CVE-2025-40040: mm/ksm: fix flag-dropping behavior in ksm_madvise

In the Linux kernel, the following vulnerability has been resolved:

mm/ksm: fix flag-dropping behavior in ksmmadvise

syzkaller discovered the following crash: (kernel BUG)

[ 44.607039] ------------[ cut here ]------------ [ 44.607422] kernel BUG at mm/userfaultfd.c:2067! [ 44.608148] Oops: invalid opcode: 0000 [#1] SMP DEBUGPAGEALLOC KASAN NOPTI [ 44.608814] CPU: 1 UID: 0 PID: 2475 Comm: reproducer Not tainted 6.16.0-rc6 #1 PREEMPT(none) [ 44.609635] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 44.610695] RIP: 0010:userfaultfdreleaseall+0x3a8/0x460

<snip other registers, drop unreliable trace>

[ 44.617726] Call Trace: [ 44.617926] <TASK> [ 44.619284] userfaultfdrelease+0xef/0x1b0 [ 44.620976] fput+0x3f9/0xb60 [ 44.621240] fputclosesync+0x110/0x210 [ 44.622222] x64sysclose+0x8f/0x120 [ 44.622530] dosyscall64+0x5b/0x2f0 [ 44.622840] entrySYSCALL64afterhwframe+0x76/0x7e [ 44.623244] RIP: 0033:0x7f365bb3f227

Kernel panics because it detects UFFD inconsistency during userfaultfdreleaseall(). Specifically, a VMA which has a valid pointer to vma->vmuserfaultfdctx, but no UFFD flags in vma->vmflags.

The inconsistency is caused in ksmmadvise(): when user calls madvise() with MADVUNMEARGEABLE on a VMA that is registered for UFFD in MINOR mode, it accidentally clears all flags stored in the upper 32 bits of vma->vmflags.

Assuming x8664 kernel build, unsigned long is 64-bit and unsigned int and int are 32-bit wide. This setup causes the following mishap during the &= ~VMMERGEABLE assignment.

VMMERGEABLE is a 32-bit constant of type unsigned int, 0x8000'0000. After ~ is applied, it becomes 0x7fff'ffff unsigned int, which is then promoted to unsigned long before the & operation. This promotion fills upper 32 bits with leading 0s, as we're doing unsigned conversion (and even for a signed conversion, this wouldn't help as the leading bit is 0). & operation thus ends up AND-ing vmflags with 0x0000'0000'7fff'ffff instead of intended 0xffff'ffff'7fff'ffff and hence accidentally clears the upper 32-bits of its value.

Fix it by changing VMMERGEABLE constant to unsigned long, using the BIT() macro.

Note: other VM flags are not affected: This only happens to the VMMERGEABLE flag, as the other VM flags are all constants of type int and after ~ operation, they end up with leading 1 and are thus converted to unsigned long with leading 1s.

Note 2: After commit 31defc3b01d9 ("userfaultfd: remove (VM)BUGON()s"), this is no longer a kernel BUG, but a WARNING at the same place:

[ 45.595973] WARNING: CPU: 1 PID: 2474 at mm/userfaultfd.c:2067

but the root-cause (flag-drop) remains the same.

[akpm@linux-foundation.org: rust bindgen wasn't able to handle BIT(), from Miguel]