CVE-2026-74632: mm/huge_memory: fix huge_zero_pfn race

In the Linux kernel, the following vulnerability has been resolved:

mm/hugememory: fix hugezeropfn race

Patch series "mm/hugememory: fix hugezeropfn race", v2.

There is a subtle race in the reference-counted hugezerofolio implementation.

The fast path atomic logic fails to account for the fact that the shrinker (which drops the final hugezerorefcount pin) can overwrite hugezeropfn with the ~0UL sentinel value in shrinkhugezerofolioscan() after a racing gethugezerofolio() installed a valid value there.

This results in hugezerofolio being correctly set but hugezeropfn being set incorrectly and thus ishugezeropfn() and consequently ishugezeropmd() will misidentify the huge zero folio as being an ordinary THP folio.

This can result in the huge zero folio being split and otherwise treated incorrectly.

The solution to this is very subtle as there is an atomic fast path, and thus ordering in weakly ordered architectures has to be treated very carefully.

The first commit fixes the issue by introducing a spinlock around hugezero[pfn, folio, refcount] write, with careful consideration paid to load/store ordering in the fast path. It is placed first and kept as small as possible so that it can be backported on its own.

The second commit is a pure cleanup which reworks the CONFIGPERSISTENTHUGEZEROFOLIO logic to better separate the persistent logic from the dynamically allocated one.

This patch (of 2):

If !CONFIGPERSISTENTHUGEZEROFOLIO, the hugezerofolio is refcounted by hugezerorefcount and returned by mmgethugezerofolio().

When the caller is done with the huge zero page, its reference count is decremented. Only a shrinker can set the reference count to zero.

A race can unfortunately occur between a shrinker decrementing the reference count to zero and a concurrent page fault.

This is because shrinkhugezerofolioscan() might, if very unlucky, be preempted between setting hugezerorefcount to zero and writing an invalid value.

During this time gethugezerofolio() could write to hugezeropfn before shrinkhugezerofolioscan() resumes.

In this event the huge zero folio will be persistently misidentified causing the THP code path to be entered inappropriately for the huge zero folio:

CPU 0 CPU 1 =======================================|================================= shrinkhugezerofolioscan() | atomiccmpxchg() sets refcount to 0 | xchg() sets hugezerofolio to NULL | gethugezerofolio() | | atomicincnotzero() -> zero preempted for a long time | Allocate new huge zero folio | | Write valid hugezerofolio v | Write valid hugezeropfn Overwrite hugezeropfn with ~0UL <--- Invalid overwrite!

This results in ishugezeropfn() and ishugezeropmd() incorrectly returning false for a huge zero page which could result in issues like the huge zero folio being incorrectly split.

Note that the issue is with hugezeropfn not hugezerofolio, as gethugezerofolio() uses cmpxchg() gated on hugezerofolio being NULL with a retry loop and shrinkhugezerofolioscan() uses xchg() to set hugezerofolio.

Fix the issue by introducing a spinlock, hugezerolock, to prevent concurrent write of hugezerofolio, hugezeropfn and hugezerorefcount.

There needs to be significant care taken here to ensure correctness:

The fast path in gethugezerofolio() uses atomicincnotzero(), which is outside of the critical section, and means huge zero allocation is gated on zero hugezerorefcount.

The fast path doesn't use hugezerolock, so the critical section is irrelevant to it.

So invariants are required - hugezerorefcount MUST:

Only be set in the hugezerolock critical section to ensure serialisation of hugezeropfn, hugezerofolio and ---truncated---