CVE-2024-56559: mm/vmalloc: combine all TLB flush operations of KASAN shadow virtual address into one operation
In the Linux kernel, the following vulnerability has been resolved:
mm/vmalloc: combine all TLB flush operations of KASAN shadow virtual address into one operation
When compiling kernel source 'make -j $(nproc)' with the up-and-running KASAN-enabled kernel on a 256-core machine, the following soft lockup is shown:
watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [kworker/28:1:1760] CPU: 28 PID: 1760 Comm: kworker/28:1 Kdump: loaded Not tainted 6.10.0-rc5 #95 Workqueue: events drainvmapareawork RIP: 0010:smpcallfunctionmanycond+0x1d8/0xbb0 Code: 38 c8 7c 08 84 c9 0f 85 49 08 00 00 8b 45 08 a8 01 74 2e 48 89 f1 49 89 f7 48 c1 e9 03 41 83 e7 07 4c 01 e9 41 83 c7 03 f3 90 <0f> b6 01 41 38 c7 7c 08 84 c0 0f 85 d4 06 00 00 8b 45 08 a8 01 75 RSP: 0018:ffffc9000cb3fb60 EFLAGS: 00000202 RAX: 0000000000000011 RBX: ffff8883bc4469c0 RCX: ffffed10776e9949 RDX: 0000000000000002 RSI: ffff8883bb74ca48 RDI: ffffffff8434dc50 RBP: ffff8883bb74ca40 R08: ffff888103585dc0 R09: ffff8884533a1800 R10: 0000000000000004 R11: ffffffffffffffff R12: ffffed1077888d39 R13: dffffc0000000000 R14: ffffed1077888d38 R15: 0000000000000003 FS: 0000000000000000(0000) GS:ffff8883bc400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00005577b5c8d158 CR3: 0000000004850000 CR4: 0000000000350ef0 Call Trace: <IRQ> ? watchdogtimerfn+0x2cd/0x390 ? pfxwatchdogtimerfn+0x10/0x10 ? hrtimerrunqueues+0x300/0x6d0 ? schedclockcpu+0x69/0x4e0 ? pfxhrtimerrunqueues+0x10/0x10 ? srsoreturnthunk+0x5/0x5f ? ktimegetupdateoffsetsnow+0x7f/0x2a0 ? srsoreturnthunk+0x5/0x5f ? srsoreturnthunk+0x5/0x5f ? hrtimerinterrupt+0x2ca/0x760 ? sysvecapictimerinterrupt+0x8c/0x2b0 ? sysvecapictimerinterrupt+0x6a/0x90 </IRQ> <TASK> ? asmsysvecapictimerinterrupt+0x16/0x20 ? smpcallfunctionmanycond+0x1d8/0xbb0 ? pfxdokernelrangeflush+0x10/0x10 oneachcpucondmask+0x20/0x40 flushtlbkernelrange+0x19b/0x250 ? srsoreturnthunk+0x5/0x5f ? kasanreleasevmalloc+0xa7/0xc0 purgevmapnode+0x357/0x820 ? pfxpurgevmapnode+0x10/0x10 purgevmaparealazy+0x5b8/0xa10 drainvmapareawork+0x21/0x30 processonework+0x661/0x10b0 workerthread+0x844/0x10e0 ? srsoreturnthunk+0x5/0x5f ? kthreadparkme+0x82/0x140 ? pfxworkerthread+0x10/0x10 kthread+0x2a5/0x370 ? pfxkthread+0x10/0x10 retfromfork+0x30/0x70 ? pfxkthread+0x10/0x10 retfromforkasm+0x1a/0x30 </TASK>
Debugging Analysis:
1. The following ftrace log shows that the lockup CPU spends too much time iterating vmapnodes and flushing TLB when purging vmarea structures. (Some info is trimmed).
kworker: funcgraphentry: | drainvmapareawork() { kworker: funcgraphentry: | mutexlock() { kworker: funcgraphentry: 1.092 us | condresched(); kworker: funcgraphexit: 3.306 us | } ... ... kworker: funcgraphentry: | flushtlbkernelrange() { ... ... kworker: funcgraphexit: # 7533.649 us | } ... ... kworker: funcgraphentry: 2.344 us | mutexunlock(); kworker: funcgraphexit: $ 23871554 us | }
The drainvmapareawork() spends over 23 seconds.
There are 2805 flushtlbkernelrange() calls in the ftrace log. One is called in purgevmaparealazy(). Others are called by purgevmapnode->kasanreleasevmalloc. purgevmapnode() iteratively releases kasan vmalloc allocations and flushes TLB for each vmaparea. - [Rough calculation] Each flushtlbkernelrange() runs about 7.5ms. -- 2804 7.5ms = 21.03 seconds. -- That's why a soft lock is triggered.
2. Extending the soft lockup time can work around the issue (For example, # echo ---truncated---
Other sources
This CVE was automatically created from a reference found in an email or other text. If you are reading this, then this CVE entry is probably erroneous, since this text should be replaced by the official CVE description automatically.
— Launchpad
Affected Software
Remediation
Event History
Frequently Asked Questions
What is the severity of CVE-2024-56559?
CVE-2024-56559 is rated as a moderate severity vulnerability in the Linux kernel.
How do I fix CVE-2024-56559?
To fix CVE-2024-56559, update your Linux kernel to one of the patched versions such as 5.10.223-1 or 6.12.20-1.
What type of vulnerability is CVE-2024-56559?
CVE-2024-56559 is an issue related to TLB flush operations in the memory management subsystem of the Linux kernel.
Which versions of the Linux kernel are affected by CVE-2024-56559?
CVE-2024-56559 affects multiple versions of the Linux kernel prior to the release of the patched versions specified.
What is KASAN in relation to CVE-2024-56559?
KASAN, or Kernel Address Sanitizer, is a feature in the Linux kernel that helps detect memory errors, and CVE-2024-56559 involves its operation.