CVE-2025-68310: s390/pci: Avoid deadlock between PCI error recovery and mlx5 crdump
In the Linux kernel, the following vulnerability has been resolved:
s390/pci: Avoid deadlock between PCI error recovery and mlx5 crdump
Do not block PCI config accesses through pcicfgaccesslock() when executing the s390 variant of PCI error recovery: Acquire just devicelock() instead of pcidevlock() as powerpc's EEH and generig PCI AER processing do.
During error recovery testing a pair of tasks was reported to be hung:
mlx5core 0000:00:00.1: mlx5healthtryrecover:338:(pid 5553): health recovery flow aborted, PCI reads still not working INFO: task kmcheck:72 blocked for more than 122 seconds. Not tainted 5.14.0-570.12.1.bringup7.el9.s390x #1 "echo 0 > /proc/sys/kernel/hungtasktimeoutsecs" disables this message. task:kmcheck state:D stack:0 pid:72 tgid:72 ppid:2 flags:0x00000000 Call Trace: [<000000065256f030>] schedule+0x2a0/0x590 [<000000065256f356>] schedule+0x36/0xe0 [<000000065256f572>] schedulepreemptdisabled+0x22/0x30 [<0000000652570a94>] mutexlock.constprop.0+0x484/0x8a8 [<000003ff800673a4>] mlx5unloadone+0x34/0x58 [mlx5core] [<000003ff8006745c>] mlx5pcierrdetected+0x94/0x140 [mlx5core] [<0000000652556c5a>] zpcieventattempterrorrecovery+0xf2/0x398 [<0000000651b9184a>] zpcieventerror+0x23a/0x2c0 INFO: task kworker/u1664:6:1514 blocked for more than 122 seconds. Not tainted 5.14.0-570.12.1.bringup7.el9.s390x #1 "echo 0 > /proc/sys/kernel/hungtasktimeoutsecs" disables this message. task:kworker/u1664:6 state:D stack:0 pid:1514 tgid:1514 ppid:2 flags:0x00000000 Workqueue: mlx5health0000:00:00.0 mlx5fwfatalreportererrwork [mlx5core] Call Trace: [<000000065256f030>] schedule+0x2a0/0x590 [<000000065256f356>] schedule+0x36/0xe0 [<0000000652172e28>] pciwaitcfg+0x80/0xe8 [<0000000652172f94>] pcicfgaccesslock+0x74/0x88 [<000003ff800916b6>] mlx5vscgwlock+0x36/0x178 [mlx5core] [<000003ff80098824>] mlx5crdumpcollect+0x34/0x1c8 [mlx5core] [<000003ff80074b62>] mlx5fwfatalreporterdump+0x6a/0xe8 [mlx5core] [<0000000652512242>] devlinkhealthdodump.part.0+0x82/0x168 [<0000000652513212>] devlinkhealthreport+0x19a/0x230 [<000003ff80075a12>] mlx5fwfatalreportererrwork+0xba/0x1b0 [mlx5core]
No kernel log of the exact same error with an upstream kernel is available - but the very same deadlock situation can be constructed there, too:
- task: kmcheck mlx5unloadone() tries to acquire devlink lock while the PCI error recovery code has set pdev->blockcfgaccess by way of pcicfgaccesslock() - task: kworker mlx5crdumpcollect() tries to set blockcfgaccess through pcicfgaccesslock() while devlinkhealthreport() had acquired the devlink lock.
A similar deadlock situation can be reproduced by requesting a crdump with > devlink health dump show pci/<BDF> reporter fwfatal
while PCI error recovery is executed on the same <BDF> physical function by mlx5core's pcierrorhandlers. On s390 this can be injected with > zpcictl --reset-fw <BDF>
Tests with this patch failed to reproduce that second deadlock situation, the devlink command is rejected with "kernel answers: Permission denied" - and we get a kernel log message of:
mlx5core 1ed0:00:00.1: mlx5crdumpcollect:50:(pid 254382): crdump: failed to lock vsc gw err -5
because the config read of VSCSEMAPHORE is rejected by the underlying hardware.
Two prior attempts to address this issue have been discussed and ultimately rejected [see link], with the primary argument that s390's implementation of PCI error recovery is imposing restrictions that neither powerpc's EEH nor PCI AER handling need. Tests show that PCI error recovery on s390 is running to completion even without blocking access to PCI config space.
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
ubuntu/linux-aws-6.8to a version that resolves this vulnerability.Fixed in 6.8.0-1050.53~22.04.1 - Upgrade
Upgrade
ubuntu/linux-azure-6.8to a version that resolves this vulnerability.Fixed in 6.8.0-1051.57~22.04.1 - Upgrade
Upgrade
ubuntu/linux-azure-fde-6.8to a version that resolves this vulnerability.Fixed in 6.8.0-1058.65~22.04.1 - Upgrade
Upgrade
ubuntu/linux-gcp-6.8to a version that resolves this vulnerability.Fixed in 6.8.0-1052.55~22.04.1 - Upgrade
Upgrade
ubuntu/linux-hwe-6.8to a version that resolves this vulnerability.Fixed in 6.8.0-106.106~22.04.1 - Upgrade
Upgrade
ubuntu/linux-ibm-6.8to a version that resolves this vulnerability.Fixed in 6.8.0-1049.49~22.04.1 - Upgrade
Upgrade
ubuntu/linux-lowlatency-hwe-6.8to a version that resolves this vulnerability.Fixed in 6.8.0-106.106.1~22.04.1 - Upgrade
Upgrade
ubuntu/linux-nvidia-6.8to a version that resolves this vulnerability.Fixed in 6.8.0-1049.52~22.04.1 - Upgrade
Upgrade
ubuntu/linux-oracle-6.8to a version that resolves this vulnerability.Fixed in 6.8.0-1047.48~22.04.1 - Upgrade
Upgrade
ubuntu/linux-riscv-6.8to a version that resolves this vulnerability.Fixed in 6.8.0-106.106~22.04.1 - Upgrade
Upgrade
ubuntu/linux-realtime-6.8to a version that resolves this vulnerability.Fixed in 6.8.1-1045.46~22.04.1 - Upgrade
Upgrade
ubuntu/linuxto a version that resolves this vulnerability.Fixed in 6.8.0-106.106 - Upgrade
Upgrade
ubuntu/linux-awsto a version that resolves this vulnerability.Fixed in 6.8.0-1050.53 - Upgrade
Upgrade
ubuntu/linux-aws-6.17to a version that resolves this vulnerability.Fixed in 6.17.0-1007.7~24.04.1 - Upgrade
Upgrade
ubuntu/linux-azureto a version that resolves this vulnerability.Fixed in 6.8.0-1051.57 - Upgrade
Upgrade
ubuntu/linux-azure-fdeto a version that resolves this vulnerability.Fixed in 6.8.0-1058.65 - Upgrade
Upgrade
ubuntu/linux-azure-nvidiato a version that resolves this vulnerability.Fixed in 6.8.0-1039.42 - Upgrade
Upgrade
ubuntu/linux-gcpto a version that resolves this vulnerability.Fixed in 6.8.0-1052.55 - Upgrade
Upgrade
ubuntu/linux-gcp-6.17to a version that resolves this vulnerability.Fixed in 6.17.0-1008.8~24.04.1 - Upgrade
Upgrade
ubuntu/linux-gketo a version that resolves this vulnerability.Fixed in 6.8.0-1048.53 - Upgrade
Upgrade
ubuntu/linux-gkeopto a version that resolves this vulnerability.Fixed in 6.8.0-1035.38 - Upgrade
Upgrade
ubuntu/linux-ibmto a version that resolves this vulnerability.Fixed in 6.8.0-1049.49 - Upgrade
Upgrade
ubuntu/linux-lowlatencyto a version that resolves this vulnerability.Fixed in 6.8.0-106.106.1 - Upgrade
Upgrade
ubuntu/linux-nvidiato a version that resolves this vulnerability.Fixed in 6.8.0-1049.52 - Upgrade
Upgrade
ubuntu/linux-nvidia-6.17to a version that resolves this vulnerability.Fixed in 6.17.0-1008.8 - Upgrade
Upgrade
ubuntu/linux-nvidia-lowlatencyto a version that resolves this vulnerability.Fixed in 6.8.0-1049.52.1 - Upgrade
Upgrade
ubuntu/linux-nvidia-tegrato a version that resolves this vulnerability.Fixed in 6.8.0-1020.20 - Upgrade
Upgrade
ubuntu/linux-oem-6.17to a version that resolves this vulnerability.Fixed in 6.17.0-1011.11 - Upgrade
Upgrade
ubuntu/linux-oracleto a version that resolves this vulnerability.Fixed in 6.8.0-1047.48 - Upgrade
Upgrade
ubuntu/linux-oracle-6.17to a version that resolves this vulnerability.Fixed in 6.17.0-1007.7~24.04.1 - Upgrade
Upgrade
ubuntu/linux-raspito a version that resolves this vulnerability.Fixed in 6.8.0-1050.54 - Upgrade
Upgrade
ubuntu/linux-xilinxto a version that resolves this vulnerability.Fixed in 6.8.0-1028.29 - Upgrade
Upgrade
ubuntu/linux-bluefieldto a version that resolves this vulnerability.Fixed in 6.8.0-1017.21 - Upgrade
Upgrade
ubuntu/linux-raspi-realtimeto a version that resolves this vulnerability.Fixed in 6.8.0-2040.41 - Upgrade
Upgrade
ubuntu/linux-realtimeto a version that resolves this vulnerability.Fixed in 6.8.1-1045.46 - Upgrade
Upgrade
ubuntu/linux-realtime-6.17to a version that resolves this vulnerability.Fixed in 6.17.0-1006.7~24.04.1 - Upgrade
Upgrade
ubuntu/linux-aws-fipsto a version that resolves this vulnerability.Fixed in 6.8.0-1050.53+fips1 - Upgrade
Upgrade
ubuntu/linux-azure-fipsto a version that resolves this vulnerability.Fixed in 6.8.0-1052.58+fips1 - Upgrade
Upgrade
ubuntu/linux-fipsto a version that resolves this vulnerability.Fixed in 6.8.0-106.106+fips1 - Upgrade
Upgrade
ubuntu/linux-gcp-fipsto a version that resolves this vulnerability.Fixed in 6.8.0-1052.55+fips1 - Upgrade
Upgrade
ubuntu/linuxto a version that resolves this vulnerability.Fixed in 6.17.0-14.14 - Upgrade
Upgrade
ubuntu/linux-awsto a version that resolves this vulnerability.Fixed in 6.17.0-1007.7 - Upgrade
Upgrade
ubuntu/linux-azureto a version that resolves this vulnerability.Fixed in 6.17.0-1008.8 - Upgrade
Upgrade
ubuntu/linux-gcpto a version that resolves this vulnerability.Fixed in 6.17.0-1007.7 - Upgrade
Upgrade
ubuntu/linux-oracleto a version that resolves this vulnerability.Fixed in 6.17.0-1007.7 - Upgrade
Upgrade
ubuntu/linux-raspito a version that resolves this vulnerability.Fixed in 6.17.0-1008.8 - Upgrade
Upgrade
ubuntu/linux-realtimeto a version that resolves this vulnerability.Fixed in 6.17.0-1006.7 - Upgrade
Upgrade
ubuntu/linux-riscvto a version that resolves this vulnerability.Fixed in 6.17.0-14.14.1 - Compensating control
Modify the s390 PCI error-recovery implementation so it acquires device_lock() instead of pci_dev_lock() and does not block PCI configuration accesses through pci_cfg_access_lock() during recovery.
Event History
Frequently Asked Questions
What is the severity of CVE-2025-68310?
CVE-2025-68310 is classified as a medium severity vulnerability in the Linux kernel.
How do I fix CVE-2025-68310?
To fix CVE-2025-68310, ensure that your Linux kernel is updated to the latest stable version where this vulnerability has been patched.
What systems are affected by CVE-2025-68310?
CVE-2025-68310 affects systems running the Linux kernel that utilize PCI error recovery functionality.
What is the impact of CVE-2025-68310?
The impact of CVE-2025-68310 includes potential deadlock situations during PCI error recovery processes.
Who discovered CVE-2025-68310?
CVE-2025-68310 was reported by contributors to the Linux kernel development community.