CVE-2024-26670: arm64: entry: fix ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD

In the Linux kernel, the following vulnerability has been resolved:

arm64: entry: fix ARM64WORKAROUNDSPECULATIVEUNPRIVLOAD

Currently the ARM64WORKAROUNDSPECULATIVEUNPRIVLOAD workaround isn't quite right, as it is supposed to be applied after the last explicit memory access, but is immediately followed by an LDR.

The ARM64WORKAROUNDSPECULATIVEUNPRIVLOAD workaround is used to handle Cortex-A520 erratum 2966298 and Cortex-A510 erratum 3117295, which are described in:

https://developer.arm.com/documentation/SDEN2444153/0600/?lang=en https://developer.arm.com/documentation/SDEN1873361/1600/?lang=en

In both cases the workaround is described as:

| If pagetable isolation is disabled, the context switch logic in the | kernel can be updated to execute the following sequence on affected | cores before exiting to EL0, and after all explicit memory accesses: | | 1. A non-shareable TLBI to any context and/or address, including | unused contexts or addresses, such as a TLBI VALE1 Xzr. | | 2. A DSB NSH to guarantee completion of the TLBI.

The important part being that the TLBI+DSB must be placed "after all explicit memory accesses".

Unfortunately, as-implemented, the TLBI+DSB is immediately followed by an LDR, as we have:

| alternativeif ARM64WORKAROUNDSPECULATIVEUNPRIVLOAD | tlbi vale1, xzr | dsb nsh | alternativeelsenopendif | alternativeifnot ARM64UNMAPKERNELATEL0 | ldr lr, [sp, #SLR] | add sp, sp, #PTREGSSIZE // restore sp | eret | alternativeelsenopendif | | [ ... KPTI exception return path ... ]

This patch fixes this by reworking the logic to place the TLBI+DSB immediately before the ERET, after all explicit memory accesses.

The ERET is currently in a separate alternative block, and alternatives cannot be nested. To account for this, the alternative block for ARM64UNMAPKERNELATEL0 is replaced with a single alternative branch to skip the KPTI logic, with the new shape of the logic being:

| alternativeinsn "b .Lskiptrampexit\@", nop, ARM64UNMAPKERNELATEL0 | [ ... KPTI exception return path ... ] | .Lskiptrampexit\@: | | ldr lr, [sp, #SLR] | add sp, sp, #PTREGSSIZE // restore sp | | alternativeif ARM64WORKAROUNDSPECULATIVEUNPRIVLOAD | tlbi vale1, xzr | dsb nsh | alternativeelsenopendif | eret

The new structure means that the workaround is only applied when KPTI is not in use; this is fine as noted in the documented implications of the erratum:

| Pagetable isolation between EL0 and higher level ELs prevents the | issue from occurring.

... and as per the workaround description quoted above, the workaround is only necessary "If pagetable isolation is disabled".