CVE-2021-47639: KVM: x86/mmu: Zap _all_ roots when unmapping gfn range in TDP MMU

In the Linux kernel, the following vulnerability has been resolved:

KVM: x86/mmu: Zap all roots when unmapping gfn range in TDP MMU

Zap both valid and invalid roots when zapping/unmapping a gfn range, as KVM must ensure it holds no references to the freed page after returning from the unmap operation. Most notably, the TDP MMU doesn't zap invalid roots in mmunotifier callbacks. This leads to use-after-free and other issues if the mmunotifier runs to completion while an invalid root zapper yields as KVM fails to honor the requirement that there must be no references to the page after the mmunotifier returns.

The bug is most easily reproduced by hacking KVM to cause a collision between setnxhugepages() and kvmmmunotifierrelease(), but the bug exists between kvmmmunotifierinvalidaterangestart() and memslot updates as well. Invalidating a root ensures pages aren't accessible by the guest, and KVM won't read or write page data itself, but KVM will trigger e.g. kvmsetpfndirty() when zapping SPTEs, and thus completing a zap of an invalid root after the mmunotifier returns is fatal.

WARNING: CPU: 24 PID: 1496 at arch/x86/kvm/../../../virt/kvm/kvmmain.c:173 [kvm] RIP: 0010:kvmiszonedevicepfn+0x96/0xa0 [kvm] Call Trace: <TASK> kvmsetpfndirty+0xa8/0xe0 [kvm] handlechangedspte+0x2ab/0x5e0 [kvm] handlechangedspte+0x2ab/0x5e0 [kvm] handlechangedspte+0x2ab/0x5e0 [kvm] zapgfnrange+0x1f3/0x310 [kvm] kvmtdpmmuzapinvalidatedroots+0x50/0x90 [kvm] kvmmmuzapallfast+0x177/0x1a0 [kvm] setnxhugepages+0xb4/0x190 [kvm] paramattrstore+0x70/0x100 moduleattrstore+0x19/0x30 kernfsfopwriteiter+0x119/0x1b0 newsyncwrite+0x11c/0x1b0 vfswrite+0x1cc/0x270 ksyswrite+0x5f/0xe0 dosyscall64+0x38/0xc0 entrySYSCALL64afterhwframe+0x44/0xae </TASK>