CVE-2026-53373: mm/vma: do not try to unmap a VMA if mmap_prepare() invoked from mmap()

In the Linux kernel, the following vulnerability has been resolved:

mm/vma: do not try to unmap a VMA if mmapprepare() invoked from mmap()

The mmapprepare hook functionality includes the ability to invoke mmapprepare() from the mmap() hook of existing 'stacked' drivers, that is ones which are capable of calling the mmap hooks of other drivers/file systems (e.g. overlayfs, shm).

As part of the mmapprepare action functionality, we deal with errors by unmapping the VMA should one arise. This works in the usual mmapprepare case, as we invoke this action at the last moment, when the VMA is established in the maple tree.

However, the mmap() hook passes a not-fully-established VMA pointer to the caller (which is the motivation behind the mmapprepare() work), which is detached.

So attempting to unmap a VMA in this state will be problematic, with the most obvious symptom being a warning in vmamarkdetached(), because the VMA is already detached.

It's also unncessary - the mmap() handler will clean up the VMA on error.

So to fix this issue, this patch propagates whether or not an mmap action is being completed via the compatibility layer or directly.

If the former, then we do not attempt VMA cleanup, if the latter, then we do.

This patch also updates the userland VMA tests to reflect the change.