An issue was discovered in Xen XAPI before 2020-12-15. Certain xenstore keys provide feedback from the guest, and are therefore watched by toolstack. Specifically, keys are watched by xenopsd, and data are forwarded via RPC through message-switch to xapi. The watching logic in xenopsd sends one RPC update containing all data, any time any single xenstore key is updated, and therefore has O(N^2) time complexity. Furthermore, message-switch retains recent (currently 128) RPC messages for diagnostic purposes, yielding O(MN) space complexity. The quantity of memory a single guest can monopolise is bounded by xenstored quota, but the quota is fairly large. It is believed to be in excess of 1G per malicious guest. In practice, this manifests as a host denial of service, either through message-switch thrashing against swap, or OOMing entirely, depending on dom0's configuration. (There are no quotas in xenopsd to limit the quantity of keys that result in RPC traffic.) A buggy or malicious guest can cause unreasonable memory usage in dom0, resulting in a host denial of service. All versions of XAPI are vulnerable. Systems that are not using the XAPI toolstack are not vulnerable.
Addressing certain issues, in particular related to operations which may take excessively long and therefore would need preemption, has turned out overly costly. Since alternatives (HVM/PVH: HAP, PV: shim) are commonly available, the decision was to deprecate the functionality, while still retaining it for people to use at their own (security) risk. Memory-wise small enough guests may still be okay to run.
XAPI open file limit DoS It is possible for an unauthenticated client on the network to cause XAPI to hit its file-descriptor limit. This causes XAPI to be unable to accept new requests for other (trusted) clients, and blocks XAPI from carrying out any tasks that require the opening of file descriptors.
The ssmi emulation in Xen 5.1 on IA64 architectures allows attackers to cause a denial of service (dom0 panic) via certain traffic, as demonstrated using an FTP stress test tool.