Summary

The Phi-4 multimodal audio encoder (phi4mmaudio.py / phi4mmutils.py) runs its conformer attention in eager mode by default: the official HF config of microsoft/Phi-4-multimodal-instruct does not set useptscaleddotproductattention, so the default False applies (phi4mmaudio.py:166, :876; phi4mmutils.py:1669). The eager path fully materializes the attention probability matrix of shape [nchunks, 16, 500, 500] per layer — and then stores it on the module: self.attn = attn (phi4mmutils.py:1802, a torch.jit.Attribute declared at :1693), keeping a live reference for every one of the 24 encoder layers.

The only gates on audio length are the server-side limits VLLMMAXAUDIOCLIPFILESIZEMB=25 and VLLMMAXAUDIODECODEDURATIONS=600 (envs.py:83-84) — i.e. a 10-minute clip is a legal input by default. A 600 s clip becomes ~60,000 mel frames, which the encoder unfolds into 500-frame chunks (phi4mmaudio.py:996, maxseqlen = 500), giving nchunks ≈ 120. Each layer therefore materializes a 120×16×500×500 matrix (~1–2 GiB depending on dtype), and the module attribute keeps all completed layers' matrices alive, so GPU memory climbs monotonically within a single request until an allocation fails.

Verified end-to-end on the official vllm/vllm-openai:v0.31.0 image: one 19.2 MB silent 600 s WAV → worker torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.29 GiB ... 501.75 MiB free ... 22.09 GiB allocated by PyTorch inside getaudiofeatures (phi4mmaudio.py:1279 → :1332 → phi4mmutils.py:1386) → EngineCore encountered a fatal error → API EngineDeadError → whole server down; /health refuses connections. Total time from request to dead service: ~6 seconds.

A secondary effect: because self.attn persists on the module after the request ends, the matrices of the last audio request stay resident indefinitely (observed +50 MiB pinned after a single 5 s control request); they are only overwritten by the next audio request, so the resident footprint equals the largest audio processed since startup.

Details

1. Root cause. MultiHeadedAttention.forwardattention in eager mode computes attn = maskedsoftmax(scores, mask) and unconditionally executes self.attn = attn (phi4mmutils.py:1795-1802). The attribute is a leftover inference/debug hook (used by torch.jit.Attribute(None, Tensor | None) at :1693); in serving it serves no purpose but pins the full [batch·chunks, heads, T, T] tensor per layer past both the layer's forward and the request's lifetime. The SDP attention branch (useptscaleddotproductattention=True) never materializes this matrix — but the HF config does not enable it and the code default is False.

2. Why one request suffices. The conformer chunks long audio to maxseqlen = 500 frames (phi4mmaudio.py:996). Duration is bounded only by VLLMMAXAUDIODECODEDURATIONS=600, so nchunks reaches ~120 for a maximal clip. With 16 heads and fp32 scores, one layer's attention tensor is ~1.8 GiB (the observed failing allocation was 2.29 GiB, consistent with the scores+bias materialization). The encoder has numblocks=24, attentionheads=16, attentiondim=1024, timereduction=8 (HF config). Weights plus a handful of retained per-layer matrices exhaust the ~2.3 GiB headroom that --gpu-memory-utilization 0.9 leaves on a 24 GB card — mid-forward, on the first request.

4. Deployment reality. --trust-remote-code is the documented, standard way to serve this model (the HF model card instructs it), so the finding applies to default deployments. The crash is deterministic, not load-dependent: scheduler stats at death show a single running request and near-zero KV cache usage.

Impact

Any unauthenticated client that can reach the chat/completions API of a default deployment serving Phi-4-multimodal-instruct (or any model reusing this audio tower) can kill the entire engine with one request carrying a perfectly legal 10-minute audio clip (silent noise suffices — no crafted content needed). Full denial of service, requiring a process restart. On GPUs with more headroom (e.g. 80 GB at default utilization) a single request may survive, but the eager path still materializes GiB-scale per-layer matrices and retains the last request's tensors indefinitely, so a few sequential max-length requests — or normal traffic following one — achieve the same result.

Relation to prior art: the only public Phi-4-mm advisory we found, GHSA-vc6m-hm49-g9qg, concerns a CPU-side quadratic-cost path — different subsystem, different mechanism. GitHub search for "phi4mm audio OOM/memory" (31 results, 2026-10-08) shows no report of this attention-retention issue; main (2026-10-08) is unchanged.

Suggested remediation (any one breaks the chain; 1+2 recommended):

1. Remove self.attn = attn from the serving path (or gate it behind an explicit debug/outputattentions flag) so attention matrices are freed when the layer returns. 2. Default useptscaleddotproductattention to True (flash/SDPA kernels never materialize the [B, H, T, T] matrix); keep eager only as an opt-in. 3. Additionally bound the encoder input (frames or duration) for this model so the eager path cannot reach GiB-scale single-layer allocations even if enabled.

This advisory was fixed in MR60614.

A security flaw has been discovered in vllm-project vLLM up to 0.31.0. This impacts the function gettokenbincountsandmask of the file vllm/modelexecutor/layers/utils.py of the component Penalty Handler. Performing a manipulation results in denial of service. Remote exploitation of the attack is possible. The exploit has been released to the public and may be used for attacks. The project was informed of the problem early through an issue report but has not responded yet.