GHSA-p92p-rxj5-7p2x: Race Condition

Summary

Multimodal content parts (imageurl, videourl, inputaudio, ...) accept a user-supplied uuid field that vLLM uses verbatim as the item's cache identity (mmhash). The two frontend/engine processor caches (P0 "sender" cache in the API process, P1 "receiver" cache in the EngineCore process) are independent LRU caches whose evictions are not synchronized, and the P1 hit path performs zero consistency validation — it silently substitutes the cached item's tensor data for whatever the current request actually contains.

A single unauthenticated client can exploit this to kill the entire engine, deterministically, with default server flags:

1. Seed: send a small image with uuid="U". Both caches now map U to an item with NA visual tokens (e.g. 4). 2. Drift: send ~70 requests, each carrying a unique uuid, a max-size image, and an over-long text prompt. Each request is rejected with 400 — but after its image has been inserted into the P0 cache (insertion happens during chat rendering, before prompt-length validation), so these requests fill P0 without ever reaching the engine/P1. U is evicted from P0 only. 3. Kill: send a large image with uuid="U". P0 misses → the image is freshly processed and the prompt is spliced with NB (e.g. 2048+) placeholder tokens; the request carries full data to the engine, but P1 still has U → the engine discards the fresh data and substitutes the stale NA-row tensor. The scheduler plans for NB encoder slots while the worker merges NA embeddings into NB placeholders → ValueError: Attempted to assign 4 = 4 multimodal tokens to 2048 placeholders at vllm/modelexecutor/models/utils.py:711 → uncaught in the EngineCore busy loop → sendenginedead() → the EngineCore process dies, every API server reports EngineDeadError and shuts down. /health never recovers; the deployment needs a full restart.

Verified end-to-end on the official vllm/vllm-openai:v0.29.0 image with Qwen2.5-VL-3B-Instruct, default configuration (no flags beyond --served-model-name/--gpu-memory-utilization): after 70 flood requests the kill request returned 500 EngineCore encountered an issue and /health went dead permanently. The vulnerable code is byte-identical on main (post-0.29.0).

Details

1. The client-controlled uuid becomes the cache hash verbatim. vllm/entrypoints/chatutils.py:1677 / :1884-1886 read part["uuid"] per content part and pass it through as mmuuids. In vllm/multimodal/processing/inputs.py:62-63, when no hfprocessormmkwargs are present (the default for chat requests), the uuid string is the item hash — no content binding at all:

python vllm/multimodal/processing/inputs.py:62-63 else: hashes.append(uuiditem) # client string used verbatim as mmhash

2. P0 insertion happens before request validation. In the serving path, renderchatrequest (vllm/entrypoints/openai/chatcompletion/serving.py:275) runs full mm processing — mmprocessor.apply (vllm/renderers/base.py:759) → cachedapplyhfprocessor → mergemmkwargs → cache.getandupdateitem (vllm/multimodal/processing/processor.py:1338), which inserts the processed item into the P0 sender cache (default mmprocessorcachetype="lru", 4 GiB). Only afterwards does getmaxtokens (vllm/entrypoints/serve/utils/apiutils.py:183-186) reject over-long prompts with 400. A request can therefore write to P0 while never reaching the engine — the primitive that desynchronizes P0 from P1. (Public issue #43941 documented this same asymmetry for the opposite drift direction, which was later handled via MultiModalCacheMissError; the P0-miss/P1-hit direction below was never addressed.)

3. P1 substitutes cached data on hash hit with no consistency check. In the EngineCore input thread, preprocessaddrequest (vllm/v1/engine/core.py:997-1000) calls mmreceivercache.getandupdatefeatures, which only replaces feature.data (vllm/multimodal/cache.py:673-690) and keeps the request's own mmposition (placeholder layout). On a cache hit the receiver cache returns the stale item wholesale:

python vllm/multimodal/cache.py:739-740 (MultiModalReceiverCache.getandupdateitem) if (cacheditem := self.cache.get(mmhash)) is not None: return cacheditem # fresh data silently discarded; no size/modality check

Only the opposite drift (P0 hit / P1 miss, data=None) is handled, via the retryable MultiModalCacheMissError. Nothing validates that the cached item's modality, shape, or token count matches the current request's placeholders.

4. The mismatch is fatal to the engine. The scheduler plans encoder work from the request's (fresh) placeholder count NB, while the worker gathers embeddings from the stale NA-row tensor (encoder-cache hit under the same forged hash, or re-encoding the stale data — both yield NA rows). mergemultimodalembeddings then raises:

python vllm/modelexecutor/models/utils.py:703-714 inputsembeds[ismultimodal] = mmembedsflat.to(dtype=inputdtype) # RuntimeError ... raise ValueError( f"Attempted to assign {expr} = {numactualtokens} " f"multimodal tokens to {numexpectedtokens} placeholders") from e

This exception is raised inside the worker's input preparation with no per-request containment, propagates to the EngineCore busy loop's top-level handler (vllm/v1/engine/core.py:1372), and triggers sendenginedead(). The API server's outputhandler receives EngineDeadError (vllm/v1/engine/asyncllm.py:819) and the whole process tree shuts down. (Issue #55546 independently reported this sink — "one bad multimodal item kills the entire V1 engine tree" — from an organically occurring stale cache entry; this report provides the missing deterministic, remote, single-attacker trigger.)

Cost of the attack. With the default mmprocessorcachegb=4, one max-size Qwen2.5-VL image (12845056 pixels) caches as ~77 MB, so ~56 flood requests suffice; we used 70. Flood requests are CPU-only image decodes (~0.3 s each here) and can be issued sequentially or concurrently; the total wall-clock cost of the kill was under one minute. Models with larger maxpixels fill the cache in even fewer requests. Note that issuing the flood with successful requests does not work — they would populate P1 as well; rejection after P0 insertion is essential, and the over-long-prompt 400 is one always-available way to get it (no size limits depend on non-default flags).

Impact

Any unauthenticated client that can reach a multimodal vLLM endpoint (/v1/chat/completions, /v1/responses, etc. — all accept content-part uuid) can deterministically crash the entire serving instance: the EngineCore process dies, every in-flight request fails, and the API server exits; the deployment stays down until restarted. In data-parallel or multi-API-server deployments the blast radius extends to the whole engine group (cf. the DP8 outage in #55546). The attack requires no special privileges, no non-default server flags, no race conditions, and costs under a minute of sequential requests at default cache sizes.

This is distinct from the documented "Multimodal Media UUID Security" guidance in docs/usage/security.md, which covers only integrity/confidentiality consequences of uuid collisions (a caller silently receiving another caller's media). The guidance does not mention availability, and following it does not help: the attacker here deliberately collides with their own uuid, and the fatal element is the missing consistency check on the P1 hit path plus the unsynchronized eviction of the two caches.

Suggested remediation (any one of these breaks the kill chain):

1. On a P1 receiver-cache hit, validate the cached item against the request's mmposition (modality and expected embedding count); on mismatch, treat it as a miss (MultiModalCacheMissError) or reject the request — never substitute silently. 2. Make the feature/placeholder count mismatch in mergemultimodalembeddings a per-request error instead of a fatal engine exception (also requested by #55546). 3. Bind server-side cache identity to content even when a uuid is provided (e.g. hash(content) + uuid), eliminating forged-hash cache hits entirely. 4. Synchronize P0/P1 evictions (the engine already pushes invalidate to P0 on cache-miss; eviction-direction sync would close the remaining drift).

A public fix is available here: https://github.com/vllm-project/vllm/pull/57833