Summary

The extracthiddenstates speculative decoding proposer in vLLM returns a tensor with an incorrect shape after the first decode step, causing a RuntimeError that crashes the EngineCore process. The crash is triggered when any request in the batch uses sampling penalty parameters (repetitionpenalty, frequencypenalty, or presencepenalty).

A single request with a penalty parameter (e.g., "repetitionpenalty": 1.1) is sufficient to crash the server. The crash is deterministic and immediate — no concurrency, race condition, or special workload is required.

Details

In vLLM v0.17.0, the extracthiddenstates proposer's propose() method returned sampledtokenids.unsqueeze(-1), producing a tensor of shape (batchsize, 1).

In PR #37013 (first released in v0.18.0), the KV connector interface was refactored out of propose(). The return type changed from tuple[Tensor, KVConnectorOutput | None] to Tensor, and the .unsqueeze(-1) call was removed along with the KV connector output:

python Before (v0.17.0): return sampledtokenids.unsqueeze(-1), kvconnectoroutput # shape (batchsize, 1)

After (v0.18.0+): return sampledtokenids # shape (batchsize, 2) after first decode step

The refactor missed that sampledtokenids changed semantics between the first and subsequent decode steps. After the first decode step, the rejection sampler allocates its output as (batchsize, maxspeclen + 1). With numspeculativetokens=1, this produces shape (batchsize, 2) instead of the expected (batchsize, 1), causing a broadcast shape mismatch during penalty application.

Impact

Any vLLM deployment between v0.18.0 and v0.19.1 (inclusive) configured with extracthiddenstates speculative decoding is affected. A single API request containing any penalty parameter immediately and permanently crashes the EngineCore process, resulting in complete loss of service availability.

Patches

Fixed in PR #38610, first included in vLLM v0.20.0. The fix slices the return value to sampledtokenids[:, :1], ensuring the correct (batchsize, 1) shape regardless of the rejection sampler's output dimensions.

Workarounds

- Upgrade to vLLM v0.20.0 or later. - If upgrading is not possible, avoid using extracthiddenstates as the speculative decoding method on affected versions. - Alternatively, reject or strip penalty parameters (repetitionpenalty, frequencypenalty, presencepenalty) from incoming requests at an API gateway before they reach vLLM.