CVE-2026-85670: tokenizers BpeBuilder Buffer Overflow via merge token
tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::fromfile/fromstr, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuingsubwordprefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
tokenizersto a version that resolves this vulnerability.Fixed in 0.23.1Patch tokenizers BpeBuilder Buffer Overflow via merge token
Event History
Frequently Asked Questions
What attacker-controlled input is required to trigger the issue?
An attacker must be able to supply a crafted tokenizer.json that the application loads through Tokenizer::from_file or Tokenizer::from_str. No encoding operation is required; the failure occurs while loading the tokenizer.
What is the practical impact in Rust and embedded use cases?
The out-of-bounds buffer access is converted by Rust into a panic that aborts the process, including in FFI embeddings, causing denial of service. A related condition involving continuing_subword_prefix and a shorter merge token panics in debug builds and may result in memory corruption in release builds.
How can I determine whether a tokenizer file is likely to trigger the primary condition?
Inspect merge rules in the tokenizer.json and compare each concatenated merge token with the longest vocabulary key. A concatenated merge token longer than every vocabulary key can overrun the scratch buffer during BPE model construction.
What configuration condition triggers the secondary defect?
The secondary defect requires continuing_subword_prefix to be set and a merge token shorter than that prefix. Under those conditions, length calculation can underflow.
Which version has been observed to be affected?
The issue was observed in tokenizers version 0.23.1.