GHSA-x99w-6fgc-pmfw: Pip/nltk vulnerability
Summary
The current source tree still allows arbitrary code execution during supposedly safer allowlisted pickle loading. The allowlist trusts whole module namespaces instead of exact safe globals, so crafted pickles can invoke dangerous in-namespace callables through pickle REDUCE.
Details
- Vulnerability type: Remote code execution via unsafe deserialization - Affected component: nltk.picklesec.allowlistedpickleload, nltk.tokenize.punkt.punktpickleload, nltk.parse.transitionparser.TransitionParser.parse - Affected versions: Current source v3.10.0-rc2; published 3.9.4 was not the claim target for this bypass. - Patched versions: Not yet patched - Root cause: Module-prefix allowlists include dangerous callables such as nltk.tokenize.repp.ReppTokenizer.execute and numpy.f2py.crackfortran.myeval.
punktpickleload() allowlists both nltk.tokenize.punkt and the whole nltk.tokenize namespace, which exposes ReppTokenizer.execute() and its subprocess.Popen(...) sink during unpickling. TransitionParser.parse() uses allowlistedpickleload(..., allowedmodules=("numpy", "scipy", "sklearn")), which permits numpy.f2py.crackfortran.myeval() and its attacker-controlled eval(...) path. I confirmed both gadgets create marker files before the caller returns or later aborts on type misuse.
PoC
Preconditions - The application loads an attacker-controlled tokenizer or model artifact through these public loaders.
Steps 1. Create a pickle whose REDUCE callable is ReppTokenizer.execute and point its command to a harmless marker-file write. 2. Pass that payload to punktpickleload(BytesIO(payload)) and observe the marker file is created during unpickling. 3. Create a second pickle whose REDUCE callable is numpy.f2py.crackfortran.myeval and load it through TransitionParser.parse(). 4. Observe the second marker file is created before TransitionParser.parse() later fails on the returned object type.
Minimal reproducible excerpt
text {'punktmarker': 'PUNKTRCE', 'transitionparsermarker': 'TPRCE'}
Impact
Any caller that trusts these current allowlisted loaders can still execute attacker-controlled commands while loading model or tokenizer artifacts. This defeats the protection mechanism that replaced unrestricted pickle loading and creates a dangerous false sense of safety.
Remediation
Replace broad module-prefix allowlists with exact (module, qualname) pairs for the few safe classes or functions genuinely required. Do not allow entire namespaces such as nltk.tokenize or numpy, and keep post-load type validation only as a secondary defense.
Resources
- https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/tokenize/punkt.py#L120-L134 - https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/tokenize/repp.py#L111-L115 - https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/parse/transitionparser.py#L26-L30 - https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/parse/transitionparser.py#L565-L571
---
Fix + attack demonstration (verified)
+ tightened callers findclass now, before the allowlists: 1. Rejects any dotted name → closes 4489 with zero legit impact. 2. Denies dangerous modules (os, subprocess, sys, builtins, numpy.f2py, nltk.tokenize.repp, …) even under a broad allowedmodules — a defense-in-depth backstop so a future too-broad allowlist can't silently reopen RCE. 3. builtins denied wholesale; safe primitives (int, str, …) must be named exactly via allowedglobals.
Callers tightened: punkt drops the broad nltk.tokenize (keeps nltk.tokenize.punkt + exact collections.defaultdict/builtins.int); transitionparser keeps numpy/scipy/sklearn (array unpickling needs their submodules) with the new guards blocking the gadgets.
Full pickle-sink audit Every deserialization sink in the tree was reviewed: no raw pickle.load anywhere, and no joblib/numpy/torch/dill/yaml/marshal loaders. data.load + wordnetapp use RestrictedUnpickler (blocks all globals — safe); the remaining pickleload sites (chartparserapp, tbl/demo) load user-selected or self-written files and keep their warning.
Attack demonstration (captured; fork clone) === EXPLOITS blocked === 4489 sklearn.os.system (dotted) -> BLOCKED x99w numpy.f2py.crackfortran.myeval -> BLOCKED x99w nltk.tokenize.repp.execute -> BLOCKED backstop os.system (os allowlisted) -> BLOCKED backstop builtins.eval (exact global)-> BLOCKED === LEGIT loads still work === punkt round-trip via punktpickleload -> OK builtins.int (safe primitive) -> OK
Tests testpickleallowlistsecurity.py — added 5 regressions (dotted traversal, both namespace gadgets, denied-module backstop, legit round-trip). Suite: 122 passed / 9 skipped (sklearn-dependent) across pickle/punkt/transition/tokenize. pre-commit (black/isort/ruff) clean.
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
pip/nltkto a version that resolves this vulnerability.Fixed in 3.10.3 - Configuration
Replace the punkt_pickle_load allowlist that currently permits the whole "nltk.tokenize" namespace with an allowlist restricted to "nltk.tokenize.punkt" (dropping "nltk.tokenize"). This prevents gadgets like nltk.tokenize.repp.ReppTokenizer._execute from being reachable during unpickling.
nltk.tokenize.punkt.punkt_pickle_load allowed_modules = ("nltk.tokenize.punkt" only) and remove broad allow of "nltk.tokenize" - Configuration
Tighten TransitionParser.parse() deserialization allowlists so they do not permit dangerous in-module callables such as numpy.f2py.crackfortran.myeval (and its attacker-controlled eval(...) path). Specifically, replace broad module-prefix allowlists with exact (module, qualname) pairs for only the safe classes/functions required for parsing.
nltk.parse.transitionparser.TransitionParser.parse allowed_modules = (keep only ("numpy", "scipy", "sklearn") but ensure allowlists use exact (module, qualname) pairs and block numpy.f2py.crackfortran.myeval gadget) - Compensating control
Ensure the builtins backstop remains active during allowlisted pickle loading: deny builtins wholesale and only permit exact names via allowed_globals (e.g., allow builtins.int but block builtins.eval and dotted names). This provides a defense-in-depth against future too-broad allowlists.
- Operational
Add/maintain regression tests to verify the allowlist security behavior: block dotted traversal (e.g., "sklearn.os.system"), block the namespace gadgets (nltk.tokenize.repp._execute and numpy.f2py.crackfortran.myeval), and confirm legitimate round-trips still work (e.g., punkt round-trip via punkt_pickle_load).
Event History
Frequently Asked Questions
Who is exposed to this issue?
Applications using the current NLTK source v3.10.0-rc2 are exposed if they invoke the affected pickle-loading paths on attacker-controlled pickle data. The identified paths are allowlisted_pickle_load, punkt_pickle_load, and TransitionParser.parse.
What does an attacker need to exploit it?
An attacker needs to cause a crafted pickle to be processed by one of the affected loading paths. The bypass uses pickle REDUCE to invoke dangerous callables that are reachable because entire module namespaces are allowlisted.
Is the published 3.9.4 release affected by this specific bypass?
Published version 3.9.4 was not the claim target for this bypass. The affected version identified in the advisory is the current source tree at v3.10.0-rc2.
What should be done if a patch is not yet available?
Do not process untrusted pickle data through the affected NLTK loading paths. No patched version is listed.
How can I determine whether my application is affected?
Check whether it uses current-source NLTK v3.10.0-rc2 and calls allowlisted_pickle_load, punkt_pickle_load, or TransitionParser.parse with pickle data that an attacker could influence. In particular, punkt_pickle_load allows the full nltk.tokenize namespace, and TransitionParser.parse allows numpy, scipy, and sklearn namespaces.