Summary

The current source tree still allows arbitrary code execution during supposedly safer allowlisted pickle loading. The allowlist trusts whole module namespaces instead of exact safe globals, so crafted pickles can invoke dangerous in-namespace callables through pickle REDUCE.

Details

- Vulnerability type: Remote code execution via unsafe deserialization - Affected component: nltk.picklesec.allowlistedpickleload, nltk.tokenize.punkt.punktpickleload, nltk.parse.transitionparser.TransitionParser.parse - Affected versions: Current source v3.10.0-rc2; published 3.9.4 was not the claim target for this bypass. - Patched versions: Not yet patched - Root cause: Module-prefix allowlists include dangerous callables such as nltk.tokenize.repp.ReppTokenizer.execute and numpy.f2py.crackfortran.myeval.

punktpickleload() allowlists both nltk.tokenize.punkt and the whole nltk.tokenize namespace, which exposes ReppTokenizer.execute() and its subprocess.Popen(...) sink during unpickling. TransitionParser.parse() uses allowlistedpickleload(..., allowedmodules=("numpy", "scipy", "sklearn")), which permits numpy.f2py.crackfortran.myeval() and its attacker-controlled eval(...) path. I confirmed both gadgets create marker files before the caller returns or later aborts on type misuse.

PoC

Preconditions - The application loads an attacker-controlled tokenizer or model artifact through these public loaders.

Steps 1. Create a pickle whose REDUCE callable is ReppTokenizer.execute and point its command to a harmless marker-file write. 2. Pass that payload to punktpickleload(BytesIO(payload)) and observe the marker file is created during unpickling. 3. Create a second pickle whose REDUCE callable is numpy.f2py.crackfortran.myeval and load it through TransitionParser.parse(). 4. Observe the second marker file is created before TransitionParser.parse() later fails on the returned object type.

Minimal reproducible excerpt

text {'punktmarker': 'PUNKTRCE', 'transitionparsermarker': 'TPRCE'}

Impact

Any caller that trusts these current allowlisted loaders can still execute attacker-controlled commands while loading model or tokenizer artifacts. This defeats the protection mechanism that replaced unrestricted pickle loading and creates a dangerous false sense of safety.

Remediation

Replace broad module-prefix allowlists with exact (module, qualname) pairs for the few safe classes or functions genuinely required. Do not allow entire namespaces such as nltk.tokenize or numpy, and keep post-load type validation only as a secondary defense.

Resources

- https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/tokenize/punkt.py#L120-L134 - https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/tokenize/repp.py#L111-L115 - https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/parse/transitionparser.py#L26-L30 - https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/parse/transitionparser.py#L565-L571

---

Fix + attack demonstration (verified)

+ tightened callers findclass now, before the allowlists: 1. Rejects any dotted name → closes 4489 with zero legit impact. 2. Denies dangerous modules (os, subprocess, sys, builtins, numpy.f2py, nltk.tokenize.repp, …) even under a broad allowedmodules — a defense-in-depth backstop so a future too-broad allowlist can't silently reopen RCE. 3. builtins denied wholesale; safe primitives (int, str, …) must be named exactly via allowedglobals.

Callers tightened: punkt drops the broad nltk.tokenize (keeps nltk.tokenize.punkt + exact collections.defaultdict/builtins.int); transitionparser keeps numpy/scipy/sklearn (array unpickling needs their submodules) with the new guards blocking the gadgets.

Full pickle-sink audit Every deserialization sink in the tree was reviewed: no raw pickle.load anywhere, and no joblib/numpy/torch/dill/yaml/marshal loaders. data.load + wordnetapp use RestrictedUnpickler (blocks all globals — safe); the remaining pickleload sites (chartparserapp, tbl/demo) load user-selected or self-written files and keep their warning.

Attack demonstration (captured; fork clone) === EXPLOITS blocked === 4489 sklearn.os.system (dotted) -> BLOCKED x99w numpy.f2py.crackfortran.myeval -> BLOCKED x99w nltk.tokenize.repp.execute -> BLOCKED backstop os.system (os allowlisted) -> BLOCKED backstop builtins.eval (exact global)-> BLOCKED === LEGIT loads still work === punkt round-trip via punktpickleload -> OK builtins.int (safe primitive) -> OK

Tests testpickleallowlistsecurity.py — added 5 regressions (dotted traversal, both namespace gadgets, denied-module backstop, legit round-trip). Suite: 122 passed / 9 skipped (sklearn-dependent) across pickle/punkt/transition/tokenize. pre-commit (black/isort/ruff) clean.

NLTK before 3.10.3 contains a remote code execution vulnerability in AllowlistUnpickler that validates only the pickle module string and not the global name, allowing attackers to resolve dotted names by attribute traversal to callables outside the allowlisted namespace. Attackers can craft untrusted transition-parser models that execute arbitrary commands when TransitionParser.parse loads the model through allowlistedpickleload.

NLTK versions before 3.10.0 default to ENFORCE=False in pathsec.py, causing all security validation functions to emit warnings instead of raising exceptions. Attackers can bypass path traversal and pickle deserialization protections by exploiting the disabled security controls that are only active when manually enabled.

Summary

Current NLTK source reopens SSRF in proxied environments. pathsec.urlopen() validates the requested hostname locally, but once proxy inheritance is enabled the real fetch is performed by the proxy rather than by the validated direct-connect socket path.

Details

- Vulnerability type: Server-side request forgery - Affected component: nltk.pathsec.urlopen, nltk.data.load, nltk.downloader.Downloader.index, nltk.downloader.Downloader.download - Affected versions: Current source v3.10.0-rc2; published 3.9.4 was a negative control and did not reproduce. - Patched versions: Not yet patched - Root cause: Proxy-handler inheritance disables SafeHTTPHandler and SafeHTTPSHandler, so the validated hostname no longer matches the actual egress destination.

The hardened direct path pins the validated numeric destination IP before opening the socket. The proxied branch instead copies ProxyHandler instances from the global opener, marks the request as proxied, and skips the pinned handlers. I confirmed that a validated public URL can be fetched from a loopback-only internal service through the proxy path via pathsec.urlopen(), nltk.data.load(), Downloader.index(), and Downloader.download().

PoC

Preconditions - The runtime has an HTTP proxy configured and the caller relies on pathsec to keep network fetches SSRF-safe.

Steps 1. Start a loopback-only HTTP server that serves secret text, a valid downloader index, and a ZIP payload. 2. Configure a proxy that forwards a validated public URL to that internal loopback service. 3. Call pathsec.urlopen() or nltk.data.load() on the public URL and observe the internal response is returned. 4. Instantiate Downloader(serverindexurl=...), call index() and download(), and observe internal-only content is parsed and installed.

Minimal reproducible excerpt

text {'urlopen': 'PROXYTEXTSECRET', 'dataload': 'PROXYTEXTSECRET', 'downloadedfile': 'INTERNALZIPSECRET'}

Impact

Consumers that trust pathsec as an SSRF barrier in proxied environments can be made to read internal-only HTTP resources, load forged downloader indexes, and install attacker-chosen package content fetched from the proxy's network view.

Remediation

Preserve destination validation for the actual proxy egress target or fail closed when the request would otherwise downgrade into an unpinned proxied path. Add regression tests across pathsec.urlopen, nltk.data.load, and downloader fetches with a configured proxy.

References

- https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/pathsec.py#L468-L518 - https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/data.py#L1247-L1283 - https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/downloader.py#L875-L889 - https://github.com/nltk/nltk/blob/v3.10.0-rc2/nltk/downloader.py#L1220-L1226 - https://github.com/nltk/nltk/blob/3.9.4/nltk/pathsec.py#L245-L250

---

Fix + attack demonstration (verified)

NLTK cannot pin the egress through a proxy, so it stops pretending to: under ENFORCE a proxied fetch is refused rather than performed unvalidated. Operators who trust their proxy opt back in with NLTKALLOWPROXIEDURLOPEN=1 or nltk.pathsec.ALLOWPROXIEDFETCH=True; under ENFORCE=False the refusal degrades to a warning. This closes the whole class (environment proxies and explicit ProxyHandler alike), because NLTK declines any fetch whose egress it cannot validate.

Attack demonstration (reproduced; captured output) A loopback HTTP server stands in for the internal target; httpproxy points at it; NLTK is asked for a public IP URL.

Before the fix — the internal secret is exfiltrated through the proxy: validatenetworkurl(public): PASSED BYPASS: pathsec.urlopen returned INTERNAL content via proxy: 'INTERNALONLYSECRET'

After the fix — five scenarios, isolated subprocesses: | Scenario | Result | |---|---| | proxied (env) + ENFORCE | PermissionError — blocked | | proxied + opt-in | returns secret — escape hatch works | | explicit ProxyHandler (not env) + ENFORCE | PermissionError — blocked (whole class) | | no proxy (direct) | internal IP still refused — pinning intact | | proxied + ENFORCE=False | returns secret + warns |

Tests nltk/test/unit/testpathsec.py: 64 passed. Added an end-to-end regression (testproxiedfetchdoesnotreachinternaltarget) plus testenvproxyfailsclosedunderenforce; the prior testenvproxyskipspinninghandlers (which encoded the vulnerable path) is re-expressed as the opt-in case. Existing direct-path DNS-rebinding and IP-policy tests unchanged and passing. pre-commit (isort/black/ruff) clean.

Note The upfront validatenetworkurl() and the direct-path IP pinning (from the earlier DNS-rebinding fixes, CVE-2026-54296 / GHSA-qvv7) are unchanged — this only closes the proxied downgrade they didn't cover.