CVE-2026-55520: Protego: Exponential backtracking ReDoS in robots.txt URL wildcard matching
Problem description
Protego constructs regular expressions to match URLs against robots.txt Allow: and Disallow: directives, see protego.urlpattern.URLPattern.preparepatternforregex(). Every in the directive value is translated into a lazy .? regex piece, thus a specially crafted directive value with many asterisks may produce a regex that freezes the parser due to exponential backtracking.
Impact
Parsing a specially crafted robots.txt with protego.Protego.parse() and then trying to match an URL with protego.Protego.canfetch() results in the latter call not returning for a period dependent on the length of the URL.
Proof of concept
python from protego import Protego
robotstxt = f""" User-agent: Disallow: /{"1" 12}Z """ rp = Protego.parse(robotstxt) url = "/" + "1" 60 rp.canfetch(url, "mybot") # freezes
Other sources
Protego is a pure-Python robots.txt parser with support for modern conventions. Prior to 0.6.2, protego.urlpattern.URLPattern.preparepatternforregex translates every asterisk in an Allow or Disallow directive into a lazy regular-expression wildcard, so a directive containing many asterisks creates exponential backtracking. After protego.Protego.parse processes a crafted robots.txt file, protego.Protego.canfetch can spend an attacker-controlled period matching a near-miss URL and deny service to the crawler. The vulnerable path is src/protego/urlpattern.py in the URLPattern match logic. This issue is fixed in version 0.6.2.
— MITRE
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
pip/Protegoto a version that resolves this vulnerability.Fixed in 0.6.2 - Upgrade
Upgrade
protegoto a version that resolves this vulnerability.Fixed in 0.6.2
Event History
Frequently Asked Questions
What must an attacker control to trigger the issue?
They need to supply a specially crafted robots.txt containing a directive value with many asterisks. The application must parse that file with Protego and subsequently evaluate a URL against it using can_fetch().
When does the application become unresponsive?
The excessive processing occurs during the can_fetch() URL-matching call, rather than when Protego.parse() parses the robots.txt file. The duration depends on the length of the URL being matched.
Which configurations are exposed?
Any use of Protego that parses attacker-controlled or otherwise untrusted robots.txt content and calls can_fetch() on it is exposed. The provided example uses a User-agent: * Disallow: directive.
What version contains the fix?
The referenced Protego release 0.6.2 includes the fix.