GHSA-wjmf-p669-5m5p: Pip/Protego vulnerability
Problem description
Protego constructs regular expressions to match URLs against robots.txt Allow: and Disallow: directives, see protego.urlpattern.URLPattern.preparepatternforregex(). Every in the directive value is translated into a lazy .? regex piece, thus a specially crafted directive value with many asterisks may produce a regex that freezes the parser due to exponential backtracking.
Impact
Parsing a specially crafted robots.txt with protego.Protego.parse() and then trying to match an URL with protego.Protego.canfetch() results in the latter call not returning for a period dependent on the length of the URL.
Proof of concept
python from protego import Protego
robotstxt = f""" User-agent: Disallow: /{"1" 12}Z """ rp = Protego.parse(robotstxt) url = "/" + "1" 60 rp.canfetch(url, "mybot") # freezes
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
pip/Protegoto a version that resolves this vulnerability.Fixed in 0.6.2
Event History
Frequently Asked Questions
What conditions are required to trigger the parser freeze?
An attacker needs to supply a specially crafted robots.txt containing a directive value with many asterisks, and the application must parse it with Protego.parse(). The slowdown occurs when can_fetch() later matches a URL whose length drives the exponential backtracking.
Which deployments are realistically exposed?
Applications using pip/Protego to parse robots.txt and then evaluate URLs with Protego.can_fetch() are exposed when they process attacker-controlled or otherwise untrusted robots.txt content. The impact is a can_fetch() call that may not return for a period dependent on URL length.
How can I check whether my application is affected?
Review whether it uses Protego.parse() on robots.txt content and subsequently calls can_fetch(). The provided proof of concept uses a Disallow directive built from repeated "*1" sequences followed by "*Z" and a URL consisting of repeated "1" characters; a freezing or severely delayed can_fetch() call indicates exposure.