Impact
If you manually define cookies on a Request object, and that Request object gets a redirect response, the new Request object scheduled to follow the redirect keeps those user-defined cookies, regardless of the target domain.
Patches
Upgrade to Scrapy 2.6.0, which resets cookies when creating Request objects to follow redirects¹, and drops the Cookie header if manually-defined if the redirect target URL domain name does not match the source URL domain name².
If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.6.0 is not an option, you may upgrade to Scrapy 1.8.2 instead.
¹ At that point the original, user-set cookies have been processed by the cookie middleware into the global or request-specific cookiejar, with their domain restricted to the domain of the original URL, so when the cookie middleware processes the new (redirect) request it will incorporate those cookies into the new request as long as the domain of the new request matches the domain of the original request.
² This prevents cookie leaks to unintended domains even if the cookies middleware is not used.
Workarounds
If you cannot upgrade, set your cookies using a list of dictionaries instead of a single dictionary, as described in the Request documentation, and set the right domain for each cookie.
Alternatively, you can disable cookies altogether, or limit target domains to domains that you trust with all your user-set cookies.
References Originally reported at huntr.dev
For more information
If you have any questions or comments about this advisory: Open an issue Email us
Scrapy 1.4 allows remote attackers to cause a denial of service (memory consumption) via large files because arbitrarily many files are read into memory, which is especially problematic if the files are then individually written in a separate thread to a slow storage resource, as demonstrated by interaction between dataReceived (in core/downloader/handlers/http11.py) and S3FilesStore.
Impact
The following parts of the Scrapy API were found to be vulnerable to a ReDoS attack:
- The XMLFeedSpider class or any subclass that uses the default node iterator: iternodes, as well as direct uses of the scrapy.utils.iterators.xmliter function.
- Scrapy 2.6.0 to 2.11.0: The openinbrowser function for a response without a base tag.
Handling a malicious response could cause extreme CPU and memory usage during the parsing of its content, due to the use of vulnerable regular expressions for that parsing.
Patches
Upgrade to Scrapy 2.11.1.
If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead.
Workarounds
For XMLFeedSpider, switch the node iterator to xml or html.
For openinbrowser, before using the function, either manually review the response content to discard a ReDos attack or manually define the base tag to avoid its automatic definition by openinbrowser later.
Acknowledgements
This security issue was reported by @nicecatch2000 through huntr.com.