Brotli versions up to 1.1.0 are vulnerable to a denial of service (DoS) attack due to decompression. This issue has been patched in Brotli version 1.2.0.

Additionally, this affects users who implement the Brotli decompression with Scrapy versions up to 2.13.2, leaving them vulnerable to a denial of service (DoS) attack. The protection mechanism against decompression bombs fails to mitigate the brotli variant, allowing remote servers to crash clients with less than 80GB of available memory. This occurs because brotli can achieve extremely high compression ratios for zero-filled data, leading to excessive memory consumption during decompression.

Impact

If you use HttpAuthMiddleware (i.e. the httpuser and httppass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target.

This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXTOBEY setting is set to True, or as requests reached through redirects.

Patches

Upgrade to Scrapy 2.5.1 and use the new httpauthdomain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead.

Workarounds

If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basicauthheader function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.

For more information If you have any questions or comments about this advisory: Open an issue Email us

Duplicate Advisory This advisory has been withdrawn because it is a duplicate of GHSA-cw9j-q3vf-hrrv. This link is maintained to preserve external references.

Original Description In scrapy versions before 2.11.1, an issue was identified where the Authorization header, containing credentials for server authentication, is leaked to a third-party site during a cross-domain redirect. This vulnerability arises from the failure to remove the Authorization header when redirecting across domains. The exposure of the Authorization header to unauthorized actors could potentially allow for account hijacking.

Duplicate Advisory This advisory has been withdrawn because it is a duplicate of GHSA-7j7m-v7m3-jqm7. This link is maintained to preserve external references.

Original Description The scrapy/scrapy project is vulnerable to XML External Entity (XXE) attacks due to the use of lxml.etree.fromstring for parsing untrusted XML data without proper validation. This vulnerability allows attackers to perform denial of service attacks, access local files, generate network connections, or circumvent firewalls by submitting specially crafted XML data.

Impact

Since version 2.11.1, Scrapy drops the Authorization header when a request is redirected to a different domain. However, it keeps the header if the domain remains the same but the scheme (http/https) or the port change, all scenarios where the header should also be dropped.

In the context of a man-in-the-middle attack, this could be used to get access to the value of that Authorization header

Patches

Upgrade to Scrapy 2.11.2.

Workarounds

There is no easy workaround for unpatched versions of Scrapy. You can replace the built-in redirect middlewares with custom ones patched for this issue, but you have to patch them yourself, manually.

References

This security issue was reported and fixed by @szarny at https://huntr.com/bounties/27f6a021-a891-446a-ada5-0226d619dd1a/.

Impact

The following parts of the Scrapy API were found to be vulnerable to a ReDoS attack:

- The XMLFeedSpider class or any subclass that uses the default node iterator: iternodes, as well as direct uses of the scrapy.utils.iterators.xmliter function.

- Scrapy 2.6.0 to 2.11.0: The openinbrowser function for a response without a base tag.

Handling a malicious response could cause extreme CPU and memory usage during the parsing of its content, due to the use of vulnerable regular expressions for that parsing.

Patches

Upgrade to Scrapy 2.11.1.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead.

Workarounds

For XMLFeedSpider, switch the node iterator to xml or html.

For openinbrowser, before using the function, either manually review the response content to discard a ReDos attack or manually define the base tag to avoid its automatic definition by openinbrowser later.

Acknowledgements

This security issue was reported by @nicecatch2000 through huntr.com.

Impact

If you manually define cookies on a Request object, and that Request object gets a redirect response, the new Request object scheduled to follow the redirect keeps those user-defined cookies, regardless of the target domain.

Patches

Upgrade to Scrapy 2.6.0, which resets cookies when creating Request objects to follow redirects¹, and drops the Cookie header if manually-defined if the redirect target URL domain name does not match the source URL domain name².

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.6.0 is not an option, you may upgrade to Scrapy 1.8.2 instead.

¹ At that point the original, user-set cookies have been processed by the cookie middleware into the global or request-specific cookiejar, with their domain restricted to the domain of the original URL, so when the cookie middleware processes the new (redirect) request it will incorporate those cookies into the new request as long as the domain of the new request matches the domain of the original request.

² This prevents cookie leaks to unintended domains even if the cookies middleware is not used.

Workarounds

If you cannot upgrade, set your cookies using a list of dictionaries instead of a single dictionary, as described in the Request documentation, and set the right domain for each cookie.

Alternatively, you can disable cookies altogether, or limit target domains to domains that you trust with all your user-set cookies.

References Originally reported at huntr.dev

For more information

If you have any questions or comments about this advisory: Open an issue Email us

Scrapy 1.4 allows remote attackers to cause a denial of service (memory consumption) via large files because arbitrarily many files are read into memory, which is especially problematic if the files are then individually written in a separate thread to a slow storage resource, as demonstrated by interaction between dataReceived (in core/downloader/handlers/http11.py) and S3FilesStore.