Search for packages
| purl | pkg:pypi/scrapy@2.11.1 |
| Vulnerability | Summary | Fixed by |
|---|---|---|
|
VCID-1k4b-pr5k-s7e5
Aliases: GHSA-cwxj-rr6w-m6w7 |
Scrapy: Arbitrary Module Import via Referrer-Policy Header in RefererMiddleware ### Impact Since version 1.4.0, Scrapy respects the `Referrer-Policy` response header to decide whether and how to set a `Referer` header on follow-up requests. If the header value looked like a valid Python import path, Scrapy would import the referenced object and call it, assuming it referred to a referrer policy class (for example, `scrapy.spidermiddlewares.referer.DefaultReferrerPolicy`) and attempting to instantiate it to handle the `Referer` header. A malicious site could exploit this by setting `Referrer-Policy` to a path such as `sys.exit`, causing Scrapy to import and execute it and potentially terminate the process. ### Patches Upgrade to Scrapy 2.14.2 (or later). ### Workarounds If you cannot upgrade to Scrapy 2.14.2, consider the following mitigations. - **Disable the middleware:** If you don't need the `Referer` header on follow-up requests, set [`REFERER_ENABLED`](https://docs.scrapy.org/en/latest/topics/spider-middleware.html#referer-enabled) to `False`. - **Set headers manually:** If you do need a `Referer`, disable the middleware and set the header explicitly on the requests that require it. - **Set `referrer_policy` in request metadata:** If disabling the middleware is not viable, set the [`referrer_policy`](https://docs.scrapy.org/en/latest/topics/spider-middleware.html#referrer-policy) request meta key on all requests to prevent evaluating preceding responses' `Referrer-Policy`. For example: ```python Request( url, meta={ "referrer_policy": "scrapy.spidermiddlewares.referer.DefaultReferrerPolicy", }, ) ``` Instead of editing requests individually, you can: - implement a custom [spider middleware](https://docs.scrapy.org/en/latest/topics/spider-middleware.html) that runs before the built-in referrer policy middleware and sets the `referrer_policy` meta key; or - set the meta key in start requests and use the [scrapy-sticky-meta-params](https://github.com/heylouiz/scrapy-sticky-meta-params) plugin to propagate it to follow-up requests. If you want to continue respecting legitimate `Referrer-Policy` headers while protecting against malicious ones, disable the built-in referrer policy middleware by setting it to `None` in [`SPIDER_MIDDLEWARES`](https://docs.scrapy.org/en/latest/topics/settings.html#std-setting-SPIDER_MIDDLEWARES) and replace it with the fixed implementation from Scrapy 2.14.2. If the Scrapy 2.14.2 implementation is incompatible with your project (for example, because your Scrapy version is older), copy the corresponding middleware from your Scrapy version, apply the same patch, and use that as a replacement. |
Affected by 0 other vulnerabilities. |
|
VCID-dc1m-rt7j-w3af
Aliases: CVE-2025-6176 GHSA-2qfp-q593-8484 |
Scrapy is vulnerable to a denial of service (DoS) attack due to flaws in brotli decompression implementation Scrapy versions up to 2.13.3 are vulnerable to a denial of service (DoS) attack due to a flaw in its brotli decompression implementation. The protection mechanism against decompression bombs fails to mitigate the brotli variant, allowing remote servers to crash clients with less than 80GB of available memory. This occurs because brotli can achieve extremely high compression ratios for zero-filled data, leading to excessive memory consumption during decompression. Mitigation for this vulnerability needs security enhancement added in brotli v1.2.0. |
Affected by 1 other vulnerability. |
|
VCID-nekz-z7zw-mfgz
Aliases: GHSA-23j4-mw76-5v7h |
Scrapy allows redirect following in protocols other than HTTP ### Impact Scrapy was following redirects regardless of the URL protocol, so redirects were working for `data://`, `file://`, `ftp://`, `s3://`, and any other scheme defined in the `DOWNLOAD_HANDLERS` setting. However, HTTP redirects should only work between URLs that use the `http://` or `https://` schemes. A malicious actor, given write access to the start requests (e.g. ability to define `start_urls`) of a spider and read access to the spider output, could exploit this vulnerability to: - Redirect to any local file using the `file://` scheme to read its contents. - Redirect to an `ftp://` URL of a malicious FTP server to obtain the FTP username and password configured in the spider or project. - Redirect to any `s3://` URL to read its content using the S3 credentials configured in the spider or project. For `file://` and `s3://`, how the spider implements its parsing of input data into an output item determines what data would be vulnerable. A spider that always outputs the entire contents of a response would be completely vulnerable, while a spider that extracted only fragments from the response could significantly limit vulnerable data. ### Patches Upgrade to Scrapy 2.11.2. ### Workarounds Replace the built-in retry middlewares (`RedirectMiddleware` and `MetaRefreshMiddleware`) with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended. ### References This security issue was reported by @mvsantos at https://github.com/scrapy/scrapy/issues/457. |
Affected by 2 other vulnerabilities. |
|
VCID-t5cn-a543-nyag
Aliases: GHSA-cg34-w3fm-82h3 |
Duplicate Advisory: Scrapy leaks the authorization header on same-domain but cross-origin redirects ## Duplicate Advisory This advisory has been withdrawn because it is a duplicate of GHSA-4qqq-9vqf-3h3f. This link is maintained to preserve external references. ## Original Description In scrapy/scrapy, an issue was identified where the Authorization header is not removed during redirects that only change the scheme (e.g., HTTPS to HTTP) but remain within the same domain. This behavior contravenes the Fetch standard, which mandates the removal of Authorization headers in cross-origin requests when the scheme, host, or port changes. Consequently, when a redirect downgrades from HTTPS to HTTP, the Authorization header may be inadvertently exposed in plaintext, leading to potential sensitive information disclosure to unauthorized actors. The flaw is located in the _build_redirect_request function of the redirect middleware. |
Affected by 2 other vulnerabilities. |
|
VCID-urb1-hv1z-duga
Aliases: CVE-2024-1968 GHSA-4qqq-9vqf-3h3f PYSEC-2024-258 |
In scrapy/scrapy, an issue was identified where the Authorization header is not removed during redirects that only change the scheme (e.g., HTTPS to HTTP) but remain within the same domain. This behavior contravenes the Fetch standard, which mandates the removal of Authorization headers in cross-origin requests when the scheme, host, or port changes. Consequently, when a redirect downgrades from HTTPS to HTTP, the Authorization header may be inadvertently exposed in plaintext, leading to potential sensitive information disclosure to unauthorized actors. The flaw is located in the _build_redirect_request function of the redirect middleware. |
Affected by 2 other vulnerabilities. |
|
VCID-veaw-n6vt-zfgu
Aliases: GHSA-jm3v-qxmh-hxwv |
Scrapy's redirects ignoring scheme-specific proxy settings ### Impact When using system proxy settings, which are scheme-specific (i.e. specific to `http://` or `https://` URLs), Scrapy was not accounting for scheme changes during redirects. For example, an HTTP request would use the proxy configured for HTTP and, when redirected to an HTTPS URL, the new HTTPS request would still use the proxy configured for HTTP instead of switching to the proxy configured for HTTPS. Same the other way around. If you have different proxy configurations for HTTP and HTTPS in your system for security reasons (e.g., maybe you don’t want one of your proxy providers to be aware of the URLs that you visit with the other one), this would be a security issue. ### Patches Upgrade to Scrapy 2.11.2. ### Workarounds Replace the built-in retry middlewares (`RedirectMiddleware` and `MetaRefreshMiddleware`) and the `HttpProxyMiddleware` middleware with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended. ### References This security issue was reported by @redapple at https://github.com/scrapy/scrapy/issues/767. |
Affected by 2 other vulnerabilities. |
| Vulnerability | Summary | Aliases |
|---|---|---|
| VCID-385b-344t-23es | Scrapy decompression bomb vulnerability ### Impact Scrapy limits allowed response sizes by default through the [`DOWNLOAD_MAXSIZE`](https://docs.scrapy.org/en/latest/topics/settings.html#download-maxsize) and [`DOWNLOAD_WARNSIZE`](https://docs.scrapy.org/en/latest/topics/settings.html#download-warnsize) settings. However, those limits were only being enforced during the download of the raw, usually-compressed response bodies, and not during decompression, making Scrapy vulnerable to [decompression bombs](https://cwe.mitre.org/data/definitions/409.html). A malicious website being scraped could send a small response that, on decompression, could exhaust the memory available to the Scrapy process, potentially affecting any other process sharing that memory, and affecting disk usage in case of uncompressed response caching. ### Patches Upgrade to Scrapy 2.11.1. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead. ### Workarounds There is no easy workaround. Disabling HTTP decompression altogether is impractical, as HTTP compression is a rather common practice. However, it is technically possible to manually backport the 2.11.1 or 1.8.4 fix, replacing the corresponding components of an unpatched version of Scrapy with patched versions copied into your own code. ### Acknowledgements This security issue was reported by @dmandefy [through huntr.com](https://huntr.com/bounties/c4a0fac9-0c5a-4718-9ee4-2d06d58adabb/). |
CVE-2024-3572
GHSA-7j7m-v7m3-jqm7 GMS-2024-327 |
| VCID-64nx-aruy-q7gy | A Regular Expression Denial of Service (ReDoS) vulnerability exists in the XMLFeedSpider class of the scrapy/scrapy project, specifically in the parsing of XML content. By crafting malicious XML content that exploits inefficient regular expression complexity used in the parsing process, an attacker can cause a denial-of-service (DoS) condition. This vulnerability allows for the system to hang and consume significant resources, potentially rendering services that utilize Scrapy for XML processing unresponsive. |
CVE-2024-1892
GHSA-cc65-xxvf-f7r9 GMS-2024-287 PYSEC-2024-162 |
| VCID-8wu8-qh8m-7ubm | Duplicate Advisory: ReDos vulnerability of XMLFeedSpider ## Duplicate Advisory This advisory has been withdrawn because it is a duplicate of GHSA-cc65-xxvf-f7r9. This link is maintained to preserve external references. ## Original Description Parts of the Scrapy API were found to be vulnerable to a ReDoS attack. Handling a malicious response could cause extreme CPU and memory usage during the parsing of its content, due to the use of vulnerable regular expressions for that parsing. |
GHSA-7c9g-vj9m-8pm6
|
| VCID-auya-u2uf-27gn | Duplicate Advisory: Scrapy decompression bomb vulnerability ## Duplicate Advisory This advisory has been withdrawn because it is a duplicate of GHSA-7j7m-v7m3-jqm7. This link is maintained to preserve external references. ## Original Description The scrapy/scrapy project is vulnerable to XML External Entity (XXE) attacks due to the use of lxml.etree.fromstring for parsing untrusted XML data without proper validation. This vulnerability allows attackers to perform denial of service attacks, access local files, generate network connections, or circumvent firewalls by submitting specially crafted XML data. |
GHSA-rmqv-7v3j-mr7p
|
| VCID-epjc-cfvv-s3dq | Duplicate Advisory: Scrapy authorization header leakage on cross-domain redirect ## Duplicate Advisory This advisory has been withdrawn because it is a duplicate of GHSA-cw9j-q3vf-hrrv. This link is maintained to preserve external references. ## Original Description In scrapy versions before 2.11.1, an issue was identified where the Authorization header, containing credentials for server authentication, is leaked to a third-party site during a cross-domain redirect. This vulnerability arises from the failure to remove the Authorization header when redirecting across domains. The exposure of the Authorization header to unauthorized actors could potentially allow for account hijacking. |
GHSA-4q82-j5c2-g2c5
|
| VCID-kgf5-wu3r-pqc6 | Scrapy authorization header leakage on cross-domain redirect ### Impact When you send a request with the `Authorization` header to one domain, and the response asks to redirect to a different domain, Scrapy’s built-in redirect middleware creates a follow-up redirect request that keeps the original `Authorization` header, leaking its content to that second domain. The [right behavior](https://fetch.spec.whatwg.org/#ref-for-cors-non-wildcard-request-header-name) would be to drop the `Authorization` header instead, in this scenario. ### Patches Upgrade to Scrapy 2.11.1. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead. ### Workarounds If you cannot upgrade, make sure that you are not using the `Authentication` header, either directly or through some third-party plugin. If you need to use that header in some requests, add `"dont_redirect": True` to the `request.meta` dictionary of those requests to disable following redirects for them. If you need to keep (same domain) redirect support on those requests, make sure you trust the target website not to redirect your requests to a different domain. ### Acknowledgements This security issue was reported by @ranjit-git [through huntr.com](https://huntr.com/bounties/49974321-2718-43e3-a152-62b16eed72a9/). |
CVE-2024-3574
GHSA-cw9j-q3vf-hrrv GMS-2024-288 |