Package Instance

Duplicate Advisory: Scrapy leaks the authorization header on same-domain but cross-origin redirects
## Duplicate Advisory
This advisory has been withdrawn because it is a duplicate of GHSA-4qqq-9vqf-3h3f. This link is maintained to preserve external references.

## Original Description
In scrapy/scrapy, an issue was identified where the Authorization header is not removed during redirects that only change the scheme (e.g., HTTPS to HTTP) but remain within the same domain. This behavior contravenes the Fetch standard, which mandates the removal of Authorization headers in cross-origin requests when the scheme, host, or port changes. Consequently, when a redirect downgrades from HTTPS to HTTP, the Authorization header may be inadvertently exposed in plaintext, leading to potential sensitive information disclosure to unauthorized actors. The flaw is located in the _build_redirect_request function of the redirect middleware.

Scrapy authorization header leakage on cross-domain redirect
### Impact

When you send a request with the `Authorization` header to one domain, and the response asks to redirect to a different domain, Scrapy’s built-in redirect middleware creates a follow-up redirect request that keeps the original `Authorization` header, leaking its content to that second domain.

The [right behavior](https://fetch.spec.whatwg.org/#ref-for-cors-non-wildcard-request-header-name) would be to drop the `Authorization` header instead, in this scenario.

### Patches

Upgrade to Scrapy 2.11.1.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead.

### Workarounds

If you cannot upgrade, make sure that you are not using the `Authentication` header, either directly or through some third-party plugin.

If you need to use that header in some requests, add `"dont_redirect": True` to the `request.meta` dictionary of those requests to disable following redirects for them.

If you need to keep (same domain) redirect support on those requests, make sure you trust the target website not to redirect your requests to a different domain.

### Acknowledgements

This security issue was reported by @ranjit-git [through huntr.com](https://huntr.com/bounties/49974321-2718-43e3-a152-62b16eed72a9/).

Cookie-setting is not restricted based on the public suffix list
Responses from domain names whose public domain name suffix contains 1 or more periods (e.g. responses from `example.co.uk`, given its public domain name suffix is `co.uk`) are able to set cookies that are included in requests to any other domain sharing the same domain name suffix.

Scrapy decompression bomb vulnerability
### Impact

Scrapy limits allowed response sizes by default through the [`DOWNLOAD_MAXSIZE`](https://docs.scrapy.org/en/latest/topics/settings.html#download-maxsize) and [`DOWNLOAD_WARNSIZE`](https://docs.scrapy.org/en/latest/topics/settings.html#download-warnsize) settings.

However, those limits were only being enforced during the download of the raw, usually-compressed response bodies, and not during decompression, making Scrapy vulnerable to [decompression bombs](https://cwe.mitre.org/data/definitions/409.html).

A malicious website being scraped could send a small response that, on decompression, could exhaust the memory available to the Scrapy process, potentially affecting any other process sharing that memory, and affecting disk usage in case of uncompressed response caching.

### Patches

Upgrade to Scrapy 2.11.1.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead.

### Workarounds

There is no easy workaround.

Disabling HTTP decompression altogether is impractical, as HTTP compression is a rather common practice.

However, it is technically possible to manually backport the 2.11.1 or 1.8.4 fix, replacing the corresponding components of an unpatched version of Scrapy with patched versions copied into your own code.

### Acknowledgements

This security issue was reported by @dmandefy [through huntr.com](https://huntr.com/bounties/c4a0fac9-0c5a-4718-9ee4-2d06d58adabb/).

Scrapy before 2.6.2 and 1.8.3 vulnerable to one proxy sending credentials to another
### Impact

When the [built-in HTTP proxy downloader middleware](https://docs.scrapy.org/en/2.6/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.httpproxy) processes a request with `proxy` metadata, and that `proxy` metadata includes proxy credentials, the built-in HTTP proxy downloader middleware sets the `Proxy-Authentication` header, but only if that header is not already set.

There are third-party proxy-rotation downloader middlewares that set different `proxy` metadata every time they process a request.

Because of request retries and redirects, the same request can be processed by downloader middlewares more than once, including both the built-in HTTP proxy downloader middleware and any third-party proxy-rotation downloader middleware.

These third-party proxy-rotation downloader middlewares could change the `proxy` metadata of a request to a new value, but fail to remove the `Proxy-Authentication` header from the previous value of the `proxy` metadata, causing the credentials of one proxy to be leaked to a different proxy.

If you rotate proxies from different proxy providers, and any of those proxies requires credentials, you are affected, unless you are handling proxy rotation as described under **Workarounds** below. If you use a third-party downloader middleware for proxy rotation, the same applies to that downloader middleware, and installing a patched version of Scrapy may not be enough; patching that downloader middlware may be necessary as well.

### Patches

Upgrade to Scrapy 2.6.2.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.6.2 is not an option, you may upgrade to Scrapy 1.8.3 instead.

### Workarounds

If you cannot upgrade, make sure that any code that changes the value of the `proxy` request meta also removes the `Proxy-Authorization` header from the request if present.

### For more information

If you have any questions or comments about this advisory:
* [Open an issue](https://github.com/scrapy/scrapy/issues)
* [Email us](mailto:opensource@zyte.com)

Scrapy's redirects ignoring scheme-specific proxy settings
When using system proxy settings, which are scheme-specific (i.e. specific to `http://` or `https://` URLs), Scrapy was not accounting for scheme changes during redirects.

For example, an HTTP request would use the proxy configured for HTTP and, when redirected to an HTTPS URL, the new HTTPS request would still use the proxy configured for HTTP instead of switching to the proxy configured for HTTPS. Same the other way around.

If you have different proxy configurations for HTTP and HTTPS in your system for security reasons (e.g., maybe you don’t want one of your proxy providers to be aware of the URLs that you visit with the other one), this would be a security issue.

Duplicate
This advisory duplicates another.

Duplicate
This advisory duplicates another.

Scrapy is vulnerable to a denial of service (DoS) attack due to flaws in brotli decompression implementation
Scrapy versions up to 2.13.3 are vulnerable to a denial of service (DoS) attack due to a flaw in its brotli decompression implementation. The protection mechanism against decompression bombs fails to mitigate the brotli variant, allowing remote servers to crash clients with less than 80GB of available memory. This occurs because brotli can achieve extremely high compression ratios for zero-filled data, leading to excessive memory consumption during decompression. Mitigation for this vulnerability needs security enhancement added in brotli v1.2.0.