⬆️ Update scrapy to v2.14.2 [SECURITY]

[renovate]

This PR contains the following updates:

Package	Change	Age	Adoption	Passing	Confidence
scrapy (changelog)	==2.13.4 → ==2.14.2

---

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

GitHub Vulnerability Alerts

GHSA-cwxj-rr6w-m6w7

Impact

Since version 1.4.0, Scrapy respects the Referrer-Policy response header to decide whether and how to set a Referer header on follow-up requests.

If the header value looked like a valid Python import path, Scrapy would import the referenced object and call it, assuming it referred to a referrer policy class (for example, scrapy.spidermiddlewares.referer.DefaultReferrerPolicy) and attempting to instantiate it to handle the Referer header.

A malicious site could exploit this by setting Referrer-Policy to a path such as sys.exit, causing Scrapy to import and execute it and potentially terminate the process.

Patches

Upgrade to Scrapy 2.14.2 (or later).

Workarounds

If you cannot upgrade to Scrapy 2.14.2, consider the following mitigations.

• Disable the middleware: If you don't need the Referer header on follow-up requests, set REFERER_ENABLED to False.
• Set headers manually: If you do need a Referer, disable the middleware and set the header explicitly on the requests that require it.
• Set referrer_policy in request metadata: If disabling the middleware is not viable, set the referrer_policy request meta key on all requests to prevent evaluating preceding responses' Referrer-Policy. For example:

Request(
    url,
    meta={
        "referrer_policy": "scrapy.spidermiddlewares.referer.DefaultReferrerPolicy",
    },
)

Instead of editing requests individually, you can:

• implement a custom spider middleware that runs before the built-in referrer policy middleware and sets the referrer_policy meta key; or
• set the meta key in start requests and use the scrapy-sticky-meta-params plugin to propagate it to follow-up requests.

If you want to continue respecting legitimate Referrer-Policy headers while protecting against malicious ones, disable the built-in referrer policy middleware by setting it to None in SPIDER_MIDDLEWARES and replace it with the fixed implementation from Scrapy 2.14.2.

If the Scrapy 2.14.2 implementation is incompatible with your project (for example, because your Scrapy version is older), copy the corresponding middleware from your Scrapy version, apply the same patch, and use that as a replacement.

Severity

• CVSS Score: 7.5 / 10 (High)
• Vector String: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

---

Release Notes

scrapy/scrapy (scrapy)

v2.14.2

Compare Source

• Values from the Referrer-Policy header of HTTP responses are no longer executed as Python callables. See the cwxj-rr6w-m6w7 security advisory for details.
• In line with the standard, 301 redirects of POST requests are converted into GET requests.

Full Changelog

v2.14.1

Compare Source

• Deprecate maybeDeferred_coro()
• Pass the spider arg to custom stat collectors {open,close}_spider()
• Replace deprecated Codecov CI action

Full Changelog

v2.14.0

Compare Source

• More coroutine-based replacements for Deferred-based APIs
• The default priority queue is now DownloaderAwarePriorityQueue
• Dropped support for Python 3.9 and PyPy 3.10
• Improved and documented the API for custom download handlers

Full changelog

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • ""
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.