24 — curl_cffi: keep proxy credentials on the tunnel, off the origin

[MirjamOdile]

What was broken: CURL_CFFI_ENABLED: true and an authenticating proxy could never be used together — every single request failed with 407 Proxy Authentication Required. Both settings are documented, both work on their own, and the combination has never worked. Worse, the proxy's username and password were being sent onward to every site we crawled. Observed in production on a site that blocks plain HTTP from the server IP and blocks curl_cffi without a proxy — so only the broken combination could have worked: the crawl enqueued 486 URLs and scraped 0 items, and even the robots.txt/llms.txt compliance witnesses failed.

What it does now: the credentials are put back where curl_cffi looks for them, and removed from where they don't belong. Proxied curl_cffi crawls authenticate normally, and the crawled site never sees the proxy credentials.

Details

Three components hand the proxy along, and the credentials fell through the gap:

1. SmartProxyMiddleware sets request.meta['proxy'] to the full credentialed URL.
2. Scrapy's built-in HttpProxyMiddleware then strips those credentials out of meta['proxy'] and moves them into a Proxy-Authorization header — the contract its own downloader honours.
3. proxies_from_request() read meta['proxy'] only — now credential-free — and handed it to curl_cffi, which authenticates from the URL and ignores that header. The proxy answered 407.

• handlers/curl_cffi_handler.py, proxies_from_request(): when meta['proxy'] carries no credentials but a Proxy-Authorization header is present, decode the Basic payload and put the credentials back into the URL. Guarded on "@" not in proxy, so a proxy URL that still has its own credentials is left untouched.
• handlers/curl_cffi_handler.py, _fetch_sync(): drop Proxy-Authorization from the headers sent to the origin server. The handler built its outgoing headers from request.headers, which still contained it — so the credentials went to every site on curl_cffi crawls. They belong to the CONNECT tunnel, not the target.
• 16 lines, one file. No new settings.
• Request doc: docs/requests/24-curl-cffi-proxy-auth.md.

Behavior changes

• CURL_CFFI_ENABLED + proxy crawls now work instead of failing 407 on every request. On the affected site: 407 count 0, robots.txt captured, items extracting, 403 rate 26% (was 42%).
• Proxy credentials are no longer disclosed to crawled sites.
• No change when no proxy is in use — meta['proxy'] absent still returns None, exactly as before.
• inspect was never affected: it calls the transport directly with the credentialed URL and never enters the middleware chain. That asymmetry is what made this expensive to diagnose — the same site "works in inspect, fails in crawl".

Verified

5 unit tests (tests/unit/test_curl_cffi_proxy.py, 3 new): credentials restored from the header, a URL carrying its own credentials left alone, Proxy-Authorization absent from the outgoing request while other headers still go out, plus the two pre-existing meta-proxy cases. Full gate green against main (black, flake8, 473 unit tests).