perf(leadgen): reuse pooled httpx clients in plain fetch

[debpalash]

Part of #33. This is the cheap fix: Tier-1 plain HTTP fetches now reuse their transports.

What changed

StealthClient._fetch_plain used to build and close a new httpx.AsyncClient on every fetch, so every fetch paid a fresh TCP/TLS handshake. It now borrows a pooled client:

• Keyed by (running event loop, proxy). httpx clients are bound to the loop they were first used on, and the proxy is a client-level setting. Loops are held in a WeakKeyDictionary, so a finished loop (tests, asyncio.run in workers) drops its clients and they are never handed to another loop.
• Bounded: an LRU of at most 16 clients per loop (_MAX_PLAIN_CLIENTS). An evicted client is closed once its in-flight requests finish (tracked with an in-use refcount).
• Shutdown: a new aclose_plain_clients() closes the current loop's clients. It is called from the FastAPI lifespan shutdown in apps/api/main.py.
• The aiohttp fallback (used only if httpx can't be imported) is unchanged and still creates a session per call.

Safety properties (per the RFC: "may reuse transports, but must not reuse tenant-specific credentials or cookies")

• follow_redirects=False. check_url(current, allow_http=True, resolve=True) still runs on every hop. _MAX_REDIRECTS is still enforced and BlockedUrlError behaves the same.
• The timeout is applied per request (build_request(..., timeout=timeout)), so callers' timeouts are still honored on the shared client.
• No cookie persistence: the shared client's jar uses a cookie policy that rejects every cookie, so a Set-Cookie from one fetch can never reach a later fetch, tenant or workspace. Cookies set during a single redirect chain are replayed from a per-fetch jar, which matches the old per-call-client behaviour. One thing to know: httpx copies an httpx.Cookies argument into a fresh jar with the default policy, so the raw CookieJar has to be passed. The new tests caught exactly this leak during development.
• Default headers, verify=True and the proxy are unchanged. No credentials are stored on the client.

Tests

New tests/test_leadgen_http_client_reuse.py (11 tests, offline via httpx.MockTransport):

• the client is reused across fetches with the same proxy and loop
• different proxies get different clients
• clients are not shared across event loops
• the timeout is applied per request
• a Set-Cookie is not sent on a later fetch, and the shared jar stays empty
• cookies are replayed within one redirect chain only
• check_url is called on every redirect hop with allow_http=True, resolve=True
• a redirect to a private target is still blocked
• too many redirects are still rejected
• the LRU bound holds and evicted clients are closed
• aclose_plain_clients closes and clears the pool

Commands run:

• uv run pytest -q tests/test_leadgen_http_client_reuse.py tests/test_security_floor.py tests/test_browser_tier_escalation.py: 51 passed
• uv run pytest -q -x: 2226 passed, 139 skipped
• uvx ruff check on the changed files: no new findings on the changed lines. The remaining findings were already there.

🤖 Generated with Claude Code