fix(security): P0-5 SSRF-safe webfetch — resolve-validate-pin transport, robots + streaming

[shoom1]

Summary

Fixes P0-5 (Critical/High) from the 2026-07-10 comprehensive review: the web_fetch path was exposed to SSRF. The validator resolved a single IPv4 via gethostbyname and httpx re-resolved the hostname at connect time (a validate→connect TOCTOU / DNS-rebinding window); AAAA and several non-global ranges were unchecked; the robots.txt fetch was a completely unvalidated request; and response bodies were read whole before any size cap. This PR centralizes transport with resolve → validate → pin semantics so the address validated is the address connected to, and every request (initial fetch, each redirect hop, robots) is pinned and validated. 7 commits; offline suite 1969 passed, 0 failed.

What's fixed

Validator (webfetch/validator.py). resolve_and_validate(host, port) resolves every A+AAAA via getaddrinfo and requires each to be globally routable — ipaddress.is_global plus a supplementary deny-list for the two SSRF-risky ranges is_global marks global (NAT64 64:ff9b::/96, 6to4-relay 192.88.99.0/24) and an IPv4-mapped-IPv6 embedded-v4 check. It rejects the whole host if any resolved IP is unsafe (defeating split-horizon / rebinding). validate(url) now does scheme / blocked-domain / IP-literal policy only (no DNS). New BlockedAddressError(httpx.RequestError).

PinnedTransport (webfetch/transport.py, new). A composition wrapper that, per request, resolves+validates the host and rewrites the request to connect to the validated pinned IP while preserving the Host header and setting sni_hostname to the original host — so TLS certificate verification stays bound to the hostname, not the IP (verified against httpcore: it sets server_hostname = sni_hostname, so the pin fails closed if ever dropped). Because it sits at the transport layer, the initial fetch, every manually-followed redirect hop, and the robots fetch are pinned+validated by construction.

Fetcher (webfetch/fetcher.py). Injects the shared transport, streams the body under a hard cap (max_content_bytes / max_pdf_bytes, buffer never grows unbounded), and removes the racy post-fetch gethostbyname re-check (the transport now guarantees the connected IP was validated). The manual redirect loop, cross-host-redirect blocking, and per-hop Location + robots revalidation are preserved.

Robots (webfetch/robots.py). Fetches robots.txt through the same pinned transport (closing the unvalidated-request hole), with a capped body; permissive-on-error behavior unchanged.

Factory (webfetch_tool.py). get_or_create_fetcher builds one validator + one PinnedTransport and shares it across the fetcher and robots. PinnedTransport.aclose() is a no-op so a per-call AsyncClient's close doesn't tear down the shared pool (fixes a concurrent-web_fetch fragility; also enables keep-alive).

Verification

• Offline suite: 1969 passed, 1 skipped, 26 xfailed, 0 failed. All new/migrated tests are fully offline — socket.getaddrinfo is monkeypatched and an httpx.MockTransport is the PinnedTransport inner, so no real DNS or sockets. The migration replaced patch("httpx.AsyncClient.get") mocks (which bypassed the transport) with the real pinned path, asserting pinning actually happens (url.host == IP, Host == original, sni == original).
• Built via subagent-driven development (per-task TDD + spec/quality review) and an opus whole-branch review that verified the load-bearing httpx/httpcore behaviors empirically and found no SSRF bypass end-to-end. Its one new finding (shared-transport concurrency) and two endorsed test-hygiene items were fixed.

Not in scope (separate cycles / deferred)

• P1-7 (http.read capability split for fetch vs. search; built-in secret-path read denials).
• web_search still uses a bare client, but its URL is the fixed Tavily/Brave API endpoint (not agent-controlled), so it is not the P0-5 SSRF vector.
• Minor deferred test-hygiene items (documented in the review), all opus-triaged as safe to defer.

This closes the last Tier-3 Critical — with P0-1, P0-2, and P0-5 done, the Tier-3 arc from the 2026-07-10 review is complete.

https://claude.ai/code/session_01SwkKXQpy3JLEqrPkQA8QRv