feat(jobs): AUTH-004 synthetic prober for the prod login loop

[mastermanas805]

Summary

• Adds a 5-minute River periodic job (AuthProbeWorker) that drives the full browser-shaped login loop against prod and pages on regression — closes the gap that hid prod login broken for ~24h on 2026-05-29 (the AUTH-004 chain: client never POSTed /auth/exchange, Accept: application/json forced rejected preflight, api response missing Access-Control-Allow-Credentials).
• Three legs: POST /auth/email/start (assert 202 + {"ok":true} + latency < 2s), OPTIONS + POST /auth/exchange (assert Access-Control-Allow-Origin ∈ allow-list AND Access-Control-Allow-Credentials: true on BOTH preflight and real POST), GET /auth/me with a probe-account bearer (assert 200 + body has email).
• Wires instant_auth_probe_outcome_total{leg,result} counter + instant_auth_probe_latency_seconds{leg} histogram; emits structured auth_probe_failed leg=... reason=... ERROR log + audit_log row (kind=auth_probe_failed, actor=system:auth_probe) on every fail outcome.

Rule-17 coverage block

Symptom:       AUTH-004 chain (3 stacked failures hid prod login broken ~24h on 2026-05-29)
Enumeration:   rg -F 'Access-Control-Allow-Credentials' under api/ +
               rg 'exchangeCookieName' under api/internal/handlers/
Sites found:   1 CORS emit site (api/internal/router/router.go),
               1 cookie name const (api/internal/handlers/auth.go),
               3 setExchangeCookie call sites (auth.go x2, magic_link.go)
Sites touched: 0 in api / instanode-web. Those fixes already shipped:
                 api PR #198 (added ACAC header),
                 instanode-web PR #150 (added POST /auth/exchange),
                 instanode-web PR #151 (dropped Accept: application/json).
               This PR adds the detection layer that should have caught all three.
Coverage test: TestAuthProbe_ExchangeHeadersMissing_FailsLeg2 fails immediately if
               api ever stops emitting ACAC. Sibling tests cover preflight 403
               (PR-#151 class), wildcard origin (CORS misconfig), 5xx email_start,
               401 me, DNS fail (histogram-pollution guard), and bearer-unset
               degraded path.
Live verified: pending first deploy. After auto-deploy lands, will append a
               prod /metrics scrape showing instant_auth_probe_outcome_total{result="pass"}>0
               to the PR description. Per CLAUDE.md rule 14, the deploy gate also asserts
               the worker /healthz commit_id matches HEAD.

Rule-25 observability surface

Surface	Location	Status
Counter	worker/internal/metrics/metrics.go::AuthProbeOutcomeTotal	included in this PR
Histogram	worker/internal/metrics/metrics.go::AuthProbeLatencySeconds	included in this PR
NR alert	infra/newrelic/alerts/auth-probe-fail.json	infra repo follow-up PR (same branch name)
Prom rule	infra/k8s/prometheus-rules.yaml::AuthProbeFail	infra repo follow-up PR
Dashboard tile	infra/newrelic/dashboards/instanode-reliability.json (3 tiles: per-leg outcomes, fails billboard, P95 latency)	infra repo follow-up PR
Catalog row	infra/observability/METRICS-CATALOG.md	infra repo follow-up PR

Infra changes live in a sibling PR on the infra repo (feat/auth-004-synthetic-prober) because the two repos auto-deploy on separate workflows.

Synthetic identity choice (leg 2)

Chose the CORS-headers-only approach over the JWT-mint alternative — asserting that OPTIONS and POST /auth/exchange carry the right Access-Control-Allow-Origin AND Access-Control-Allow-Credentials: true is the precise surface the AUTH-004 chain broke, and requires no JWT-secret distribution to the worker pod or probe-only-claim allowlist in the api (both of which would expand the secret blast radius for marginal gain over what leg 3 already covers via a real probe-account bearer).

Leg 3 (/auth/me) uses AUTH_PROBE_BEARER_TOKEN (env, k8s secret). When unset, leg 3 reports result="degraded" (not fail) so a missing config is visible in monitoring without triggering the page.

Test plan

• make gate GREEN locally (15 packages, ~95s).
• go test ./internal/jobs/ -run AuthProbe -short -count=1 GREEN (12 tests).
• CI deploy.yml test step GREEN on push.
• After auto-deploy: curl https://api.instanode.dev/healthz | jq .commit_id matches HEAD (rule 14).
• After auto-deploy: prod /metrics scrape shows instant_auth_probe_outcome_total{result="pass"} > 0 for all 3 legs within 10 min.
• Infra PR landed: kubectl get prometheusrule -n instant instant-platform -o yaml | grep AuthProbeFail returns the new rule.
• Operator wires AUTH_PROBE_BEARER_TOKEN k8s secret so leg 3 leaves the degraded state.

🤖 Generated with Claude Code