Backend recovery check: mendys-prod degraded after cluster outage (524s, runners down)

[izzywdev]

@claude Please check the shared cluster's recovery state — MendysRobotics prod is only half-recovered after today's outage (began ~14:46 UTC, partially restored ~15:40).

Observed from the MendysRobotics side (16:15–16:30 UTC):

• Edge/ingress is fine: Cloudflare + Traefik serve both SPAs; the WordPress storefront responds.
• live.mendysrobotics.com portal shell loads, but the backend API (mendys-backend, ns mendys-prod) intermittently stops responding — Cloudflare 524 origin timeouts (Ray a1fc02708dcd6108, 16:22:00 UTC); some API calls hang indefinitely.
• Self-hosted ARC runners (mendysrobotics + FuzeInfra cluster-query) are not picking up jobs — every relay/cluster-query queued forever (e.g. FuzeInfra cluster-query run 30021981643 never started). Prod introspection from our side is dead.

Requested (read-only checks + whatever remediation is on your side):

1. kubectl -n mendys-prod get pods -o wide — are backend/consumer pods Running and Ready? Node pressure/evictions after the outage?
2. ARC runner namespaces — why aren't runners registering/picking up jobs?
3. kubectl -n fuzeinfra get pods — datastores (Postgres/Mongo/Kafka/Redis) healthy? The backend 524s suggest it may be blocked on a datastore.
4. Anything needing action on the MendysRobotics side (e.g. rollout restart via GitOps), please report back on this issue.

Context: chart appVersion 1.8.11 was merged ~15:02 UTC but the outage began at 14:46, before the roll — we did not roll back. Reported from a MendysRobotics agent session (repo-scoped; cannot operate the cluster).

[izzywdev]

⚠️ Claude responder did not run to completion

The responder errored before finishing your request, so no changes were made and no branch was created.

• Reason: Anthropic API rejected preflight (HTTP 400): Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.
• Run log: https://github.com/izzywdev/FuzeInfra/actions/runs/30031488227

This notice is automated and will not retry on its own — the responder stops after one failure and waits. If this looks like an API-side problem (key revoked / out of credits / model or 1M-context beta not entitled / Anthropic outage), an operator should check the ANTHROPIC_API_KEY secret and fund/repair the Anthropic account first, then re-trigger by posting a new comment mentioning the responder.

[izzywdev]

Status check 2026-07-30:

• ARC runners: recovered — recent FuzeInfra workflows complete normally on self-hosted runners; the perpetual-queue symptom is gone.
• Datastores: no current evidence of pressure (cluster has been stable since the Longhorn/OOM incidents were resolved).
• mendys-prod crashloop (root cause of the 524s): the mendys_svc Postgres credential mismatch is still open — tracked in izzywdev/MendysRobotics#258. The initial @claude run there failed at 0s; re-nudged @claude in that issue just now.

Keeping this open until MendysRobotics#258 resolves and the 4 crashlooping pods (mendys-bom-service, mendys-datasets, mendys-design, consumer-attribute-proposals-writer) are confirmed Running with restarts frozen. No FuzeInfra-side change required.

[izzywdev]

⚠️ Claude responder did not run to completion

The responder errored before finishing your request, so no changes were made and no branch was created.

• Reason: Anthropic API rejected preflight (HTTP 400): Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.
• Run log: https://github.com/izzywdev/FuzeInfra/actions/runs/30493690090

This notice is automated and will not retry on its own — the responder stops after one failure and waits. If this looks like an API-side problem (key revoked / out of credits / model or 1M-context beta not entitled / Anthropic outage), an operator should check the ANTHROPIC_API_KEY secret and fund/repair the Anthropic account first, then re-trigger by posting a new comment mentioning the responder.

[izzywdev]

Status as of 2026-08-02:

The mendys-prod recovery is pending resolution of the credential mismatch root cause. Current state:

• MendysRobotics#258 (Postgres password fix) — open, @claude re-nudged
• MendysRobotics#263 (LiteLLM egress fix for AI keywords) — open, just delegated
• ARC runners: recovered per earlier observation in this session

Leaving open as a tracker until MendysRobotics#258 and #263 merge and mendys-prod pods stabilize. Once those merge: verify kubectl -n mendys-prod get pods shows all deployments stable, then close this issue.

[izzywdev]

⚠️ Claude responder did not run to completion

The responder errored before finishing your request, so no changes were made and no branch was created.

• Reason: Anthropic API rejected preflight (HTTP 400): Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.
• Run log: https://github.com/izzywdev/FuzeInfra/actions/runs/30754768287

This notice is automated and will not retry on its own — the responder stops after one failure and waits. If this looks like an API-side problem (key revoked / out of credits / model or 1M-context beta not entitled / Anthropic outage), an operator should check the ANTHROPIC_API_KEY secret and fund/repair the Anthropic account first, then re-trigger by posting a new comment mentioning the responder.