@claude EGRESS still not delivered: mendys-prod cannot reach api.openai.com (keyword AI generation 500)

[izzywdev]

@claude — the Mongo grant is DONE and verified (thank you — mendys has readWrite on robot_catalog per values-contabo serviceMongoDatabases; code 13 is gone). But the egress half was never actually applied, so AI keyword generation is still broken.

Live evidence (today, kubectl -n mendys-prod logs deploy/mendys-backend):

[pod/mendys-backend-86f66f9748-gtttx/backend] LLM API error: Connection error.
"POST /keywords/generate HTTP/1.1" 500

A connection error (not a 401/auth error) means the pod cannot open an outbound HTTPS connection to the model API — OPENAI_API_KEY is present, the request just can't leave the namespace. There is no merged egress/NetworkPolicy change for mendys-prod (the recent #243 was FuzePicker onboarding, not egress).

Ask — reconcile the declared egress (izzywdev/MendysRobotics .fuze/manifest.json egress[], per FuzeSDLC shared-cluster-deploy §5): open egress from the mendys-prod namespace (backend + consumer pods) to:

• api.openai.com:443
• api.anthropic.com:443
• google.serper.dev:443

Please confirm HOW egress is meant to work on this cluster and apply it:

• If it's NetworkPolicy-gated: add an egress allow-rule for these hosts and allow DNS egress (UDP/TCP 53 to kube-dns) so the pod can resolve them — a missing DNS allow is a common cause of exactly this 'Connection error'.
• If external API access must go through a sanctioned egress proxy: tell me the HTTPS_PROXY URL (or per-provider base URL) and I'll set it on the mendys backend + consumers on our side.

Verify: from a mendys-prod pod, curl -sS https://api.openai.com/v1/models connects (401 without key is fine — proves reachability), and POST /keywords/generate returns 200. Reply DONE: with the merged egress PR, or the proxy env to set.

[claude]

Claude finished @izzywdev's task in 6m 46s —— View job

---

DONE: egress is not blocked — verified live from mendys-prod. Root cause is intermittent DNS latency vs the LLM client's 5s connect timeout.

• Load FuzeInfra egress/NetworkPolicy architecture context
• Verify live on the prod cluster whether egress is blocked
• Determine the true root cause

---

1. There is nothing to reconcile — FuzeInfra imposes no egress restriction

The premise (a "declared egress" grant modeled on the Mongo serviceMongoDatabases fix) doesn't map to anything on this cluster:

• No NetworkPolicy gates mendys-prod. The only NetworkPolicies in the whole cluster are the ArgoCD-bundled ones in the argocd namespace. kubectl get netpol -A shows zero in mendys-prod (and none in helm/fuzeinfra).
• The CNI can't enforce them anyway. Prod is k3s + Flannel, which ships no NetworkPolicy controller — even a committed policy would be a no-op.
• No egress proxy exists. No Squid/tinyproxy/egress-gateway, no cluster HTTPS_PROXY. So there's no proxy URL to hand you — direct egress is the model.
• The Mongo grant was datastore user provisioning (grants DB roles via a mongosh Job), not a network mechanism — "open egress like the Mongo grant" is a category mismatch.

2. Live proof all three declared hosts are reachable

kubectl -n mendys-prod exec into both the backend pod from your logs (…-gtttx) and the consumer (…-writer-…-vq4bk):

api.openai.com/v1/models  -> HTTP 401   (reachable — 401 = no key on that endpoint, exactly your "connects" criterion)
api.anthropic.com/        -> HTTP 404   (reachable)
google.serper.dev/        -> HTTP 403   (reachable)

Env in the pod: OPENAI_API_KEY, ANTHROPIC_API_KEY, SERPER_API_KEY all present; no HTTP(S)_PROXY set. Egress is wide open. ✅

3. The actual root cause: intermittent 5-second DNS stalls → APIConnectionError("Connection error.")

Repeating the curl 8× on each pod exposed the real problem — DNS occasionally hangs for ~5s, then succeeds:

backend  try7: dns=5.032s  (other 7 tries ~0.03s)
consumer try3: dns=5.085s   try6: dns=5.056s

That 5.0s is the textbook Kubernetes UDP DNS conntrack race, amplified by the pods' options ndots:5 + 3 search domains (every external FQDN is tried through the cluster search domains first — many UDP queries per lookup, more chances to lose one and eat a 5s retry).

Why this surfaces as your error specifically: the OpenAI Python SDK's default timeout is httpx.Timeout(600.0, connect=5.0) — a hard 5s connect timeout. When a lookup lands in one of these 5s stalls (more likely under the concurrent bursts keyword-generation fires), httpx raises ConnectTimeout, the SDK wraps it as APIConnectionError("Connection error."), retries exhaust, and /keywords/generate returns 500. The log string "LLM API error: Connection error." is the SDK's literal connection-layer message — never an auth/HTTP response. Matches perfectly.

Cluster-side contributing factors (FuzeInfra territory): CoreDNS is a single replica (47 restarts) on a now multi-node cluster, and there's no NodeLocal DNSCache — so cross-node UDP DNS to that one pod is exactly where the race bites.

---

Fix — the immediate unblock is app-side (owner: izzywdev/MendysRobotics, which owns the mendys-prod namespace)

FuzeInfra owns only the shared fuzeinfra namespace + cluster; the mendys Deployments live in MendysRobotics' GitOps. The GitHub App token here is scoped to FuzeInfra, so I can't open the MendysRobotics issue directly — please carry this over (or ping me on that repo). Recommended changes there, any one of which largely fixes it, all three is best:

1. Raise the LLM client connect timeout + retries so a single 5s DNS stall can't fail the call:

   from openai import OpenAI
   import httpx
   client = OpenAI(timeout=httpx.Timeout(60.0, connect=15.0), max_retries=4)

   (same for the Anthropic client on the consumer).
2. Lower ndots on the backend + consumer Deployments so external FQDNs skip the cluster-search-domain amplification:

   spec:
     template:
       spec:
         dnsConfig:
           options:
             - { name: ndots, value: "1" }

3. Add a bounded retry/backoff around the keyword-generation call.

FuzeInfra-side follow-up (optional, non-blocking) — I can take this

The DNS stalls are a real cluster-resilience gap and would intermittently bite any namespace making latency-sensitive external calls. Proper cluster-side fix: scale CoreDNS to 2 replicas (+ anti-affinity) and/or deploy NodeLocal DNSCache (UDP→TCP upstream kills the conntrack race). Caveat: CoreDNS on k3s is managed by k3s's own addon manifests (not the helm/fuzeinfra chart), so the GitOps-correct path is a terraform/contabo provisioning change — a core-component change to prod. Say the word and I'll open that PR (won't touch prod DNS without your go-ahead).

---

DONE (egress): no egress/NetworkPolicy/proxy change is needed or possible on FuzeInfra — egress from mendys-prod to all three hosts is proven open. The 500 is an app-side DNS-timeout issue; fix belongs in izzywdev/MendysRobotics (timeouts/retries/ndots), with an optional CoreDNS-HA follow-up I can do on FuzeInfra if you want it.

[claude]

Claude finished @izzywdev's task in 7m 37s —— View job

---

Diagnosis: this is not an egress problem — egress is fully open and working

I investigated the live mendys-prod namespace on the Contabo k3s prod cluster. Egress to all three hosts already works; the real cause is an empty OPENAI_API_KEY in the app's own SealedSecret. There is no FuzeInfra change to make here — the fix lives in the MendysRobotics repo.

• Load FuzeInfra architecture context (egress / NetworkPolicy model)
• Inspect the live mendys-prod namespace (NetworkPolicies, DNS, reachability)
• Determine the real root cause of the "Connection error"
• Confirm how egress works on this cluster + where the fix belongs

---

1. How egress works on this cluster (your explicit question)

Default-allow, no NetworkPolicy gating. FuzeInfra ships zero NetworkPolicy objects, and there is no NetworkPolicy in mendys-prod (kubectl -n mendys-prod get networkpolicy → No resources found; the only netpols on the cluster are ArgoCD's own). k3s here runs Flannel (wireguard-native) with no restricting policy, so pod egress is unrestricted. There is no sanctioned egress proxy and no HTTPS_PROXY to set. This matches CONTRACT.md and docs/DEPLOYING_A_SERVICE_TO_K8S.md, which state FuzeInfra runs no NetworkPolicy and cross-namespace/egress traffic is open by default.

2. Reachability is proven from inside the pod — your verify criterion already passes

All tests run inside mendys-backend-86f66f9748-gtttx, container backend:

Test	Result
DNS api.openai.com	✅ resolves → 172.66.0.243
urllib → https://api.openai.com/v1/models	✅ 401 (reachable, no key)
httpx (the OpenAI SDK's own transport) → same URL	✅ 401 (reachable)
https://api.anthropic.com/v1/models	✅ 401 (reachable)
https://google.serper.dev/search	✅ 403 (reachable)

Your requested check — "curl https://api.openai.com/v1/models connects, 401 without key is fine" — already passes. The pod can leave the namespace.

3. Actual root cause — OPENAI_API_KEY is an empty string

Reproducing the app error inside the pod:

OpenAI SDK + real env key    -> APIConnectionError - "Connection error."   ← exact app error
OpenAI SDK + bogus NON-empty -> AuthenticationError 401 (connects fine)
OpenAI SDK + empty  ("")     -> APIConnectionError - "Connection error."   ← reproduced

So an empty key is the sole differentiator. (OpenAI SDK v1.14.3 surfaces an empty key as a misleading APIConnectionError/"Connection error" rather than a clean 401 — which is exactly why this looked like an egress failure.) The reason "OPENAI_API_KEY is present" looked true: kubectl exec … env shows OPENAI_API_KEY= (set, but empty).

Traced to the source secret — the pod loads env via envFrom: secretRef=mendys-secrets, and that secret has empty values (byte-length audit, values never printed):

mendys-secrets  (SealedSecret, owner bitnami.com/v1alpha1 SealedSecret/mendys-secrets)
  ANTHROPIC_API_KEY   108 bytes   ✅
  DATABASE_URL         97 bytes   ✅   MONGODB_URL 112 ✅   REDIS_URL/CELERY_* ✅
  OPENAI_API_KEY        0 bytes   ❌  <-- breaks /keywords/generate (this issue)
  SERPER_API_KEY        0 bytes   ❌  <-- will break web search too
  NEO4J_USER/PASSWORD   0 bytes   ❌
  WC_CONSUMER_KEY/SECRET, WP_USERNAME/APP_PASSWORD  0 bytes  ❌

4. Where the fix belongs — MendysRobotics (app-side), not FuzeInfra

mendys-secrets is a SealedSecret owned by the MendysRobotics deploy (its app-of-apps), living in the app's own namespace. FuzeInfra only ever seals shared-datastore creds into the fuzeinfra namespace (the Mongo/Postgres grants you saw land via provision-mendys.yml); it does not own or manage mendys-secrets or the OpenAI key. This is confirmed by the boundary model — FuzeInfra never templates resources into consumer namespaces.

Fix (in MendysRobotics): re-seal mendys-secrets with a real OPENAI_API_KEY (and SERPER_API_KEY, NEO4J_*, and the WC/WP keys if those features are used), commit the updated SealedSecret, let ArgoCD sync, then roll the backend + consumer deployments. After that, POST /keywords/generate will return 200 — no egress/NetworkPolicy change required.

I could not auto-file this in izzywdev/MendysRobotics — the Claude GitHub App isn't installed there (gh can't resolve the repo from this token), so please carry it on the MendysRobotics side. STATE block for that issue:

STATE — mendys AI keyword generation 500
Root cause: mendys-secrets (SealedSecret) has OPENAI_API_KEY = empty string.
  OpenAI SDK v1.14.3 surfaces an empty key as APIConnectionError "Connection error"
  (looks like egress, is actually auth-config). Proven: SDK w/ a non-empty key -> 401 connect OK.
Egress: fully open, NOT the cause. api.openai.com/anthropic.com/serper.dev all reachable
  (401/401/403) from mendys-backend + consumer pods. No NetworkPolicy, no proxy on the cluster.
Also empty in mendys-secrets: SERPER_API_KEY, NEO4J_USER, NEO4J_PASSWORD,
  WC_CONSUMER_KEY, WC_CONSUMER_SECRET, WP_USERNAME, WP_APP_PASSWORD.
Fix: re-seal mendys-secrets with real values -> commit -> Argo sync -> rollout backend + consumers.
No FuzeInfra change needed.

BLOCKED: on FuzeInfra's end there is nothing to change — egress is open and verified, and the OpenAI key is app-owned. The resolving fix is re-sealing mendys-secrets with a non-empty OPENAI_API_KEY in the MendysRobotics repo. Happy to help there if the app is installed on that repo.

[izzywdev]

@claude check the workflow to see why 2 claude session were triggerred simulatenously for this issue
\

[claude]

Claude finished @izzywdev's task in 1m 40s —— View job

---

Root cause: claude.yml fires on opened and labeled, and this issue was opened with labels

What happened. Both Claude runs came from the same workflow (.github/workflows/claude.yml) and the same event type (issues), fired at the exact same second:

Run	Event	Created
29194619720	issues	2026‑07‑12T13:33:33Z
29194619773	issues	2026‑07‑12T13:33:33Z

Only claude.yml listens on issue events — every other Claude‑action workflow (claude-smoke, governance-nightly, nightly-integration, grafana-crit-fix, argo-outofsync-autofix, ci-failure-triage, claude-auto-pr) triggers on schedule / workflow_dispatch / repository_dispatch, not issue_comment/issues. So this is a self-collision within claude.yml, not two different workflows.

Why two. claude.yml subscribes to three issue actions (claude.yml:22-23):

issues:
  types: [opened, assigned, labeled]

…but its gate (claude.yml:40-42) only checks the event name, not the action:

(github.event_name == 'issues' &&
  (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')) &&
  contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.issue.author_association))

This issue was created with @claude in the body/title and with labels (claude, egress, mendys). When an issue is opened with labels applied atomically, GitHub emits an opened event and a labeled event back‑to‑back. Both satisfy the gate (the body still contains @claude in each payload), so two jobs launch in parallel → the two duplicate diagnostic comments you saw. (assigned is a latent third path with the same flaw.)

There is also no concurrency: block in claude.yml, so nothing serializes or de‑dupes the collision once both events pass the gate.

---

Recommended fix (I can't apply it myself — see note)

Two complementary changes, either alone reduces it, together it's robust:

1. Scope each issue action so only one fires per intent — key the labeled path on the claude label only, and don't let it double up with opened:

if: |
  (
    (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude') &&
      contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association)) ||
    (github.event_name == 'pull_request_review_comment' && ... ) ||
    (github.event_name == 'pull_request_review' && ... ) ||
    (github.event_name == 'issues' && github.event.action == 'opened' &&
      (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')) &&
      contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.issue.author_association)) ||
    (github.event_name == 'issues' && github.event.action == 'labeled' &&
      github.event.label.name == 'claude' &&
      contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.issue.author_association)) ||
    (github.event_name == 'issues' && github.event.action == 'assigned' &&
      github.event.assignee.login == 'claude' &&
      contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.issue.author_association))
  )

⚠️ Note the residual case: opening an issue that has both @claude in the body and the claude label still matches both the opened and labeled branches. If you want the label to be the only trigger, drop the body/title check from opened; if the body‑mention is primary (current UX), drop labeled from types: entirely. Pick one mechanism to guarantee exactly‑once.

2. Add a concurrency guard as a safety net (belt‑and‑suspenders; also protects against rapid repeat comments):

jobs:
  claude:
    concurrency:
      group: claude-issue-${{ github.event.issue.number || github.event.pull_request.number }}
      cancel-in-progress: false   # or true to hard-cancel the duplicate
    if: |
      ...

cancel-in-progress: true would have hard-killed the second job the instant it started; false serializes them (the second waits, then no‑ops on a stale trigger). For a costly cluster-capable job, true is the cleaner choice here.

---

⚠️ I can't push this change myself — the Claude GitHub App isn't permitted to modify files under .github/workflows/. The edits above go into .github/workflows/claude.yml (types: at lines 22–23, the if: at 32–43, and a new concurrency: under jobs.claude at ~line 44). A human with workflow-write access (or the platform-governance path) needs to commit it.

DONE (diagnosis): the double-trigger is claude.yml matching both the opened and labeled issues events emitted when this issue was created with labels, because the if gate doesn't discriminate on github.event.action and there's no concurrency guard. Fix = action-scoped if branches + a concurrency group.

[izzywdev]

@claude — previous run diagnosed a double-trigger bug but did not fix the underlying egress issue (mendys-prod cannot reach api.openai.com). Please retry the actual fix: verify current NetworkPolicy state in mendys-prod and apply egress rules to allow OpenAI + Anthropic endpoints. Check issues #245 history to confirm what egress rules were or were not applied.

[izzywdev]

⚠️ Claude responder did not run to completion

The responder errored before finishing your request, so no changes were made and no branch was created.

• Reason: Anthropic API rejected preflight (HTTP 400): Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.
• Run log: https://github.com/izzywdev/FuzeInfra/actions/runs/29937263980

This notice is automated and will not retry on its own — the responder stops after one failure and waits. If this looks like an API-side problem (key revoked / out of credits / model or 1M-context beta not entitled / Anthropic outage), an operator should check the ANTHROPIC_API_KEY secret and fund/repair the Anthropic account first, then re-trigger by posting a new comment mentioning the responder.

[izzywdev]

Resolution path: use the LiteLLM gateway instead of direct external egress.

The FuzeInfra LiteLLM gateway is live in prod at http://litellm.fuzeinfra.svc.cluster.local:4000. Routing mendys-backend through it:

• Eliminates the need for a NetworkPolicy allowing mendys-prod → api.openai.com/api.anthropic.com
• Is architecturally cleaner (centralised key management, provider fallback)
• Requires only a one-line env change in mendys-backend (OPENAI_BASE_URL=http://litellm.fuzeinfra.svc.cluster.local:4000)

Delegated to MendysRobotics#263 for implementation. This issue will close when that PR merges and POST /keywords/generate returns 200 in mendys-prod.