[Bug]: Intermittent Astra native stream failures lack actionable error evidence #3657

Activity

1. Ingwannu

   added

   bugSomething isn't workingSomething isn't working

   on Sep 5, 2026
2. github-actions

   added

   streamingSSE, WebSocket, terminal stream framesSSE, WebSocket, terminal stream frames

   on Sep 5, 2026
3. lidge-jun commented on Sep 5, 2026

   

   lidge-jun

   on Sep 5, 2026

   Owner

   More actions

   리뷰 · 우선순위 49 / 80

   설명

   이 이슈는 로컬에서 패치된 OpenCodex 2.35.0 위에서 네이티브 openai / gpt-6-astra 요청이 가끔 502와 upstream_server_error로 끝나고, 성공한 요청도 섞여 있다는 조사 보고입니다. 리포터가 강조한 핵심은 “아스트라가 지금 dev에서 깨졌다”가 아니라, 실패 행에 업스트림 상세 오류가 남지 않아 OCX 전송/릴레이 실패와 업스트림 실패를 구별할 수 없다는 점입니다. 샘플(약 24시간)은 성공 1,256건·실패 41건이고, 실패는 status=502, terminalStatus=failed, closeReason=terminal, adapter=openai-responses, sendCount=1, streamAborted=true, 복구 종류 없음입니다. first-output 타임스탬프가 없는 행도 있지만, 그것이 업스트림이 일을 안 했다는 뜻은 아니라고 분명히 적혀 있습니다.

   지금 dev HEAD는 ba9a45570(package 2.43.0)이고, 리포터가 따로 본 ebad8ebff보다 앞선 커밋들도 이미 들어와 있습니다. 카탈로그에는 src/codex/catalog/native-models.ts의 NATIVE_GPT6_ASTRA_MODEL(gpt-6-astra)이 핀으로 있고, src/providers/registry.ts와 가격표(src/usage/expected-prices.ts)에도 슬롯이 있습니다. 즉 “카탈로그에 아스트라가 없어서” 생기는 문제로 보이지는 않습니다. 요청 로그 쪽에는 이미 src/usage/log.ts / src/server/request-log.ts에 upstreamError 필드가 있고, SSE/페이로드에서 메시지를 뽑으면 최대 500자로 저장합니다. 그런데 스트림이 본문 오류 프레임 없이 끊기면 이 필드가 비어 upstream_server_error 코드만 남는 경로가 있습니다. 그래서 리포터가 본 “detailed upstream error: unavailable”은 현재 HEAD 계약과도 맞을 수 있습니다.

   같은 날 머지된 #3643은 업스트림 WebSocket 경로에서 HTTP 메타데이터를 보존하는 수정입니다. 전송 계층 관측에는 도움이 될 수 있지만, 요청 로그에 안전한 close/error 카테고리를 남기는 작업과는 다릅니다. #3384/#3389는 zero-output mid-stream 재시도 축인데, 이 이슈 본문이 말하듯 등치는 아직 증명되지 않았고, first-output이 없다고 해서 무조건 재실행하면 안 됩니다. 재현도 아직 결정적이지 않고, 실행 환경이 패치된 2.35.0이라 제품 회귀로 바로 받기엔 증거가 부족합니다.

   라인 - 없음(조사 이슈). 실패 샘플에 줄 번호가 없고, 공개된 재현 패치도 없다.
   경로 src/server/request-log.ts upstreamError - 파싱 가능한 업스트림 메시지/불완전 사유가 있을 때만 채운다. 본문 없이 끊긴 502·streamAborted면 코드만 남고 상세가 비는 것이 현재 동작이다.
   경로 src/lib/errors.ts - 많은 5xx가 upstream_server_error로 묶인다. 로그만 보면 업스트림·릴레이·합성 터미널이 같은 코드로 보일 수 있다.
   경로 src/server/relay.ts - 스트림 중단 시 streamAborted와 합성 502를 찍는 경로가 있다. 여기에 안전한 close 카테고리(예: ws-reset / idle-timeout / upstream-5xx-frame)를 남기지 않으면 조사 로그가 지금처럼 얇아진다.
   경로 NATIVE_GPT6_ASTRA_MODEL / registry - HEAD에는 아스트라가 이미 있다. 로컬 2.35.0 카탈로그 패치와 HEAD를 같은 버그로 묶으면 안 된다.
   관련 #3643·#3389 - 전자는 WS 메타데이터, 후자는 zero-output 재시도. 이 이슈가 원하는 “실패 증거 카테고리”와는 겹치지만 동일 수정이 아니다.

   메인테이너의 판단이 필요한 지점

   • 이 이슈를 “아스트라 회귀”가 아니라 “네이티브 스트림 실패 관측성” 버킷으로 둘지
   • 요청 로그에 넣을 최소 카테고리 집합(전송 계층 / HTTP 상태 / incomplete reason / 합성 502 여부)을 어디까지 공개할지. 원문 업스트림 본문은 넣지 않는 전제
   • 재현을 패치된 2.35.0 집계만으로 받을지, 아니면 깨끗한 2.43/dev 빌드에서 같은 비율이 나오는지 먼저 볼지
   • feat(retry): refetch once on zero-output mid-stream socket reset #3389 재시도와 묶을지, 관측성 PR을 먼저 받고 재시도는 금지(리포터 권고)를 유지할지

   너의 추천
   열어 두되 우선순위는 낮게 둔다. 다음 단계는 (1) 민감정보 없는 close/error 카테고리를 요청 로그·usage 행에 붙이는 작은 관측성 PR, (2) 가능하면 비패치 dev/2.43에서 같은 실패 비율 확인, (3) 그 증거 전에는 재시도 정책·아스트라 카탈로그 변경 PR은 받지 않는다. #3644 진단 PR(#3658)과 주제를 섞지 말 것.

   이 댓글은 grok-bot이 작성했습니다
4. GuillaumeOuint commented on Sep 6, 2026

   

   GuillaumeOuint

   on Sep 6, 2026

   More actions

   This bug make opencodex unusable with gpt-6-astra. It sometimes fail compaction with this 502 error with no further details. And as compaction don't work, the thread is "deadlocked". So a new thread must continue, and it use more quota than continue the old thread because it have to catch up with the old thread before.
5. Ingwannu commented on Sep 6, 2026

   

   Ingwannu

   on Sep 6, 2026

   OwnerAuthor

   More actions

   Thanks—compaction being the failing operation is an important additional detail. Please provide the OCX version (and whether it is locally patched), OS, the failing endpoint if known, and only the error code/status plus timing from the matching row. In particular, distinguish /v1/responses/compact from a compact-shaped request sent through /v1/responses. Do not post the conversation, encrypted payload, account identifiers, or raw capture.

   There is a separate large-compaction report in #3573 with an explicit 413 and a 256 MiB decompression-limit error. A generic 502 here is not evidence of that same cause, nor is it evidence of Astra entitlement failure. I will not recommend increasing the body limit, changing models, or replaying an already-sent request without identifying the failing boundary.

   This remains an open investigation. The earlier baseline in this issue was a locally patched 2.35.0 installation; your exact version/endpoint would help establish whether the problem also affects a current unmodified build.
6. GuillaumeOuint commented on Sep 6, 2026

   

   GuillaumeOuint

   on Sep 6, 2026

   Last edited by GuillaumeOuint

   More actions

   Thanks—compaction being the failing operation is an important additional detail. Please provide the OCX version (and whether it is locally patched), OS, the failing endpoint if known, and only the error code/status plus timing from the matching row. In particular, distinguish /v1/responses/compact from a compact-shaped request sent through /v1/responses. Do not post the conversation, encrypted payload, account identifiers, or raw capture.

   There is a separate large-compaction report in #3573 with an explicit 413 and a 256 MiB decompression-limit error. A generic 502 here is not evidence of that same cause, nor is it evidence of Astra entitlement failure. I will not recommend increasing the body limit, changing models, or replaying an already-sent request without identifying the failing boundary.

   This remains an open investigation. The earlier baseline in this issue was a locally patched 2.35.0 installation; your exact version/endpoint would help establish whether the problem also affects a current unmodified build.

   I use 2.42.0 upstream, on arch linux, codex/openai backend provider, the endpoint is response as compact don't exist anymore.
   Response code is 502 with no further details like the original post is telling.
   #3573 is more for the "deadlock" part, but the 502 is still appearing sometimes... The request fail but can continue the thread (even after compaction). PS: I use native default context, so this problem that happend with large context shouldn't occurs to me.
   Edit: my issue seems more related to openai/codex#43062
7. ryoshun49 commented on Sep 7, 2026

   

   ryoshun49

   on Sep 7, 2026

   More actions

   Adding a macOS observation in response to the request for version, endpoint, and error details.

   • Environment: macOS 26.5.1, arm64, Codex desktop, gpt-6-astra.
   • Version: OCX CLI is currently 2.46.0, but I cannot tie this failure to the running proxy build or verify its patch status. This is not a confirmed 2.46.0 reproduction.
   • Endpoint: The log records responses / openai-responses; the exact endpoint and whether compaction was involved are unknown.

   One example from the local usage log:

   UTC: 2026-09-07T06:28:59.282Z
   status=502, errorCode=upstream_server_error
   terminalStatus=failed, closeReason=terminal
   upstreamError=empty/absent
   durationMs=12325, firstOutputMs=12189
   sendCount=1, streamAborted=true, recoveryKinds=[]
   

   Other 502 rows contain usage-limit errors with terminalStatus=incomplete; I am treating those separately.

   What minimal, privacy-safe diagnostic would help identify the running build, whether compaction was involved, and the transport failure reason? No prompts, payloads, account identifiers, or raw captures are attached.
8. Ingwannu commented on Sep 7, 2026

   

   Ingwannu

   on Sep 7, 2026

   OwnerAuthor

   More actions

   Thanks for keeping the quota-incomplete rows separate and not attributing this to the CLI version alone. The minimal next diagnostic is the running/CLI version pair, not a full config or raw capture.

   Please run ocx status --json locally and share only versionSkew.cliVersion, versionSkew.proxyVersion, and versionSkew.skewed. Current source exposes these fields in src/cli/status.ts; ocx --version alone identifies only the CLI on PATH. If the fields are missing or proxyVersion is null, report that rather than restarting or repairing the service just to collect evidence. Please do not paste the complete status output, runtime files, tokens, account labels, or paths.

   For the same occurrence, keep the UTC timestamp and the scalar fields already supplied. inboundProtocol=responses identifies the API family; it does not distinguish the exact path or establish that compaction was involved. If the client already shows the request path or a compaction event, report only that path/event (no query strings or body); otherwise leave it unknown. A version observation collected now also cannot prove which build served the historical failure unless the process was unchanged.

   The provided firstOutputMs followed by a failed terminal is consistent with failure after output began, but does not establish a socket reset, an HTTP-versus-WebSocket cause, or permission to replay the request. I am leaving this open; no automatic retry change or new payload logging is justified by this row alone.
9. ryoshun49 commented on Sep 8, 2026

   

   ryoshun49

   on Sep 8, 2026

   More actions

   Thanks. I ran ocx status --json at 2026-09-08 02:02 UTC:

   {
     "versionSkew.cliVersion": "2.46.0",
     "versionSkew.proxyVersion": null,
     "versionSkew.skewed": false
   }

   proxyVersion returned null. I did not restart or repair the service to collect this.

   This version check was collected separately from the failure reported above, so it does not establish which proxy build served that request. The original failure timestamp and diagnostic fields remain as reported. The exact request path and whether compaction was involved remain unknown.
10. Ingwannu commented on Sep 8, 2026

    

    Ingwannu

    on Sep 8, 2026

    OwnerAuthor

    More actions

    Thanks — this is useful, and you are right to keep it separate from the historical failure.

    In current src/cli/version-skew.ts, computeVersionSkew explicitly returns skewed=false when proxyVersion is unavailable. Therefore this result means the version comparison could not establish a mismatch; it does not prove a version match, and null alone does not prove the proxy was stopped. src/cli/status.ts derives this version from the live runtime record, separately from its health probe.

    If you still have the JSON captured at 02:02 UTC, please share only proxy.health.ok and whether proxy.pid was non-null (a boolean is enough). No full status output, actual PID, paths, credentials, new restart or repair is needed. If that capture is no longer available, leave these unknown.

    The originating proxy build, exact request path and compaction involvement remain unproven. Keeping this issue open; this observation does not justify an automatic retry or configuration change.
11. ryoshun49 commented on Sep 8, 2026

    

    ryoshun49

    on Sep 8, 2026

    More actions

    Thanks for clarifying. I checked the collection command and retained output from 2026-09-08 02:02 UTC. Only the three versionSkew fields were retained; the full status JSON was not saved.

    For that observation:

    • proxy.health.ok: unknown
    • Whether proxy.pid was non-null: unknown

    I understand that skewed=false with an unavailable proxyVersion does not establish a version match, and that proxyVersion=null alone does not establish that the proxy was stopped. I’m leaving both requested fields unknown, as requested.
12. Ingwannu commented on Sep 9, 2026

    

    Ingwannu

    on Sep 9, 2026

    OwnerAuthor

    More actions

    The requested historical health/PID fields can remain unknown; there is no need to reconstruct them from a new process state.

    A relevant update is now on dev: #4061 was merged by @lidge-jun and preserves transportPhase/terminalSource through usage-log persistence and restart hydration. I reviewed that persistence path. It improves evidence for the next incident; it does not identify or fix the original Astra 502 and I am leaving this issue open.

    If the failure occurs again on a build containing that change, a sanitized per-request summary of the version, ingress/transport, errorCode, terminalStatus, transportPhase and terminalSource would help distinguish an upstream terminal from a proxy-generated tail. Do not post full status/config JSON, prompt content, headers or credentials, and no daemon restart is needed solely to answer this thread.
13. cb8010d6

    added a commit that references this issue on Sep 9, 2026

    fix(logs): persist transport finality evidence

    ...

    This issue will close once commit e12dfb1 is merged into the 'main' branch.

    e12dfb1
14. lidge-jun

    closed this as completedin e12dfb1on Sep 9, 2026
15. agentHits

    added a commit that references this issue on Sep 17, 2026

    fix(logs): persist transport finality evidence

    ...

    This issue will close once commit 3425438 is merged into the 'main' branch.

    3425438