Skip to content

[Bug][claude-surface] message_start reports usage.input_tokens: 0 — third-party clients reading the first frame under-report context usage (Paseo) #4857

Closed
Closed
[Bug][claude-surface] message_start reports usage.input_tokens: 0 — third-party clients reading the first frame under-report context usage (Paseo)#4857
Labels
bugSomething isn't workingstreamingSSE, WebSocket, terminal stream frames

Description

@yu381792
yu381792
opened on Sep 17, 2026

Client or integration

Claude Code

Area

Streaming

Summary

The Anthropic Messages stream emitted by opencodex reports usage: { input_tokens: 0, output_tokens: 0 } in the message_start frame. The real input size (including cache read/creation) only arrives in the terminal message_delta.usage.

Real Anthropic populates message_start.message.usage.input_tokens with the actual prompt size at the start of the turn. Clients that follow the documented contract and read usage from the first frame — e.g. Paseo's context-window meter (packages/server/src/server/agent/providers/claude/agent.ts in getpaseo/paseo: readStreamRequestInputTokens reads message_start, readStreamRequestOutputTokens reads message_delta, and the sum drives contextWindowUsedTokens) — therefore see only the output tokens of the previous turn and show a nearly-empty context ring (e.g. "a few hundred tokens used" of 1M) while /context in Claude Code correctly shows ~97k.

Claude Code itself does not rely on the first frame, so compaction and cost accounting are unaffected — this is a display-contract gap for third-party clients only.

Reproduction

  1. Start opencodex (ocx start) with any provider behind the Claude surface (reproduced with an openai-chat adapter provider; version 2.56.0, macOS, bun runtime).
  2. Send a small streaming request directly through the proxy:
curl -sN http://127.0.0.1:10100/v1/messages \
  -H "Authorization: Bearer <token>" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model":"<provider>/<model>","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"reply with one word"}]}'

Observed:

event: message_start
data: {"type":"message_start","message":{...,"usage":{"input_tokens":0,"output_tokens":0}}}
...
event: message_delta
data: {"type":"message_delta","delta":{...},"usage":{"input_tokens":17,"output_tokens":147,"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}

Expected (real Anthropic): message_start.message.usage.input_tokens already carries the full prompt size for this turn (and cache_read_input_tokens / cache_creation_input_tokens when caching applies).

  1. In a real session, ~/.opencodex/usage.jsonl confirms the terminal usage is correct — latest turn recorded inputTokens: 87567, cachedInputTokens: 57920 — while Paseo's meter showed only a few hundred tokens for the same session. So the data exists; only the first frame is zero.

Root cause (2.56.0 source)

src/claude/outbound.ts:

  • messageSnapshot(model) (L192) hardcodes usage: { input_tokens: 0, output_tokens: 0 }.
  • responsesSseToAnthropicSse emits message_start with that snapshot (L283) as soon as the stream starts, long before upstream usage is known.
  • The real usage only reaches the client in finish() via anthropicUsage() on message_delta (L355).

I understand the upstream Responses stream may not provide prompt usage until the end, so a faithful fix may not always be possible at message_start time. But two intermediate options would already unblock first-frame readers:

  • If the adapter knows the prompt size early (e.g. from an estimate or an early upstream usage frame), emit it in message_start.
  • Alternatively, emit a synthetic message_delta with a usage update as soon as input usage becomes known, since Paseo-style readers sum message_start input + latest message_delta output.

Related prior art: #508 (Kiro under-reporting fixed in v2.7.41 via contextTotalTokens) — same user-visible symptom class, different code path.

Version

2.56.0

Operating system

macOS 15.5 (Darwin 25.5.0)

Provider and model

tianyi / glm-5.3-oc (openai-chat adapter, surface: claude)

Logs or error output

event: message_start
data: {"type":"message_start","message":{"id":"msg_7629b195d7e044088b9fbda77ff5e9c4","type":"message","role":"assistant","content":[],"model":"tianyi/glm-5.3-oc","stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":0,"output_tokens":0}}}
event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"input_tokens":17,"output_tokens":147,"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}

Checks

  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.

Activity

  1. github-actions
    added
    bugSomething isn't working
    on Sep 17, 2026
  2. github-actions
    closed this as not plannedon Sep 17, 2026

  3. github-actions commented on Sep 17, 2026

    @github-actions
    github-actionsbot
    on Sep 17, 2026 – with GitHub Actions
    Contributor

    Issue reopened

    The report now contains the information required by the automated check. Thanks for updating it.

  4. github-actions
    added
    streamingSSE, WebSocket, terminal stream frames
    on Sep 17, 2026
  5. github-actions
    reopened this on Sep 17, 2026

  6. lidge-jun commented on Sep 17, 2026

    @lidge-jun
    lidge-jun
    on Sep 17, 2026
    Owner

    리뷰 · 우선순위 74 / 80

    설명

    이 이슈는 Claude surface SSE에서 message_start 프레임의 usage.input_tokens 가 항상 0으로 나가서, 첫 프레임만 읽는 서드파티 클라이언트(예: Paseo context meter)가 컨텍스트 사용량을 거의 비어 있다고 보여주는 버그 보고다. 보고자 말대로 진짜 Anthropic은 message_start.message.usage.input_tokens 에 이번 턴 prompt 크기를 넣고, message_delta.usage 는 누적/최종 쪽이다. Claude Code 본체는 첫 프레임에 의존하지 않아서 compaction·비용 계산은 안 깨지고, 표시 계약 갭이다.

    현재 dev checkout(7a29e7b6630652d5228b207b1b3244312ec336de, package 2.58.0)의 src/claude/outbound.ts 를 그대로 읽어 보면 보고의 원인 지점이 아직 그대로다. messageSnapshot(model) (약 L192–201)이 usage: { input_tokens: 0, output_tokens: 0 } 을 하드코딩하고, responsesSseToAnthropicSse 의 ensureStarted() (약 L282–286)가 스트림이 의미 출력을 내는 순간 그 스냅샷으로 message_start 를 먼저 보낸다. 실제 usage는 finish() (약 L352–361)에서 response.completed / incomplete 경로의 response.usage 를 anthropicUsage() (L90–105, cache read/write를 Anthropic 계약에 맞게 빼는 변환)로 넣어 message_delta 에만 실어 보낸다. upstream Responses가 prompt usage를 끝까지 안 주는 경우가 많아서 시작 시점에 항상 채울 수는 없다는 보고자의 이해도 코드와 맞다.

    관련 테스트 tests/claude-integration/claude-outbound.test.ts 도 message_start 의 role/model/stop_reason은 보고, usage 숫자는 주로 message_delta 쪽에서 assert한다(예: cache 매핑 케이스에서 md.usage 의 input/cache 필드). 즉 지금 suite는 첫 프레임 input_tokens=0을 허용하는 상태이고, Paseo처럼 message_start input + message_delta output을 합치는 독자는 문서 계약을 따라가도 빈 링을 본다. 증상 계급은 닫힌 #508 (Kiro under-report / contextTotalTokens) 와 비슷하지만 코드 경로는 Claude outbound translator 쪽이다 별개다.

    왜 지금 고칠 가치가 있나: opencodex가 Anthropic Messages 표면을 흉내 내는 한, 서드파티가 공개 계약을 믿을 수 있어야 한다. 데이터는 이미 터미널 프레임과 ~/.opencodex/usage.jsonl 에 있다(보고 재현). 완벽한 Anthropic 동치가 어렵다면, 보고자가 제안한 중간안 — (a) adapter가 일찍 아는 prompt size/추정치를 message_start 에 넣기, 또는 (b) input usage가 생기는 즉시 usage-only message_delta 를 한 번 더 내기 — 가 first-frame reader를 풀어 준다. 최근 tip 방향(#4860 behaviour oracle)과도 맞다: ‘0이 아닌 숫자 문자열’이 아니라 언제 어떤 프레임에 usage가 채워지는지를 테스트해야 한다.

    라인 단위 / 경로 단위 문제

    src/claude/outbound.ts L192–201 messageSnapshot - usage를 항상 0으로 고정해서 Anthropic message_start 계약을 충족하지 못한다
    src/claude/outbound.ts L282–286 ensureStarted - upstream usage를 알기 전에 message_start를 내보내므로, 나중에 알게 된 input usage를 첫 프레임에 되돌릴 경로가 없다
    src/claude/outbound.ts L352–361 finish / L618+ response.completed - usage가 여기서만 anthropicUsage로 나가 message_delta에 실림; 중간 usage-only delta 경로가 없음
    src/claude/outbound.ts L90–105 anthropicUsage - 변환 자체(캐시 차감)는 올바르고 terminal 쪽 정확도의 근거다. 문제는 변환 시점이 너무 늦다는 것
    tests/claude-integration/claude-outbound.test.ts - message_start.usage.input_tokens 기대를 거의 안 걸어서 회귀가 suite에 안 잡힌다. 0 고정을 ‘맞는 동작’처럼 고정하면 안 된다
    재현 버전 2.56.0 vs 현재 2.58.0 - 해당 하드코딩은 tip에도 남아 있어 version drift로 사라지지 않았다

    메인테이너의 판단이 필요한 지점

    • 완벽한 early Anthropic parity를 목표로 할지, Paseo급 first-frame reader만 푸는 중간안(추정 start 또는 early message_delta)으로 갈지
    • openai-chat / responses / 기타 adapter별로 early usage 가용성이 다를 때, 추정 토큰을 넣어도 되는지(과대/과소 표시 허용 범위)
    • Claude Code 비의존 이슈이므로 P1로 당장 패치할지, streaming 계약 묶음에 넣어 릴리즈 트레인에 실을지

    너의 추천

    고치는 쪽으로 진행(keep open). 중복 close 대상 아님(#508은 다른 경로). 선호 순서: (1) upstream/adapter가 input usage를 일찍 주면 message_start 에 anthropicUsage 결과를 넣고, (2) 늦게 알면 usage-bearing message_delta 를 stop_reason 없이(또는 문서화된 누적 규칙으로) 한 번 더 emit, (3) tests/claude-integration/claude-outbound.test.ts 에 ‘첫 프레임 또는 early delta에 input_tokens>0’ behaviour oracle 추가 — 0 하드코딩을 영구 정답으로 assert하지 말 것. 점수 74/80: 재현·원인·영향 범위가 tip 코드와 정확히 맞고, 고치면 서드파티 표시 계약이 바로 좋아진다.

    이 댓글은 grok-bot이 작성했습니다

  7. lidge-jun commented on Sep 17, 2026

    @lidge-jun
    lidge-jun
    on Sep 17, 2026
    Owner

    Partially fixed, and deliberately left open with a narrower scope.

    #4891 (6b6d6e6294, in 2.58.0) puts a real input-token count on message_start when the upstream sends one early — on response.created or response.in_progress. That covers the destinations that report usage up front.

    It does not cover the path this report came from. The internal bridge attaches usage: null to its lifecycle frames (src/bridge/sse.ts), so a tianyi/glm-5.3-oc request through the openai-chat adapter on the Claude surface still shows zero at the start of the turn. #4891's own description says this is by design rather than an oversight.

    Closing this now would tell the reporter their symptom was fixed when it was not. What remains is the option this issue already proposed: emit a synthetic message_delta carrying input usage at the moment it becomes known, for paths where it is not available at message_start. Scope is therefore narrowed to the internal bridge lifecycle frames; the early-usage destinations are done.

  8. lidge-jun
    added a commit that references this issue on Sep 18, 2026

    fix(claude): report the prompt this proxy counted on message_start (#…

    c5b39d4

  9. lidge-jun commented on Sep 18, 2026

    @lidge-jun
    lidge-jun
    on Sep 18, 2026
    Owner

    Fixed on dev by #5057.

    message_start now carries this proxy's own count of the prompt it forwarded, whenever the upstream reported no usage before the first content frame. #4891 had covered the destinations that report early; the internal bridge attaches usage: null to its lifecycle frames, so the path your report came from had nothing to publish and kept sending zero. The floor comes from estimateClaudeRequestTokens, the same estimate the usage log already trusts, computed once per request.

    Ordering is explicit and tested: confirmed early upstream usage always outranks the floor, and message_delta remains the authoritative count. The floor is never a claim about the upstream tokenizer.

    This narrowed a decision the repository had recorded, and the test comment now says so rather than having been quietly deleted. The pinned case read that zero is the honest placeholder and an estimate must not replace it. Its second half stands — the first content frame is still never delayed to await usage. The first half does not: zero is not honest about a prompt that exists, it asserts an empty one, and because the Anthropic schema makes usage required on message_start, "absent" was never available. The real choice was between a false measurement and a true measurement of the request side.

    A floor that measures nothing is still not published: zero, negative, and NaN all fall back to the previous zeros, so a caller that cannot count cannot turn that into a number on the wire. Your Paseo-style reader should now see the prompt size at the start of the turn instead of a nearly-empty ring.

  10. lidge-jun
    closed this as completedon Sep 18, 2026
  11. Ingwannu
    reopened this on Sep 18, 2026

  12. Ingwannu commented on Sep 18, 2026

    @Ingwannu
    Ingwannu
    on Sep 18, 2026
    Owner

    Reopening because #5057 fixes the translated streaming path but current dev still emits zero input usage for the JSON-upstream-to-streaming fallback.

    responsesJsonToAnthropicMessage() preserves known body.usage on the synthesized message. In claude-messages.ts, the fallback then replaces that known usage in message_start with { input_tokens: 0, output_tokens: 0 } and exposes the real usage only later in message_delta. Clients that read the first frame still under-report exactly as this issue describes.

    Preserve the known input/cache usage fields in message_start and set only output_tokens to zero for stream start. Add a regression with a JSON upstream carrying nonzero input and cache usage, proving message_start contains those known values and message_delta retains final output usage. The estimate added by #5057 should remain a fallback only when confirmed usage is unavailable.

  13. lidge-jun
    closed this as completedin c5b39d4on Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstreamingSSE, WebSocket, terminal stream frames

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions