Skip to content

Streaming /v1/messages: message_start reports input_tokens 0 and no cache fields #93

Closed

Description

@gdevenyi

Streamed /v1/messages sends message_start with usage = {"input_tokens": 0, "output_tokens": 0} and no cache fields, also with --enable-cache-report. The real prompt and cache counts arrive only in the final message_delta. Clients that read input usage from message_start see 0 on every streamed turn.

Upstream: FlashML-org#718 (repro, cause and proposed fix). The repro there gives the same output on 5080deploy c23d2d5:

cache_report=True message_start: {'input_tokens': 0, 'output_tokens': 0}
cache_report=True message_delta: {'input_tokens': 176, 'output_tokens': 5, 'cache_read_input_tokens': 1024}

Proposed: emit a prompt-admitted event from _generate_events_impl when the admission ack arrives, and send message_start with the real usage on it.

Activity

  1. gdevenyi commented on Oct 20, 2026

    @gdevenyi
    OwnerAuthor

    Fixed by 76f6f1e on 5080deploy (round 7, PR to follow in the round-7 rebuild PR); message_start now carries the admitted prompt usage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions