[dreamtalen]

Why and what changed

With engine.detokenization a closed-loop client sends the request that waited on a finished response as soon as it receives the end of that response. A real load generator such as AIPerf first parses the response's chunks and returns the concurrency slot to its scheduler, which takes time that grows with the response length. We found this gap while running the paper's vLLM replay experiments.

• traffic.load.response_overhead (base_ms, per_output_token_us) releases a request that waits on a finished response (a concurrency slot's next request, a session's next turn, a dependent agentic request) that long after the client receives the end of the response.
• It is a cost of the load generator, not an authored wait like AIPerf's think time or our inter_turn_delay_ms: a session's next turn is sent at the response end, plus the response overhead, plus inter_turn_delay_ms.
• A request that ends without output (a rejection, or a failed or canceled P/D handoff) reaches the client when it ends and waits base_ms.
• Builds on engine.detokenization but also works alone: without detokenization the client receives every output and terminal at its pass end, and acts on a finished request after the overhead.
• Worker time and gpu_hours include the overhead between requests but, like duration_ms, end with the last response.
• Off by default and keeps outputs request for request. Offline token replay on --stack engine only; online replay, worker scaling, AFD, analytical EPD, agentic profiles and SDK workloads without source_type are rejected. Rust: Replayer::with_response_overhead.
• A missing collector output length is now an error instead of a silent zero.

Review map

• Risk level: medium (opt-in; touches the release path of closed-loop, session and agentic traffic)
• Stacked on feat: model response detokenization in replay #634: review only the top commit (or this PR against its base branch).
• Start with: crates/core/src/replay/components/client_path.rs (ResponseOverhead, ClientStep::{Receive, Act}), then crates/core/src/replay/agg.rs and disagg.rs (terminal release, client_overlap, only_response_overhead_in_flight, commit_response_overhead_tail), then crates/core/tests/replay_detokenization.rs.
• Public or serialized contract changed: yes, additive. traffic.load.response_overhead YAML block, Replayer::with_response_overhead.
• Compatibility or rollback concern: none when unset.

Evidence

• Tests (Linux x86 VM, Rust 1.96.1, commit f188f46 on 4f6264f): cargo fmt --all -- --check; cargo test --workspace (core 2,373 passed) and cargo test --workspace --features embed-python,replay-bench with AIC_REQUIRE_EMBEDDED_ROUND_TRIP=1 (core 2,461 passed) pass with no failures; cargo test --manifest-path crates/tests/public-api/Cargo.toml passes (21). Perfmodel parity suites pass with goldens unchanged (test_engine_step_parity.py 304, test_compile_engine_parity.py 85). ruff check, ruff format --check and scripts/ci/check_packaged_legal_files.py pass. Python tests/ and python/aisimulate/tests -m unit fail exactly the same tests as main on that host (204 and 72; they fail on main too) and run 29 new cases.
• Fast CI:
• Full CI:
• New tests: the_client_releases_its_next_request_after_receiving_and_response_overhead, a_longer_response_has_a_longer_response_overhead, a_rejected_request_waits_for_the_response_overhead, a_rejection_holds_one_slot_until_the_client_acts_on_it (concurrency 2), worker_time_ends_with_the_last_response_not_its_response_overhead, zero_detokenization_cost_and_response_overhead_keep_pass_end_delivery, invalid configs (all in aggregated and disaggregated replay); ClientPath unit tests (overhead alone keeps outputs at the pass end, Receive then Act, atomic rejection of a non-finite action time); Python compiler/runner cases, rejected modes and a native end-to-end replay.
• Calibration (independent of the evaluation sweep): the same two B200 calibration launches as engine.detokenization. Response end → next request start at concurrency 1 (54 requests, 64–2048 output tokens) fits base_ms 1.40 + per_output_token_us 3.65 (RMSE 0.31 ms). Higher-concurrency points measure 0.2–3.4 ms above the c1 fit (AIPerf's scheduler contends with in-flight streams); this PR keeps the c1 fit.
• Exp1 (MiniMax-M3-NVFP4, TP4 B200, 18 configs, 10,220 requests), MAPE over 9 concurrency points, TTFT mean / p50 / p90 · TPOT mean · throughput, with feat: model vLLM async-scheduling admission in replay #624 and engine.detokenization on both sides, without → with response overhead (plus the measured 0.255 ms request send leg in engine.tokenization):
  • G1k: 7.9 / 4.6 / 11.2 · 4.2 · 4.6 → 7.6 / 5.3 / 10.7 · 4.1 · 4.5
  • G8k: 4.4 / 5.9 / 8.8 · 1.5 · 1.6 → 4.2 / 5.9 / 7.9 · 1.5 · 1.6
  • Response overhead alone, without detokenization: G1k 10.3 / 8.8 / 11.1 · 3.5 · 3.9 and G8k 5.3 / 8.4 / 11.0 · 1.5 · 1.6, against 11.1 / 8.7 / 16.7 · 3.7 · 4.1 and 6.3 / 9.2 / 13.7 · 1.5 · 1.7 without either.
• With the overhead off, the 18 Exp1 replays equal the detokenization-only replays request for request; all numbers above reproduce request for request from the merge of this PR and feat: model vLLM async-scheduling admission in replay #624.
• Cost: one heap entry and one hash lookup per finished request. Native wall_time_ms, release builds of f45d14a (main) and this PR on an idle VM, paired alternating runs. Off: Exp1 1k c1 / 1k c256 / 8k c64 median ratio to main 0.949 / 0.965 / 1.006 (5 runs). Synthetic 512/128 closed loops of 1k / 4k / 16k and a 400 req/s Poisson overload (3 runs, every request completed, same simulated outcome across variants), host cost per output token main → this PR off → with detokenization and response overhead: 0.49 → 0.51 → 0.53, 0.66 → 0.67 → 0.61, 0.72 → 0.68 → 0.66, 0.72 → 0.67 → 0.67 µs.
• Reviews: fresh Claude reviews (rudy-review) before and after the rename from turnaround. Fixed: gpu_hours tail, docs rows, a request held by both the disaggregated flow and the client is counted once against the concurrency limit, leftover naming. Documented: requests ending without output wait base_ms.

Modeling or data provenance

• One base + per-output-token cost; the concurrency-dependent part of AIPerf's overhead is not modeled.

Tracking

• Closes: N/A
• Related PRs or issues: stacked on feat: model response detokenization in replay #634 (engine.detokenization); Exp1 numbers also include feat: model vLLM async-scheduling admission in replay #624.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}