Skip to content

docs: distinguish benchmark request and session usage #7890

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
huangruiteng merged 1 commit into from
Nov 2, 2026
Merged

docs: distinguish benchmark request and session usage #7890

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions benchmark/runtime/session_observation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
"""Content-free, bounded observations from a live Codex session rollout.

This is an operational signal for liveness and capacity control. Exact usage
accounting still uses the complete streaming reader owned by
``loopx.control_plane.quota.codex_session_usage``.
This is an operational signal for liveness and capacity control.
``usage_cumulative`` retains the native ``token_count`` basis, which may exclude
compaction. Goal usage collection uses the complete streaming reader in
``loopx.control_plane.quota.codex_session_usage`` with that same counter basis.
For recorded request/compaction coverage, use the benchmark-toolkit trajectory
capture's ``request_coverage`` projection. Session counters and reconciled
all-request totals are separate observations, not additive charges.
"""

from __future__ import annotations
Expand Down
12 changes: 12 additions & 0 deletions skills/loopx-benchmark/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,18 @@ separate: fewer bytes, calls or tokens are not proof of a better task outcome.
analysis. Do not substitute profiler hotspots for outcome evidence or require
profiling to inspect duplicate content and unnecessary interactions.

For recorded model-request totals, use `loopx benchmark trajectory-pack-register`
with `--usage-jsonl` and `--usage-session-id`, then read
`solver_usage.projection.request_coverage`. Registration does not run an analyzer;
an unfinished final Turn requires the explicit `--include-incomplete-usage`
option. Live-tail `usage_cumulative` and `token_count` observations retain the
native session-counter basis, which may exclude compaction. Only verified
`request_coverage.all_requests` supplies the reconciled recorded-request total;
retain its `counter_basis`, capture window and limitations. Compaction is a
subset and its mirrored record is not another request. Do not add either to
the session totals. Missing or unavailable coverage remains unknown. This
source selection does not rewrite Goal accounting or establish billing coverage.

## Experiment sequence

1. **Read the experiment board before launching or selecting a case.**
Expand Down
Loading