diff --git a/benchmark/runtime/session_observation.py b/benchmark/runtime/session_observation.py index 023240ad6d..b1dd0b3a1c 100644 --- a/benchmark/runtime/session_observation.py +++ b/benchmark/runtime/session_observation.py @@ -1,8 +1,12 @@ """Content-free, bounded observations from a live Codex session rollout. -This is an operational signal for liveness and capacity control. Exact usage -accounting still uses the complete streaming reader owned by -``loopx.control_plane.quota.codex_session_usage``. +This is an operational signal for liveness and capacity control. +``usage_cumulative`` retains the native ``token_count`` basis, which may exclude +compaction. Goal usage collection uses the complete streaming reader in +``loopx.control_plane.quota.codex_session_usage`` with that same counter basis. +For recorded request/compaction coverage, use the benchmark-toolkit trajectory +capture's ``request_coverage`` projection. Session counters and reconciled +all-request totals are separate observations, not additive charges. """ from __future__ import annotations diff --git a/skills/loopx-benchmark/SKILL.md b/skills/loopx-benchmark/SKILL.md index 07db8a72c3..3d146d4943 100644 --- a/skills/loopx-benchmark/SKILL.md +++ b/skills/loopx-benchmark/SKILL.md @@ -125,6 +125,18 @@ separate: fewer bytes, calls or tokens are not proof of a better task outcome. analysis. Do not substitute profiler hotspots for outcome evidence or require profiling to inspect duplicate content and unnecessary interactions. +For recorded model-request totals, use `loopx benchmark trajectory-pack-register` +with `--usage-jsonl` and `--usage-session-id`, then read +`solver_usage.projection.request_coverage`. Registration does not run an analyzer; +an unfinished final Turn requires the explicit `--include-incomplete-usage` +option. Live-tail `usage_cumulative` and `token_count` observations retain the +native session-counter basis, which may exclude compaction. Only verified +`request_coverage.all_requests` supplies the reconciled recorded-request total; +retain its `counter_basis`, capture window and limitations. Compaction is a +subset and its mirrored record is not another request. Do not add either to +the session totals. Missing or unavailable coverage remains unknown. This +source selection does not rewrite Goal accounting or establish billing coverage. + ## Experiment sequence 1. **Read the experiment board before launching or selecting a case.**