Skip to content

docs: distinguish benchmark request and session usage - #7890

Merged
huangruiteng merged 1 commit into
mainfrom
codex/benchmark-request-usage-source-20261102
Nov 2, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/benchmark-request-usage-source-20261102

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Live session counters retain the native token_count basis, which can exclude compaction responses. Reading them as recorded request totals understates that separate measurement even though the existing trajectory reader already supplies reconciled request coverage.

Clarify the existing benchmark workflow: register a trajectory usage capture, explicitly opt into an unfinished tail when applicable, and read verified solver_usage.projection.request_coverage.all_requests with its counter basis, window and limitations. Keep session counters, request coverage and billing coverage distinct; compaction subsets and mirrored records are not additional charges. The live-observer module documentation now makes the same distinction.

Validation at c90f3a83342770fc12d37660d334c6e6df6a838a:

  • 14 focused Python usage and session-observation tests passed.
  • 15 TypeScript request reconciliation and complete/observed-tail tests passed.
  • The documented source CLI options were verified; the observer executable AST is unchanged.
  • Standard premerge: all 9 selected canaries and 5 direct checks passed. The initial semantic smoke failed because the clean checkout lacked declared Node development dependencies; it passed after npm ci --ignore-scripts. The benchmark-sensitive maintainer-review hold remains.
  • Diff-scoped semantic advisory found no supported new vocabulary; the public-boundary scan found no private additions. No RFC pair changed.

This is a documentation correction in the existing benchmark-toolkit workflow. It adds no runtime behavior, quota-accounting change, protocol field, evaluation or model call. It does not qualify billing completeness or experiment benefit. No installed behavior or UI layout changes.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1) 问题定义(为什么改、要解决什么)

  • 明确区分实时会话观测(session_observation.py 的 usage_cumulative / token_count 基准)与轨迹请求级覆盖统计(trajectory-pack-register 的 request_coverage.all_requests),防止分析时将两者重复累加或漏记 compaction 请求成本。

2) 改动摘要(按模块分组)

  • 观测口径文档与 Skill 规范 (benchmark/runtime/session_observation.py, skills/loopx-benchmark/SKILL.md):说明 usage_cumulative 沿用原生 token_count 口径(可能不含 compaction),而完整请求与 compaction 覆盖应通过 loopx benchmark trajectory-pack-register --usage-jsonl ... 读取 solver_usage.projection.request_coverage,并明确禁止将两者相加。

3) 逐点审阅结论(架构、语义、正确性、测试、风险)

  • 语义准确性:准确对齐了 trajectory_solver_usage.py 的实际度量口径与边界。

4) 建议修改项(分 P0/P1/P2,带文件路径和行号)

  • 无阻塞项。

5) 最终结论(Verdict)

Approved — ready to merge (exact head c90f3a83342770fc12d37660d334c6e6df6a838a verified).

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent — gpt-6.1-sol / OpenAI; runtime_reported; reasoning_effort=max

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

精确 head:c90f3a83342770fc12d37660d334c6e6df6a838a;base:867615e2c3aa2a32516a71d46f11a20d6fcb3289。未发现阻塞的文档/代码问题,给出 APPROVE 结论;本地 premerge 的 benchmark-sensitive maintainer-review hold 保留,整体 gate 仍为 manual_review_required。

动机 / Motivation

分析 LoopX benchmark 轨迹与实时 session usage、需要报告记录到的请求总量的操作者。
同一段 synthetic capture 的原生 session counters 可以是140 input/15 output,而对账后的全部请求是160/22,compaction 子集为20/7。把前者称作全部请求会低估另一种测量;把这些总量相加又会重复计算。
新指引指向既有 trajectory capture 的 verified request_coverage.all_requests,并保留 counter_basis、窗口与限制;真实 CLI/reader 验证了完整、未完成尾段、未知来源和 mirror 去重的原有行为。
本 PR 只纠正文档与 Agent 的来源选择;Goal 计量口径、评分、run admission 和 observer 的执行代码不变。计费完整性、模型采用、实验收益与安装采用均未验证,本轮不启动模型或评测。

改动思路 / Approach

现有 trajectory capture 与 TypeScript reducer 已提供对账结果,修正 skill 和 observer 的来源说明即可;无需新计量 store、重算 Goal 账或调用 analyzer。
本 PR 保留现有完整 session 默认、显式未完成尾段、source hash、unknown 与不可相加规则,只修正分析指引;benchmark-sensitive maintainer-review hold 保持。
这里区分的是同一捕获中的不同观测:live-tail/Goal collection 继续读 native token_count;请求总量则读取已存在的 TypeScript 对账结果。无操作会保留“exact accounting”容易混同全部请求的说明;重写 Goal collector、增加计量 store 或启动分析模型都超出这次问题。新段落只对需要 recorded request totals 的既有 analysis lane 给出入口、字段和限制,原来的 solver/operator、权限、hidden evaluator 与 matched-run 边界完整保留,无新增确认轮次。

具体改动 / Changes

完整 diff 是两文件 +19/-3,内容全为 module docstring 与 skill prose。规范为 docs/reference/trajectory-assessment.md,不可变 spec_revision:867615e2c3aa2a32516a71d46f11a20d6fcb3289,对应 complete/observed session 和 request/compaction coverage 条款;本轮映射 request-source、tail-admission、unknown-isolation、accounting-boundary,四项 implemented。整份 Python AST 只排除开头 module docstring 和位置属性后与 base 完全相同,未排除任何其他节点;Agent 指令的来源选择则是本 PR 有意改变的行为。

关键代码讲解

以下现有执行函数均未被本 PR 修改,是新指引必须对齐的实际 owner。

  1. observe_codex_session_tail 保持 bounded、content-free live observation。module docstring 现在说明其 usage_cumulative 使用 native counter basis;完整 Goal reader 同样取最新 token_count cumulative,不能据此假设所有请求已被记录或计费。
  2. register_pack 的现有 --usage-jsonl / --usage-session-id 路径校验 source、identity、窗口并保存 capture;未完成尾段要显式 --include-incomplete-usage。真实 CLI 注册/回读中 job 为 None,注册未调用 analyzer。
  3. reconcileTrajectoryRequests 按 response identity 去重并核对 prefix、稳定 counter basis 和 compaction mirror,verified 时提供 all_requests 与 subset;冲突时 coverage unavailable,独立有效的 session counters 继续可读。skill 第128–138行指向此结果,并要求保留 counter_basis、capture window 和 limitations。

对主干的风险 / Risks

最强反例是把 native ordinary basis 当作全部请求,或把 all_requests、compaction 与 session totals 直接相加。当前真实 source/CLI/typed tests 同时覆盖 native ordinary 140/15 和 native all 160/22:对账的 all 为 160/22、compaction 为 20/7,mirror 与重复 request 不再记一笔。缺失/冲突 request、不同 basis、最后 counter 后新增的零 token request 均不能制造 verified 总量;legacy absent coverage 保留 unknown,observed tail 保留不完整成本。changed/missing source 的历史读回不会重写旧 capture。

PASS:usage/session 相关 Python 14 passed(其余39项未选),两项 TypeScript suite 15 passed,含实际 CLI complete 默认拒绝、显式 unfinished opt-in、source/privacy/readback 与无 model dispatch。PASS:本地 premerge 5 direct + 9 canaries 全部通过、零 check failure,Ruff 通过;但因 1项 benchmark_sensitive hold,premerge 退出1,整体 merge gate 保持 manual_review_required,没有把它描述为完整通过。最初 reviewer TS 入口误用了未声明的 tsx,preflight 也误用不支持的 --dry-run,均未执行对应检查;已改用仓库 Node24 native strip-types 与 --no-execute,原错误保留。作者此前缺 Node 依赖的失败也没有改记成历史通过。

我的整体评价 / Verdict

本次文档来源纠正为 goal_achieved;long_horizon preserved,user_experience improved:已有分析入口、字段与未知边界现在能对应真实输出,原生计量和模型/实验权限继续由原 owner 决定。内容比例合适,直接复用现有 reader 和 reducer,运行时代码重构在此边界不需要;semantic vocabulary 沿用已有字段/flag。完整两文件只修改 module docstring 与 benchmark skill 指引;已有 Research Sources/usage 页面、布局与控件没有变化。真实 Agent 采用、原始私人轨迹、计费完整性、实验收益和安装采用未在本轮验证。CI 未查询或等待;当前批准不清除 maintainer hold,也不授予自合并、模型或评测权限。

English verdict: APPROVE — c90f3a83342770fc12d37660d334c6e6df6a838a accurately separates recorded-request coverage from native session counters; the benchmark maintainer-review hold remains.

@huangruiteng
huangruiteng merged commit 529a2fc into main Nov 2, 2026
19 of 31 checks passed
@huangruiteng
huangruiteng deleted the codex/benchmark-request-usage-source-20261102 branch November 2, 2026 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants