Repository navigation
fix(benchmark): declare the resolved mode for native context delivery - #7516
Conversation
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; model=gpt-6.1-sol; provider=OpenAI; reasoning_effort=max; runtime_reported
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机 / Motivation
受影响的是开启 context delivery 的 Codex heartbeat worker。省略 execution mode 时,worker 已解析为 heartbeat,CLI 预检却收到缺失声明而退出;本改动把同一解析值传入子进程,使原 guard 和后续恢复可以继续。实际 native Host 的原 guard 捕获、fork/恢复与重复抑制通过,且默认关闭路径保持。本 PR 不改变评分、任务、评测预算、权限或启动实验。外部 checkpoint supervisor、Linux 联合恢复、完整 workspace 和真实模型长期收益尚未独立资格。
改动思路 / Approach
沿既有 Execution 和 activation owner 传递已验证的模式即可,无需新增开关、授权或运行状态。本次边界是 worker 环境传递及两个既有 native fixture 的 canonical 初始化。模式由 Execution 构造时校验并解析,环境只是向已有 CLI adapter 运输同一声明;子进程仍独立校验 host/scheduler/mode。关闭时不新增这个环境值。两个测试先完成源编写,再调用原有 initializer;未恢复生产 legacy fallback,也未改为重放 guard 来规避失败。
具体改动 / Changes
关键代码讲解
run_once:仅context_delivery == "on"时写入execution.mode。省略 mode 和显式 heartbeat 使用同一解析结果,原生 hook 设置与 solver 子进程共用该环境。test_real_quota_output_is_held_across_native_fork:在配置完成后、第一次 guard 前初始化原 canonical authority,保留捕获 digest、原 receipt 数与 fork 未重放断言。test_native_cli_interruption_retains_guard_beyond_tool_yield:resume、捕获失败、packet 改写三个分支均保留原失败语义,未弱化超出 tool yield 后的原结果恢复要求。
没有为此 bug 单独命名的书面 specification;评审先核既有 codex_context.py::context_delivery_activation、Execution 及 smoke 合同,再执行实际反例。immutable base 为 b69021498ef7483b1d2b190bc3f4c7bb6675c882,实审 head 为 3f13fd19c6efdefc9ff95745d90b3f89994f6123。未新增词汇或额外验收条目。
对主干的风险 / Risks
PASS:真实 Codex CLI/app-server 与本地确定性 provider 的模块为 18 passed、4 skipped;相关 worker/预算/context CLI 为 183 passed、48 skipped。4 个 skip 需要外部 checkpoint supervisor,48 个 skip 缺可选 Harbor;它们不记为通过。相同 base 的四个 opt-in worker 场景在实际 CLI 预检失败,head 对应场景通过;base 与 head 的默认 off native 场景均通过。初始化顺序修复还经过真实 fork、12 秒跨 yield 恢复及失败分支。
风险检查的 5 direct + 10 selected 自动检查零失败,整体仍返回 manual_review_required,保留 benchmark-sensitive 人工门。没有查询或等待 CI。Linux 联合 capture/receiver、完整 workspace、正式安装和付费模型未测试;本地脚本化 provider 不代表真实科研资格。可整体回退这两个文件;不得以新建 admission 代替原 capture 恢复。
我的整体评价 / Verdict
APPROVE,作为有用的运行时修复增量。沿既有 Execution 和 activation owner 传递已验证的模式即可,无需新增开关、授权或运行状态。本次边界是 worker 环境传递及两个既有 native fixture 的 canonical 初始化。长程路径判断为 improved,范围仅限已验证的原 guard/恢复能继续;用户使用判断为 improved,因为已有合法配置不再要求补填已解析的信息。完整研究效果与总成本未知。视觉审阅不适用:完整 diff 仅 worker 环境和测试初始化,没有渲染页面;未验证范围为安装 UI 与真实模型行为。
外部 checkpoint supervisor、Linux 联合恢复、完整 workspace 和真实模型长期收益尚未独立资格。现有 benchmark/checkpoint owner 继续承担这些资格,本评审不结清它们。自动检查、代码结论与合并权限分别判断。
English verdict: APPROVE 3f13fd19c6efdefc9ff95745d90b3f89994f6123: resolved-mode propagation and native fixture ordering are verified; external supervisor and full research qualification remain unverified.
Problem and result
An opted-in Codex heartbeat worker resolves an omitted execution mode to heartbeat, but did not carry that resolved declaration to its CLI subprocess. The existing native-context activation adapter therefore rejected the actual worker before its guard ran. The worker now passes its validated mode when context delivery is on; the default-off environment remains unchanged.
Two native handoff fixtures also read before canonical initialization. They now initialize the existing authority once, after complete source authoring and before the first guard. Existing midwake CAS, full capture, no-replay and unique-settlement assertions remain.
Scope and validation
The existing differential-context configuration and benchmark-toolkit entrypoints retain activation and authority ownership. No scoring, task, evaluation budget, cohort, job launch or business permission changes.
Actual native Host module: 18 passed, 4 skipped because the optional external checkpoint supervisor is not attached. Related worker/budget/CLI suites: 91 passed, 43 skipped; skip counts remain explicit. Required Ruff, mypy, syntax and public-boundary checks passed. Risk-based premerge ran 5 direct and 10 selected checks with zero check failures; its benchmark-sensitive manual hold is reviewed separately before merge.
Final integrated release, model, installed artifacts and complete workspace qualification remain separate.
Implemented against
Demonstrated native qualification failures and the existing context_delivery_activation declaration contract in loopx/control_plane/turn_driver/codex_context.py. The independent native worker cases exercise omitted-mode opt-in, default off, original guard capture, recovery and duplicate suppression. Repository constraints: docs/development/good-smokes.md and benchmark/runtime/worker.py's existing Execution owner.