Repository navigation
Phase 0 baseline: strengthen turn-benchmark oracles (read-only verifiers, positive refactor oracle) #701
Description
Activity
- added a commit that references this issue
on Jul 16, 2026 Tracking one more non-blocker from the PR #700 re-review: rename the
LatencySourceJSON fieldshare→exclusiveShareso the contract makes explicit that the share is computed on exclusive time, not a raw duration sum. It's a v2 schema field (nothing downstream consumes it yet), so it's a safe deliberate rename for the hardening pass rather than mid-baseline. Pulling it in here alongside the refactor structural verifier so the next baseline report ships a self-describing contract.- added a commit that references this issue
on Jul 16, 2026 - added a commit that references this issue
on Jul 17, 2026 Closing: this landed in #712, with #730, #737 and #763 refining it since. I checked each of the three asks against
mainrather than going by the PR title.Edit oracles are no longer bare substring greps. They now compile a stamped oracle and assert the old name is gone:
bash -c go test ./... && ! grep --exclude=.zero-answer.txt -RIn 'MaxRetries =' .The refactor oracle is positive. A no-op refactor fails it, because the count is asserted rather than the build:
bash -c go test ./... && [ $(grep -c 'hello, %s' main.go) -eq 1 ]buildOnlyClassesis now[], so refactor has graduated out of the build tier into correctness, which is what made the build-only tier empty.The read-only tasks are settled both ways. nav carries a deterministic verifier against the stamped answer file and has graduated from latency-only to correctness. longproc, longctx and parallel remain oracle-free and are documented as permanently latency-only in
MANIFEST.md, with the reasoning stated: the answer is free-form prose and a contains-check would rubber-stamp it.Counts on
main: 48 tasks, 34 carrying an oracle (edit 10, fix 8, nav 10, refactor 6). The 14 without are exactly longproc, longctx and parallel.internal/perfbenchpasses.The part worth keeping is that the manifest documents why three classes have no oracle rather than leaving them silently unverified. A pass rate computed from oracles that cannot fail is the same defect as a test that passes with the code deleted, and harder to notice because it looks like a number.
Follow-up to the Phase 0 turn-trace baseline (#700). The baseline lands with a per-oracle-tier pass/fail split so the report cannot be misread as a blanket correctness verdict:
go test). The only pass rate that can move with model quality.go build ./..., reported asbuildPassRate, excluded fromcorrectnessPassRate.latencyOnlyTasks, excluded from every pass rate.The tier split keeps the report honest, so this hardening is not a blocker — but the oracles are intentionally lightweight for Phase 0 and should be strengthened before a baseline report is treated as a correctness signal.
Work
grep -R Label .) only proves the string appears, not that it landed on the right struct. Addgo vet/go buildplus a structural assertion (e.g. the field on the named struct) for add-field/add-method tasks. Rename oracles already assert the old name is gone.go build ./...passes a no-op refactor. Add a structural verifier that proves the refactor happened (extracted helper exists, call sites updated), or document refactor as permanently structure-uncorrected.diff -q <(expected) <(agent-output)if the harness pipes the model's final text to a known path). longproc/longctx/parallel summarize with no canonical answer — document them as permanently latency-only.Where
internal/perfbench/manifests/baseline.jsoninternal/perfbench/turn_bench.go(NewTurnExecRunner,RunTurnBench)internal/perfbench/MANIFEST.md