Skip to content

fix(model-bench): reject sessions without a final answer - #299

Merged
enkerewpo merged 1 commit into
devfrom
fix/bench-unfinished-sessions
Oct 27, 2026
Merged

enkerewpo merged 1 commit into
devfrom
fix/bench-unfinished-sessions

Conversation

@enkerewpo

Copy link
Copy Markdown
Member

Cases without expect.answered could pass after Pilot dispatched the expected calls even when the session ended without a final answer. Require a non-whitespace final answer, no reported Pilot failure, and no explicit finished: false in both the runner checks and the summary of saved results. Existing execution checks and judge verdicts remain unchanged.

The saved dev regression for case 16 (decomp.named_room) recorded pilot_failed: false, fails: [], final: "", and pass: true; its CLI session exited 1 without a final event. A snapshot of all 16 live board submissions contained 646 cases, including 10 passing cases with empty finals. The same completion gate rejects those cases without another model or judge call.

Older result files lack finished; their recorded nonempty final remains eligible. They do not retain CLI exit codes or error logs, so this fix cannot retrospectively identify every hidden Pilot error. Failed HTTP requests before a successful retry remain eligible.

Validation: on work_pc, python3 -m unittest discover -s testing -p test_model_bench.py -v passed all 11 tests. Tests cover empty and whitespace finals, explicit unfinished sessions, reported failures, old files without finished, existing call failures, and cached judge verdicts without new judge requests. git diff --check passed. No capability contracts, generated code, or configuration changes.

@github-actions github-actions Bot added type:bug Bug fix (fix:) comp:docs docs/ and READMEs and removed type:bug Bug fix (fix:) labels Oct 25, 2026
@enkerewpo
enkerewpo marked this pull request as ready for review October 27, 2026 09:12
@enkerewpo
enkerewpo merged commit c9c159e into dev Oct 27, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:docs docs/ and READMEs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant