Repository navigation
feat(agent): strengthen headless contract verification - #47
Merged
Merged
Conversation
minixalpha
marked this pull request as ready for review
October 8, 2026 13:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Headless runs now identify deliverables, interfaces, and constraints; verify output behavior through the real entry point with proportionate independent or boundary checks; diagnose and retest failures; and report observed evidence and remaining limitations. The prompt permits task-supplied source, executables, and tests while restricting externally obtained task-specific answers and hidden evaluation tests. Only
HEADLESS_SYSTEM_PROMPTchanges in runtime code.The PR includes frozen experiment configurations, prompt snapshots, all trial results, reference-use audits, a tool-limit diagnostic, and synchronized development notes.
Experiment evidence
6664f7b)03cfd53)The candidate adds a regex-chess pass with no observed per-task regression. Both variants still fail pipeline backward-activation checks. The main MTEB pass involved reference-answer and hidden-test retrieval, so its raw score is retained and the whole pair is excluded from the eligible comparison. The candidate's MTEB pass used task-authorized public data. Costs rise 30.3% and summed execution time rises 21.6%; usage records are complete.
The final 385-word prompt (
e61558f) clarifies permission to use supplied workspace files. Its separately registered path-tracing follow-up passed in 340.510 seconds with $0.114759 estimated cost and used the supplied executable. The full twenty-task score belongs to the frozen 368-word candidate; the final clarification was checked on one selected task only. This adaptive, single-sample pilot has no held-out cohort and does not establish a stable general gain. Abandoned-candidate results and historical contamination findings remain separately recorded.Validation and documentation
not_configured.Experiment report · All paired results and audits · Clarification follow-up