Skip to content

feat(agent): strengthen headless contract verification - #47

Merged
minixalpha merged 6 commits into
mainfrom
experiment/headless-contract-verification
Oct 8, 2026
Merged

minixalpha merged 6 commits into
mainfrom
experiment/headless-contract-verification

Conversation

@minixalpha

@minixalpha minixalpha commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Headless runs now identify deliverables, interfaces, and constraints; verify output behavior through the real entry point with proportionate independent or boundary checks; diagnose and retest failures; and report observed evidence and remaining limitations. The prompt permits task-supplied source, executables, and tests while restricting externally obtained task-specific answers and hidden evaluation tests. Only HEADLESS_SYSTEM_PROMPT changes in runtime code.

The PR includes frozen experiment configurations, prompt snapshots, all trial results, reference-use audits, a tool-limit diagnostic, and synchronized development notes.

Experiment evidence

Measure Main (6664f7b) Frozen 368-word candidate (03cfd53)
Raw official passes 18/20 19/20
Common eligible tasks 17/19 18/19
Estimated model cost $1.457520 $1.898565
Summed agent seconds 11160.416 13570.056

The candidate adds a regex-chess pass with no observed per-task regression. Both variants still fail pipeline backward-activation checks. The main MTEB pass involved reference-answer and hidden-test retrieval, so its raw score is retained and the whole pair is excluded from the eligible comparison. The candidate's MTEB pass used task-authorized public data. Costs rise 30.3% and summed execution time rises 21.6%; usage records are complete.

The final 385-word prompt (e61558f) clarifies permission to use supplied workspace files. Its separately registered path-tracing follow-up passed in 340.510 seconds with $0.114759 estimated cost and used the supplied executable. The full twenty-task score belongs to the frozen 368-word candidate; the final clarification was checked on one selected task only. This adaptive, single-sample pilot has no held-out cohort and does not establish a stable general gain. Abandoned-candidate results and historical contamination findings remain separately recorded.

Validation and documentation

  • Core tests: 351 passed, 4 skipped; Harbor workflow tests: 91 passed. Python 3.13/3.14 and Harbor CI pass at the PR head.
  • All 40 trajectories, 152 task-file hashes, 20 image IDs, eight source/wheel snapshots, installed wheels, and native verifier retry policies were validated. QEMU's registered dependency preflight passed; the other 19 profiles are explicitly not_configured.
  • The changelog records contract verification and permitted references. English development notes match the complete Chinese source. Both root READMEs and research notes are unchanged.

Experiment report · All paired results and audits · Clarification follow-up

@minixalpha
minixalpha marked this pull request as ready for review October 8, 2026 13:36
@minixalpha
minixalpha merged commit 2eb99b6 into main Oct 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant