feat(agent): strengthen headless contract verification

benchmarks/harbor/configs/headless-p0-20261007/README.md

@@ -0,0 +1,73 @@
+# Headless contract and verification prompt comparison
+
+**Execution status:** the original candidate at `7ea2d75` was abandoned after
+retrieving a task-specific reference solution and evaluation source. Preserve all
+five started candidate trials in the [audit](../../results/headless-p0-original-candidate-audit-20261007.json);
+the other 15 never started. All main-control trials continue unchanged. The
+[guarded-candidate plan](../headless-p0-guarded-20261007/README.md) is a separate,
+explicitly adaptive comparison, with 20 fresh candidate trials. The original
+predeclared plan and prompt hashes remain unchanged below.
+
+This [predeclared experiment](experiment.json) evaluates the first candidate from
+the [prompt study](../../../../docs/research/en/agent_system_prompt.md). Compare
+current main (`6664f7bac9d7f9b26f49e523ac6ee7e4e8a6b9ac`, including truncation
+recovery) with a candidate that changes only `HEADLESS_SYSTEM_PROMPT`.
+The exact [control](control-prompt.txt) and [treatment](treatment-prompt.txt)
+strings are saved without trailing newlines and hashed in the plan.
+
+The candidate adds task-contract identification, representative and boundary
+checks, independent references or invariants, diagnosis and retesting, and
+evidence-based completion. Verification remains proportional to task and budget.
+It contains no task names, hidden verifier parameters, or prescribed algorithms.
+Earlier executable feedback is a separate hypothesis, excluded from this test.
+
+Run all 20 pinned pilot tasks once per arm: **40 fresh model runs**. The released
+18/20 result is historical context, not the control arm. There is no held-out
+cohort or within-task replication, so this is an exploratory comparison.
+Keep every score and infrastructure failure. Do not tune the prompt or add
+favorable replacement runs after observing results.
+
+## Execution
+
+Use a dedicated control checkout at the recorded main revision and the candidate
+checkout for treatment. Both use the standard repository adapter; do not copy or
+modify adapters per job. Configure the official DeepSeek credentials in each
+command's environment, set `ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic`,
+and leave `ANTHROPIC_MODEL` unset. Credentials must not enter the configuration.
+
+Before model work, run the 20-task install-only configuration from the control
+repository root. Check the candidate's installation separately. Full trials
+repeat setup in fresh containers. QEMU's registered profile must pass; the other
+19 tasks report `not_configured`, not verified dependencies.
+
+```bash
+uv run --project benchmarks/harbor python -m harbor_adapter run \
+  --config /absolute/path/to/headless-p0-20261007/install-only.json \
+  --job-name headless-p0-install-20261007 --install-only
+```
+
+Run the two arms of each budget group concurrently, with **one trial per arm**
+and at most **two concurrent trials total**. Wait for both arms before starting
+the next group. The plan fixes group order and alternates arm launch order.
+For example, from each arm's repository root:
+
+```bash
+uv run --project benchmarks/harbor python -m harbor_adapter run \
+  --config /absolute/path/to/headless-p0-20261007/budget-720s.json \
+  --job-name headless-p0-control-720s-20261007
+```
+
+Use the corresponding treatment job name in the treatment checkout. Share the
+existing verified APT cache with `--cache-dir` when using separate checkouts.
+Use fresh names to reproduce the experiment; existing outputs are immutable.
+Every runner records its source snapshot, wheel hash, config, and working-tree
+patch. Verify that the two source snapshots differ only in the headless prompt.
+
+Preserve native task images, package sources, official scripts, model, endpoint,
+100-reply limit, 65,536-token request cap, work budgets, and verifier limits.
+Whole-trial retries stay disabled. Only verifier timeouts may retry, at most
+three times on the same output using the original per-attempt timeout.
+
+Report paired task outcomes, regressions, unscored trials, time, token estimates,
+accounting completeness, and observed checking behavior. A normal terminal state
+or a final claim of testing is not a substitute for an official passing result.

benchmarks/harbor/configs/headless-p0-20261007/budget-1620s.json

@@ -0,0 +1,30 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 1620
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/dna-assembly",
+      "ref": "sha256:e41a8e94d86019949b08d3b5f88a85f6d943ba0fd85d5e1d5ebb95cb8f66223f"
+    },
+    {
+      "name": "terminal-bench/path-tracing",
+      "ref": "sha256:cf56094c881a488b27e9f204a638a7e78ed7d55e12dc3064108c93357190314c"
+    },
+    {
+      "name": "terminal-bench/llm-inference-batching-scheduler",
+      "ref": "sha256:a3bf47589118daec5124fe3689b4439802c805b2f4ed9d402a48ae73ca2fea67"
+    }
+  ]
+}

benchmarks/harbor/configs/headless-p0-20261007/budget-2220s.json

@@ -0,0 +1,22 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 2220
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/schemelike-metacircular-eval",
+      "ref": "sha256:58130c2166c3115276dc8592f358e326ff2d81ea852e3d88636c82fd1dff57e6"
+    }
+  ]
+}

benchmarks/harbor/configs/headless-p0-20261007/budget-3420s.json

@@ -0,0 +1,34 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 3420
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/regex-chess",
+      "ref": "sha256:e763e0ac1c9759081af0a4a82ba51b8cf9ae5485a93de3bbe42d7d344597bd78"
+    },
+    {
+      "name": "terminal-bench/caffe-cifar-10",
+      "ref": "sha256:7b0045106d7d5af724efe96b610ba64f7893f5c88528401c573c4d47e384e2bf"
+    },
+    {
+      "name": "terminal-bench/mteb-leaderboard",
+      "ref": "sha256:484f6d7008a05b5b8640fc6618a384b8c9447cd76f85416c8a595028d29bff9c"
+    },
+    {
+      "name": "terminal-bench/circuit-fibsqrt",
+      "ref": "sha256:9bcffe1054bb33249aa578a9a2a74f3c8cca66b0cb7aa1328233f1d31822aae3"
+    }
+  ]
+}

benchmarks/harbor/configs/headless-p0-20261007/budget-720s.json

@@ -0,0 +1,66 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 720
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/write-compressor",
+      "ref": "sha256:d9ddd9a8e925e2c566b37b2492cbf995afecefe58874e4043ef78d7f3c892c7e"
+    },
+    {
+      "name": "terminal-bench/torch-tensor-parallelism",
+      "ref": "sha256:f32ce74a5aeb6638480247ab799fe46127bbee631acdd0921b0f394ec49b3684"
+    },
+    {
+      "name": "terminal-bench/kv-store-grpc",
+      "ref": "sha256:973c5d4c111fb61a344457936f1c36400acd2d9e44389e7b319586fe23a7a307"
+    },
+    {
+      "name": "terminal-bench/pypi-server",
+      "ref": "sha256:1a1e0542f58e2d3362fec17a9bbb98667717d9a4a3e9a4c8413d3150a4fa0ff1"
+    },
+    {
+      "name": "terminal-bench/torch-pipeline-parallelism",
+      "ref": "sha256:db605337c749a872cea7b5b413429b3915bb4c3efe0f7875f0c46ce81bd8c4fb"
+    },
+    {
+      "name": "terminal-bench/qemu-alpine-ssh",
+      "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28"
+    },
+    {
+      "name": "terminal-bench/openssl-selfsigned-cert",
+      "ref": "sha256:d4afa2bd2a9ba1420db8d6cfde42ffdb4873ae2d955c35014e8da94444c83302"
+    },
+    {
+      "name": "terminal-bench/log-summary-date-ranges",
+      "ref": "sha256:27b074a2f10fff7606e096f3abd8dced418ad8fda0f53d88acbe477f2d9ceaf6"
+    },
+    {
+      "name": "terminal-bench/model-extraction-relu-logits",
+      "ref": "sha256:1ae5045ad68b5d34c3398b612066a07c4a08b6dc330d28868ec4021e17c94b17"
+    },
+    {
+      "name": "terminal-bench/regex-log",
+      "ref": "sha256:802c16cfd132e6c457529cb864be5a757c1b23b6cadc57f2d01983cb0110292a"
+    },
+    {
+      "name": "terminal-bench/pytorch-model-recovery",
+      "ref": "sha256:2e628841cff93290919172398e573e794f34c95d9382b7425adde4364022decc"
+    },
+    {
+      "name": "terminal-bench/merge-diff-arc-agi-task",
+      "ref": "sha256:6aab6511a5344ce87698293bb1ce4cc51d9a45f1ad9f0c075d2a83197b36727d"
+    }
+  ]
+}

benchmarks/harbor/configs/headless-p0-20261007/control-prompt.txt

@@ -0,0 +1 @@
+You are nanoPyCodeAgent, a coding agent running non-interactively on a task handed to you up front. There is no user to reply to you: never ask a clarifying question, never stop to wait for confirmation, and never present a plan for approval — decide on your own and carry it out. Work the task through to the end, then check the result with the tools instead of assuming it worked. When it is done, answer with a short summary and no further tool calls: that reply is what ends the run. Runtime budget reminders report remaining time and model replies; when either is running low, prioritize saving the required deliverables, necessary verification, and a final summary. Do not start optional work after the requirements are satisfied. Prefer the read tool for viewing files, the edit tool for changing part of an existing file, and the write tool for creating files or rewriting them whole. Use the bash tool to run commands, search with grep, and complete tasks that need real command output instead of guessing.