test(benchmarks): establish a 12-task main baseline

benchmarks/harbor/README.md

@@ -89,6 +89,33 @@ when it runs; a successful preflight is not a guarantee against later outages.
 Full standard runs enable the timeout retry policy below by default, with zero
 whole-trial retries. CI checks these contracts without model calls or Docker.
 
+### Cache original task images
+
+If Docker image pulls repeatedly stall, the optional cache utility downloads the
+same public Docker Hub image in bounded parallel HTTP ranges:
+
+```bash
+python benchmarks/harbor/scripts/cache_image.py \
+  alexgshaw/git-multibranch:20251031 \
+  --cache .cache/nanopy-harbor/images --load
+```
+
+The first request records the registry manifest digest; later requests reject a
+changed tag. Every downloaded blob must match its size and SHA256, and each
+uncompressed layer must match the original image configuration. The resulting
+[OCI archive](https://github.com/opencontainers/image-spec/blob/main/image-layout.md)
+preserves the original manifest, configuration, and compressed layers byte for
+byte. Docker import must retain the original repository digest.
+
+This Linux utility supports public single-platform `linux/amd64` images with
+gzip layers and SHA256 descriptors. `--load` requires Docker's containerd image
+store. Use one cache writer at a time; `--workers` controls parallel range
+requests (default eight, maximum sixteen). Completed blobs and partial ranges
+can be reused after an interrupted download, with complete blobs revalidated.
+Keep the manifest lock records with the run evidence. This only warms Docker's
+image cache: continue using the standard benchmark entry point and installation
+preflight, and retain any earlier environment-start failures separately.
+
 ### Direct Harbor invocation
 
 For adapter development or comparison with published versions, the lower-level

benchmarks/harbor/README.zh-CN.md

@@ -71,6 +71,28 @@ uv 0.9.5、Python 3.13、pytest 8.4.1 和 pytest-json-ctrf 0.3.5，不执行题
 通过不能保证后续没有网络故障。完整标准运行默认启用下述验证超时重试，整题重试
 次数为零。CI 会验证这些约定，不调用模型、不启动 Docker。
 
+### 缓存原始任务镜像
+
+如果 Docker 拉取镜像反复停滞，可用缓存工具从同一 Docker Hub 公共仓库分段并行下载：
+
+```bash
+python benchmarks/harbor/scripts/cache_image.py \
+  alexgshaw/git-multibranch:20251031 \
+  --cache .cache/nanopy-harbor/images --load
+```
+
+首次请求记录仓库的 manifest 摘要；后续请求若发现 tag 指向的摘要改变，会拒绝继续。
+每个下载对象都必须匹配大小和 SHA256，每层解压后的摘要也必须匹配原始镜像配置。
+生成的 [OCI 归档](https://github.com/opencontainers/image-spec/blob/main/image-layout.md)
+逐字节保留原始 manifest、配置和压缩层；Docker 导入后必须保留原始仓库摘要。
+
+该 Linux 工具支持使用 gzip 层和 SHA256 描述符的公共单平台 `linux/amd64` 镜像。
+`--load` 需要 Docker 的 containerd 镜像存储。同一缓存目录同时只允许一个写入进程；
+`--workers` 控制分段请求并发数，默认 8、最多 16。下载中断后可复用已完成的对象和分段，
+完整对象会重新校验。请将 manifest 锁定记录纳入实验的证据归档。
+这个步骤只准备 Docker 镜像缓存；随后仍需使用标准 benchmark 入口并执行安装预检，
+之前出现的环境启动失败应单独保留。
+
 ### 直接调用 Harbor
 
 开发 adapter 或对照已发布版本时，仍可使用底层 Harbor 入口，并固定容器内 agent 版本：

benchmarks/harbor/configs/dev12-baseline-20261011/README.md

@@ -0,0 +1,116 @@
+# Twelve-task development baseline from main
+
+This cohort adds twelve tasks to the 22 previously exercised tasks. Selection
+was fixed by public task names and desired coverage before downloading task
+packages, reading their instructions, or seeing outcomes. It is a purposive
+**development** cohort, not a representative random sample of Terminal-Bench.
+The other 55 tasks are left unopened by this experiment.
+
+Completed result: **10/12 passed**, with two scored correctness failures and no
+final setup or verifier errors. See the
+[baseline report](../../reports/dev12-main-baseline-20261011.md) and
+[machine-readable evidence](../../results/dev12-main-baseline-20261011.json).
+
+The runtime is main at `2eb99b608dfb39464d17d109a4a8f11afda607a9`, isolated in
+`jobs/dev12-baseline-main-source`. This branch records configuration and evidence;
+it does not change the agent or adapter. Neither prompt candidate from PR #48
+is included. The fixed dataset revision is
+`sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a`.
+
+[Selection](selection.json) records the existing 22 tasks, selected revisions and
+rationales. The [original registered plan](experiment.json) records runtime,
+prompt, configuration and task-file hashes, work limits, preflight and queue
+order. The active [pre-model revision](experiment-v3.json) accepts twelve
+successful native installation trials with hash-bound provenance. All model
+configurations, tasks, runtime, budgets and queue entries remain identical.
+
+The [first preflight evidence](preflight-v1-evidence.json) retains five successful
+installations, six image-pull/environment-start errors, and one agent setup
+timeout while `apt-get update` was still downloading package indexes. No models
+or official verifiers ran. All twelve original images are now cached, including
+six downloaded with the verified image-cache utility. Their original registry
+digests are recorded.
+
+The [second preflight](experiment-v2.json) completed eleven successful
+installations; POV-Ray's uv download ended with a partial-transfer error.
+[Its evidence](preflight-v2-evidence.json) retains every result. The final plan
+uses these eleven successes and POV-Ray's earlier successful installation,
+without replaying successful installations because of another trial's network
+failure. Before model work, the orchestrator requires exactly one accepted
+installation-only result for every queued task and source arm. It verifies the
+task revision, source snapshot, installed wheel, bootstrap status, input mode,
+and recorded artifact hashes. A scored trial, failed installation, pending
+registered verifier profile, incomplete set, or changed evidence is rejected.
+Unsupported verifier profiles remain explicit. Both failed jobs and all eight
+failed preflight trials remain in the evidence; none is a scored agent failure.
+
+| Task | Coverage | Work budget (seconds) |
+| --- | --- | ---: |
+| `build-pov-ray` | Legacy native compilation | 11,820 |
+| `cobol-modernization` | Language migration and behavioral compatibility | 720 |
+| `custom-memory-heap-crash` | C++ memory debugging | 1,620 |
+| `fix-code-vulnerability` | Security repair and behavior preservation | 720 |
+| `nginx-request-logging` | Service configuration and observable behavior | 720 |
+| `git-multibranch` | Git deployment and service integration | 720 |
+| `db-wal-recovery` | Database recovery and data integrity | 720 |
+| `multi-source-data-merger` | Data integration and conflict resolution | 720 |
+| `query-optimize` | SQL correctness and performance | 720 |
+| `adaptive-rejection-sampler` | Numerical algorithms and statistical checks | 720 |
+| `constraints-scheduling` | Calendar constraints and temporal reasoning | 1,020 |
+| `pytorch-model-cli` | C inference and model CLI contracts | 720 |
+
+All trials use `deepseek/deepseek-flash` at the official DeepSeek
+Anthropic-compatible endpoint, 100 replies and 65,536 output tokens per request.
+Temperature and reasoning effort are unset. Each task runs once. Work budgets
+follow the existing policy: native task agent timeout minus 180 seconds.
+POV-Ray has a 12,000-second native limit; it is retained rather than capped to
+match the shorter tasks. Setup independently allows 1,800 seconds.
+
+The queue is fixed by ascending work budget, preserving selection order on ties.
+Consecutive pairs finish before the next pair starts; at most two trials run
+concurrently. There are no outcome-based task replacements or whole-trial
+retries. Verifiers retain their original task limits and may retry only after a
+timeout, at most three total attempts using the same agent output.
+
+## Run and preserve evidence
+
+Run from the repository root after creating the exact source checkout. Pass
+credentials through the process environment, never configuration or logs. Set
+`ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic` and leave
+`ANTHROPIC_MODEL` unset so the recorded configuration selects the model.
+
+```bash
+python benchmarks/harbor/scripts/run_comparison.py \
+  --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v3.json \
+  --phase preflight
+
+python benchmarks/harbor/scripts/run_comparison.py \
+  --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v3.json \
+  --phase model
+```
+
+The reusable orchestrator calls only the standard
+`uv run --project benchmarks/harbor python -m harbor_adapter run --config ...`
+workflow from the frozen repository root. It creates an immutable registration,
+requires all twelve successful install-only records before model work, and
+refuses to retry a job already recorded as started. With the final plan,
+`--phase preflight` validates the recorded successful installations without
+launching another installation job. Source snapshots and wheel manifests remain
+in the normal Harbor job-input directories. A full model trial still installs
+the agent in its own fresh container; preflight does not eliminate later
+network failures.
+
+All twelve selected verifier dependency profiles are explicitly
+`not_configured`. Agent installation preflight does **not** certify verifier
+dependencies. Preserve original images, package sources and official scripts.
+Record setup failures, verifier errors and scored failures separately; preserve
+raw rewards even when an unchanged verifier writes zero before tests start.
+
+Inspect only public metadata/instructions before every model output freezes.
+Hashing task files does not require inspecting their contents. Afterwards, audit
+traces for external benchmark answers, verification quality and incomplete
+accounting. A single observed pass or failure is not a stability estimate.
+Keep these twelve results separate from previous pilot20 and prompt comparisons.
+
+For a later reproduction, use new job names and a new registration directory;
+never overwrite this baseline's artifacts.

benchmarks/harbor/configs/dev12-baseline-20261011/adaptive-rejection-sampler.json

@@ -0,0 +1,22 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 720
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/adaptive-rejection-sampler",
+      "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de"
+    }
+  ]
+}

benchmarks/harbor/configs/dev12-baseline-20261011/build-pov-ray.json

@@ -0,0 +1,22 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 11820
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/build-pov-ray",
+      "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb"
+    }
+  ]
+}

benchmarks/harbor/configs/dev12-baseline-20261011/cobol-modernization.json

@@ -0,0 +1,22 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 720
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/cobol-modernization",
+      "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88"
+    }
+  ]
+}

benchmarks/harbor/configs/dev12-baseline-20261011/constraints-scheduling.json

@@ -0,0 +1,22 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 1020
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/constraints-scheduling",
+      "ref": "sha256:91dd6f87e1ee508328d5aafbd99a6bd1ccc47bc22671b4aaa4060bda492deb53"
+    }
+  ]
+}

benchmarks/harbor/configs/dev12-baseline-20261011/custom-memory-heap-crash.json

@@ -0,0 +1,22 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 1620
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/custom-memory-heap-crash",
+      "ref": "sha256:61e021e0304c9818a9be2c9173c8a0938ce5d136e95c5422e099f1fdb2dc080e"
+    }
+  ]
+}

benchmarks/harbor/configs/dev12-baseline-20261011/db-wal-recovery.json

@@ -0,0 +1,22 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 720
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/db-wal-recovery",
+      "ref": "sha256:07e9193513bfce038b9f9d84962ecc85683eee0a642d14ee6fb7104b086ddab4"
+    }
+  ]
+}