test(benchmarks): establish a common-main 34-task baseline

benchmarks/harbor/README.md

@@ -89,6 +89,104 @@ when it runs; a successful preflight is not a guarantee against later outages.
 Full standard runs enable the timeout retry policy below by default, with zero
 whole-trial retries. CI checks these contracts without model calls or Docker.
 
+### Cache original task images
+
+If Docker image pulls repeatedly stall, the optional cache utility downloads the
+same public Docker Hub image in bounded parallel HTTP ranges:
+
+```bash
+python benchmarks/harbor/scripts/cache_image.py \
+  alexgshaw/git-multibranch:20251031 \
+  --cache .cache/nanopy-harbor/images --load
+```
+
+The first request records the registry manifest digest; later requests reject a
+changed tag. Every downloaded blob must match its size and SHA256, and each
+uncompressed layer must match the original image configuration. The resulting
+[OCI archive](https://github.com/opencontainers/image-spec/blob/main/image-layout.md)
+preserves the original manifest, configuration, and compressed layers byte for
+byte. Docker import must retain the original repository digest.
+
+This Linux utility supports public single-platform `linux/amd64` images with
+gzip layers and SHA256 descriptors. `--load` requires Docker's containerd image
+store. Use one cache writer at a time; `--workers` controls parallel range
+requests (default eight, maximum sixteen). Completed blobs and partial ranges
+can be reused after an interrupted download, with complete blobs revalidated.
+Keep the manifest lock records with the run evidence. This only warms Docker's
+image cache: continue using the standard benchmark entry point and installation
+preflight, and retain any earlier environment-start failures separately.
+
+### Cache immutable APT indexes
+
+If original APT index downloads repeatedly stall, an optional host proxy can
+fetch immutable `/by-hash/SHA256/<digest>` indexes using verified parallel ranges:
+
+```bash
+python benchmarks/harbor/scripts/apt_index_proxy.py \
+  --bind 172.17.0.1 --port 18742 \
+  --cache .cache/nanopy-harbor/apt-indexes --workers 16
+```
+
+Choose the host's actual Docker bridge address. Register a read-only APT
+configuration mount through Docker's `extra_docker_compose`, with per-host
+`Acquire::http::Proxy::<original-host>` entries pointing to the proxy. The
+[34-task registration](configs/dev34-main-baseline-20261012/README.md) contains
+an example. Avoid global proxy environment variables: model/API traffic and
+non-APT tools must retain their normal network path.
+
+The proxy accepts only the original public Debian/Ubuntu APT hosts. It validates
+exact ranges, object sizes, and whole-object SHA256 before serving an index;
+cache hits are rehashed. Mutable release metadata and package downloads are
+forwarded uncached. APT still checks signed release metadata and current package
+hashes. Original repositories, task images, and test scripts stay unchanged.
+Keep the proxy running through preflight and model trials, record the mount and
+utility hashes in the experiment plan, and retain earlier setup failures.
+
+If containers cannot reach the host listener, run the utility in an isolated
+sidecar with `network_mode: service:main`, bind `127.0.0.1`, and use a readiness
+health check. The registered example uses this mode. Mount only the proxy code
+and index cache into the sidecar; do not share task files or credentials. Multiple
+proxy processes coordinate cache writes with per-digest file locks. No host
+proxy process or published port is needed in this mode.
+
+### Cache original uv release archives
+
+When the official uv installer repeatedly receives truncated GitHub downloads,
+prepare its exact original release archives using the public release index:
+
+```bash
+python benchmarks/harbor/scripts/uv_release_cache.py prepare \
+  --cache .cache/nanopy-harbor/uv-releases \
+  --manifest /path/to/new-run/uv-release-manifest.json \
+  --version 0.9.11 --version 0.9.5
+```
+
+The utility requires the original asset filename, size, and SHA256 from GitHub's
+release metadata. It downloads verified ranges, checks the whole archive, and
+preserves the metadata by content hash. Existing manifests cannot be overwritten.
+Only the `x86_64-unknown-linux-gnu` archive is prepared; task images stay original.
+
+Run `uv_release_cache.py serve --cache /cache --manifest /manifest.json` in a
+loopback sidecar sharing `service:main`, with a readiness health check and both
+cache and manifest mounted read-only. Set only the official installer's supported
+`UV_INSTALLER_GITHUB_BASE_URL=http://127.0.0.1:18743` environment variable in the
+task container. Installer scripts, versions, package contents, and normal model
+traffic remain unchanged. Registered archives are rehashed before serving;
+uncached uv versions/platforms redirect to their original HTTPS GitHub URLs.
+Record this transport setting, utility, manifest, and native preflight evidence
+before model work. This cache does not certify unsupported verifier dependencies.
+
+For an observed managed-Python download failure, add its exact original build
+and archive to a new manifest, for example `--python-build 20251014
+--python-asset cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz`.
+The same checks apply to the original `astral-sh/python-build-standalone` release
+metadata. Set the documented
+[`UV_PYTHON_INSTALL_MIRROR`](https://docs.astral.sh/uv/configuration/environment/#uv_python_install_mirror)
+to `http://127.0.0.1:18743/astral-sh/python-build-standalone/releases/download`.
+This changes the download route while retaining uv's original Python selection
+and archive contents. Uncached Python builds redirect to their original GitHub
+URLs; the mirror does not replace uv's embedded Python download index.
+
 ### Direct Harbor invocation
 
 For adapter development or comparison with published versions, the lower-level

benchmarks/harbor/README.zh-CN.md

@@ -71,6 +71,88 @@ uv 0.9.5、Python 3.13、pytest 8.4.1 和 pytest-json-ctrf 0.3.5，不执行题
 通过不能保证后续没有网络故障。完整标准运行默认启用下述验证超时重试，整题重试
 次数为零。CI 会验证这些约定，不调用模型、不启动 Docker。
 
+### 缓存原始任务镜像
+
+如果 Docker 拉取镜像反复停滞，可用缓存工具从同一 Docker Hub 公共仓库分段并行下载：
+
+```bash
+python benchmarks/harbor/scripts/cache_image.py \
+  alexgshaw/git-multibranch:20251031 \
+  --cache .cache/nanopy-harbor/images --load
+```
+
+首次请求记录仓库的 manifest 摘要；后续请求若发现 tag 指向的摘要改变，会拒绝继续。
+每个下载对象都必须匹配大小和 SHA256，每层解压后的摘要也必须匹配原始镜像配置。
+生成的 [OCI 归档](https://github.com/opencontainers/image-spec/blob/main/image-layout.md)
+逐字节保留原始 manifest、配置和压缩层；Docker 导入后必须保留原始仓库摘要。
+
+该 Linux 工具支持使用 gzip 层和 SHA256 描述符的公共单平台 `linux/amd64` 镜像。
+`--load` 需要 Docker 的 containerd 镜像存储。同一缓存目录同时只允许一个写入进程；
+`--workers` 控制分段请求并发数，默认 8、最多 16。下载中断后可复用已完成的对象和分段，
+完整对象会重新校验。请将 manifest 锁定记录纳入实验的证据归档。
+这个步骤只准备 Docker 镜像缓存；随后仍需使用标准 benchmark 入口并执行安装预检，
+之前出现的环境启动失败应单独保留。
+
+### 缓存不可变的 APT 索引
+
+如果原始 APT 索引下载反复停滞，可用可选的宿主机代理，通过校验过的并行分段请求
+获取 `/by-hash/SHA256/<digest>` 不可变索引：
+
+```bash
+python benchmarks/harbor/scripts/apt_index_proxy.py \
+  --bind 172.17.0.1 --port 18742 \
+  --cache .cache/nanopy-harbor/apt-indexes --workers 16
+```
+
+地址应使用本机实际的 Docker 网桥地址。通过 Docker 的 `extra_docker_compose`
+登记只读 APT 配置挂载，以 `Acquire::http::Proxy::<原始主机名>` 分别指定代理。
+[34 题实验登记](configs/dev34-main-baseline-20261012/README.md) 提供了示例。
+不要设置全局代理环境变量，模型 API 和非 APT 工具应保留正常的网络路径。
+
+代理仅允许原始公共 Debian/Ubuntu APT 主机。索引必须通过精确分段、大小和完整
+SHA256 校验后才能提供给客户端；缓存命中也会重新校验哈希。可变的 release 元数据
+和软件包下载直接转发、不由代理缓存。APT 仍校验签名元数据及当前软件包哈希。
+软件源、任务镜像和测试脚本保持原样。预检及模型运行期间需保持代理运行，将挂载
+文件与工具哈希登记进实验计划，并保留之前的安装失败记录。
+
+如果容器无法访问宿主机监听端口，可将该工具运行在独立辅助容器中，设置
+`network_mode: service:main`，绑定 `127.0.0.1`，并配置就绪健康检查。上述登记示例
+采用这一方式。辅助容器只挂载代理代码和索引缓存，不共享题目文件或凭据。多个
+代理进程通过按摘要划分的文件锁协调缓存写入；此模式不需要宿主机代理进程，也
+不向宿主机发布端口。
+
+### 缓存原始 uv 发布压缩包
+
+如果官方 uv 安装器反复遇到 GitHub 下载不完整，可按公共发布索引缓存对应的原始版本：
+
+```bash
+python benchmarks/harbor/scripts/uv_release_cache.py prepare \
+  --cache .cache/nanopy-harbor/uv-releases \
+  --manifest /path/to/new-run/uv-release-manifest.json \
+  --version 0.9.11 --version 0.9.5
+```
+
+工具要求压缩包匹配 GitHub 发布元数据中的文件名、大小和 SHA256，通过分段及完整
+文件校验后才接受，并按内容哈希保留原始元数据。已有 manifest 不可覆盖。目前只准备
+`x86_64-unknown-linux-gnu` 压缩包，题目镜像保持原样。
+
+在共享 `service:main` 网络命名空间的回环辅助容器中运行
+`uv_release_cache.py serve --cache /cache --manifest /manifest.json`，配置就绪健康
+检查，并以只读方式挂载缓存和 manifest。在题目容器中仅设置官方安装器支持的
+`UV_INSTALLER_GITHUB_BASE_URL=http://127.0.0.1:18743` 环境变量。安装脚本、版本、
+软件包内容及正常模型流量保持原样。已登记压缩包在提供下载前重新校验；未缓存的
+uv 版本或平台重定向至其原始 HTTPS GitHub 地址。在模型运行前登记此下载配置、
+工具、manifest 和原生预检证据；缓存不代表未配置的验证器依赖已通过预检。
+
+若已观察到 uv 管理的 Python 下载失败，可将对应原始构建及文件名加入新的 manifest，
+例如 `--python-build 20251014
+--python-asset cpython-3.13.9+20251014-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz`。
+同样按原始 `astral-sh/python-build-standalone` 发布元数据校验。将官方文档中的
+[`UV_PYTHON_INSTALL_MIRROR`](https://docs.astral.sh/uv/configuration/environment/#uv_python_install_mirror)
+设置为 `http://127.0.0.1:18743/astral-sh/python-build-standalone/releases/download`。
+这只改变下载路径，保留 uv 原来的 Python 版本选择和压缩包内容。未缓存的 Python
+构建重定向至其原始 GitHub 地址，不替换 uv 内置的 Python 下载索引。
+
 ### 直接调用 Harbor
 
 开发 adapter 或对照已发布版本时，仍可使用底层 Harbor 入口，并固定容器内 agent 版本：

benchmarks/harbor/configs/dev12-baseline-20261011/README.md

@@ -0,0 +1,116 @@
+# Twelve-task development baseline from main
+
+This cohort adds twelve tasks to the 22 previously exercised tasks. Selection
+was fixed by public task names and desired coverage before downloading task
+packages, reading their instructions, or seeing outcomes. It is a purposive
+**development** cohort, not a representative random sample of Terminal-Bench.
+The other 55 tasks are left unopened by this experiment.
+
+Completed result: **10/12 passed**, with two scored correctness failures and no
+final setup or verifier errors. See the
+[baseline report](../../reports/dev12-main-baseline-20261011.md) and
+[machine-readable evidence](../../results/dev12-main-baseline-20261011.json).
+
+The runtime is main at `2eb99b608dfb39464d17d109a4a8f11afda607a9`, isolated in
+`jobs/dev12-baseline-main-source`. This branch records configuration and evidence;
+it does not change the agent or adapter. Neither prompt candidate from PR #48
+is included. The fixed dataset revision is
+`sha256:7d7bdc1cbedad549fc1140404bd4dc45e5fd0ea7c4186773687d177ad3a0699a`.
+
+[Selection](selection.json) records the existing 22 tasks, selected revisions and
+rationales. The [original registered plan](experiment.json) records runtime,
+prompt, configuration and task-file hashes, work limits, preflight and queue
+order. The active [pre-model revision](experiment-v3.json) accepts twelve
+successful native installation trials with hash-bound provenance. All model
+configurations, tasks, runtime, budgets and queue entries remain identical.
+
+The [first preflight evidence](preflight-v1-evidence.json) retains five successful
+installations, six image-pull/environment-start errors, and one agent setup
+timeout while `apt-get update` was still downloading package indexes. No models
+or official verifiers ran. All twelve original images are now cached, including
+six downloaded with the verified image-cache utility. Their original registry
+digests are recorded.
+
+The [second preflight](experiment-v2.json) completed eleven successful
+installations; POV-Ray's uv download ended with a partial-transfer error.
+[Its evidence](preflight-v2-evidence.json) retains every result. The final plan
+uses these eleven successes and POV-Ray's earlier successful installation,
+without replaying successful installations because of another trial's network
+failure. Before model work, the orchestrator requires exactly one accepted
+installation-only result for every queued task and source arm. It verifies the
+task revision, source snapshot, installed wheel, bootstrap status, input mode,
+and recorded artifact hashes. A scored trial, failed installation, pending
+registered verifier profile, incomplete set, or changed evidence is rejected.
+Unsupported verifier profiles remain explicit. Both failed jobs and all eight
+failed preflight trials remain in the evidence; none is a scored agent failure.
+
+| Task | Coverage | Work budget (seconds) |
+| --- | --- | ---: |
+| `build-pov-ray` | Legacy native compilation | 11,820 |
+| `cobol-modernization` | Language migration and behavioral compatibility | 720 |
+| `custom-memory-heap-crash` | C++ memory debugging | 1,620 |
+| `fix-code-vulnerability` | Security repair and behavior preservation | 720 |
+| `nginx-request-logging` | Service configuration and observable behavior | 720 |
+| `git-multibranch` | Git deployment and service integration | 720 |
+| `db-wal-recovery` | Database recovery and data integrity | 720 |
+| `multi-source-data-merger` | Data integration and conflict resolution | 720 |
+| `query-optimize` | SQL correctness and performance | 720 |
+| `adaptive-rejection-sampler` | Numerical algorithms and statistical checks | 720 |
+| `constraints-scheduling` | Calendar constraints and temporal reasoning | 1,020 |
+| `pytorch-model-cli` | C inference and model CLI contracts | 720 |
+
+All trials use `deepseek/deepseek-flash` at the official DeepSeek
+Anthropic-compatible endpoint, 100 replies and 65,536 output tokens per request.
+Temperature and reasoning effort are unset. Each task runs once. Work budgets
+follow the existing policy: native task agent timeout minus 180 seconds.
+POV-Ray has a 12,000-second native limit; it is retained rather than capped to
+match the shorter tasks. Setup independently allows 1,800 seconds.
+
+The queue is fixed by ascending work budget, preserving selection order on ties.
+Consecutive pairs finish before the next pair starts; at most two trials run
+concurrently. There are no outcome-based task replacements or whole-trial
+retries. Verifiers retain their original task limits and may retry only after a
+timeout, at most three total attempts using the same agent output.
+
+## Run and preserve evidence
+
+Run from the repository root after creating the exact source checkout. Pass
+credentials through the process environment, never configuration or logs. Set
+`ANTHROPIC_BASE_URL=https://api.deepseek.com/anthropic` and leave
+`ANTHROPIC_MODEL` unset so the recorded configuration selects the model.
+
+```bash
+python benchmarks/harbor/scripts/run_comparison.py \
+  --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v3.json \
+  --phase preflight
+
+python benchmarks/harbor/scripts/run_comparison.py \
+  --plan benchmarks/harbor/configs/dev12-baseline-20261011/experiment-v3.json \
+  --phase model
+```
+
+The reusable orchestrator calls only the standard
+`uv run --project benchmarks/harbor python -m harbor_adapter run --config ...`
+workflow from the frozen repository root. It creates an immutable registration,
+requires all twelve successful install-only records before model work, and
+refuses to retry a job already recorded as started. With the final plan,
+`--phase preflight` validates the recorded successful installations without
+launching another installation job. Source snapshots and wheel manifests remain
+in the normal Harbor job-input directories. A full model trial still installs
+the agent in its own fresh container; preflight does not eliminate later
+network failures.
+
+All twelve selected verifier dependency profiles are explicitly
+`not_configured`. Agent installation preflight does **not** certify verifier
+dependencies. Preserve original images, package sources and official scripts.
+Record setup failures, verifier errors and scored failures separately; preserve
+raw rewards even when an unchanged verifier writes zero before tests start.
+
+Inspect only public metadata/instructions before every model output freezes.
+Hashing task files does not require inspecting their contents. Afterwards, audit
+traces for external benchmark answers, verification quality and incomplete
+accounting. A single observed pass or failure is not a stability estimate.
+Keep these twelve results separate from previous pilot20 and prompt comparisons.
+
+For a later reproduction, use new job names and a new registration directory;
+never overwrite this baseline's artifacts.

benchmarks/harbor/configs/dev12-baseline-20261011/adaptive-rejection-sampler.json

@@ -0,0 +1,22 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 720
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/adaptive-rejection-sampler",
+      "ref": "sha256:bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de"
+    }
+  ]
+}

benchmarks/harbor/configs/dev12-baseline-20261011/build-pov-ray.json

@@ -0,0 +1,22 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 11820
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/build-pov-ray",
+      "ref": "sha256:4f08dab602fe2e9e2f42051cc6aef684a88cf2822e8b499b3ca19e670759fddb"
+    }
+  ]
+}

benchmarks/harbor/configs/dev12-baseline-20261011/cobol-modernization.json

@@ -0,0 +1,22 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "override_setup_timeout_sec": 1800,
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 720
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/cobol-modernization",
+      "ref": "sha256:e03aa03965eee48c9f772d566ff9bcd741fceeac9693123f82a13fdd46be0d88"
+    }
+  ]
+}