feat: add DeepSeek cost estimates and reliable Harbor runs

.github/workflows/ci.yml

@@ -41,3 +41,19 @@ jobs:
         run: uv run --frozen pytest
       - name: Test the httpx2 SDK transport
         run: uv run --with anthropic==1.5.0 pytest
+
+  harbor:
+    name: Harbor workflow contracts
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+      - name: Install uv
+        uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
+        with:
+          python-version: "3.13"
+      - name: Test cache integrity, preflight, and verifier retries
+        run: >-
+          uv run --frozen --project benchmarks/harbor pytest
+          -c benchmarks/harbor/pyproject.toml benchmarks/harbor/tests

AGENTS.md

@@ -19,6 +19,24 @@ the matching one:
 
 See `docs/RELEASING.md` for the release process and prerequisites.
 
+## Benchmarks
+
+For new Harbor benchmark runs, use the standard workflow from the repository
+root: `uv run --project benchmarks/harbor python -m harbor_adapter run --config
+<config.json>`. See `benchmarks/harbor/README.md` for configuration, verified APT
+cache import, and the `--install-only` dependency preflight option.
+
+- Do not create per-job adapter copies or ad hoc bootstrap scripts. Add reusable
+  fixes and reviewed task-revision profiles to `benchmarks/harbor` with tests.
+- Preserve task images, package sources, and official test scripts. Cache hits
+  must match the current package index's filename, size, and SHA256 digest.
+- Run dependency preflight before model work when a task has a registered
+  profile. Record unsupported profiles explicitly; do not claim they passed.
+- Keep setup failures and verifier errors separate from scored failures. Only
+  verifier timeouts may retry, at most three total attempts using the same agent
+  output and the task's original per-attempt time limit.
+
+
 ## Language
 
 Write everything in **English** by default — source code (identifiers,

benchmarks/harbor/README.md

@@ -18,15 +18,90 @@ export ANTHROPIC_API_KEY="..."
 export ANTHROPIC_BASE_URL="https://gateway.example"
 ```
 
-From the repository root, run Harbor through this workspace and pin the agent
-installed in the task container:
+Run new benchmarks through the standard entry point, from the repository root:
+
+```bash
+uv run --project benchmarks/harbor python -m harbor_adapter run \
+  --config benchmarks/harbor/configs/qemu-alpine-ssh.json
+```
+
+This example pins the QEMU task revision and uses `deepseek/deepseek-flash`.
+Set the endpoint and credentials for that model, or copy the configuration and
+change `agents[].model_name`, task selection, and budgets for another run. These
+are ordinary Harbor JSON configurations; the entry point supplies the standard
+verifier and cache policy. It accepts the repository's `NanoPyCodeAgent` adapter
+with budget kwargs; credentials belong in the environment.
+
+The entry point builds the current working tree, including uncommitted changes,
+into a wheel and checks its Python sources against the recorded snapshot. It
+also snapshots the adapter, records source and wheel hashes, and uses that
+snapshot throughout the run. Inputs and logs live in `jobs/<job-name>-input/`;
+Harbor outputs remain in `jobs/<job-name>/`. Existing jobs are never overwritten.
+`summary.json` records scores, setup/preflight status, cache use, and verifier
+attempts. An infrastructure exception has no score; it is not converted to zero.
+The command exits nonzero for execution errors or incomplete results, while a
+completed score of zero is a valid benchmark result.
+
+Use `--job-name <unique-name>` for a readable run name. Add `--prepare-only` to
+build and record inputs without starting containers. Add `--install-only` to
+exercise dependency installation and preflight without calling the model or
+executing the official tests. A full run still performs setup and preflight in
+its own fresh container; an install-only check does not resume into model work.
+
+### Verified dependencies and preflight
+
+The standard adapter automatically uses `.cache/nanopy-harbor/`, shared across
+runs and ignored by Git. Override it with `NANOPY_HARBOR_CACHE` or the entry
+point's `--cache-dir`. On APT systems it refreshes the package index, resolves a
+SHA256 download plan, restores matching cached files, and downloads missing
+ones. Successful downloads are verified and saved before installation, so
+APT cleanup hooks cannot erase the shared copies. A different version or digest
+does not reuse an old cached file. Corrupt files fail setup before model work.
+Other package managers use Harbor's regular dependency installation.
+
+To import previously verified downloads once:
+
+```bash
+uv run --project benchmarks/harbor python -m harbor_adapter import-apt \
+  /path/to/bootstrap-cache/manifest.json
+```
+
+The manifest is a JSON array of objects with `url`, `filename`, `size`, and
+`sha256`, accompanied by `packages/<filename>` files. Import validates every
+file. At use time, the current APT index must independently match each entry.
+Copying the entire shared cache to another machine is also supported. A new
+machine with no matching cache still needs the original downloads to be
+available; this mechanism does not replace package sources or invent missing
+versions.
+
+Verifier dependency profiles are tied to reviewed task revisions in
+`src/harbor_adapter/bootstrap.py`. The current QEMU profile installs `sshpass`
+and warms the original verifier's uv 0.9.5, Python 3.13, pytest 8.4.1, and
+pytest-json-ctrf 0.3.5 dependencies using `pytest --version`. It runs no task
+assertions or solution code. Unknown QEMU revisions fail setup pending profile
+review; tasks without a profile explicitly report `not_configured`. Add a
+profile and regression tests when another task needs verifier preflight.
+
+Every installation writes `agent/bootstrap.json`, including partial progress
+on failure. Preflight failure stops the trial before any model call. The
+official verifier script remains unchanged and may still access the network
+when it runs; a successful preflight is not a guarantee against later outages.
+Full standard runs enable the timeout retry policy below by default, with zero
+whole-trial retries. CI checks these contracts without model calls or Docker.
+
+### Direct Harbor invocation
+
+For adapter development or comparison with published versions, the lower-level
+Harbor interface remains available. Pin the agent installed in the container:
 
 ```bash
 uv run --project benchmarks/harbor harbor run \
   --task terminal-bench/openssl-selfsigned-cert \
   --agent harbor_adapter:NanoPyCodeAgent \
   --agent-kwarg git_ref=<commit-sha> \
   --model anthropic/claude-sonnet-4-6 \
+  --verifier harbor_adapter:RetryingVerifier \
+  --verifier-timeout-multiplier 4 \
   --env docker \
   --n-concurrent 1 \
   --n-attempts 1
@@ -65,6 +140,11 @@ The adapter also asks the agent to write an ATIF-v1.7 trajectory directly to
 ATIF output and backfills prompt, completion, cache-token, and cost totals into
 the agent result. Missing, invalid, or partial trajectories remain explicitly
 diagnosed; unknown usage or cost is not reported as zero.
+If any costs are estimated, `final_metrics.extra` records
+`cost_is_estimated: true` and the estimated portion in `estimated_cost_usd`.
+The adapter preserves both fields in the agent result's `metadata.trajectory`,
+including for partial totals. Estimates with a generation ID remain eligible
+for billing reconciliation; a successful lookup replaces the estimate.
 
 By default, the adapter strips the first provider prefix from `--model` and
 passes the result to nanoPyCodeAgent as `ANTHROPIC_MODEL`. Set
@@ -74,6 +154,39 @@ precedence. Harbor-native provider credentials and configured base URLs are
 also normalized to the `ANTHROPIC_*` variables expected by nanoPyCodeAgent's
 SDK.
 
+## Retry verifier timeouts
+
+The example above uses `harbor_adapter:RetryingVerifier` for single-step Linux
+tasks. It runs the original test script up to **three times total**, retrying
+only when the script exceeds the task's native `[verifier].timeout_sec`. A
+900-second task therefore gets at most three 900-second attempts. A completed
+verification stops immediately, including a reward of zero. Script errors,
+missing or invalid rewards, and cancellation do not trigger another attempt.
+
+`--verifier-timeout-multiplier 4` gives Harbor's outer watchdog room for all
+three attempts plus test upload, process cleanup, and log collection. It does
+not extend each test-script attempt. Conflicting timeout overrides or caps
+that leave insufficient overall time are rejected. Use
+`--verifier-kwarg max_attempts=2` for two total attempts, or `max_attempts=1` for
+one; values above three are rejected.
+
+The runner reuses the same container, agent output, and dependency caches;
+the agent is run once. Test scripts must tolerate being rerun in that
+environment; verifier-side filesystem changes are not rolled back. Timed-out
+process groups and other members of their Linux session are killed before
+retrying. Each attempt's logs and any reward files are archived under
+`verifier/attempts/1/`, `2/`, and `3/`. The runner clears current verifier
+outputs before the next attempt, so a stale reward cannot determine its score.
+`verifier/retry-summary.json` records each attempt's status, duration, and exit
+code when available. Exhausting the attempts still produces Harbor's
+`VerifierTimeoutError`.
+
+This verifier requires `python3` and `bash` in the container, which the
+nanoPyCodeAgent adapter installs. These options apply to new runs; they cannot
+restore a deleted container or missing task outputs from an old trial.
+Harbor's ordinary `--max-retries` setting reruns the entire trial, including
+the agent, and is independent of these verifier attempts.
+
 ## Test the adapter
 
 ```bash

benchmarks/harbor/README.zh-CN.md

@@ -17,14 +17,72 @@ export ANTHROPIC_API_KEY="..."
 export ANTHROPIC_BASE_URL="https://gateway.example"
 ```
 
-从仓库根目录通过这个 workspace 运行 Harbor，并固定安装到 task 容器中的 agent：
+新的 benchmark 统一从仓库根目录通过标准入口运行：
+
+```bash
+uv run --project benchmarks/harbor python -m harbor_adapter run \
+  --config benchmarks/harbor/configs/qemu-alpine-ssh.json
+```
+
+此示例固定了 QEMU 题目版本，模型为 `deepseek/deepseek-flash`。请配置对应的
+endpoint 和凭证；测试其他模型或题目时，复制配置并修改 `agents[].model_name`、
+题目选择及预算。这是普通的 Harbor JSON 配置，入口会补齐标准验证器和缓存策略。
+入口接受本仓库的 `NanoPyCodeAgent` adapter 和预算参数，凭证通过环境变量传入。
+
+入口会将当前工作区（包括未提交修改）构建为 wheel，并核对其 Python 源码与保存的
+快照一致；同时保存 adapter 快照及源码、wheel 的哈希，整个运行都使用该快照。
+输入和日志保存在 `jobs/<job-name>-input/`，Harbor 输出保存在
+`jobs/<job-name>/`，不会覆盖已有运行。`summary.json` 记录评分、安装和预检状态、
+缓存使用情况及验证次数。基础设施异常保留为无评分，不会转换为 0 分。
+执行错误或结果不完整时命令返回非零退出码；正常完成验证得到 0 分仍是有效结果。
+
+可用 `--job-name <unique-name>` 命名运行。`--prepare-only` 只构建并保存输入，
+不启动容器；`--install-only` 只执行依赖安装和预检，不调用模型、不运行官方测试。
+完整运行仍会在自己的新容器里执行安装和预检，不能把 install-only 容器续跑为解题。
+
+### 依赖缓存与预检
+
+标准 adapter 自动使用被 Git 忽略的 `.cache/nanopy-harbor/` 跨运行共享缓存，
+可通过 `NANOPY_HARBOR_CACHE` 或入口的 `--cache-dir` 覆盖。对于 APT 系统，先更新
+索引并解析 SHA256 下载计划，再恢复匹配的缓存文件、下载缺失文件。下载成功后会先
+校验并保存缓存，再安装，避免 APT 清理钩子删除共享副本。版本或哈希不匹配时不会
+复用旧包，文件损坏会在模型调用前终止安装。其他包管理器使用 Harbor 的常规安装。
+
+历史下载只需导入一次：
+
+```bash
+uv run --project benchmarks/harbor python -m harbor_adapter import-apt \
+  /path/to/bootstrap-cache/manifest.json
+```
+
+manifest 是包含 `url`、`filename`、`size`、`sha256` 的 JSON 对象数组，同目录下
+配套 `packages/<filename>` 文件。导入时逐个校验，使用时还必须独立匹配当前 APT
+索引。也可将整个共享缓存复制到其他机器。新机器没有匹配缓存时仍依赖原下载地址
+可用；此机制不会更换软件源或伪造缺失版本。
+
+验证器依赖预检配置位于 `src/harbor_adapter/bootstrap.py`，绑定已检查的题目版本。
+目前 QEMU 配置会安装 `sshpass`，并通过 `pytest --version` 预热原验证器所需的
+uv 0.9.5、Python 3.13、pytest 8.4.1 和 pytest-json-ctrf 0.3.5，不执行题目断言或
+参考解。未知的 QEMU 版本会在安装阶段报错，要求先检查对应预检配置；没有预检配置
+的其他题目明确记录为 `not_configured`。其他题目需要预检时，在这里新增配置及回归测试。
+
+每次安装都会写入 `agent/bootstrap.json`，失败时也保留已完成的步骤。预检失败会
+在模型调用前终止 trial。官方验证脚本保持原样，执行时仍可能访问网络，所以预检
+通过不能保证后续没有网络故障。完整标准运行默认启用下述验证超时重试，整题重试
+次数为零。CI 会验证这些约定，不调用模型、不启动 Docker。
+
+### 直接调用 Harbor
+
+开发 adapter 或对照已发布版本时，仍可使用底层 Harbor 入口，并固定容器内 agent 版本：
 
 ```bash
 uv run --project benchmarks/harbor harbor run \
   --task terminal-bench/openssl-selfsigned-cert \
   --agent harbor_adapter:NanoPyCodeAgent \
   --agent-kwarg git_ref=<commit-sha> \
   --model anthropic/claude-sonnet-4-6 \
+  --verifier harbor_adapter:RetryingVerifier \
+  --verifier-timeout-multiplier 4 \
   --env docker \
   --n-concurrent 1 \
   --n-attempts 1
@@ -56,13 +114,41 @@ adapter 还会要求 agent 将 ATIF-v1.7 trajectory 直接写入
 `/logs/agent/trajectory.json`。Harbor 会把该文件作为 trial 的原生 ATIF 输出采集，
 并将 prompt、completion、cache token 和 cost 汇总回填到 agent result。trajectory
 缺失、无效或指标不完整时会保留明确诊断；未知 usage 或 cost 不会被记成零。
+只要费用中包含估算，`final_metrics.extra` 就会记录 `cost_is_estimated: true`，
+并通过 `estimated_cost_usd` 单独记录估算部分的金额。adapter 会将这两个字段保留到
+agent result 的 `metadata.trajectory`，费用仅部分已知时也一样。带 generation ID
+的估算费用仍会进行账单补查；补查成功后，真实费用会替换估算值。
 
 默认情况下，adapter 会移除 `--model` 中的第一个 provider 前缀，并将结果作为
 `ANTHROPIC_MODEL` 传给 nanoPyCodeAgent。只有自定义 endpoint 要求的实际模型名与
 Harbor 的 `provider/model` 身份不同时，才显式设置 `ANTHROPIC_MODEL`；该覆盖值优先。
 Harbor 原生的 provider 凭证和已经配置的 base URL 也会转换为 nanoPyCodeAgent SDK
 所需的 `ANTHROPIC_*` 变量。
 
+## 验证器超时重试
+
+上面的示例使用 `harbor_adapter:RetryingVerifier`，适用于单阶段 Linux 题目。
+原始测试脚本**总共最多执行 3 次**，仅当脚本超过题目原生的
+`[verifier].timeout_sec` 时才重试。900 秒时限的题目，每次仍最多执行 900 秒。
+验证完成后立即停止，包括得分为 0 的情况。脚本错误、评分缺失或格式错误、取消执行
+都不会触发下一次尝试。
+
+`--verifier-timeout-multiplier 4` 为 Harbor 外层时限预留 3 次尝试以及上传测试、
+清理进程和收集日志的时间，不会延长每次测试脚本的时限。如果其他超时覆盖或上限
+导致总时间不足，会报错。可用 `--verifier-kwarg max_attempts=2` 将总尝试次数设为
+2 次，或用 `max_attempts=1` 只执行一次；不允许设置为超过 3 次。
+
+重试复用同一容器、agent 产物和依赖缓存，agent 只运行一次。测试脚本需要能够在该
+环境中重复执行，验证过程对文件系统的修改不会回滚。超时后会先终止测试进程组以及
+同一 Linux session 中的其他进程，再开始下一次尝试。每次日志及评分文件归档到
+`verifier/attempts/1/`、`2/`、`3/`。下一次尝试开始前会清理当前验证输出，避免旧
+评分被误用。`verifier/retry-summary.json` 记录每次状态、耗时及可用的退出码。
+所有尝试均超时后，仍以 Harbor 的 `VerifierTimeoutError` 结束。
+
+容器需要 `python3` 和 `bash`，nanoPyCodeAgent adapter 会安装它们。这些选项用于
+新运行，无法恢复旧 trial 已删除的容器或缺失的任务产物。Harbor 原生的
+`--max-retries` 会重新执行整道题（包括 agent），与这里的验证尝试次数相互独立。
+
 ## 测试 adapter
 
 ```bash

benchmarks/harbor/configs/qemu-alpine-ssh.json

@@ -0,0 +1,21 @@
+{
+  "n_concurrent_trials": 1,
+  "n_attempts": 1,
+  "agents": [
+    {
+      "name": "harbor_adapter:NanoPyCodeAgent",
+      "model_name": "deepseek/deepseek-flash",
+      "kwargs": {
+        "max_turns": 100,
+        "max_tokens": 65536,
+        "time_budget_seconds": 720
+      }
+    }
+  ],
+  "tasks": [
+    {
+      "name": "terminal-bench/qemu-alpine-ssh",
+      "ref": "sha256:60b7050b0e0aa51641208cf65766743d340e59575db2c4d2f8628240846c2a28"
+    }
+  ]
+}