chore(eval): put the evaluation harness, ledger and current results in the repo

docs/reports/2026-09-19-v0.4.32-evaluation.md

@@ -0,0 +1,128 @@
+# Orca 评测报告 — v0.4.32（2026-09-19）
+
+来源 `origin/main` @ `5970b40c`（v0.4.32，含本轮 26 个 issue 修复与你那批 CI 稳定化），从源码重建 host
+与 musl 二进制，跑 L1 全量套件 + Terminal-Bench 2.0 全量（88 题）。
+
+## 协议（数字可比的前提）
+
+| 项 | 值 |
+|---|---|
+| Source | `5970b40c`（tag 附近，v0.4.32） |
+| Binary | 静态 `x86_64-unknown-linux-musl`（`messense/rust-musl-cross` 构建）+ host `target/release/orca` |
+| Harness | Harbor 0.20.0，`terminal_bench.orca_agent:OrcaInstalledAgent`（仓库内 adapter） |
+| Model | adapter 默认 → routed `deepseek-v4-pro`（`reason: default_pro`） |
+| Settings | `-k 1 -n 6`，一题一次尝试 |
+| Tasks | 88 / 89，`terminal-bench/qemu-startup` 按 `terminal_bench/quarantine.json` 排除 |
+| Window | 2026-09-19 23:31 → 2026-09-20 03:13（本地，含 13 个 900–3600 s 长任务） |
+
+## 结论速览
+
+| 层 | 结果 | 对比 |
+|---|---|---|
+| L1（26 次运行，24 个套件） | **全绿**，0 失败、0 "blocked by" 注解 | 上一基线 155 通过 / 32 失败 |
+| TB2 2.0 | **mean 0.7955**（70/88）；计分题内均值 0.8434 | v0.4.31 修正后 0.761 |
+| 新 issue | **#114**、**#115** | 之前 0 个 open |
+
+明细：[`2026-09-19-tb2-v0432-summary.json`](2026-09-19-tb2-v0432-summary.json)（每题 reward / exception /
+agent exit / turns / 轨迹字节 / 耗时），台账：[`evaluation-ledger.md`](evaluation-ledger.md)。
+
+## L1：26 次运行全绿（24 个套件；`sandbox_probe.py` 运行两种模式）
+
+`bash scripts/eval/sweep.sh jobs/eval-sweep/20260919-v0432`（约 40 分钟，无网络/无 API key）。
+上一轮为红、这轮转绿的关键项：
+
+- `fault_injection.py` **13/13**（#67 重复 tool-call id、#68 5xx 重试、#62 截断重试）
+- `wire_param_probe.py` **14/14**（#77 裸字符串 input、#78 请求 id 保留、#79 请求错误不再杀 server）
+- `acp_param_probe.py` **20/20**（#80 参数错误 `-32602`）、`daemon_probe.py` 12/12
+- `signal_probe.py` **2/2**（#72：130/143 + terminal + 无孤儿）
+- `permission_probe.py` 16/16（#74）、`trust_probe.py` 8/8 + `trust_race_probe.py`（#75/#81）
+- `workflow_probe.py` 19/19（#82–#85）、`linux_shell_probe.py`（#70）、`sandbox_probe.py`（#69）
+
+## TB2 2.0：mean 0.7955
+
+| 分类 | 数量 | 说明 |
+|---|---|---|
+| reward 1.0 | 70 | |
+| reward 0.0 | 13 | 其中 3 题是本报告的两个产品缺陷所致，6 题是超时，4 题是纯能力 |
+| `AgentTimeoutError` | 13 | 其中 **7 题验证器已通过**（活儿干完了没收手）：`filter-js-from-html`、`gpt2-codegolf`、`largest-eigenval`、`mcmc-sampling-stan`、`model-extraction-relu-logits`、`regex-chess`、`tune-mjcf` |
+| `EnvironmentStartTimeoutError` | 4 | 本机 Docker 6 并发噪音，已复跑（见下） |
+| `NonZeroAgentExitCodeError` | 1 | exit 137 = **#114 自我 kill** |
+| `RuntimeError` | 1 | 拉镜像 EOF，环境噪音 |
+
+### 产品缺陷 1：instruction 在 argv 里，进程匹配命令杀掉 agent 自己（#114）
+
+`terminal-bench/make-mips-interpreter`：agent 用 `/proc` 扫描并 `kill -9` cmdline 含 `vm.js` 的进程，
+而 `orca exec … -- '<instruction>'` 的 argv 里正带着含 `vm.js` 的指令 → 自杀，exit 137，26 MB 轨迹断在
+第 43 轮，TB2 记成 agent error。issue 内给出 19 行本地复现（`pgrep -fl $MARKER` 能看到 orca 自己，
+`pkill -9 -f $MARKER` → 137）。建议 adapter 把 prompt 走 stdin（`printf '%s' "$instruction" | orca exec …`）。
+
+### 产品缺陷 2：`lifetime: "workspace"` 的服务在会话结束时被杀（#115）
+
+文档承诺 *"It survives the task that started it and is stopped explicitly"*，任务取消路径也照此过滤
+（`crates/orca-runtime/src/tasks.rs:5185`），但会话结束走的 `RuntimeShellSessionManager::terminate_all()`
+（`crates/orca-runtime/src/shell_session.rs:955`，由 `terminal_service.rs:665/672` 触发）对所有 shell 一律
+`Cancelled`，不看 lifetime。受害任务：`install-windows-3-11`（QEMU+VNC，会话内已用 QMP screendump 验证画面）、
+`kv-store-grpc`（`server.py` 会话内客户端往返成功）。新探针
+[`scripts/eval/workspace_lifetime_probe.py`](../../scripts/eval/workspace_lifetime_probe.py) 直接测契约：
+
+```
+[FAIL] lifetime=workspace exit=0 terminal=True ticks_at_exit=31 ticks_after=31 — 会话结束后应继续运行
+[PASS] lifetime=task      exit=0 terminal=True ticks_at_exit=31 ticks_after=31 — 随会话停止（符合预期）
+```
+
+### 基础设施噪音与复跑
+
+4 次 `EnvironmentStartTimeoutError`（`pytorch-model-recovery`、`mteb-leaderboard`、`crack-7z-hash`、
+`video-processing`）+ 1 次拉镜像 EOF（`vulnerable-secret`）在 `-n 1` 下逐题复跑：**5/5 验证器通过**
+（其中 `pytorch-model-recovery` 满分，等于 #61 的 `--` prompt 修复在 TB2 端到端确认）。按验证器通过计入，
+本轮解法率为 **75/88 = 85.2%**（harbor 正式 mean 仍按全部 88 题计为 0.7955）。
+
+## Agent 水平定位
+
+| 维度 | 数值 / 题目 |
+|---|---|
+| TB2 mean（单次尝试） | 0.7955（70/88）；计分题内 0.8434 |
+| 三连跑（同协议） | 0.494 → 0.761 → **0.7955** |
+| 超时 | 13（15%），其中 7 题验证器已通过 |
+| 产品缺陷丢分 | 3 题（#114 ×1、#115 ×2） |
+| 纯能力不足 | 4 题（`chess-best-move`、`pypi-server`、`sam-cell-seg`、`adaptive-rejection-sampler`） |
+| 可靠性 | L1 24/24；TB2 无 orca 崩溃/泄漏/轨迹丢失 |
+
+- **工程可靠性是生产级**：协议、ACP、信号、信任并发、工作流、沙箱、MCP、恢复、保留全部有自动化契约；
+  TB2 里没有一次崩溃类错误。
+- **能力层约 4/5 任务可解**：70 题覆盖仓库恢复/编辑、构建、协议与服务实现、密码学/逆向、数据处理。
+- **瓶颈是长时程效率**：13 次超时里 7 次任务已达标只是没停手，6 次是真算不完（训练、模拟器、大编译）。
+
+外部对比：TB2 有公开榜（[tbench.ai](https://www.tbench.ai/leaderboard/terminal-bench/2.0)、
+[benchlm.ai](https://benchlm.ai/benchmarks/terminal-bench-2)、[BenchmarkList](https://benchmarklist.com/benchmarks/vals_terminal_bench_2/)），
+但本机网络把外网域名解析到非公网 IP、抓不到具体数值，因此本报告不给不可核实的对比；且公开条目多为
+k≥5 + 厂商 scaffold，与本仓库"单次尝试 + `orca exec` + routed `deepseek-v4-pro`"不同口径。
+
+现实上限估计 **0.87–0.90**：修 #114/#115 立即回收 3 题（≈ +3.4 pt），超时池 13 题靠收尾纪律与 effort
+实验逐步回收，剩下 4 题属模型能力。
+
+## 复现
+
+```bash
+# L1（无网络）
+bash scripts/eval/sweep.sh jobs/eval-sweep/$(date +%Y%m%d)
+
+# TB2（Docker + API key）
+harbor run -d "terminal-bench/terminal-bench-2" \
+  --agent "terminal_bench.orca_agent:OrcaInstalledAgent" \
+  -k 1 -n 6 --exclude-task-name "terminal-bench/qemu-startup" \
+  --mounts '[{"type":"bind","source":"'"$PWD"'/target/x86_64-unknown-linux-musl/release","target":"/mnt/orca-bin","read_only":true}]'
+
+# 产出可提交的 summary（原始 jobs/ 与轨迹不入库）
+python3 scripts/eval/summarize.py jobs/full-88-v0432/<timestamp> \
+  --out docs/reports/2026-09-19-tb2-v0432-summary.json --source-commit 5970b40c \
+  --model "deepseek-flash (routed: deepseek-v4-pro)" --attempts 1 --concurrency 6
+```
+
+## 产物说明
+
+- **入库**：本报告 + [`summary.json`](2026-09-19-tb2-v0432-summary.json)（34 KB，88 行 per-task）+
+  [`evaluation-ledger.md`](evaluation-ledger.md) / [`.json`](evaluation-ledger.json) + `scripts/eval/` 夹具。
+- **不入库**：`jobs/` 原始轨迹与容器日志（GB 级、含 workspace 内容）。三个关键题目的完整轨迹
+  （#114 的 `make-mips-interpreter`、#115 的 `install-windows-3-11` 与 `kv-store-grpc`）已 gzip 归档在
+  仓库外，需要时可附到对应 issue。

docs/reports/evaluation-ledger.json

@@ -0,0 +1,60 @@
+{
+  "schema_version": 1,
+  "updated_at": "2026-09-19",
+  "definition": {
+    "tb2_mean": "Harbor aggregation over every scheduled trial; a trial without a verifier reward counts as 0",
+    "tb2_mean_scored_only": "mean over the trials that produced a verifier reward",
+    "l1": "scripts/eval/sweep.sh suites, no network or API key required"
+  },
+  "entries": [
+    {
+      "date": "2026-09-15",
+      "source_commit": "3aa4efdfa",
+      "version": "v0.4.31",
+      "tasks": 89,
+      "settings": { "attempts": 1, "concurrency": 3, "environment_build_timeout_multiplier": 2.0 },
+      "tb2_mean": 0.494,
+      "tb2_passed": 44,
+      "report": "docs/reports/2026-09-15-terminal-bench-2-v0.4.31.md"
+    },
+    {
+      "date": "2026-09-16",
+      "source_commit": "48fa0a458",
+      "version": "v0.4.31+fixes",
+      "tasks": 88,
+      "excluded_tasks": ["terminal-bench/qemu-startup"],
+      "settings": { "attempts": 1, "concurrency": 3, "environment_build_timeout_multiplier": 2.0 },
+      "tb2_mean": 0.727,
+      "tb2_mean_infra_corrected": 0.761,
+      "report": "docs/reports/2026-09-16-terminal-bench-2-v0.4.31-fixed.md"
+    },
+    {
+      "date": "2026-09-19",
+      "source_commit": "5970b40c",
+      "version": "v0.4.32",
+      "tasks": 88,
+      "excluded_tasks": ["terminal-bench/qemu-startup"],
+      "settings": { "attempts": 1, "concurrency": 6 },
+      "tb2_mean": 0.7955,
+      "tb2_mean_scored_only": 0.8434,
+      "tb2_passed": 70,
+      "tb2_scored": 83,
+      "tb2_errored": 19,
+      "exceptions": {
+        "AgentTimeoutError": 13,
+        "EnvironmentStartTimeoutError": 4,
+        "NonZeroAgentExitCodeError": 1,
+        "RuntimeError": 1
+      },
+      "infra_reruns": {
+        "tasks": 5,
+        "verifier_passed": 5,
+        "mean_if_counted": 0.8523
+      },
+      "l1": { "runs": 26, "unique_suites": 24, "failed": 0, "note": "sandbox_probe.py runs in default and privileged modes" },
+      "issues_filed": [114, 115],
+      "report": "docs/reports/2026-09-19-v0.4.32-evaluation.md",
+      "summary": "docs/reports/2026-09-19-tb2-v0432-summary.json"
+    }
+  ]
+}

docs/reports/evaluation-ledger.md

@@ -0,0 +1,54 @@
+# Evaluation ledger
+
+One row per evaluation round. **The protocol matters more than the number**: two entries are
+only comparable when the source commit, harness, model, attempts/concurrency and task set
+agree. Every entry links to the run's committed summary (per-task outcomes) and to the human
+report; raw trajectories, container logs and `jobs/` output stay out of git.
+
+| date | source | tasks | settings | TB2 mean | L1 | report / summary |
+|---|---|---|---|---|---|---|
+| 2026-09-15 | `3aa4efdfa` (v0.4.31) | 89 | `-k 1 -n 3`, build-timeout ×2 | **0.494** (44/89) | — | [report](2026-09-15-terminal-bench-2-v0.4.31.md) |
+| 2026-09-16 | `48fa0a458` (v0.4.31 + evidence fixes) | 88 (`qemu-startup` excluded) | `-k 1 -n 3`, build-timeout ×2 | **0.727** raw · **0.761** infra-corrected | — | [report](2026-09-16-terminal-bench-2-v0.4.31-fixed.md) |
+| 2026-09-19 | `5970b40c` (v0.4.32, 26 merged fixes) | 88 (`qemu-startup` excluded) | `-k 1 -n 6` | **0.7955** (70/88) · scored-only 0.8434 | **26/26 runs green** (24 suites; `sandbox_probe.py` twice) | [report](2026-09-19-v0.4.32-evaluation.md) · [summary](2026-09-19-tb2-v0432-summary.json) |
+
+Machine-readable form: [`evaluation-ledger.json`](evaluation-ledger.json).
+
+## How a number is defined
+
+- **TB2 mean** = Harbor's own aggregation: every scheduled trial counts, and a trial that
+  never produced a reward (infrastructure failure, timeout kill) counts as 0. The
+  scored-only mean (denominator = trials that produced a reward) is reported next to it
+  because it separates "the agent failed the task" from "the run lost the trial".
+- **L1** = the local suite sweep (`scripts/eval/sweep.sh`): protocol contracts, ACP, signals,
+  trust races, workflow, sandbox, MCP, recovery, retention, plus the fault-injection battery.
+  It needs no network or API key and takes ~30 minutes.
+- **Infra corrections** are called out explicitly in each report rather than silently folded
+  into the mean (e.g. the 5 environment failures in the 2026-09-19 run were re-run one at a
+  time and all five passed).
+
+## Reproducing a row
+
+```bash
+# L1 (no network, no API key)
+bash scripts/eval/sweep.sh jobs/eval-sweep/$(date +%Y%m%d)
+
+# TB2 (Harbor + Docker + API key), then turn the run into the committed summary
+harbor run -d "terminal-bench/terminal-bench-2" \
+  --agent "terminal_bench.orca_agent:OrcaInstalledAgent" \
+  -k 1 -n 6 --exclude-task-name "terminal-bench/qemu-startup" \
+  --mounts '[{"type":"bind","source":"'"$PWD"'/target/x86_64-unknown-linux-musl/release","target":"/mnt/orca-bin","read_only":true}]'
+python3 scripts/eval/summarize.py jobs/full-88-<version>/<timestamp> \
+  --out docs/reports/<date>-tb2-<version>-summary.json --source-commit "$(git rev-parse --short HEAD)"
+```
+
+## Findings that left a test behind
+
+Evaluation only pays off when a finding becomes a permanent check:
+
+| finding | permanent check |
+|---|---|
+| #72 SIGINT/SIGTERM killed the run without a terminal record | `tests/signal_contract.rs`, `scripts/eval/signal_probe.py` |
+| #69 `bwrap` never probed when `cwd` is an ancestor of `/usr/bin` | `crates/orca-tools` bwrap guard unit tests, `scripts/eval/sandbox_probe.py` |
+| #81 concurrent `orca trust add/remove` lost decisions | `tests/jsonl_surface_differential.rs`, `scripts/eval/trust_race_probe.py` |
+| #115 a `lifetime: workspace` service is killed at session end | `scripts/eval/workspace_lifetime_probe.py` (in the sweep) |
+| #67 duplicate provider tool-call id aborted the session | `scripts/eval/fault_injection.py` (`duplicate_tool_call_id`), `model_response` unit tests |