feat(evals): add coding-task evaluation framework (#2084)

package.json

@@ -19,6 +19,8 @@
 		"solheim-smoke:check-types": "tsc -p scripts/solheim-smoke/tsconfig.json",
 		"test:agent-evals:unit": "node --import tsx --test scripts/agent-evals/*.test.ts",
 		"agent-evals:check-types": "tsc -p scripts/agent-evals/tsconfig.json",
+		"coding-evals:side": "node --import tsx scripts/agent-evals/coding-side.mts",
+		"coding-evals:compare": "node --import tsx scripts/agent-evals/coding-compare.mts",
 		"test:mutation-ci": "node --test scripts/stryker-diff.test.mjs",
 		"lifecycle:model-check": "tsx scripts/check-task-lifecycle.ts && tsx scripts/check-task-store-concurrency.ts && tsx scripts/check-provider-handoff-scheduler.ts && pnpm cleanup-protocol:model-check && pnpm parser-scope:model-check && tsx scripts/check-completion-persistence.ts && tsx scripts/check-delegated-mode-readers.ts && pnpm tool-turn:model-check",
 		"fanout-protocol:model-check": "tsx scripts/check-task-fanout-protocol.ts",

scripts/agent-evals/README.md

@@ -80,8 +80,10 @@ them:
   message still appears in `completionMessageSeen`.
 - `SESSION_LOST` covers disconnects and interrupts. A manual stop after
   completion output stays `SESSION_LOST`.
-- The usage drain runs only after a `taskCompleted` event. It stays inside the
-  bounded drain window.
+- The usage drain runs after a `taskCompleted` event and after an observer stop
+  that followed the final `completion_result` message. It stays inside the
+  bounded drain window. A plain stop, a run timeout, and a session loss get no
+  drain.
 
 ## Checks
 
@@ -92,13 +94,94 @@ pnpm agent-evals:check-types
 
 Both run offline. No check spawns VS Code or touches a provider.
 
-## Follow-up milestones
+## Coding evaluation framework
+
+The coding framework layers an evaluation on the substrate. It grades runs,
+repeats them, and compares two exact commits. It stays independent from any
+release process: a maintainer runs it locally or on demand.
+
+| File                   | Purpose                                                                                                                                                  |
+| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `coding-fixtures.ts`   | Five frozen coding fixtures. Each ships a known-good workspace, one prompt, a tool policy, limits, hidden acceptance tests, and two reference solutions. |
+| `coding-suite.ts`      | Suite identity `coding-basic-v1`, registration, and the suite content digest.                                                                            |
+| `coding-workspace.ts`  | The immutable initial-state digest, fresh workspace builds, and fresh grading copies.                                                                    |
+| `coding-grader.ts`     | The grader contract, version, driver invocation, and the local subprocess runtime.                                                                       |
+| `grader-driver.mjs`    | The acceptance-test driver. Runs the workspace tests and writes one result JSON file.                                                                    |
+| `container-runtime.ts` | The default grading runtime. Builds and runs the locked-down container invocation.                                                                       |
+| `coding-outcomes.ts`   | Outcome rules: `correct`, `incorrect`, `timed out`, `grading failure`, `infrastructure failure`.                                                         |
+| `coding-pricing.ts`    | The operator-maintained price table and the run cost estimate.                                                                                           |
+| `coding-metrics.ts`    | Per-run metrics: requests, usage, tool calls, cost, unavailable telemetry, cost per successful task.                                                     |
+| `coding-run.ts`        | One repetition: substrate run, final-workspace capture, grading, outcome, metrics.                                                                       |
+| `coding-pins.ts`       | Pinned comparison inputs and the incompatibility check.                                                                                                  |
+| `coding-stats.ts`      | Three repetitions, success rates with Wilson intervals, paired differences.                                                                              |
+| `coding-report.ts`     | The self-contained JSON report and the Markdown renderer.                                                                                                |
+| `coding-compare.ts`    | The comparison orchestration and the side result shape.                                                                                                  |
+| `coding-side.mts`      | One comparison side: rebuilds the evaluated tree, then all fixtures times N repetitions on it, graded, with pinned inputs.                               |
+| `coding-compare.mts`   | Two-commit comparison driver. Writes `comparison-report.json` and `comparison-report.md`.                                                                |
+| `coding-trust.ts`      | The comparison trust policy: approved full commit SHAs, resolution, the approval gate, and the side environment allowlist.                               |
+| `fs-atomic.ts`         | Atomic file publication through same-directory temp files and renames.                                                                                   |
+
+### Grading
+
+- The grader is model-free and credential-free. It judges the final workspace state only.
+- The final workspace is captured as bounded file data and rebuilt as a fresh copy. Hidden acceptance tests enter only that copy, so the agent can never read or change them, and one run can never inherit another run's files.
+- The default runtime executes the driver in a uniquely named disposable container with no network, no capabilities, no privilege escalation, a read-only root filesystem, and fixed CPU, memory, and process limits. No environment variable passes through. Only trusted repo code runs with the provider credential. Fixtures and extension code from outside pull requests never do.
+- The grader has its own wall-clock limit. On expiry the runtime removes the container by name and verifies the removal before it deletes any output. A grader that hangs, crashes, or loses its runner is a `grading failure`. It is never an `incorrect` solution. The image pull happens before the timed run and never consumes grading time. The runtime resolves the immutable grader image digest from the local engine and the side records it in the comparison pins; it never fabricates one.
+- The driver's result file is untrusted data: it is written inside the container where agent-written test code executes. The host reads it with a size bound and validates shape, bounds, and internal consistency. An invalid or inconsistent result is a `grading failure`, never a verdict.
+- Each fixture ships two reference solutions. The known-correct solution must pass. The deliberately incorrect one passes the visible tests and must fail. Unit tests enforce both on every fixture.
+- `createLocalSubprocessGradingRuntime` runs the same driver directly on the host. It does not give container isolation. It exists for unit tests and explicit local checks only and must never back a recorded comparison.
+- The live `coding-evals:side` and `coding-evals:compare` entry points reject `--allow-unisolated-grading` instead of ignoring it. A recorded comparison always grades in the container runtime.
+- The driver publishes its result file atomically: the content lands in a unique same-directory temp file and is renamed into place. A failed write or rename removes the temp file, and the host treats a missing or invalid result as a grading failure.
+
+### Comparisons
+
+A paired comparison pins both commit SHAs, the suite digest, the settings digest, the model identifier, the grader version, the resolved grader image digest, and the runner environment on both sides. Any difference in a pinned value marks the pair incompatible. The report records the values and flags the pair instead of dropping it.
+
+The default is three repetitions per fixture per commit. Reports show success rates with Wilson 95% intervals and paired differences. They make no significance claim: three repetitions can show large regressions, not small improvements.
+
+Runs with incomplete token or cost accounting are flagged and excluded from aggregate cost comparisons. Missing token or cost values stay missing. The report states the price-table identifier and date and how cache reads and writes are priced. Cost per successful task divides the total attempt cost by the number of `correct` runs. It is unavailable when no run is correct, when a correct run lacks complete cost data, or when any attempt, including a failed one, lacks complete cost data. Duplicate-call detection and HTTP retry counts are reported as unavailable.
+
+### Trust policy for comparisons
+
+A comparison runs the extension code of two commits with the provider credential. A commit runs only when its full 40-hex SHA is listed in an explicit trust policy file. Ref names are resolved first and checked against the policy before any worktree is created or any process starts. A policy entry is a full SHA; short SHAs and ref names are rejected.
+
+```json
+{ "version": 1, "approvedCommits": ["<full-40-hex-commit-sha>"] }
+```
+
+The comparison driver, fixtures, grader, and report builder all run from the trusted checkout that invokes `coding-evals:compare`. Only the extension host loads an evaluated worktree. The side process receives only the allowlisted environment plus the one named credential variable. The driver creates each side's fresh detached worktree of the approved commit. Each side then constructs its own fresh detached worktree of that same commit, installs the frozen lockfile dependencies and builds inside it, and launches every host against it, so each side runs only its own compiled extension of the approved commit. The install and the build consume the side's own wall clock (`--side-timeout-minutes`, default 120).
+
+The side script runs the same gate on its own inputs. It requires `--trust-file`, resolves the full commit of its `--evaluated-root`, and rejects an unapproved or unreadable commit before it creates a host or reads the credential. It never executes a caller-prepared tree. After the gate it constructs a fresh detached worktree of the approved commit in private scratch space, verifies that the worktree resolves to the approved commit and holds no extra content, and then installs the dependencies from the frozen lockfile and builds inside that worktree with credential-free, allowlisted children. The host loads only build output this side produced from the approved commit, so dirty files, untracked files, ignored dependencies, stale build output, and symlinks in the caller's tree can never execute. The side removes its worktree on completion, on failure, and on interruption. The `--credential-env` name must come from the supported provider variable allowlist; an arbitrary name can never forward an unrelated parent secret. Worktree install and build children also receive only an allowlisted environment, so the driver's secrets never reach them.
+
+Every driver child, the side included, runs detached in its own process group and is tracked by one guard. The driver's guard tracks the side; the side's guard tracks its own worktree, install, and build. A SIGINT or SIGTERM that reaches only the driver terminates the side group through a bounded SIGTERM-then-SIGKILL escalation. The side starts the same escalation for its preparation children and waits for it before it removes its worktree or exits. The driver's worktree cleanup also waits for the driver's escalation before it removes any tree.
+
+### Output publication contract
+
+Artifacts are published atomically: content lands in a same-directory temp file and is renamed into place. A reader never sees a partial file.
+
+A side run invalidates its output directory first. Before the first repetition executes, the side removes any previous `side-results.json` and then the whole `artifacts` tree, so stale results from an earlier rerun can never survive. The authoritative result goes first, so an error or an interruption during invalidation leaves the generation incomplete, and consumers fail closed until the new atomic publication. `side-results.json` is the completion marker: it appears only after every repetition of the current invocation finished, and consumers must require it and treat its absence as a failed side. A side rerun that cannot remove a stale artifact fails before any repetition runs.
+
+The side process arms one flag when it receives SIGINT or SIGTERM. The flag starts no further repetition and blocks the completion marker, so an interrupted side exits with the conventional signal code and publishes nothing.
+
+The comparison reports are published as one generation. `comparison-report.json` and `comparison-report.md` carry the report's `generatedAt` stamp, and `comparison-report.complete` holds that stamp; it is written last. Consumers must require the marker and match its stamp against the report files. `readPublishedComparisonReport` enforces this: it returns the report only when the marker exists and its stamp matches both files. The marker is removed before publication starts, so a crash between the two file publications leaves a detectable incomplete generation instead of a mixed one. A thrown failure rolls both files back; a failed comparison leaves no report files.
+
+### Commands
+
+```sh
+# Unit tests and type checks. Offline. No VS Code, no provider, no containers.
+pnpm test:agent-evals:unit
+pnpm agent-evals:check-types
+
+# One comparison side against the real provider. Requires the trust policy,
+# the credential environment variable, and the container runtime for grading.
+pnpm coding-evals:side --reps 3 --out /tmp/side-out --trust-file trust-policy.json
+
+# Full paired comparison of two approved commits.
+pnpm coding-evals:compare --baseline <sha> --candidate <sha> --trust-file trust-policy.json --reps 3 --out /tmp/compare-out
+```
 
-The substrate supports these planned steps. None of them exist in this
-change:
+### Follow-up milestones
 
-1. Five frozen, real coding task fixtures with known-good workspaces.
-2. Independent credential-free graders that judge workspace state only.
-3. Paired exact trusted commit comparisons on the same fixtures.
-4. Repeated runs with uncertainty reporting.
-5. Advisory release integration aligned to the actual release workflows.
+1. Release integration: trigger runs, store baselines, and promote them. See issue #2085.
+2. Later suites: repository navigation, debugging, and cost stress, toward the 21-task set.
+3. Optimization experiments: per-request context instrumentation, non-progress loop detection, tool-output filtering, compaction strategies, and dynamic tool-schema loading.