Standardize ExploitBench task configuration and harnesses

.gitignore

@@ -24,6 +24,7 @@ venv/
 
 # Inspect logs
 logs/
+run-artifacts*/
 
 # OS
 .DS_Store

docs/checkpointing.md

@@ -0,0 +1,64 @@
+# Durable Hawk recovery
+
+Enable checkpoints explicitly for long runs. Each harness must run the checkpoint-aware ExploitBench wrapper; older task commits cannot be made resumable just by adding configuration.
+
+```yaml
+# Merge these settings into the selected Hawk eval-set configuration.
+checkpoint:
+  enabled: true
+  trigger:
+    type: time
+    every: 600
+  sandbox_paths:
+    default:
+      - /rlenv/workspace
+      - /home/agent
+  max_consecutive_failures: 3
+retry_attempts: 10
+retry_immediate: false
+```
+
+Also set `fail_on_error: true` in each `exploit_bench` task's arguments. Without it, a task with an errored sample can appear complete to the evaluation-set retry loop. Use one vulnerability per task item and an independent evaluation set per sample when using delayed retries: otherwise retries wait for the other tasks to finish. Pin the task commit, native CLI versions and provider route for reproducible restarts. Worker placement is controlled separately by Kubernetes; this configuration does not enforce one sandbox per node.
+
+The example allows ten attempts, not unlimited retries. Inspect retries the task after increasing delays; successful samples are reused and failed samples restore their last usable checkpoint. A provider 405 is not one of Inspect's normal request-level retry statuses, so the task checkpoint retry supplies recovery. Persistent provider rejection, broken credentials, exhausted quotas and incompatible code still require intervention. A checkpoint failure becomes visible after the configured failure threshold.
+
+A dead image MCP connection also needs an attempt restart. Inspect reports a closed stdout reader or EOF as a tool error, which would otherwise leave the agent spending tokens on a permanently unusable connection. The ExploitBench tool wrapper promotes these specific transport failures to `GraderConnectionError`. Native CLI bridges are cancelled before that error is raised in the agent task, so they cannot swallow it and mark the agent complete. With the checkpoint/retry configuration above, the next attempt creates a fresh grader and restores previous work. Ordinary submission errors and ambiguous tool timeouts remain model feedback. This detects connection death; it does not fix the underlying cause of a repeatedly crashing grader.
+
+Monitor successful authoritative tool responses as well as model calls and checkpoint writes. A running container, a completed model request, or a trace entry saying a tool call exited is insufficient evidence of functional grading. The `mcp_recovery_probe` task kills the real image MCP process after a saved grade and requires successful grading again after restoration; it makes no paid inference calls.
+
+Inspect 0.3.265 fixes a ZIP decompressor compatibility issue exposed by the Hawk Python 3.13.15 runtime. The older version could write a successful final log but fail when rereading it, causing unnecessary retries. Real-image ReAct and native Codex tests on the fixed version each completed after one injected failure and one restored attempt. Check both the durable log and aggregate completion when validating a new runtime. If Hawk's package-age cutoff hides a required upstream fix, an exact official wheel URL with its verified SHA-256 can select that dependency without relaxing the cutoff for all packages.
+
+## What is saved
+
+Hawk writes Inspect state and declared sandbox paths to S3. This preserves conversation, grading history, cumulative token usage and native CLI session files. A restore creates a fresh sandbox from the pinned image, restores the files, and restarts the agent process. It does not restore RAM, background processes, open sockets, or arbitrary changes outside the listed paths. Keep durable work in those paths, or explicitly add the additional paths before launching. Capturing the whole image is much more expensive and has not been validated here.
+
+Time triggers fire at a safe checkpoint boundary. A ten-minute setting is not a guarantee that at most ten minutes of work can be lost: a long model request or tool call can delay the next checkpoint. The final `agent_complete` checkpoint allows scoring to be retried without executing the agent again.
+
+Claude Code and Codex can return voluntarily before the benchmark is complete. Their wrapper owns one checkpoint session across these continuations, rebinds bridge callbacks to the active CLI invocation, and writes completion only after the benchmark finishes. Inspect SWE 0.2.71 and Inspect 0.3.265 were checked: repeatedly opening their native checkpoint scope still registers duplicate bridge keys. A context-local adapter supplies the enclosing checkpoint without changing other samples. The `nudge` fault probe exercises a voluntary exit followed by a provider failure.
+
+The wrapper also recognizes the older premature `agent_complete` checkpoint when continuation was enabled and no benchmark exit reason was recorded. It restores the saved bridge conversation, adds the pending continuation, and resumes the native session. A separate completion marker preserves scorer-only retries after this migration. Normal completed samples must still be preserved by the evaluation-set selection policy.
+
+Checkpoints persist independently of live containers. `runner.cleanup: false` keeps resources for debugging indefinitely until deleted; it does not implement an hours-based retention policy and is unnecessary for durable recovery. Hawk documents persistent checkpoint storage, but the deployment's S3 lifecycle policy determines ultimate retention. Keeping only the `.eval` file is insufficient for restoring sandbox files; retain its associated checkpoint data too.
+
+## Resume and scorer repairs
+
+From an authenticated Hawk session:
+
+```text
+hawk eval-set resume EVAL_SET_ID
+hawk eval-set resume EVAL_SET_ID --config repaired-config.yaml
+```
+
+For a scorer-only repair, publish a compatible task commit, point the repaired config at it, preserve sample IDs/epochs/harness/checkpoint paths, and resume the failed evaluation set. With an `agent_complete` checkpoint, Inspect restores the completed agent state and runs the repaired scorer. This cannot retroactively create a checkpoint for an old run. Arbitrary edits to a solver's tracked state or native CLI version can make a checkpoint incompatible; validate those changes separately.
+
+The automated fault probe uses the real task image and grading tools with a scripted model. It writes random files, injects a provider error or runner crash, then requires a fresh container, identical file hashes, preserved grading/conversation, and unchanged cumulative token usage. Scorer-failure probes require a restart with `EXPLOITBENCH_PROBE_SCORER_FIXED=1` and no repeated model work.
+
+See [Hawk checkpointing](https://hawk.metr.org/user-guide/checkpointing/) and [live observation / ACP](https://hawk.metr.org/user-guide/babysitting-evals/). Observation and intervention are separate: steering a scored benchmark agent changes the measurement and should be restricted to designated tests.
+
+## Avoiding CLI release API rate limits
+
+Explicit Codex `0.154.0` and OpenCode `1.18.31` pins on Linux x64 prefetch the exact official release archive into Inspect SWE's native archive cache. The checked-in SHA-256 digest is verified on both cold and warm reads, and cache replacement is atomic. This avoids GitHub's rate-limited release-metadata API during installation while retaining the complete Codex package and OpenCode's baseline binary. Downloading the archive still requires its release CDN; transport and server errors use Inspect SWE's bounded download retries. Other versions and platforms retain the upstream installer. Verify and add a new digest when changing either pin; this does not silently substitute versions.
+
+The current Inspect SWE release was checked before adding this prefetch: it still resolves these pinned versions through the release API on an empty cache. The integration uses its existing archive-cache mechanism and is covered by real-image tests with release-API resolution deliberately disabled.
+
+The recommended launch profile targets a checkpoint every ten minutes (Hawk's default when checkpointing is enabled); fault tests save every turn to make the injected failures deterministic. Increase the interval to twenty minutes if representative measurements show material checkpoint overhead. Hawk currently exposes no rolling keep-last policy: retain the checkpoint data until a separate, coordinated pruning workflow is validated. Keep multiple recent checkpoints and the final pre-scoring checkpoint when defining that workflow; do not expire individual restic objects while newer snapshots still reference them.

pyproject.toml

@@ -10,7 +10,7 @@ where = ["src"]
 include = ["exploitbench*"]
 
 [tool.setuptools.package-data]
-"*" = ["compose.yaml", "eval.yaml", "run_configs/*.yaml"]
+"*" = ["compose.yaml", "k8s.yaml", "eval.yaml", "run_configs/*.yaml"]
 
 [tool.ruff]
 src = ["src"]
@@ -99,24 +99,27 @@ classifiers = [
 ]
 dependencies = [
     "anthropic",
+    "anyio",
     "google-genai",
-    "inspect_ai==0.3.263",
-    "inspect_swe==0.2.70",
+    "inspect_ai==0.3.265",
     "openai",
     "pyyaml>=5.1.0",
     "mcp>=1.0.0",
 ]
 
+[project.optional-dependencies]
+cli = [
+    "inspect-swe @ git+https://github.com/meridianlabs-ai/inspect_swe.git@9a6e92b614fc224b157d7a7bed8df175ea13f7d4",
+    "nodejs-wheel==24.19.0",
+]
+
 [project.urls]
 "Source Code" = "https://github.com/Generality-Labs/exploitbench-eval"
 "Issue Tracker" = "https://github.com/Generality-Labs/exploitbench-eval/issues"
 
 [project.entry-points.inspect_ai]
 exploitbench = "exploitbench"
 
-[project.scripts]
-exploitbench-report = "exploitbench.reporting:main"
-
 [dependency-groups]
 dev = [
     "mypy",

src/exploitbench/__init__.py

@@ -1,4 +1,11 @@
-from exploitbench.parity_agent import parity_agent
-from exploitbench.v8 import v8
+from exploitbench.checkpoint_probe import CheckpointProbe, checkpoint_probe
+from exploitbench.mcp_recovery_probe import MCPRecoveryProbe, mcp_recovery_probe
+from exploitbench.task import exploit_bench
 
-__all__ = ["parity_agent", "v8"]
+__all__ = [
+    "exploit_bench",
+    "checkpoint_probe",
+    "CheckpointProbe",
+    "mcp_recovery_probe",
+    "MCPRecoveryProbe",
+]