Demonstrate task-attached default run configuration

README.md

@@ -45,23 +45,23 @@ model API requests, and `.eval` logs local while hosting the challenge image.
 
 Replace `provider/model` with your Inspect model identifier. Run the ReAct: **(Recommended way to run the eval)**
 
-Review the settings in [default.yaml](src/exploitbench/run_configs/default.yaml) and adjust them as needed before running the evaluation.
+The task automatically applies the maintained settings in [default.yaml](src/exploitbench/run_configs/default.yaml). Review them before running the evaluation.
 
 ```bash
 uv run inspect eval \
-  --run-config src/exploitbench/run_configs/default.yaml \
+  exploitbench/exploit_bench \
   --model provider/model \
   -T vulnerability_ids=cve-2024-1939 \
   --log-dir logs
 ```
 
 Or a provider CLI (claude_code, codex_cli, gemini_cli, kimi_code, opencode):
 
-Use the same config and change `task.args.agent`:
+Override the attached config's `task.args.agent`:
 
 ```bash
 uv run --extra cli inspect eval \
-  --run-config src/exploitbench/run_configs/default.yaml \
+  exploitbench/exploit_bench \
   --model provider/model \
   -T vulnerability_ids=cve-2024-1939 \
   -T agent=inspect_swe/codex_cli \
@@ -97,7 +97,7 @@ Docker remains the default; Kubernetes execution requires the host's
 
 ## Options
 
-Edit or copy one of the config files linked below and pass its path to `--run-config`. Override task arguments with `-T` and generation/evaluation settings with CLI flags, e.g. `--token-limit 1000000000 --epochs 1`. Use `uv run inspect eval --help` for all options.
+The maintained config is attached to `exploitbench/exploit_bench` and applies automatically. Copy a config and pass it to `--run-config` to replace the attached settings, or pass `--no-default-config` to opt out. Override task arguments with `-T` and generation/evaluation settings with CLI flags, e.g. `--token-limit 1000000000 --epochs 1`. Use `uv run inspect eval --help` for all options.
 
 To change benchmark prompt wording, edit the named `Prompt` objects in [prompts.py](src/exploitbench/prompts.py).
 
@@ -176,9 +176,11 @@ The original config selects `exploitbench/original_agent`, with `agent_args.tool
 
 ### Native Inspect configuration
 
-Pass either YAML file directly to `inspect eval --run-config <path>`. Standard Inspect flags override YAML, including `--solver`, `--token-limit`, `--max-tokens`, and `-T` task arguments.
+Running `inspect eval exploitbench/exploit_bench` applies `default.yaml` through Inspect's task-default configuration support. Pass either YAML file to `inspect eval --run-config <path>` to replace the attached config. Standard Inspect flags override YAML, including `--solver`, `--token-limit`, `--max-tokens`, and `-T` task arguments.
 
-Token and time limits belong to Inspect's evaluation settings. Supply the run config or explicit limits when launching; a bare `exploit_bench()` task has no token or time cap. The task reads its agent and generation defaults directly from `default.yaml`. Replacement solvers remain responsible for attaching the wrapped benchmark tools and their own continuation policy.
+For the Python API, use Inspect's public `read_run_config(path).to_params()` and pass `default_config=False` to `eval()` when an explicit file should replace the attached default.
+
+Token and time limits belong to Inspect's evaluation settings. Inspect applies them when it resolves the registered task or task callable. A direct `exploit_bench(...)` call or preconstructed `Task` intentionally bypasses the attached config; direct callers must supply every task argument explicitly and receive no attached generation, token, or time settings. Replacement solvers remain responsible for attaching the wrapped benchmark tools and their own continuation policy.
 
 ## Code layout
 
@@ -190,7 +192,6 @@ Token and time limits belong to Inspect's evaluation settings. Supply the run co
 | [cli.py](src/exploitbench/cli.py) | Claude Code, Codex, Gemini, Kimi, and OpenCode adapters, context settings, and request filtering. |
 | [tools.py](src/exploitbench/tools.py) | Shared MCP connection, tool wrappers, and optional submission tool. |
 | [grading.py](src/exploitbench/grading.py) / [scorers.py](src/exploitbench/scorers.py) | Validated grading history, final capability scores, and epoch reduction. |
-| [run_config.py](src/exploitbench/run_config.py) | Read native YAML settings for task and agent defaults. |
 
 ## Dataset
 

_test_config.py

@@ -0,0 +1,27 @@
+from pathlib import Path
+from typing import Any
+
+import yaml
+from inspect_ai import read_run_config
+
+ROOT = Path(__file__).parent
+RUN_CONFIGS = ROOT / "src/exploitbench/run_configs"
+
+
+def load_config(path: str | Path = RUN_CONFIGS / "default.yaml") -> dict[str, Any]:
+    """Read a YAML mapping for tests that modify configuration fixtures."""
+    source = Path(path)
+    if not source.is_absolute():
+        source = ROOT / "src/exploitbench" / source
+    config = yaml.safe_load(source.read_text())
+    if not isinstance(config, dict):
+        raise ValueError("Configuration must be a YAML mapping")
+    return config
+
+
+def load_eval_params(path: str | Path) -> dict[str, Any]:
+    """Prepare an explicit replacement run config for Inspect's Python API."""
+    return {
+        **read_run_config(str(path)).to_params(),
+        "default_config": False,
+    }

docs/modal-root-issue-handover.md

@@ -66,7 +66,7 @@ The failing evaluation used:
 export EXPLOITBENCH_ACKNOWLEDGE_RISKS=1
 
 uv run --python 3.12 --extra modal inspect eval \
-  --run-config src/exploitbench/run_configs/default.yaml \
+  exploitbench/exploit_bench \
   --sandbox exploitbench_modal \
   --model anthropic/claude-5-sonnet \
   -T vulnerability_ids=cve-2024-10231 \

docs/modal.md

@@ -70,7 +70,7 @@ Then run one sample while validating the integration:
 export EXPLOITBENCH_ACKNOWLEDGE_RISKS=1
 
 uv run --python 3.12 --extra modal inspect eval \
-  --run-config src/exploitbench/run_configs/default.yaml \
+  exploitbench/exploit_bench \
   --sandbox exploitbench_modal \
   --model provider/model \
   -T vulnerability_ids=cve-2024-10231 \

pyproject.toml

@@ -52,6 +52,7 @@ ban-relative-imports = "all"
 [tool.pytest.ini_options]
 minversion = "7.0"
 addopts = "--import-mode=importlib"
+pythonpath = ["."]
 testpaths = ["tests"]
 asyncio_mode = "auto"
 log_level = "WARNING"
@@ -105,7 +106,9 @@ dependencies = [
     "anthropic",
     "anyio",
     "google-genai",
-    "inspect_ai==0.3.263",
+    # PR 5294 adds task-attached default run configurations. Pin its reviewed
+    # head until the feature is available in a published Inspect release.
+    "inspect_ai @ git+https://github.com/UKGovernmentBEIS/inspect_ai.git@105a3ed037ae546b6835d6ab78c8d7876bf9ae90",
     "openai",
     "pyyaml>=5.1.0",
     "mcp>=1.0.0",

src/exploitbench/cli.py

@@ -33,10 +33,8 @@
     GRADE_REMINDER,
 )
 from exploitbench.reminders import cli_reminders, nudge
-from exploitbench.run_config import load_config
 from exploitbench.tools import benchmark_server, recorded_tool, submit_tool
 
-DEFAULT_AGENT_ARGS = load_config()["task"]["args"]
 CLI_HARNESSES = ("claude_code", "codex_cli", "gemini_cli", "kimi_code", "opencode")
 OPENCODE_CONTEXT_FRACTION = 2
 OPENCODE_MIN_RESERVED_TOKENS = 8_000
@@ -50,16 +48,14 @@
 def cli_agent(
     harness: str,
     harness_args: dict[str, Any] | None = None,
-    submit: bool = DEFAULT_AGENT_ARGS["submit"],
-    nudge_prompt: bool = DEFAULT_AGENT_ARGS["nudge_prompt"],
-    token_budget_reminder: bool = DEFAULT_AGENT_ARGS["token_budget_reminder"],
-    grade_submit_reminder: bool = DEFAULT_AGENT_ARGS["grade_submit_reminder"],
-    grade_submit_reminder_interval: int = DEFAULT_AGENT_ARGS[
-        "grade_submit_reminder_interval"
-    ],
-    grade_timeout: int | None = DEFAULT_AGENT_ARGS["grade_timeout"],
-    context_window: int | None = DEFAULT_AGENT_ARGS["context_window"],
-    time_limit_reminder: bool = DEFAULT_AGENT_ARGS["time_limit_reminder"],
+    submit: bool = False,
+    nudge_prompt: bool = True,
+    token_budget_reminder: bool = True,
+    grade_submit_reminder: bool = True,
+    grade_submit_reminder_interval: int = 10,
+    grade_timeout: int | None = 7200,
+    context_window: int | None = None,
+    time_limit_reminder: bool = False,
 ) -> Agent:
     """Resume an off-the-shelf Inspect SWE CLI with the same benchmark tools and reminders."""
     if harness not in CLI_HARNESSES:

src/exploitbench/dataset.py

@@ -1,12 +1,13 @@
 from collections.abc import Mapping
 from dataclasses import dataclass
+from importlib.resources import files
 from typing import cast
 
+import yaml
 from inspect_ai.dataset import Sample
 from inspect_ai.util import ComposeConfig, ComposeService, SandboxEnvironmentSpec
 
 from exploitbench.prompts import ORIGINAL
-from exploitbench.run_config import load_config
 
 
 @dataclass(frozen=True)
@@ -20,7 +21,7 @@ class V8Environment:
 
 def get_v8_environments() -> tuple[V8Environment, ...]:
     """Read the ordered V8 challenge manifest from eval.yaml."""
-    assets = load_config("eval.yaml").get("external_assets")
+    assets = get_eval_manifest().get("external_assets")
     if not isinstance(assets, list):
         raise ValueError("eval.yaml external_assets must be a list")
 
@@ -39,6 +40,14 @@ def get_v8_environments() -> tuple[V8Environment, ...]:
     return environments
 
 
+def get_eval_manifest() -> dict[str, object]:
+    """Read the package's evaluation metadata manifest."""
+    manifest = yaml.safe_load(files("exploitbench").joinpath("eval.yaml").read_text())
+    if not isinstance(manifest, dict):
+        raise ValueError("eval.yaml must contain a mapping")
+    return cast(dict[str, object], manifest)
+
+
 def get_v8_environment_by_id() -> dict[str, V8Environment]:
     """Index the eval.yaml V8 challenge manifest by vulnerability ID."""
     return {

src/exploitbench/harness_default.py

@@ -9,11 +9,8 @@
 from exploitbench.cli import cli_agent as cli_agent
 from exploitbench.prompts import GRADE_REMINDER
 from exploitbench.reminders import react_continuation
-from exploitbench.run_config import load_config
 from exploitbench.tools import benchmark_tools as benchmark_tools
 
-DEFAULT_AGENT_ARGS = load_config()["task"]["args"]
-
 
 def configured_agent(
     name: str,
@@ -54,23 +51,17 @@ def configured_agent(
 
 
 def react_agent(
-    tools: Sequence[str | Tool | ToolDef | ToolSource] | None = DEFAULT_AGENT_ARGS[
-        "react"
-    ]["tools"],
-    compaction_threshold: float | None = DEFAULT_AGENT_ARGS["react"]["compaction"][
-        "threshold"
-    ],
-    token_budget_reminder: bool = DEFAULT_AGENT_ARGS["token_budget_reminder"],
-    grade_submit_reminder: bool = DEFAULT_AGENT_ARGS["grade_submit_reminder"],
-    grade_submit_reminder_interval: int = DEFAULT_AGENT_ARGS[
-        "grade_submit_reminder_interval"
-    ],
-    grade_timeout: int | None = DEFAULT_AGENT_ARGS["grade_timeout"],
-    submit: bool = DEFAULT_AGENT_ARGS["submit"],
-    nudge_prompt: bool = DEFAULT_AGENT_ARGS["nudge_prompt"],
-    context_window: int | None = DEFAULT_AGENT_ARGS["context_window"],
-    time_limit_reminder: bool = DEFAULT_AGENT_ARGS["time_limit_reminder"],
-    tool_timeout: int | None = DEFAULT_AGENT_ARGS["react"].get("tool_timeout"),
+    tools: Sequence[str | Tool | ToolDef | ToolSource] | None = ("bash", "python"),
+    compaction_threshold: float | None = 0.75,
+    token_budget_reminder: bool = True,
+    grade_submit_reminder: bool = True,
+    grade_submit_reminder_interval: int = 10,
+    grade_timeout: int | None = 7200,
+    submit: bool = False,
+    nudge_prompt: bool = True,
+    context_window: int | None = None,
+    time_limit_reminder: bool = False,
+    tool_timeout: int | None = 7200,
     **options: Any,
 ) -> Agent:
     """Return Inspect's native ReAct agent with benchmark tools and its public continuation callback."""

src/exploitbench/harness_original.py

@@ -13,13 +13,8 @@
     render_nudge_prompt,
     time_reminder,
 )
-from exploitbench.run_config import load_config
 from exploitbench.tools import benchmark_tools
 
-DEFAULT_AGENT_ARGS = load_config("run_configs/original.yaml")["task"]["args"][
-    "agent_args"
-]
-
 _OVERFLOW_MARKERS = (
     "prompt is too long",
     "input is too long",
@@ -46,20 +41,12 @@ def _model_matches(requested: str, served: str) -> bool:
 
 @agent
 def original_agent(
-    turn_budget: int = DEFAULT_AGENT_ARGS["turn_budget"],
-    nudge_prompt: bool = load_config("run_configs/original.yaml")["task"]["args"][
-        "nudge_prompt"
-    ],
-    tools: Sequence[str | Tool | ToolDef | ToolSource] | None = DEFAULT_AGENT_ARGS[
-        "tools"
-    ],
-    time_limit_reminder: bool = load_config("run_configs/original.yaml")["task"][
-        "args"
-    ]["time_limit_reminder"],
-    tool_timeout: int | None = DEFAULT_AGENT_ARGS.get("tool_timeout"),
-    grade_timeout: int | None = load_config("run_configs/original.yaml")["task"][
-        "args"
-    ]["grade_timeout"],
+    turn_budget: int = 300,
+    nudge_prompt: bool = True,
+    tools: Sequence[str | Tool | ToolDef | ToolSource] | None = None,
+    time_limit_reminder: bool = False,
+    tool_timeout: int | None = 7200,
+    grade_timeout: int | None = 7200,
 ) -> Agent:
     """Reproduce the upstream host-side conversation loop using the image's MCP tools."""
     if tool_timeout is not None and tool_timeout < 1:

src/exploitbench/run_configs/default.yaml

@@ -3,6 +3,9 @@ eval_config:
   time_limit: null
   limit: null
   epochs: 1
+  epochs_reducer: [exploitbench/capability_union]
+  fail_on_error: false
+  score_on_error: true
 
 generate_config:
   reasoning_effort: xhigh

src/exploitbench/run_configs/original.yaml

@@ -3,6 +3,9 @@ eval_config:
   time_limit: 18000
   limit: null
   epochs: 3
+  epochs_reducer: [exploitbench/capability_union]
+  fail_on_error: false
+  score_on_error: true
 
 generate_config:
   reasoning_effort: xhigh

src/exploitbench/sandbox.py

@@ -1,13 +1,12 @@
 import hashlib
 from functools import cache
+from importlib.resources import files
 from pathlib import Path
 from tempfile import TemporaryDirectory
 
 import yaml
 from inspect_ai.util import SandboxEnvironmentSpec
 
-from exploitbench.run_config import load_config
-
 
 @cache
 def _sandbox_config_directory() -> TemporaryDirectory[str]:
@@ -18,7 +17,9 @@ def _sandbox_config_directory() -> TemporaryDirectory[str]:
 @cache
 def kubernetes_sandbox(image: str) -> SandboxEnvironmentSpec:
     """Create native Kubernetes values for one digest-pinned challenge image."""
-    values = load_config("k8s.yaml")
+    values = yaml.safe_load(files("exploitbench").joinpath("k8s.yaml").read_text())
+    if not isinstance(values, dict):
+        raise ValueError("k8s.yaml must contain a mapping")
     values["services"]["default"]["image"] = image
     filename = hashlib.sha256(image.encode()).hexdigest() + ".yaml"
     path = Path(_sandbox_config_directory().name) / filename