Skip to content

Demonstrate task-attached default run configuration - #2

Draft
MattFisher wants to merge 1 commit into
codex/exploitbench-review-fixesfrom
codex/exploitbench-task-default-config
Draft

MattFisher wants to merge 1 commit into
codex/exploitbench-review-fixesfrom
codex/exploitbench-task-default-config

Conversation

@MattFisher

Copy link
Copy Markdown

Purpose

Demonstrates what the task-attached default configuration from Inspect AI PR 5294 enables for ExploitBench.

The normal invocation becomes:

inspect eval exploitbench/exploit_bench --model provider/model

Inspect discovers and applies the maintained YAML automatically, while an explicit --run-config still replaces it.

What this changes

  • attaches run_configs/default.yaml with @task(default_config=...)
  • makes task parameters config-supplied instead of loading YAML into Python defaults
  • moves generation, limits, epoch reduction, and error-scoring policy into the YAML profiles
  • deletes the production run_config.py loader
  • retains package-resource loading only for eval.yaml and k8s.yaml metadata
  • updates CLI, Modal, Python API, provenance, and override coverage
  • pins the reviewed head of Inspect AI PR 5294 until the feature is released

Why draft

This is a demonstration stacked on codex/exploitbench-review-fixes and depends on the unreleased Inspect AI feature. It is intended to make the resulting API and code shape easy to review before choosing whether to adopt it.

Validation

  • 300 passed, 23 skipped
  • Ruff formatting and lint pass
  • mypy passes
  • uv lock check passes

@pabloRom2004

Copy link
Copy Markdown

Will we have to wait here for the unreleased feature to Inspect AI to be merged before we can merge this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants