Skip to content

feat(benchmarks): add local tasks runner adapter - #160

Open
Cyberpradeep wants to merge 1 commit into
google-research:mainfrom
Cyberpradeep:feat/local-benchmark-runner
Open

Cyberpradeep wants to merge 1 commit into
google-research:mainfrom
Cyberpradeep:feat/local-benchmark-runner

Conversation

@Cyberpradeep

Copy link
Copy Markdown

Summary

This PR implements --bench local as proposed in #5, allowing Harness² to run recursive adaptation sweeps on an arbitrary local directory of instructions and source documents without requiring JobBench, LAB, or WorkBuddy installations.

Implementation Overview

  1. LocalAdapter (harness2/benchmarks/local.py):
    • Task Discovery: Discovers tasks in <tasks-dir>/<domain>/<task_id>/task.md.
    • Domain Mapping: Tasks are mapped to their <domain> parent folder, preserving per-domain experience library growth across user tasks.
    • Zero-Dependency Metadata: Parses optional task.toml via Python 3.12 standard library tomllib (outputs, check, timeout).
    • Workspace Isolation: Every rollout receives a clean, isolated workspace with a fresh copy of files/. Rollout edits never bleed into subsequent rollouts.
    • Substrate Launchers: Concrete launchers for OpenCode (run_argv) and Codex (exec).
    • Answer-Free Post-Selection Scoring: If check is configured, it executes strictly post-selection via harness2-judge. Tasks without a check return status="not_configured" and are reported as ungraded rather than judge failures.
    • Baseline Harness Generation: Automatically generates a minimal, valid c1..c8 starter tree or adopts a user-supplied --base-harness.
  2. CLI & Preflight Integration:
    • Integrated --bench local, --tasks-dir, and --base-harness across _common.py, run_sweep.py, judge.py, report.py, and check.py.
    • Decoupled harness2/preflight.py from academic benchmark and third-party runner dependencies.
  3. Core Seam Purity:
    • Zero changes inside harness2/core/. The implementation proves the benchmark seam in core/spec.py works as intended.

Dependency & Coordination Notes


Validation

  • python -m unittest -v tests.test_local_bench: All 7 unit/integration tests passed in 0.47s (OK).
  • Tested harness2 --bench local --tasks-dir <path> --dry-run: correctly discovers domains, tasks, and builds isolated namespaces.
  • Preflight and CLI option handling tested across run_sweep, judge, report, and check.
  • Documentation added in docs/LOCAL_BENCHMARK.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant