[mstaats-nvidia]

Run each prepared case through a local Harbor loop: baseline first, then each mutant, with one full task inventory per agent, explicit task concurrency, and no automatic retries. run_evals.py owns the CLI and calls the execution library. execute_benchmark prepares all jobs before execution, iterates cases and agents, and publishes results; run_or_reuse_job handles one agent’s execution, receipt verification, and result collection.

Produce the results contract from #105: a flat task inventory, execution observations, and an experiment manifest mapping case IDs to results indexes. Preserve native Harbor evidence and exact task bindings. Completed jobs can be verified and reused after relocation; incomplete jobs require explicit Harbor recovery. Signed inputs are required by default, while --allow-unsigned permits iteration and includes unpinned contents in the reuse identity. Changed inputs require a fresh output directory. Reuse currently binds the entire benchmark configuration, including analysis settings.

Provide a dedicated Markdown usage guide for each shipped command and link from the shared tools page. Document execution prerequisites, output layout, reuse, and interruption recovery. Public execution functions have concise docstrings and explicit case/agent variable names.

Validation on final head 792b8f9: all 21 execution tests passed, including the real Docker smoke (nine Oracle/NOP trials), CLI-to-library reuse, and relocation. Python 3.12/3.13 CI, native compatibility, static, licensing, and other applicable checks passed; optional workflows were skipped. The earlier local full suite passed 2,616 tests with 26 skips. No live model authoring was invoked.

After this PR merges, the remaining draft stack is main → #95 mutant scores → #96 coverage → #97 correlation → #99 orchestration.

[asutermo]

nit: worth adding an eventual call out to Helix for this part? We can also leave as is for now

[mstaats-nvidia]

Let's avoid until it's in Helix, but yea 100 we wanna add that!

[mstaats-nvidia]

Recommended file review order for the current head (792b8f9):

1. labs/metric-benchmark/docs/run-evals.md — Start with the execution command, inputs, output layout, reuse rules, and interruption recovery.
2. labs/metric-benchmark/metric_benchmark/execution.py — Read execute_benchmark for the outer flow, then run_or_reuse_job for one agent; follow job_configuration, run_harbor, collect_job, and benchmark_identity as needed.
3. labs/metric-benchmark/run_evals.py — Review main(), argument parsing, and CLI error handling; the entry point calls the execution library.
4. tests/test_metric_benchmark_execution.py — Check the two-case matrix and resume, relocation, partial-job handling, and changed unsigned inputs.
5. labs/metric-benchmark/docs/stages.md — Check execution-stage responsibilities.
6. labs/metric-benchmark/docs/architecture.md — Check the execution and results-contract description.
7. labs/metric-benchmark/docs/assets/experiment-flow.svg — Check the local Harbor and future Helix labels.
8. labs/metric-benchmark/docs/validate-results.md — Review the validator usage and library instructions moved from the shared pages.
9. labs/metric-benchmark/docs/validation-report.md — Check the link to the command guide; report-format documentation stays here.
10. labs/metric-benchmark/docs/sha-sign.md — Review the signing instructions moved into their own command guide.
11. labs/metric-benchmark/docs/tools.md — Check the command-guide links and shared configuration-loading instructions.
12. labs/metric-benchmark/README.md — Check the entry links and description of the implemented local runner.