feat(metric-benchmark): execute local Harbor case and mutant loops

labs/metric-benchmark/README.md

@@ -21,8 +21,8 @@ Execution and score analysis comprise four executable stages, each exposed as a
 Python tool and library API:
 
 1. **Run mutants x eval tasks:** run the baseline and every mutant against the
-   benchmark tasks and save the results. The first execution script will be a
-   basic local Harbor loop; support for **NeMo Helix** will follow.
+   benchmark tasks and save the results. The [local execution script](docs/run-evals.md) uses a
+   basic Harbor loop; support for **NeMo Helix** will follow.
 2. **Compute mutant scores:** measure degradation across the mutants for each
    predefined task subset.
 3. **Compute coverage:** evaluate our metric on those same subsets.
@@ -33,3 +33,5 @@ for how the experiment works.
 
 For preparing bundles, use the [benchmark format specification](docs/benchmark-format.md).
 For execution outputs, use the [results format specification](docs/results-format.md).
+
+See [benchmark tools](docs/tools.md) for the command guides and configuration-loading API.

labs/metric-benchmark/docs/architecture.md

@@ -20,8 +20,8 @@ The experiment is a program with supplied inputs, a fixed sequence of operations
 and defined outputs. It executes benchmark tasks against the baseline agent and
 a fixed cohort of mutants. Its analysis consumes predefined task subsets, computes a
 mutant score for each subset, invokes the coverage tool on the same subsets, and
-correlates the two scores. The current implementation starts from saved execution
-results. Execution is a replaceable component whose output is a shared results
+correlates the two scores. The local Harbor implementation produces these execution results; analysis can
+also consume saved results. Execution is a replaceable component whose output is a shared results
 directory consumed by analysis.
 
 Comparisons between coverage tools use the same subsets, mutant cohort, and
@@ -99,7 +99,8 @@ Baseline and mutant bundles are prepared and checked before release.
 Tasks and their environments are ready for execution. At present only Harbor tasks are used.
 
 The configuration supplies the choices that determine execution: the fixed subset manifest,
-selected agent runs, scoring rules, and the coverage-tool command and settings.
+scoring rules, and the coverage-tool command and settings. Analysis derives one
+agent run per configured agent from the results index.
 The program validates these inputs before using them. Missing or inconsistent
 inputs cause an error.
 
@@ -133,8 +134,10 @@ are the **execution implementation** and the **results-directory interface**.
 | Execution implementation | Runs the configured agent-by-task matrix and assembles its outputs. The initial implementation is a simple local Harbor loop, one agent's full benchmark at a time. A future NeMo Helix implementation will orchestrate evaluations at scale. |
 | Results directory | Carries the task inventory, agent/task identities, statuses, measurements, and native evidence into analysis. Both execution implementations produce the same contract. |
 
-The proposed layout is agent-first: `agents/<agent>/<run>/<native-trial>/`, with
-`results.json` at the root indexing those trial results. Each run contains the
+The experiment root contains `execution.json`, mapping each case ID to its
+`results.json`. Within each case, the local layout is agent-first:
+`agents/<agent>/<run>/<native-trial>/`. The case index lists its task inventory
+and execution results as arrays and locates each trial’s native evidence. Each run contains the
 native output of that agent running the benchmark. The
 [results-directory contract](results-format.md) specifies the
 layout. Invocation, scheduling, and collection are internal to the execution
@@ -216,7 +219,7 @@ configuration and coordinates their execution.
 
 | Stage | Python tool / library operation |
 | --- | --- |
-| Execute evals | Local Harbor execution tool, or a future NeMo Helix execution tool; both expose the same results-directory contract |
+| Execute evals | `run_evals.py` / `metric_benchmark.execution.execute_benchmark`; a future NeMo Helix tool will produce the same results-directory contract |
 | Compute mutant scores | `compute_mutant_scores.py` / `metric_benchmark.scoring.compute_mutant_scores` |
 | Compute coverage | `compute_coverage.py` / `metric_benchmark.coverage.invoke_coverage` |
 | Correlation | `correlate.py` / `metric_benchmark.correlation.correlate` |

labs/metric-benchmark/docs/run-evals.md

@@ -0,0 +1,107 @@
+<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+
+# run_evals.py
+
+Run each prepared case’s baseline and mutants against its Harbor task inventory.
+
+## Usage
+
+```bash
+uv run python labs/metric-benchmark/run_evals.py \
+  --benchmark experiment.yml --run-id trial-1 --output private-run
+```
+
+Supply `--benchmark` for the benchmark YAML, `--run-id` for this execution's
+identifier, and `--output` for the experiment results directory. Run from the
+repository root after `uv sync --locked`; the default execution environment
+requires Docker. Prepared agents may have additional runtime or credential
+requirements defined by their benchmark bundle.
+
+The benchmark YAML is the command's only configuration file. Its referenced
+manifests supply the agents and tasks. All inputs require SHA pins by default;
+add `--allow-unsigned` while iterating to permit missing pins, while still
+verifying any supplied pins.
+
+## Execution and outputs
+
+The loop follows case order, then baseline and mutant order. Each job runs the
+whole task inventory once, with retries disabled and configured task concurrency.
+Outputs are indexed by case ID in `execution.json`:
+
+```text
+private-run/
+  execution-inputs.json
+  execution.json
+  cases/case-0000/execution-results/
+    results.json
+    agents/agent-0000/
+      request.json
+      native-request.json
+      receipt.json
+      run-0001/                 # Intact native Harbor job
+        result.json
+        config.json
+        lock.json
+        task-name__suffix/      # Native trial, indexed using metadata
+          result.json
+          agent/
+          verifier/
+```
+
+Numeric folder aliases are stable within a definition. The index carries actual
+case, agent, task and run identities. Each case index has `run_id` from
+`--run-id`, a `task_inventory` array, and an `execution_results` array. Each observation has an
+`agent_run_id` for its agent’s job; `native_results_path` locates the trial evidence.
+See the [results format](results-format.md) for the complete contract.
+No model or private service is involved in
+the orchestration itself. Prepared agents may call models when executed.
+
+## Reusing completed work
+
+Rerunning the command verifies completed job receipts and skips those jobs.
+Completed outputs and prepared inputs can be moved together. Reuse currently
+requires the entire benchmark configuration to match, including analysis settings
+and subset manifests, as well as the run ID and prepared inputs. Changing an
+analysis-only setting therefore also requires a fresh execution output directory.
+With `--allow-unsigned`, the runner hashes inputs without pins when computing this identity, so editing an
+unsigned bundle also requires a fresh output directory. This does not sign or
+rewrite the prepared manifests. The retained
+`native-request.json` records effective Harbor settings and exact original task
+paths for collecting saved trials; collection does not guess identities from
+path suffixes. Harbor version remains pinned. Native evidence is never rewritten.
+This supports collection/reuse of completed jobs, not migration of an active
+Harbor job. Older draft output formats require explicit conversion before reuse;
+the runner does not rewrite their requests or receipts.
+Changed evidence requires a new output directory. Do not run two writers against
+the same output directory.
+
+## Interrupted jobs
+
+If execution stopped between jobs, rerun the original command. If a native Harbor
+job is incomplete, complete that job first with Harbor's resume command, using
+its original input and output locations. For example:
+
+```bash
+uv run --locked harbor jobs resume --job-path \
+  private-run/cases/case-0000/execution-results/agents/agent-0001/run-0001
+```
+
+Replace the path with the interrupted job's directory, then rerun `run_evals.py`
+with the original arguments. Harbor reuses completed trials and reruns unfinished
+trials; its default resume behavior removes and reruns cancelled trials. Do not
+resume a completed job that this runner has already sealed with a receipt, since
+changing its evidence makes reuse fail.
+
+## Python library
+
+`execute_benchmark(loaded_benchmark, run_id, output)` runs the same stage as the
+CLI and returns the experiment manifest. Obtain `loaded_benchmark` with
+[`load_benchmark`](tools.md#reading-benchmarks). A future NeMo Helix implementation
+can produce the same [results contract](results-format.md).
+
+## Local smoke test
+
+The optional `METRIC_BENCHMARK_DOCKER_TEST=1` execution test runs nine local
+Oracle/NOP trials and checks CLI-to-library resume. It requires Docker and the
+`python:3.12-slim` image; it does not call a model or a benchmark service.

labs/metric-benchmark/docs/sha-sign.md

@@ -0,0 +1,74 @@
+<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+
+# sha-sign
+
+Sign or verify the prepared inputs referenced by a benchmark YAML file.
+
+Iterate with SHA fields omitted. When the prepared inputs are ready, sign the
+bundle from the repository root:
+
+```bash
+uv run labs/metric-benchmark/tools/sha-sign benchmark/experiment.yml
+```
+
+`sha-sign` reads the experiment and walks every case's referenced manifests and
+inputs. It checks paths, file types, schemas and subset task IDs before writing
+anything. It adds `sha256` to task sources, agent inputs, descriptions, subsets,
+and agent/tasks manifest references, working from input files up to the
+experiment. Shared manifests are updated once. It does not run agents or the
+metric tool, and it does not modify task files, agent files or JSON inputs.
+
+If any reference already has a SHA field, signing stops without writing files
+and lists those fields' locations. Delete those fields, or explicitly re-sign
+the current contents with `--resign`:
+
+```bash
+uv run labs/metric-benchmark/tools/sha-sign benchmark/experiment.yml --resign
+```
+
+`--resign` replaces old pins, including ones that no longer match edited inputs.
+Signing edits only SHA fields, preserving comments, key order, quoting and
+formatting elsewhere. New SHA fields are inserted into their reference mapping;
+existing pins keep their plain or quoted scalar style. Block-scalar, anchored or
+tagged SHA values must be changed to plain or quoted scalars before re-signing.
+The input files should not be edited while signing. Inputs cannot include
+manifests that the signer itself rewrites, since doing so would invalidate their
+hashes.
+
+Once signed, readers verify each supplied digest as they load its reference.
+`load_benchmark()` verifies all inputs; `read_benchmark()` verifies the manifests
+it reads, and individual artifact loaders verify their own inputs. Omitting a
+digest requires explicit `allow_unsigned=True` when loading during iteration.
+SHA signing fixes expected contents; it does not authenticate a publisher.
+
+To check that the entire benchmark is signed correctly without changing files:
+
+```bash
+uv run labs/metric-benchmark/tools/sha-sign benchmark/experiment.yml --check
+```
+
+`--check` requires a SHA on every input reference and verifies it against the
+current contents. It stops at the first validation error, including a missing,
+null or incorrect digest, and reports the input context. It uses the same strict
+`load_benchmark()` path as normal full loading. To fill missing pins throughout
+an unsigned bundle, run signing without `--check`; use `--resign` if some pins
+already exist. Success exits with status zero; validation failures exit nonzero.
+`--check` and `--resign` are mutually exclusive.
+The top-level experiment has no self-hash, and inline cases have no separate
+hash: their agent, tasks and subsets references are checked instead.
+
+To sign or check from Python:
+
+```python
+from pathlib import Path
+from metric_benchmark.signing import check_benchmark, sign_benchmark
+
+experiment = Path("benchmark/experiment.yml")
+changed_files = sign_benchmark(experiment, resign=False)
+check_benchmark(experiment)
+```
+
+`sign_benchmark()` returns the paths of YAML files it changed.
+`check_benchmark()` returns normally on success and raises on missing or invalid
+pins or inputs. Re-signing does not require existing pins to match.

labs/metric-benchmark/docs/stages.md

@@ -15,7 +15,7 @@ fixed subset manifest.
 
 | Stage | Tool |
 | --- | --- |
-| 1. Run mutants x eval tasks | Planned local Harbor loop; future NeMo Helix implementation |
+| 1. Run mutants x eval tasks | `run_evals.py` (local Harbor); future NeMo Helix implementation |
 | 2. Compute mutant scores | `compute_mutant_scores.py` |
 | 3. Compute coverage | `compute_coverage.py` |
 | 4. Correlation | `correlate.py` |
@@ -39,7 +39,7 @@ mutants and produce the results directory consumed by analysis.
 **Inputs:** prepared agent bundles, benchmark tasks, and experiment execution
 settings.
 
-**Implementations:** the first execution script will run Harbor tasks in a
+**Implementations:** the execution script runs Harbor tasks in a
 basic local loop: baseline first, then the configured mutants, with one Harbor job per agent
 covering the full benchmark. Agent jobs run sequentially. Task concurrency within
 a Harbor job is an explicit execution setting; a fully sequential run uses one
@@ -49,8 +49,8 @@ the selected run; repetition aggregation remains future work.
 This phase is expected to be expensive: with `M` mutants and `T` tasks, even a
 single execution per pair requires `(M + 1) × T` task executions before retries.
 The baseline is included because mutant scoring compares its outcomes with each
-mutant's outcomes. The repository currently supplies the analysis tools; the
-local Harbor loop is the first execution implementation to add here.
+mutant's outcomes. The local Harbor loop produces the shared index and native evidence used by
+the analysis tools.
 
 A future NeMo Helix implementation will use an orchestrator to run evaluations at
 scale. Each implementation owns invocation, scheduling, retries, and collection