diff --git a/labs/metric-benchmark/README.md b/labs/metric-benchmark/README.md index b8ba8310..a0041b87 100644 --- a/labs/metric-benchmark/README.md +++ b/labs/metric-benchmark/README.md @@ -21,8 +21,8 @@ Execution and score analysis comprise four executable stages, each exposed as a Python tool and library API: 1. **Run mutants x eval tasks:** run the baseline and every mutant against the - benchmark tasks and save the results. The first execution script will be a - basic local Harbor loop; support for **NeMo Helix** will follow. + benchmark tasks and save the results. The [local execution script](docs/run-evals.md) uses a + basic Harbor loop; support for **NeMo Helix** will follow. 2. **Compute mutant scores:** measure degradation across the mutants for each predefined task subset. 3. **Compute coverage:** evaluate our metric on those same subsets. @@ -33,3 +33,5 @@ for how the experiment works. For preparing bundles, use the [benchmark format specification](docs/benchmark-format.md). For execution outputs, use the [results format specification](docs/results-format.md). + +See [benchmark tools](docs/tools.md) for the command guides and configuration-loading API. diff --git a/labs/metric-benchmark/docs/architecture.md b/labs/metric-benchmark/docs/architecture.md index 55fb4997..3501553a 100644 --- a/labs/metric-benchmark/docs/architecture.md +++ b/labs/metric-benchmark/docs/architecture.md @@ -20,8 +20,8 @@ The experiment is a program with supplied inputs, a fixed sequence of operations and defined outputs. It executes benchmark tasks against the baseline agent and a fixed cohort of mutants. Its analysis consumes predefined task subsets, computes a mutant score for each subset, invokes the coverage tool on the same subsets, and -correlates the two scores. The current implementation starts from saved execution -results. Execution is a replaceable component whose output is a shared results +correlates the two scores. The local Harbor implementation produces these execution results; analysis can +also consume saved results. Execution is a replaceable component whose output is a shared results directory consumed by analysis. Comparisons between coverage tools use the same subsets, mutant cohort, and @@ -99,7 +99,8 @@ Baseline and mutant bundles are prepared and checked before release. Tasks and their environments are ready for execution. At present only Harbor tasks are used. The configuration supplies the choices that determine execution: the fixed subset manifest, -selected agent runs, scoring rules, and the coverage-tool command and settings. +scoring rules, and the coverage-tool command and settings. Analysis derives one +agent run per configured agent from the results index. The program validates these inputs before using them. Missing or inconsistent inputs cause an error. @@ -133,8 +134,10 @@ are the **execution implementation** and the **results-directory interface**. | Execution implementation | Runs the configured agent-by-task matrix and assembles its outputs. The initial implementation is a simple local Harbor loop, one agent's full benchmark at a time. A future NeMo Helix implementation will orchestrate evaluations at scale. | | Results directory | Carries the task inventory, agent/task identities, statuses, measurements, and native evidence into analysis. Both execution implementations produce the same contract. | -The proposed layout is agent-first: `agents////`, with -`results.json` at the root indexing those trial results. Each run contains the +The experiment root contains `execution.json`, mapping each case ID to its +`results.json`. Within each case, the local layout is agent-first: +`agents////`. The case index lists its task inventory +and execution results as arrays and locates each trial’s native evidence. Each run contains the native output of that agent running the benchmark. The [results-directory contract](results-format.md) specifies the layout. Invocation, scheduling, and collection are internal to the execution @@ -216,7 +219,7 @@ configuration and coordinates their execution. | Stage | Python tool / library operation | | --- | --- | -| Execute evals | Local Harbor execution tool, or a future NeMo Helix execution tool; both expose the same results-directory contract | +| Execute evals | `run_evals.py` / `metric_benchmark.execution.execute_benchmark`; a future NeMo Helix tool will produce the same results-directory contract | | Compute mutant scores | `compute_mutant_scores.py` / `metric_benchmark.scoring.compute_mutant_scores` | | Compute coverage | `compute_coverage.py` / `metric_benchmark.coverage.invoke_coverage` | | Correlation | `correlate.py` / `metric_benchmark.correlation.correlate` | diff --git a/labs/metric-benchmark/docs/assets/experiment-flow.svg b/labs/metric-benchmark/docs/assets/experiment-flow.svg index c85eecb1..f6a4f1ba 100644 --- a/labs/metric-benchmark/docs/assets/experiment-flow.svg +++ b/labs/metric-benchmark/docs/assets/experiment-flow.svg @@ -3,7 +3,7 @@ Metric benchmark: from case examples to correlation - Three phases: prepare case examples for the experiment; run the baseline and mutants against all eval tasks; analyze the saved results. Prep supplies the agent, mutants, case description, benchmark tasks, and experiment configuration. It constructs subsets using a fixed seed and bundles their exact memberships in an authoritative task-inventory-bound manifest. Execution produces a shared results directory. Analysis consumes the prepared subset manifest, computes mutant scores from baseline and mutant outcomes, and computes coverage from the case description and baseline evidence using metric computation such as Eval Author. The two scores are joined by subset for correlation. Configuration governs execution and analysis. Prep is outside the experiment; the local Harbor loop and future NeMo Helix implementation are planned execution backends. + Three phases: prepare case examples for the experiment; run the baseline and mutants against all eval tasks; analyze the saved results. Prep supplies the agent, mutants, case description, benchmark tasks, and experiment configuration. It constructs subsets using a fixed seed and bundles their exact memberships in an authoritative task-inventory-bound manifest. Execution produces a shared results directory. Analysis consumes the prepared subset manifest, computes mutant scores from baseline and mutant outcomes, and computes coverage from the case description and baseline evidence using metric computation such as Eval Author. The two scores are joined by subset for correlation. Configuration governs execution and analysis. Prep is outside the experiment; execution uses the local Harbor loop; NeMo Helix is a future backend. @@ -83,7 +83,7 @@ Execute evals Mutants + baseline x eval tasks One full benchmark per agent; local jobs run in sequence. - Planned Python tools: local Harbor, then NeMo Helix. + Local Harbor tool now; NeMo Helix in the future. Results directory diff --git a/labs/metric-benchmark/docs/run-evals.md b/labs/metric-benchmark/docs/run-evals.md new file mode 100644 index 00000000..65dac2f6 --- /dev/null +++ b/labs/metric-benchmark/docs/run-evals.md @@ -0,0 +1,107 @@ + + + +# run_evals.py + +Run each prepared case’s baseline and mutants against its Harbor task inventory. + +## Usage + +```bash +uv run python labs/metric-benchmark/run_evals.py \ + --benchmark experiment.yml --run-id trial-1 --output private-run +``` + +Supply `--benchmark` for the benchmark YAML, `--run-id` for this execution's +identifier, and `--output` for the experiment results directory. Run from the +repository root after `uv sync --locked`; the default execution environment +requires Docker. Prepared agents may have additional runtime or credential +requirements defined by their benchmark bundle. + +The benchmark YAML is the command's only configuration file. Its referenced +manifests supply the agents and tasks. All inputs require SHA pins by default; +add `--allow-unsigned` while iterating to permit missing pins, while still +verifying any supplied pins. + +## Execution and outputs + +The loop follows case order, then baseline and mutant order. Each job runs the +whole task inventory once, with retries disabled and configured task concurrency. +Outputs are indexed by case ID in `execution.json`: + +```text +private-run/ + execution-inputs.json + execution.json + cases/case-0000/execution-results/ + results.json + agents/agent-0000/ + request.json + native-request.json + receipt.json + run-0001/ # Intact native Harbor job + result.json + config.json + lock.json + task-name__suffix/ # Native trial, indexed using metadata + result.json + agent/ + verifier/ +``` + +Numeric folder aliases are stable within a definition. The index carries actual +case, agent, task and run identities. Each case index has `run_id` from +`--run-id`, a `task_inventory` array, and an `execution_results` array. Each observation has an +`agent_run_id` for its agent’s job; `native_results_path` locates the trial evidence. +See the [results format](results-format.md) for the complete contract. +No model or private service is involved in +the orchestration itself. Prepared agents may call models when executed. + +## Reusing completed work + +Rerunning the command verifies completed job receipts and skips those jobs. +Completed outputs and prepared inputs can be moved together. Reuse currently +requires the entire benchmark configuration to match, including analysis settings +and subset manifests, as well as the run ID and prepared inputs. Changing an +analysis-only setting therefore also requires a fresh execution output directory. +With `--allow-unsigned`, the runner hashes inputs without pins when computing this identity, so editing an +unsigned bundle also requires a fresh output directory. This does not sign or +rewrite the prepared manifests. The retained +`native-request.json` records effective Harbor settings and exact original task +paths for collecting saved trials; collection does not guess identities from +path suffixes. Harbor version remains pinned. Native evidence is never rewritten. +This supports collection/reuse of completed jobs, not migration of an active +Harbor job. Older draft output formats require explicit conversion before reuse; +the runner does not rewrite their requests or receipts. +Changed evidence requires a new output directory. Do not run two writers against +the same output directory. + +## Interrupted jobs + +If execution stopped between jobs, rerun the original command. If a native Harbor +job is incomplete, complete that job first with Harbor's resume command, using +its original input and output locations. For example: + +```bash +uv run --locked harbor jobs resume --job-path \ + private-run/cases/case-0000/execution-results/agents/agent-0001/run-0001 +``` + +Replace the path with the interrupted job's directory, then rerun `run_evals.py` +with the original arguments. Harbor reuses completed trials and reruns unfinished +trials; its default resume behavior removes and reruns cancelled trials. Do not +resume a completed job that this runner has already sealed with a receipt, since +changing its evidence makes reuse fail. + +## Python library + +`execute_benchmark(loaded_benchmark, run_id, output)` runs the same stage as the +CLI and returns the experiment manifest. Obtain `loaded_benchmark` with +[`load_benchmark`](tools.md#reading-benchmarks). A future NeMo Helix implementation +can produce the same [results contract](results-format.md). + +## Local smoke test + +The optional `METRIC_BENCHMARK_DOCKER_TEST=1` execution test runs nine local +Oracle/NOP trials and checks CLI-to-library resume. It requires Docker and the +`python:3.12-slim` image; it does not call a model or a benchmark service. diff --git a/labs/metric-benchmark/docs/sha-sign.md b/labs/metric-benchmark/docs/sha-sign.md new file mode 100644 index 00000000..e795a1fa --- /dev/null +++ b/labs/metric-benchmark/docs/sha-sign.md @@ -0,0 +1,74 @@ + + + +# sha-sign + +Sign or verify the prepared inputs referenced by a benchmark YAML file. + +Iterate with SHA fields omitted. When the prepared inputs are ready, sign the +bundle from the repository root: + +```bash +uv run labs/metric-benchmark/tools/sha-sign benchmark/experiment.yml +``` + +`sha-sign` reads the experiment and walks every case's referenced manifests and +inputs. It checks paths, file types, schemas and subset task IDs before writing +anything. It adds `sha256` to task sources, agent inputs, descriptions, subsets, +and agent/tasks manifest references, working from input files up to the +experiment. Shared manifests are updated once. It does not run agents or the +metric tool, and it does not modify task files, agent files or JSON inputs. + +If any reference already has a SHA field, signing stops without writing files +and lists those fields' locations. Delete those fields, or explicitly re-sign +the current contents with `--resign`: + +```bash +uv run labs/metric-benchmark/tools/sha-sign benchmark/experiment.yml --resign +``` + +`--resign` replaces old pins, including ones that no longer match edited inputs. +Signing edits only SHA fields, preserving comments, key order, quoting and +formatting elsewhere. New SHA fields are inserted into their reference mapping; +existing pins keep their plain or quoted scalar style. Block-scalar, anchored or +tagged SHA values must be changed to plain or quoted scalars before re-signing. +The input files should not be edited while signing. Inputs cannot include +manifests that the signer itself rewrites, since doing so would invalidate their +hashes. + +Once signed, readers verify each supplied digest as they load its reference. +`load_benchmark()` verifies all inputs; `read_benchmark()` verifies the manifests +it reads, and individual artifact loaders verify their own inputs. Omitting a +digest requires explicit `allow_unsigned=True` when loading during iteration. +SHA signing fixes expected contents; it does not authenticate a publisher. + +To check that the entire benchmark is signed correctly without changing files: + +```bash +uv run labs/metric-benchmark/tools/sha-sign benchmark/experiment.yml --check +``` + +`--check` requires a SHA on every input reference and verifies it against the +current contents. It stops at the first validation error, including a missing, +null or incorrect digest, and reports the input context. It uses the same strict +`load_benchmark()` path as normal full loading. To fill missing pins throughout +an unsigned bundle, run signing without `--check`; use `--resign` if some pins +already exist. Success exits with status zero; validation failures exit nonzero. +`--check` and `--resign` are mutually exclusive. +The top-level experiment has no self-hash, and inline cases have no separate +hash: their agent, tasks and subsets references are checked instead. + +To sign or check from Python: + +```python +from pathlib import Path +from metric_benchmark.signing import check_benchmark, sign_benchmark + +experiment = Path("benchmark/experiment.yml") +changed_files = sign_benchmark(experiment, resign=False) +check_benchmark(experiment) +``` + +`sign_benchmark()` returns the paths of YAML files it changed. +`check_benchmark()` returns normally on success and raises on missing or invalid +pins or inputs. Re-signing does not require existing pins to match. diff --git a/labs/metric-benchmark/docs/stages.md b/labs/metric-benchmark/docs/stages.md index 4543ded7..2c6e8760 100644 --- a/labs/metric-benchmark/docs/stages.md +++ b/labs/metric-benchmark/docs/stages.md @@ -15,7 +15,7 @@ fixed subset manifest. | Stage | Tool | | --- | --- | -| 1. Run mutants x eval tasks | Planned local Harbor loop; future NeMo Helix implementation | +| 1. Run mutants x eval tasks | `run_evals.py` (local Harbor); future NeMo Helix implementation | | 2. Compute mutant scores | `compute_mutant_scores.py` | | 3. Compute coverage | `compute_coverage.py` | | 4. Correlation | `correlate.py` | @@ -39,7 +39,7 @@ mutants and produce the results directory consumed by analysis. **Inputs:** prepared agent bundles, benchmark tasks, and experiment execution settings. -**Implementations:** the first execution script will run Harbor tasks in a +**Implementations:** the execution script runs Harbor tasks in a basic local loop: baseline first, then the configured mutants, with one Harbor job per agent covering the full benchmark. Agent jobs run sequentially. Task concurrency within a Harbor job is an explicit execution setting; a fully sequential run uses one @@ -49,8 +49,8 @@ the selected run; repetition aggregation remains future work. This phase is expected to be expensive: with `M` mutants and `T` tasks, even a single execution per pair requires `(M + 1) × T` task executions before retries. The baseline is included because mutant scoring compares its outcomes with each -mutant's outcomes. The repository currently supplies the analysis tools; the -local Harbor loop is the first execution implementation to add here. +mutant's outcomes. The local Harbor loop produces the shared index and native evidence used by +the analysis tools. A future NeMo Helix implementation will use an orchestrator to run evaluations at scale. Each implementation owns invocation, scheduling, retries, and collection diff --git a/labs/metric-benchmark/docs/tools.md b/labs/metric-benchmark/docs/tools.md index 253a4121..26b76518 100644 --- a/labs/metric-benchmark/docs/tools.md +++ b/labs/metric-benchmark/docs/tools.md @@ -24,73 +24,8 @@ directory layout, manifest fields, relative paths, and content pins: ## Signing a benchmark -Iterate with SHA fields omitted. When the prepared inputs are ready, sign the -bundle from the repository root: - -```bash -uv run labs/metric-benchmark/tools/sha-sign benchmark/experiment.yml -``` - -`sha-sign` reads the experiment and walks every case's referenced manifests and -inputs. It checks paths, file types, schemas and subset task IDs before writing -anything. It adds `sha256` to task sources, agent inputs, descriptions, subsets, -and agent/tasks manifest references, working from input files up to the -experiment. Shared manifests are updated once. It does not run agents or the -metric tool, and it does not modify task files, agent files or JSON inputs. - -If any reference already has a SHA field, signing stops without writing files -and lists those fields' locations. Delete those fields, or explicitly re-sign -the current contents with `--resign`: - -```bash -uv run labs/metric-benchmark/tools/sha-sign benchmark/experiment.yml --resign -``` - -`--resign` replaces old pins, including ones that no longer match edited inputs. -Signing edits only SHA fields, preserving comments, key order, quoting and -formatting elsewhere. New SHA fields are inserted into their reference mapping; -existing pins keep their plain or quoted scalar style. Block-scalar, anchored or -tagged SHA values must be changed to plain or quoted scalars before re-signing. -The input files should not be edited while signing. Inputs cannot include -manifests that the signer itself rewrites, since doing so would invalidate their -hashes. - -Once signed, readers verify each supplied digest as they load its reference. -`load_benchmark()` verifies all inputs; `read_benchmark()` verifies the manifests -it reads, and individual artifact loaders verify their own inputs. Omitting a -digest requires explicit `allow_unsigned=True` when loading during iteration. -SHA signing fixes expected contents; it does not authenticate a publisher. - -To check that the entire benchmark is signed correctly without changing files: - -```bash -uv run labs/metric-benchmark/tools/sha-sign benchmark/experiment.yml --check -``` - -`--check` requires a SHA on every input reference and verifies it against the -current contents. It stops at the first validation error, including a missing, -null or incorrect digest, and reports the input context. It uses the same strict -`load_benchmark()` path as normal full loading. To fill missing pins throughout -an unsigned bundle, run signing without `--check`; use `--resign` if some pins -already exist. Success exits with status zero; validation failures exit nonzero. -`--check` and `--resign` are mutually exclusive. -The top-level experiment has no self-hash, and inline cases have no separate -hash: their agent, tasks and subsets references are checked instead. - -To sign or check from Python: - -```python -from pathlib import Path -from metric_benchmark.signing import check_benchmark, sign_benchmark - -experiment = Path("benchmark/experiment.yml") -changed_files = sign_benchmark(experiment, resign=False) -check_benchmark(experiment) -``` - -`sign_benchmark()` returns the paths of YAML files it changed. -`check_benchmark()` returns normally on success and raises on missing or invalid -pins or inputs. Re-signing does not require existing pins to match. +See the [`sha-sign` command guide](sha-sign.md) for signing, re-signing, checking +pins, and the Python API. ## Reading benchmarks @@ -120,9 +55,8 @@ benchmark = load_benchmark(Path("benchmark/experiment.yml"), allow_unsigned=True This permits missing pins but still verifies every supplied digest. The lighter `read_benchmark()` and individual artifact loaders accept the same keyword, -also defaulting to `False`. The downstream execution and analysis PRs add -`--allow-unsigned` to their benchmark-reading CLIs; `sha-sign --check` always -requires all pins. +also defaulting to `False`. Benchmark-reading CLIs accept `--allow-unsigned` +for iteration; `sha-sign --check` always requires all pins. Use `read_benchmark(path)` from the same module when only the experiment and inline cases and referenced agent/task manifests are needed. It does not open @@ -139,28 +73,11 @@ expose their format through `model_json_schema()`. ## Validate a results directory -See [Generating the report](validation-report.md#generating-the-report) for the -CLI invocation, required arguments, and output behavior. - -Library: `validate_experiment_results(loaded, root, *, case_id=None)` returns the -same report as the CLI. Pass a `LoadedBenchmark` from `load_benchmark` and the -whole experiment output root; `case_id` optionally selects one case. - -The per-case `validate_results(raw, case, root, input_root, scoring, *, allow_unsigned=False)` -function remains available for execution and analysis libraries. Its `root` is -that case’s results directory, and it returns the individual case report. +See the [`validate_results.py` command guide](validate-results.md) for arguments, +output behavior, and the Python API. The [validation report](validation-report.md) +defines the output format. -The [results format specification](results-format.md) defines the index, native -evidence requirements and matrix accounting. The [validation report](validation-report.md) -defines the tool’s output. Start with the [mock results](results-format.md#mock-results) -to review the checked-in mock directory. +## Execute the eval tasks -`test_contract_and_cli_parity` in -[the validation tests](../../../tests/test_metric_benchmark_validation.py) -materializes the synthetic input tasks, copies the mock output to a temporary -directory, and checks that the library and CLI reproduce the checked-in report. -To exercise that example without Docker or model calls: - -```bash -uv run --locked pytest tests/test_metric_benchmark_validation.py -k contract_and_cli_parity -q -``` +See the [`run_evals.py` command guide](run-evals.md) for arguments, the output +layout, completed-job reuse, interrupted-job recovery, and the Python API. diff --git a/labs/metric-benchmark/docs/validate-results.md b/labs/metric-benchmark/docs/validate-results.md new file mode 100644 index 00000000..e34efd31 --- /dev/null +++ b/labs/metric-benchmark/docs/validate-results.md @@ -0,0 +1,62 @@ + + + +# validate_results.py + +Validate saved Harbor evidence against the prepared benchmark inputs. This command +does not execute agents or compute coverage. + +## Usage + +From the repository root, invoke the validation CLI with: + +```bash +uv run python labs/metric-benchmark/validate_results.py \ + --benchmark experiment.yml \ + --results-root experiment-results \ + --output validation.json +``` + +Supply these required arguments: + +- `--benchmark`: path to the experiment YAML file and its references to prepared inputs. +- `--results-root`: whole experiment output directory containing `execution.json` and the case results directories. +- `--output`: path for the validation report, outside the results root; `validation.json` is an example filename. + +By default, the tool validates every configured case. Add `--case case-id` to +validate only that case example, using the same `--results-root` path. The +experiment’s `execution.json` maps each case ID to its results index. + +Optionally, add `--allow-unsigned` when input SHA pins are missing during iteration; +any supplied pins are still verified. + +The CLI checks all selected cases and writes one report only if every selected +case passes. A filtered invocation uses the same report structure with one case. +Explicitly unavailable observations do not themselves cause validation to fail. If any constraint fails, the command exits nonzero and +no new report is written. An existing output file can be reused only if its +contents are identical; choose a new output path for changed results. + +## Python library + +`validate_experiment_results(loaded, root, *, case_id=None)` returns the +same report as the CLI. Pass a `LoadedBenchmark` from `load_benchmark` and the +whole experiment output root; `case_id` optionally selects one case. + +The per-case `validate_results(raw, case, root, input_root, scoring, *, allow_unsigned=False)` +function remains available for execution and analysis libraries. Its `root` is +that case’s results directory, and it returns the individual case report. + +The [results format specification](results-format.md) defines the index, native +evidence requirements and matrix accounting. The [validation report](validation-report.md) +defines the tool’s output. Start with the [mock results](results-format.md#mock-results) +to review the checked-in mock directory. + +`test_contract_and_cli_parity` in +[the validation tests](../../../tests/test_metric_benchmark_validation.py) +materializes the synthetic input tasks, copies the mock output to a temporary +directory, and checks that the library and CLI reproduce the checked-in report. +To exercise that example without Docker or model calls: + +```bash +uv run --locked pytest tests/test_metric_benchmark_validation.py -k contract_and_cli_parity -q +``` diff --git a/labs/metric-benchmark/docs/validation-report.md b/labs/metric-benchmark/docs/validation-report.md index 929f2fdb..f4700d11 100644 --- a/labs/metric-benchmark/docs/validation-report.md +++ b/labs/metric-benchmark/docs/validation-report.md @@ -14,36 +14,8 @@ pass, it produces the report described here. ## Generating the report -From the repository root, invoke the validation CLI with: - -```bash -uv run python labs/metric-benchmark/validate_results.py \ - --benchmark experiment.yml \ - --results-root experiment-results \ - --output validation.json -``` - -Supply these required arguments: - -- `--benchmark`: path to the experiment YAML file and its references to prepared inputs. -- `--results-root`: whole experiment output directory containing `execution.json` and the case results directories. -- `--output`: path for the validation report, outside the results root; `validation.json` is an example filename. - -By default, the tool validates every configured case. Add `--case case-id` to -validate only that case example, using the same `--results-root` path. The -experiment’s `execution.json` maps each case ID to its results index. - -Optionally, add `--allow-unsigned` when input SHA pins are missing during iteration; -any supplied pins are still verified. - -The CLI checks all selected cases and writes one report only if every selected -case passes. A filtered invocation uses the same report structure with one case. -Explicitly unavailable observations do not themselves cause validation to fail. If any constraint fails, the command exits nonzero and -no new report is written. An existing output file can be reused only if its -contents are identical; choose a new output path for changed results. - -The [Python library](tools.md#validate-a-results-directory) returns the same report -as a dictionary. +Use the [`validate_results.py` command guide](validate-results.md) for the CLI +invocation, arguments, output behavior, and Python API. ## Report format: `validation.json` diff --git a/labs/metric-benchmark/metric_benchmark/execution.py b/labs/metric-benchmark/metric_benchmark/execution.py new file mode 100644 index 00000000..d114996d --- /dev/null +++ b/labs/metric-benchmark/metric_benchmark/execution.py @@ -0,0 +1,258 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Sequential case/agent orchestration of native Harbor jobs.""" + +import asyncio +import sys +from collections.abc import Callable +from importlib.metadata import version +from pathlib import Path + +from harbor.models.job.config import JobConfig +from harbor.models.trial.config import EnvironmentConfig +from harbor.models.trial.result import TrialResult + +from metric_benchmark.config import Case, LoadedBenchmark, LoadedCase, file_manifest +from metric_benchmark.io import digest_bytes, digest_file, input_digest, json_bytes, read_json, write_json +from metric_benchmark.models import Execution, ExecutionManifest, RawResults, Text, resolve_evidence +from metric_benchmark.validation import agent_config, native_observation, validate_results + + +def benchmark_identity(loaded: LoadedBenchmark) -> str: + """Compute the benchmark reuse fingerprint, including the contents of inputs without SHA pins.""" + unsigned = {} + for entry in loaded.definition.cases: + loaded_case = loaded.case(entry.case_id) + case = loaded_case.case + sources = [entry.agent, entry.tasks, entry.subsets, case.case_description, *case.task_sources.values()] + sources.extend(source for agent in case.agents() for source in agent.inputs.values()) + for source in sources: + if source.sha256 is None and source.path not in unsigned: + unsigned[source.path] = input_digest( + source.resolve(loaded_case.root, allow_unsigned=loaded_case.allow_unsigned) + ) + if not unsigned: + return loaded.identity + return digest_bytes(json_bytes({"benchmark_sha256": loaded.identity, "unsigned_inputs": unsigned})) + + +def case_root(output: Path, case_index: int) -> Path: + """Locate a case’s results directory by its zero-based position in the benchmark.""" + return output / "cases" / f"case-{case_index:04d}" / "execution-results" + + +def agent_run_id(agent_index: int) -> str: + """Return the local executor’s logical run ID for an agent’s zero-based position.""" + return f"agent-{agent_index:04d}/run-0001" + + +def run_harbor(config: JobConfig) -> None: + """Run one agent’s full task inventory through Harbor and wait for completion. + Harbor schedules the task trials using the configured concurrency. + """ + from harbor.job import Job + + async def run(): + job = await Job.create(config) + await job.run() + + asyncio.run(run()) + + +def job_configuration( + loaded: LoadedBenchmark, loaded_case: LoadedCase, agent_index: int, root: Path, paths: list[Path] +) -> JobConfig: + """Build and validate a Harbor job for one agent and all supplied task paths. + Resolve the agent’s input files and execution settings without starting the job. + """ + case = loaded_case.case + settings = loaded.definition.execution + if set(settings.environment) - EnvironmentConfig.model_fields.keys(): + raise ValueError("unknown Harbor environment setting") + return JobConfig.model_validate( + { + "job_name": "run-0001", + "jobs_dir": str(root / "agents" / f"agent-{agent_index:04d}"), + "n_attempts": 1, + "retry": {"max_retries": 0, "exclude_exceptions": []}, + "n_concurrent_trials": settings.n_concurrent_trials, + "environment": settings.environment, + "agents": [ + agent_config( + case.agents()[agent_index], loaded_case.root, allow_unsigned=loaded_case.allow_unsigned + ).model_dump(mode="json") + ], + "tasks": [{"path": str(path)} for path in paths], + "quiet": True, + } + ) + + +def collect_job(case: Case, agent_index: int, root: Path, request: JobConfig, metric: str) -> list[Execution]: + """Read a finished Harbor job into one Execution record per task, including status, score, and evidence path. + Use saved task paths to identify tasks; reject unfinished jobs and missing, duplicate, or unexpected tasks. + """ + agent = case.agents()[agent_index] + directory = root / "agents" / f"agent-{agent_index:04d}" / "run-0001" + job = read_json(resolve_evidence(directory, "result.json")) + if not isinstance(job, dict) or not job.get("finished_at"): + raise ValueError("Harbor job has not finished; preserve this run and resume it explicitly in Harbor") + task_paths = { + native.path: task + for native, task in zip(request.tasks, sorted(case.task_inventory.tasks, key=lambda t: t.key()), strict=True) + } + rows = [] + seen = set() + for result_path in sorted(directory.glob("*/result.json")): + native = TrialResult.model_validate( + read_json(resolve_evidence(directory, result_path.relative_to(directory).as_posix())) + ) + path = native.config.task.path + task = task_paths.get(path) if path is not None else None + if task is None or task.task_id in seen: + raise ValueError("unexpected or repeated native task in Harbor job") + seen.add(task.task_id) + status, score = native_observation(native, metric) + rows.append( + Execution.model_validate( + { + "agent_run_id": agent_run_id(agent_index), + "agent_bundle_id": agent.agent_bundle_id, + "evaluation": { + "evaluation_id": f"agent-{agent_index:04d}/{task.task_id}", + "execution": 1, + "task": task.model_dump(), + }, + "status": status, + "metrics": {metric: score.model_dump()}, + "native_results_path": result_path.parent.relative_to(root).as_posix(), + "native_results_format": "directory", + } + ) + ) + if seen != {t.task_id for t in case.task_inventory.tasks}: + raise ValueError("finished Harbor job does not account for every configured task") + return sorted(rows, key=lambda r: r.evaluation.task.key()) + + +def run_or_reuse_job( + loaded: LoadedBenchmark, + case: Case, + agent_index: int, + root: Path, + config: JobConfig, + identity: str, + *, + runner: Callable[[JobConfig], None] = run_harbor, +) -> list[Execution]: + """Run or verify one agent’s Harbor job, collect its task observations, and seal completed evidence. + Preserve partial jobs for explicit recovery and report failures with case and agent context. + """ + agent = case.agents()[agent_index] + try: + parent = config.jobs_dir + directory = parent / config.job_name + receipt_path = parent / "receipt.json" + sealed = receipt_path.exists() + request = { + "benchmark_sha256": identity, + "case_id": case.case_id, + "agent_bundle_id": agent.agent_bundle_id, + "execution": loaded.definition.execution.model_dump(mode="json"), + } + write_json(parent / "request.json", request) + native_path = parent / "native-request.json" + if not native_path.exists(): + if directory.exists() or sealed: + raise ValueError("existing job lacks native-request.json; original task bindings are required") + write_json(native_path, config) + native_request = JobConfig.model_validate(read_json(resolve_evidence(parent, "native-request.json"))) + expected = { + "configuration_sha256": digest_bytes(json_bytes(request)), + "native_request_sha256": digest_file(native_path), + } + print( + f"case {case.case_id}, agent {agent.agent_bundle_id} ({agent_index + 1}/{len(case.agents())}): " + + ("checking saved job" if directory.exists() else "running Harbor"), + file=sys.stderr, + ) + if sealed: + receipt = read_json(resolve_evidence(parent, "receipt.json")) + if receipt != {**expected, "files": file_manifest(directory)}: + raise ValueError("completed Harbor job changed") + elif not directory.exists(): + # An unstarted request is reusable only at its original paths; + # relocation support applies to completed jobs. + if native_request != config: + raise ValueError("unstarted job request moved; use a new output for this execution") + runner(config) + collected = collect_job(case, agent_index, root, native_request, loaded.definition.scoring.metric) + if not sealed: + write_json(receipt_path, {**expected, "files": file_manifest(directory)}) + return collected + except (OSError, ValueError) as error: + raise ValueError(f"case {case.case_id}, agent {agent.agent_bundle_id}: {error}") from error + + +def execute_benchmark( + loaded: LoadedBenchmark, + run_id: Text, + output: Path, + *, + runner: Callable[[JobConfig], None] = run_harbor, +) -> dict: + """Run one Harbor job per case/agent and publish validated case results plus an experiment manifest. + Reuse verified completed jobs; leave incomplete jobs for explicit Harbor recovery. + """ + if not run_id.strip(): + raise ValueError("run_id must be nonempty") + if version("harbor") != loaded.definition.execution.harbor_version: + raise ValueError("installed Harbor version differs from benchmark execution version") + identity = benchmark_identity(loaded) + output = output.resolve() + # Resolve and validate all jobs before launching any, so a bad later case + # cannot spend a valid earlier case's execution budget. + jobs = {} + for case_index, loaded_case in enumerate(loaded.cases.values()): + case = loaded_case.case + paths = [ + case.task_sources[t.task_id].resolve(loaded_case.root, allow_unsigned=loaded_case.allow_unsigned) + for t in sorted(case.task_inventory.tasks, key=lambda t: t.key()) + ] + if len(set(paths)) != len(paths): + raise ValueError(f"case {case.case_id}: task sources must have distinct paths") + for agent_index in range(len(case.agents())): + jobs[case_index, agent_index] = job_configuration( + loaded, loaded_case, agent_index, case_root(output, case_index), paths + ) + inputs_path = output / "execution-inputs.json" + inputs = { + "benchmark_sha256": identity, + "benchmark": loaded.definition.model_dump(mode="json"), + "run_id": run_id, + } + if inputs_path.exists() and read_json(inputs_path) != inputs: + raise ValueError("benchmark inputs, configuration, or run ID changed; use a new output directory") + write_json(inputs_path, inputs) + indexes = {} + # Each agent gets one job containing all case tasks; Harbor schedules the individual trials. + for case_index, loaded_case in enumerate(loaded.cases.values()): + case = loaded_case.case + root = case_root(output, case_index) + rows = [] + for agent_index in range(len(case.agents())): + rows.extend( + run_or_reuse_job( + loaded, case, agent_index, root, jobs[case_index, agent_index], identity, runner=runner + ) + ) + raw = RawResults(run_id=run_id, task_inventory=case.task_inventory.tasks, execution_results=rows) + validate_results( + raw, case, root, loaded_case.root, loaded.definition.scoring, allow_unsigned=loaded.allow_unsigned + ) + write_json(root / "results.json", raw) + indexes[case.case_id] = (root / "results.json").relative_to(output).as_posix() + result = ExecutionManifest(benchmark_sha256=identity, run_id=run_id, cases=indexes).model_dump() + write_json(output / "execution.json", result) + return result diff --git a/labs/metric-benchmark/run_evals.py b/labs/metric-benchmark/run_evals.py new file mode 100644 index 00000000..4a5ff61d --- /dev/null +++ b/labs/metric-benchmark/run_evals.py @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Sequential case/agent orchestration of native Harbor jobs.""" + +import argparse +from pathlib import Path + +from metric_benchmark.config import load_benchmark +from metric_benchmark.execution import execute_benchmark +from metric_benchmark.io import cli + + +def main() -> None: + """Parse execution arguments, load the benchmark, and invoke the execution library.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--benchmark", type=Path, required=True) + parser.add_argument( + "--allow-unsigned", action="store_true", help="allow missing input pins; still verify supplied SHAs" + ) + parser.add_argument("--run-id", required=True) + parser.add_argument("--output", type=Path, required=True) + + def run(args): + execute_benchmark(load_benchmark(args.benchmark, allow_unsigned=args.allow_unsigned), args.run_id, args.output) + + cli(parser, run) + + +if __name__ == "__main__": + main() diff --git a/tests/test_metric_benchmark_execution.py b/tests/test_metric_benchmark_execution.py new file mode 100644 index 00000000..1d0a7375 --- /dev/null +++ b/tests/test_metric_benchmark_execution.py @@ -0,0 +1,350 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from pathlib import Path + +import pytest +from metric_benchmark.config import load_benchmark +from metric_benchmark.execution import case_root, execute_benchmark +from metric_benchmark.io import read_json, write_json +from test_metric_benchmark_config import case_definition, prepared_benchmark, write_case +from test_metric_benchmark_validation import native_trial + + +def fake_harbor(config): + directory = config.jobs_dir / config.job_name + directory.mkdir(parents=True) + write_json(directory / "result.json", {"finished_at": "2026-01-01T00:00:00"}) + for task in config.tasks: + native = directory / (task.path.name + "__arbitrary") + native.mkdir() + result = native_trial(task.path, native, config.agents[0], float(config.agents[0].name == "oracle")) + write_json(native / "result.json", result) + + +def test_order_full_matrix_and_verified_resume(tmp_path): + path, _ = prepared_benchmark(tmp_path / "inputs") + loaded = load_benchmark(path) + calls = [] + + def runner(config): + calls.append(config.agents[0].name) + assert config.n_attempts == 1 and config.retry.max_retries == 0 + fake_harbor(config) + + output = tmp_path / "out" + first = execute_benchmark(loaded, "run", output, runner=runner) + assert calls == ["oracle", "nop", "nop"] + assert execute_benchmark(loaded, "run", output, runner=runner) == first + assert calls == ["oracle", "nop", "nop"] + raw = read_json(case_root(output, 0) / "results.json") + assert isinstance(raw, dict) + assert len(raw["execution_results"]) == 9 + with pytest.raises(ValueError, match="changed; use a new output directory"): + execute_benchmark(loaded, "other-run", output, runner=runner) + evidence = case_root(output, 0) / raw["execution_results"][0]["native_results_path"] / "result.json" + evidence.write_text("{}") + with pytest.raises(ValueError, match="job changed"): + execute_benchmark(loaded, "run", output, runner=runner) + + +def test_two_cases_produce_distinct_results_and_resume(tmp_path): + import yaml + from metric_benchmark.io import input_digest + from metric_benchmark.validation import validate_experiment_results + + path, definition = prepared_benchmark(tmp_path / "inputs") + first = definition["cases"][0] + agent_path = path.parent / "second-agent.yml" + agents = yaml.safe_load((path.parent / first["agent"]["path"]).read_text()) + agents["baseline"]["agent_bundle_id"] = "second-baseline" + agents["baseline"]["harbor"] = {"name": "nop"} + agents["mutants"] = [{**agents["mutants"][0], "agent_bundle_id": "second-mutant", "harbor": {"name": "oracle"}}] + agent_path.write_text(yaml.safe_dump(agents)) + definition["cases"].append( + {**first, "case_id": "second-case", "agent": {"path": agent_path.name, "sha256": input_digest(agent_path)}} + ) + path.write_text(yaml.safe_dump(definition)) + loaded = load_benchmark(path) + output = tmp_path / "out" + calls = [] + + def runner(config): + calls.append((config.jobs_dir.relative_to(output).parts[1], config.agents[0].name)) + fake_harbor(config) + + manifest = execute_benchmark(loaded, "run", output, runner=runner) + assert calls == [ + ("case-0000", "oracle"), + ("case-0000", "nop"), + ("case-0000", "nop"), + ("case-0001", "nop"), + ("case-0001", "oracle"), + ] + assert manifest["cases"] == { + first["case_id"]: "cases/case-0000/execution-results/results.json", + "second-case": "cases/case-0001/execution-results/results.json", + } + for ordinal, prepared in enumerate(loaded.cases.values()): + raw = read_json(case_root(output, ordinal) / "results.json") + assert isinstance(raw, dict) + rows = raw["execution_results"] + assert {(row["agent_bundle_id"], row["evaluation"]["task"]["task_id"]) for row in rows} == { + (agent.agent_bundle_id, task.task_id) + for agent in prepared.case.agents() + for task in prepared.case.task_inventory.tasks + } + assert [row["metrics"]["reward"]["value"] for row in rows] == ( + [1.0] * 3 + [0.0] * 6 if ordinal == 0 else [0.0] * 3 + [1.0] * 3 + ) + validate_experiment_results(loaded, output) + before = {p.relative_to(output): p.read_bytes() for p in output.rglob("*") if p.is_file()} + assert execute_benchmark(loaded, "run", output, runner=lambda _: pytest.fail("must not replay")) == manifest + assert {p.relative_to(output): p.read_bytes() for p in output.rglob("*") if p.is_file()} == before + + +def test_failure_between_agents_reuses_completed_job(tmp_path): + path, _ = prepared_benchmark(tmp_path / "inputs") + loaded = load_benchmark(path) + output = tmp_path / "out" + + def failing(config): + if config.agents[0].name == "nop": + raise ValueError("worker unavailable") + fake_harbor(config) + + with pytest.raises(ValueError, match="worker unavailable"): + execute_benchmark(loaded, "run", output, runner=failing) + calls = [] + + def resume(config): + calls.append(config.agents[0].name) + fake_harbor(config) + + execute_benchmark(loaded, "run", output, runner=resume) + assert calls == ["nop", "nop"] + + +def test_partial_native_job_is_not_rerun(tmp_path): + path, _ = prepared_benchmark(tmp_path / "inputs") + loaded = load_benchmark(path) + output = tmp_path / "out" + + def partial(config): + folder = config.jobs_dir / config.job_name + folder.mkdir(parents=True) + write_json(folder / "result.json", {"finished_at": None}) + + with pytest.raises(ValueError, match="not finished"): + execute_benchmark(loaded, "run", output, runner=partial) + + def forbidden(config): + raise AssertionError("must not replay") + + with pytest.raises(ValueError, match="not finished"): + execute_benchmark(loaded, "run", output, runner=forbidden) + + +def test_native_harbor_cli_smoke(tmp_path): + """Opt-in: nine Docker trials, no model calls or remote benchmark service.""" + import os + import subprocess + import sys + + from metric_benchmark.config import input_digest + from test_metric_benchmark_config import LAB + + if os.environ.get("METRIC_BENCHMARK_DOCKER_TEST") != "1": + pytest.skip("set METRIC_BENCHMARK_DOCKER_TEST=1 for local Docker smoke") + path, definition = prepared_benchmark(tmp_path / "inputs") + case_data = case_definition(path) + for task_id, source in case_data["task_sources"].items(): + task = path.parent / source["path"] + (task / "environment").mkdir(exist_ok=True) + (task / "environment/Dockerfile").write_text("FROM python:3.12-slim\nWORKDIR /app\n") + (task / "solution").mkdir() + (task / "solution/solve.sh").write_text("#!/bin/bash\nprintf ready > /app/answer\n") + (task / "tests").mkdir(exist_ok=True) + (task / "tests/test.sh").write_text( + "#!/bin/bash\nmkdir -p /logs/verifier\n" + "if test -f /app/answer; then echo 1; else echo 0; fi > /logs/verifier/reward.txt\n" + ) + (task / "task.toml").write_text('version = "1.0"\n[agent]\ntimeout_sec = 30\n[verifier]\ntimeout_sec = 30\n') + source["sha256"] = input_digest(task) + write_case(path, case_data) + output = tmp_path / "native-run" + subprocess.run( + [ + sys.executable, + str(LAB / "run_evals.py"), + "--benchmark", + str(path), + "--run-id", + "docker-smoke", + "--output", + str(output), + ], + check=True, + timeout=300, + ) + loaded = load_benchmark(path) + before = (case_root(output, 0) / "results.json").read_bytes() + + def forbidden(config): + raise AssertionError("native jobs must be reused") + + execute_benchmark(loaded, "docker-smoke", output, runner=forbidden) + assert (case_root(output, 0) / "results.json").read_bytes() == before + raw = read_json(case_root(output, 0) / "results.json") + assert isinstance(raw, dict) + assert [r["metrics"]["reward"]["value"] for r in raw["execution_results"]] == [1.0] * 3 + [0.0] * 6 + import shutil + + relocated_inputs = tmp_path / "relocated-inputs" + relocated_output = tmp_path / "relocated-output" + shutil.move(path.parent, relocated_inputs) + shutil.move(output, relocated_output) + execute_benchmark(load_benchmark(relocated_inputs / path.name), "docker-smoke", relocated_output, runner=forbidden) + assert (case_root(relocated_output, 0) / "results.json").read_bytes() == before + + +@pytest.mark.parametrize("unsigned", [False, True]) +@pytest.mark.parametrize("move_inputs,move_output", [(False, False), (True, False), (False, True), (True, True)]) +def test_move_completed_jobs_and_prepared_bundles_without_reexecution(tmp_path, move_inputs, move_output, unsigned): + import shutil + + from metric_benchmark.config import input_digest + + original = tmp_path / "producer" + path, definition = prepared_benchmark(original / "inputs") + bundle = path.parent / "bundle.txt" + bundle.write_text("prepared dependency") + case_data = case_definition(path) + for agent in [case_data["baseline"], *case_data["mutants"]]: + agent["inputs"] = {"bundle": {"path": bundle.name, "sha256": input_digest(bundle)}} + agent["harbor"]["kwargs"] = {"bundle": "{input:bundle}"} + write_case(path, case_data) + if unsigned: + from test_metric_benchmark_signing import remove_pins + + remove_pins(path.parent) + loaded = load_benchmark(path, allow_unsigned=unsigned) + output = original / "out" + execute_benchmark(loaded, "run", output, runner=fake_harbor) + before = {p.relative_to(output): p.read_bytes() for p in output.rglob("*") if p.is_file()} + moved = tmp_path / "consumer" + moved.mkdir() + inputs = path.parent + if move_inputs: + inputs = Path(shutil.move(inputs, moved / "inputs")) + if move_output: + output = Path(shutil.move(output, moved / "out")) + + def forbidden(config): + raise AssertionError("relocation must reuse completed native jobs") + + execute_benchmark( + load_benchmark(inputs / "experiment.yml", allow_unsigned=unsigned), "run", output, runner=forbidden + ) + assert {p.relative_to(output): p.read_bytes() for p in output.rglob("*") if p.is_file()} == before + evidence = next(output.glob("cases/*/execution-results/agents/agent-0000/run-0001/*/result.json")) + evidence.write_text("{}") + with pytest.raises(ValueError, match="case rho-agent, agent baseline: completed Harbor job changed"): + execute_benchmark( + load_benchmark(inputs / "experiment.yml", allow_unsigned=unsigned), "run", output, runner=forbidden + ) + + +def test_duplicate_task_sources_rejected_before_execution(tmp_path): + + path, definition = prepared_benchmark(tmp_path / "inputs") + case_data = case_definition(path) + sources = case_data["task_sources"] + sources["read-config"] = sources["inspect-files"] + write_case(path, case_data) + with pytest.raises(ValueError, match="distinct paths"): + execute_benchmark( + load_benchmark(path), "run", tmp_path / "out", runner=lambda _: pytest.fail("must not execute") + ) + + +def test_logical_request_and_exact_nested_task_bindings(tmp_path): + import shutil + + path, definition = prepared_benchmark(tmp_path / "inputs") + case_data = case_definition(path) + sources = case_data["task_sources"] + # One source suffix is another's suffix; collection must use the exact saved binding. + for task_id, relative in [("inspect-files", "tasks/a"), ("read-config", "x/tasks/a")]: + destination = path.parent / relative + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.move(path.parent / sources[task_id]["path"], destination) + sources[task_id]["path"] = relative + write_case(path, case_data) + loaded = load_benchmark(path) + output = tmp_path / "out" + + def runner(config): + directory = config.jobs_dir / config.job_name + directory.mkdir(parents=True) + write_json(directory / "result.json", {"finished_at": "2026-01-01T00:00:00"}) + for i, task in enumerate(config.tasks): + native = directory / f"trial-{i}" + native.mkdir() + write_json(native / "result.json", native_trial(task.path, native, config.agents[0], float(i))) + + execute_benchmark(loaded, "run", output, runner=runner) + parent = case_root(output, 0) / "agents/agent-0000" + request = read_json(parent / "request.json") + assert request == { + "benchmark_sha256": loaded.identity, + "case_id": "rho-agent", + "agent_bundle_id": "baseline", + "execution": loaded.definition.execution.model_dump(mode="json"), + } + assert str(tmp_path) not in (parent / "request.json").read_text() + assert str(path.parent) in (parent / "native-request.json").read_text() + raw = read_json(case_root(output, 0) / "results.json") + assert isinstance(raw, dict) + assert [ + (row["evaluation"]["task"]["task_id"], row["metrics"]["reward"]["value"]) + for row in raw["execution_results"][:3] + ] == [("inspect-files", 0.0), ("read-config", 1.0), ("search-workspace", 2.0)] + (parent / "native-request.json").write_text("{}") + with pytest.raises(ValueError): + execute_benchmark(loaded, "run", output, runner=lambda _: pytest.fail("must not execute")) + + +@pytest.mark.parametrize( + "changed", + [ + "agents/rho-agent/agent.yml", + "tasks/workspace/tasks.yml", + "tasks/workspace/subsets.json", + "agents/rho-agent/description.json", + "tasks/workspace/inspect-files/instruction.md", + "bundle.txt", + ], +) +def test_unsigned_input_changes_refuse_completed_job_reuse(tmp_path, changed): + from test_metric_benchmark_signing import remove_pins + + path, _ = prepared_benchmark(tmp_path / "inputs") + bundle = path.parent / "bundle.txt" + bundle.write_text("original agent dependency") + case = case_definition(path) + case["baseline"]["inputs"] = {"bundle": {"path": bundle.name}} + case["baseline"]["harbor"]["kwargs"] = {"bundle": "{input:bundle}"} + write_case(path, case) + remove_pins(path.parent) + output = tmp_path / "out" + execute_benchmark(load_benchmark(path, allow_unsigned=True), "run", output, runner=fake_harbor) + with (path.parent / changed).open("a") as stream: + stream.write("\n") + with pytest.raises(ValueError, match="changed; use a new output directory"): + execute_benchmark( + load_benchmark(path, allow_unsigned=True), + "run", + output, + runner=lambda _: pytest.fail("changed inputs must not resume old work"), + )