diff --git a/labs/metric-benchmark/docs/architecture.md b/labs/metric-benchmark/docs/architecture.md index 3501553a..6db7e13f 100644 --- a/labs/metric-benchmark/docs/architecture.md +++ b/labs/metric-benchmark/docs/architecture.md @@ -45,30 +45,20 @@ coverage, and correlation. Prep is external to those stages. ## Benchmark definition -A metric benchmark consists of an experiment with inline cases and linked manifests. SHA-256 pins are optional during -iteration with explicit unsigned loading; default loading requires every input -to be pinned. Established benchmarks pin every reference to fix their inputs: - -- **Experiment YAML (`experiment.yml`):** defines inline cases and configures execution, - metric computation, and scoring. -- **Case entry (inside `experiment.yml`):** combines an agent manifest, a tasks manifest, - and a prepared subset manifest. -- **Agent manifest (`agent.yml`):** defines the baseline, mutants, and intended - behavior description, with references to prepared dependencies. -- **Tasks manifest (`tasks.yml`):** defines the fixed inventory and task locations. -- **Subset manifest:** records the list of exact task memberships and their - preparation seed. It is bound to the task inventory, independently of case IDs. - -Prep produces these fixed inputs. Cases can share agents, tasks and subsets; -experiments combine these inputs through their inline case definitions. -Different metric configurations can reuse the same prepared inputs. - -After unpacking a benchmark, all paths resolve relative to the bundle root—the -folder containing the experiment YAML. Nested manifests can reference shared -inputs without `../`. Supplied SHA-256 pins verify their links. The -[benchmark format specification](benchmark-format.md) specifies the directory -layout and fields. Each execution produces its own results while retaining the -configured input identities. +A metric benchmark is one experiment referencing three configuration files: + +- **Cases (`cases.yml`):** selects agent and task manifests for each case ID. +- **Execution (`execution.yml`):** configures the local Harbor loop. +- **Analysis (`analysis.yml`):** selects descriptions and fixed subsets by case ID, + and configures coverage, scoring, and missing-data handling. + +`experiment.yml` references these three files directly. Standalone execution takes +cases plus execution settings; analysis takes cases plus analysis settings and +saved results. The orchestrator passes the same loaded objects to stage libraries. +Analysis reads case metadata without requiring executable agent/task directories. +Execution reuse depends only on execution inputs and settings, so analysis-only +changes can reuse saved observations. See the [format specification](benchmark-format.md) +for references, content pins, and the example layout. ## 1. Prep @@ -83,7 +73,7 @@ consumes them; it does not regenerate or select mutants based on its results. ### Supplied inputs -A run receives five supplied inputs. Case packages provide the first four; +A run receives five supplied inputs. Preparation supplies the first four; the experiment configuration specifies how to use them. The table describes their logical contents. @@ -91,15 +81,15 @@ their logical contents. | --- | --- | | Baseline agent | An executable bundle with an immutable identity and a documented invocation contract | | Mutants | Executable bundles with distinct identities, their baseline relationship, and mutation descriptions | -| Case description | Stable case identity and intended agent behavior; an Ethos is one possible representation | +| Case description | Prepared description of intended agent behavior, referenced by case ID through `analysis.case_settings`; an Ethos is one possible representation | | Benchmark tasks | Fixed task membership, task identities, environments, correctness verifiers, and a versioned subset manifest | -| Experiment configuration | `experiment.yml`: inline cases and their input references, execution settings, coverage-system settings and permitted evidence, and scoring parameters | +| Experiment configuration | `experiment.yml`: references to cases, execution settings, and analysis settings | Baseline and mutant bundles are prepared and checked before release. Tasks and their environments are ready for execution. At present only Harbor tasks are used. -The configuration supplies the choices that determine execution: the fixed subset manifest, -scoring rules, and the coverage-tool command and settings. Analysis derives one +Execution settings determine the Harbor loop. Analysis settings supply the fixed +subset manifest, scoring rules, and coverage-tool command and settings. Analysis derives one agent run per configured agent from the results index. The program validates these inputs before using them. Missing or inconsistent inputs cause an error. diff --git a/labs/metric-benchmark/docs/assets/experiment-flow.svg b/labs/metric-benchmark/docs/assets/experiment-flow.svg index f6a4f1ba..7eaf7ad3 100644 --- a/labs/metric-benchmark/docs/assets/experiment-flow.svg +++ b/labs/metric-benchmark/docs/assets/experiment-flow.svg @@ -57,7 +57,7 @@ Benchmark tasks Tasks + verifiers experiment.yml - Defines cases inline + References cases and settings Construct eval subsets Pinned task inventory + fixed seed @@ -67,7 +67,7 @@ Subset manifest Named subsets + explicit task IDs Versioned subsets.json · algorithm + fixed seed recorded - Experiment YAML defines cases; each selects agent, tasks, and subset manifests. + Experiment YAML references cases, execution settings, and analysis settings. diff --git a/labs/metric-benchmark/docs/benchmark-format.md b/labs/metric-benchmark/docs/benchmark-format.md index 4d85283a..eaccbd9a 100644 --- a/labs/metric-benchmark/docs/benchmark-format.md +++ b/labs/metric-benchmark/docs/benchmark-format.md @@ -3,156 +3,78 @@ # Benchmark format -This is the format contract for externally prepared metric benchmark bundles. -Use the recommended layout below for new bundles. The Python reader implements -this contract; [Benchmark tools](tools.md#reading-benchmarks) explains how to -load and verify a bundle. Preparation is external to the experiment. - -A bundle connects the inputs in this order: +One experiment composes three files. Cases define the agents and tasks; execution +and analysis settings do not refer to cases themselves. ```text experiment.yml - -> cases (inline list) - -> agent.yml -> baseline files, mutant files, description.json - -> tasks.yml -> task directories, each with its own verifier - -> subsets.json -> task IDs from the selected tasks.yml + -> cases.yml -> agent.yml -> baseline and mutant files + -> tasks.yml -> Harbor task directories + -> execution.yml + -> analysis.yml -> description.json and subsets.json, selected by case ID ``` -Cases select agent, task, and subset manifests. The experiment defines its -cases inline and supplies execution and analysis settings. Reusing a manifest does not -copy or regenerate its contents. Execution results belong in a separate output -directory; they are not prepared inputs in this bundle. +Preparation happens outside the experiment. Agent and task files are materialized +locally before execution; analysis can use case metadata and saved results without +those executable files. The [tool guides](tools.md) explain the library and CLI interfaces. -Use YAML for experiment, agent and task manifests so authors can edit and -comment on configuration. Use JSON for subset memberships and agent descriptions -as data exchanged with preparation and metric tools. The examples use `.yml` -and `.json` accordingly; changing a filename does not change its expected format. +## Experiment: `experiment.yml` -## Recommended directory layout +```yaml +benchmark_id: rho-metric-validity +revision: 1 +cases: {path: cases.yml} +execution: {path: execution.yml} +analysis: {path: analysis.yml} +``` -Use this **recommended layout** when preparing a benchmark. The names below -are examples: replace `rho-agent`, `workspace`, task IDs, and mutant IDs with -those of the case being packaged. Keep agents under `agents/`, task collections -under `tasks/`, and case combinations in `experiment.yml`. Keep `subsets.json` beside -its `tasks.yml` and each agent version in its own directory. +All five fields are required. `benchmark_id` is a nonempty string; `revision` is a +positive integer. The three references must name files. They are relative to the +experiment file's directory. Signed experiments include a `sha256` on each reference. +The experiment's identity and revision describe the composition; they do not +invalidate execution reuse when only analysis changes. -The loader enforces the manifest relationships and reference rules below, not -these directory names. Other layouts are accepted when their references satisfy -the same contract. -The directory containing `experiment.yml` is the bundle root: +## Cases: `cases.yml` -```text -benchmark/ - experiment.yml - agents/rho-agent/ - agent.yml - description.json - baseline/ - ... agent files and dependencies - mutants/ - mutant/ - ... mutant agent files and dependencies - mutant-no-search/ - ... mutant agent files and dependencies - tasks/workspace/ - tasks.yml - subsets.json - inspect-files/ - ... Harbor task files, environment and verifier - read-config/ - ... Harbor task files, environment and verifier - search-workspace/ - ... Harbor task files, environment and verifier +```yaml +- case_id: rho-agent + agent: {path: agents/rho-agent/agent.yml} + tasks: {path: tasks/workspace/tasks.yml} ``` -## References and content pins +The document is a nonempty list, with unique case IDs. Every case selects an agent +manifest and a task manifest; there is no `inputs` wrapper and no analysis settings +in this file. Pairing the same agent with a different task collection uses another +case ID. Shared manifests need not be copied. -Every reference requires `path`. `sha256` (`sha256:`) is optional, as is -`kind` (`file` by default, or `directory`). **All paths are relative to the bundle root**, -including references inside nested manifests. Cases can share agent, task and -subset manifests without copying them or using `../`. Absolute paths, traversal, -and symlinks in reference paths are rejected. When supplied, file digests verify -exact bytes. -Directory digests hash the canonical JSON mapping of relative file paths to -SHA-256 digests using `input_digest(path)`. Moving the bundle preserves its -references and identity. Task membership is explicit; adding a directory does -not add a task. - -Use `/`-separated relative paths without `..`, backslashes, colons, or NUL -characters. The referenced file or directory must exist when verified and match -its declared `kind`. File references use `kind: file` or omit `kind`; task -sources are always directories and have no `kind` field in `tasks.yml`. Agent -inputs may use either kind. - -Omit `sha256` while iterating and load with `allow_unsigned=True` in Python. -The default is `False`: unsigned inputs fail loading. With the opt-in, the loader -still checks that the referenced path exists and has the expected kind, but does -not hash unpinned contents. When `sha256` is supplied, a mismatch is an error. A -digest is `sha256:` followed by 64 lowercase hexadecimal characters. - -For an established benchmark, supply digests on every reference, including nested -manifests and agent/task inputs, to fix the expected contents. These are content -pins, not cryptographic signatures identifying a publisher. The experiment's -configuration digest (`benchmark_sha256` in downstream artifacts) can remain -unchanged when unsigned inputs change. It is not proof that the bundle is -unchanged: require a successful strict `load_benchmark()` or `sha-sign --check` -to verify the referenced contents against all pins. - -For example, a case's `agent` reference in `experiment.yml` points to -`agents/rho-agent/agent.yml`. Its digest -covers that file's exact bytes, including whitespace and comments. - -Directory pins cover relative file names and file contents. They do not cover -permission bits or empty directories. - -Use [`sha-sign`](tools.md#signing-a-benchmark) to write the pins. It hashes -agent/task files, descriptions and subsets first, writes the hashes into the -agent/tasks manifests, then hashes those manifests and updates the inline case -references in `experiment.yml`. The experiment file has no self-hash. Changing -an input requires re-signing its referencing manifests; revision labels do not -replace content verification. For routine updates after editing a signed bundle, -use `sha-sign experiment.yml --resign`. Signing changes only SHA fields and -preserves comments and other formatting, so the diff shows which pins changed. -Use plain or quoted SHA scalar values; block-scalar, anchored and tagged SHA -values are not supported for rewriting. - -Use `sha-sign experiment.yml --check` to require and verify every pin without -writing files. Checking follows the normal strict `load_benchmark()` path and -stops at the first validation error. Only explicit unsigned loading allows -missing pins. The [tool guide](tools.md#signing-a-benchmark) provides CLI and -library usage. +The directory containing `cases.yml` is the input bundle root. All references in +cases, agent/task manifests, and analysis case settings resolve from that root, +regardless of the settings file's location. The experiment's three file references +are resolved from its own directory. This gives standalone stage commands the +same interpretation of paths as the orchestrator. ## Agents: `agent.yml` -This file defines `baseline`, `mutants`, and a `description` reference. Each agent -version supplies `agent_bundle_id`, `revision`, native `harbor` settings, and -optional `inputs` references for its prepared files and dependencies. In the example -layout above, these inputs reference `agents/rho-agent/baseline/` or the -corresponding directory under `agents/rho-agent/mutants/`, with `kind: directory` -and, when pinning the benchmark, that directory's SHA-256 digest. Listing those -files does not install or launch them; the selected Harbor integration must support doing so. - -`baseline`, `mutants`, and `description` are required. There must be at least -one mutant, and the baseline and mutants must have distinct `agent_bundle_id` -values within a case. An agent `revision` is a nonempty **string**, so quote -numeric labels such as `'1'`. `inputs` defaults to an empty mapping; built-in -agents can use this when they have no prepared files to reference. - -The agent is the executable attempting the task; the task owns its verifier. -The `harbor` field configures how the local executor invokes that agent through -a Harbor integration (`name` or `import_path`, plus native settings). Prepared -agent contents and dependencies are separate input references. - -The agent's description is a JSON object describing intended behavior for the -coverage tool. The runner passes it through as `case_description`; the configured -metric tool determines how to interpret it. An Ethos-derived description is one -possible input. The synthetic fixture's `required` list is only a test convention, -not a required description schema. +```yaml +baseline: + agent_bundle_id: baseline + revision: '1' + harbor: {name: oracle} +mutants: + - agent_bundle_id: mutant + revision: '1' + harbor: {name: nop} +``` -## Tasks: `tasks.yml` +These built-in agents illustrate the format, not a real mutation experiment. +Each version has a distinct nonempty `agent_bundle_id`, a string `revision`, native +`harbor` settings, and optional `inputs` references to files or directories. A real +agent's Harbor integration can reference a named dependency using `{input:NAME}` +in its settings. References describe prepared dependencies; they do not install them. +The selected integration must support loading them. There must be at least one mutant. +The intended-behavior description belongs in analysis settings, not this manifest. -`tasks.yml` contains one nonempty `tasks` list. Each task specifies its identity -and source together: +## Tasks: `tasks.yml` ```yaml tasks: @@ -164,127 +86,121 @@ tasks: path: tasks/workspace/read-config ``` -Each entry requires a unique `task_id`, a positive integer `revision`, and a -bundle-relative `path` to a Harbor task directory. `name` and `sha256` are -optional. `sha-sign` adds the SHA field. There is no `kind` field: all task -sources are directories. The loader derives the task inventory used by scoring -and execution results from this list; authors do not repeat it in the YAML. - -Task IDs are stable labels. The example uses readable names matching the task -directories, but each task's explicit `path` determines its location. Each -Harbor task directory contains `task.toml`, `instruction.md`, environment files -and a verifier. The task owns the verifier; the agent attempts the task. - -This list can package an existing task benchmark. Archives, Git repositories -and remote datasets must be materialized as local directories during prep; -the runner does not download or unpack them. There is no additional collection -benchmark ID, deployment, or revision block. - -Generated tasks use this same format. A preparation tool can write `tasks.yml` -or `tasks.generated.yml`, materialize its task directories, and point the case's -`tasks.path` at that manifest. Its subsets reference those task IDs. Generation -needs no separate schema or `generated_tasks` field. - -## Subsets: `subsets.json` - -The subset manifest sits beside `tasks.yml` and contains -`artifact_type: eval_subset_definitions`, `sampling_parameters` (including the -preparation seed), and `eval_subsets`. Each subset has an `eval_subset_id` (a -readable name or number) and a `task_ids` list. It does not repeat task revisions, -display names, or the task inventory. The selected `tasks.yml` supplies those -task definitions; analysis checks that recorded task IDs and revisions match it. -Unknown task IDs, duplicate subset IDs, and repeated task IDs within a subset -are rejected. Identical membership across different subsets is allowed; choosing -which subsets belong in the benchmark is a preparation-time decision. -Cases and experiments can share prepared subsets. Membership stays fixed; -runtime never samples from the seed. - -`sampling_parameters` and `eval_subsets` are required; `artifact_type` may be -omitted. Include a fixed integer `seed` in `sampling_parameters` as the prep -convention (the loader does not require that specific key). Other sampling -metadata values must be integers or strings. There must be at least one subset, -and each subset must contain at least one task. Subset IDs are strings, including -numbered labels such as `"subset-000001"`. Subsets may overlap. - -## Experiment and inline cases: `experiment.yml` - -The experiment defines its cases directly. Each case requires a unique `case_id` -and `agent`, `tasks`, and `subsets` file references. There is no separate -`case.yml`. Cases can reuse manifests while choosing different combinations of -agents, tasks and subsets. - -This unsigned example shows the full shape. The coverage command is illustrative; -replace it with the installed metric tool's command: +The task list is nonempty. IDs are unique, revisions are positive integers, and +`name` and `sha256` are optional. Each path identifies a Harbor task directory, +including its environment and verifier. Task sources are always directories; +there is no `kind` field on these entries. The inventory is derived from this +manifest and is not repeated in the experiment. Adding a directory does not add +a task. Preparation must materialize archives or remote datasets first. + +## Execution: `execution.yml` ```yaml -benchmark_id: rho-metric-validity -revision: 1 -cases: - - case_id: rho-agent - agent: - path: agents/rho-agent/agent.yml - tasks: - path: tasks/workspace/tasks.yml - subsets: - path: tasks/workspace/subsets.json +n_concurrent_trials: 4 +environment: + type: docker +``` + +An empty mapping selects defaults: `backend: local_harbor`, `harbor_version: 0.20.0`, +`n_concurrent_trials: 1`, and Docker. No other backend or Harbor version is currently +supported. The environment mapping accepts native Harbor environment settings. +Execution always runs each case's baseline then mutants against its complete task +inventory, with one attempt per pair and no automatic retries. That matrix is +already defined by the cases and is not configured a second time. + +## Analysis: `analysis.yml` + +```yaml +case_settings: + rho-agent: + description: {path: agents/rho-agent/description.json} + subsets: {path: tasks/workspace/subsets.json} coverage: system_id: eval-author system_revision: '1' command: [python, /path/to/coverage-adapter.py] +scoring: + metric: reward + direction: higher_is_better +missing_policy: error ``` -When the same agent is paired with two task collections, give each combination -a distinct case ID. `sha-sign` adds SHA fields beside each reference's `path`; -there is no SHA on the inline case itself. - -| Field | Requirement or default | -| --- | --- | -| `benchmark_id` | Required nonempty string identifying the metric benchmark. | -| `revision` | Required positive integer identifying this experiment definition. | -| `cases` | Required nonempty list of inline cases, with distinct case IDs. | -| `coverage` | Required metric-tool configuration, described below. | -| `execution` | Optional; defaults to the local Harbor settings below. | -| `scoring` | Optional; defaults to the policy below. | -| `missing_policy` | `error` (default) or `exclude` for analysis. | - -`coverage` requires `system_id`, `system_revision` (both nonempty strings), and -`command` (a nonempty list of argument strings). Optional `settings` is a JSON -object, default `{}`; `timeout_seconds` is a positive integer, default `3600`. -The command identifies the metric tool to invoke, not a prepared input reference. -The bundle-relative path rules apply to input references; they do not rewrite -arbitrary command arguments or native Harbor settings. - -The default `scoring` policy is `metric: reward`, -`direction: higher_is_better`, `completed_statuses: [succeeded]`, and -`attempt_policy: only`. The accepted alternatives are `lower_is_better` for -`direction` and `latest` for `attempt_policy`. `completed_statuses` is a nonempty -list of status strings. These are analysis settings; the local execution stage -runs one trial per agent/task pair. - -The initial backend is `local_harbor`, pinned to Harbor 0.20.0. Its settings are -`n_concurrent_trials` (default 1) and native `environment` configuration (Docker -by default). NeMo Helix is a future backend. Native Harbor settings and coverage -commands are trusted local programs, not a sandbox or secret store. Supply -credentials through the runtime environment, not benchmark YAML committed to Git. - -## Validation and examples - -Manifest fields have strict types: unknown fields, duplicate YAML or JSON keys, -and non-finite JSON numbers are rejected. Native `harbor`, `coverage.settings`, -and `execution.environment` values must be JSON-compatible; quote YAML date-like -strings. There is no `schema_version` field. Revisions identify experiment, -agent, task, or metric-tool inputs rather than versions of this file format. - -`load_benchmark()` reads the experiment and its inline cases, verifies the referenced manifests and -all prepared inputs (requiring and checking digests by default), and loads subsets and -descriptions. It checks subset task IDs against the selected task inventory. It does not execute the agents, verify -that a launch integration works, or validate the metric-specific meaning of the -description. See [Reading benchmarks](tools.md#reading-benchmarks) for the API. - -For concrete YAML and JSON syntax, start with the -[example experiment](../../../tests/fixtures/metric_benchmark/experiment.yml) -and follow its manifest references. This is a synthetic configuration fixture: -its Oracle baseline and two NOP mutants have no prepared agent directories, -its task files are supplied by the tests, and its coverage command is a -placeholder. A released bundle using the recommended layout includes the -referenced task and agent files and a working launch configuration. +`case_settings` must contain exactly the configured case IDs. Each entry references +a description JSON object and a prepared subset manifest. The coverage tool defines +the description's contents; an Ethos-derived description is one possible input. +The coverage command above is illustrative and must name an installed tool. + +Coverage requires `system_id`, `system_revision`, and a nonempty `command` argument +list. Optional `settings` is a JSON object and `timeout_seconds` defaults to 3600. +Scoring defaults to `metric: reward`, `direction: higher_is_better`, +`completed_statuses: [succeeded]`, and `attempt_policy: only`. Alternatives include +`lower_is_better` and `latest` (select the highest attempt, not the best reward). +`missing_policy` is `error` by default or `exclude`. Analysis selects from recorded +measurements; execution preserves all native reward metrics independently of this policy. + +### Subsets: `subsets.json` + +A subset manifest contains `artifact_type: eval_subset_definitions` (optional), +`sampling_parameters`, and a nonempty `eval_subsets` list. Each entry supplies a +unique string `eval_subset_id` and nonempty `task_ids`. Task IDs come from the selected +task manifest; unknown or repeated IDs within one subset are rejected. Subsets may +overlap, and different subset IDs may have identical memberships. + +Preparation records a fixed integer seed in `sampling_parameters`; other metadata +values may be integers or strings. Runtime consumes the explicit memberships and +never samples from the seed. Analysis checks the recorded task identities and revisions +against the case metadata. + +## References and content pins + +References require `path`, optionally `sha256` (`sha256:` plus 64 lowercase hex digits) +and `kind` (`file` by default, or `directory` for agent dependencies). Use relative +POSIX paths without `..`, backslashes, colons, or NUL characters. Absolute paths, +symlinks, and special files are rejected. Settings and manifest references name files. + +File pins cover exact bytes. Directory pins hash a canonical map of relative file +paths to content digests; permission bits and empty directories are not covered. +Use [`sha-sign`](sha-sign.md) to add or update pins from leaves to the experiment. +It preserves comments and formatting while editing SHA fields. Pins identify content, +not a publisher. Omit pins while iterating and pass `--allow-unsigned`; supplied +pins are still verified. `sha-sign --check` requires and checks every reference. + +## Stage boundaries and reuse + +`load_cases` verifies the case and agent/task metadata without opening task or +agent files. `load_execution` additionally validates executable inputs. +`load_analysis` reads case metadata and analysis settings; its artifact loaders +open subsets or descriptions only when needed. `load_benchmark` composes stage +objects without executing or requiring executable inputs. Full bundle verification +is provided by `sha-sign --check`. + +Execution identity covers the resolved case definitions, executable content pins +(or actual contents during unsigned iteration), and effective execution settings. +It excludes analysis settings, descriptions, subsets, and experiment labels/revision. +Signing unchanged inputs preserves both execution and case identities, so saved +jobs remain reusable and their results remain valid. Unsigned case identities +require executable inputs to compute missing content digests; sign the bundle +before removing those inputs for analysis elsewhere. +Reformatting an unsigned manifest alone does not change its execution meaning. +Analysis records its settings and inputs alongside the selected execution evidence. +Use a new analysis output directory for a new hypothesis; saved executions remain +unchanged. Analysis can choose another recorded reward metric without rerunning tasks. + +Fields have strict types; unknown fields, duplicate YAML/JSON keys, non-finite +numbers, and unquoted YAML dates in native settings are rejected. Credentials belong +in the runtime environment, not in committed configuration. Native Harbor settings +and coverage commands run trusted local code. + +## Examples and migration + +The [example experiment](../../../tests/fixtures/metric_benchmark/experiment.yml) +links all three files. It is a synthetic configuration fixture; tests supply its +local task sources and its coverage command is a placeholder. Released benchmarks +must include executable inputs and a working metric command. + +This replaces the experimental inline-case format. Move the old experiment's +`cases` list into `cases.yml`, move `execution` into `execution.yml`, and move +coverage/scoring/missing policy into `analysis.yml`. Move each case's subsets and +its agent manifest's description into `analysis.case_settings[case_id]`, then sign +again. Existing execution artifacts using `benchmark_sha256` are not silently +converted; preserve them separately or explicitly migrate and validate their provenance. diff --git a/labs/metric-benchmark/docs/results-format.md b/labs/metric-benchmark/docs/results-format.md index cdde69ac..63083c34 100644 --- a/labs/metric-benchmark/docs/results-format.md +++ b/labs/metric-benchmark/docs/results-format.md @@ -76,13 +76,14 @@ are storage labels, not case identities. | Field | Required / default | Meaning | Source | | --- | --- | --- | --- | | `run_id` | Required | Execution ID shared by all case indexes. | CLI: the runner’s `--run-id`. | -| `benchmark_sha256` | Required | Benchmark identity recorded by the execution runner. | Runner-generated SHA-256 digest. | -| `cases` | Required, nonempty | Map from configured case IDs to relative `results.json` paths. | YAML: `experiment.yml` → `cases[].case_id`; paths are runner-generated. | +| `cases_sha256` | Required | Resolved case metadata identity, checked by analysis without executable files. | Runner-generated digest of the case definitions. | +| `execution_sha256` | Required | Execution-only identity recorded by the runner. | Runner-generated SHA-256 digest. | +| `cases` | Required, nonempty | Map from configured case IDs to relative `results.json` paths. | YAML: `cases.yml` → each `case_id`; paths are runner-generated. | For example, `cases` may contain `{"rho-agent": "cases/case-0000/execution-results/results.json"}`. The validator uses this mapping and checks the supplied prepared inputs; it does not recompute -the runner’s `benchmark_sha256` identity. +the runner’s `execution_sha256` identity. ### Results index: `results.json` @@ -94,7 +95,7 @@ also show their explicit defaults. | Field | Required / default | Meaning | Source | | --- | --- | --- | --- | | `run_id` | Required | ID supplied by the caller when starting this execution of the experiment; the local runner takes it from `--run-id`. | CLI: `--run-id`. | -| `task_inventory` | Required, nonempty | Array of tasks configured for this case example, including tasks whose executions failed or are missing. | YAML: `tasks.yml` → `tasks`, referenced by `experiment.yml` → `cases[].tasks.path`. | +| `task_inventory` | Required, nonempty | Array of tasks configured for this case example, including tasks whose executions failed or are missing. | YAML: `tasks.yml` → `tasks`, referenced by `cases.yml` → `[].tasks.path`. | | `execution_results` | Required, nonempty | Outcomes for each agent/task pair: one entry per attempt, including explicit entries for missing executions. | Runner-generated from the configured agents/tasks and Harbor trial results. | For example, starting the runner with `--run-id trial-001` produces @@ -136,7 +137,7 @@ case’s results index; consumers do not need to parse their components. | `framework_profile_id` | Defaults to `null` | Optional profile identity; it must stay consistent within a run. | Optional producer metadata; the local runner leaves it `null`. | | `evaluation` | Required | Task and attempt identity, defined below. | Runner-generated identity and attempt, plus a task reference from `tasks.yml`; see below. | | `status` | Required | Native outcome: `succeeded`, `error`, or an explicit `missing` execution. | Harbor: derived from the native trial outcome. A producer explicitly declares `missing`. | -| `metrics` | Defaults to `{}` in the model | Map from measurement names to score objects. Validation requires the experiment's configured metric to be present. | YAML: `experiment.yml` → `scoring.metric` (default `reward`) selects the key. Harbor: `result.json` → `verifier_result.rewards` supplies the value; the runner derives unavailable reasons. | +| `metrics` | Defaults to `{}` in the model | Map from all native reward names to score objects; if no rewards exist, a null `reward` records unavailability. | Harbor: `result.json` → `verifier_result.rewards` supplies the value; the runner derives unavailable reasons. | | `native_results_path` | Required | Relative trial-directory path, or a nonexistent location for a missing execution. | Runner-generated relative path to the Harbor trial directory; a producer supplies a placeholder for missing evidence. | | `native_results_format` | Required | Use `directory` for the current native validator. The model also recognizes `tar.gz`, but observed archives are not supported by this validator. | Runner-generated; the local runner sets `directory`. | | `unavailable_evidence` | Defaults to `{}` | Map from evidence names to reasons. Missing executions require a `native_results` reason. | Optional producer-supplied reasons; the local runner leaves this empty and rejects incomplete jobs. | @@ -163,7 +164,8 @@ A numeric value must be finite and have no non-null reason. An unavailable value must be `null` and have a nonblank reason. Zero and negative numbers are valid measurements; neither means that evidence is absent. -For the configured metric, the index must agree with native Harbor observations: +For every native reward metric, the index must agree with native Harbor observations. +When no rewards are present, a null `reward` entry records unavailability: | Situation | Status | Metric value and reason | | --- | --- | --- | @@ -204,10 +206,10 @@ Every `(evaluation_id, execution)` pair is unique across the index. Reusing an one agent/profile combination; different agents cannot share it. One run per agent and one logical `evaluation_id` per agent/task pair are -allowed. The experiment's `scoring.attempt_policy` controls retries: +allowed. Analysis's `scoring.attempt_policy` controls which recorded attempts it accepts: - `only` (default): exactly one execution entry per pair. -- `latest`: multiple attempts of the same evaluation are permitted. Validation checks all supplied attempts; downstream scoring selects the highest attempt number, not the best reward. +- `latest`: multiple attempts of the same evaluation are permitted. Native validation checks all supplied attempts; downstream scoring selects the highest attempt number, not the best reward. Independent repetitions with different evaluation IDs for the same pair are ambiguous and rejected. Native trial evidence cannot be reused by multiple @@ -222,13 +224,13 @@ relative POSIX path within that case directory. Keep the configuration, results, available agent/verifier evidence; symlinks and special files are not allowed. The validator reads each observed trial’s Harbor 0.20.0 `result.json` and checks the task identity/checksum, trial directory name, agent configuration, status and -configured measurement against the prepared inputs and results index. +all native measurements against the prepared inputs and results index. A missing execution still occupies its agent/task position in the index, and its `native_results_path` must not exist. An unfinished native trial is rejected; it cannot be presented as a completed observation. Multi-step trials without an execution exception are rejected because no aggregation policy is implemented. -Validation compares the configured metric, including its reason, against the +Validation compares every indexed metric, including its reason, against the native observation. Other metric keys are schema-checked but are not compared against native measurements by this validator. diff --git a/labs/metric-benchmark/docs/run-evals.md b/labs/metric-benchmark/docs/run-evals.md index 65dac2f6..e4780bfc 100644 --- a/labs/metric-benchmark/docs/run-evals.md +++ b/labs/metric-benchmark/docs/run-evals.md @@ -9,17 +9,17 @@ Run each prepared case’s baseline and mutants against its Harbor task inventor ```bash uv run python labs/metric-benchmark/run_evals.py \ - --benchmark experiment.yml --run-id trial-1 --output private-run + --cases cases.yml --execution execution.yml --run-id trial-1 --output private-run ``` -Supply `--benchmark` for the benchmark YAML, `--run-id` for this execution's +Supply `--cases` for case definitions, `--execution` for Harbor settings, `--run-id` for this execution's identifier, and `--output` for the experiment results directory. Run from the repository root after `uv sync --locked`; the default execution environment requires Docker. Prepared agents may have additional runtime or credential requirements defined by their benchmark bundle. -The benchmark YAML is the command's only configuration file. Its referenced -manifests supply the agents and tasks. All inputs require SHA pins by default; +The cases file references manifests supplying the agents and tasks. Execution +settings do not reference cases or analysis. All inputs require SHA pins by default; add `--allow-unsigned` while iterating to permit missing pins, while still verifying any supplied pins. @@ -51,7 +51,7 @@ private-run/ Numeric folder aliases are stable within a definition. The index carries actual case, agent, task and run identities. Each case index has `run_id` from -`--run-id`, a `task_inventory` array, and an `execution_results` array. Each observation has an +`--run-id`, a `task_inventory` array, and an `execution_results` array. Each observation retains all native reward metrics and has an `agent_run_id` for its agent’s job; `native_results_path` locates the trial evidence. See the [results format](results-format.md) for the complete contract. No model or private service is involved in @@ -61,9 +61,9 @@ the orchestration itself. Prepared agents may call models when executed. Rerunning the command verifies completed job receipts and skips those jobs. Completed outputs and prepared inputs can be moved together. Reuse currently -requires the entire benchmark configuration to match, including analysis settings -and subset manifests, as well as the run ID and prepared inputs. Changing an -analysis-only setting therefore also requires a fresh execution output directory. +requires the execution inputs, effective execution settings, and run ID to match. +Analysis settings, subsets, descriptions, and experiment revision may change +without invalidating completed jobs. With `--allow-unsigned`, the runner hashes inputs without pins when computing this identity, so editing an unsigned bundle also requires a fresh output directory. This does not sign or rewrite the prepared manifests. The retained @@ -95,9 +95,9 @@ changing its evidence makes reuse fail. ## Python library -`execute_benchmark(loaded_benchmark, run_id, output)` runs the same stage as the -CLI and returns the experiment manifest. Obtain `loaded_benchmark` with -[`load_benchmark`](tools.md#reading-benchmarks). A future NeMo Helix implementation +`execute_benchmark(loaded_execution, run_id, output)` runs the same stage as the +CLI and returns the experiment manifest. Obtain `loaded_execution` with +[`load_execution`](tools.md#reading-benchmarks). A future NeMo Helix implementation can produce the same [results contract](results-format.md). ## Local smoke test diff --git a/labs/metric-benchmark/docs/sha-sign.md b/labs/metric-benchmark/docs/sha-sign.md index e795a1fa..d029ff54 100644 --- a/labs/metric-benchmark/docs/sha-sign.md +++ b/labs/metric-benchmark/docs/sha-sign.md @@ -15,8 +15,8 @@ uv run labs/metric-benchmark/tools/sha-sign benchmark/experiment.yml `sha-sign` reads the experiment and walks every case's referenced manifests and inputs. It checks paths, file types, schemas and subset task IDs before writing anything. It adds `sha256` to task sources, agent inputs, descriptions, subsets, -and agent/tasks manifest references, working from input files up to the -experiment. Shared manifests are updated once. It does not run agents or the +agent/task manifest references, and cases/execution/analysis references, working +from input files up to the experiment. Shared manifests are updated once. It does not run agents or the metric tool, and it does not modify task files, agent files or JSON inputs. If any reference already has a SHA field, signing stops without writing files @@ -37,8 +37,8 @@ manifests that the signer itself rewrites, since doing so would invalidate their hashes. Once signed, readers verify each supplied digest as they load its reference. -`load_benchmark()` verifies all inputs; `read_benchmark()` verifies the manifests -it reads, and individual artifact loaders verify their own inputs. Omitting a +`load_cases()` verifies metadata; stage loaders and artifact loaders verify the +inputs they consume. `check_benchmark()` verifies every prepared input. Omitting a digest requires explicit `allow_unsigned=True` when loading during iteration. SHA signing fixes expected contents; it does not authenticate a publisher. @@ -51,12 +51,12 @@ uv run labs/metric-benchmark/tools/sha-sign benchmark/experiment.yml --check `--check` requires a SHA on every input reference and verifies it against the current contents. It stops at the first validation error, including a missing, null or incorrect digest, and reports the input context. It uses the same strict -`load_benchmark()` path as normal full loading. To fill missing pins throughout +`check_benchmark()` path as full bundle verification. To fill missing pins throughout an unsigned bundle, run signing without `--check`; use `--resign` if some pins already exist. Success exits with status zero; validation failures exit nonzero. `--check` and `--resign` are mutually exclusive. -The top-level experiment has no self-hash, and inline cases have no separate -hash: their agent, tasks and subsets references are checked instead. +The top-level experiment has no self-hash; it pins the three configuration files. +The cases and analysis files pin their own referenced inputs. To sign or check from Python: diff --git a/labs/metric-benchmark/docs/tools.md b/labs/metric-benchmark/docs/tools.md index 26b76518..c3cf68d4 100644 --- a/labs/metric-benchmark/docs/tools.md +++ b/labs/metric-benchmark/docs/tools.md @@ -13,63 +13,41 @@ These tools consume externally prepared cases. A prepared case contains: ## Benchmark config -The [benchmark format specification](benchmark-format.md) defines the recommended -directory layout, manifest fields, relative paths, and content pins: - -- [Agents](benchmark-format.md#agents-agentyml): baseline, mutants and intended behavior. -- [Tasks](benchmark-format.md#tasks-tasksyml): task inventory and source directories. -- [Subsets](benchmark-format.md#subsets-subsetsjson): fixed task memberships. -- [Cases](benchmark-format.md#experiment-and-inline-cases-experimentyml): inline combinations of agents, tasks and subsets. -- [Experiment](benchmark-format.md#experiment-and-inline-cases-experimentyml): case list and execution/analysis settings. +The [format specification](benchmark-format.md) defines the experiment's three +references: [cases](benchmark-format.md#cases-casesyml), +[execution settings](benchmark-format.md#execution-executionyml), and +[analysis settings](benchmark-format.md#analysis-analysisyml). ## Signing a benchmark -See the [`sha-sign` command guide](sha-sign.md) for signing, re-signing, checking -pins, and the Python API. +See [`sha-sign`](sha-sign.md) for signing, re-signing, checking pins, and the Python API. ## Reading benchmarks -Unpack a benchmark bundle and pass its `experiment.yml` to the Python library. -From the repository root: +From the repository root, put `labs/metric-benchmark` on `PYTHONPATH` to import the +libraries. Standalone stage loaders take the cases file and their settings file: -```bash -PYTHONPATH=labs/metric-benchmark uv run python - <<'PYTHON' +```python from pathlib import Path -from metric_benchmark.config import load_benchmark - -benchmark = load_benchmark(Path("benchmark/experiment.yml")) -prepared = benchmark.case("rho-agent") -subsets = benchmark.subsets["rho-agent"] -PYTHON -``` - -`load_benchmark(path)` is the full entry point: it reads the experiment, verifies -all referenced manifests, task sources and agent inputs, and loads every case's -subsets and description. The returned object holds the experiment definition, -resolved cases, subsets, and descriptions. Loading requires SHA-256 digests by -default. During iteration, explicitly opt in to unsigned inputs: +from metric_benchmark.config import load_execution, load_analysis, load_benchmark -```python -benchmark = load_benchmark(Path("benchmark/experiment.yml"), allow_unsigned=True) +execution = load_execution(Path("benchmark/cases.yml"), Path("benchmark/execution.yml")) +analysis = load_analysis(Path("benchmark/cases.yml"), Path("benchmark/analysis.yml")) +subsets = analysis.load_subsets("rho-agent") +experiment = load_benchmark(Path("benchmark/experiment.yml")) ``` -This permits missing pins but still verifies every supplied digest. The lighter -`read_benchmark()` and individual artifact loaders accept the same keyword, -also defaulting to `False`. Benchmark-reading CLIs accept `--allow-unsigned` -for iteration; `sha-sign --check` always requires all pins. - -Use `read_benchmark(path)` from the same module when only the experiment and -inline cases and referenced agent/task manifests are needed. It does not open -task sources, agent bundles, subsets, or descriptions. With this lighter reader, -load a selected case's subsets or description through -`prepared.case.load_subsets(prepared.root, allow_unsigned=prepared.allow_unsigned)` or -`prepared.case.load_description(prepared.root, allow_unsigned=prepared.allow_unsigned)`; -each verifies the referenced artifact and requires a digest unless unsigned -loading was explicitly enabled. - -The [format specification](benchmark-format.md) describes validation rules. -`BenchmarkDefinition`, `CaseDefinition`, `AgentManifest`, and `TaskManifest` also -expose their format through `model_json_schema()`. +`load_execution` verifies executable inputs. `load_analysis` reads only metadata +and settings; subset and description loaders verify the artifacts they read. +`load_cases(path)` reads case/agent/task metadata for full native result validation. +`load_benchmark` composes execution and analysis objects for orchestration; execution +verifies its own inputs before launching jobs. It does not eagerly open executable +inputs, subsets, or descriptions. Use `sha-sign --check` to verify the entire bundle. + +Loading requires pins by default. All these entrypoints accept +`allow_unsigned=True` during iteration, still verifying supplied pins. The CLI +spelling is `--allow-unsigned`. Metadata loading requires the referenced YAML +manifests, but analysis does not require their executable agent/task directories. ## Validate a results directory diff --git a/labs/metric-benchmark/docs/validate-results.md b/labs/metric-benchmark/docs/validate-results.md index e34efd31..f180da0b 100644 --- a/labs/metric-benchmark/docs/validate-results.md +++ b/labs/metric-benchmark/docs/validate-results.md @@ -12,14 +12,14 @@ From the repository root, invoke the validation CLI with: ```bash uv run python labs/metric-benchmark/validate_results.py \ - --benchmark experiment.yml \ + --cases cases.yml \ --results-root experiment-results \ --output validation.json ``` Supply these required arguments: -- `--benchmark`: path to the experiment YAML file and its references to prepared inputs. +- `--cases`: path to the case definitions and their prepared agent/task manifests. - `--results-root`: whole experiment output directory containing `execution.json` and the case results directories. - `--output`: path for the validation report, outside the results root; `validation.json` is an example filename. @@ -39,10 +39,10 @@ contents are identical; choose a new output path for changed results. ## Python library `validate_experiment_results(loaded, root, *, case_id=None)` returns the -same report as the CLI. Pass a `LoadedBenchmark` from `load_benchmark` and the +same report as the CLI. Pass `LoadedCases` from `load_cases` and the whole experiment output root; `case_id` optionally selects one case. -The per-case `validate_results(raw, case, root, input_root, scoring, *, allow_unsigned=False)` +The per-case `validate_results(raw, case, root, input_root, *, allow_unsigned=False)` function remains available for execution and analysis libraries. Its `root` is that case’s results directory, and it returns the individual case report. diff --git a/labs/metric-benchmark/docs/validation-report.md b/labs/metric-benchmark/docs/validation-report.md index f4700d11..1f78e349 100644 --- a/labs/metric-benchmark/docs/validation-report.md +++ b/labs/metric-benchmark/docs/validation-report.md @@ -32,11 +32,11 @@ Each entry in `cases` contains: | Field | Meaning | Source | | --- | --- | --- | -| `case_id` | Selected prepared case. | YAML: `experiment.yml` → `cases[].case_id`, optionally selected by the validator’s `--case` flag. | +| `case_id` | Selected prepared case. | YAML: `cases.yml` → `[].case_id`, optionally selected by the validator’s `--case` flag. | | `run_id` | Execution identity copied from the index. | Copied from `results.json` → `run_id`, originally supplied by the execution CLI’s `--run-id`. | | `expected_pairs` | Number of configured agents multiplied by number of tasks. | Validator-generated from the selected case’s `agent.yml` and `tasks.yml` manifests. | | `observations` | Number of execution entries, including missing entries and retries. | Validator-generated count of `results.json` → `execution_results`. | -| `unavailable` | Entries with null configured measurements, each containing `agent`, `task`, `attempt`, and `reason`. | Validator-generated from execution entries whose configured metric value is `null`. | +| `unavailable` | Entries with null native measurements, each containing `agent`, `task`, `attempt`, `metric`, and `reason`. | Validator-generated from execution entries whose native metric value is `null`. | | `files` | Map of relative evidence-file paths to `sha256:` digests. | Validator-generated by hashing files under that case’s results directory. | Each case’s file manifest covers ordinary files under its results directory diff --git a/labs/metric-benchmark/metric_benchmark/config.py b/labs/metric-benchmark/metric_benchmark/config.py index e3755916..caf180d7 100644 --- a/labs/metric-benchmark/metric_benchmark/config.py +++ b/labs/metric-benchmark/metric_benchmark/config.py @@ -3,12 +3,12 @@ """Read a versioned benchmark definition and verify its prepared local inputs.""" -from dataclasses import dataclass, field +from dataclasses import dataclass from pathlib import Path from typing import Literal, Self import yaml -from pydantic import Field, JsonValue, model_validator +from pydantic import Field, JsonValue, TypeAdapter, model_validator from metric_benchmark.io import digest_bytes, json_bytes, read_json, verify_input from metric_benchmark.io import file_manifest as file_manifest @@ -88,7 +88,6 @@ class CoverageConfig(Model): class AgentManifest(Model): baseline: Agent mutants: list[Agent] = Field(min_length=1) - description: PinnedInput class TaskSource(Task): @@ -120,12 +119,11 @@ class CaseDefinition(Model): case_id: Text agent: PinnedInput tasks: PinnedInput - subsets: PinnedInput @model_validator(mode="after") def file_references(self) -> Self: - if any(source.kind != "file" for source in (self.agent, self.tasks, self.subsets)): - raise ValueError("agent, tasks and subsets must reference manifest files") + if any(source.kind != "file" for source in (self.agent, self.tasks)): + raise ValueError("agent and tasks must reference manifest files") return self @@ -133,8 +131,6 @@ class Case(Model): case_id: Text task_inventory: Tasks task_sources: dict[Text, PinnedInput] - case_description: PinnedInput - subsets: PinnedInput baseline: Agent mutants: list[Agent] = Field(min_length=1) @@ -142,10 +138,8 @@ class Case(Model): def from_manifests(cls, definition: CaseDefinition, agent: AgentManifest, tasks: TaskManifest) -> Self: return cls( case_id=definition.case_id, - subsets=definition.subsets, baseline=agent.baseline, mutants=agent.mutants, - case_description=agent.description, task_inventory=tasks.task_inventory, task_sources=tasks.task_sources, ) @@ -160,27 +154,8 @@ def consistent(self) -> Self: raise ValueError("task_sources must match the task inventory") if any(source.kind != "directory" for source in self.task_sources.values()): raise ValueError("Harbor task sources must be directories") - if self.subsets.kind != "file" or self.case_description.kind != "file": - raise ValueError("subsets and case description must be files") return self - def load_subsets(self, root: Path, *, allow_unsigned: bool = False) -> Subsets: - try: - subsets = Subsets.model_validate(read_json(self.subsets.resolve(root, allow_unsigned=allow_unsigned))) - subsets.resolve_tasks(self.task_inventory) - return subsets - except (OSError, ValueError) as error: - raise ValueError(f"case {self.case_id}, subsets: {error}") from error - - def load_description(self, root: Path, *, allow_unsigned: bool = False) -> dict: - try: - description = read_json(self.case_description.resolve(root, allow_unsigned=allow_unsigned)) - if not isinstance(description, dict): - raise ValueError("case description must be a JSON object") - return description - except (OSError, ValueError) as error: - raise ValueError(f"case {self.case_id}, case_description: {error}") from error - def selection(self, run_ids: dict[str, str], scoring: ScoringPolicy) -> ScoringConfig: return ScoringConfig( **scoring.model_dump(), @@ -198,18 +173,35 @@ class LocalExecution(Model): environment: dict[str, JsonValue] = Field(default_factory=lambda: dict[str, JsonValue](type="docker")) -class BenchmarkDefinition(Model): - benchmark_id: Text - revision: int = Field(ge=1) - cases: list[CaseDefinition] = Field(min_length=1) +class CaseAnalysis(Model): + subsets: PinnedInput + description: PinnedInput + + @model_validator(mode="after") + def file_references(self) -> Self: + if self.subsets.kind != "file" or self.description.kind != "file": + raise ValueError("subsets and description must reference files") + return self + + +class AnalysisSettings(Model): + case_settings: dict[Text, CaseAnalysis] = Field(min_length=1) coverage: CoverageConfig scoring: ScoringPolicy = Field(default_factory=ScoringPolicy) - execution: LocalExecution = Field(default_factory=LocalExecution) missing_policy: Literal["error", "exclude"] = "error" + +class BenchmarkDefinition(Model): + benchmark_id: Text + revision: int = Field(ge=1) + cases: PinnedInput + execution: PinnedInput + analysis: PinnedInput + @model_validator(mode="after") - def unique_cases(self) -> Self: - unique([case.case_id for case in self.cases], "case IDs") + def file_references(self) -> Self: + if any(source.kind != "file" for source in (self.cases, self.execution, self.analysis)): + raise ValueError("cases, execution and analysis must reference files") return self @@ -221,12 +213,9 @@ class LoadedCase: @dataclass(frozen=True) -class LoadedBenchmark: - definition: BenchmarkDefinition - root: Path +class LoadedCases: cases: dict[str, LoadedCase] - subsets: dict[str, Subsets] = field(default_factory=dict) - descriptions: dict[str, dict] = field(default_factory=dict) + root: Path allow_unsigned: bool = False def case(self, case_id: str) -> LoadedCase: @@ -236,7 +225,97 @@ def case(self, case_id: str) -> LoadedCase: @property def identity(self) -> str: - return digest_bytes(json_bytes(self.definition)) + return digest_bytes(json_bytes(self.identity_payload())) + + def identity_payload(self, *, verify: bool = False) -> list[dict]: + """Use content digests whether inputs were signed before or after execution. + + Signed metadata suffices for analysis without executable inputs. Unsigned + sources must still be available to establish their content identity. + """ + cases = [] + for prepared in self.cases.values(): + case = prepared.case.model_dump() + sources = list(case["task_sources"].values()) + sources.extend( + source for agent in [case["baseline"], *case["mutants"]] for source in agent["inputs"].values() + ) + for source in sources: + reference = PinnedInput.model_validate(source) + if verify or reference.sha256 is None: + path = reference.resolve(prepared.root, allow_unsigned=prepared.allow_unsigned) + source["sha256"] = reference.sha256 or input_digest(path) + cases.append(case) + return cases + + def verify_execution_inputs(self) -> None: + for prepared in self.cases.values(): + case = prepared.case + sources = [(f"task {name}", source) for name, source in case.task_sources.items()] + sources.extend( + (f"agent {agent.agent_bundle_id}, input {name}", source) + for agent in case.agents() + for name, source in agent.inputs.items() + ) + for label, source in sources: + try: + source.resolve(self.root, allow_unsigned=self.allow_unsigned) + except (OSError, ValueError) as error: + raise ValueError(f"case {case.case_id}: {label}: {error}") from error + + +@dataclass(frozen=True) +class LoadedExecution: + inputs: LoadedCases + settings: LocalExecution + + +@dataclass(frozen=True) +class LoadedAnalysis: + inputs: LoadedCases + settings: AnalysisSettings + + def load_subsets(self, case_id: str) -> Subsets: + prepared = self.inputs.case(case_id) + try: + source = self.settings.case_settings[case_id].subsets + subsets = Subsets.model_validate( + read_json(source.resolve(self.inputs.root, allow_unsigned=self.inputs.allow_unsigned)) + ) + subsets.resolve_tasks(prepared.case.task_inventory) + return subsets + except (OSError, ValueError) as error: + raise ValueError(f"case {case_id}, subsets: {error}") from error + + def load_description(self, case_id: str) -> dict: + self.inputs.case(case_id) + try: + source = self.settings.case_settings[case_id].description + description = read_json(source.resolve(self.inputs.root, allow_unsigned=self.inputs.allow_unsigned)) + if not isinstance(description, dict): + raise ValueError("case description must be a JSON object") + return description + except (OSError, ValueError) as error: + raise ValueError(f"case {case_id}, case_description: {error}") from error + + @property + def identity(self) -> str: + artifacts = {} + for case_id in self.inputs.cases: + artifacts[case_id] = { + "subsets": self.load_subsets(case_id).model_dump(), + "description": self.load_description(case_id), + } + return digest_bytes( + json_bytes({"cases": self.inputs.identity, "settings": self.settings.model_dump(), "artifacts": artifacts}) + ) + + +@dataclass(frozen=True) +class LoadedBenchmark: + definition: BenchmarkDefinition + execution: LoadedExecution + analysis: LoadedAnalysis def read_yaml(path: Path) -> object: @@ -246,12 +325,15 @@ def read_yaml(path: Path) -> object: raise ValueError(f"invalid YAML in {path.name}: {error}") from error -def read_benchmark(path: Path, *, allow_unsigned: bool = False) -> LoadedBenchmark: - """Read inline cases and referenced agent/task manifests without execution inputs.""" - definition = BenchmarkDefinition.model_validate(read_yaml(path)) +def load_cases(path: Path, *, allow_unsigned: bool = False) -> LoadedCases: + """Read case and agent/task metadata without opening executable inputs.""" + definitions = TypeAdapter(list[CaseDefinition]).validate_python(read_yaml(path)) + if not definitions: + raise ValueError("cases must not be empty") + unique([entry.case_id for entry in definitions], "case IDs") root = path.resolve().parent cases = {} - for entry in definition.cases: + for entry in definitions: try: agent = AgentManifest.model_validate(read_yaml(entry.agent.resolve(root, allow_unsigned=allow_unsigned))) tasks = TaskManifest.model_validate(read_yaml(entry.tasks.resolve(root, allow_unsigned=allow_unsigned))) @@ -259,28 +341,31 @@ def read_benchmark(path: Path, *, allow_unsigned: bool = False) -> LoadedBenchma except (OSError, ValueError) as error: raise ValueError(f"case {entry.case_id}: {error}") from error cases[case.case_id] = LoadedCase(case, root, allow_unsigned=allow_unsigned) - return LoadedBenchmark(definition, root, cases, allow_unsigned=allow_unsigned) + return LoadedCases(cases, root, allow_unsigned) + + +def load_execution(cases: Path, execution: Path, *, allow_unsigned: bool = False) -> LoadedExecution: + inputs = load_cases(cases, allow_unsigned=allow_unsigned) + settings = LocalExecution.model_validate(read_yaml(execution)) + inputs.verify_execution_inputs() + return LoadedExecution(inputs, settings) + + +def load_analysis(cases: Path, analysis: Path, *, allow_unsigned: bool = False) -> LoadedAnalysis: + inputs = load_cases(cases, allow_unsigned=allow_unsigned) + settings = AnalysisSettings.model_validate(read_yaml(analysis)) + if settings.case_settings.keys() != inputs.cases.keys(): + raise ValueError("analysis case_settings must match the cases exactly") + return LoadedAnalysis(inputs, settings) def load_benchmark(path: Path, *, allow_unsigned: bool = False) -> LoadedBenchmark: - """Load and verify all prepared inputs for execution and full validation.""" - loaded = read_benchmark(path, allow_unsigned=allow_unsigned) - for prepared in loaded.cases.values(): - case, root = prepared.case, prepared.root - try: - for task_id, source in case.task_sources.items(): - try: - source.resolve(root, allow_unsigned=allow_unsigned) - except (OSError, ValueError) as error: - raise ValueError(f"task {task_id}: {error}") from error - for agent in case.agents(): - for name, source in agent.inputs.items(): - try: - source.resolve(root, allow_unsigned=allow_unsigned) - except (OSError, ValueError) as error: - raise ValueError(f"agent {agent.agent_bundle_id}, input {name}: {error}") from error - except (OSError, ValueError) as error: - raise ValueError(f"case {case.case_id}: {error}") from error - loaded.subsets[case.case_id] = case.load_subsets(root, allow_unsigned=allow_unsigned) - loaded.descriptions[case.case_id] = case.load_description(root, allow_unsigned=allow_unsigned) - return loaded + """Read the experiment and stage configurations without opening executable inputs.""" + definition = BenchmarkDefinition.model_validate(read_yaml(path)) + root = path.resolve().parent + cases_path = definition.cases.resolve(root, allow_unsigned=allow_unsigned) + execution_path = definition.execution.resolve(root, allow_unsigned=allow_unsigned) + analysis_path = definition.analysis.resolve(root, allow_unsigned=allow_unsigned) + analysis = load_analysis(cases_path, analysis_path, allow_unsigned=allow_unsigned) + execution = LoadedExecution(analysis.inputs, LocalExecution.model_validate(read_yaml(execution_path))) + return LoadedBenchmark(definition, execution, analysis) diff --git a/labs/metric-benchmark/metric_benchmark/execution.py b/labs/metric-benchmark/metric_benchmark/execution.py index d114996d..339ff40c 100644 --- a/labs/metric-benchmark/metric_benchmark/execution.py +++ b/labs/metric-benchmark/metric_benchmark/execution.py @@ -13,28 +13,17 @@ from harbor.models.trial.config import EnvironmentConfig from harbor.models.trial.result import TrialResult -from metric_benchmark.config import Case, LoadedBenchmark, LoadedCase, file_manifest -from metric_benchmark.io import digest_bytes, digest_file, input_digest, json_bytes, read_json, write_json +from metric_benchmark.config import Case, LoadedCase, LoadedExecution, file_manifest +from metric_benchmark.io import digest_bytes, digest_file, json_bytes, read_json, write_json from metric_benchmark.models import Execution, ExecutionManifest, RawResults, Text, resolve_evidence -from metric_benchmark.validation import agent_config, native_observation, validate_results +from metric_benchmark.validation import agent_config, native_measurements, validate_results -def benchmark_identity(loaded: LoadedBenchmark) -> str: - """Compute the benchmark reuse fingerprint, including the contents of inputs without SHA pins.""" - unsigned = {} - for entry in loaded.definition.cases: - loaded_case = loaded.case(entry.case_id) - case = loaded_case.case - sources = [entry.agent, entry.tasks, entry.subsets, case.case_description, *case.task_sources.values()] - sources.extend(source for agent in case.agents() for source in agent.inputs.values()) - for source in sources: - if source.sha256 is None and source.path not in unsigned: - unsigned[source.path] = input_digest( - source.resolve(loaded_case.root, allow_unsigned=loaded_case.allow_unsigned) - ) - if not unsigned: - return loaded.identity - return digest_bytes(json_bytes({"benchmark_sha256": loaded.identity, "unsigned_inputs": unsigned})) +def execution_identity(loaded: LoadedExecution) -> str: + """Identify execution inputs independently of analysis and experiment metadata.""" + return digest_bytes( + json_bytes({"cases": loaded.inputs.identity_payload(verify=True), "execution": loaded.settings.model_dump()}) + ) def case_root(output: Path, case_index: int) -> Path: @@ -61,13 +50,13 @@ async def run(): def job_configuration( - loaded: LoadedBenchmark, loaded_case: LoadedCase, agent_index: int, root: Path, paths: list[Path] + loaded: LoadedExecution, loaded_case: LoadedCase, agent_index: int, root: Path, paths: list[Path] ) -> JobConfig: """Build and validate a Harbor job for one agent and all supplied task paths. Resolve the agent’s input files and execution settings without starting the job. """ case = loaded_case.case - settings = loaded.definition.execution + settings = loaded.settings if set(settings.environment) - EnvironmentConfig.model_fields.keys(): raise ValueError("unknown Harbor environment setting") return JobConfig.model_validate( @@ -89,7 +78,7 @@ def job_configuration( ) -def collect_job(case: Case, agent_index: int, root: Path, request: JobConfig, metric: str) -> list[Execution]: +def collect_job(case: Case, agent_index: int, root: Path, request: JobConfig) -> list[Execution]: """Read a finished Harbor job into one Execution record per task, including status, score, and evidence path. Use saved task paths to identify tasks; reject unfinished jobs and missing, duplicate, or unexpected tasks. """ @@ -113,7 +102,7 @@ def collect_job(case: Case, agent_index: int, root: Path, request: JobConfig, me if task is None or task.task_id in seen: raise ValueError("unexpected or repeated native task in Harbor job") seen.add(task.task_id) - status, score = native_observation(native, metric) + status, metrics = native_measurements(native) rows.append( Execution.model_validate( { @@ -125,7 +114,7 @@ def collect_job(case: Case, agent_index: int, root: Path, request: JobConfig, me "task": task.model_dump(), }, "status": status, - "metrics": {metric: score.model_dump()}, + "metrics": {name: score.model_dump() for name, score in metrics.items()}, "native_results_path": result_path.parent.relative_to(root).as_posix(), "native_results_format": "directory", } @@ -137,7 +126,7 @@ def collect_job(case: Case, agent_index: int, root: Path, request: JobConfig, me def run_or_reuse_job( - loaded: LoadedBenchmark, + loaded: LoadedExecution, case: Case, agent_index: int, root: Path, @@ -156,10 +145,10 @@ def run_or_reuse_job( receipt_path = parent / "receipt.json" sealed = receipt_path.exists() request = { - "benchmark_sha256": identity, + "execution_sha256": identity, "case_id": case.case_id, "agent_bundle_id": agent.agent_bundle_id, - "execution": loaded.definition.execution.model_dump(mode="json"), + "execution": loaded.settings.model_dump(mode="json"), } write_json(parent / "request.json", request) native_path = parent / "native-request.json" @@ -187,7 +176,7 @@ def run_or_reuse_job( if native_request != config: raise ValueError("unstarted job request moved; use a new output for this execution") runner(config) - collected = collect_job(case, agent_index, root, native_request, loaded.definition.scoring.metric) + collected = collect_job(case, agent_index, root, native_request) if not sealed: write_json(receipt_path, {**expected, "files": file_manifest(directory)}) return collected @@ -196,7 +185,7 @@ def run_or_reuse_job( def execute_benchmark( - loaded: LoadedBenchmark, + loaded: LoadedExecution, run_id: Text, output: Path, *, @@ -207,14 +196,14 @@ def execute_benchmark( """ if not run_id.strip(): raise ValueError("run_id must be nonempty") - if version("harbor") != loaded.definition.execution.harbor_version: + if version("harbor") != loaded.settings.harbor_version: raise ValueError("installed Harbor version differs from benchmark execution version") - identity = benchmark_identity(loaded) + identity = execution_identity(loaded) output = output.resolve() # Resolve and validate all jobs before launching any, so a bad later case # cannot spend a valid earlier case's execution budget. jobs = {} - for case_index, loaded_case in enumerate(loaded.cases.values()): + for case_index, loaded_case in enumerate(loaded.inputs.cases.values()): case = loaded_case.case paths = [ case.task_sources[t.task_id].resolve(loaded_case.root, allow_unsigned=loaded_case.allow_unsigned) @@ -228,16 +217,16 @@ def execute_benchmark( ) inputs_path = output / "execution-inputs.json" inputs = { - "benchmark_sha256": identity, - "benchmark": loaded.definition.model_dump(mode="json"), + "execution_sha256": identity, + "execution": loaded.settings.model_dump(mode="json"), "run_id": run_id, } if inputs_path.exists() and read_json(inputs_path) != inputs: - raise ValueError("benchmark inputs, configuration, or run ID changed; use a new output directory") + raise ValueError("execution inputs, settings, or run ID changed; use a new output directory") write_json(inputs_path, inputs) indexes = {} # Each agent gets one job containing all case tasks; Harbor schedules the individual trials. - for case_index, loaded_case in enumerate(loaded.cases.values()): + for case_index, loaded_case in enumerate(loaded.inputs.cases.values()): case = loaded_case.case root = case_root(output, case_index) rows = [] @@ -248,11 +237,11 @@ def execute_benchmark( ) ) raw = RawResults(run_id=run_id, task_inventory=case.task_inventory.tasks, execution_results=rows) - validate_results( - raw, case, root, loaded_case.root, loaded.definition.scoring, allow_unsigned=loaded.allow_unsigned - ) + validate_results(raw, case, root, loaded_case.root, allow_unsigned=loaded.inputs.allow_unsigned) write_json(root / "results.json", raw) indexes[case.case_id] = (root / "results.json").relative_to(output).as_posix() - result = ExecutionManifest(benchmark_sha256=identity, run_id=run_id, cases=indexes).model_dump() + result = ExecutionManifest( + cases_sha256=loaded.inputs.identity, execution_sha256=identity, run_id=run_id, cases=indexes + ).model_dump() write_json(output / "execution.json", result) return result diff --git a/labs/metric-benchmark/metric_benchmark/models.py b/labs/metric-benchmark/metric_benchmark/models.py index fabcfec1..74b143cf 100644 --- a/labs/metric-benchmark/metric_benchmark/models.py +++ b/labs/metric-benchmark/metric_benchmark/models.py @@ -96,7 +96,8 @@ def safe_path(self) -> Self: class ExecutionManifest(Model): - benchmark_sha256: Digest + cases_sha256: Digest + execution_sha256: Digest run_id: Text cases: dict[Text, Text] = Field(min_length=1) diff --git a/labs/metric-benchmark/metric_benchmark/signing.py b/labs/metric-benchmark/metric_benchmark/signing.py index ec972908..228822cc 100644 --- a/labs/metric-benchmark/metric_benchmark/signing.py +++ b/labs/metric-benchmark/metric_benchmark/signing.py @@ -8,18 +8,22 @@ from pathlib import Path import yaml +from pydantic import TypeAdapter from metric_benchmark.config import ( AgentManifest, + AnalysisSettings, BenchmarkDefinition, Case, + CaseDefinition, + LocalExecution, PinnedInput, TaskManifest, load_benchmark, read_yaml, ) from metric_benchmark.io import digest_bytes, input_digest, read_json -from metric_benchmark.models import Subsets +from metric_benchmark.models import Subsets, unique def _sha_edits(node: yaml.Node, before, after, text: str) -> list[tuple[int, int, str]]: @@ -60,7 +64,7 @@ def _sha_edits(node: yaml.Node, before, after, text: str) -> list[tuple[int, int return edits -def _yaml_bytes(path: Path, data: dict) -> bytes: +def _yaml_bytes(path: Path, data: dict | list) -> bytes: # Parser positions let us change pins without reformatting YAML or losing comments. text = path.read_bytes().decode("utf-8") node = yaml.compose(text, Loader=yaml.SafeLoader) @@ -84,39 +88,56 @@ def _replace(path: Path, data: bytes) -> None: temporary.unlink(missing_ok=True) -def _collect(path: Path) -> tuple[dict[Path, dict], dict[tuple[Path, str], tuple[dict, Path, bool]]]: +def _collect(path: Path) -> tuple[dict[Path, dict | list], dict[tuple[Path, str], tuple[dict, Path, bool]]]: root = path.parent - documents: dict[Path, dict] = {} + documents: dict[Path, dict | list] = {} references: dict[tuple[Path, str], tuple[dict, Path, bool]] = {} - def document(target: Path) -> dict: + def document(target: Path): if target not in documents: data = read_yaml(target) - if not isinstance(data, dict): - raise ValueError(f"{target.relative_to(root)} must be a YAML mapping") + if not isinstance(data, (dict, list)): + raise ValueError(f"{target.name} must be a YAML mapping or list") documents[target] = data return documents[target] - def reference(owner: Path, label: str, data: dict, source: PinnedInput, *, manifest: bool = False) -> Path: + def reference(owner: Path, label: str, data: dict, source: PinnedInput, *, manifest=False) -> Path: try: target = source.resolve(root, verify_digest=False) except (OSError, ValueError) as error: - raise ValueError(f"{owner.relative_to(root)}:{label}: {error}") from error + raise ValueError(f"{owner.name}:{label}: {error}") from error references[owner, label] = data, target, manifest return target experiment = document(path) definition = BenchmarkDefinition.model_validate(experiment) - for entry, raw_case in zip(definition.cases, experiment["cases"], strict=True): + cases_path = reference(path, "cases", experiment["cases"], definition.cases, manifest=True) + execution_path = reference(path, "execution", experiment["execution"], definition.execution, manifest=True) + analysis_path = reference(path, "analysis", experiment["analysis"], definition.analysis, manifest=True) + LocalExecution.model_validate(document(execution_path)) + analysis_data = document(analysis_path) + analysis = AnalysisSettings.model_validate(analysis_data) + cases_data = document(cases_path) + entries = TypeAdapter(list[CaseDefinition]).validate_python(cases_data) + if not entries: + raise ValueError("cases must not be empty") + unique([entry.case_id for entry in entries], "case IDs") + if set(analysis.case_settings) != {entry.case_id for entry in entries}: + raise ValueError("analysis case_settings must match the cases exactly") + root = cases_path.parent + for entry, raw_case in zip(entries, cases_data, strict=True): label = f"cases[{entry.case_id}]" - agent_path = reference(path, f"{label}.agent", raw_case["agent"], entry.agent, manifest=True) - tasks_path = reference(path, f"{label}.tasks", raw_case["tasks"], entry.tasks, manifest=True) + agent_path = reference(cases_path, f"{label}.agent", raw_case["agent"], entry.agent, manifest=True) + tasks_path = reference(cases_path, f"{label}.tasks", raw_case["tasks"], entry.tasks, manifest=True) agent_data, tasks_data = document(agent_path), document(tasks_path) agent, tasks = AgentManifest.model_validate(agent_data), TaskManifest.model_validate(tasks_data) case = Case.from_manifests(entry, agent, tasks) - subset_path = reference(path, f"{label}.subsets", raw_case["subsets"], entry.subsets) + settings, raw_settings = analysis.case_settings[entry.case_id], analysis_data["case_settings"][entry.case_id] + subset_path = reference(analysis_path, f"{label}.subsets", raw_settings["subsets"], settings.subsets) Subsets.model_validate(read_json(subset_path)).resolve_tasks(case.task_inventory) - description = reference(agent_path, "description", agent_data["description"], agent.description) + description = reference( + analysis_path, f"{label}.description", raw_settings["description"], settings.description + ) if not isinstance(read_json(description), dict): raise ValueError(f"case {entry.case_id}: case description must be a JSON object") for i, (task, raw_task) in enumerate(zip(tasks.tasks, tasks_data["tasks"], strict=True)): @@ -126,13 +147,16 @@ def reference(owner: Path, label: str, data: dict, source: PinnedInput, *, manif agent_label = "baseline" if i == 0 else f"mutants[{i - 1}]" for name, source in version.inputs.items(): reference(agent_path, f"{agent_label}.inputs.{name}", raw_version["inputs"][name], source) - return documents, references def check_benchmark(path: Path) -> None: """Require and verify every input pin without writing files.""" - load_benchmark(path) + loaded = load_benchmark(path) + loaded.execution.inputs.verify_execution_inputs() + for case_id in loaded.analysis.inputs.cases: + loaded.analysis.load_subsets(case_id) + loaded.analysis.load_description(case_id) def sign_benchmark(path: Path, *, resign: bool = False) -> list[Path]: @@ -156,13 +180,22 @@ def sign_benchmark(path: Path, *, resign: bool = False) -> list[Path]: if any(target == item or target in item.parents for item in documents): raise ValueError(f"input overlaps a manifest being signed: {target.relative_to(root)}") data["sha256"] = input_digest(target) - # The schema has two manifest levels: experiment -> agent/tasks -> leaf inputs. - # If nested manifests are added, finalize their children before hashing parents. - planned = {target: _yaml_bytes(target, data) for target, data in documents.items() if target != path} - for data, target, manifest in references.values(): - if manifest: - data["sha256"] = digest_bytes(planned[target]) - planned[path] = _yaml_bytes(path, documents[path]) + planned = {} + active = set() + + def finalize(target): + if target in active: + raise ValueError("cyclic manifest references") + if target not in planned: + active.add(target) + for (owner, _), (data, child, manifest) in references.items(): + if owner == target and manifest: + data["sha256"] = digest_bytes(finalize(child)) + planned[target] = _yaml_bytes(target, documents[target]) + active.remove(target) + return planned[target] + + finalize(path) changed = [] for target, content in planned.items(): if target.read_bytes() != content: diff --git a/labs/metric-benchmark/metric_benchmark/validation.py b/labs/metric-benchmark/metric_benchmark/validation.py index 29c0ddb7..625b7a42 100644 --- a/labs/metric-benchmark/metric_benchmark/validation.py +++ b/labs/metric-benchmark/metric_benchmark/validation.py @@ -14,11 +14,11 @@ from metric_benchmark.config import ( Agent, Case, - LoadedBenchmark, + LoadedCases, ScoringConfig, ScoringPolicy, file_manifest, - load_benchmark, + load_cases, ) from metric_benchmark.io import cli, read_json, write_json from metric_benchmark.models import ExecutionManifest, RawResults, Score, resolve_evidence @@ -70,6 +70,13 @@ def native_observation(trial: TrialResult, metric: str) -> tuple[str, Score]: return "succeeded", Score(value=rewards[metric]) +def native_measurements(trial: TrialResult) -> tuple[str, dict[str, Score]]: + rewards = trial.verifier_result.rewards if trial.verifier_result else None + measurements = {metric: native_observation(trial, metric)[1] for metric in (rewards or {"reward": None})} + status, _ = native_observation(trial, next(iter(measurements))) + return status, measurements + + def selection_for_results(case: Case, raw: RawResults, scoring: ScoringPolicy) -> ScoringConfig: if raw.task_inventory != case.task_inventory.tasks: raise ValueError("results task inventory differs from benchmark case") @@ -100,7 +107,7 @@ def recorded_input_root(task_path: Path | None, relative: str) -> Path: def validate_results( - raw: RawResults, case: Case, root: Path, input_root: Path, scoring: ScoringPolicy, *, allow_unsigned: bool = False + raw: RawResults, case: Case, root: Path, input_root: Path, *, allow_unsigned: bool = False ) -> dict: """Validate an accounted matrix; unavailable observations remain unavailable. @@ -109,7 +116,7 @@ def validate_results( """ # Revalidate callers' mutable model instances as well as file inputs. raw = RawResults.model_validate(raw.model_dump()) - selection = selection_for_results(case, raw, scoring) + selection_for_results(case, raw, ScoringPolicy()) agents = {a.agent_bundle_id: a for a in case.agents()} expected = {(a, t.task_id, t.revision) for a in agents for t in case.task_inventory.tasks} tasks = { @@ -127,11 +134,12 @@ def validate_results( if key not in expected: raise ValueError("unexpected agent or task in results") groups.setdefault(key, []).append(row) - if scoring.metric not in row.metrics: - raise ValueError("measurement must be numeric or explicitly unavailable") - score = row.metrics[scoring.metric] if row.status == "missing": - if score.value is not None or not row.unavailable_evidence.get("native_results"): + if ( + not row.metrics + or any(score.value is not None for score in row.metrics.values()) + or not row.unavailable_evidence.get("native_results") + ): raise ValueError("missing execution needs unavailable score and evidence reason") if (root / row.native_results_path).exists(): raise ValueError("missing execution unexpectedly has native evidence") @@ -160,18 +168,20 @@ def validate_results( ) if trial.config.agent != configs[config_key]: raise ValueError("native agent configuration differs from benchmark") - status, observed = native_observation(trial, scoring.metric) - if status != row.status or score != observed: + status, observed = native_measurements(trial) + if status != row.status or row.metrics != observed: raise ValueError("native status or measurement differs from index") - if score.value is None: - unavailable.append( - { - "agent": row.agent_bundle_id, - "task": key[1], - "attempt": row.evaluation.execution, - "reason": score.reason, - } - ) + for metric, score in row.metrics.items(): + if score.value is None: + unavailable.append( + { + "agent": row.agent_bundle_id, + "task": key[1], + "attempt": row.evaluation.execution, + "metric": metric, + "reason": score.reason, + } + ) except (OSError, ValueError) as error: raise ValueError( f"case {case.case_id}, agent {row.agent_bundle_id}, task {row.evaluation.task.task_id}, " @@ -182,8 +192,6 @@ def validate_results( for rows in groups.values(): if len({r.evaluation.evaluation_id for r in rows}) != 1: raise ValueError("ambiguous repetitions for agent/task") - if len(rows) != 1 and selection.attempt_policy == "only": - raise ValueError("multiple attempts with attempt_policy=only") manifest = file_manifest(root) # Derived reports are outside this root. The index is hashed separately by # consumers, so it may be published after validation by an execution backend. @@ -198,8 +206,10 @@ def validate_results( } -def validate_experiment_results(loaded: LoadedBenchmark, root: Path, *, case_id: str | None = None) -> dict: +def validate_experiment_results(loaded: LoadedCases, root: Path, *, case_id: str | None = None) -> dict: manifest = ExecutionManifest.model_validate(read_json(resolve_evidence(root, "execution.json"))) + if manifest.cases_sha256 != loaded.identity: + raise ValueError("execution case definitions differ from supplied cases") selected = {case_id: loaded.case(case_id)} if case_id is not None else loaded.cases missing = selected.keys() - manifest.cases.keys() unexpected = manifest.cases.keys() - loaded.cases.keys() @@ -225,7 +235,6 @@ def validate_experiment_results(loaded: LoadedBenchmark, root: Path, *, case_id: prepared.case, index.parent, prepared.root, - loaded.definition.scoring, allow_unsigned=prepared.allow_unsigned, ) except (OSError, ValueError) as error: @@ -235,7 +244,7 @@ def validate_experiment_results(loaded: LoadedBenchmark, root: Path, *, case_id: def main() -> None: parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--benchmark", type=Path, required=True) + parser.add_argument("--cases", type=Path, required=True) parser.add_argument( "--allow-unsigned", action="store_true", help="allow missing input pins; still verify supplied SHAs" ) @@ -244,7 +253,7 @@ def main() -> None: parser.add_argument("--output", type=Path, required=True) def run(args): - loaded = load_benchmark(args.benchmark, allow_unsigned=args.allow_unsigned) + loaded = load_cases(args.cases, allow_unsigned=args.allow_unsigned) write_json(args.output, validate_experiment_results(loaded, args.results_root, case_id=args.case)) cli(parser, run) diff --git a/labs/metric-benchmark/run_evals.py b/labs/metric-benchmark/run_evals.py index 4a5ff61d..ce08063c 100644 --- a/labs/metric-benchmark/run_evals.py +++ b/labs/metric-benchmark/run_evals.py @@ -6,7 +6,7 @@ import argparse from pathlib import Path -from metric_benchmark.config import load_benchmark +from metric_benchmark.config import load_execution from metric_benchmark.execution import execute_benchmark from metric_benchmark.io import cli @@ -14,7 +14,8 @@ def main() -> None: """Parse execution arguments, load the benchmark, and invoke the execution library.""" parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--benchmark", type=Path, required=True) + parser.add_argument("--cases", type=Path, required=True) + parser.add_argument("--execution", type=Path, required=True) parser.add_argument( "--allow-unsigned", action="store_true", help="allow missing input pins; still verify supplied SHAs" ) @@ -22,7 +23,9 @@ def main() -> None: parser.add_argument("--output", type=Path, required=True) def run(args): - execute_benchmark(load_benchmark(args.benchmark, allow_unsigned=args.allow_unsigned), args.run_id, args.output) + execute_benchmark( + load_execution(args.cases, args.execution, allow_unsigned=args.allow_unsigned), args.run_id, args.output + ) cli(parser, run) diff --git a/tests/fixtures/metric_benchmark/agents/rho-agent/agent.yml b/tests/fixtures/metric_benchmark/agents/rho-agent/agent.yml index d4647de6..38714f62 100644 --- a/tests/fixtures/metric_benchmark/agents/rho-agent/agent.yml +++ b/tests/fixtures/metric_benchmark/agents/rho-agent/agent.yml @@ -16,7 +16,3 @@ mutants: revision: '1' harbor: name: nop -description: - path: agents/rho-agent/description.json - sha256: sha256:fcb4a1e2e0a78fc9dd624d72c123d73b5547d8185a567874b2581114357a957a - kind: file diff --git a/tests/fixtures/metric_benchmark/analysis.yml b/tests/fixtures/metric_benchmark/analysis.yml new file mode 100644 index 00000000..a60e4211 --- /dev/null +++ b/tests/fixtures/metric_benchmark/analysis.yml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +case_settings: + rho-agent: + description: + path: agents/rho-agent/description.json + sha256: sha256:fcb4a1e2e0a78fc9dd624d72c123d73b5547d8185a567874b2581114357a957a + kind: file + subsets: + path: tasks/workspace/subsets.json + sha256: sha256:c9b05fb9ca48d0f7ca01fd1dc755d68b909eceadf3d094edcadd7aa3b162a786 +coverage: + system_id: synthetic + system_revision: '1' + command: + - python + - /adapter.py diff --git a/tests/fixtures/metric_benchmark/cases.yml b/tests/fixtures/metric_benchmark/cases.yml new file mode 100644 index 00000000..8aa9e44b --- /dev/null +++ b/tests/fixtures/metric_benchmark/cases.yml @@ -0,0 +1,10 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +- case_id: rho-agent + agent: + path: agents/rho-agent/agent.yml + sha256: sha256:dd5ad215b00f9d5e726c48ea80097db48a7fe7e28060e23194a95844b6b6ffc3 + tasks: + path: tasks/workspace/tasks.yml + sha256: sha256:247e20d159875c87f97c01b58fe7d9115c9e849f0b3779a1a5081788d9d1f93f diff --git a/tests/fixtures/metric_benchmark/execution.yml b/tests/fixtures/metric_benchmark/execution.yml new file mode 100644 index 00000000..fdc11add --- /dev/null +++ b/tests/fixtures/metric_benchmark/execution.yml @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +n_concurrent_trials: 1 diff --git a/tests/fixtures/metric_benchmark/experiment.yml b/tests/fixtures/metric_benchmark/experiment.yml index 09281445..e979bcca 100644 --- a/tests/fixtures/metric_benchmark/experiment.yml +++ b/tests/fixtures/metric_benchmark/experiment.yml @@ -1,23 +1,14 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# Prepared case contents and subsets stay fixed when experiment settings change. benchmark_id: rho-metric-validity revision: 1 cases: -- case_id: rho-agent - agent: - path: agents/rho-agent/agent.yml - sha256: sha256:d5f2748f94d1a3c14054bb7a6f10ffd6a3888956b2e96f948e53db42fe8db662 - tasks: - path: tasks/workspace/tasks.yml - sha256: sha256:247e20d159875c87f97c01b58fe7d9115c9e849f0b3779a1a5081788d9d1f93f - subsets: - path: tasks/workspace/subsets.json - sha256: sha256:c9b05fb9ca48d0f7ca01fd1dc755d68b909eceadf3d094edcadd7aa3b162a786 -coverage: - system_id: synthetic - system_revision: '1' - command: - - python - - /adapter.py + path: cases.yml + sha256: sha256:cf58ce28cb663c8262733ca171b5b4ce59f4adce251e985838bd4e9fcb1fb910 +execution: + path: execution.yml + sha256: sha256:3e7868e54726a1816ef6539bc6c7c6f4bdc82e9d31b9f82219b85330ff25e1d0 +analysis: + path: analysis.yml + sha256: sha256:a014a0633291ec1f6ea1227c5e60244608a87af75805c88dc3aade384422830a diff --git a/tests/fixtures/metric_benchmark_results/experiment-results/execution.json b/tests/fixtures/metric_benchmark_results/experiment-results/execution.json index 47949770..2dbd6942 100644 --- a/tests/fixtures/metric_benchmark_results/experiment-results/execution.json +++ b/tests/fixtures/metric_benchmark_results/experiment-results/execution.json @@ -1,7 +1,8 @@ { - "benchmark_sha256": "sha256:bbd8b9dd1fbdbfe2ccf8fce5b106254cbfdd8b13e05b9d613347aa976203be6f", + "execution_sha256": "sha256:bbd8b9dd1fbdbfe2ccf8fce5b106254cbfdd8b13e05b9d613347aa976203be6f", "cases": { "rho-agent": "cases/case-0000/execution-results/results.json" }, - "run_id": "mock-run-001" + "run_id": "mock-run-001", + "cases_sha256": "sha256:670d1cac62a4938f17f44b0d50e915472947847542ce7cb3a3ca76599d202d8a" } diff --git a/tests/test_metric_benchmark_config.py b/tests/test_metric_benchmark_config.py index 0e82c439..6aa08227 100644 --- a/tests/test_metric_benchmark_config.py +++ b/tests/test_metric_benchmark_config.py @@ -8,7 +8,7 @@ import pytest import yaml -from metric_benchmark.config import input_digest, load_benchmark, read_benchmark +from metric_benchmark.config import input_digest, load_analysis, load_benchmark, load_execution from metric_benchmark.io import read_json, write_json from metric_benchmark.models import Score, Subsets, Tasks @@ -62,15 +62,16 @@ def prepared_benchmark(root): def case_definition(experiment): - manifest = yaml.safe_load(experiment.read_text())["cases"][0] + manifest = yaml.safe_load((experiment.parent / "cases.yml").read_text())[0] agent = yaml.safe_load((experiment.parent / manifest["agent"]["path"]).read_text()) tasks = yaml.safe_load((experiment.parent / manifest["tasks"]["path"]).read_text()) + settings = yaml.safe_load((experiment.parent / "analysis.yml").read_text())["case_settings"][manifest["case_id"]] return { "case_id": manifest["case_id"], - "subsets": manifest["subsets"], + "subsets": settings["subsets"], "baseline": agent["baseline"], "mutants": agent["mutants"], - "case_description": agent["description"], + "case_description": settings["description"], "task_inventory": { "tasks": [ {key: value for key, value in task.items() if key in ("task_id", "revision", "name")} @@ -84,17 +85,25 @@ def case_definition(experiment): } -def write_case(experiment, case): +def write_definition(experiment, name, data): + target = experiment.parent / name + target.write_text(yaml.safe_dump(data, sort_keys=False)) definition = yaml.safe_load(experiment.read_text()) - manifest = definition["cases"][0] + for source in (definition["cases"], definition["execution"], definition["analysis"]): + source["sha256"] = input_digest(experiment.parent / source["path"]) + experiment.write_text(yaml.safe_dump(definition, sort_keys=False)) + + +def write_case(experiment, case): + cases_path = experiment.parent / "cases.yml" + cases = yaml.safe_load(cases_path.read_text()) + manifest = cases[0] + original_id = manifest["case_id"] parts = { - "agent": {"baseline": case["baseline"], "mutants": case["mutants"], "description": case["case_description"]}, + "agent": {"baseline": case["baseline"], "mutants": case["mutants"]}, "tasks": { "tasks": [ - { - **task, - **{key: value for key, value in case["task_sources"][task["task_id"]].items() if key != "kind"}, - } + {**task, **{k: v for k, v in case["task_sources"][task["task_id"]].items() if k != "kind"}} for task in case["task_inventory"]["tasks"] ] }, @@ -103,56 +112,12 @@ def write_case(experiment, case): target = experiment.parent / manifest[name]["path"] target.write_text(yaml.safe_dump(data, sort_keys=False)) manifest[name]["sha256"] = input_digest(target) - manifest.update(case_id=case["case_id"], subsets=case["subsets"]) - experiment.write_text(yaml.safe_dump(definition, sort_keys=False)) - - -def test_prepared_inputs_and_fixed_memberships(tmp_path): - path, _ = prepared_benchmark(tmp_path) - loaded = load_benchmark(path) - assert loaded == load_benchmark(path) - assert loaded.subsets["rho-agent"].sampling_parameters == {"seed": 42} - assert loaded.case("rho-agent").case.selection( - {"baseline": "baseline-job", "mutant": "mutant-job", "mutant-no-search": "search-mutant-job"}, - loaded.definition.scoring, - ).run_ids == { - "baseline": "baseline-job", - "mutant": "mutant-job", - "mutant-no-search": "search-mutant-job", - } - (tmp_path / TASK_DIR / "inspect-files/instruction.md").write_text("changed") - with pytest.raises(ValueError, match="digest differs"): - load_benchmark(path) - - -@pytest.mark.parametrize("change", ["unknown", "duplicate-case", "agent-pin", "source-missing"]) -def test_bad_definition(tmp_path, change): - path, config = prepared_benchmark(tmp_path) - if change == "unknown": - config["surprise"] = True - elif change == "duplicate-case": - config["cases"] *= 2 - elif change == "agent-pin": - (tmp_path / AGENT_DIR / "agent.yml").write_text("changed") - else: - case = case_definition(path) - case["task_sources"]["inspect-files"]["path"] = "missing-task" - write_case(path, case) - config = yaml.safe_load(path.read_text()) - path.write_text(yaml.safe_dump(config)) - with pytest.raises(ValueError): - load_benchmark(path) - - -def test_duplicate_yaml_and_paths(tmp_path): - path, config = prepared_benchmark(tmp_path) - path.write_text(path.read_text() + "benchmark_id: duplicate\n") - with pytest.raises(ValueError, match="unique strings"): - load_benchmark(path) - config["cases"][0]["agent"]["path"] = "../outside.yml" - path.write_text(yaml.safe_dump(config)) - with pytest.raises(ValueError, match="relative"): - load_benchmark(path) + manifest["case_id"] = case["case_id"] + cases_path.write_text(yaml.safe_dump(cases, sort_keys=False)) + analysis = yaml.safe_load((experiment.parent / "analysis.yml").read_text()) + analysis["case_settings"].pop(original_id) + analysis["case_settings"][case["case_id"]] = {"description": case["case_description"], "subsets": case["subsets"]} + write_definition(experiment, "analysis.yml", analysis) def test_json_numbers_and_immutable_outputs(tmp_path): @@ -180,40 +145,6 @@ def test_manifest_allows_duplicate_memberships(tmp_path): assert subsets.eval_subsets[0].task_ids == subsets.eval_subsets[1].task_ids -@pytest.mark.parametrize("value", [float("inf"), float("-inf"), float("nan")]) -@pytest.mark.parametrize("field", ["harbor", "settings"]) -def test_nonfinite_native_settings_rejected_before_serialization(tmp_path, value, field): - path, definition = prepared_benchmark(tmp_path) - case = case_definition(path) - target = case["baseline"]["harbor"] if field == "harbor" else definition["coverage"].setdefault("settings", {}) - target["nested"] = {"values": [value]} - if field == "harbor": - write_case(path, case) - else: - path.write_text(yaml.safe_dump(definition)) - with pytest.raises(ValueError, match="baseline.harbor" if field == "harbor" else "coverage.settings"): - load_benchmark(path) - - -def test_yaml_date_rejected_but_explicit_string_and_null_preserved(tmp_path): - path, definition = prepared_benchmark(tmp_path) - settings = {"when": date(2020, 1, 1)} - definition["coverage"]["settings"] = settings - path.write_text(yaml.safe_dump(definition)) - with pytest.raises(ValueError, match=r"coverage\.settings\.when"): - load_benchmark(path) - settings.update(when="2020-01-01", limit=None) - path.write_text(yaml.safe_dump(definition), encoding="utf-8") - assert load_benchmark(path).definition.coverage.settings == settings - - -def test_changed_task_input_names_case_and_task(tmp_path): - path, _ = prepared_benchmark(tmp_path) - (tmp_path / TASK_DIR / "inspect-files/instruction.md").unlink() - with pytest.raises(ValueError, match="case rho-agent: task inspect-files: input digest differs"): - load_benchmark(path) - - @pytest.mark.parametrize("value", [float("inf"), float("nan"), date(2020, 1, 1)]) def test_library_configuration_rejects_non_json_values(value): from metric_benchmark.config import Agent, CoverageConfig, LocalExecution @@ -238,122 +169,185 @@ def test_mutated_configuration_cannot_serialize_invalid_json(): json_bytes(config) -def test_analysis_definition_and_subsets_do_not_need_execution_inputs(tmp_path): +def test_stage_loading_and_fixed_memberships(tmp_path): + path, _ = prepared_benchmark(tmp_path) + benchmark = load_benchmark(path) + execution = load_execution(tmp_path / "cases.yml", tmp_path / "execution.yml") + analysis = load_analysis(tmp_path / "cases.yml", tmp_path / "analysis.yml") + assert benchmark.execution == execution + assert benchmark.analysis == analysis + assert analysis.load_subsets("rho-agent").sampling_parameters == {"seed": 42} + assert analysis.load_description("rho-agent") + assert set( + execution.inputs.case("rho-agent") + .case.selection({"baseline": "a", "mutant": "b", "mutant-no-search": "c"}, analysis.settings.scoring) + .run_ids + ) == {"baseline", "mutant", "mutant-no-search"} + (tmp_path / TASK_DIR / "inspect-files/instruction.md").write_text("changed") + with pytest.raises(ValueError, match="case rho-agent: task inspect-files: input digest differs"): + load_execution(tmp_path / "cases.yml", tmp_path / "execution.yml") + + +@pytest.mark.parametrize( + "change", ["unknown", "duplicate-case", "agent-pin", "source-missing", "empty-cases", "analysis-cases"] +) +def test_bad_definition(tmp_path, change): + path, config = prepared_benchmark(tmp_path) + cases = yaml.safe_load((tmp_path / "cases.yml").read_text()) + if change == "unknown": + config["surprise"] = True + path.write_text(yaml.safe_dump(config)) + elif change in ("duplicate-case", "empty-cases"): + write_definition(path, "cases.yml", cases * 2 if change == "duplicate-case" else []) + elif change == "agent-pin": + (tmp_path / AGENT_DIR / "agent.yml").write_text("changed") + elif change == "analysis-cases": + analysis = yaml.safe_load((tmp_path / "analysis.yml").read_text()) + analysis["case_settings"]["unknown"] = analysis["case_settings"].pop("rho-agent") + write_definition(path, "analysis.yml", analysis) + else: + case = case_definition(path) + case["task_sources"]["inspect-files"]["path"] = "missing-task" + write_case(path, case) + with pytest.raises(ValueError): + loaded = load_benchmark(path) + loaded.execution.inputs.verify_execution_inputs() + + +def test_duplicate_yaml_keys(tmp_path): + path, _ = prepared_benchmark(tmp_path) + path.write_text(path.read_text() + "benchmark_id: duplicate\n") + with pytest.raises(ValueError, match="unique strings"): + load_benchmark(path) + + +@pytest.mark.parametrize("value", [float("inf"), float("-inf"), float("nan"), date(2020, 1, 1)]) +@pytest.mark.parametrize("field", ["harbor", "settings"]) +def test_non_json_native_settings_rejected(tmp_path, value, field): + path, _ = prepared_benchmark(tmp_path) + if field == "harbor": + case = case_definition(path) + case["baseline"]["harbor"]["nested"] = {"values": [value]} + write_case(path, case) + else: + analysis = yaml.safe_load((tmp_path / "analysis.yml").read_text()) + analysis["coverage"]["settings"] = {"nested": [value]} + write_definition(path, "analysis.yml", analysis) + with pytest.raises(ValueError): + load_benchmark(path) + + +def test_yaml_string_dates_and_null_preserved(tmp_path): + path, _ = prepared_benchmark(tmp_path) + settings = {"when": "2020-01-01", "limit": None} + analysis = yaml.safe_load((tmp_path / "analysis.yml").read_text()) + analysis["coverage"]["settings"] = settings + write_definition(path, "analysis.yml", analysis) + assert load_benchmark(path).analysis.settings.coverage.settings == settings + + +def test_analysis_does_not_open_executable_inputs_or_execution_settings(tmp_path): path, _ = prepared_benchmark(tmp_path) for source in case_definition(path)["task_sources"].values(): shutil.rmtree(tmp_path / source["path"]) - (tmp_path / DESCRIPTION_PATH).unlink() - loaded = read_benchmark(path) - prepared = loaded.case("rho-agent") - case = prepared.case - assert len(case.load_subsets(prepared.root).eval_subsets) == 3 - (tmp_path / SUBSETS_PATH).write_text("{}") - with pytest.raises(ValueError, match="input digest differs"): - case.load_subsets(prepared.root) - with pytest.raises(ValueError, match="unknown case: absent"): - loaded.case("absent") - - -@pytest.mark.parametrize("standalone", [False, True]) + (tmp_path / "execution.yml").unlink() + analysis = load_analysis(tmp_path / "cases.yml", tmp_path / "analysis.yml") + assert len(analysis.load_subsets("rho-agent").eval_subsets) == 3 + assert analysis.load_description("rho-agent") + for lookup in (analysis.inputs.case, analysis.load_subsets, analysis.load_description): + with pytest.raises(ValueError, match="unknown case: absent"): + lookup("absent") + + +def test_execution_does_not_open_analysis_inputs_or_settings(tmp_path): + path, _ = prepared_benchmark(tmp_path) + for name in ("analysis.yml", SUBSETS_PATH, DESCRIPTION_PATH): + (tmp_path / name).unlink() + assert load_execution(tmp_path / "cases.yml", tmp_path / "execution.yml").inputs.cases + + @pytest.mark.parametrize("field", ["subsets", "case_description"]) @pytest.mark.parametrize("failure", ["missing", "invalid"]) -def test_case_artifact_errors_have_one_context(tmp_path, standalone, field, failure): - path, definition = prepared_benchmark(tmp_path) - case_data = case_definition(path) - source = case_data[field] +def test_case_artifact_errors_have_context(tmp_path, field, failure): + path, _ = prepared_benchmark(tmp_path) + case = case_definition(path) + source = case[field] artifact = tmp_path / source["path"] if failure == "missing": artifact.unlink() - detail = f"missing pinned input: {source['path']}" else: + data = [] if field == "case_description" else json.loads(artifact.read_text()) if field == "subsets": - data = json.loads(artifact.read_text()) + assert isinstance(data, dict) data["eval_subsets"][0]["task_ids"] = ["unknown-task"] - detail = "subset subset-000001 contains unknown task IDs: ['unknown-task']" - else: - data = [] - detail = "case description must be a JSON object" artifact.write_text(json.dumps(data)) source["sha256"] = input_digest(artifact) - write_case(path, case_data) - with pytest.raises(ValueError) as error: - if standalone: - benchmark = read_benchmark(path) - prepared = benchmark.case("rho-agent") - case = prepared.case - if field == "subsets": - case.load_subsets(prepared.root) - else: - case.load_description(prepared.root) - else: - load_benchmark(path) - assert str(error.value) == f"case rho-agent, {field}: {detail}" + write_case(path, case) + loaded = load_benchmark(path).analysis + with pytest.raises(ValueError, match=f"case rho-agent, {field}:"): + (loaded.load_subsets if field == "subsets" else loaded.load_description)("rho-agent") def test_permission_error_is_not_reported_as_missing(tmp_path, monkeypatch): path, _ = prepared_benchmark(tmp_path) - prepared = read_benchmark(path).case("rho-agent") - case = prepared.case + analysis = load_benchmark(path).analysis def denied(root, value): raise PermissionError("permission denied for prepared input") monkeypatch.setattr("metric_benchmark.config.resolve_evidence", denied) - with pytest.raises(ValueError, match="case rho-agent, subsets: permission denied for prepared input"): - case.load_subsets(prepared.root) + with pytest.raises(ValueError, match="case rho-agent, subsets: permission denied"): + analysis.load_subsets("rho-agent") -def test_fixed_case_can_be_reused_by_different_experiments(tmp_path): - path, definition = prepared_benchmark(tmp_path) - first = load_benchmark(path) - before = {p.relative_to(tmp_path): p.read_bytes() for p in (tmp_path / AGENT_DIR).rglob("*") if p.is_file()} - definition["benchmark_id"] = "rho-alternative-metric" - definition["coverage"]["system_id"] = "another-metric" - definition["scoring"] = {"direction": "lower_is_better"} - alternative = tmp_path / "alternative.yml" - alternative.write_text(yaml.safe_dump(definition)) - second = load_benchmark(alternative) - assert first.identity != second.identity - assert first.cases == second.cases - assert first.subsets == second.subsets - assert first.definition.cases == second.definition.cases - assert before == {p.relative_to(tmp_path): p.read_bytes() for p in (tmp_path / AGENT_DIR).rglob("*") if p.is_file()} - - -def test_unpacked_bundle_resolves_relative_to_bundle_root(tmp_path, monkeypatch): +def test_unpacked_bundle_resolves_from_cases_file(tmp_path, monkeypatch): path, _ = prepared_benchmark(tmp_path / "original") original = load_benchmark(path) + original_analysis_identity = original.analysis.identity destination = tmp_path / "unpacked" shutil.copytree(path.parent, destination) shutil.rmtree(path.parent) monkeypatch.chdir(tmp_path) moved = load_benchmark(destination / path.name) - assert moved.identity == original.identity - assert moved.subsets == original.subsets - prepared = moved.case("rho-agent") + assert moved.analysis.identity == original_analysis_identity + assert moved.execution.inputs.identity == original.execution.inputs.identity + prepared = moved.execution.inputs.case("rho-agent") assert prepared.root == destination assert ( - prepared.case.task_sources["inspect-files"].resolve(prepared.root) == prepared.root / TASK_DIR / "inspect-files" + prepared.case.task_sources["inspect-files"].resolve(prepared.root) == destination / TASK_DIR / "inspect-files" ) -@pytest.mark.parametrize("level", ["agent", "tasks", "subsets"]) +@pytest.mark.parametrize("level", ["cases", "execution", "analysis", "agent", "tasks", "subsets", "description"]) @pytest.mark.parametrize("failure", ["absolute", "traversal", "symlink", "missing"]) -def test_manifest_references_stay_inside_their_package(tmp_path, level, failure): +def test_manifest_references_stay_inside_package(tmp_path, level, failure): path, definition = prepared_benchmark(tmp_path) - source = definition["cases"][0][level] - root = tmp_path - target = root / source["path"] + if level in ("cases", "execution", "analysis"): + data, owner = definition, "experiment.yml" + source = data[level] + elif level in ("agent", "tasks"): + owner = "cases.yml" + data = yaml.safe_load((tmp_path / owner).read_text()) + source = data[0][level] + else: + owner = "analysis.yml" + data = yaml.safe_load((tmp_path / owner).read_text()) + source = data["case_settings"]["rho-agent"][level] + target = tmp_path / source["path"] if failure == "absolute": source["path"] = str(target) elif failure == "traversal": source["path"] = "../" + source["path"] elif failure == "symlink": - link = root / "alias" - link.symlink_to(target) + (tmp_path / "alias").symlink_to(target) source["path"] = "alias" else: target.unlink() - path.write_text(yaml.safe_dump(definition)) + if owner == "experiment.yml" or failure == "missing": + (tmp_path / owner).write_text(yaml.safe_dump(data)) if failure != "missing" else None + else: + write_definition(path, owner, data) with pytest.raises(ValueError, match="relative|symlink|missing pinned input"): - load_benchmark(path) + loaded = load_benchmark(path) + loaded.analysis.load_subsets("rho-agent") + loaded.analysis.load_description("rho-agent") diff --git a/tests/test_metric_benchmark_execution.py b/tests/test_metric_benchmark_execution.py index 1d0a7375..fc3b1097 100644 --- a/tests/test_metric_benchmark_execution.py +++ b/tests/test_metric_benchmark_execution.py @@ -1,13 +1,14 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 +import json from pathlib import Path import pytest -from metric_benchmark.config import load_benchmark -from metric_benchmark.execution import case_root, execute_benchmark +from metric_benchmark.config import load_execution +from metric_benchmark.execution import case_root, execute_benchmark, execution_identity from metric_benchmark.io import read_json, write_json -from test_metric_benchmark_config import case_definition, prepared_benchmark, write_case +from test_metric_benchmark_config import case_definition, prepared_benchmark, write_case, write_definition from test_metric_benchmark_validation import native_trial @@ -24,7 +25,7 @@ def fake_harbor(config): def test_order_full_matrix_and_verified_resume(tmp_path): path, _ = prepared_benchmark(tmp_path / "inputs") - loaded = load_benchmark(path) + loaded = load_execution(path.parent / "cases.yml", path.parent / "execution.yml") calls = [] def runner(config): @@ -54,18 +55,19 @@ def test_two_cases_produce_distinct_results_and_resume(tmp_path): from metric_benchmark.validation import validate_experiment_results path, definition = prepared_benchmark(tmp_path / "inputs") - first = definition["cases"][0] + cases = yaml.safe_load((path.parent / "cases.yml").read_text()) + first = cases[0] agent_path = path.parent / "second-agent.yml" agents = yaml.safe_load((path.parent / first["agent"]["path"]).read_text()) agents["baseline"]["agent_bundle_id"] = "second-baseline" agents["baseline"]["harbor"] = {"name": "nop"} agents["mutants"] = [{**agents["mutants"][0], "agent_bundle_id": "second-mutant", "harbor": {"name": "oracle"}}] agent_path.write_text(yaml.safe_dump(agents)) - definition["cases"].append( + cases.append( {**first, "case_id": "second-case", "agent": {"path": agent_path.name, "sha256": input_digest(agent_path)}} ) - path.write_text(yaml.safe_dump(definition)) - loaded = load_benchmark(path) + write_definition(path, "cases.yml", cases) + loaded = load_execution(path.parent / "cases.yml", path.parent / "execution.yml") output = tmp_path / "out" calls = [] @@ -85,7 +87,7 @@ def runner(config): first["case_id"]: "cases/case-0000/execution-results/results.json", "second-case": "cases/case-0001/execution-results/results.json", } - for ordinal, prepared in enumerate(loaded.cases.values()): + for ordinal, prepared in enumerate(loaded.inputs.cases.values()): raw = read_json(case_root(output, ordinal) / "results.json") assert isinstance(raw, dict) rows = raw["execution_results"] @@ -97,7 +99,7 @@ def runner(config): assert [row["metrics"]["reward"]["value"] for row in rows] == ( [1.0] * 3 + [0.0] * 6 if ordinal == 0 else [0.0] * 3 + [1.0] * 3 ) - validate_experiment_results(loaded, output) + validate_experiment_results(loaded.inputs, output) before = {p.relative_to(output): p.read_bytes() for p in output.rglob("*") if p.is_file()} assert execute_benchmark(loaded, "run", output, runner=lambda _: pytest.fail("must not replay")) == manifest assert {p.relative_to(output): p.read_bytes() for p in output.rglob("*") if p.is_file()} == before @@ -105,7 +107,7 @@ def runner(config): def test_failure_between_agents_reuses_completed_job(tmp_path): path, _ = prepared_benchmark(tmp_path / "inputs") - loaded = load_benchmark(path) + loaded = load_execution(path.parent / "cases.yml", path.parent / "execution.yml") output = tmp_path / "out" def failing(config): @@ -127,7 +129,7 @@ def resume(config): def test_partial_native_job_is_not_rerun(tmp_path): path, _ = prepared_benchmark(tmp_path / "inputs") - loaded = load_benchmark(path) + loaded = load_execution(path.parent / "cases.yml", path.parent / "execution.yml") output = tmp_path / "out" def partial(config): @@ -177,8 +179,10 @@ def test_native_harbor_cli_smoke(tmp_path): [ sys.executable, str(LAB / "run_evals.py"), - "--benchmark", - str(path), + "--cases", + str(path.parent / "cases.yml"), + "--execution", + str(path.parent / "execution.yml"), "--run-id", "docker-smoke", "--output", @@ -187,7 +191,7 @@ def test_native_harbor_cli_smoke(tmp_path): check=True, timeout=300, ) - loaded = load_benchmark(path) + loaded = load_execution(path.parent / "cases.yml", path.parent / "execution.yml") before = (case_root(output, 0) / "results.json").read_bytes() def forbidden(config): @@ -204,7 +208,12 @@ def forbidden(config): relocated_output = tmp_path / "relocated-output" shutil.move(path.parent, relocated_inputs) shutil.move(output, relocated_output) - execute_benchmark(load_benchmark(relocated_inputs / path.name), "docker-smoke", relocated_output, runner=forbidden) + execute_benchmark( + load_execution(relocated_inputs / "cases.yml", relocated_inputs / "execution.yml"), + "docker-smoke", + relocated_output, + runner=forbidden, + ) assert (case_root(relocated_output, 0) / "results.json").read_bytes() == before @@ -228,7 +237,7 @@ def test_move_completed_jobs_and_prepared_bundles_without_reexecution(tmp_path, from test_metric_benchmark_signing import remove_pins remove_pins(path.parent) - loaded = load_benchmark(path, allow_unsigned=unsigned) + loaded = load_execution(path.parent / "cases.yml", path.parent / "execution.yml", allow_unsigned=unsigned) output = original / "out" execute_benchmark(loaded, "run", output, runner=fake_harbor) before = {p.relative_to(output): p.read_bytes() for p in output.rglob("*") if p.is_file()} @@ -244,14 +253,20 @@ def forbidden(config): raise AssertionError("relocation must reuse completed native jobs") execute_benchmark( - load_benchmark(inputs / "experiment.yml", allow_unsigned=unsigned), "run", output, runner=forbidden + load_execution(inputs / "cases.yml", inputs / "execution.yml", allow_unsigned=unsigned), + "run", + output, + runner=forbidden, ) assert {p.relative_to(output): p.read_bytes() for p in output.rglob("*") if p.is_file()} == before evidence = next(output.glob("cases/*/execution-results/agents/agent-0000/run-0001/*/result.json")) evidence.write_text("{}") with pytest.raises(ValueError, match="case rho-agent, agent baseline: completed Harbor job changed"): execute_benchmark( - load_benchmark(inputs / "experiment.yml", allow_unsigned=unsigned), "run", output, runner=forbidden + load_execution(inputs / "cases.yml", inputs / "execution.yml", allow_unsigned=unsigned), + "run", + output, + runner=forbidden, ) @@ -264,7 +279,10 @@ def test_duplicate_task_sources_rejected_before_execution(tmp_path): write_case(path, case_data) with pytest.raises(ValueError, match="distinct paths"): execute_benchmark( - load_benchmark(path), "run", tmp_path / "out", runner=lambda _: pytest.fail("must not execute") + load_execution(path.parent / "cases.yml", path.parent / "execution.yml"), + "run", + tmp_path / "out", + runner=lambda _: pytest.fail("must not execute"), ) @@ -281,7 +299,7 @@ def test_logical_request_and_exact_nested_task_bindings(tmp_path): shutil.move(path.parent / sources[task_id]["path"], destination) sources[task_id]["path"] = relative write_case(path, case_data) - loaded = load_benchmark(path) + loaded = load_execution(path.parent / "cases.yml", path.parent / "execution.yml") output = tmp_path / "out" def runner(config): @@ -297,10 +315,10 @@ def runner(config): parent = case_root(output, 0) / "agents/agent-0000" request = read_json(parent / "request.json") assert request == { - "benchmark_sha256": loaded.identity, + "execution_sha256": execution_identity(loaded), "case_id": "rho-agent", "agent_bundle_id": "baseline", - "execution": loaded.definition.execution.model_dump(mode="json"), + "execution": loaded.settings.model_dump(mode="json"), } assert str(tmp_path) not in (parent / "request.json").read_text() assert str(path.parent) in (parent / "native-request.json").read_text() @@ -320,8 +338,6 @@ def runner(config): [ "agents/rho-agent/agent.yml", "tasks/workspace/tasks.yml", - "tasks/workspace/subsets.json", - "agents/rho-agent/description.json", "tasks/workspace/inspect-files/instruction.md", "bundle.txt", ], @@ -338,13 +354,115 @@ def test_unsigned_input_changes_refuse_completed_job_reuse(tmp_path, changed): write_case(path, case) remove_pins(path.parent) output = tmp_path / "out" - execute_benchmark(load_benchmark(path, allow_unsigned=True), "run", output, runner=fake_harbor) - with (path.parent / changed).open("a") as stream: - stream.write("\n") + execute_benchmark( + load_execution(path.parent / "cases.yml", path.parent / "execution.yml", allow_unsigned=True), + "run", + output, + runner=fake_harbor, + ) + if changed.endswith(".yml"): + import yaml + + target = path.parent / changed + data = yaml.safe_load(target.read_text()) + if "baseline" in data: + data["baseline"]["revision"] = "changed" + else: + data["tasks"][0]["revision"] += 1 + target.write_text(yaml.safe_dump(data)) + else: + with (path.parent / changed).open("a") as stream: + stream.write("\n") with pytest.raises(ValueError, match="changed; use a new output directory"): execute_benchmark( - load_benchmark(path, allow_unsigned=True), + load_execution(path.parent / "cases.yml", path.parent / "execution.yml", allow_unsigned=True), "run", output, runner=lambda _: pytest.fail("changed inputs must not resume old work"), ) + + +def test_analysis_changes_reuse_execution_and_preserve_all_rewards(tmp_path): + import yaml + from metric_benchmark.config import load_benchmark + from metric_benchmark.signing import sign_benchmark + + path, _ = prepared_benchmark(tmp_path / "inputs") + output = tmp_path / "out" + + def runner(config): + fake_harbor(config) + for result in (config.jobs_dir / config.job_name).glob("*/result.json"): + data = read_json(result) + assert isinstance(data, dict) + data["verifier_result"]["rewards"]["cost"] = 7.0 + result.write_text(json.dumps(data)) + + loaded = load_benchmark(path) + first = execute_benchmark(loaded.execution, "run", output, runner=runner) + before = {p.relative_to(output): p.read_bytes() for p in output.rglob("*") if p.is_file()} + analysis = yaml.safe_load((path.parent / "analysis.yml").read_text()) + analysis["scoring"] = {"metric": "cost", "direction": "lower_is_better"} + analysis["coverage"]["system_id"] = "another-system" + write_definition(path, "analysis.yml", analysis) + (path.parent / "agents/rho-agent/description.json").write_text('{"new": "hypothesis"}') + subsets = path.parent / "tasks/workspace/subsets.json" + data = read_json(subsets) + assert isinstance(data, dict) + data["eval_subsets"] = data["eval_subsets"][:1] + subsets.write_text(json.dumps(data)) + definition = yaml.safe_load(path.read_text()) + definition["revision"] += 1 + path.write_text(yaml.safe_dump(definition)) + sign_benchmark(path, resign=True) + changed = load_benchmark(path) + assert execute_benchmark(changed.execution, "run", output, runner=lambda _: pytest.fail("must reuse")) == first + assert {p.relative_to(output): p.read_bytes() for p in output.rglob("*") if p.is_file()} == before + raw = read_json(case_root(output, 0) / "results.json") + assert isinstance(raw, dict) + assert all(set(row["metrics"]) == {"reward", "cost"} for row in raw["execution_results"]) + assert changed.analysis.settings.scoring.metric == "cost" + + +@pytest.mark.parametrize("changed", [None, "bundle.txt", "tasks/workspace/inspect-files/instruction.md"]) +def test_signing_unsigned_execution_preserves_resume_and_validation(tmp_path, changed): + from metric_benchmark.io import file_manifest + from metric_benchmark.signing import sign_benchmark + from metric_benchmark.validation import validate_experiment_results + from test_metric_benchmark_signing import remove_pins + + path, _ = prepared_benchmark(tmp_path / "inputs") + bundle = path.parent / "bundle.txt" + bundle.write_text("prepared agent dependency") + case = case_definition(path) + case["baseline"]["inputs"] = {"bundle": {"path": bundle.name}} + case["baseline"]["harbor"]["kwargs"] = {"bundle": "{input:bundle}"} + write_case(path, case) + remove_pins(path.parent) + unsigned = load_execution(path.parent / "cases.yml", path.parent / "execution.yml", allow_unsigned=True) + output = tmp_path / "out" + first = execute_benchmark(unsigned, "run", output, runner=fake_harbor) + report = validate_experiment_results(unsigned.inputs, output) + before = file_manifest(output) + assert unsigned.inputs.case("rho-agent").case.baseline.inputs["bundle"].sha256 is None + if changed: + with (path.parent / changed).open("a") as stream: + stream.write("\nchanged contents\n") + sign_benchmark(path) + signed = load_execution(path.parent / "cases.yml", path.parent / "execution.yml") + assert signed.inputs.case("rho-agent").case.baseline.inputs["bundle"].sha256 is not None + + def forbidden(_): + pytest.fail("saved jobs must not execute again") + + if changed: + assert signed.inputs.identity != first["cases_sha256"] + with pytest.raises(ValueError, match="changed; use a new output directory"): + execute_benchmark(signed, "run", output, runner=forbidden) + with pytest.raises(ValueError, match="execution case definitions differ"): + validate_experiment_results(signed.inputs, output) + else: + assert signed.inputs.identity == first["cases_sha256"] + assert execute_benchmark(signed, "run", output, runner=forbidden) == first + assert validate_experiment_results(signed.inputs, output) == report + assert file_manifest(output) == before diff --git a/tests/test_metric_benchmark_signing.py b/tests/test_metric_benchmark_signing.py index b512c66f..c097e4c6 100644 --- a/tests/test_metric_benchmark_signing.py +++ b/tests/test_metric_benchmark_signing.py @@ -49,8 +49,8 @@ def test_unsigned_to_signed_library_and_cli_round_trip(tmp_path): with pytest.raises(ValueError, match="missing SHA"): load_benchmark(experiment) unsigned = load_benchmark(experiment, allow_unsigned=True) - assert unsigned.allow_unsigned - assert unsigned.case("rho-agent").allow_unsigned + assert unsigned.execution.inputs.allow_unsigned + assert unsigned.execution.inputs.case("rho-agent").allow_unsigned before = manifest_bytes(tmp_path) check = sha_cli(experiment, "--check") assert check.returncode == 2 and "missing SHA" in check.stderr @@ -58,12 +58,12 @@ def test_unsigned_to_signed_library_and_cli_round_trip(tmp_path): signed = sha_cli(experiment) assert signed.returncode == 0, signed.stderr check_benchmark(experiment) - assert not load_benchmark(experiment).allow_unsigned + assert not load_benchmark(experiment).execution.inputs.allow_unsigned assert sha_cli(experiment, "--check").returncode == 0 assert sign_benchmark(experiment, resign=True) == [] (bundle / "agent.py").write_text("print('changed agent')\n") with pytest.raises(ValueError, match="input digest differs"): - load_benchmark(experiment, allow_unsigned=True) + load_benchmark(experiment, allow_unsigned=True).execution.inputs.verify_execution_inputs() with pytest.raises(ValueError, match="input digest differs"): check_benchmark(experiment) assert sha_cli(experiment, "--resign").returncode == 0 @@ -74,14 +74,15 @@ def test_unsigned_to_signed_library_and_cli_round_trip(tmp_path): @pytest.mark.parametrize("newline", [b"\n", b"\r\n"]) def test_resign_preserves_comments_quotes_and_formatting(tmp_path, flow, newline): experiment, _ = prepared_benchmark(tmp_path) + cases_path = tmp_path / "cases.yml" if flow: - raw = experiment.read_text() + raw = cases_path.read_text() raw = re.sub( r" agent:\n path: ([^\n]+)\n sha256: ([^\n]+)\n", r""" agent: {path: '\1', sha256: "\2"} # keep inline comment\n""", raw, ) - experiment.write_text(raw) + cases_path.write_text(raw) agent_path = tmp_path / AGENT_DIR / "agent.yml" agent_path.write_text( re.sub(r"sha256: (sha256:[a-f0-9]+)", r"sha256: '\1' # keep SHA comment", agent_path.read_text()) @@ -106,7 +107,8 @@ def test_resign_preserves_comments_quotes_and_formatting(tmp_path, flow, newline @pytest.mark.parametrize("style", ["block", "flow", "null"]) def test_sign_inserts_missing_sha_without_losing_comments(tmp_path, style): experiment = unsigned_benchmark(tmp_path) - raw = experiment.read_text() + cases_path = tmp_path / "cases.yml" + raw = cases_path.read_text() if style == "flow": raw = raw.replace( " agent:\n path: agents/rho-agent/agent.yml\n", @@ -118,10 +120,10 @@ def test_sign_inserts_missing_sha_without_losing_comments(tmp_path, style): ) else: raw = raw.replace(" path: agents/rho-agent/agent.yml\n", " path: agents/rho-agent/agent.yml # retained\n") - experiment.write_text(raw) + cases_path.write_text(raw) sign_benchmark(experiment, resign=style == "null") check_benchmark(experiment) - assert "# retained" in experiment.read_text() + assert "# retained" in cases_path.read_text() assert sign_benchmark(experiment, resign=True) == [] @@ -131,8 +133,8 @@ def test_signing_refusal_and_invalid_input_leave_every_manifest_unchanged(tmp_pa result = sha_cli(experiment) assert result.returncode == 2 assert "--resign" in result.stderr - assert "experiment.yml:cases[rho-agent].agent" in result.stderr - assert "agents/rho-agent/agent.yml:description" in result.stderr + assert "cases.yml:cases[rho-agent].agent" in result.stderr + assert "analysis.yml:cases[rho-agent].description" in result.stderr assert manifest_bytes(tmp_path) == before # A later missing task directory must be discovered before any earlier pins are written. shutil.rmtree(tmp_path / TASK_DIR / "search-workspace") diff --git a/tests/test_metric_benchmark_validation.py b/tests/test_metric_benchmark_validation.py index 0ca6d364..8bae879f 100644 --- a/tests/test_metric_benchmark_validation.py +++ b/tests/test_metric_benchmark_validation.py @@ -12,11 +12,11 @@ import yaml from harbor.models.task.task import Task as HarborTask from harbor.models.trial.result import TrialResult -from metric_benchmark.config import ScoringPolicy, load_benchmark +from metric_benchmark.config import ScoringPolicy, load_cases from metric_benchmark.io import read_json, write_json from metric_benchmark.models import RawResults from metric_benchmark.validation import agent_config, task_checksum, validate_experiment_results, validate_results -from test_metric_benchmark_config import FIXTURE, LAB, case_definition, prepared_benchmark, write_case +from test_metric_benchmark_config import FIXTURE, LAB, case_definition, prepared_benchmark, write_case, write_definition RESULT_FIXTURE = FIXTURE.parent / "metric_benchmark_results" CASE_RESULTS = Path("cases/case-0000/execution-results") @@ -55,7 +55,7 @@ def result_example(tmp_path, *, bundle=False): agent["inputs"] = {"bundle": {"path": "bundle.txt", "sha256": input_digest(source)}} agent["harbor"]["kwargs"] = {"bundle": "{input:bundle}"} write_case(path, case_data) - loaded = load_benchmark(path) + loaded = load_cases(path.parent / "cases.yml") case = loaded.case("rho-agent").case root = tmp_path / "results" rows = [] @@ -119,14 +119,12 @@ def test_results_task_inventory_rejects_duplicate_ids(): @pytest.mark.parametrize("case_id", [None, "rho-agent"]) def test_contract_and_cli_parity(tmp_path, case_id): path, _ = prepared_benchmark(tmp_path / "inputs") - loaded = load_benchmark(path) + loaded = load_cases(path.parent / "cases.yml") case = loaded.case("rho-agent").case root = tmp_path / "experiment-results" shutil.copytree(RESULT_FIXTURE / "experiment-results", root) raw = RawResults.model_validate(read_json(root / CASE_RESULTS / "results.json")) - case_report = validate_results( - raw, case, root / CASE_RESULTS, loaded.case(case.case_id).root, loaded.definition.scoring - ) + case_report = validate_results(raw, case, root / CASE_RESULTS, loaded.case(case.case_id).root) report = validate_experiment_results(loaded, root, case_id=case_id) assert case_report["expected_pairs"] == 9 assert report["cases"]["rho-agent"] == case_report @@ -136,8 +134,8 @@ def test_contract_and_cli_parity(tmp_path, case_id): [ sys.executable, str(LAB / "validate_results.py"), - "--benchmark", - str(path), + "--cases", + str(path.parent / "cases.yml"), *(["--case", case_id] if case_id is not None else []), "--results-root", str(root), @@ -174,10 +172,11 @@ def test_inconsistent_matrix_or_evidence_rejected(tmp_path, change): raw.execution_results.append(raw.execution_results[0].model_copy(deep=True)) native_path.write_text(json.dumps(native)) with pytest.raises(ValueError): - validate_results(raw, case, root, loaded.case(case.case_id).root, loaded.definition.scoring) + validate_results(raw, case, root, loaded.case(case.case_id).root) -def test_explicit_missing_observation_is_not_zero(tmp_path): +@pytest.mark.parametrize("empty_metrics", [False, True]) +def test_explicit_missing_observation_is_not_zero(tmp_path, empty_metrics): loaded, case, raw, root, _ = result_example(tmp_path) row = raw.execution_results[0] row.native_results_path = "unavailable/inspect-files" @@ -185,28 +184,30 @@ def test_explicit_missing_observation_is_not_zero(tmp_path): row.metrics["reward"].value = None row.metrics["reward"].reason = "worker never started" row.unavailable_evidence = {"native_results": "worker never started"} - report = validate_results(raw, case, root, loaded.case(case.case_id).root, loaded.definition.scoring) + if empty_metrics: + row.metrics = {} + with pytest.raises(ValueError, match="missing execution needs unavailable score"): + validate_results(raw, case, root, loaded.case(case.case_id).root) + return + report = validate_results(raw, case, root, loaded.case(case.case_id).root) assert report["unavailable"][0]["reason"] == "worker never started" def test_relocated_bundle_and_native_evidence_validate_without_original_paths(tmp_path): original = tmp_path / "producer" loaded, case, raw, root, path = result_example(original, bundle=True) - before = validate_results(raw, case, root, loaded.case(case.case_id).root, loaded.definition.scoring) + before = validate_results(raw, case, root, loaded.case(case.case_id).root) moved = tmp_path / "consumer" shutil.move(original, moved) - loaded = load_benchmark(moved / "inputs/experiment.yml") + loaded = load_cases(moved / "inputs/cases.yml") case = loaded.case("rho-agent").case - assert ( - validate_results(raw, case, moved / "results", loaded.case(case.case_id).root, loaded.definition.scoring) - == before - ) + assert validate_results(raw, case, moved / "results", loaded.case(case.case_id).root) == before native_path = moved / "results" / raw.execution_results[0].native_results_path / "result.json" native = json.loads(native_path.read_text()) native["config"]["agent"]["kwargs"]["bundle"] = "/unrelated/bundle.txt" native_path.write_text(json.dumps(native)) with pytest.raises(ValueError, match="case rho-agent, agent baseline, task inspect-files.*agent configuration"): - validate_results(raw, case, moved / "results", loaded.case(case.case_id).root, loaded.definition.scoring) + validate_results(raw, case, moved / "results", loaded.case(case.case_id).root) def test_empty_agent_does_not_default_to_oracle(tmp_path): @@ -220,10 +221,7 @@ def test_executor_run_names_are_not_directory_conventions(tmp_path): loaded, case, raw, root, _ = result_example(tmp_path) for row in raw.execution_results: row.agent_run_id = "external-job:" + row.agent_bundle_id - assert ( - validate_results(raw, case, root, loaded.case(case.case_id).root, loaded.definition.scoring)["expected_pairs"] - == 9 - ) + assert validate_results(raw, case, root, loaded.case(case.case_id).root)["expected_pairs"] == 9 @pytest.mark.parametrize("unexpected", [False, True]) @@ -252,23 +250,21 @@ def test_distinct_worker_roots_are_not_experiment_identity(tmp_path): trial["config"]["task"]["path"] = producer + "/" + case.task_sources[row.evaluation.task.task_id].path trial["config"]["agent"]["kwargs"]["bundle"] = producer + "/bundle.txt" path.write_text(json.dumps(trial)) - assert ( - validate_results(raw, case, root, loaded.case(case.case_id).root, loaded.definition.scoring)["expected_pairs"] - == 9 - ) + assert validate_results(raw, case, root, loaded.case(case.case_id).root)["expected_pairs"] == 9 def test_all_cases_and_case_filter_use_the_same_root(tmp_path): path, definition = prepared_benchmark(tmp_path / "inputs") - definition["cases"].append({**definition["cases"][0], "case_id": "second-case"}) - path.write_text(yaml.safe_dump(definition, sort_keys=False)) - loaded = load_benchmark(path) + cases = yaml.safe_load((path.parent / "cases.yml").read_text()) + cases.append({**cases[0], "case_id": "second-case"}) + write_definition(path, "cases.yml", cases) + loaded = load_cases(path.parent / "cases.yml") root = tmp_path / "experiment-results" shutil.copytree(RESULT_FIXTURE / "experiment-results", root) second = Path("cases/case-0001/execution-results") shutil.copytree(root / CASE_RESULTS, root / second) manifest = json.loads((root / "execution.json").read_text()) - manifest["benchmark_sha256"] = loaded.identity + manifest["cases_sha256"] = loaded.identity manifest["cases"]["second-case"] = (second / "results.json").as_posix() (root / "execution.json").write_text(json.dumps(manifest)) report = validate_experiment_results(loaded, root) @@ -311,8 +307,8 @@ def test_invalid_experiment_index_rejected_without_report(tmp_path, change): [ sys.executable, str(LAB / "validate_results.py"), - "--benchmark", - str(path), + "--cases", + str(path.parent / "cases.yml"), "--results-root", str(root), "--output",