From 4610543fc315261b67d2f96b00c0eb83552fe899 Mon Sep 17 00:00:00 2001 From: mstaats Date: Tue, 20 Oct 2026 16:46:40 +0000 Subject: [PATCH 1/5] docs: draft external metric benchmark architecture Signed-off-by: mstaats --- labs/metric-benchmark/docs/README.md | 30 ++++ labs/metric-benchmark/docs/architecture.md | 171 ++++++++++++++++++ labs/metric-benchmark/docs/stages.md | 195 +++++++++++++++++++++ 3 files changed, 396 insertions(+) create mode 100644 labs/metric-benchmark/docs/README.md create mode 100644 labs/metric-benchmark/docs/architecture.md create mode 100644 labs/metric-benchmark/docs/stages.md diff --git a/labs/metric-benchmark/docs/README.md b/labs/metric-benchmark/docs/README.md new file mode 100644 index 00000000..f41a4a6a --- /dev/null +++ b/labs/metric-benchmark/docs/README.md @@ -0,0 +1,30 @@ + + + +# Metric benchmark + +**Draft for review.** This directory describes an external benchmark for testing +whether Eval Author's coverage measurements predict how well an evaluation set +detects deliberately weakened agents. It contains a proposed design, not an +implemented runner or a completed experiment. + +Start with the [architecture](architecture.md) for the supplied inputs, component +boundaries, and experiment flow. The [stage descriptions](stages.md) explain +execution, scoring, and completion criteria. + +The benchmark comes with prepared case examples: a baseline agent, its mutants, +and an associated task set. Building those inputs happens before a benchmark +release. A run starts with those inputs and an explicit experiment configuration, +executes the agent-by-task matrix, then analyzes the resulting evidence. + +The intended runner is deterministic program code. The distinction between +deterministic orchestration and repeatable model outputs remains explicit in the +[determinism requirements](architecture.md#determinism). + +This draft adapts the metric efficacy experiment design from the fixtures +repository, as present at revision `f1f72b7a6a5085f9e369905417b3171f30879b92`. +It retains the scientific question and the execution-to-analysis artifact flow. +Preparation, publication, service-specific identifiers, and internal execution +infrastructure are outside this version's runtime design. The +[open decisions](architecture.md#open-decisions) identify where the external +benchmark contract still needs work. diff --git a/labs/metric-benchmark/docs/architecture.md b/labs/metric-benchmark/docs/architecture.md new file mode 100644 index 00000000..df505149 --- /dev/null +++ b/labs/metric-benchmark/docs/architecture.md @@ -0,0 +1,171 @@ + + + +# Architecture + +## Scientific question + +Does a coverage measurement predict how effectively an evaluation set detects +agent degradation? + +For each supplied case, run the baseline agent and a fixed cohort of mutants +against the same benchmark tasks. Construct several subsets of those tasks. +For each subset, measure coverage from the baseline's traces and sensitivity to +mutants from the baseline and mutant outcomes. Correlate those paired scores +across subsets. + +Compare coverage definitions using the same subsets, mutant cohort, and outcome +scoring policy. Higher mutant scores mean greater detected degradation, so the +comparison seeks a positive relationship with coverage. Correctness and resource +use are separate outcome variations. Report results per case and variation; +combining cases into a leaderboard is an open decision. + +## Supplied inputs + +A benchmark release provides prepared case packages. Their exact packaging format +is not yet specified; the following are logical contents, not a file schema. + +| Input | Required contents | +| --- | --- | +| Case description | Stable case identity, intended agent behavior, and its associated benchmark | +| Baseline agent | An executable bundle with an immutable identity and a documented invocation contract | +| Mutants | Executable bundles with distinct identities, their baseline relationship, and mutation descriptions | +| Benchmark tasks | Fixed task membership, task identities, environments, and correctness verifiers | +| Coverage rubric | A frozen definition of the coverage items and denominator for each selected metric configuration | +| Experiment configuration | Case selection, fixed mutant cohort, metric implementation versions, execution settings, subset construction, and scoring parameters | + +Baseline and mutant bundles are prepared and checked before release. Any +qualification that determines the released cohort also happens before release; +the benchmark run does not select mutants based on its observed scores. +Tasks and their environments are supplied ready for the selected execution +backend. The rubric is treated as another supplied input in this draft, so rubric +generation is outside the measured run. + +The configuration records all effective settings, including defaults. Each run +preserves the exact configuration and input identities. A changed bundle, task +set, rubric, or metric version defines a different comparison. + +## Experiment flow + +The preparation portion of the original experiment becomes a supplied-input +boundary. The runnable experiment has two parts: **Running Evals** and +**Statistical Analysis**. + +```mermaid +flowchart TD + CASE["Supplied case: baseline and mutant bundles"] + TASKS["Supplied benchmark tasks and verifiers"] + RUBRIC["Supplied frozen rubric"] + CONFIG["Experiment configuration"] + VALIDATE["Validate inputs and configuration"] + + CASE --> VALIDATE + TASKS --> VALIDATE + RUBRIC --> VALIDATE + CONFIG --> VALIDATE + + subgraph EXEC["Running Evals"] + RUN["Run Eval Matrix"] + COLLECT["Collect Eval Results"] + RAW["Raw execution results, traces, and task inventory"] + RUN --> COLLECT --> RAW + end + + VALIDATE --> RUN + + subgraph ANALYSIS["Statistical Analysis"] + SUBSETS["Eval Subset Construction"] + MUTANT["Compute Mutant Scores"] + COVERAGE["Compute Coverage"] + CORRELATE["Spearman Correlation"] + RECORD["Record Experiment"] + SUBSETS --> MUTANT + SUBSETS --> COVERAGE + MUTANT --> CORRELATE + COVERAGE --> CORRELATE + CORRELATE --> RECORD + end + + RAW -->|"task inventory only"| SUBSETS + RAW -->|"baseline and mutant outcomes"| MUTANT + RAW -->|"baseline traces only"| COVERAGE + RUBRIC --> COVERAGE + CONFIG -.-> SUBSETS + CONFIG -.-> MUTANT + CONFIG -.-> COVERAGE + CONFIG -.-> CORRELATE +``` + +Record Experiment retains the inputs and every stage's outputs, including raw +execution evidence. The diagram omits those archive edges for readability. +Subset construction uses task identities and the configured sampling rule, never +observed rewards or coverage. Coverage measurement receives baseline evidence; +mutant outcomes are available only to mutant scoring and the final analysis. + +## Component responsibilities + +| Component | Responsibility | +| --- | --- | +| Case package | Supply the prepared baseline, mutants, tasks, and case metadata | +| Configuration validator | Check input identities, declared relationships, settings, and required capabilities before execution | +| Runner | Execute the fixed stage sequence, validate outputs, and record stage completion | +| Execution adapter | Invoke the supplied agents and tasks through the selected backend and preserve attempts and outcomes | +| Results collector | Produce a common results index while retaining native traces and verifier evidence | +| Metric implementation | Measure baseline traces against the supplied coverage definition | +| Analysis functions | Construct subsets, compute mutant scores, join score dictionaries, and compute correlations | +| Experiment record | Preserve the configuration, input identities, evidence, scores, exclusions, and completion status | + +The execution adapter is a portability boundary. This design requires no +particular hosted service, registry, deployment, or orchestration product. +Supporting a backend requires defining and testing its invocation and evidence +contract; the adapter box alone does not establish portability. + +Owned configuration and artifact formats should use shared, versioned data +models with one validation and serialization implementation. Native trace and +task formats retain their established readers. Concrete schemas and interfaces +are not defined by this documentation. + +## Determinism + +The runner uses explicit code and configuration for stage ordering, retries, +completion checks, and analysis. No coordinating language model chooses the +next step, changes scoring rules, or repairs an experiment during execution. + +For fixed input evidence, subset membership and numerical results must be +reproducible. This requires stable task ordering, a specified sampling algorithm +and seed or supplied memberships, fixed scoring rules, pinned implementations, +and explicit treatment of missing values and ties. Completion order must not +change which attempts or observations are scored. + +Live agent calls and model-based coverage judgments can still vary. Pinning a +model name, setting a seed, or choosing temperature zero does not by itself +establish identical results. Wall-clock measurements can also vary between runs. +The draft therefore does not yet promise full reproducibility of fresh live +executions. + +To support an end-to-end deterministic claim, we must choose whether the benchmark +requires deterministic agents and metric implementations, replays supplied +execution and judgment evidence, or defines determinism only for orchestration +and analysis over frozen observations. Replay would test analysis +reproducibility; by itself it would not test new live agent behavior or new +model-generated judgments. + +## Open decisions + +- **Determinism scope:** choose the guarantee for live execution, coverage + judgments, and replay before describing the benchmark as fully deterministic. +- **Distribution:** choose the case package format, bundle invocation contract, + supported task format, and initial execution backend. +- **Rubric ownership:** confirm that frozen rubrics ship with case packages, + or define how independently supplied metric configurations bind to them. +- **Execution policy:** specify repetitions, seeds, timeouts, retry limits, + attempt selection, and treatment of incomplete agent-task pairs. +- **Coverage interface:** define how a metric consumes baseline traces, which + evidence is required, and whether it may make model calls. +- **Scoring:** choose the primary mutant score, mutant weighting, coverage + aggregation, resource measurements, and treatment of partially observed subsets. +- **Statistical reporting:** choose subset sizes and overlap constraints, + uncertainty estimates that account for shared tasks, and any cross-case summary. + +These choices are intentionally unresolved. No command-line interface, public +dataset release, or runnable experiment is implied by this draft. diff --git a/labs/metric-benchmark/docs/stages.md b/labs/metric-benchmark/docs/stages.md new file mode 100644 index 00000000..caee29da --- /dev/null +++ b/labs/metric-benchmark/docs/stages.md @@ -0,0 +1,195 @@ + + + +# Experiment stages + +This draft starts with the prepared inputs defined in the +[architecture](architecture.md#supplied-inputs). Agent bundling, mutant generation +and qualification, benchmark construction and publication, and rubric generation +are release preparation rather than stages of a benchmark run. + +Before starting, validate the declared input identities, task inventory, baseline +and mutant relationships, frozen rubric, and configuration. Check that the chosen +backend can invoke the bundles and retain the evidence needed by the configured +metrics. Validation does not execute agents or establish benchmark quality. + +## Stage contract + +Each stage has declared inputs, fixed parameters, outputs, and completion +criteria. Outputs become available downstream only after validation. +The program controls transitions and handles failures according to configured +rules. An execution error must remain distinct from a legitimate low reward. + +Resume at completed stage boundaries. An incomplete stage can reconcile its +recorded attempts when invoked again, but must not blindly duplicate executions +or pick the most favorable attempt. Changing an input invalidates dependent +outputs; earlier observations remain in the record under their original identities. + +The following descriptions specify behavior, not implemented APIs. + +## Running Evals + +### 1. Run Eval Matrix + +**Purpose:** execute every task against the baseline and every supplied mutant +selected in the configuration. + +**Inputs:** prepared agent bundles, benchmark tasks, and execution settings. + +**Operation:** enumerate the agent-by-task matrix, including configured +repetitions. Execute it through the selected backend. Use the same task and +verifier versions and comparable execution settings for baseline and mutants; +record intentional differences that define a mutation. Apply a fixed retry +policy to execution failures without retrying legitimate agent failures to +improve their scores. + +**Outputs:** an execution manifest identifying each requested pairing and its +attempts, status, and evidence locations. These are portable experiment +identities; backend-specific metadata can accompany them. + +**Completion:** every requested pairing and repetition is accounted for, with no +pending or unresolved attempts. A completed execution can have a failing task +reward. Missing execution evidence is not a zero reward or a completed result. +The treatment of exhausted execution failures must be fixed in the run policy. + +### 2. Collect Eval Results + +**Purpose:** make execution evidence available in a consistent form for analysis. + +**Inputs:** the execution manifest and the native outputs of the backend. + +**Operation:** collect results and traces from their recorded locations. Preserve +native evidence and produce an index keyed by case, agent, task, repetition, +and attempt. Record correctness and any available resource measurements, their +units, and evidence locations. Verify the recorded task inventory against the +supplied benchmark. + +**Outputs:** Raw Execution Results and Task References. The shared index locates +native evidence rather than replacing it. + +**Completion:** all requested executions are represented, evidence can be read +and its identity checked, and measurements or traces absent from the source are +explicitly marked. Collection failures leave the stage incomplete. The collector +does not manufacture measurements or turn missing data into agent failures. + +## Statistical Analysis + +### 3. Eval Subset Construction + +**Purpose:** construct simulated evaluation sets from the benchmark task inventory. + +**Inputs:** Task References and the configured subset count, sizes, sampling +algorithm, seed, and constraints, or explicit supplied memberships. + +**Operation:** select task identities without consulting outcomes, resource +measurements, or coverage scores. Preserve exact memberships so subsequent +analysis need not resample. + +**Outputs:** Eval Subset Definitions with stable subset IDs, source benchmark +identity, membership, and construction parameters. + +**Completion:** all memberships match the requested count, sizes, and constraints, +and every task belongs to the supplied benchmark. An impossible sampling request +fails explicitly rather than silently relaxing its constraints. + +### 4. Compute Mutant Scores + +**Purpose:** measure each subset's sensitivity to the supplied mutations. + +**Inputs:** Eval Subset Definitions, Raw Execution Results for the baseline and +mutants, the fixed mutant cohort, and the scoring policy. + +**Operation:** compare baseline and mutant outcomes on identical task memberships. +Compute one score per subset and scoring variation. Correctness and efficiency +remain separate variations; their units and score direction must be explicit. + +A candidate correctness score inherited from the original experiment is the mean +signed reward loss: + +```text +S(agent, subset) = mean task reward for that agent on that subset +M(subset) = mean over selected mutants of + [S(baseline, subset) - S(mutant, subset)] +``` + +This candidate weights mutants equally and preserves negative differences when +a mutant outperforms the baseline. It is an example policy, not a finalized +default. Repetition aggregation, incomplete evidence, and any alternative +weighting must be specified before use. + +**Outputs:** Mutant Scores, keyed by subset ID, with the case, baseline, mutant +cohort, aggregation, and scoring variation identified. Retain per-mutant +comparisons as supporting evidence. + +**Completion:** every requested subset has a score or an explicit unavailable +value and reason. Never silently change task membership or mutant composition +to obtain a computable score. + +### 5. Compute Coverage + +**Purpose:** measure coverage for the same subsets using baseline behavior. + +**Inputs:** Eval Subset Definitions, baseline traces from Raw Execution Results, +the frozen rubric, and the versioned metric implementation and settings. + +**Operation:** measure baseline traces against the rubric and aggregate per-task +measurements for each subset. Reuse identical per-task measurements across +overlapping subsets within a metric configuration. Keep the rubric and +denominator fixed within the comparison. Mutant outcomes and correlation results +are not inputs to this stage. + +**Outputs:** Coverage Scores, keyed by subset ID, with rubric and metric +identities and supporting per-task measurements or judgments. + +**Completion:** every requested subset and metric configuration has a score or +an explicit unavailable value and reason. If measurement requires model-generated +judgments, their reproducibility must satisfy the chosen +[determinism contract](architecture.md#determinism); deterministic aggregation +alone does not make those judgments deterministic. + +### 6. Spearman Correlation + +**Purpose:** determine whether coverage predicts mutation sensitivity across +evaluation subsets. + +**Inputs:** Coverage Scores and Mutant Scores with matching subset definitions. + +**Operation:** verify matching case, baseline, benchmark, and subset identities. +Join scores by subset ID and compute Spearman rank correlation separately for +each metric configuration and mutant-score variation, using average ranks for +ties. Retain each paired score and every exclusion with its reason. + +**Outputs:** Correlation Results containing the coefficient, number of included +subsets, paired scores, configuration identities, and exclusions. + +**Completion:** each comparison has a result or an explicit undefined result. +Constant scores or too few usable pairs do not become zero correlation. +Unexpected identity or membership mismatches are errors, not silently dropped +observations. Any allowed exclusion policy must be specified in advance. + +Overlapping subsets share observations. Treating them as independent samples +would overstate the strength of evidence. The uncertainty method remains an +open decision; an observed coefficient alone is not a general validity claim. + +### 7. Record Experiment + +**Purpose:** preserve enough evidence to inspect and reproduce the analysis. + +**Inputs:** the frozen configuration, supplied input identities, and all stage +outputs, including the paired scores and correlation results. + +**Operation:** retain an inventory of inputs, execution attempts, native evidence, +subset memberships, scores, exclusions, implementation versions, and completion +status. A rerun creates a distinct record linked to its reused inputs. + +**Outputs:** an Experiment Record containing the inventory and analysis report, +with evidence stored or referenced according to the eventual distribution format. + +**Completion:** every input and output is accounted for and accessible to the +intended reader. Unavailable values and failed attempts retain their explanations. +Partial runs should retain diagnostic records, but must not be reported as +completed benchmark results. + +Mutant scoring and coverage measurement can run independently after subset +construction. Their results meet at correlation. None of the statistical +analysis stages reruns the benchmark agents. From b321d2dba4b8b732cb488e907bf2056743f9adc5 Mon Sep 17 00:00:00 2001 From: mstaats Date: Tue, 20 Oct 2026 17:04:49 +0000 Subject: [PATCH 2/5] docs: make coverage the output of the system under evaluation Signed-off-by: mstaats --- labs/metric-benchmark/docs/README.md | 15 ++-- labs/metric-benchmark/docs/architecture.md | 100 ++++++++++++++------- labs/metric-benchmark/docs/stages.md | 80 +++++++++++------ 3 files changed, 128 insertions(+), 67 deletions(-) diff --git a/labs/metric-benchmark/docs/README.md b/labs/metric-benchmark/docs/README.md index f41a4a6a..8f10d47a 100644 --- a/labs/metric-benchmark/docs/README.md +++ b/labs/metric-benchmark/docs/README.md @@ -4,7 +4,7 @@ # Metric benchmark **Draft for review.** This directory describes an external benchmark for testing -whether Eval Author's coverage measurements predict how well an evaluation set +whether a system's coverage measurements predict how well an evaluation set detects deliberately weakened agents. It contains a proposed design, not an implemented runner or a completed experiment. @@ -12,10 +12,15 @@ Start with the [architecture](architecture.md) for the supplied inputs, componen boundaries, and experiment flow. The [stage descriptions](stages.md) explain execution, scoring, and completion criteria. -The benchmark comes with prepared case examples: a baseline agent, its mutants, -and an associated task set. Building those inputs happens before a benchmark -release. A run starts with those inputs and an explicit experiment configuration, -executes the agent-by-task matrix, then analyzes the resulting evidence. +The supplied inputs are a baseline agent, its mutants, a case description, +benchmark tasks, and an experiment configuration. Building the case packages +happens before a benchmark release. The system under evaluation is supplied +separately: Eval Author or another system that measures coverage. + +A run executes tasks against the baseline and mutants, obtains coverage scores +from the system under evaluation, and analyzes coverage versus mutant scores +across task subsets. Coverage scores and any rubric the system generates are +experiment outputs, not supplied benchmark inputs. The intended runner is deterministic program code. The distinction between deterministic orchestration and repeatable model outputs remains explicit in the diff --git a/labs/metric-benchmark/docs/architecture.md b/labs/metric-benchmark/docs/architecture.md index df505149..2312d008 100644 --- a/labs/metric-benchmark/docs/architecture.md +++ b/labs/metric-benchmark/docs/architecture.md @@ -10,9 +10,11 @@ agent degradation? For each supplied case, run the baseline agent and a fixed cohort of mutants against the same benchmark tasks. Construct several subsets of those tasks. -For each subset, measure coverage from the baseline's traces and sensitivity to -mutants from the baseline and mutant outcomes. Correlate those paired scores -across subsets. +For each subset, ask the system under evaluation to measure coverage using the +case description and permitted baseline evidence. Measure sensitivity to mutants +from the baseline and mutant outcomes. Correlate those paired scores across +subsets. The system under evaluation may be Eval Author or another coverage +system. Compare coverage definitions using the same subsets, mutant cohort, and outcome scoring policy. Higher mutant scores mean greater detected degradation, so the @@ -22,28 +24,43 @@ combining cases into a leaderboard is an open decision. ## Supplied inputs -A benchmark release provides prepared case packages. Their exact packaging format -is not yet specified; the following are logical contents, not a file schema. +A run receives five supplied inputs. Case packages provide the first four; +the experiment configuration specifies how to use them. Their exact packaging +format is not yet specified; these are logical contents, not a file schema. | Input | Required contents | | --- | --- | -| Case description | Stable case identity, intended agent behavior, and its associated benchmark | | Baseline agent | An executable bundle with an immutable identity and a documented invocation contract | | Mutants | Executable bundles with distinct identities, their baseline relationship, and mutation descriptions | +| Case description | Stable case identity and intended agent behavior; an Ethos is one possible representation | | Benchmark tasks | Fixed task membership, task identities, environments, and correctness verifiers | -| Coverage rubric | A frozen definition of the coverage items and denominator for each selected metric configuration | -| Experiment configuration | Case selection, fixed mutant cohort, metric implementation versions, execution settings, subset construction, and scoring parameters | +| Experiment configuration | Case-to-benchmark associations, fixed mutant cohort, execution settings, subset construction, coverage-system settings and permitted evidence, and scoring parameters | + +### System under evaluation + +In addition to those inputs, a run receives the versioned system whose coverage +measurements are being evaluated: Eval Author or another implementation. Record +its version, dependencies, model configuration when applicable, and effective +settings independently of the case package. Different systems can be compared +against the same cases, execution evidence, subsets, and mutant-scoring policy. + +Coverage is produced during the experiment. A system that uses a rubric may +generate it as part of its coverage workflow; a rubric is not required of every +system. Retain any generated rubric, judgments, and other measurement artifacts +with the scores. The benchmark supplies the case description as context, not a +precomputed coverage answer. Whether that description must follow the Ethos +format remains open. Baseline and mutant bundles are prepared and checked before release. Any qualification that determines the released cohort also happens before release; the benchmark run does not select mutants based on its observed scores. Tasks and their environments are supplied ready for the selected execution -backend. The rubric is treated as another supplied input in this draft, so rubric -generation is outside the measured run. +backend. The configuration records all effective settings, including defaults. Each run preserves the exact configuration and input identities. A changed bundle, task -set, rubric, or metric version defines a different comparison. +set, case description, or coverage-system version defines a different comparison. +Generated coverage artifacts are recorded as outputs with their input identities. ## Experiment flow @@ -53,16 +70,18 @@ boundary. The runnable experiment has two parts: **Running Evals** and ```mermaid flowchart TD - CASE["Supplied case: baseline and mutant bundles"] + AGENTS["Supplied baseline and mutant bundles"] + DESCRIPTION["Supplied case description"] TASKS["Supplied benchmark tasks and verifiers"] - RUBRIC["Supplied frozen rubric"] CONFIG["Experiment configuration"] + SYSTEM["System under evaluation: Eval Author or another coverage system"] VALIDATE["Validate inputs and configuration"] - CASE --> VALIDATE + AGENTS --> VALIDATE + DESCRIPTION --> VALIDATE TASKS --> VALIDATE - RUBRIC --> VALIDATE CONFIG --> VALIDATE + SYSTEM --> VALIDATE subgraph EXEC["Running Evals"] RUN["Run Eval Matrix"] @@ -76,11 +95,13 @@ flowchart TD subgraph ANALYSIS["Statistical Analysis"] SUBSETS["Eval Subset Construction"] MUTANT["Compute Mutant Scores"] - COVERAGE["Compute Coverage"] + COVERAGE["Compute Coverage with the system under evaluation"] + EVIDENCE["Coverage artifacts: rubric if used, judgments, and evidence"] CORRELATE["Spearman Correlation"] RECORD["Record Experiment"] SUBSETS --> MUTANT SUBSETS --> COVERAGE + COVERAGE --> EVIDENCE MUTANT --> CORRELATE COVERAGE --> CORRELATE CORRELATE --> RECORD @@ -89,7 +110,8 @@ flowchart TD RAW -->|"task inventory only"| SUBSETS RAW -->|"baseline and mutant outcomes"| MUTANT RAW -->|"baseline traces only"| COVERAGE - RUBRIC --> COVERAGE + DESCRIPTION --> COVERAGE + SYSTEM --> COVERAGE CONFIG -.-> SUBSETS CONFIG -.-> MUTANT CONFIG -.-> COVERAGE @@ -99,21 +121,25 @@ flowchart TD Record Experiment retains the inputs and every stage's outputs, including raw execution evidence. The diagram omits those archive edges for readability. Subset construction uses task identities and the configured sampling rule, never -observed rewards or coverage. Coverage measurement receives baseline evidence; -mutant outcomes are available only to mutant scoring and the final analysis. +observed rewards or coverage. Coverage measurement receives the case description +and explicitly permitted baseline evidence, never mutant outcomes or correlation +results. The precise access to agent source, task contents, traces, and baseline +verifier results must be fixed in the configuration and held comparable across +systems. Passing a path to the entire results directory would not enforce that +boundary. ## Component responsibilities | Component | Responsibility | | --- | --- | | Case package | Supply the prepared baseline, mutants, tasks, and case metadata | -| Configuration validator | Check input identities, declared relationships, settings, and required capabilities before execution | +| Configuration validator | Check input and system identities, declared relationships, settings, and required capabilities before execution | | Runner | Execute the fixed stage sequence, validate outputs, and record stage completion | | Execution adapter | Invoke the supplied agents and tasks through the selected backend and preserve attempts and outcomes | | Results collector | Produce a common results index while retaining native traces and verifier evidence | -| Metric implementation | Measure baseline traces against the supplied coverage definition | +| System under evaluation | Produce coverage scores from the case description and permitted baseline evidence, including rubric generation if its method requires it | | Analysis functions | Construct subsets, compute mutant scores, join score dictionaries, and compute correlations | -| Experiment record | Preserve the configuration, input identities, evidence, scores, exclusions, and completion status | +| Experiment record | Preserve the configuration, input and system identities, execution evidence, generated coverage artifacts, scores, exclusions, and completion status | The execution adapter is a portability boundary. This design requires no particular hosted service, registry, deployment, or orchestration product. @@ -131,24 +157,26 @@ The runner uses explicit code and configuration for stage ordering, retries, completion checks, and analysis. No coordinating language model chooses the next step, changes scoring rules, or repairs an experiment during execution. -For fixed input evidence, subset membership and numerical results must be -reproducible. This requires stable task ordering, a specified sampling algorithm +For fixed execution evidence and recorded coverage outputs, subset membership +and statistical results must be reproducible. This requires stable task ordering, +a specified sampling algorithm and seed or supplied memberships, fixed scoring rules, pinned implementations, and explicit treatment of missing values and ties. Completion order must not change which attempts or observations are scored. -Live agent calls and model-based coverage judgments can still vary. Pinning a -model name, setting a seed, or choosing temperature zero does not by itself +Live agent calls, generated rubrics, and model-based coverage judgments can still +vary. Pinning a model name, setting a seed, or choosing temperature zero does not by itself establish identical results. Wall-clock measurements can also vary between runs. The draft therefore does not yet promise full reproducibility of fresh live executions. To support an end-to-end deterministic claim, we must choose whether the benchmark -requires deterministic agents and metric implementations, replays supplied -execution and judgment evidence, or defines determinism only for orchestration +requires deterministic agents and coverage systems, replays previously recorded +execution and coverage evidence, or defines determinism only for orchestration and analysis over frozen observations. Replay would test analysis reproducibility; by itself it would not test new live agent behavior or new -model-generated judgments. +model-generated rubrics and judgments. Replaying an earlier coverage score does +not evaluate a new version of the coverage system. ## Open decisions @@ -156,16 +184,22 @@ model-generated judgments. judgments, and replay before describing the benchmark as fully deterministic. - **Distribution:** choose the case package format, bundle invocation contract, supported task format, and initial execution backend. -- **Rubric ownership:** confirm that frozen rubrics ship with case packages, - or define how independently supplied metric configurations bind to them. +- **Case description:** decide whether an Ethos is required or whether other + descriptions of intended agent behavior are accepted. - **Execution policy:** specify repetitions, seeds, timeouts, retry limits, attempt selection, and treatment of incomplete agent-task pairs. -- **Coverage interface:** define how a metric consumes baseline traces, which - evidence is required, and whether it may make model calls. +- **Coverage interface and evidence:** define the common score interface, which + baseline artifacts each system can see, and whether it can inspect the full + benchmark or only the subset being scored. For systems that generate a rubric, + specify its generation scope and reuse across subsets. Keep mutant outcomes + and correlation results inaccessible to coverage computation. - **Scoring:** choose the primary mutant score, mutant weighting, coverage aggregation, resource measurements, and treatment of partially observed subsets. - **Statistical reporting:** choose subset sizes and overlap constraints, uncertainty estimates that account for shared tasks, and any cross-case summary. +- **Meaning of mutant score:** mutation sensitivity is a proxy for evaluation + quality. Baseline failures, irrelevant mutants, and ceiling or floor effects + can weaken that proxy even when the execution and analysis are correct. These choices are intentionally unresolved. No command-line interface, public dataset release, or runnable experiment is implied by this draft. diff --git a/labs/metric-benchmark/docs/stages.md b/labs/metric-benchmark/docs/stages.md index caee29da..57fad571 100644 --- a/labs/metric-benchmark/docs/stages.md +++ b/labs/metric-benchmark/docs/stages.md @@ -5,13 +5,16 @@ This draft starts with the prepared inputs defined in the [architecture](architecture.md#supplied-inputs). Agent bundling, mutant generation -and qualification, benchmark construction and publication, and rubric generation -are release preparation rather than stages of a benchmark run. +and qualification, and benchmark construction and publication are release +preparation rather than stages of a benchmark run. The system under evaluation +is a separate input. Coverage computation, including rubric generation when the +system uses it, is part of the experiment. Before starting, validate the declared input identities, task inventory, baseline -and mutant relationships, frozen rubric, and configuration. Check that the chosen -backend can invoke the bundles and retain the evidence needed by the configured -metrics. Validation does not execute agents or establish benchmark quality. +and mutant relationships, case description, system version, and configuration. +Check that the chosen backend can invoke the bundles and retain the evidence +needed by the system under evaluation. Validation does not execute agents or +establish benchmark quality. ## Stage contract @@ -127,25 +130,42 @@ to obtain a computable score. ### 5. Compute Coverage -**Purpose:** measure coverage for the same subsets using baseline behavior. - -**Inputs:** Eval Subset Definitions, baseline traces from Raw Execution Results, -the frozen rubric, and the versioned metric implementation and settings. - -**Operation:** measure baseline traces against the rubric and aggregate per-task -measurements for each subset. Reuse identical per-task measurements across -overlapping subsets within a metric configuration. Keep the rubric and -denominator fixed within the comparison. Mutant outcomes and correlation results -are not inputs to this stage. - -**Outputs:** Coverage Scores, keyed by subset ID, with rubric and metric -identities and supporting per-task measurements or judgments. - -**Completion:** every requested subset and metric configuration has a score or -an explicit unavailable value and reason. If measurement requires model-generated -judgments, their reproducibility must satisfy the chosen +**Purpose:** obtain the system under evaluation's coverage scores for the same +subsets. + +**Inputs:** the case description, Eval Subset Definitions, permitted baseline +evidence from Raw Execution Results, and the versioned system under evaluation +with its settings. Additional inputs such as agent source or task contents must +be explicitly allowed by the configured evidence policy. + +**Operation:** invoke Eval Author or another coverage system through its declared +interface to produce scores. If that system needs a rubric, let it generate the +rubric within this stage using only permitted inputs, and retain the resulting +artifact. Rubric generation and measurement are parts of the system being tested, +not assumed benchmark answers. + +For a method that uses one case-level rubric, freeze its generated contents and +denominator before scoring subsets and reuse them within that comparison. +A method that derives its coverage definition separately for each subset must +declare that behavior and retain each definition; it must not be silently treated +as a shared-rubric measurement. The allowed generation scope is an open protocol +decision. + +Reuse per-task measurements across overlapping subsets only when the system's +method makes them independent of the other tasks in the subset. Systems that +score whole subsets can instead produce scores directly. Mutant outcomes, +mutant scores, and correlation results are never inputs to coverage computation. + +**Outputs:** Coverage Scores, keyed by subset ID, with system and configuration +identities and supporting measurement evidence. Retain any generated rubrics, +per-task measurements, and judgments with their exact inputs and versions. + +**Completion:** every requested subset and system configuration has a score or +an explicit unavailable value and reason. Generated coverage artifacts are +accounted for. If computation generates rubrics or judgments with a model, +their reproducibility must satisfy the chosen [determinism contract](architecture.md#determinism); deterministic aggregation -alone does not make those judgments deterministic. +alone does not make those outputs deterministic. ### 6. Spearman Correlation @@ -156,8 +176,8 @@ evaluation subsets. **Operation:** verify matching case, baseline, benchmark, and subset identities. Join scores by subset ID and compute Spearman rank correlation separately for -each metric configuration and mutant-score variation, using average ranks for -ties. Retain each paired score and every exclusion with its reason. +each coverage-system configuration and mutant-score variation, using average +ranks for ties. Retain each paired score and every exclusion with its reason. **Outputs:** Correlation Results containing the coefficient, number of included subsets, paired scores, configuration identities, and exclusions. @@ -175,12 +195,14 @@ open decision; an observed coefficient alone is not a general validity claim. **Purpose:** preserve enough evidence to inspect and reproduce the analysis. -**Inputs:** the frozen configuration, supplied input identities, and all stage -outputs, including the paired scores and correlation results. +**Inputs:** the frozen configuration, supplied input and system identities, and +all stage outputs, including generated coverage artifacts, paired scores, and +correlation results. **Operation:** retain an inventory of inputs, execution attempts, native evidence, -subset memberships, scores, exclusions, implementation versions, and completion -status. A rerun creates a distinct record linked to its reused inputs. +subset memberships, generated rubrics and judgments when present, scores, +exclusions, implementation versions, and completion status. A rerun creates a +distinct record linked to its reused inputs. **Outputs:** an Experiment Record containing the inventory and analysis report, with evidence stored or referenced according to the eventual distribution format. From 180ab442e37960b8c78bc9027420f22f155a5534 Mon Sep 17 00:00:00 2001 From: mstaats Date: Wed, 21 Oct 2026 20:21:38 +0000 Subject: [PATCH 3/5] docs(metric-benchmark): define architecture and fixed benchmark contract Signed-off-by: mstaats --- labs/metric-benchmark/docs/README.md | 57 ++- labs/metric-benchmark/docs/architecture.md | 379 ++++++++-------- .../docs/assets/experiment-flow.svg | 134 ++++++ .../docs/assets/score-analysis.svg | 142 ++++++ labs/metric-benchmark/docs/stages.md | 421 +++++++++++------- 5 files changed, 775 insertions(+), 358 deletions(-) create mode 100644 labs/metric-benchmark/docs/assets/experiment-flow.svg create mode 100644 labs/metric-benchmark/docs/assets/score-analysis.svg diff --git a/labs/metric-benchmark/docs/README.md b/labs/metric-benchmark/docs/README.md index 8f10d47a..d878cbed 100644 --- a/labs/metric-benchmark/docs/README.md +++ b/labs/metric-benchmark/docs/README.md @@ -3,33 +3,62 @@ # Metric benchmark -**Draft for review.** This directory describes an external benchmark for testing -whether a system's coverage measurements predict how well an evaluation set -detects deliberately weakened agents. It contains a proposed design, not an -implemented runner or a completed experiment. +**Draft for review.** This directory specifies the architecture of an external +benchmark for testing whether coverage measurements predict how well an +evaluation set detects deliberately weakened agents. This base PR contains the +design and figures. Separate implementation PRs build the stage tools on top: +[subset preparation (#94)](https://github.com/NVIDIA-NeMo/eval-author/pull/94), +[mutant scoring (#95)](https://github.com/NVIDIA-NeMo/eval-author/pull/95), +[coverage computation (#96)](https://github.com/NVIDIA-NeMo/eval-author/pull/96), +[correlation (#97)](https://github.com/NVIDIA-NeMo/eval-author/pull/97), and +[orchestration and packaging (#99)](https://github.com/NVIDIA-NeMo/eval-author/pull/99). +The tool guide in that implementation stack documents available commands and +current limits. Execution uses a shared results directory, with a local Harbor +loop as the initial backend and Helix planned later. Start with the [architecture](architecture.md) for the supplied inputs, component boundaries, and experiment flow. The [stage descriptions](stages.md) explain execution, scoring, and completion criteria. +A fixed benchmark is a specific `benchmark.yaml` and its versioned input +artifacts. Prep builds those artifacts and emits the YAML definition. The +orchestrator reads it and invokes the stage tools through their Python APIs; +the same tools can also be used independently from the command line. Versioned +configuration variants support controlled comparisons of experimental designs. + The supplied inputs are a baseline agent, its mutants, a case description, -benchmark tasks, and an experiment configuration. Building the case packages +benchmark tasks with a fixed subset manifest, and an experiment configuration. Building the case packages happens before a benchmark release. The system under evaluation is supplied separately: Eval Author or another system that measures coverage. -A run executes tasks against the baseline and mutants, obtains coverage scores -from the system under evaluation, and analyzes coverage versus mutant scores -across task subsets. Coverage scores and any rubric the system generates are -experiment outputs, not supplied benchmark inputs. +The architecture has three stages: + +1. **Prep:** prepare case examples for the experiment, including the agent, + mutants, case description, tasks, and configuration. Construct subsets with a + fixed seed and bundle their explicit memberships in a versioned manifest. + This work is outside the experiment. +2. **Run mutants x eval tasks:** every task runs against the baseline and each mutant. The initial + implementation is a simple local Harbor loop, processing one agent at a time; + this is expected to be expensive. A future Helix implementation will provide + orchestration for evaluation at scale. +3. **Score analysis:** most of the tools here consume the execution results, + use the prepared subset manifest, compute mutant scores and coverage, and correlate them. + +Execution and analysis share a [results-directory contract](stages.md#results-directory-contract). +The proposed layout groups native results by agent, then benchmark run, with a +common index for looking up individual task executions. Coverage scores and any +rubric the system generates are experiment outputs, not supplied benchmark inputs. -The intended runner is deterministic program code. The distinction between -deterministic orchestration and repeatable model outputs remains explicit in the -[determinism requirements](architecture.md#determinism). +The analysis runner is deterministic program code. Its control flow, validation, +and output handling work the same whether a person, an agent, or another program +invokes it. These documents describe program behavior and interfaces; execution +does not depend on an agent interpreting the prose. Model use belongs to the +system being evaluated. See the +[benchmark orchestration](architecture.md#benchmark-orchestration). This draft adapts the metric efficacy experiment design from the fixtures repository, as present at revision `f1f72b7a6a5085f9e369905417b3171f30879b92`. It retains the scientific question and the execution-to-analysis artifact flow. Preparation, publication, service-specific identifiers, and internal execution infrastructure are outside this version's runtime design. The -[open decisions](architecture.md#open-decisions) identify where the external -benchmark contract still needs work. +[tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/README.md#current-implementation-limits) describes current implementation limits. diff --git a/labs/metric-benchmark/docs/architecture.md b/labs/metric-benchmark/docs/architecture.md index 2312d008..b8fc0f0b 100644 --- a/labs/metric-benchmark/docs/architecture.md +++ b/labs/metric-benchmark/docs/architecture.md @@ -3,203 +3,212 @@ # Architecture -## Scientific question +## Evaluating Eval Coverage -Does a coverage measurement predict how effectively an evaluation set detects -agent degradation? +Our formulation of eval authoring is based on (1) defining +a list of coverage goals and (2) generating tasks to achieve these goals. We +evaluate our ability to do this separately. -For each supplied case, run the baseline agent and a fixed cohort of mutants -against the same benchmark tasks. Construct several subsets of those tasks. -For each subset, ask the system under evaluation to measure coverage using the -case description and permitted baseline evidence. Measure sensitivity to mutants -from the baseline and mutant outcomes. Correlate those paired scores across -subsets. The system under evaluation may be Eval Author or another coverage -system. +The question we answer here is **Is our coverage metric a good predictor for agentic evaluation quality?** -Compare coverage definitions using the same subsets, mutant cohort, and outcome -scoring policy. Higher mutant scores mean greater detected degradation, so the -comparison seeks a positive relationship with coverage. Correctness and resource -use are separate outcome variations. Report results per case and variation; -combining cases into a leaderboard is an open decision. +This is hard question to answer. Benchmarking a _metric_ is a more complicated ask than simply benchmarking an +agent's ability to perform veriable tasks. Our approach is to conducting an experiment based on mutation +testing (see [Mutation testing — Wikipedia](https://en.wikipedia.org/wiki/Mutation_testing)), +measuring the correlations between the ability of eval tasks to detect mutants and their coverage. -## Supplied inputs +The experiment is a program with supplied inputs, a fixed sequence of operations, +and defined outputs. It executes benchmark tasks against the baseline agent and +a fixed cohort of mutants. Its analysis consumes predefined task subsets, computes a +mutant score for each subset, invokes the coverage tool on the same subsets, and +correlates the two scores. The current implementation starts from saved execution +results. Execution is a replaceable component whose output is a shared results +directory consumed by analysis. + +Comparisons between coverage tools use the same subsets, mutant cohort, and +outcome scoring policy. Higher mutant scores mean greater detected degradation, +so the comparison seeks a positive relationship with coverage. Each analysis +produces a result for one case and one outcome measurement, such as correctness +or resource use. Results are reported per case and measurement. + +## Experiment flow + +The architecture has three stages: **prep**, **run mutants x eval tasks**, and +**score analysis**. Prep prepares case examples for the experiment. The experiment +consumes those inputs, runs the baseline and mutants against the eval tasks, +and analyzes the resulting directory. + +![Metric benchmark architecture: prepare case examples, run mutants x eval tasks, and analyze coverage versus mutant sensitivity.](assets/experiment-flow.svg) + +[Open the SVG diagram](assets/experiment-flow.svg). + +## Benchmark definition + +A fixed metric benchmark is defined by a specific YAML file, `benchmark.yaml`, +and the versioned inputs it references. It identifies the baseline, mutant +cohort, case description, Harbor tasks, prepared subset manifest, execution +implementation, metric computation, and scoring settings. Here, the benchmark +means this complete experiment configuration; the Harbor task set is one of +its inputs. + +Prep builds that benchmark: it produces the case artifacts and fixed subsets, +then emits the YAML definition that ties them together. Running the benchmark +means reading that definition and executing the configured experiment. Each run +has its own results while retaining the same benchmark definition and input +identities. Deliberate changes produce another versioned configuration, making +it possible to compare experimental designs as well as metric tools while +keeping the unchanged inputs fixed. + +## 1. Prep + +Prep prepares case examples for the experiment. This case-construction process +is outside the experiment itself. It produces the +baseline agent, creates and qualifies its mutants, and assembles the case +description, task set, and configuration in the benchmark's input format. Prep +also constructs the eval subsets and bundles their manifest with the task set. +Prep emits `benchmark.yaml` referencing these outputs. Released benchmarks +include the YAML definition and its prepared case examples. The experiment +consumes them; it does not regenerate or select mutants based on its results. + +### Supplied inputs A run receives five supplied inputs. Case packages provide the first four; -the experiment configuration specifies how to use them. Their exact packaging -format is not yet specified; these are logical contents, not a file schema. +the experiment configuration specifies how to use them. The table describes +their logical contents. | Input | Required contents | | --- | --- | | Baseline agent | An executable bundle with an immutable identity and a documented invocation contract | | Mutants | Executable bundles with distinct identities, their baseline relationship, and mutation descriptions | | Case description | Stable case identity and intended agent behavior; an Ethos is one possible representation | -| Benchmark tasks | Fixed task membership, task identities, environments, and correctness verifiers | -| Experiment configuration | Case-to-benchmark associations, fixed mutant cohort, execution settings, subset construction, coverage-system settings and permitted evidence, and scoring parameters | - -### System under evaluation - -In addition to those inputs, a run receives the versioned system whose coverage -measurements are being evaluated: Eval Author or another implementation. Record -its version, dependencies, model configuration when applicable, and effective -settings independently of the case package. Different systems can be compared -against the same cases, execution evidence, subsets, and mutant-scoring policy. - -Coverage is produced during the experiment. A system that uses a rubric may -generate it as part of its coverage workflow; a rubric is not required of every -system. Retain any generated rubric, judgments, and other measurement artifacts -with the scores. The benchmark supplies the case description as context, not a -precomputed coverage answer. Whether that description must follow the Ethos -format remains open. - -Baseline and mutant bundles are prepared and checked before release. Any -qualification that determines the released cohort also happens before release; -the benchmark run does not select mutants based on its observed scores. -Tasks and their environments are supplied ready for the selected execution -backend. - -The configuration records all effective settings, including defaults. Each run -preserves the exact configuration and input identities. A changed bundle, task -set, case description, or coverage-system version defines a different comparison. -Generated coverage artifacts are recorded as outputs with their input identities. +| Benchmark tasks | Fixed task membership, task identities, environments, correctness verifiers, and a versioned subset manifest | +| Experiment configuration | `benchmark.yaml`: case-to-task-set associations, fixed mutant cohort, execution implementation and settings, subset-manifest identity, coverage-system settings and permitted evidence, and scoring parameters | -## Experiment flow +Baseline and mutant bundles are prepared and checked before release. +Tasks and their environments are ready for execution. At present only Harbor tasks are used. + +The configuration supplies the choices that determine execution: the fixed subset manifest, +selected agent runs, scoring rules, and the coverage-tool command and settings. +The program validates these inputs before using them. Missing or inconsistent +inputs cause an error. + +### Subset manifest -The preparation portion of the original experiment becomes a supplied-input -boundary. The runnable experiment has two parts: **Running Evals** and -**Statistical Analysis**. - -```mermaid -flowchart TD - AGENTS["Supplied baseline and mutant bundles"] - DESCRIPTION["Supplied case description"] - TASKS["Supplied benchmark tasks and verifiers"] - CONFIG["Experiment configuration"] - SYSTEM["System under evaluation: Eval Author or another coverage system"] - VALIDATE["Validate inputs and configuration"] - - AGENTS --> VALIDATE - DESCRIPTION --> VALIDATE - TASKS --> VALIDATE - CONFIG --> VALIDATE - SYSTEM --> VALIDATE - - subgraph EXEC["Running Evals"] - RUN["Run Eval Matrix"] - COLLECT["Collect Eval Results"] - RAW["Raw execution results, traces, and task inventory"] - RUN --> COLLECT --> RAW - end - - VALIDATE --> RUN - - subgraph ANALYSIS["Statistical Analysis"] - SUBSETS["Eval Subset Construction"] - MUTANT["Compute Mutant Scores"] - COVERAGE["Compute Coverage with the system under evaluation"] - EVIDENCE["Coverage artifacts: rubric if used, judgments, and evidence"] - CORRELATE["Spearman Correlation"] - RECORD["Record Experiment"] - SUBSETS --> MUTANT - SUBSETS --> COVERAGE - COVERAGE --> EVIDENCE - MUTANT --> CORRELATE - COVERAGE --> CORRELATE - CORRELATE --> RECORD - end - - RAW -->|"task inventory only"| SUBSETS - RAW -->|"baseline and mutant outcomes"| MUTANT - RAW -->|"baseline traces only"| COVERAGE - DESCRIPTION --> COVERAGE - SYSTEM --> COVERAGE - CONFIG -.-> SUBSETS - CONFIG -.-> MUTANT - CONFIG -.-> COVERAGE - CONFIG -.-> CORRELATE -``` - -Record Experiment retains the inputs and every stage's outputs, including raw -execution evidence. The diagram omits those archive edges for readability. -Subset construction uses task identities and the configured sampling rule, never -observed rewards or coverage. Coverage measurement receives the case description -and explicitly permitted baseline evidence, never mutant outcomes or correlation -results. The precise access to agent source, task contents, traces, and baseline -verifier results must be fixed in the configuration and held comparable across -systems. Passing a path to the entire results directory would not enforce that -boundary. - -## Component responsibilities - -| Component | Responsibility | +Prep constructs subsets from the pinned task inventory, before inspecting +execution outcomes or coverage scores. Each case bundles `subsets.json`, a +versioned manifest containing stable subset IDs, exact task identities and +revisions for each subset, the source benchmark identity, and the construction +algorithm, subset count, subset size, and fixed seed. + +The manifest is authoritative: every run and metric tool uses the same explicit +memberships. The seed documents construction and stays fixed for that benchmark +version; analysis does not resample or regenerate subsets. Changing the seed, +inventory, or memberships creates a new benchmark version. Analysis validates +the manifest against the execution inventory and rejects mismatched identities. +Missing results follow the scoring policy without changing subset membership. + +## 2. Run mutants x eval tasks + +This stage executes the same benchmark tasks against the baseline and every +selected mutant. Its two architectural components +are the **execution implementation** and the **results-directory interface**. + +| Component | Role | +| --- | --- | +| Execution implementation | Runs the configured agent-by-task matrix and assembles its outputs. The initial implementation is a simple local Harbor loop, one agent's full benchmark at a time. A future Helix implementation will orchestrate evaluations at scale. | +| Results directory | Carries the task inventory, agent/task identities, statuses, measurements, and native evidence into analysis. Both execution implementations produce the same contract. | + +The proposed layout is agent-first: `agents////`, with +`results.json` at the root indexing those trial results. Each run contains the +native output of that agent running the benchmark. The +[results-directory contract](stages.md#results-directory-contract) specifies the +layout. Invocation, scheduling, and collection are internal to the execution +implementation; they do not add generic experiment stages. + +## 3. Score analysis + +Score analysis consumes the results directory and produces the benchmark's score. +It consists of three components. Mutant scoring and coverage consume the prepared +subset manifest and execution outputs; correlation consumes their derived scores. None of +these components reruns the benchmark tasks. + +![How score analysis works: each predefined task subset produces a coverage score and a mutant score; each pair becomes one point for correlation.](assets/score-analysis.svg) + +For each subset in the manifest, analysis selects those tasks' saved results for +the baseline and every mutant. Comparing their outcomes produces one mutant +score for the subset. Separately, metric computation uses that same subset's +baseline evidence and the case description to produce one coverage score. +Repeating this for all predefined subsets gives one pair of scores per subset. +Each point in the figure represents a subset; correlation measures whether +subsets with higher coverage tend to detect more degradation. The example +values are illustrative. Subsets may overlap, so these points are not independent +observations. The reported correlation is descriptive, without independent-sample +significance claims. + +| Component | Inputs | Output | +| --- | --- | --- | +| Compute Mutant Scores | Subset manifest and baseline/mutant measurements from `results.json` | A mutant-sensitivity score for each subset | +| Compute Coverage | Subset manifest, case description, and indexed baseline evidence, evaluated with the configured system | A coverage score for each subset and any artifacts generated by that method | +| Correlation | Coverage and mutant scores joined by subset ID | Correlation across subsets, with paired values and any exclusions | + +The Python analysis runner validates the inputs and invokes these components in +a fixed order. Subset membership is fixed during prep. Coverage receives baseline evidence without mutant outcomes or +correlation results. The experiment ends with correlation; output storage and +packaging are implementation details. + +Mutant scoring uses mean signed baseline-to-mutant degradation, with equal +weight for tasks and mutants. Correlation compares the ranks of the paired +scores, with average ranks for ties. The [component descriptions](stages.md#3-score-analysis) +specify these calculations and the handling of missing or undefined values. +The [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/README.md) documents the commands and their current +implementation status. + +### Metric computation (e.g. Eval Author) + +Eval Author is the primary system under evaluation. For each task subset, the +runner supplies the case description, task identities, and baseline execution +evidence, and obtains a coverage score. The correlation stage compares those +scores with the independently computed mutant scores. + +Coverage computation may include rubric generation or model calls as part of +the system's method. Coverage scores and any generated rubrics are experiment +outputs. The Python runner controls stage ordering, validation, and result +storage. + +Tool-specific adapters provide the common invocation interface needed to +evaluate other systems, such as skill-based or LangChain-based workflows. +They handle differences in native inputs, runtimes, and outputs; the experiment +still performs the same Compute Coverage stage. Integration details and current +implementation limits are described in the [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/README.md#coverage-system-invocation). + +## Benchmark orchestration + +This benchmark runs the same Harbor tasks against multiple versions of an agent: +the baseline and its mutants. A lightweight experiment runner coordinates those +runs and analyzes their combined results. Harbor's verifiers score individual +task executions; the metric benchmark's score is computed across many executions +by comparing coverage with mutant sensitivity across task subsets. That analysis +sits above Harbor, so the runner provides the orchestration and scoring that a +single Harbor job or task verifier does not. This control flow is deterministic +Python code. + +### Stage tools and Python libraries + +Each executable experiment stage has a Python tool and an importable Python +API. The orchestrator reads and validates `benchmark.yaml`, resolves its inputs, +and calls these APIs in dependency order, passing each stage's outputs to the +next. The command-line tools expose the same operations for standalone use. +Stage behavior lives in these reusable tools; the orchestrator supplies the +configuration and coordinates their execution. + +| Stage | Python tool / library operation | | --- | --- | -| Case package | Supply the prepared baseline, mutants, tasks, and case metadata | -| Configuration validator | Check input and system identities, declared relationships, settings, and required capabilities before execution | -| Runner | Execute the fixed stage sequence, validate outputs, and record stage completion | -| Execution adapter | Invoke the supplied agents and tasks through the selected backend and preserve attempts and outcomes | -| Results collector | Produce a common results index while retaining native traces and verifier evidence | -| System under evaluation | Produce coverage scores from the case description and permitted baseline evidence, including rubric generation if its method requires it | -| Analysis functions | Construct subsets, compute mutant scores, join score dictionaries, and compute correlations | -| Experiment record | Preserve the configuration, input and system identities, execution evidence, generated coverage artifacts, scores, exclusions, and completion status | - -The execution adapter is a portability boundary. This design requires no -particular hosted service, registry, deployment, or orchestration product. -Supporting a backend requires defining and testing its invocation and evidence -contract; the adapter box alone does not establish portability. - -Owned configuration and artifact formats should use shared, versioned data -models with one validation and serialization implementation. Native trace and -task formats retain their established readers. Concrete schemas and interfaces -are not defined by this documentation. - -## Determinism - -The runner uses explicit code and configuration for stage ordering, retries, -completion checks, and analysis. No coordinating language model chooses the -next step, changes scoring rules, or repairs an experiment during execution. - -For fixed execution evidence and recorded coverage outputs, subset membership -and statistical results must be reproducible. This requires stable task ordering, -a specified sampling algorithm -and seed or supplied memberships, fixed scoring rules, pinned implementations, -and explicit treatment of missing values and ties. Completion order must not -change which attempts or observations are scored. - -Live agent calls, generated rubrics, and model-based coverage judgments can still -vary. Pinning a model name, setting a seed, or choosing temperature zero does not by itself -establish identical results. Wall-clock measurements can also vary between runs. -The draft therefore does not yet promise full reproducibility of fresh live -executions. - -To support an end-to-end deterministic claim, we must choose whether the benchmark -requires deterministic agents and coverage systems, replays previously recorded -execution and coverage evidence, or defines determinism only for orchestration -and analysis over frozen observations. Replay would test analysis -reproducibility; by itself it would not test new live agent behavior or new -model-generated rubrics and judgments. Replaying an earlier coverage score does -not evaluate a new version of the coverage system. - -## Open decisions - -- **Determinism scope:** choose the guarantee for live execution, coverage - judgments, and replay before describing the benchmark as fully deterministic. -- **Distribution:** choose the case package format, bundle invocation contract, - supported task format, and initial execution backend. -- **Case description:** decide whether an Ethos is required or whether other - descriptions of intended agent behavior are accepted. -- **Execution policy:** specify repetitions, seeds, timeouts, retry limits, - attempt selection, and treatment of incomplete agent-task pairs. -- **Coverage interface and evidence:** define the common score interface, which - baseline artifacts each system can see, and whether it can inspect the full - benchmark or only the subset being scored. For systems that generate a rubric, - specify its generation scope and reuse across subsets. Keep mutant outcomes - and correlation results inaccessible to coverage computation. -- **Scoring:** choose the primary mutant score, mutant weighting, coverage - aggregation, resource measurements, and treatment of partially observed subsets. -- **Statistical reporting:** choose subset sizes and overlap constraints, - uncertainty estimates that account for shared tasks, and any cross-case summary. -- **Meaning of mutant score:** mutation sensitivity is a proxy for evaluation - quality. Baseline failures, irrelevant mutants, and ceiling or floor effects - can weaken that proxy even when the execution and analysis are correct. - -These choices are intentionally unresolved. No command-line interface, public -dataset release, or runnable experiment is implied by this draft. +| Execute evals | Local Harbor execution tool, or a future Helix execution tool; both expose the same results-directory contract | +| Compute mutant scores | `compute_mutant_scores.py` / `metric_benchmark.scoring.compute_mutant_scores` | +| Compute coverage | `compute_coverage.py` / `metric_benchmark.coverage.invoke_coverage` | +| Correlation | `correlate.py` / `metric_benchmark.correlation.correlate` | + +Subset construction is also a reusable Python tool, invoked when building the +benchmark. The experiment runner consumes its fixed manifest. Execution tools +can have multiple implementations without changing the analysis tools or their +input contract. Available commands and implementation status are documented in +the [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/README.md#fixed-python-pipeline). diff --git a/labs/metric-benchmark/docs/assets/experiment-flow.svg b/labs/metric-benchmark/docs/assets/experiment-flow.svg new file mode 100644 index 00000000..9114dacd --- /dev/null +++ b/labs/metric-benchmark/docs/assets/experiment-flow.svg @@ -0,0 +1,134 @@ + + + + + Metric benchmark: from case examples to correlation + Three phases: prepare case examples for the experiment; run the baseline and mutants against all eval tasks; analyze the saved results. Prep supplies the agent, mutants, case description, benchmark tasks, and experiment configuration. It constructs subsets using a fixed seed and bundles their exact memberships in an authoritative versioned manifest. Execution produces a shared results directory. Analysis consumes the prepared subset manifest, computes mutant scores from baseline and mutant outcomes, and computes coverage from the case description and baseline evidence using metric computation such as Eval Author. The two scores are joined by subset for correlation. Configuration governs execution and analysis. Prep is outside the experiment; the local Harbor loop and future Helix implementation are planned execution backends. + + + + + + + + + + + + Metric benchmark + Does coverage predict how well eval tasks detect agent degradation? + + + + + 1 + Prep + Prepare case examples for the experiment + + Outside the experiment + + + + + + + + + Agent + Baseline bundle + Mutants + Fixed variant cohort + Case description + Intended behavior + Benchmark tasks + Tasks + verifiers + benchmark.yaml + Fixed configuration + + Construct eval subsets + Pinned task inventory + fixed seed + Freeze membership before inspecting outcomes. + + + Subset manifest + Stable subset IDs + exact task membership + Versioned subsets.json · algorithm + fixed seed recorded + Prep emits benchmark.yaml + its input artifacts. The orchestrator reads the YAML and invokes Python stage tools. + + + + + + + + 2 + Run mutants x eval tasks + Same tasks · baseline + every mutant + + + Execute evals + Mutants + baseline x eval tasks + One full benchmark per agent; local jobs run in sequence. + Planned Python tools: local Harbor, then Helix. + + + Results directory + Shared index + native task results + results.json + agents/<agent>/<run>/<native-trial>/ + + + + + + + 3 + Score analysis + Read the saved results. + Compare the same subsets. + + + Subset manifest + Fixed membership from prep + + + + + + METHOD BEING EVALUATED + Metric computation + (e.g. Eval Author) + + + + Compute mutant scores + Baseline + mutant outcomes + compute_mutant_scores.py + + + Compute coverage + Case description + baseline evidence + compute_coverage.py + + + + Join by subset + + Correlation + correlate.py + + diff --git a/labs/metric-benchmark/docs/assets/score-analysis.svg b/labs/metric-benchmark/docs/assets/score-analysis.svg new file mode 100644 index 00000000..23ccaa53 --- /dev/null +++ b/labs/metric-benchmark/docs/assets/score-analysis.svg @@ -0,0 +1,142 @@ + + + + +Each task subset produces one pair of scores +First, take a predefined subset such as tasks 2, 5, and 8. Read its saved execution results for the baseline and every mutant. Compare baseline and mutant outcomes to compute one mutant score for the subset. Separately, use its baseline evidence and the case description to compute one coverage score. Repeat for every subset in the fixed manifest. Each subset contributes one coverage and mutant score pair, shown as one point on a scatter plot. Correlation measures whether subsets with higher coverage tend to detect more degradation. Example scores and plotted points are illustrative, not experiment results. Analysis does not rerun tasks. + + + + + + +One task subset → two scores → one point +Analysis reuses saved executions; tasks do not run again for each subset. + + +1. Score the same subset in two ways +Example subset A: tasks 2, 5, and 8 +All mutants + +Baseline + +M1 +M2 +… +Mm +Task 2 + + + + + + + +… + + +Task 5 + + + + + + + +… + + +Task 8 + + + + + + + +… + + +Each cell represents a saved task result. + + + + +Compute mutant score +Compare baseline vs. each mutant on these tasks. +Average the degradation → one mutant score. + + + + +Compute coverage +Use baseline evidence + the case description. +Metric computation → one coverage score. +Together: one (coverage, mutant score) pair for subset A + + +2. Repeat for every predefined subset, then compute correlation + +SUBSET +TASKS +COVERAGE +MUTANT SCORE + + + +A +2, 5, 8 +0.25 +0.10 + + +B +1, 4, 8 +0.55 +0.40 + + +C +3, 5, 7 +0.80 +0.70 +… many fixed subsets → many pairs +Illustrative scores; not experiment results. + +Each row +is one point + + + + +Coverage score → +Mutant score → + + + + + + + + + + + + + +A + +B + +C +Do subsets with higher coverage also detect more degradation? + diff --git a/labs/metric-benchmark/docs/stages.md b/labs/metric-benchmark/docs/stages.md index 57fad571..69e21ae3 100644 --- a/labs/metric-benchmark/docs/stages.md +++ b/labs/metric-benchmark/docs/stages.md @@ -5,109 +5,224 @@ This draft starts with the prepared inputs defined in the [architecture](architecture.md#supplied-inputs). Agent bundling, mutant generation -and qualification, and benchmark construction and publication are release +and qualification, subset construction, and benchmark construction and publication are release preparation rather than stages of a benchmark run. The system under evaluation is a separate input. Coverage computation, including rubric generation when the -system uses it, is part of the experiment. +system uses it, is part of the experiment. The [three-stage overview](architecture.md#experiment-flow) +shows preparation of case examples as stage 1. Run mutants x eval tasks is stage 2; +score analysis is stage 3, with its individual components numbered 3.1–3.3. -Before starting, validate the declared input identities, task inventory, baseline -and mutant relationships, case description, system version, and configuration. -Check that the chosen backend can invoke the bundles and retain the evidence -needed by the system under evaluation. Validation does not execute agents or -establish benchmark quality. +Each stage has explicit inputs and outputs. Execution produces a results +directory through one of its supported implementations. The Python analysis +runner consumes that directory and calls its operations in a fixed order. An +agent or a person can invoke the runner, but neither supplies missing control +flow or decides whether a stage has completed. ## Stage contract +The benchmark's YAML definition selects the inputs, stage settings, and execution +implementation. Each executable stage provides a Python API and a command-line +tool. The orchestrator validates that definition and invokes the APIs in +dependency order. Prep produces the definition and its input artifacts when +building the benchmark; experiment runs consume them. + Each stage has declared inputs, fixed parameters, outputs, and completion criteria. Outputs become available downstream only after validation. -The program controls transitions and handles failures according to configured -rules. An execution error must remain distinct from a legitimate low reward. - -Resume at completed stage boundaries. An incomplete stage can reconcile its -recorded attempts when invoked again, but must not blindly duplicate executions -or pick the most favorable attempt. Changing an input invalidates dependent -outputs; earlier observations remain in the record under their original identities. - -The following descriptions specify behavior, not implemented APIs. - -## Running Evals - -### 1. Run Eval Matrix - -**Purpose:** execute every task against the baseline and every supplied mutant -selected in the configuration. - -**Inputs:** prepared agent bundles, benchmark tasks, and execution settings. - -**Operation:** enumerate the agent-by-task matrix, including configured -repetitions. Execute it through the selected backend. Use the same task and -verifier versions and comparable execution settings for baseline and mutants; -record intentional differences that define a mutation. Apply a fixed retry -policy to execution failures without retrying legitimate agent failures to -improve their scores. - -**Outputs:** an execution manifest identifying each requested pairing and its -attempts, status, and evidence locations. These are portable experiment -identities; backend-specific metadata can accompany them. - -**Completion:** every requested pairing and repetition is accounted for, with no -pending or unresolved attempts. A completed execution can have a failing task -reward. Missing execution evidence is not a zero reward or a completed result. -The treatment of exhausted execution failures must be fixed in the run policy. - -### 2. Collect Eval Results - -**Purpose:** make execution evidence available in a consistent form for analysis. - -**Inputs:** the execution manifest and the native outputs of the backend. - -**Operation:** collect results and traces from their recorded locations. Preserve -native evidence and produce an index keyed by case, agent, task, repetition, -and attempt. Record correctness and any available resource measurements, their -units, and evidence locations. Verify the recorded task inventory against the -supplied benchmark. - -**Outputs:** Raw Execution Results and Task References. The shared index locates -native evidence rather than replacing it. +The program controls transitions and stops on a stage error. Scoring accepts +only the configured execution statuses, so an unsuccessful execution's numeric +reward is not automatically treated as a valid observation. + +On a repeated invocation, the analysis pipeline recomputes inexpensive +deterministic stages and reuses completed coverage requests after verifying +their inputs and saved files. Changed inputs cause an error in the existing +output directory. A coverage request without a completion receipt runs again; +the adapter owns any external side effects from an interrupted request. + +The implementation stack provides standalone Python tools for analysis +components 3.1–3.3; see its [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/README.md). Stage 2 defines the execution boundary. The current +tools consume the fixtures v1 saved-results index and native evidence, and +never rerun benchmark agents. + +| Component | Command in the implementation stack | +| --- | --- | +| Prep: subset construction | `construct_subsets.py` | +| 3.1. Mutant scoring | `compute_mutant_scores.py` | +| 3.2. Coverage | `compute_coverage.py` | +| 3.3. Correlation | `correlate.py` | +| Analysis runner (manifest input pending) | `run_analysis.py` | + +## 1. Prep + +Prep prepares case examples for the experiment. The +[prepared-input contract](architecture.md#supplied-inputs) describes +the baseline, mutants, case description, tasks with their subset manifest, and configuration supplied to +the experiment. The experiment does not execute this preparation process. + +### Subset construction + +**Purpose:** prepare fixed evaluation subsets shared by every run and metric tool. + +**Inputs:** the pinned benchmark task inventory and fixed subset count, size, +and seed. Execution results and coverage scores are not inputs to prep. + +**Operation:** the sampler sorts task identities and applies its versioned, +seeded algorithm to select distinct subsets of the configured size. The seed is +fixed for the benchmark version and recorded as construction provenance. +Subsets are frozen before inspecting outcomes or coverage scores. + +**Output:** a versioned `subsets.json` manifest with the source benchmark identity, +task inventory and revisions, stable subset IDs, explicit membership, algorithm, +count, size, and seed. The manifest's memberships are authoritative; analysis +reads them rather than regenerating them from the seed. Changing the inventory, +seed, or memberships creates a new benchmark version. + +**Completion:** all subset IDs and memberships are unique, every member belongs +to the pinned inventory, and the requested count and size are satisfied. +Impossible requests fail. Analysis validates the manifest against the execution +inventory; missing observations never cause it to alter the prepared subsets. + +**Current implementation:** the existing `eval_subset_definitions` artifact +contains the memberships and sampling parameters. The Python sampler accepts a +task inventory directly, but its CLI currently obtains that inventory from saved +results. A prep CLI accepting the inventory without results is still needed. +The combined `run_analysis.py` also still generates subsets; it needs to accept +and pin a supplied manifest instead. Standalone mutant scoring and coverage +already accept `--subsets`. The current artifact includes an `experiment_id`; +its binding must be separated from benchmark identity to reuse one unchanged +manifest across independently identified runs. + +## 2. Run mutants x eval tasks + +**Purpose:** execute the benchmark tasks against the baseline and supplied +mutants and produce the results directory consumed by analysis. + +**Inputs:** prepared agent bundles, benchmark tasks, and experiment execution +settings. + +**Implementations:** the initial local implementation uses a simple deterministic +loop: baseline first, then the configured mutants, with one Harbor job per agent +covering the full benchmark. Agent jobs run sequentially. Task concurrency within +a Harbor job is an explicit execution setting; a fully sequential run uses one +concurrent trial. The initial analysis supports one trial per agent/task within +the selected run; repetition aggregation remains future work. + +This phase is expected to be expensive: with `M` mutants and `T` tasks, even a +single execution per pair requires `(M + 1) × T` task executions before retries. +The baseline is included because mutant scoring compares its outcomes with each +mutant's outcomes. The repository currently supplies the analysis tools; the +local Harbor loop is the first execution implementation to add here. + +A future Helix implementation will use an orchestrator to run evaluations at +scale. Each implementation owns invocation, scheduling, retries, and collection +and produces the same output contract. Analysis can rerun against saved results +without repeating this execution phase. + +### Results-directory contract + +**Proposed layout:** agent first, then benchmark run, with native trial outputs +inside each run. One results root represents one case and benchmark revision. -**Completion:** all requested executions are represented, evidence can be read -and its identity checked, and measurements or traces absent from the source are -explicitly marked. Collection failures leave the stage incomplete. The collector -does not manufacture measurements or turn missing data into agent failures. - -## Statistical Analysis - -### 3. Eval Subset Construction - -**Purpose:** construct simulated evaluation sets from the benchmark task inventory. - -**Inputs:** Task References and the configured subset count, sizes, sampling -algorithm, seed, and constraints, or explicit supplied memberships. - -**Operation:** select task identities without consulting outcomes, resource -measurements, or coverage scores. Preserve exact memberships so subsequent -analysis need not resample. - -**Outputs:** Eval Subset Definitions with stable subset IDs, source benchmark -identity, membership, and construction parameters. +```text +execution-results/ + results.json # Shared analysis index and task inventory + agents/ + baseline/ + run-0001/ # One intact Harbor job directory + config.json + lock.json + result.json # Harbor's job-level result + job.log + task-a__/ + config.json + lock.json + result.json # Harbor's per-trial result + agent/ # Agent execution evidence + verifier/ # Verifier outputs + ... # Other native Harbor artifacts + task-b__/ + ... + mutant-001/ + run-0001/ + ... # Same benchmark, this mutant's results + mutant-002/ + run-0001/ + ... +``` -**Completion:** all memberships match the requested count, sizes, and constraints, -and every task belongs to the supplied benchmark. An impossible sampling request -fails explicitly rather than silently relaxing its constraints. +The native portion shown is illustrative of Harbor's single-step trial layout; +its actual files depend on the pinned Harbor version and task type. The local +implementation chooses the agent/run parent directory and preserves the native +job contents, including native trial names. Explicit run names avoid using a +timestamp as the benchmark's logical run identity. + +Agent-first matches the proposed execution loop and keeps each Harbor job's +configuration, aggregate results, and task evidence together. A task-first tree +would split that job across multiple directories. A second physical copy is +unnecessary: the shared index supports either agent-wise or task-wise views. +Separate run directories allow later executions without overwriting earlier +ones; the current scorer selects one run per agent. + +**Index:** `results.json` uses the existing v1 `raw_execution_results` schema. +It includes the complete benchmark/task inventory and each execution's agent, +run, task revision, attempt, status, measurements, and native-evidence location. +This shared index is distinct from Harbor's own job and trial `result.json` +files. All evidence paths are relative to `execution-results/`. For example: + +```json +{ + "native_results_path": "agents/mutant-001/run-0001/task-a__x7k2p9m", + "native_results_format": "directory" +} +``` -### 4. Compute Mutant Scores +This is an excerpt of an execution entry, not a complete results index. It points +to one trial, not the whole job directory. Folder labels are filesystem-safe +aliases; the index carries the actual agent and task identities. A run identity +is unique across the results index even when agents reuse the folder label +`run-0001`. Native trial names can contain generated suffixes, and analysis never +parses those names to recover task identities, repetitions, or retry order. + +The current reader already supports these relative paths and native directories +or archives. Existing commands accept explicit index and evidence-root paths; +the new producers will adopt this layout, but the reader does not yet enforce +these folder names. The local Harbor implementation still needs the code that +builds the shared index from native job/trial metadata. + +The Helix implementation will export the same agent/run layout and index with +task-level evidence in a supported format. Its scheduling and storage can differ +internally. Analysis reads the common index and evidence without a Helix-specific +service API. An archive can replace a trial directory when the index explicitly +declares its path and format; the native evidence contents still need to be +readable by the coverage implementation. + +**Completion contract:** the output accounts for the configured agent-by-task +matrix and identifies failed attempts and unavailable measurements explicitly. +A completed task with a zero reward remains distinct from an execution error. +The output index and referenced evidence agree on identities and formats. +A full execution-completeness validator is still to be implemented; the existing +analysis tools validate the index and handle unavailable observations according +to their scoring policy. + +The shared contract covers directory contents as well as paths: measurements +have consistent meanings, and the evidence formats are readable by the coverage +implementation. A Helix implementation will produce that same contract rather +than making analysis interpret Helix-specific result metadata. + +## 3. Score analysis + +### 3.1. Compute Mutant Scores **Purpose:** measure each subset's sensitivity to the supplied mutations. -**Inputs:** Eval Subset Definitions, Raw Execution Results for the baseline and +**Inputs:** the prepared subset manifest, Raw Execution Results for the baseline and mutants, the fixed mutant cohort, and the scoring policy. -**Operation:** compare baseline and mutant outcomes on identical task memberships. -Compute one score per subset and scoring variation. Correctness and efficiency -remain separate variations; their units and score direction must be explicit. +**Operation:** the scoring function selects the configured runs and attempts, +pairs baseline and mutant outcomes by task, and computes one score per subset +for the configured measurement. It rejects ambiguous repetitions. The metric +name and higher-is-better or lower-is-better direction are configuration fields. -A candidate correctness score inherited from the original experiment is the mean -signed reward loss: +For reward, the implemented score is mean signed reward loss: ```text S(agent, subset) = mean task reward for that agent on that subset @@ -115,103 +230,91 @@ M(subset) = mean over selected mutants of [S(baseline, subset) - S(mutant, subset)] ``` -This candidate weights mutants equally and preserves negative differences when -a mutant outperforms the baseline. It is an example policy, not a finalized -default. Repetition aggregation, incomplete evidence, and any alternative -weighting must be specified before use. +The scorer uses this equally weighted signed score and preserves +negative differences when a mutant outperforms the baseline. Missing required +pairs make a subset unavailable; no tasks or mutants are silently dropped. +One run per agent and an explicit attempt-selection rule are required. +This analysis does not pool repetitions or apply alternative weighting. **Outputs:** Mutant Scores, keyed by subset ID, with the case, baseline, mutant -cohort, aggregation, and scoring variation identified. Retain per-mutant -comparisons as supporting evidence. +cohort, aggregation, and scoring variation identified. The output also contains +the individual mutant effects used in the aggregate. **Completion:** every requested subset has a score or an explicit unavailable -value and reason. Never silently change task membership or mutant composition -to obtain a computable score. +value and reason. Missing required pairs produce unavailable scores; the +function does not remove tasks or mutants to obtain a numeric result. -### 5. Compute Coverage +### 3.2. Compute Coverage **Purpose:** obtain the system under evaluation's coverage scores for the same subsets. -**Inputs:** the case description, Eval Subset Definitions, permitted baseline -evidence from Raw Execution Results, and the versioned system under evaluation -with its settings. Additional inputs such as agent source or task contents must -be explicitly allowed by the configured evidence policy. - -**Operation:** invoke Eval Author or another coverage system through its declared -interface to produce scores. If that system needs a rubric, let it generate the -rubric within this stage using only permitted inputs, and retain the resulting -artifact. Rubric generation and measurement are parts of the system being tested, -not assumed benchmark answers. - -For a method that uses one case-level rubric, freeze its generated contents and -denominator before scoring subsets and reuse them within that comparison. -A method that derives its coverage definition separately for each subset must -declare that behavior and retain each definition; it must not be silently treated -as a shared-rubric measurement. The allowed generation scope is an open protocol -decision. - -Reuse per-task measurements across overlapping subsets only when the system's -method makes them independent of the other tasks in the subset. Systems that -score whole subsets can instead produce scores directly. Mutant outcomes, -mutant scores, and correlation results are never inputs to coverage computation. +**Inputs:** the case description, the prepared subset manifest, permitted baseline +evidence from Raw Execution Results, and the configured system under evaluation +with its tool-specific adapter and settings. The current request supplies task +identities and native baseline evidence for the subset; additional source or +task-content interfaces are not implemented. + +**Operation:** the coverage stage copies the subset's selected baseline evidence +into a work directory, writes a JSON request, and invokes the configured adapter +command. The adapter translates the request into its system's native inputs, +executes that system through its native interface, validates its outputs, and +converts the result to the common response. The coverage stage validates the +adapter's JSON response as a numeric score or an unavailable value with a reason. +Missing baseline evidence, invalid responses, +input modification, timeouts, and nonzero exits fail the stage. + +The system may generate a rubric as part of its method. Its adapter places the +supporting outputs in the work directory. The runner does not infer which +rubrics or judgments the tool should have produced; required adapter-specific +outputs need an adapter-specific validator. The current implementation provides +the common invocation protocol and a synthetic adapter; it does not yet supply +production adapters for Eval Author or other systems. + +Each subset is evaluated independently. The runner has no shared-rubric +generation pass or cross-subset measurement cache. Those methods would require +explicit additional interfaces and validation. Mutant outcomes, mutant scores, +and correlation results are absent from the coverage request. Staging inputs +does not sandbox the tool's access to the host. **Outputs:** Coverage Scores, keyed by subset ID, with system and configuration -identities and supporting measurement evidence. Retain any generated rubrics, -per-task measurements, and judgments with their exact inputs and versions. +identities, plus the saved requests, responses, and files produced in each work +directory. The stage writes a completion receipt with file digests for each +successful request. -**Completion:** every requested subset and system configuration has a score or -an explicit unavailable value and reason. Generated coverage artifacts are -accounted for. If computation generates rubrics or judgments with a model, -their reproducibility must satisfy the chosen -[determinism contract](architecture.md#determinism); deterministic aggregation -alone does not make those outputs deterministic. +**Completion:** every subset has a validated response and completion receipt. +The stage writes the aggregate Coverage Scores artifact only after all subset +requests complete. If the tool generates rubrics or judgments with a model, +those files are observations. The [benchmark orchestration](architecture.md#benchmark-orchestration) +is deterministic; fresh model outputs are not assumed identical. -### 6. Spearman Correlation +### 3.3. Correlation **Purpose:** determine whether coverage predicts mutation sensitivity across evaluation subsets. **Inputs:** Coverage Scores and Mutant Scores with matching subset definitions. -**Operation:** verify matching case, baseline, benchmark, and subset identities. -Join scores by subset ID and compute Spearman rank correlation separately for -each coverage-system configuration and mutant-score variation, using average -ranks for ties. Retain each paired score and every exclusion with its reason. +**Operation:** the correlation function verifies matching source identities and +subset IDs, joins scores by subset ID, and computes Spearman rank correlation +using average ranks for ties. It rejects unavailable pairs by default. With the +explicit exclusion policy, it omits those pairs and writes their reasons into +the result alongside the included pairs. -**Outputs:** Correlation Results containing the coefficient, number of included -subsets, paired scores, configuration identities, and exclusions. +**Outputs:** Correlation Results containing the coefficient, included pairs, +configuration identities, and exclusions. The included count is the length of +the pair list. **Completion:** each comparison has a result or an explicit undefined result. Constant scores or too few usable pairs do not become zero correlation. Unexpected identity or membership mismatches are errors, not silently dropped -observations. Any allowed exclusion policy must be specified in advance. +observations. The missing-value policy is an explicit function input. Overlapping subsets share observations. Treating them as independent samples -would overstate the strength of evidence. The uncertainty method remains an -open decision; an observed coefficient alone is not a general validity claim. - -### 7. Record Experiment - -**Purpose:** preserve enough evidence to inspect and reproduce the analysis. - -**Inputs:** the frozen configuration, supplied input and system identities, and -all stage outputs, including generated coverage artifacts, paired scores, and -correlation results. - -**Operation:** retain an inventory of inputs, execution attempts, native evidence, -subset memberships, generated rubrics and judgments when present, scores, -exclusions, implementation versions, and completion status. A rerun creates a -distinct record linked to its reused inputs. - -**Outputs:** an Experiment Record containing the inventory and analysis report, -with evidence stored or referenced according to the eventual distribution format. - -**Completion:** every input and output is accounted for and accessible to the -intended reader. Unavailable values and failed attempts retain their explanations. -Partial runs should retain diagnostic records, but must not be reported as -completed benchmark results. +would overstate the strength of evidence. The reported coefficient is +descriptive; the analysis does not report independent-sample significance or +claim general validity from that coefficient alone. -Mutant scoring and coverage measurement can run independently after subset -construction. Their results meet at correlation. None of the statistical -analysis stages reruns the benchmark agents. +Mutant scoring and coverage measurement depend on the subset definitions, and +their results meet at correlation. The current runner executes them sequentially +in the order shown above. None of the analysis stages reruns benchmark agents. From 409cc4c6b2cd622c439f056169133167d278b90b Mon Sep 17 00:00:00 2001 From: mstaats Date: Wed, 21 Oct 2026 20:27:46 +0000 Subject: [PATCH 4/5] docs(metric-benchmark): publish concise overview and runtime stages Signed-off-by: mstaats --- labs/metric-benchmark/README.md | 31 ++++ labs/metric-benchmark/docs/README.md | 64 --------- labs/metric-benchmark/docs/architecture.md | 18 +-- .../docs/assets/experiment-flow.svg | 6 +- labs/metric-benchmark/docs/stages.md | 136 +++++------------- 5 files changed, 81 insertions(+), 174 deletions(-) create mode 100644 labs/metric-benchmark/README.md delete mode 100644 labs/metric-benchmark/docs/README.md diff --git a/labs/metric-benchmark/README.md b/labs/metric-benchmark/README.md new file mode 100644 index 00000000..e569d86a --- /dev/null +++ b/labs/metric-benchmark/README.md @@ -0,0 +1,31 @@ + + + +# Metric benchmark + +An experimental benchmark for evaluating how well our coverage metric predicts +an evaluation set's effectiveness at detecting mutations in an agent. This +project outlines the architecture and develops deterministic Python tooling for +running a growing set of case examples and evaluating our metric through +mutation testing. + +**This is a work in progress.** The tools and benchmark will change. Case examples +will grow, change, and may be dropped as we learn. We believe in building in plain +sight. + +Prep happens outside this repository. The benchmark consumes prepared case +examples, fixed task-subset manifests, and a YAML experiment configuration. + +The experiment has four executable stages, each exposed as a Python tool and +library API: + +1. **Run mutants x eval tasks:** run the baseline and every mutant against the + benchmark tasks and save the results. The first execution script will be a + basic local Harbor loop; support for **NeMo Helix** will follow. +2. **Compute mutant scores:** measure degradation across the mutants for each + predefined task subset. +3. **Compute coverage:** evaluate our metric on those same subsets. +4. **Correlation:** compare the paired coverage and mutant scores across subsets. + +See the [architecture](docs/architecture.md) and [stage descriptions](docs/stages.md) +for how the experiment works. diff --git a/labs/metric-benchmark/docs/README.md b/labs/metric-benchmark/docs/README.md deleted file mode 100644 index d878cbed..00000000 --- a/labs/metric-benchmark/docs/README.md +++ /dev/null @@ -1,64 +0,0 @@ - - - -# Metric benchmark - -**Draft for review.** This directory specifies the architecture of an external -benchmark for testing whether coverage measurements predict how well an -evaluation set detects deliberately weakened agents. This base PR contains the -design and figures. Separate implementation PRs build the stage tools on top: -[subset preparation (#94)](https://github.com/NVIDIA-NeMo/eval-author/pull/94), -[mutant scoring (#95)](https://github.com/NVIDIA-NeMo/eval-author/pull/95), -[coverage computation (#96)](https://github.com/NVIDIA-NeMo/eval-author/pull/96), -[correlation (#97)](https://github.com/NVIDIA-NeMo/eval-author/pull/97), and -[orchestration and packaging (#99)](https://github.com/NVIDIA-NeMo/eval-author/pull/99). -The tool guide in that implementation stack documents available commands and -current limits. Execution uses a shared results directory, with a local Harbor -loop as the initial backend and Helix planned later. - -Start with the [architecture](architecture.md) for the supplied inputs, component -boundaries, and experiment flow. The [stage descriptions](stages.md) explain -execution, scoring, and completion criteria. - -A fixed benchmark is a specific `benchmark.yaml` and its versioned input -artifacts. Prep builds those artifacts and emits the YAML definition. The -orchestrator reads it and invokes the stage tools through their Python APIs; -the same tools can also be used independently from the command line. Versioned -configuration variants support controlled comparisons of experimental designs. - -The supplied inputs are a baseline agent, its mutants, a case description, -benchmark tasks with a fixed subset manifest, and an experiment configuration. Building the case packages -happens before a benchmark release. The system under evaluation is supplied -separately: Eval Author or another system that measures coverage. - -The architecture has three stages: - -1. **Prep:** prepare case examples for the experiment, including the agent, - mutants, case description, tasks, and configuration. Construct subsets with a - fixed seed and bundle their explicit memberships in a versioned manifest. - This work is outside the experiment. -2. **Run mutants x eval tasks:** every task runs against the baseline and each mutant. The initial - implementation is a simple local Harbor loop, processing one agent at a time; - this is expected to be expensive. A future Helix implementation will provide - orchestration for evaluation at scale. -3. **Score analysis:** most of the tools here consume the execution results, - use the prepared subset manifest, compute mutant scores and coverage, and correlate them. - -Execution and analysis share a [results-directory contract](stages.md#results-directory-contract). -The proposed layout groups native results by agent, then benchmark run, with a -common index for looking up individual task executions. Coverage scores and any -rubric the system generates are experiment outputs, not supplied benchmark inputs. - -The analysis runner is deterministic program code. Its control flow, validation, -and output handling work the same whether a person, an agent, or another program -invokes it. These documents describe program behavior and interfaces; execution -does not depend on an agent interpreting the prose. Model use belongs to the -system being evaluated. See the -[benchmark orchestration](architecture.md#benchmark-orchestration). - -This draft adapts the metric efficacy experiment design from the fixtures -repository, as present at revision `f1f72b7a6a5085f9e369905417b3171f30879b92`. -It retains the scientific question and the execution-to-analysis artifact flow. -Preparation, publication, service-specific identifiers, and internal execution -infrastructure are outside this version's runtime design. The -[tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/README.md#current-implementation-limits) describes current implementation limits. diff --git a/labs/metric-benchmark/docs/architecture.md b/labs/metric-benchmark/docs/architecture.md index b8fc0f0b..1a0fbd04 100644 --- a/labs/metric-benchmark/docs/architecture.md +++ b/labs/metric-benchmark/docs/architecture.md @@ -61,7 +61,7 @@ keeping the unchanged inputs fixed. ## 1. Prep Prep prepares case examples for the experiment. This case-construction process -is outside the experiment itself. It produces the +is outside this repository and the experiment itself. It produces the baseline agent, creates and qualifies its mutants, and assembles the case description, task set, and configuration in the benchmark's input format. Prep also constructs the eval subsets and bundles their manifest with the task set. @@ -114,7 +114,7 @@ are the **execution implementation** and the **results-directory interface**. | Component | Role | | --- | --- | -| Execution implementation | Runs the configured agent-by-task matrix and assembles its outputs. The initial implementation is a simple local Harbor loop, one agent's full benchmark at a time. A future Helix implementation will orchestrate evaluations at scale. | +| Execution implementation | Runs the configured agent-by-task matrix and assembles its outputs. The initial implementation is a simple local Harbor loop, one agent's full benchmark at a time. A future NeMo Helix implementation will orchestrate evaluations at scale. | | Results directory | Carries the task inventory, agent/task identities, statuses, measurements, and native evidence into analysis. Both execution implementations produce the same contract. | The proposed layout is agent-first: `agents////`, with @@ -157,9 +157,9 @@ packaging are implementation details. Mutant scoring uses mean signed baseline-to-mutant degradation, with equal weight for tasks and mutants. Correlation compares the ranks of the paired -scores, with average ranks for ties. The [component descriptions](stages.md#3-score-analysis) +scores, with average ranks for ties. The [component descriptions](stages.md#2-compute-mutant-scores) specify these calculations and the handling of missing or undefined values. -The [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/README.md) documents the commands and their current +The [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/docs/tools.md) documents the commands and their current implementation status. ### Metric computation (e.g. Eval Author) @@ -178,7 +178,7 @@ Tool-specific adapters provide the common invocation interface needed to evaluate other systems, such as skill-based or LangChain-based workflows. They handle differences in native inputs, runtimes, and outputs; the experiment still performs the same Compute Coverage stage. Integration details and current -implementation limits are described in the [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/README.md#coverage-system-invocation). +implementation limits are described in the [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/docs/tools.md#coverage-system-invocation). ## Benchmark orchestration @@ -202,13 +202,13 @@ configuration and coordinates their execution. | Stage | Python tool / library operation | | --- | --- | -| Execute evals | Local Harbor execution tool, or a future Helix execution tool; both expose the same results-directory contract | +| Execute evals | Local Harbor execution tool, or a future NeMo Helix execution tool; both expose the same results-directory contract | | Compute mutant scores | `compute_mutant_scores.py` / `metric_benchmark.scoring.compute_mutant_scores` | | Compute coverage | `compute_coverage.py` / `metric_benchmark.coverage.invoke_coverage` | | Correlation | `correlate.py` / `metric_benchmark.correlation.correlate` | -Subset construction is also a reusable Python tool, invoked when building the -benchmark. The experiment runner consumes its fixed manifest. Execution tools +External prep builds the benchmark and its fixed subset manifest. The +experiment runner consumes those artifacts. Execution tools can have multiple implementations without changing the analysis tools or their input contract. Available commands and implementation status are documented in -the [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/README.md#fixed-python-pipeline). +the [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/docs/tools.md#fixed-python-pipeline). diff --git a/labs/metric-benchmark/docs/assets/experiment-flow.svg b/labs/metric-benchmark/docs/assets/experiment-flow.svg index 9114dacd..94a4b9de 100644 --- a/labs/metric-benchmark/docs/assets/experiment-flow.svg +++ b/labs/metric-benchmark/docs/assets/experiment-flow.svg @@ -3,7 +3,7 @@ Metric benchmark: from case examples to correlation - Three phases: prepare case examples for the experiment; run the baseline and mutants against all eval tasks; analyze the saved results. Prep supplies the agent, mutants, case description, benchmark tasks, and experiment configuration. It constructs subsets using a fixed seed and bundles their exact memberships in an authoritative versioned manifest. Execution produces a shared results directory. Analysis consumes the prepared subset manifest, computes mutant scores from baseline and mutant outcomes, and computes coverage from the case description and baseline evidence using metric computation such as Eval Author. The two scores are joined by subset for correlation. Configuration governs execution and analysis. Prep is outside the experiment; the local Harbor loop and future Helix implementation are planned execution backends. + Three phases: prepare case examples for the experiment; run the baseline and mutants against all eval tasks; analyze the saved results. Prep supplies the agent, mutants, case description, benchmark tasks, and experiment configuration. It constructs subsets using a fixed seed and bundles their exact memberships in an authoritative versioned manifest. Execution produces a shared results directory. Analysis consumes the prepared subset manifest, computes mutant scores from baseline and mutant outcomes, and computes coverage from the case description and baseline evidence using metric computation such as Eval Author. The two scores are joined by subset for correlation. Configuration governs execution and analysis. Prep is outside the experiment; the local Harbor loop and future NeMo Helix implementation are planned execution backends. @@ -39,7 +39,7 @@ Prep Prepare case examples for the experiment - Outside the experiment + Prepared externally @@ -83,7 +83,7 @@ Execute evals Mutants + baseline x eval tasks One full benchmark per agent; local jobs run in sequence. - Planned Python tools: local Harbor, then Helix. + Planned Python tools: local Harbor, then NeMo Helix. Results directory diff --git a/labs/metric-benchmark/docs/stages.md b/labs/metric-benchmark/docs/stages.md index 69e21ae3..034577f8 100644 --- a/labs/metric-benchmark/docs/stages.md +++ b/labs/metric-benchmark/docs/stages.md @@ -3,95 +3,37 @@ # Experiment stages -This draft starts with the prepared inputs defined in the -[architecture](architecture.md#supplied-inputs). Agent bundling, mutant generation -and qualification, subset construction, and benchmark construction and publication are release -preparation rather than stages of a benchmark run. The system under evaluation -is a separate input. Coverage computation, including rubric generation when the -system uses it, is part of the experiment. The [three-stage overview](architecture.md#experiment-flow) -shows preparation of case examples as stage 1. Run mutants x eval tasks is stage 2; -score analysis is stage 3, with its individual components numbered 3.1–3.3. - -Each stage has explicit inputs and outputs. Execution produces a results -directory through one of its supported implementations. The Python analysis -runner consumes that directory and calls its operations in a fixed order. An -agent or a person can invoke the runner, but neither supplies missing control -flow or decides whether a stage has completed. +The experiment runs the four stages below using the supplied YAML configuration. +Each stage has a Python tool and an importable API; the orchestrator validates +inputs, calls the tools in dependency order, and passes their outputs onward. -## Stage contract +**Prep:** case construction and subset preparation happen outside this repository. +Runs consume the [prepared inputs](architecture.md#supplied-inputs), including a +fixed subset manifest. -The benchmark's YAML definition selects the inputs, stage settings, and execution -implementation. Each executable stage provides a Python API and a command-line -tool. The orchestrator validates that definition and invokes the APIs in -dependency order. Prep produces the definition and its input artifacts when -building the benchmark; experiment runs consume them. - -Each stage has declared inputs, fixed parameters, outputs, and completion -criteria. Outputs become available downstream only after validation. -The program controls transitions and stops on a stage error. Scoring accepts -only the configured execution statuses, so an unsuccessful execution's numeric -reward is not automatically treated as a valid observation. - -On a repeated invocation, the analysis pipeline recomputes inexpensive -deterministic stages and reuses completed coverage requests after verifying -their inputs and saved files. Changed inputs cause an error in the existing -output directory. A coverage request without a completion receipt runs again; -the adapter owns any external side effects from an interrupted request. - -The implementation stack provides standalone Python tools for analysis -components 3.1–3.3; see its [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/README.md). Stage 2 defines the execution boundary. The current -tools consume the fixtures v1 saved-results index and native evidence, and -never rerun benchmark agents. - -| Component | Command in the implementation stack | +| Stage | Tool | | --- | --- | -| Prep: subset construction | `construct_subsets.py` | -| 3.1. Mutant scoring | `compute_mutant_scores.py` | -| 3.2. Coverage | `compute_coverage.py` | -| 3.3. Correlation | `correlate.py` | -| Analysis runner (manifest input pending) | `run_analysis.py` | - -## 1. Prep - -Prep prepares case examples for the experiment. The -[prepared-input contract](architecture.md#supplied-inputs) describes -the baseline, mutants, case description, tasks with their subset manifest, and configuration supplied to -the experiment. The experiment does not execute this preparation process. - -### Subset construction - -**Purpose:** prepare fixed evaluation subsets shared by every run and metric tool. - -**Inputs:** the pinned benchmark task inventory and fixed subset count, size, -and seed. Execution results and coverage scores are not inputs to prep. - -**Operation:** the sampler sorts task identities and applies its versioned, -seeded algorithm to select distinct subsets of the configured size. The seed is -fixed for the benchmark version and recorded as construction provenance. -Subsets are frozen before inspecting outcomes or coverage scores. - -**Output:** a versioned `subsets.json` manifest with the source benchmark identity, -task inventory and revisions, stable subset IDs, explicit membership, algorithm, -count, size, and seed. The manifest's memberships are authoritative; analysis -reads them rather than regenerating them from the seed. Changing the inventory, -seed, or memberships creates a new benchmark version. - -**Completion:** all subset IDs and memberships are unique, every member belongs -to the pinned inventory, and the requested count and size are satisfied. -Impossible requests fail. Analysis validates the manifest against the execution -inventory; missing observations never cause it to alter the prepared subsets. - -**Current implementation:** the existing `eval_subset_definitions` artifact -contains the memberships and sampling parameters. The Python sampler accepts a -task inventory directly, but its CLI currently obtains that inventory from saved -results. A prep CLI accepting the inventory without results is still needed. -The combined `run_analysis.py` also still generates subsets; it needs to accept -and pin a supplied manifest instead. Standalone mutant scoring and coverage -already accept `--subsets`. The current artifact includes an `experiment_id`; -its binding must be separated from benchmark identity to reuse one unchanged -manifest across independently identified runs. - -## 2. Run mutants x eval tasks +| 1. Run mutants x eval tasks | Planned local Harbor loop; future NeMo Helix implementation | +| 2. Compute mutant scores | `compute_mutant_scores.py` | +| 3. Compute coverage | `compute_coverage.py` | +| 4. Correlation | `correlate.py` | + +The analysis tools are provided in the implementation PRs above this design. +Their [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/docs/tools.md) +describes available commands and implementation status. + +## Stage contract + +Each stage has declared inputs, parameters, outputs, and completion criteria. +Outputs become available downstream only after validation. The runner stops on a +stage error. Scoring accepts only configured execution statuses, so a failed +execution's numeric reward is not automatically treated as a valid observation. + +The analysis pipeline can reuse verified coverage results when their inputs and +saved files match. Changed inputs require a new output directory. Task-subset +membership stays fixed; analysis never resamples subsets or reruns benchmark tasks. + +## 1. Run mutants x eval tasks **Purpose:** execute the benchmark tasks against the baseline and supplied mutants and produce the results directory consumed by analysis. @@ -99,8 +41,8 @@ mutants and produce the results directory consumed by analysis. **Inputs:** prepared agent bundles, benchmark tasks, and experiment execution settings. -**Implementations:** the initial local implementation uses a simple deterministic -loop: baseline first, then the configured mutants, with one Harbor job per agent +**Implementations:** the first execution script will run Harbor tasks in a +basic local loop: baseline first, then the configured mutants, with one Harbor job per agent covering the full benchmark. Agent jobs run sequentially. Task concurrency within a Harbor job is an explicit execution setting; a fully sequential run uses one concurrent trial. The initial analysis supports one trial per agent/task within @@ -112,7 +54,7 @@ The baseline is included because mutant scoring compares its outcomes with each mutant's outcomes. The repository currently supplies the analysis tools; the local Harbor loop is the first execution implementation to add here. -A future Helix implementation will use an orchestrator to run evaluations at +A future NeMo Helix implementation will use an orchestrator to run evaluations at scale. Each implementation owns invocation, scheduling, retries, and collection and produces the same output contract. Analysis can rerun against saved results without repeating this execution phase. @@ -188,9 +130,9 @@ the new producers will adopt this layout, but the reader does not yet enforce these folder names. The local Harbor implementation still needs the code that builds the shared index from native job/trial metadata. -The Helix implementation will export the same agent/run layout and index with +The NeMo Helix implementation will export the same agent/run layout and index with task-level evidence in a supported format. Its scheduling and storage can differ -internally. Analysis reads the common index and evidence without a Helix-specific +internally. Analysis reads the common index and evidence without a NeMo Helix-specific service API. An archive can replace a trial directory when the index explicitly declares its path and format; the native evidence contents still need to be readable by the coverage implementation. @@ -205,12 +147,10 @@ to their scoring policy. The shared contract covers directory contents as well as paths: measurements have consistent meanings, and the evidence formats are readable by the coverage -implementation. A Helix implementation will produce that same contract rather -than making analysis interpret Helix-specific result metadata. - -## 3. Score analysis +implementation. A NeMo Helix implementation will produce that same contract rather +than making analysis interpret NeMo Helix-specific result metadata. -### 3.1. Compute Mutant Scores +## 2. Compute mutant scores **Purpose:** measure each subset's sensitivity to the supplied mutations. @@ -244,7 +184,7 @@ the individual mutant effects used in the aggregate. value and reason. Missing required pairs produce unavailable scores; the function does not remove tasks or mutants to obtain a numeric result. -### 3.2. Compute Coverage +## 3. Compute coverage **Purpose:** obtain the system under evaluation's coverage scores for the same subsets. @@ -288,7 +228,7 @@ requests complete. If the tool generates rubrics or judgments with a model, those files are observations. The [benchmark orchestration](architecture.md#benchmark-orchestration) is deterministic; fresh model outputs are not assumed identical. -### 3.3. Correlation +## 4. Correlation **Purpose:** determine whether coverage predicts mutation sensitivity across evaluation subsets. From 04fce7cc110015a8e7c638e5ed0378aafecc6b9a Mon Sep 17 00:00:00 2001 From: mstaats Date: Fri, 23 Oct 2026 14:32:07 +0000 Subject: [PATCH 5/5] docs(metric-benchmark): address architecture review feedback Signed-off-by: mstaats --- labs/metric-benchmark/README.md | 5 +++-- labs/metric-benchmark/docs/architecture.md | 19 +++++++++---------- labs/metric-benchmark/docs/stages.md | 18 ++++++++++-------- 3 files changed, 22 insertions(+), 20 deletions(-) diff --git a/labs/metric-benchmark/README.md b/labs/metric-benchmark/README.md index e569d86a..8abf8a3f 100644 --- a/labs/metric-benchmark/README.md +++ b/labs/metric-benchmark/README.md @@ -16,8 +16,9 @@ sight. Prep happens outside this repository. The benchmark consumes prepared case examples, fixed task-subset manifests, and a YAML experiment configuration. -The experiment has four executable stages, each exposed as a Python tool and -library API: +The architecture has three phases: external prep, execution, and score analysis. +Execution and score analysis comprise four executable stages, each exposed as a +Python tool and library API: 1. **Run mutants x eval tasks:** run the baseline and every mutant against the benchmark tasks and save the results. The first execution script will be a diff --git a/labs/metric-benchmark/docs/architecture.md b/labs/metric-benchmark/docs/architecture.md index 1a0fbd04..def95441 100644 --- a/labs/metric-benchmark/docs/architecture.md +++ b/labs/metric-benchmark/docs/architecture.md @@ -11,8 +11,8 @@ evaluate our ability to do this separately. The question we answer here is **Is our coverage metric a good predictor for agentic evaluation quality?** -This is hard question to answer. Benchmarking a _metric_ is a more complicated ask than simply benchmarking an -agent's ability to perform veriable tasks. Our approach is to conducting an experiment based on mutation +This is a hard question to answer. Benchmarking a _metric_ is a more complicated ask than simply benchmarking an +agent's ability to perform verifiable tasks. Our approach is to conduct an experiment based on mutation testing (see [Mutation testing — Wikipedia](https://en.wikipedia.org/wiki/Mutation_testing)), measuring the correlations between the ability of eval tasks to detect mutants and their coverage. @@ -32,10 +32,12 @@ or resource use. Results are reported per case and measurement. ## Experiment flow -The architecture has three stages: **prep**, **run mutants x eval tasks**, and +The architecture has three phases: **prep**, **run mutants x eval tasks**, and **score analysis**. Prep prepares case examples for the experiment. The experiment consumes those inputs, runs the baseline and mutants against the eval tasks, -and analyzes the resulting directory. +and analyzes the resulting directory. Execution and score analysis contain four +executable stages: run mutants x eval tasks, compute mutant scores, compute +coverage, and correlation. Prep is external to those stages. ![Metric benchmark architecture: prepare case examples, run mutants x eval tasks, and analyze coverage versus mutant sensitivity.](assets/experiment-flow.svg) @@ -159,8 +161,6 @@ Mutant scoring uses mean signed baseline-to-mutant degradation, with equal weight for tasks and mutants. Correlation compares the ranks of the paired scores, with average ranks for ties. The [component descriptions](stages.md#2-compute-mutant-scores) specify these calculations and the handling of missing or undefined values. -The [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/docs/tools.md) documents the commands and their current -implementation status. ### Metric computation (e.g. Eval Author) @@ -177,8 +177,8 @@ storage. Tool-specific adapters provide the common invocation interface needed to evaluate other systems, such as skill-based or LangChain-based workflows. They handle differences in native inputs, runtimes, and outputs; the experiment -still performs the same Compute Coverage stage. Integration details and current -implementation limits are described in the [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/docs/tools.md#coverage-system-invocation). +still performs the same Compute Coverage stage. Each adapter defines and validates the native inputs and outputs required by +its system. ## Benchmark orchestration @@ -210,5 +210,4 @@ configuration and coordinates their execution. External prep builds the benchmark and its fixed subset manifest. The experiment runner consumes those artifacts. Execution tools can have multiple implementations without changing the analysis tools or their -input contract. Available commands and implementation status are documented in -the [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/docs/tools.md#fixed-python-pipeline). +input contract. diff --git a/labs/metric-benchmark/docs/stages.md b/labs/metric-benchmark/docs/stages.md index 034577f8..19e5f95b 100644 --- a/labs/metric-benchmark/docs/stages.md +++ b/labs/metric-benchmark/docs/stages.md @@ -3,7 +3,9 @@ # Experiment stages -The experiment runs the four stages below using the supplied YAML configuration. +The architecture has three phases: external prep, execution, and score analysis. +The experiment runs the four executable stages below using the supplied YAML +configuration: one execution stage and three analysis stages. Each stage has a Python tool and an importable API; the orchestrator validates inputs, calls the tools in dependency order, and passes their outputs onward. @@ -18,10 +20,6 @@ fixed subset manifest. | 3. Compute coverage | `compute_coverage.py` | | 4. Correlation | `correlate.py` | -The analysis tools are provided in the implementation PRs above this design. -Their [tool guide](https://github.com/NVIDIA-NeMo/eval-author/blob/feat/metric-benchmark-record/labs/metric-benchmark/docs/tools.md) -describes available commands and implementation status. - ## Stage contract Each stage has declared inputs, parameters, outputs, and completion criteria. @@ -171,7 +169,11 @@ M(subset) = mean over selected mutants of ``` The scorer uses this equally weighted signed score and preserves -negative differences when a mutant outperforms the baseline. Missing required +negative differences when a mutant outperforms the baseline. Negative subset +scores and individual mutant effects are flagged for inspection in the scoring +output. These diagnostics do not fail the stage, clip values, or exclude pairs +from correlation. A negative value can reflect execution variability or a +mutation that improves performance on those tasks. Missing required pairs make a subset unavailable; no tasks or mutants are silently dropped. One run per agent and an explicit attempt-selection rule are required. This analysis does not pool repetitions or apply alternative weighting. @@ -201,8 +203,8 @@ command. The adapter translates the request into its system's native inputs, executes that system through its native interface, validates its outputs, and converts the result to the common response. The coverage stage validates the adapter's JSON response as a numeric score or an unavailable value with a reason. -Missing baseline evidence, invalid responses, -input modification, timeouts, and nonzero exits fail the stage. +Missing baseline evidence, invalid responses, input modification, +timeouts, and nonzero exits fail the stage. The system may generate a rubric as part of its method. Its adapter places the supporting outputs in the work directory. The runner does not infer which