diff --git a/labs/metric-benchmark/README.md b/labs/metric-benchmark/README.md new file mode 100644 index 00000000..8abf8a3f --- /dev/null +++ b/labs/metric-benchmark/README.md @@ -0,0 +1,32 @@ + + + +# Metric benchmark + +An experimental benchmark for evaluating how well our coverage metric predicts +an evaluation set's effectiveness at detecting mutations in an agent. This +project outlines the architecture and develops deterministic Python tooling for +running a growing set of case examples and evaluating our metric through +mutation testing. + +**This is a work in progress.** The tools and benchmark will change. Case examples +will grow, change, and may be dropped as we learn. We believe in building in plain +sight. + +Prep happens outside this repository. The benchmark consumes prepared case +examples, fixed task-subset manifests, and a YAML experiment configuration. + +The architecture has three phases: external prep, execution, and score analysis. +Execution and score analysis comprise four executable stages, each exposed as a +Python tool and library API: + +1. **Run mutants x eval tasks:** run the baseline and every mutant against the + benchmark tasks and save the results. The first execution script will be a + basic local Harbor loop; support for **NeMo Helix** will follow. +2. **Compute mutant scores:** measure degradation across the mutants for each + predefined task subset. +3. **Compute coverage:** evaluate our metric on those same subsets. +4. **Correlation:** compare the paired coverage and mutant scores across subsets. + +See the [architecture](docs/architecture.md) and [stage descriptions](docs/stages.md) +for how the experiment works. diff --git a/labs/metric-benchmark/docs/architecture.md b/labs/metric-benchmark/docs/architecture.md new file mode 100644 index 00000000..def95441 --- /dev/null +++ b/labs/metric-benchmark/docs/architecture.md @@ -0,0 +1,213 @@ + + + +# Architecture + +## Evaluating Eval Coverage + +Our formulation of eval authoring is based on (1) defining +a list of coverage goals and (2) generating tasks to achieve these goals. We +evaluate our ability to do this separately. + +The question we answer here is **Is our coverage metric a good predictor for agentic evaluation quality?** + +This is a hard question to answer. Benchmarking a _metric_ is a more complicated ask than simply benchmarking an +agent's ability to perform verifiable tasks. Our approach is to conduct an experiment based on mutation +testing (see [Mutation testing — Wikipedia](https://en.wikipedia.org/wiki/Mutation_testing)), +measuring the correlations between the ability of eval tasks to detect mutants and their coverage. + +The experiment is a program with supplied inputs, a fixed sequence of operations, +and defined outputs. It executes benchmark tasks against the baseline agent and +a fixed cohort of mutants. Its analysis consumes predefined task subsets, computes a +mutant score for each subset, invokes the coverage tool on the same subsets, and +correlates the two scores. The current implementation starts from saved execution +results. Execution is a replaceable component whose output is a shared results +directory consumed by analysis. + +Comparisons between coverage tools use the same subsets, mutant cohort, and +outcome scoring policy. Higher mutant scores mean greater detected degradation, +so the comparison seeks a positive relationship with coverage. Each analysis +produces a result for one case and one outcome measurement, such as correctness +or resource use. Results are reported per case and measurement. + +## Experiment flow + +The architecture has three phases: **prep**, **run mutants x eval tasks**, and +**score analysis**. Prep prepares case examples for the experiment. The experiment +consumes those inputs, runs the baseline and mutants against the eval tasks, +and analyzes the resulting directory. Execution and score analysis contain four +executable stages: run mutants x eval tasks, compute mutant scores, compute +coverage, and correlation. Prep is external to those stages. + +![Metric benchmark architecture: prepare case examples, run mutants x eval tasks, and analyze coverage versus mutant sensitivity.](assets/experiment-flow.svg) + +[Open the SVG diagram](assets/experiment-flow.svg). + +## Benchmark definition + +A fixed metric benchmark is defined by a specific YAML file, `benchmark.yaml`, +and the versioned inputs it references. It identifies the baseline, mutant +cohort, case description, Harbor tasks, prepared subset manifest, execution +implementation, metric computation, and scoring settings. Here, the benchmark +means this complete experiment configuration; the Harbor task set is one of +its inputs. + +Prep builds that benchmark: it produces the case artifacts and fixed subsets, +then emits the YAML definition that ties them together. Running the benchmark +means reading that definition and executing the configured experiment. Each run +has its own results while retaining the same benchmark definition and input +identities. Deliberate changes produce another versioned configuration, making +it possible to compare experimental designs as well as metric tools while +keeping the unchanged inputs fixed. + +## 1. Prep + +Prep prepares case examples for the experiment. This case-construction process +is outside this repository and the experiment itself. It produces the +baseline agent, creates and qualifies its mutants, and assembles the case +description, task set, and configuration in the benchmark's input format. Prep +also constructs the eval subsets and bundles their manifest with the task set. +Prep emits `benchmark.yaml` referencing these outputs. Released benchmarks +include the YAML definition and its prepared case examples. The experiment +consumes them; it does not regenerate or select mutants based on its results. + +### Supplied inputs + +A run receives five supplied inputs. Case packages provide the first four; +the experiment configuration specifies how to use them. The table describes +their logical contents. + +| Input | Required contents | +| --- | --- | +| Baseline agent | An executable bundle with an immutable identity and a documented invocation contract | +| Mutants | Executable bundles with distinct identities, their baseline relationship, and mutation descriptions | +| Case description | Stable case identity and intended agent behavior; an Ethos is one possible representation | +| Benchmark tasks | Fixed task membership, task identities, environments, correctness verifiers, and a versioned subset manifest | +| Experiment configuration | `benchmark.yaml`: case-to-task-set associations, fixed mutant cohort, execution implementation and settings, subset-manifest identity, coverage-system settings and permitted evidence, and scoring parameters | + +Baseline and mutant bundles are prepared and checked before release. +Tasks and their environments are ready for execution. At present only Harbor tasks are used. + +The configuration supplies the choices that determine execution: the fixed subset manifest, +selected agent runs, scoring rules, and the coverage-tool command and settings. +The program validates these inputs before using them. Missing or inconsistent +inputs cause an error. + +### Subset manifest + +Prep constructs subsets from the pinned task inventory, before inspecting +execution outcomes or coverage scores. Each case bundles `subsets.json`, a +versioned manifest containing stable subset IDs, exact task identities and +revisions for each subset, the source benchmark identity, and the construction +algorithm, subset count, subset size, and fixed seed. + +The manifest is authoritative: every run and metric tool uses the same explicit +memberships. The seed documents construction and stays fixed for that benchmark +version; analysis does not resample or regenerate subsets. Changing the seed, +inventory, or memberships creates a new benchmark version. Analysis validates +the manifest against the execution inventory and rejects mismatched identities. +Missing results follow the scoring policy without changing subset membership. + +## 2. Run mutants x eval tasks + +This stage executes the same benchmark tasks against the baseline and every +selected mutant. Its two architectural components +are the **execution implementation** and the **results-directory interface**. + +| Component | Role | +| --- | --- | +| Execution implementation | Runs the configured agent-by-task matrix and assembles its outputs. The initial implementation is a simple local Harbor loop, one agent's full benchmark at a time. A future NeMo Helix implementation will orchestrate evaluations at scale. | +| Results directory | Carries the task inventory, agent/task identities, statuses, measurements, and native evidence into analysis. Both execution implementations produce the same contract. | + +The proposed layout is agent-first: `agents////`, with +`results.json` at the root indexing those trial results. Each run contains the +native output of that agent running the benchmark. The +[results-directory contract](stages.md#results-directory-contract) specifies the +layout. Invocation, scheduling, and collection are internal to the execution +implementation; they do not add generic experiment stages. + +## 3. Score analysis + +Score analysis consumes the results directory and produces the benchmark's score. +It consists of three components. Mutant scoring and coverage consume the prepared +subset manifest and execution outputs; correlation consumes their derived scores. None of +these components reruns the benchmark tasks. + +![How score analysis works: each predefined task subset produces a coverage score and a mutant score; each pair becomes one point for correlation.](assets/score-analysis.svg) + +For each subset in the manifest, analysis selects those tasks' saved results for +the baseline and every mutant. Comparing their outcomes produces one mutant +score for the subset. Separately, metric computation uses that same subset's +baseline evidence and the case description to produce one coverage score. +Repeating this for all predefined subsets gives one pair of scores per subset. +Each point in the figure represents a subset; correlation measures whether +subsets with higher coverage tend to detect more degradation. The example +values are illustrative. Subsets may overlap, so these points are not independent +observations. The reported correlation is descriptive, without independent-sample +significance claims. + +| Component | Inputs | Output | +| --- | --- | --- | +| Compute Mutant Scores | Subset manifest and baseline/mutant measurements from `results.json` | A mutant-sensitivity score for each subset | +| Compute Coverage | Subset manifest, case description, and indexed baseline evidence, evaluated with the configured system | A coverage score for each subset and any artifacts generated by that method | +| Correlation | Coverage and mutant scores joined by subset ID | Correlation across subsets, with paired values and any exclusions | + +The Python analysis runner validates the inputs and invokes these components in +a fixed order. Subset membership is fixed during prep. Coverage receives baseline evidence without mutant outcomes or +correlation results. The experiment ends with correlation; output storage and +packaging are implementation details. + +Mutant scoring uses mean signed baseline-to-mutant degradation, with equal +weight for tasks and mutants. Correlation compares the ranks of the paired +scores, with average ranks for ties. The [component descriptions](stages.md#2-compute-mutant-scores) +specify these calculations and the handling of missing or undefined values. + +### Metric computation (e.g. Eval Author) + +Eval Author is the primary system under evaluation. For each task subset, the +runner supplies the case description, task identities, and baseline execution +evidence, and obtains a coverage score. The correlation stage compares those +scores with the independently computed mutant scores. + +Coverage computation may include rubric generation or model calls as part of +the system's method. Coverage scores and any generated rubrics are experiment +outputs. The Python runner controls stage ordering, validation, and result +storage. + +Tool-specific adapters provide the common invocation interface needed to +evaluate other systems, such as skill-based or LangChain-based workflows. +They handle differences in native inputs, runtimes, and outputs; the experiment +still performs the same Compute Coverage stage. Each adapter defines and validates the native inputs and outputs required by +its system. + +## Benchmark orchestration + +This benchmark runs the same Harbor tasks against multiple versions of an agent: +the baseline and its mutants. A lightweight experiment runner coordinates those +runs and analyzes their combined results. Harbor's verifiers score individual +task executions; the metric benchmark's score is computed across many executions +by comparing coverage with mutant sensitivity across task subsets. That analysis +sits above Harbor, so the runner provides the orchestration and scoring that a +single Harbor job or task verifier does not. This control flow is deterministic +Python code. + +### Stage tools and Python libraries + +Each executable experiment stage has a Python tool and an importable Python +API. The orchestrator reads and validates `benchmark.yaml`, resolves its inputs, +and calls these APIs in dependency order, passing each stage's outputs to the +next. The command-line tools expose the same operations for standalone use. +Stage behavior lives in these reusable tools; the orchestrator supplies the +configuration and coordinates their execution. + +| Stage | Python tool / library operation | +| --- | --- | +| Execute evals | Local Harbor execution tool, or a future NeMo Helix execution tool; both expose the same results-directory contract | +| Compute mutant scores | `compute_mutant_scores.py` / `metric_benchmark.scoring.compute_mutant_scores` | +| Compute coverage | `compute_coverage.py` / `metric_benchmark.coverage.invoke_coverage` | +| Correlation | `correlate.py` / `metric_benchmark.correlation.correlate` | + +External prep builds the benchmark and its fixed subset manifest. The +experiment runner consumes those artifacts. Execution tools +can have multiple implementations without changing the analysis tools or their +input contract. diff --git a/labs/metric-benchmark/docs/assets/experiment-flow.svg b/labs/metric-benchmark/docs/assets/experiment-flow.svg new file mode 100644 index 00000000..94a4b9de --- /dev/null +++ b/labs/metric-benchmark/docs/assets/experiment-flow.svg @@ -0,0 +1,134 @@ + + + + + Metric benchmark: from case examples to correlation + Three phases: prepare case examples for the experiment; run the baseline and mutants against all eval tasks; analyze the saved results. Prep supplies the agent, mutants, case description, benchmark tasks, and experiment configuration. It constructs subsets using a fixed seed and bundles their exact memberships in an authoritative versioned manifest. Execution produces a shared results directory. Analysis consumes the prepared subset manifest, computes mutant scores from baseline and mutant outcomes, and computes coverage from the case description and baseline evidence using metric computation such as Eval Author. The two scores are joined by subset for correlation. Configuration governs execution and analysis. Prep is outside the experiment; the local Harbor loop and future NeMo Helix implementation are planned execution backends. + + + + + + + + + + + + Metric benchmark + Does coverage predict how well eval tasks detect agent degradation? + + + + + 1 + Prep + Prepare case examples for the experiment + + Prepared externally + + + + + + + + + Agent + Baseline bundle + Mutants + Fixed variant cohort + Case description + Intended behavior + Benchmark tasks + Tasks + verifiers + benchmark.yaml + Fixed configuration + + Construct eval subsets + Pinned task inventory + fixed seed + Freeze membership before inspecting outcomes. + + + Subset manifest + Stable subset IDs + exact task membership + Versioned subsets.json · algorithm + fixed seed recorded + Prep emits benchmark.yaml + its input artifacts. The orchestrator reads the YAML and invokes Python stage tools. + + + + + + + + 2 + Run mutants x eval tasks + Same tasks · baseline + every mutant + + + Execute evals + Mutants + baseline x eval tasks + One full benchmark per agent; local jobs run in sequence. + Planned Python tools: local Harbor, then NeMo Helix. + + + Results directory + Shared index + native task results + results.json + agents/<agent>/<run>/<native-trial>/ + + + + + + + 3 + Score analysis + Read the saved results. + Compare the same subsets. + + + Subset manifest + Fixed membership from prep + + + + + + METHOD BEING EVALUATED + Metric computation + (e.g. Eval Author) + + + + Compute mutant scores + Baseline + mutant outcomes + compute_mutant_scores.py + + + Compute coverage + Case description + baseline evidence + compute_coverage.py + + + + Join by subset + + Correlation + correlate.py + + diff --git a/labs/metric-benchmark/docs/assets/score-analysis.svg b/labs/metric-benchmark/docs/assets/score-analysis.svg new file mode 100644 index 00000000..23ccaa53 --- /dev/null +++ b/labs/metric-benchmark/docs/assets/score-analysis.svg @@ -0,0 +1,142 @@ + + + + +Each task subset produces one pair of scores +First, take a predefined subset such as tasks 2, 5, and 8. Read its saved execution results for the baseline and every mutant. Compare baseline and mutant outcomes to compute one mutant score for the subset. Separately, use its baseline evidence and the case description to compute one coverage score. Repeat for every subset in the fixed manifest. Each subset contributes one coverage and mutant score pair, shown as one point on a scatter plot. Correlation measures whether subsets with higher coverage tend to detect more degradation. Example scores and plotted points are illustrative, not experiment results. Analysis does not rerun tasks. + + + + + + +One task subset → two scores → one point +Analysis reuses saved executions; tasks do not run again for each subset. + + +1. Score the same subset in two ways +Example subset A: tasks 2, 5, and 8 +All mutants + +Baseline + +M1 +M2 +… +Mm +Task 2 + + + + + + + +… + + +Task 5 + + + + + + + +… + + +Task 8 + + + + + + + +… + + +Each cell represents a saved task result. + + + + +Compute mutant score +Compare baseline vs. each mutant on these tasks. +Average the degradation → one mutant score. + + + + +Compute coverage +Use baseline evidence + the case description. +Metric computation → one coverage score. +Together: one (coverage, mutant score) pair for subset A + + +2. Repeat for every predefined subset, then compute correlation + +SUBSET +TASKS +COVERAGE +MUTANT SCORE + + + +A +2, 5, 8 +0.25 +0.10 + + +B +1, 4, 8 +0.55 +0.40 + + +C +3, 5, 7 +0.80 +0.70 +… many fixed subsets → many pairs +Illustrative scores; not experiment results. + +Each row +is one point + + + + +Coverage score → +Mutant score → + + + + + + + + + + + + + +A + +B + +C +Do subsets with higher coverage also detect more degradation? + diff --git a/labs/metric-benchmark/docs/stages.md b/labs/metric-benchmark/docs/stages.md new file mode 100644 index 00000000..19e5f95b --- /dev/null +++ b/labs/metric-benchmark/docs/stages.md @@ -0,0 +1,262 @@ + + + +# Experiment stages + +The architecture has three phases: external prep, execution, and score analysis. +The experiment runs the four executable stages below using the supplied YAML +configuration: one execution stage and three analysis stages. +Each stage has a Python tool and an importable API; the orchestrator validates +inputs, calls the tools in dependency order, and passes their outputs onward. + +**Prep:** case construction and subset preparation happen outside this repository. +Runs consume the [prepared inputs](architecture.md#supplied-inputs), including a +fixed subset manifest. + +| Stage | Tool | +| --- | --- | +| 1. Run mutants x eval tasks | Planned local Harbor loop; future NeMo Helix implementation | +| 2. Compute mutant scores | `compute_mutant_scores.py` | +| 3. Compute coverage | `compute_coverage.py` | +| 4. Correlation | `correlate.py` | + +## Stage contract + +Each stage has declared inputs, parameters, outputs, and completion criteria. +Outputs become available downstream only after validation. The runner stops on a +stage error. Scoring accepts only configured execution statuses, so a failed +execution's numeric reward is not automatically treated as a valid observation. + +The analysis pipeline can reuse verified coverage results when their inputs and +saved files match. Changed inputs require a new output directory. Task-subset +membership stays fixed; analysis never resamples subsets or reruns benchmark tasks. + +## 1. Run mutants x eval tasks + +**Purpose:** execute the benchmark tasks against the baseline and supplied +mutants and produce the results directory consumed by analysis. + +**Inputs:** prepared agent bundles, benchmark tasks, and experiment execution +settings. + +**Implementations:** the first execution script will run Harbor tasks in a +basic local loop: baseline first, then the configured mutants, with one Harbor job per agent +covering the full benchmark. Agent jobs run sequentially. Task concurrency within +a Harbor job is an explicit execution setting; a fully sequential run uses one +concurrent trial. The initial analysis supports one trial per agent/task within +the selected run; repetition aggregation remains future work. + +This phase is expected to be expensive: with `M` mutants and `T` tasks, even a +single execution per pair requires `(M + 1) × T` task executions before retries. +The baseline is included because mutant scoring compares its outcomes with each +mutant's outcomes. The repository currently supplies the analysis tools; the +local Harbor loop is the first execution implementation to add here. + +A future NeMo Helix implementation will use an orchestrator to run evaluations at +scale. Each implementation owns invocation, scheduling, retries, and collection +and produces the same output contract. Analysis can rerun against saved results +without repeating this execution phase. + +### Results-directory contract + +**Proposed layout:** agent first, then benchmark run, with native trial outputs +inside each run. One results root represents one case and benchmark revision. + +```text +execution-results/ + results.json # Shared analysis index and task inventory + agents/ + baseline/ + run-0001/ # One intact Harbor job directory + config.json + lock.json + result.json # Harbor's job-level result + job.log + task-a__/ + config.json + lock.json + result.json # Harbor's per-trial result + agent/ # Agent execution evidence + verifier/ # Verifier outputs + ... # Other native Harbor artifacts + task-b__/ + ... + mutant-001/ + run-0001/ + ... # Same benchmark, this mutant's results + mutant-002/ + run-0001/ + ... +``` + +The native portion shown is illustrative of Harbor's single-step trial layout; +its actual files depend on the pinned Harbor version and task type. The local +implementation chooses the agent/run parent directory and preserves the native +job contents, including native trial names. Explicit run names avoid using a +timestamp as the benchmark's logical run identity. + +Agent-first matches the proposed execution loop and keeps each Harbor job's +configuration, aggregate results, and task evidence together. A task-first tree +would split that job across multiple directories. A second physical copy is +unnecessary: the shared index supports either agent-wise or task-wise views. +Separate run directories allow later executions without overwriting earlier +ones; the current scorer selects one run per agent. + +**Index:** `results.json` uses the existing v1 `raw_execution_results` schema. +It includes the complete benchmark/task inventory and each execution's agent, +run, task revision, attempt, status, measurements, and native-evidence location. +This shared index is distinct from Harbor's own job and trial `result.json` +files. All evidence paths are relative to `execution-results/`. For example: + +```json +{ + "native_results_path": "agents/mutant-001/run-0001/task-a__x7k2p9m", + "native_results_format": "directory" +} +``` + +This is an excerpt of an execution entry, not a complete results index. It points +to one trial, not the whole job directory. Folder labels are filesystem-safe +aliases; the index carries the actual agent and task identities. A run identity +is unique across the results index even when agents reuse the folder label +`run-0001`. Native trial names can contain generated suffixes, and analysis never +parses those names to recover task identities, repetitions, or retry order. + +The current reader already supports these relative paths and native directories +or archives. Existing commands accept explicit index and evidence-root paths; +the new producers will adopt this layout, but the reader does not yet enforce +these folder names. The local Harbor implementation still needs the code that +builds the shared index from native job/trial metadata. + +The NeMo Helix implementation will export the same agent/run layout and index with +task-level evidence in a supported format. Its scheduling and storage can differ +internally. Analysis reads the common index and evidence without a NeMo Helix-specific +service API. An archive can replace a trial directory when the index explicitly +declares its path and format; the native evidence contents still need to be +readable by the coverage implementation. + +**Completion contract:** the output accounts for the configured agent-by-task +matrix and identifies failed attempts and unavailable measurements explicitly. +A completed task with a zero reward remains distinct from an execution error. +The output index and referenced evidence agree on identities and formats. +A full execution-completeness validator is still to be implemented; the existing +analysis tools validate the index and handle unavailable observations according +to their scoring policy. + +The shared contract covers directory contents as well as paths: measurements +have consistent meanings, and the evidence formats are readable by the coverage +implementation. A NeMo Helix implementation will produce that same contract rather +than making analysis interpret NeMo Helix-specific result metadata. + +## 2. Compute mutant scores + +**Purpose:** measure each subset's sensitivity to the supplied mutations. + +**Inputs:** the prepared subset manifest, Raw Execution Results for the baseline and +mutants, the fixed mutant cohort, and the scoring policy. + +**Operation:** the scoring function selects the configured runs and attempts, +pairs baseline and mutant outcomes by task, and computes one score per subset +for the configured measurement. It rejects ambiguous repetitions. The metric +name and higher-is-better or lower-is-better direction are configuration fields. + +For reward, the implemented score is mean signed reward loss: + +```text +S(agent, subset) = mean task reward for that agent on that subset +M(subset) = mean over selected mutants of + [S(baseline, subset) - S(mutant, subset)] +``` + +The scorer uses this equally weighted signed score and preserves +negative differences when a mutant outperforms the baseline. Negative subset +scores and individual mutant effects are flagged for inspection in the scoring +output. These diagnostics do not fail the stage, clip values, or exclude pairs +from correlation. A negative value can reflect execution variability or a +mutation that improves performance on those tasks. Missing required +pairs make a subset unavailable; no tasks or mutants are silently dropped. +One run per agent and an explicit attempt-selection rule are required. +This analysis does not pool repetitions or apply alternative weighting. + +**Outputs:** Mutant Scores, keyed by subset ID, with the case, baseline, mutant +cohort, aggregation, and scoring variation identified. The output also contains +the individual mutant effects used in the aggregate. + +**Completion:** every requested subset has a score or an explicit unavailable +value and reason. Missing required pairs produce unavailable scores; the +function does not remove tasks or mutants to obtain a numeric result. + +## 3. Compute coverage + +**Purpose:** obtain the system under evaluation's coverage scores for the same +subsets. + +**Inputs:** the case description, the prepared subset manifest, permitted baseline +evidence from Raw Execution Results, and the configured system under evaluation +with its tool-specific adapter and settings. The current request supplies task +identities and native baseline evidence for the subset; additional source or +task-content interfaces are not implemented. + +**Operation:** the coverage stage copies the subset's selected baseline evidence +into a work directory, writes a JSON request, and invokes the configured adapter +command. The adapter translates the request into its system's native inputs, +executes that system through its native interface, validates its outputs, and +converts the result to the common response. The coverage stage validates the +adapter's JSON response as a numeric score or an unavailable value with a reason. +Missing baseline evidence, invalid responses, input modification, +timeouts, and nonzero exits fail the stage. + +The system may generate a rubric as part of its method. Its adapter places the +supporting outputs in the work directory. The runner does not infer which +rubrics or judgments the tool should have produced; required adapter-specific +outputs need an adapter-specific validator. The current implementation provides +the common invocation protocol and a synthetic adapter; it does not yet supply +production adapters for Eval Author or other systems. + +Each subset is evaluated independently. The runner has no shared-rubric +generation pass or cross-subset measurement cache. Those methods would require +explicit additional interfaces and validation. Mutant outcomes, mutant scores, +and correlation results are absent from the coverage request. Staging inputs +does not sandbox the tool's access to the host. + +**Outputs:** Coverage Scores, keyed by subset ID, with system and configuration +identities, plus the saved requests, responses, and files produced in each work +directory. The stage writes a completion receipt with file digests for each +successful request. + +**Completion:** every subset has a validated response and completion receipt. +The stage writes the aggregate Coverage Scores artifact only after all subset +requests complete. If the tool generates rubrics or judgments with a model, +those files are observations. The [benchmark orchestration](architecture.md#benchmark-orchestration) +is deterministic; fresh model outputs are not assumed identical. + +## 4. Correlation + +**Purpose:** determine whether coverage predicts mutation sensitivity across +evaluation subsets. + +**Inputs:** Coverage Scores and Mutant Scores with matching subset definitions. + +**Operation:** the correlation function verifies matching source identities and +subset IDs, joins scores by subset ID, and computes Spearman rank correlation +using average ranks for ties. It rejects unavailable pairs by default. With the +explicit exclusion policy, it omits those pairs and writes their reasons into +the result alongside the included pairs. + +**Outputs:** Correlation Results containing the coefficient, included pairs, +configuration identities, and exclusions. The included count is the length of +the pair list. + +**Completion:** each comparison has a result or an explicit undefined result. +Constant scores or too few usable pairs do not become zero correlation. +Unexpected identity or membership mismatches are errors, not silently dropped +observations. The missing-value policy is an explicit function input. + +Overlapping subsets share observations. Treating them as independent samples +would overstate the strength of evidence. The reported coefficient is +descriptive; the analysis does not report independent-sample significance or +claim general validity from that coefficient alone. + +Mutant scoring and coverage measurement depend on the subset definitions, and +their results meet at correlation. The current runner executes them sequentially +in the order shown above. None of the analysis stages reruns benchmark agents.