feat(metric-benchmark): compute coverage through an Eval Author adapter by mstaats-nvidia · Pull Request #96 · NVIDIA-NeMo/eval-author

mstaats-nvidia · 2026-10-20T17:56:13Z

Compute coverage for each fixed subset using baseline evidence and the case description. Preserve requests, authored observations, response artifacts and receipts; reject stale cached responses. Mutant outcomes are excluded from coverage requests.

compute_coverage.py is a thin CLI over invoke_coverage. It accepts cases and analysis configuration, the case ID, saved execution root, work directory and output path. Saved case identities are checked without requiring executable agent bundles or task environments. Command parsing and main() live in the CLI scripts.

The Eval Author adapter invokes the configured author command, then native measurement and reporting scripts through compute_audit_coverage. Missing judgments affect only the selected score scope. Authoring and semantic judgments may use models; experiment orchestration does not. Invalid responses and modified input evidence retain diagnostic workspaces and logs. Cache errors identify the affected subset. Linux/POSIX behavior, WSL guidance and absolute executable requirements are documented.

Usage and artifact contracts: labs/metric-benchmark/docs/compute-coverage.md and labs/metric-benchmark/docs/eval-author-coverage.md.

Validation at 11313b3: 3,436 passed, 36 skipped locally; focused coverage/adapter tests 24 passed. Ruff/format, Python 3.12/3.13 type checks, copyright, license and whitespace checks passed. Skips include optional Docker, live-model, external-runtime and platform-specific checks. No new live model or Docker trials were run for these review fixes. Hosted CI passed, including the full Python 3.12 and 3.13 jobs.

Earlier pressure testing exercised six real Harbor trials (baseline and one mutant across three tasks), native Eval Author tool-only coverage, cached resume, standalone/coordinator parity and analysis without executable inputs. Constant mutant scores correctly produced unavailable correlation. That exercise validates the pipeline behavior, not scientific metric validity.

Stack after merged #95: main → #96 coverage → #97 correlation → #99 coordinator.

Defines an experimental benchmark for evaluating how well our coverage metric predicts evaluation effectiveness through mutation testing. The concise [root README](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/README.md) describes the project, its work-in-progress status, and the expectation that case examples will grow, change, or be dropped as we build in plain sight. Prep is external to this repository; runs consume prepared case examples, fixed subset manifests, and a YAML benchmark definition. The [stage descriptions](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/docs/stages.md) focus on four executable stages: run mutants × eval tasks, compute mutant scores, compute coverage, and correlate the score pairs. The first execution script will be a basic local Harbor loop; NeMo Helix support is planned afterward. The [architecture and both SVG figures](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/docs/architecture.md) explain the inputs, component boundaries, shared results directory, and statistical comparison. This is the documentation-only base of the stack. Implementation PRs #94 → #105 → #106 → #95 → #96 → #97 → #99 inherit the overview and design unchanged. Their detailed CLI guide lives separately in `docs/tools.md`. Validation: the five files in this PR are Markdown/SVG only. Local links and anchors, Markdown fences, headers, SVG XML, and whitespace checks pass; the updated figure was rendered and visually inspected. Restacking preserves all source and test files byte-for-byte. No live evaluations or model calls were made.

asutermo · 2026-10-30T16:31:41Z

+Completed subset requests have file-digest receipts. Resume verifies those files
+and inputs before reuse. `--cached-only` prohibits new invocations. A command
+failure preserves its private diagnostic workspace without a completion receipt.
+On POSIX, timeout cancellation kills the command's process group, including its


POSIX call out needed here (have we tried any of this on windows haha)?

Agreed, yea. My move for this is to always say use "WSL" fwiw. :-)

asutermo · 2026-10-30T16:32:16Z

+    return False
+
+
+def compute_eval_author(request_path: Path) -> Score:


minor - might rename this since it's audit specifc

Agreed, done

aleckhoury

Overall sound. Verified:

Mutant outcomes never reach coverage requests. Requests carry only the case description, the subset's task IDs, copied baseline evidence with its digests, and the coverage settings.
Cached responses are reused only when the request digest, the config digest and every saved file match the receipt.
The adapter builds argv lists (no shell). The measure.py/report.py flags and output layout match skills/eval-author-audit/scripts/audit_spec.

At this head: 158 passed, 1 skipped; ruff is clean. I'd fix the score_scope gate before merge. The other two inline items are smaller.

aleckhoury · 2026-10-30T16:54:10Z

+    for details in sorted(Path("measurements").glob("task=*/run=*/*/details.json")):
+        if _unjudged(read_json(details)):
+            return Score(
+                value=None, reason=f"Eval Author evidence requires missing or unsupported judgments: {details}"
+            )


This gate ignores score_scope. Any details.json with unjudged or unsupported evidence nulls the score, even when the score covers a single item kind.

Repro with eval_author_example(non_tool=True): score_scope: tool, and the audit has capability items but no capability judgments.

Default methods (all three): every subset is null.

methods: [tool_calls]: the same tool-scope score comes out 0.5 / 0.5 / 0.0.

So whether a score is available depends on which extra methods ran, and correlation (#97) then drops those subsets or errors. The required_kinds check below is already scoped this way. Suggest skipping details whose item_kind doesn't match scope when scope != "overall".

Agreed, done.

aleckhoury · 2026-10-30T16:54:10Z

+                files = file_manifest(workspace)
+                score = Score.model_validate(read_json(workspace / "response.json"))
+                if any(files.get(name) != digest for name, digest in inputs.items()):
+                    raise ValueError("coverage system modified its supplied evidence")


Diagnostics are kept only for a nonzero exit or a timeout. If the command exits 0 but response.json is missing or invalid, or the command modified its supplied evidence, the exception raised here lets the TemporaryDirectory cleanup delete the whole workspace: the request, authored observations, the bad response and the captured stdout/stderr.

Repro: an adapter that writes authored.txt, prints diag and writes {"value": "oops"} leaves the work root empty. stages.md lists invalid responses and modified inputs as stage failures alongside nonzero exits. Suggest sending these through the same path as lines 142-146: write the logs and rename the workspace to failed-*.

Agreed, done.

aleckhoury · 2026-10-30T16:54:10Z

+    if not executable.is_absolute():
+        raise ValueError("coverage command must start with an absolute executable path")


The docs and fixtures don't reflect this absolute-executable rule:

The analysis.yml example in benchmark-format.md uses command: [python, /path/to/coverage-adapter.py], and the nearby text (~131-134) only says "nonempty command argument list".

tests/fixtures/metric_benchmark/analysis.yml uses the same python form.

compute-coverage.md doesn't mention the rule; only eval-author-coverage.md does.

Copying the documented example passes load_analysis and sha-sign, then fails here. Suggest an absolute interpreter in the example (e.g. [/usr/bin/env, python3, ...], as in the smoke example) and stating the rule in benchmark-format.md and compute-coverage.md.

Agreed, done.

…tifacts Signed-off-by: mstaats <mstaats@nvidia.com>

Signed-off-by: mstaats <mstaats@nvidia.com>

mstaats-nvidia · 2026-10-31T12:12:48Z

Agreed, updated.

Signed-off-by: mstaats <mstaats@nvidia.com>

This was referenced Oct 20, 2026

feat(metric-benchmark): correlate matched subset scores #97

Merged

feat(metric-benchmark): orchestrate configured experiments through stage libraries #99

Open

docs(metric-benchmark): define benchmark architecture #93

Merged

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch from 7851fb5 to 18545f7 Compare October 21, 2026 20:22

mstaats-nvidia mentioned this pull request Oct 21, 2026

feat(metric-benchmark): define benchmark format, reader, and verification #94

Merged

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch from 18545f7 to d25167c Compare October 21, 2026 20:28

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch from d25167c to 8ff20b2 Compare October 21, 2026 21:08

This was referenced Oct 21, 2026

feat(metric-benchmark): specify and validate experiment results #105

Merged

feat(metric-benchmark): execute local Harbor case and mutant loops #106

Merged

feat(metric-benchmark): compute paired mutant scores for fixed subsets #95

Merged

mstaats-nvidia changed the title ~~feat(metric-benchmark): invoke coverage systems through fixed artifacts~~ feat(metric-benchmark): compute coverage through an Eval Author adapter Oct 21, 2026

mstaats-nvidia added this pull request to stack #107 October 21, 2026 21:10

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch from 8ff20b2 to 99d7cbc Compare October 23, 2026 14:33

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch from 99d7cbc to 77563fe Compare October 23, 2026 14:42

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch 2 times, most recently from 94337c1 to d03ec69 Compare October 26, 2026 13:43

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch 2 times, most recently from 16cb2fb to d0c0f82 Compare October 26, 2026 19:19

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch from d0c0f82 to 1da40aa Compare October 27, 2026 10:24

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch from 0dbdd96 to aca169c Compare October 29, 2026 14:27

mstaats-nvidia removed this pull request from stack #107 October 29, 2026 14:28

mstaats-nvidia added this pull request to stack #158 October 29, 2026 14:28

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch 2 times, most recently from 40223c4 to b8219a2 Compare October 29, 2026 15:26

mstaats-nvidia marked this pull request as ready for review October 29, 2026 15:33

mstaats-nvidia requested a review from a team as a code owner October 29, 2026 15:33

This was referenced Oct 29, 2026

docs(generation-benchmark): describe the generation experiment #160

Draft

feat(generation-benchmark): run generation experiments on metric coverage #161

Draft

asutermo reviewed Oct 30, 2026

View reviewed changes

asutermo approved these changes Oct 30, 2026

View reviewed changes

aleckhoury reviewed Oct 30, 2026

View reviewed changes

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch 4 times, most recently from 347fc29 to 44c3225 Compare October 31, 2026 11:36

Base automatically changed from feat/metric-benchmark-mutant-scores to main October 31, 2026 11:57

mstaats-nvidia added 3 commits October 31, 2026 12:57

feat(metric-benchmark): compute coverage with verified Eval Author ar…

44f8dc5

…tifacts Signed-off-by: mstaats <mstaats@nvidia.com>

Use case and analysis files for standalone coverage

b5c8de5

Signed-off-by: mstaats <mstaats@nvidia.com>

fix(metric-benchmark): use missing score reason in coverage

9ed018a

Signed-off-by: mstaats <mstaats@nvidia.com>

mstaats-nvidia force-pushed the feat/metric-benchmark-coverage branch from 44c3225 to 9ed018a Compare October 31, 2026 11:57

fix(metric-benchmark): scope audit coverage and preserve failed outputs

81e697e

Signed-off-by: mstaats <mstaats@nvidia.com>

fix(metric-benchmark): identify subsets in coverage cache errors

11313b3

Signed-off-by: mstaats <mstaats@nvidia.com>

mstaats-nvidia merged commit 0cd0425 into main Oct 31, 2026
20 checks passed

mstaats-nvidia deleted the feat/metric-benchmark-coverage branch October 31, 2026 12:31

Navigation Menu

Repository navigation

feat(metric-benchmark): compute coverage through an Eval Author adapter - #96

feat(metric-benchmark): compute coverage through an Eval Author adapter#96
mstaats-nvidia merged 5 commits into
mainfrom
feat/metric-benchmark-coverage

mstaats-nvidia commented Oct 20, 2026 •

edited

Loading

Uh oh!

asutermo Oct 30, 2026

Uh oh!

mstaats-nvidia Oct 31, 2026

Uh oh!

asutermo Oct 30, 2026

Uh oh!

mstaats-nvidia Oct 31, 2026

Uh oh!

aleckhoury left a comment

Uh oh!

aleckhoury Oct 30, 2026

Uh oh!

mstaats-nvidia Oct 31, 2026

Uh oh!

aleckhoury Oct 30, 2026

Uh oh!

mstaats-nvidia Oct 31, 2026

Uh oh!

aleckhoury Oct 30, 2026

Uh oh!

mstaats-nvidia Oct 31, 2026

Uh oh!

mstaats-nvidia commented Oct 31, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		return False


		def compute_eval_author(request_path: Path) -> Score:

		if not executable.is_absolute():
		raise ValueError("coverage command must start with an absolute executable path")

Conversation

mstaats-nvidia commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aleckhoury left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mstaats-nvidia commented Oct 31, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mstaats-nvidia commented Oct 20, 2026 •

edited

Loading