Repository navigation
feat(metric-benchmark): correlate matched subset scores - #97
Merged
Merged
Conversation
This was referenced Oct 20, 2026
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 21, 2026 20:22
f311eb8 to
dff36b6
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
2 times, most recently
from
October 21, 2026 21:08
8082251 to
913bc3b
Compare
This was referenced Oct 21, 2026
mstaats-nvidia
changed the title
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 23, 2026 14:33
913bc3b to
5dde419
Compare
mstaats-nvidia
added a commit
that referenced
this pull request
Oct 23, 2026
Defines an experimental benchmark for evaluating how well our coverage metric predicts evaluation effectiveness through mutation testing. The concise [root README](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/README.md) describes the project, its work-in-progress status, and the expectation that case examples will grow, change, or be dropped as we build in plain sight. Prep is external to this repository; runs consume prepared case examples, fixed subset manifests, and a YAML benchmark definition. The [stage descriptions](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/docs/stages.md) focus on four executable stages: run mutants × eval tasks, compute mutant scores, compute coverage, and correlate the score pairs. The first execution script will be a basic local Harbor loop; NeMo Helix support is planned afterward. The [architecture and both SVG figures](https://github.com/NVIDIA-NeMo/eval-author/blob/docs/metric-benchmark-design/labs/metric-benchmark/docs/architecture.md) explain the inputs, component boundaries, shared results directory, and statistical comparison. This is the documentation-only base of the stack. Implementation PRs #94 → #105 → #106 → #95 → #96 → #97 → #99 inherit the overview and design unchanged. Their detailed CLI guide lives separately in `docs/tools.md`. Validation: the five files in this PR are Markdown/SVG only. Local links and anchors, Markdown fences, headers, SVG XML, and whitespace checks pass; the updated figure was rendered and visually inspected. Restacking preserves all source and test files byte-for-byte. No live evaluations or model calls were made.
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 23, 2026 14:42
5dde419 to
c3d31df
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 26, 2026 12:32
c3d31df to
2505187
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 26, 2026 13:43
2505187 to
a050306
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
2 times, most recently
from
October 26, 2026 19:19
5e8016d to
e5e1a94
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 27, 2026 10:24
e5e1a94 to
2d77adb
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
2 times, most recently
from
October 28, 2026 13:01
75f2728 to
47dc1fd
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
2 times, most recently
from
October 29, 2026 12:58
0eb4e76 to
11563e1
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 29, 2026 14:27
11563e1 to
996d706
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 29, 2026 14:40
996d706 to
e8ecaf7
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 29, 2026 15:26
e8ecaf7 to
597b8c9
Compare
mstaats-nvidia
marked this pull request as ready for review
October 29, 2026 15:33
asutermo
approved these changes
Oct 30, 2026
aleckhoury
approved these changes
Oct 30, 2026
aleckhoury
left a comment
aleckhoury
left a comment
Collaborator
There was a problem hiding this comment.
Correlation looks right.
- Compared
spearmanwithscipy.stats.spearmanrover 20,000 random cases (n = 0..12, both continuous and heavily tied inputs). Maximum absolute difference was 3.3e-16, and the two agreed on every case where no coefficient can be computed. - Matching on exact subset ID plus an identical
Contextis a safe join, because both inputs buildContextwithscoring.context(). missing_policyvalues and the default match config.py and the CLI.- The caveat that n = 2 always gives ±1 is documented.
At this head: 163 passed, 1 skipped.
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
2 times, most recently
from
October 31, 2026 10:09
8e4ee0f to
64f897f
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 31, 2026 11:33
64f897f to
c20650e
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 31, 2026 11:36
c20650e to
19450eb
Compare
mstaats-nvidia
force-pushed
the
feat/metric-benchmark-correlation
branch
from
October 31, 2026 11:57
19450eb to
94daac9
Compare
Signed-off-by: mstaats <mstaats@nvidia.com>
Signed-off-by: mstaats <mstaats@nvidia.com>
Move main and its argument parsing and JSON I/O imports into correlate.py. Keep the correlation library focused on models and computation. Signed-off-by: mstaats <mstaats@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Correlate coverage and mutant scores by exact subset ID and matching experiment context, using Spearman ranks with average ties. Preserve paired observations, input digests, and explicit exclusions. Constant ranks and fewer than two usable pairs produce an unavailable coefficient with
missing_score_reason; negative mutant scores remain valid.correlate.pyownsmain(), argument parsing, artifact loading, and output writing.metric_benchmark.correlationcontains the reusable models and calculation. The CLI and coordinator call the samecorrelate()function; correlation needs only the two score artifacts and the missing-pair policy.Usage and library API:
labs/metric-benchmark/docs/correlate.md. Based on main after #96; #99 consumes this library downstream.Validation at
a864afa: 3,441 passed, 36 skipped in the full local suite. The skips cover unavailable or opt-in runtimes/platforms (Gym, Node/browser rendering, Docker, pinned SkillEvaluator, Ratchet, Windows, and live Codex). All five focused correlation tests pass, including byte-identical CLI/library output. Ruff, formatting, type checks for Python 3.12/3.13, copyright, licenses, whitespace, and DCO checks pass. The CLI move preserves the function bodies and command help description. Hosted CI passed on this head, including the Python 3.12/3.13 full suites and native compatibility checks.