feat(metrics): consolidate gpt-oss/deepseek accuracy, dedup reporting, surface TPS by arekay-nv · Pull Request #400 · mlcommons/endpoints

arekay-nv · 2026-07-08T00:55:24Z

Summary

Reworks accuracy reporting into a uniform per-dataset list and aligns the run's report directory with the MLPerf endpoints submission layout (mlcommons/inference#2628).

Note: an earlier revision of this PR added a composite gpt-oss consolidation scorer; that was removed in favor of the per-dataset list below.

Per-dataset accuracy list

Report.accuracy and results.json's accuracy_scores are now an always-present list (empty for perf-only), one entry per accuracy dataset. Each entry: dataset_name, extractor, ground_truth_column, score, unit_samples, num_repeats, total_samples (= unit × repeats), duration_s (the dataset's issue-phase wall-clock), complete, and — for multi-subset scorers (DeepSeek-R1, BFCL) — a BFCL-shaped breakdown.
The single breakdown reader find_accuracy_breakdown iterates the list; compliance/checker.py + metrics/results_plots.py are unchanged behind it.
Rendered per-dataset in report.txt / results_summary.json: <name>: <score> (unit=U, repeats=R, total=T, duration=Ds).

Removed the composite gpt-oss path

Deleted the composite gptoss_120b_accuracy predefined dataset, GptOss120bAccuracyScorer, and _weighted_breakdown; the per-subset aime25::gptoss / gpqa::gptoss / livecodebench::gptoss datasets are the sole path. DeepSeek-R1 keeps its combined-subset breakdown as one list entry.

Report directory conforms to #2628

<report_dir>/
  config.yaml
  performance/results_summary.json   # the perf-metrics Report
  accuracy/accuracy_results.json     # the accuracy list
  audit/audit_<test_id>.json         # e.g. audit_output_caching_test.json (+ verify_<TEST>.txt)
  results.json                       # full run: config + perf + accuracy_scores + responses + errors
  report.txt · events.jsonl · metrics/ · sample_idx_map.json

scripts/publish_submission.py still copies the artifact trio flat into the MLPerf performance/run_1/ submission layout.

Fixes surfaced during validation

Coerce numpy scalar scores to native float: the base Scorer.score() returned np.mean(...) (numpy.float64), which crashed Report.to_json (msgspec) and results.json (json) serialization — caught on a live gpt-oss run.
Updated scripts/publish_submission.py and evaluation/bfcl_v4_multi_turn_cli.py to the list shape.

End-to-end validation (GB200 NVL72)

gpt-oss-120b (trtllm-serve): aime25 0.808 (8×) / gpqa 0.723 (5×) / livecodebench 0.849 (3×) @ 32768 OSL.
DeepSeek-R1-NVFP4 (trtllm-serve DEP4): overall 81.04 on the full 4388-sample eval (math500 95.8 / aime 83.3 / lcb 87.7 / mmlu_pro 77.3 / gpqa 66.7) — matches the MLPerf golden (81.36).

github-actions · 2026-07-08T00:55:35Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

gemini-code-assist

Code Review

This pull request introduces support for consolidating multiple accuracy datasets into a single combined score using a consolidation group tag, which is particularly useful for rolling up subsets like gpt-oss (e.g., AIME25, GPQA, and LiveCodeBench) into a sample-weighted overall score. It extracts shared accuracy breakdown helpers into a new module, refactors compliance and plotting utilities to use them, and updates the benchmark finalization process to attach consolidated accuracy breakdowns to the report. The feedback suggests a cleaner and safer approach to retrieving the dataset sample count during per-subset audit entry generation by avoiding fragile assert statements and direct internal attribute access in favor of the public num_samples() API.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Squashes the feat-consolidate-accuracy-reporting branch into one commit: - Consolidate gpt-oss/deepseek accuracy into per-dataset scoring; dedup reporting; surface TPS. - Add composite MLPerf gpt-oss-120b accuracy dataset + scorer: one results.json entry with a headline score plus a per-subset breakdown (aime25 / gpqa / livecodebench), mirroring the DeepSeek-R1 model. - Score each accuracy dataset into its own entry; drop the group dispatch/config field so examples emit one entry per dataset. - Delete the superseded GptOssAccuracyScorer orchestrator. - Enable per-dataset max-osl; cover the performance num_samples branch. - Bake the MLCommons-standard per-subset repeats (aime25 8 / gpqa 5 / livecodebench 3) into the gpt-oss-120b composite dataset: load() replicates each subset's rows in-memory by its factor, and the composite scorer divides the replication back out so the headline stays weighted by unique problem count. A config-level num_repeats multiplies on top. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… accuracy list shape Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…zation The base Scorer.score() returned np.mean(...) (numpy.float64). After the per-dataset-list refactor this raw score flows into Report.accuracy and results.json, and both msgspec (result_summary.json) and json.dump (results.json) reject numpy scalars — Report.to_json aborted the entire report write with "Encoding objects of type numpy.float64 is unsupported" (caught on a live gpt-oss-120b run). Fix at the source (return a native float) and defensively at the entry boundary in _score_accuracy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Two CI failures on PR #400, both from the accuracy dict→list refactor: - mypy: the hasattr-based numpy coercion in _score_accuracy tripped union-attr on the float|None score. numpy.float64 is a float subclass, so isinstance(score, float) + float(score) is the mypy-clean equivalent. - Tests: the integration accuracy pipeline test still indexed results["accuracy_scores"] as a dict keyed by dataset name; index the per-dataset list by dataset_name instead. (Integration tests are deselected from the default `pytest` run, so this was missed earlier.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Each accuracy entry now carries duration_s — the wall-clock of that dataset's issue phase, matched by phase name (== dataset_name; the inline-scored "performance" entry keys on the perf phase). Rendered in the report summary as `... total=T, duration=D.DDs)`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

nvzhihanj · 2026-07-10T19:34:28Z

+        ps = _load_publish_submission()
+        run = tmp_path / "acc_run"
+        run.mkdir()
+        (run / "results.json").write_text("{}", encoding="utf-8")


Leaving a note to change to accuracy/accuracy_results.json format

…folder After scoring, emit <report_dir>/accuracy/accuracy_results.json carrying the same {"accuracy_scores": [...]} list shape as results.json, so downstream accuracy consumers read one focused artifact. Written only when scoring produced entries (perf-only runs leave no accuracy/ folder). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

nvzhihanj · 2026-07-10T20:11:06Z

Thanks for the change. Can you help add some examples of the accuracy report (e.g. for gpt-oss) after the change?

Emit the perf-metrics report at <report_dir>/performance/result_summary.json (paired with the accuracy/ folder from a117e9f). Consumers updated: metrics/results_plots.py (load_run), scripts/publish_submission.py (reads it under performance/, still copies flat into the MLPerf performance/run_1 layout), and plot_results.py help text. results.json stays at the report_dir root — it is NOT performance-only: it carries accuracy_scores, per-sample responses, and errors, and compliance reads it as the accuracy artifact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rename the two summary artifacts to the names the endpoints submission checker expects: - performance/result_summary.json -> performance/results_summary.json - audit/audit_result.json -> audit/audit_<test_id>.json (e.g. audit_output_caching_test.json; auto-adapts per audit test) verify_<TEST>.txt (MLPerf validator marker) is unchanged. Consumers updated: results_plots.py, publish_submission.py (still copies flat into the MLPerf performance/run_1 layout), plot_results.py, compliance/result.py, commands/audit.py, the tests, and the audit/report docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Revert results_summary.json -> result_summary.json (benign; reduces churn vs the pre-existing name). The audit_<test_id>.json rename is kept. - result_summary.json is now purely performance: Report.to_json drops the accuracy field. Accuracy lives only in the dedicated accuracy report (accuracy/accuracy_results.json, plus results.json for compliance). The accuracy field stays on the Report struct so report.txt / the console summary still render it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

New AccuracyConfig.scale (default 1.0) multiplies the scalar accuracy score in _score_accuracy — e.g. scale: 100 reports a [0,1] pass@1 as a percentage, matching the 0-100 scale DeepSeek-R1 already uses. Applies only to the scalar score, not to any per-subset breakdown. The gpt-oss-120b example configs set scale: 100 so their aime25/gpqa/livecodebench scores render as percentages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…down fields Add an average accuracy across components and remove two now-redundant fields from the DeepSeek-R1 breakdown. - average_accuracy: plain mean of the per-dataset scalar scores (3 components for gpt-oss, 1 for DeepSeek-R1; equals the single score when there's one). The inline "performance" entry and non-numeric scores are excluded. Surfaced on Report.display (report.txt + console), accuracy/accuracy_results.json, and results.json; not in result_summary.json (perf-only by design). - DeepSeek breakdown no longer stores overall_accuracy (it's the entry's scalar score) or per_subset_status. build_breakdown drops the overall/**extra params; the block is now subset_scores + total_samples + complete. - Recognizer keys on subset_scores (shared by BFCL + DeepSeek) instead of overall_accuracy; new find_accuracy_entry exposes the owning entry so plotting sources the headline from score. BFCL breakdowns are unchanged (they build their own dicts and keep the compliance gate metrics). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Report output-token length per accuracy dataset, finalize-side (off the hot path), plus a timer for the tokenization work. - Each accuracy entry gains an `output_sequence_lengths` block with the SAME key/shape as the perf report (avg/min/max/median/std_dev/total/percentiles/ histogram). report.series_metric_dict reuses the aggregator's SeriesSampler + _series_to_metric_dict, so the block is identical to perf, computed in-process at finalize from the re-tokenized response texts (the server's completion_tokens isn't persisted — only the text is, in events.jsonl). Empty/failed completions are skipped, matching the perf-side OslTrigger. - Per-dataset `osl_tokenize_s` times just the tokenization; `osl_tokenization_s` (sum) is written to results.json + accuracy/accuracy_results.json and shown in report.txt. - The tokenizer is loaded once, only when a real accuracy dataset exists (a perf-only run pays nothing); a missing/unloadable tokenizer disables OSL rather than failing scoring. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

+    def test_osl_attached_with_tokenizer(self, tmp_path, monkeypatch):
+        """With a tokenizer, each accuracy entry gets an output_sequence_lengths
+        block (same shape as the perf report) from the phase's completions."""
+        import inference_endpoint.commands.benchmark.execute as execute_mod


+        assert entry["osl_tokenize_s"] >= 0.0
+
+    def test_osl_skipped_for_performance_entry(self, tmp_path, monkeypatch):
+        import inference_endpoint.commands.benchmark.execute as execute_mod


+
+    def test_osl_dropped_when_get_outputs_raises(self, tmp_path, monkeypatch):
+        """A read/tokenize failure only drops OSL — scoring still succeeds."""
+        import inference_endpoint.commands.benchmark.execute as execute_mod


+        assert _load_osl_tokenizer(None) is None
+
+    def test_load_failure_returns_none(self, monkeypatch):
+        import inference_endpoint.commands.benchmark.execute as execute_mod


+        assert _load_osl_tokenizer("bad/model") is None
+
+    def test_success_raises_model_max_length(self, monkeypatch):
+        import inference_endpoint.commands.benchmark.execute as execute_mod


arekay-nv requested a review from a team July 8, 2026 00:55

github-actions Bot requested a review from nvzhihanj July 8, 2026 00:55

gemini-code-assist Bot reviewed Jul 8, 2026

View reviewed changes

Comment thread src/inference_endpoint/commands/benchmark/execute.py Outdated

arekay-nv force-pushed the arekay/feat-consolidate-accuracy-reporting branch from 213b8c5 to 1ecb9e8 Compare July 8, 2026 01:01

nvzhihanj reviewed Jul 8, 2026

View reviewed changes

Comment thread src/inference_endpoint/commands/benchmark/execute.py Outdated

arekay-nv changed the title ~~feat(metrics): consolidate gpt-oss/deepseek accuracy, dedup reporting, surface TPS~~ DRAFT feat(metrics): consolidate gpt-oss/deepseek accuracy, dedup reporting, surface TPS Jul 8, 2026

nvzhihanj reviewed Jul 8, 2026

View reviewed changes

Comment thread src/inference_endpoint/evaluation/scoring.py Outdated

nvzhihanj reviewed Jul 8, 2026

View reviewed changes

Comment thread src/inference_endpoint/evaluation/scoring.py Outdated

arekay-nv requested a review from nvzhihanj July 9, 2026 01:37

arekay-nv changed the title ~~DRAFT feat(metrics): consolidate gpt-oss/deepseek accuracy, dedup reporting, surface TPS~~ feat(metrics): consolidate gpt-oss/deepseek accuracy, dedup reporting, surface TPS Jul 9, 2026

arekay-nv force-pushed the arekay/feat-consolidate-accuracy-reporting branch from 8eb098f to d29c111 Compare July 9, 2026 22:33

arekay-nv and others added 8 commits July 9, 2026 23:05

feat(accuracy): per-dataset accuracy list on Report and results.json

7da0464

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

refactor(accuracy): find_accuracy_breakdown reads the per-dataset list

a636e85

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

refactor(accuracy): remove gpt-oss-120b composite dataset and scorer

70bd76b

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(accuracy): update publish_submission + bfcl multi-turn CLI to the…

940b3a9

… accuracy list shape Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Merge branch 'main' into arekay/feat-consolidate-accuracy-reporting

b60cff1

arekay-nv mentioned this pull request Jul 10, 2026

MLPerf v6.1 LoadGen <> Endpoints++ submission checker issues mlcommons/inference#2628

Open

nvzhihanj reviewed Jul 10, 2026

View reviewed changes

arekay-nv and others added 5 commits July 10, 2026 15:34

github-code-quality Bot found potential problems Jul 11, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(metrics): consolidate gpt-oss/deepseek accuracy, dedup reporting, surface TPS#400

feat(metrics): consolidate gpt-oss/deepseek accuracy, dedup reporting, surface TPS#400
arekay-nv wants to merge 16 commits into
mainfrom
arekay/feat-consolidate-accuracy-reporting

arekay-nv commented Jul 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nvzhihanj Jul 10, 2026

Uh oh!

nvzhihanj commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

arekay-nv commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Per-dataset accuracy list

Removed the composite gpt-oss path

Report directory conforms to #2628

Fixes surfaced during validation

End-to-end validation (GB200 NVL72)

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nvzhihanj Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

nvzhihanj commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

arekay-nv commented Jul 8, 2026 •

edited

Loading