Skip to content

perf: add validated B200 vLLM 0.31 operation data - #593

Closed
simone-chen wants to merge 11 commits into
mainfrom
simonec/b200-vllm031-data
Closed

simone-chen wants to merge 11 commits into
mainfrom
simonec/b200-vllm031-data

Conversation

@simone-chen

@simone-chen simone-chen commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Measurements

  • Add 141,773 fresh B200 / vLLM 0.31.0 operation rows:113,557 GEMM,636 MLA BMM,3,256 paired quantization,10,332 GDN and13,992 MSA context measurements. Keep this PR draft and main-targeted; no defaults are promoted.
  • GEMM includes37,518 BF16,36,408 block FP8,2,113 tensor FP8 and37,518 NVFP4 rows. Preserve1,110 explained historical block-FP8 gaps. The35,405 separately qualified FP8 reuse keys and deferred GLM shape expansion are not inserted as new measurements.
  • BMM covers300 pre-BMM and336 post-BMM cases. Quantization covers1,628 paired callbacks, including four legitimate clipped-zero compute-scale estimates with positive static companions.
  • GDN covers84 callbacks /5,166 paired points with zero failures; all four historical failed callbacks recovered. Fix the0.31 logical context alias to find its measured FlashInfer recurrence; preserve packed-decode normalization and existing guards.
  • MSA continuation76851457 adds7,478 new cases with zero new failures and preserves6,514 measurements. Original17,568-case scope =13,992 measured +1,400 explained failures +2,176 source-blocked unattempted cases. Preserve per-case source groups184/6,330/7,478; aggregate finalizer metadata does not relabel seed measurements.
  • Preserve immutable image/source/runtime pins, checkpoint and case inventories, raw hashes and failure evidence. The MSA Parquet is unchanged; its sidecar is marked partial. Only affected backend facts and reuse-manifest counters change; tiers and hardware/framework membership stay unchanged.

Validation

  • Check exact identities, readable Parquets, provenance, duplicates and valid latencies. Preserve all prior accepted data byte-for-byte when adding each scope. Existing GEMM SOL floors affect39 BF16 rows; raw timings remain unchanged.
  • Python/native consumers pass with shared-layer reuse disabled:113,557 GEMM,636 BMM,3,256 quantization identities,15,036 GDN queries and13,992 MSA queries per interface. Both interfaces share the compiled engine; retired legacy APIs are not claimed as independent validation.
  • Retain bounded native eager/graph numerical probes, timing comparisons and raw review reports under benchmarks/evidence/vllm031-b200-refresh. Later compute-scale component probes do not replace missing components from the original full run.
  • All seven collector-data rules and2,452 backend-fact slices pass. Latest MSA addition passes40 focused data/MSA tests, Ruff, exact consumer checks and independent review. Earlier GDN routing validation passed22 Rust tests,21 Python tests and389 parity tests with unchanged goldens.
  • Auto collection and personal numerical checks used different host drivers. Sustained clock lock, full-grid arithmetic and end-to-end prediction accuracy are not established.
  • Signed/DCO head 0794dc832f982193397174be2f55577096b216a6 is verified by GitHub. Full CI is requested for this exact head; its result remains pending.

Remaining work

  • Other B200/vLLM campaign scopes remain open. Encoder measurements are not included in this commit. No merge or default promotion is authorized.
  • Existing shared-tier resolution can use new data for gap filling under its current policy. This PR grants no new cross-version or cross-framework performance-equivalence claim.

Signed-off-by: Simone Chen <simonec@nvidia.com>
@coderabbitai

coderabbitai Bot commented Oct 26, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@simone-chen

Copy link
Copy Markdown
Contributor Author

/ok to test 06ccfdc

Signed-off-by: Simone Chen <simonec@nvidia.com>
@simone-chen

Copy link
Copy Markdown
Contributor Author

/ok to test 7228466

Signed-off-by: Simone Chen <simonec@nvidia.com>
@simone-chen simone-chen changed the title feat: add B200 vLLM 0.31 pre-BMM measurements perf: add B200 vLLM 0.31 pre/post-BMM measurements Oct 26, 2026
@simone-chen

Copy link
Copy Markdown
Contributor Author

/ok to test ddc5eb7

Signed-off-by: Simone Chen <simonec@nvidia.com>
@simone-chen simone-chen changed the title perf: add B200 vLLM 0.31 pre/post-BMM measurements perf: add B200 vLLM 0.31 GEMM and MLA BMM data Oct 26, 2026
Signed-off-by: Simone Chen <simonec@nvidia.com>
Signed-off-by: Simone Chen <simonec@nvidia.com>
@simone-chen

Copy link
Copy Markdown
Contributor Author

/ok to test ffd7c2c

Signed-off-by: Simone Chen <simonec@nvidia.com>
@simone-chen

Copy link
Copy Markdown
Contributor Author

/ok to test 65e61e4

@simone-chen simone-chen changed the title perf: add B200 vLLM 0.31 GEMM and MLA BMM data perf: add B200 vLLM 0.31 GEMM, BMM and quantization data Oct 26, 2026
Signed-off-by: Simone Chen <simonec@nvidia.com>
@simone-chen

Copy link
Copy Markdown
Contributor Author

/ok to test 4dbc51c

Signed-off-by: Simone Chen <simonec@nvidia.com>
@simone-chen simone-chen changed the title perf: add B200 vLLM 0.31 GEMM, BMM and quantization data perf: add B200 vLLM 0.31 GEMM, GDN, BMM and quantization data Oct 26, 2026
@simone-chen

Copy link
Copy Markdown
Contributor Author

/ok to test 7530c32

Signed-off-by: Simone Chen <simonec@nvidia.com>
@simone-chen

Copy link
Copy Markdown
Contributor Author

/ok to test 8ab494f

Signed-off-by: Simone Chen <simonec@nvidia.com>
@simone-chen simone-chen changed the title perf: add B200 vLLM 0.31 GEMM, GDN, BMM and quantization data perf: add validated B200 vLLM 0.31 operation data Oct 26, 2026
@simone-chen

Copy link
Copy Markdown
Contributor Author

/ok to test 0794dc8

@simone-chen

Copy link
Copy Markdown
Contributor Author

Closing our data publication PR because the user selected Tianhao's #517 as the data update path. Retain only collector review PRs #604, #605, and #591. This does not merge these measurements or claim #517 covers vLLM 0.31. The branch, artifacts, provenance, and history remain available; collection remains stopped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant