Skip to content

feat(vllm): collect native prefill metrics for SD self-benchmarks - #16472

Draft
tedzhouhk wants to merge 1 commit into
hzhou-codex/sd-selfbench-eagle3from
hzhou-codex/sd-selfbench-native-prefill
Draft

tedzhouhk wants to merge 1 commit into
hzhou-codex/sd-selfbench-eagle3from
hzhou-codex/sd-selfbench-native-prefill

Conversation

@tedzhouhk

Copy link
Copy Markdown
Contributor

Summary

Depends on #16348 (hzhou-codex/sd-selfbench-eagle3). That PR provides native async EAGLE3 decode self-benchmarking; this stacked follow-up completes the native prefill path and artifact contract without duplicating the decode implementation.

  • Support native CUDA EAGLE3 measurements in --benchmark-mode prefill and both phases of agg. Match timing and workload snapshots by native iteration ID and keep the point alive until its timing is received; never substitute scheduler wall time.
  • Reuse existing vLLM CUDA events and rejection-corrected snapshots. No new CUDA events, synchronization, D2H, or per-iteration disk I/O.
  • Apply effective generation sampling to prefill and its preparation; warm every exact SD coordinate, including captured shapes. Keep native prefill variance unchanged.
  • Export optional immutable target/draft execution identity with explicit resolution sources or unresolved fields. Reuse the observed KV capacity including target/draft groups; do not estimate or subtract draft memory again.
  • Validate native prefill single_step as one exact retained observation (shape, timing, queue, native ID), preserve provenance/capacity, and reject CPU/CUDA relabeling. Existing AR artifacts and explicitly phase-specific legacy SD artifacts remain readable.
  • Extend the small-model GPU oracle to prefill-only and aggregate collection, including real cached prefixes, exact native producer observations, post-benchmark callback restoration and subsequent serving.

Scope

Sequential EAGLE3, tensor parallelism, DP1/PP1/CP1, no EP/offload/KV transfer, native-FPM-capable asynchronous vLLM. No additional speculative algorithms are enabled. Real-KV cached-prefill collection requires DYN_BENCH_PREFILL_REAL_SEED=1; synthetic prefixes remain labelled and are not qualified for strict AISimulate import.

The execution binding matches the profile contract in ai-dynamo/aisimulate#476 and ai-dynamo/aisimulate#477. A resolved identity is not an assertion of coverage, import readiness, or serving accuracy. Local/unknown checkpoint identities remain explicitly unresolved; no network lookup or fabricated checkpoint revision is introduced. AISimulate's initial K=1–5 consumer restriction does not silently narrow the collector's existing positive-depth support.

Validation

  • Scheduler CPU regression: 366 passed; 27 known tests requiring newer cache APIs were deselected in the retained old local runtime. Independent targeted SD rerun: 38 passed.
  • Artifact/worker-factory CPU regression: 257 passed, including native-P corruption checks, resolved/unresolved identity, preserved usable KV, legacy AR/phase-specific artifacts, and positive depth outside AISimulate's initial import range.
  • Real local GPU functional validation: prefill-only + aggregate 2/2 passed; existing eager/CUDA-graph decode matrix (AL 1, 2.78, 4) 6/6 passed. Eight starts retain 426 measured native CUDA observations: 6 P and 420 D. Cached P was not removed or weakened.
  • P/aggregate cases verified three exact zero/cached-prefix shapes, configured sampling, native timing/snapshot equality, production merge, observed usable capacity before/after, observer/factory restoration, and normal serving afterward.
  • Scoped mypy passes for both changed production modules; pre-commit passes for all six changed files. Independent source-grounded review found no outstanding actionable defects.

Qualification boundary

GPU tests used cached Qwen3-0.6B + EAGLE3 on one RTX 6000 Ada, with a retained vLLM 0.24 runtime and the previously validated native-FPM lifecycle backport from b3563fc65ae0f5359802593d78e7ea097e1fed31. Prefill cases additionally used a test-only adapter for the older two-element cache-lookup return: it reads the same cached-token element that the newer three-element API exposes, without changing the coordinator or production code. Decode cases did not need this adapter. Both adaptations remain outside this PR.

Peak measured VRAM stays within the existing 3.6 GiB test marker; all test GPU processes were reaped. No checkpoint/image downloads or cluster allocations were performed. This is functional qualification, not current-nightly/container qualification or MiniMax/B200 performance evidence. The PR remains draft pending target-runtime CI and its dependency.

Documentation: fern check passes after generating standard ignored API-reference pages. fern docs broken-links reports one pre-existing link in the unchanged DeepSeek-V4-Pro-0813 recipe; the changed page has no reported broken links. No generated API pages are included in this diff.

Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
@github-actions github-actions Bot added feat documentation Improvements or additions to documentation backend::vllm Relates to the vllm backend labels Oct 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@devin-ai-integration

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 38660904985 (attempt 1) on c468ee3fb8

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 38660904985. Updated on every full-CI run of this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend documentation Improvements or additions to documentation feat size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant