WIP: PD-DFlash Tasks 9–10 — BM3 priority band + BM4/BM5 overlap/e2e - #161
Merged
Conversation
added 9 commits
August 14, 2026 22:22
Record exact stored FP4/FP8 expert payload bytes through the route-ahead observer so the measure-first gate can report wasted_prefetch_bytes as bytes, not an expert count. RouteAheadStats.observe_layer gains an optional expert_nbytes map and RouteAheadStepSummary/as_dict expose predicted/kept/ wasted byte fields (None when unavailable -- never a fabricated average). ExpertPrefetcher.expert_nbytes_map is populated at registration time in model_offload from live params (before offload placeholders erase shape); the executor seam forwards it None-safely (isinstance-dict guard keeps mocks and resident runs at None). Strictly additive and off by default.
Opt-in RTX PRO 6000 B0-B3 route-ahead serving runner. The CLI module is import-safe (torch/moe_infinity imported only inside the GPU path), emits one JSON row per (model,baseline,B,concurrency,repeat) matching REQUIRED_METRICS, validates the device is an RTX PRO 6000 (12,0), refuses resident B0/B1/B2, and blocks B2 as BLOCKED_UNTIL_2D_SCHEDULER until the 2-D scheduler lands. Wraps draft/router/issue/verify/H2D in the frozen NVTX ranges the BM4 parser keys on. GPU test is gated on MOE_DFLASH_SERVING_GPU (1 skipped, side-effect free collection); CPU contract test locks the pure matrix/schema/writer logic.
Add BM1 summarise_row (pass iff t_router < t_verify, ratio + raw terms retained) and a CPU-safe aggregation CLI: group raw rows into §8 matrices keyed by (model,block,concurrency), permit blocked B2 via --allow-blocked, attach BM1, and emit result_matrix.json/CSV/Markdown for validate_result_matrix. Refactors the per-baseline metric check out of validate_result_matrix (behavior-preserving).
run_phase_a.sh drives the full §8 matrix for both required MoE targets on one RTX PRO 6000 with the documented env (HF_HOME, MOE_ENABLE_SM120, device-memory- ratio<0.9 to force offload) and aggregates into result_matrix.json. All inputs are documented env vars in the script header.
Task 7 of the PD-DFlash serving plan (design §10 BM2). Adds an import-safe issuance micro-benchmark that times enqueuing a saturated E_l×L block of offloaded expert tensors three ways -- python-per-expert, batched-pybind, and (reserved) cpp-internal -- reporting p50/p90/p99 microseconds and the ship gate. - benchmarks/dflash/bench_prefetch_issuance.py: pure bm2_decision / percentiles_us / build_bm2_report at module scope (torch + moe_infinity lazily imported in the GPU runner), plus the CLI runner that synthesises the real saturated tensor-id list from a loaded target and times warmup=20 / iterations=200 with perf_counter_ns. Unavailable candidate modes are reported null, never zero. - tests/python/dflash/test_prefetch_perf_reports.py: CPU-only decision-rule and report-schema tests (the plan's four exact assertions plus boundary cases). BM2 alone gates the batched-issuance C++ hop; committed independently of it.
Task 8 of the PD-DFlash serving plan (candidate hop 1), retained because BM2 passed on offloaded gpt-oss-20b: python-per-expert issuance p50=1132 µs vs batched-pybind p50=101 µs over a 768-tensor saturated block (~11× fewer pybind crossings), so per-expert issuance is exposed and the batched call hides it. - core/prefetch: add ArcherPrefetchHandle::EnqueuePrefetchTensors(tensor_ids, priority=1), which constructs and enqueues one Task per tensor entirely in C++, preserving input order and node default devices (mirrors EnqueuePrefetch). - core/python: bind prefetch_tensors -> EnqueuePrefetchTensors and retire the old no-op PrefetchTensors(request_id, buffer) binding; enqueue_prefetch retained. - expert_prefetcher: prefetch_experts_list issues one batched prefetch_tensors call when the engine exposes it, else the byte-for-byte per-expert fallback; empty input is a no-op. - tests: batch/fallback/empty coverage in test_speculative_prefetch.py; wire assertions made mechanism-agnostic (batched call carries the same ordered ids); opt-in native GPU smoke test (single module-scoped offload load).
Task 9/10 measurement harnesses (design §10), no C++ shipped: * bench_prefetch_priority.py -- BM3 three-way route-ahead prefetch priority ablation (background/route-ahead/on-demand) over median exposed-fetch seconds and tokens/s, with a pure bm3_decision ship gate: ship the dedicated route-ahead band iff it lowers exposed fetch vs background, preserves tokens/s, and on-demand stays fastest (no priority inversion). * parse_overlap.py -- BM4 expert-H2D / compute overlap from an nsys trace; pure interval arithmetic apportions memcpy bytes by overlapped-duration fraction against the union of the draft/router/verify NVTX ranges. * run_phase_c.sh -- one-command Phase-C launcher for the full BM3/BM4/BM5 matrix on both required targets. * test_prefetch_perf_reports.py -- CPU-only BM3 + BM4 decision-rule tests.
…icts) Task 10 final-gate aggregation (design §10): * summarise_bm5_equivalence -- switching Python per-expert issue to shipped C++ batched issue may only move tokens/s; acceptance, route-ahead coverage, and wasted bytes must match within tolerance. * cpp_hop_verdicts -- keep/remove each benchmark-gated C++ hop from its paired BM: batched issuance needs BM2 ship_batched, the priority band needs BM3 ship_priority_band; a missing BM removes the hop (no C++ ships without its BM). * test_pd_dflash_report.py -- CPU-only BM5 equivalence + hop-verdict tests.
drunkcoding
marked this pull request as ready for review
August 15, 2026 18:19
drunkcoding
added a commit
that referenced
this pull request
Aug 17, 2026
…s+instrumentation kept) (#169) * perf(prefetch): recover route-ahead priority band (BM3 blocked, NOT shipped) Re-implements the Task 9 route-ahead priority-band candidate that PR #161 implemented+built+validated locally then reverted (Python-only shipped) because its DFlash draft was absent. No original add/revert commit exists (verified via git log/pickaxe/fsck/stash) -- the revert was a working-tree discard -- so this restores the candidate per the plan spec to make it reviewable. C++ (core/prefetch, core/python): - task_scheduler.h: name the bands kOnDemandPriority=0, kRouteAheadPriority=1, kBackgroundPrefetchPriority=2 (NUM_PRIORITY stays 20). - EnqueuePrefetch (ordinary/background prefetch) now enqueues at background (2); EnqueuePrefetchTensors + the prefetch_tensors binding default to the dedicated route-ahead band (1). Python (expert_prefetcher.py): - add ExpertPrefetcher.route_ahead_priority (the BM3 knob, mirrors the native constants). Explicit route-ahead prefetch_experts_list issues at that band; legacy speculative_prefetch issues at background. Tests: - native GPU smoke asserts all three bands + the knob issue without raising; CPU mock tests assert explicit=route-ahead / legacy=background band mapping. BM3 was NOT run to a ship/no-ship verdict -- blocked by two issues on dev that are independent of this candidate (a 1-line EnqueuePrefetch priority + a Python knob; the native smoke swept all three bands without hanging): (A) bench_prefetch_priority._reset_cache() calls the terminal clean_up_resources() between arms, deadlocking the offload engine; the next forward hangs in fetch_tensors (model_offload.py:1670). No JSON. (B) exposed_fetch_seconds is uninstrumented, so _exposed_fetch_seconds() returns 0.0 for every arm and bm3_decision.exposed_fetch_improved can never be true -- ship_priority_band is structurally unmeasurable on dev. Per the plan's rule (no C++ ships without its paired benchmark proving the exposed window is closed) the band is NOT shipped. No false win either way. Validated (not a ship claim): SM120 build clean (0 errors); native priority smoke 5 passed on offloaded gpt-oss-20b; 60 CPU decision/mock tests passed; gpt-oss offload no-regression 9 passed. * fix(bench): unblock BM3 priority ablation (Blocker A deadlock + Blocker B exposed-fetch) Both are harness/instrumentation fixes independent of the route-ahead priority-band candidate under test; they are kept regardless of the BM3 ship/no-ship verdict. Blocker A (harness deadlock): bench_prefetch_priority._reset_cache() called the TERMINAL ArcherPrefetchHandle::CleanUpResources() between ablation arms, which reset kTaskPool/topology/memory-pool globals and set has_cleaned_up_resources_, so the next forward's fetch_tensors dereferenced a null kTaskPool and hung (model_offload.py:1670; no JSON emitted). - Add ArcherPrefetchHandle::ResetCache() (bound `reset_cache`): a non-terminal reset that drops pending prefetch work via kTaskPool->ClearQueue() while keeping the task pool, its worker threads, and all globals alive (has_cleaned_up_resources_ untouched). ClearQueue scans priority 1..NUM_PRIORITY, leaving the on-demand band (0) intact. - _reset_cache() now prefers reset_cache, falls back to replace_cache_candidates([]) on an un-rebuilt engine, and zeroes the per-arm exposed-fetch accumulator. clean_up_resources() is no longer called between arms. - native GPU smoke asserts reset_cache is non-terminal (engine still services a prefetch afterward). Blocker B (exposed-fetch instrumentation): _exposed_fetch_seconds() returned 0.0 for every arm because nothing timed the synchronous on-demand fetch, so bm3_decision.exposed_fetch_improved could never be true. - OffloadEngine._fetch_tensors_timed() times the pre-forward hook's fetch_tensors call (the exposed-fetch window: compute stalls there until an un-prefetched expert is resident) and accumulates _exposed_fetch_seconds; surfaced via get_exposed_fetch_seconds()/reset_exposed_fetch_seconds(). Read-only: the fetch behaviour is unchanged, only timed. - _measure_once() reads a per-run before/after delta so each arm records only its own on-demand stall. - CPU TDD covers the engine accumulator surface and the harness probe. * revert(prefetch): NO-SHIP the route-ahead priority band (BM3 verdict) BM3 now runs to a verdict (Blockers A+B fixed in the previous commit). The dedicated route-ahead band does NOT ship: across two independent runs on offloaded gpt-oss-20b + the DFlash draft it never satisfies all three ship conditions at once, and the exposed-fetch effect is not robust. arm (band) run1 (6 reps, seed 1408) run2 (10 reps, seed 2024) background (2) exposed 14.216ms 87.27 tok/s exposed 15.327ms 83.58 tok/s route-ahead (1) exposed 13.662ms 86.51 tok/s exposed 15.434ms 93.49 tok/s on-demand (0) exposed 13.540ms 88.92 tok/s exposed 13.758ms 91.53 tok/s run1: exposed_fetch_improved=True, throughput_preserved=False -> SHIP=False run2: exposed_fetch_improved=False, throughput_preserved=True -> SHIP=False The two runs fail on DIFFERENT gates and the exposed-fetch sign flips (route-ahead beats background in run1, loses in run2), i.e. the signal is within measurement noise. on-demand stays fastest in both (no inversion). Per the honest gate (no C++ ships without its paired benchmark proving the exposed window closes with no throughput cost), revert the one behavioural line: EnqueuePrefetch (ordinary/background prefetch) returns to band 1 (pre-candidate), so route-ahead is no longer serviced ahead of background. Default production behaviour matches pre-candidate (all prefetch at band 1; on-demand at 0). KEPT (deliberately, not part of the shipped band): the named band constants, the ExpertPrefetcher.route_ahead_priority knob, and the BM3 harness -- these are the (now inert-by-default) measurement scaffolding so BM3 stays reproducible. Also kept: the Blocker A/B harness + instrumentation fixes. --------- Co-authored-by: drunkcoding <leyang.xue@ed.ac.uk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP / DRAFT — do not merge. Stacked on #160 (
feat/pd-dflash-task7-batched-prefetch, Tasks 7/8). Review only after #160.Implements Tasks 9 & 10 of the PD-DFlash serving plan (Phase C). The plan's hard rule — no C++ ships without its paired benchmark passing — is honored: this PR ships only benchmark/parser/report/test code (pure Python + one shell launcher). No C++ change is included, because the paired benchmarks could not pass their gate on the available hardware (see below).
Task 9 — BM3 route-ahead priority band
benchmarks/dflash/bench_prefetch_priority.py: BM3 three-way priority ablation (background / route-ahead / on-demand) over median exposed-fetch seconds and tokens/s, plus a purebm3_decisionship gate — ship the dedicated route-ahead band iff it (1) lowers exposed fetch vs background, (2) does not reduce tokens/s, and (3) keeps on-demand the fastest class (no priority inversion). CPU decision tests intest_prefetch_perf_reports.py.ship_priority_band=trueon both required targets, and neitherQwen/Qwen3-Coder-30B-A3Bnoropenai/gpt-oss-20bhas its DFlash draft present in the cache — so BM3 cannot be run on either required (model, draft) pair. Gate NOT met → priority-band NOT justified → not shipped (no false win).Task 10 — BM4 overlap + BM5 e2e
benchmarks/dflash/parse_overlap.py: BM4 expert-H2D / compute overlap from an nsys trace — pure interval arithmetic apportions memcpy bytes by overlapped-duration fraction against the union of the draft/router/verify NVTX ranges (nsys stats --report cuda_gpu_trace,nvtx_pushpop_trace).benchmarks/dflash/report.py:summarise_bm5_equivalence(C++ issue may only move tokens/s; acceptance/coverage/waste must match) andcpp_hop_verdicts(keep/remove each hop from its paired BM). CPU tests intest_pd_dflash_report.py.benchmarks/dflash/run_phase_c.sh: one-command launcher for the full BM3/BM4/BM5 matrix on a fully-provisioned box (both targets + drafts + offload dirs).Why the hardware gates could not run here
Required checkpoints/offload absent on this box:
Qwen/Qwen3-Coder-30B-A3B,z-lab/Qwen3-Coder-30B-A3B-DFlash,z-lab/gpt-oss-20b-DFlash, and both offload dirs.openai/gpt-oss-20bis present (used to validate the reverted candidate builds/loads/no-regresses). Full BM3/BM4/BM5 numbers require runningbenchmarks/dflash/run_phase_c.shon a provisioned box.Verification
tests/test_gpt_oss_offload_topology.py,tests/python/unit/test_gpt_oss_mxfp4_dispatch.py): 9 passed (validated during the candidate build; shipped state removes all C++, returning to WIP: PD-DFlash Task 7 — BM2 + batched prefetch #160's already-validated base).