Skip to content

5080deploy: run the PLE graph fill on its own thread, submitted before the launch (fixes #65) - #67

Open
gdevenyi wants to merge 2 commits into
5080/26-review-fixesfrom
5080/27-ple-fill-before-launch
Open

gdevenyi wants to merge 2 commits into
5080/26-review-fixesfrom
5080/27-ple-fill-before-launch

Conversation

@gdevenyi

Copy link
Copy Markdown
Owner

Stacked on #64. Fixes #65.

Bug: the disk PLE table's wait-sync mode captures a WAIT on _flag in every decode graph, but the host set the flag only after cuGraphLaunch returned. A large graph that does not fit the launch queue makes cuGraphLaunch block until the GPU drains, and the GPU was parked on that WAIT, so it deadlocked. The hybrid MoE decode graph hit this on its first step with --moe-collect-stats, and intermittently without it. #65 has the evidence (py-spy stacks, flag watcher, nsys).

Fix (6ebfa18): all PleStore calls run FIFO on one ple-filler thread, so the store stays single-threaded.

  • A graph fill is submitted before the launch, with the request contexts snapshotted on the engine thread.
  • Eager fills are submitted and awaited, as before.
  • A failed launch cancels its fill and clears a signal that no WAIT consumed.

Test: tests/models/qwen4_exp/test_ple_disk_fill_order.py fakes a launch that blocks until the flag is set.

  • Two tests fail on the old code ("deadlock: the fill only runs after the launch returns").
  • All 4 pass with the fix.

GPU validation (RTX 5080, Qwen3.8-Flash-Next-NVFP4, disk PLE):

  • The deterministic repro (hybrid + --moe-collect-stats) used to hang on the first request. It now completes 3/3 300-token requests (8.5–8.8 s each).
  • Hybrid without stats: quality 14/15 (same as baseline); needle 32K and 128K 3/3; decode 37–42 tok/s from 2K to 128K; a 255K context fits with peak VRAM 15672 MiB.
  • Offload (production): a back-to-back A/B against the pre-fix build shows no decode regression (38.8 vs 38.95 tok/s).

Hybrid gives no decode gain over offload on this box (~38.2 vs ~38.8 tok/s), so production stays on offload. The fix still matters because offload uses the same wait-sync path.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…efore the launch

The disk PLE table's wait-sync mode captures a WAIT on _flag in every decode graph and set the
flag only after the graph launch returned. cuGraphLaunch can block until the GPU drains when the
graph does not fit the launch queue -- the GPU parked on that WAIT -- so a large decode graph
(hybrid MoE; hybrid + --moe-collect-stats every time) deadlocked on its first step (fork #65).

All PleStore calls now run FIFO on one filler thread (the store stays single-threaded). A graph
fill is submitted before the launch with the request contexts snapshotted on the engine thread;
eager fills are submitted and awaited as before. A failed launch cancels its fill and clears a
signal no WAIT consumed.

Assisted-by: Claude Opus 5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant