Repository navigation
decode deadlock: PLE disk wait-sync sets its flag only after cuGraphLaunch returns, which can block (hybrid MoE graphs) #65
Description
On 5080deploy @ 8975217 (RTX 5080, Ryzen 9 9950X), ft serve with the production flags plus --moe-strategy hybrid starts normally and logs --moe-hybrid-max-fetch auto: fetching 49.0% of each decode step's expert misses over PCIe ... the rest on the CPU. A 32K-token chat request then completes all prefill chunks (last one shows #running-req: 1) and never produces a decode step: no further log lines for 9+ minutes, GPU utilisation 0%, one CPU thread pinned at 100%.
Repro: production launch flags (see #17) + --moe-strategy hybrid; send any chat request.
Why it matters: decode here is bound by expert-cache misses (miss rate ~0.5, ~730 MB of PCIe copies per token at 1280 slots), and the saved ft bench bw profile shows CPU MoE + PCIe overlapped at ~66 GB/s vs ~53 GB/s PCIe alone -- hybrid is the one lever that could beat slot tuning.
Not investigated further yet (found during a serving-parameter sweep).
🤖 Generated with Claude Code
Activity
gdevenyi commentedon Sep 24, 2026
Narrowed down (2026-09-23). The hang is not hybrid mode itself: it needs hybrid +
--moe-collect-stats+ CUDA-graph decode.FREETOKEN_CPU_MOE_FLAG_SYNC=0(host-func handshake)--cuda-graph-max-bs 0(eager)record_decode_stats_hybridno-op'dcollect_decode_freqforced offState at the hang (gdb, native, scheduler process): main thread in
cuGraphLaunch(driversched_yieldspin = the 100% core) launching step N+1 while step N's graph is stalled on the GPU (0% utilization); the CPU MoE coordinator is in its idlesleep_for(it never saw a doorbell); all 15 CPU MoE workers idle on their condition variable; the flag watchdog asleep and silent (no pending doorbell). So the captured decode graph stalls on the GPU before the CPU submit node is reached, and only when either stats hook adds ops/allocations into the captured hybrid decode.Likely area: interaction of extra captured allocations/ops with the hybrid path in
layers/moe.py::_decode_hybrid(raw/cpu_ids/outtemporaries, the D2H into pinned IO,ensure_experts_hybridrewritingtopk_idsin place) under graph capture; root-causing further needs nsys / cuda-gdb on the replay. Workaround: don't combine--moe-collect-statswith--moe-strategy hybrid.🤖 Generated with Claude Code
https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt
[-]moe-strategy hybrid hangs at the first decode step (qwen4_exp, NVFP4 experts)[/-][+]moe-strategy hybrid + --moe-collect-stats hangs the captured decode graph (qwen4_exp)[/+]gdevenyi commentedon Sep 24, 2026
Root cause found (2026-09-23): a deadlock between the PLE disk backend's wait-sync protocol and
cuGraphLaunch, not the CPU MoE path.Evidence (instrumented reproduction, hybrid +
--moe-collect-stats):ready=0 done=1 err=0) with served count 1 each -- only the eager warmup before capture. The captured decode graph never reached a single MoE doorbell, i.e. the GPU stalls before layer 0's MoE.cuGraphLaunch(driversched_yield), GPU at 0%.DiskRowTable(models/qwen4_exp/ple_disk.py, wait-sync mode) records a stream-memop WAIT on_flaginside the decode graph; the host sets that flag inforward_host_ctx's exit path, i.e. only after the graph launch returns.cuGraphLaunchis allowed to block when the launch doesn't fit the channel's command queue. Hybrid adds per-layer doorbell writes/waits and D2H/H2D copies (x48 layers); stats adds more ops. Once the graph is big enough, the launch blocks waiting for the GPU to drain, the GPU is parked on the PLE flag, and the flag is only set after the launch returns -> deadlock. Deterministic with stats, intermittent without (near the threshold; two overlapped steps in flight tip it).One-variable confirmation:
FREETOKEN_PLE_SYNC=gate(fill the rows before the launch; the graph never waits on the host) with the otherwise-identical reproduction: 3/3 x 300-token requests complete (previously hung within seconds, every time).Fix direction: the fill-and-signal must not depend on the launch returning -- run the deferred fill on a dedicated worker thread (FIFO, so fills stay in step order) started before the launch. The same latent deadlock exists in offload mode; its graph is just smaller (never observed there).
🤖 Generated with Claude Code
https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt
[-]moe-strategy hybrid + --moe-collect-stats hangs the captured decode graph (qwen4_exp)[/-][+]decode deadlock: PLE disk wait-sync sets its flag only after cuGraphLaunch returns, which can block (hybrid MoE graphs)[/+]gdevenyi commentedon Sep 24, 2026
Directly observed (boundary logging in the scheduler, 2026-09-23 23:06): prefill (eager) completed -> decode step 1: PLE rows staged with the fill deferred ->
cuGraphLaunchENTERED and never returned (main thread still intorch.cuda.graphs.replay) -> the deferred PLE fill never started -> PLE_flag= 0 -> all 96 CPU-MoE doorbell slots at rest. The deadlock closes on the very first decode step. (cuda-gdb attached after the fact reports no CUDA contexts -- the process was not started with debugger support -- so device-side state was not inspected; the host-side boundary evidence is complete.)🤖 Generated with Claude Code
https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt
gdevenyi commentedon Sep 24, 2026
Fixed by #67 (
6ebfa18; merged into5080deployasa898557). The PLE graph fill now runs on a dedicated filler thread and is submitted beforecuGraphLaunch, so a blocking launch no longer deadlocks against the captured WAIT.Validation on the 5080: the deterministic repro (hybrid +
--moe-collect-stats) that hung on the first request now completes 3/3. Hybrid without stats also passes quality (14/15), needle 32K/128K, and a 255K context. The regression test (tests/models/qwen4_exp/test_ple_disk_fill_order.py) fails on the old code and passes with the fix.Hybrid itself gives no decode gain over offload here (~38.2 vs ~38.8 tok/s in a back-to-back A/B), so production stays on offload.
merge fork fix (issue #65): run the PLE graph fill on its own thread,…