Capture-time cuGraphUpload (#697) widens inter-kernel gaps in replayed decode graphs: ~1-2% slower decode #775

What happened

Since #697 (fix(engine): upload cuda graphs at capture …), decode steps replayed from CUDA graphs are about 1-2% slower. The extra time is idle time between kernels; the kernels themselves are not slower.

Measured on Qwen3.8-Flash-Next-NVFP4 (qwen4_exp, MoE offload, decode CUDA graphs at bs 1-2) with a like-for-like decode workload: the model copies a file verbatim from a 64K context, greedy, thinking off. The output hash is identical in every arm, so the tokens and expert misses match.

nsys over 200 decode steps:

• Kernel counts and API calls are identical.
• Kernel busy time is not higher with the upload; it is 0.37 ms per step lower.
• The median gap between consecutive kernels goes from 0.19 µs to 0.51 µs on all four streams the graph uses.

At about 1090 kernels per step, that adds 0.2-0.45 ms per step. The higher figure is with --moe-collect-stats, which adds kernels.

Cause (current main, cc56b6b)

python/freetoken/engine/graph.py:118-128 _upload_graph() calls cuGraphUpload on every captured graph, then stream.synchronize(). It is called from the capture loop at :222. Graphs uploaded this way replay with the wider inter-kernel gaps above. Uploading after all graphs are captured instead of after each one gave the same result.

Proposed fix

• Upload only where the deadlock fix(engine): upload cuda graphs at capture so a stream wait cannot block the first launch #697 fixes can occur: a disk PLE backend in wait-sync mode whose fill is released after cudaGraphLaunch returns. Skip it otherwise, or put it behind a switch.
• Or replace the upload with one warm replay at capture time. This recovered about 60% of the cost here. It needs the PLE wait released during that replay, for example by running the fill on a thread submitted before the launch.

Environment

RTX 5080 16 GB (sm_120, driver 615.78.08), torch 2.14.1+cu130, CUDA graphs for decode at bs 1 and 2, nvidia/Qwen3.8-Flash-Next-NVFP4, main cc56b6b.