Repository navigation
Conversation
… FREETOKEN_GRAPH_UPLOAD On Qwen3.8-Flash / RTX 5080 a decode graph uploaded with cuGraphUpload at capture (FlashML-org#697) replays with ~0.5 us instead of ~0.2 us between kernels: +0.19 ms/step (-1% decode), +0.43 ms with --moe-collect-stats. FREETOKEN_GRAPH_UPLOAD=0 keeps the driver's first-launch upload; default on. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
FREETOKEN_GRAPH_UPLOAD(default 1, as upstream) so FlashML-org#697's capture-timecuGraphUploadcan be turned off. Production sets it to 0 inrun-freetoken.sh. Stacked on #96.Why
A graph uploaded at capture replays with a median 0.51 µs gap between kernels instead of 0.19 µs. That holds on all four streams, from nsys over 200 decode steps. Kernel counts and API calls are identical, and kernel busy time is even 0.37 ms lower. A decode step has about 1090 kernels, so the gaps cost +0.2 to 0.45 ms per step, about 1-2% of decode.
FREETOKEN_GRAPH_UPLOAD=0A per-step fit (time = a + b·misses) gave fixed cost 12.95 ms before FlashML-org#697 and 13.36 ms after, with the same per-miss slope. FlashML-org#739's larger slot rows and FlashML-org#700's slot pools cost nothing measurable. Data:
~/projects/FreeToken-bench/steptime/.Risk
FlashML-org#697 uploads at capture to avoid a first-launch block under PLE wait-sync. This branch already runs the PLE graph fill on a filler thread submitted before the launch (fork #65 / 4f2528f) and recycles the readback events (5221ce0, 7bb5359). c23d2d5 served production without the upload. With the upload off:
test_ple_disk_fill_order.py,test_ple_disk_no_graph_fence.pyandtest_graph_upload.pypass (10), and 3 cold starts of the production launcher answered the first request in 1.24 s. More than 10 cold starts during the investigation also never hung.🤖 Generated with Claude Code
https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846