NVIDIA FP4-Local checkpoints on DGX Spark (TP=1, TP=2): format, model and speed plan in rounds #565

개요

이 이슈는 따로 관리하던 체크리스트 이슈 세 개(#493, #494, #495)를 하나로 합친 플랜입니다. 기존 플랜을 3개 이슈로 나눠놨더니 제가 라운드 처리하기가 힘들어서 통합했습니다!!

전체 작업은 NVIDIA가 내놓은 FP4-Local 체크포인트를 bloomery가 형식 그대로 읽어 DGX Spark에서 빠르게 돌리는 일입니다.
1노드에서는 Qwen3.8-Flash-Next-FP4-Local을 돌리고 2노드에서는 GLM-5.3-Flash-FP4-Local을 tensor parallel(TP=2)로 돌립니다.

크게 3라운드로 나눠놨습니당.

• 1라운드는 베이스를 닦고 있습니다. 여기에 가장 많은 시간을 부으려고 합니당. 지금 다루는 두 모델뿐 아니라 앞으로 나올 모델도 그대로 올라탈 수 있게 가중치 형식과 공용 층을 먼저 깔고 있습니다.
  형식마다 읽기와 커널, TP 분할 단위를 한 번만 만들고 모델은 텐서 이름과 역할, shape만 알려 줍니다.

• 2라운드에서는 이 기반 위에 Qwen3.8과 GLM의 FP4-Local 모델을 올리고 두 모델의 MTP draft도 함께 붙입니다.

• 3라운드는 속도 최적화입니다. 2라운드에서 잰 수치를 기준으로 삼아 PR마다 main과 A/B로 비교합니다.

라운드 안의 PR이 모두 머지되면 main을 한 번 재고 그 라운드를 닫습니다.
이때 잰 수치가 다음 라운드의 기준이 됩니다.

형식과 공용 층을 먼저 깔아 두면 모델이 바뀌어도 형식 코드를 다시 만들 필요가 없습니다.
새 모델을 붙일 때 손댈 곳은 모델의 이름 표와 shape 정도로 줄어듭니다.
라운드가 끝날 때마다 main 수치가 남으니 단계마다 무엇이 얼마나 달라졌는지도 수치로 바로 확인할 수 있습니다.

1. Goal

bloomery serves NVIDIA's FP4-Local checkpoints on DGX Spark with every part in the format the checkpoint ships, and the output matches a pinned reference:

• One DGX Spark (TP=1): nvidia/Qwen3.8-Flash-Next-FP4-Local, revision b9ce18a6.
• Two DGX Sparks (TP=2, tensors split across both nodes): nvidia/GLM-5.3-Flash-FP4-Local, revision 76348ddb.

There is no speed target. Each round is measured against the round before it, and section 2 gives the numbers this work starts from.

This issue replaces #493, #494 and #495. Items are ticked as they finish, and code items link their pull request.

2. Starting point on main

These numbers were measured on main before this work started.

One DGX Spark, main eb1b521f, Qwen3.8-Flash-Next from unsloth UD-Q4_K_XL 766911a6, one request, draft off, median of three rounds. Each cell is decode / prefill in tok/s:

• MTP with the shared Q8_0 draft, 1,024 tokens a request, against draft off [derived]: 1.52 to 1.66 times on code, 1.18 to 1.28 times on greedy English prose.
• A 4-row MTP verify costs 68 to 70 ms against a 33.6 ms plain step, 2.0 to 2.1 times [derived].
• The upload phase of loading takes 29.05 to 30.52 s. The drive reads 10.034 GB/s with eight direct readers.

Two DGX Sparks, main 7de31349 on the expert tier, where node 0 runs all of the attention alone, GLM-5.3-Flash from unsloth UD-Q4_K_XL a38483c8, prompt fed as steps, one request, 128 decoded tokens, draft off. With one observation each, this is a reference point and carries no noise floor:

Current main cannot open either FP4-Local folder. Until a round serves a checkpoint on main, its speed is compared with these GGUF numbers and labelled a comparison across files.

3. How results are measured

Round results are medians and percentiles over 20 requests a row. ISL/OSL is input and output tokens a request and C the requests in flight at once. Output tok/s is the total over all requests, output tok/s/user one request's rate and prefill tok/s one request's prompt rate. In the Text column, code is the Python function text, en the English prose and ko the Korean prose. A value in parentheses is the change in percent against the previous round, or against section 2 in round 1. Load time is reported apart.

A speed pull request names, before any code, the metrics it should improve and the metrics it must not worsen. Main and the head then run alternately for three rounds in one lease. A metric is better when all three paired rounds favour the head and the median gap is larger than main's own spread over its three runs. The change stays when a listed metric is better, none is worse and the greedy tokens match main; otherwise the code is dropped and the measurement stays on record. Fewer bytes moved, fewer kernel launches, less memory or less CPU time at the same tok/s also counts.

Prompts come from four pinned public texts: code as functions (Python), JavaScript, English prose and Korean prose. A pull request uses the function code and the English prose, and all four when it changes MTP, the tokenizer or prompt handling.

4. Round 1: the base

Weight formats and shared layers that any model uses. Each format's reader, kernels, batched paths and TP cut unit are written once; a model supplies only tensor names, roles and shapes.

• Safetensors model folder as one tensor source, and the quantization config read into one format per tensor (placement, model, gpu-gates: --place tp plans GLM-5.3-Flash's Hugging Face checkpoint over N ranks, each in its own node's pool #519)
• ModelOpt NVFP4 shards open with their U8 weights and F8_E4M3 scales held as stored (hf: a ModelOpt NVFP4 shard opens, its U8 weights and F8_E4M3 scales held as stored bytes #489)
• PLE table in F8_E4M3 rows with one tensor scale (gguf, engram, model, gpu: Qwen3.8's PLE table in F8_E4M3 rows with one tensor scale, beside IQ4_NL #490)
• Routed NVFP4 experts on the card in decode (card leg: a routed NVFP4 stack runs on the card, a gate and up and down _sel pair with a scale an expert #517)
• MXFP8 weights as a card format: a gemv and a wide prompt GEMM (gguf, gpu, placement: MXFP8 weights (E4M3 blocks of 32 with an E8M0 scale) in a gemv and the F32 tile, bloomery's own types named with rows #521)
• Prepare a GGUF split from a ModelOpt checkpoint folder (hf, model: prepare a GGUF split from a ModelOpt Qwen3.8 checkpoint folder #540)
• Compressed NVFP4 scales (nvfp4_csf, csf_32x2_v1) decoded at load, with FP4-Local's MXFP8 weights and NVFP4 PLE rows read as stored (hf, model: prepare reads Qwen3.8-Flash-Next-FP4-Local, its compressed expert scales decoded, its PLE table in NVFP4 rows and its MXFP8 weights as stored #543)
• The prompt GEMM admits NVFP4 routed experts (gemm, placement: the prompt GEMM admits NVFP4, its store scaling each slot by its expert's tensor scale #545)
• Qwen3.8 served from NVIDIA's NVFP4 checkpoint on one card, against its pinned reference (gpu, model, refset: Qwen3.8 served from the NVIDIA NVFP4 checkpoint on one card, every routed expert on the card, against its pinned oracle #561)
• TP placement over N ranks, each rank in its own node's memory (placement, model, gpu-gates: --place tp plans GLM-5.3-Flash's Hugging Face checkpoint over N ranks, each in its own node's pool #519)
• All-gather of partial rows among N ranks over the existing link (gpu, gpu-gates: the link's all-gather of f32 row spans among N ranks, and link_probe --gather #501)
• FP8 weights with 128 by 128 block scales as a card format: a gemv and a wide GEMM (quant_config: an FP8 weight of one scale a 128 by 128 block is held as FP8_B128, an f32 scale a 128 values of a row #562)
• The TP rank body, GLM first: each rank's share of the model at rank shapes, checked on one card over the loopback link (gate_glm5next_tp: the rank body on the fixture at one, two and four ranks over a loopback #589)
• BF16 weights as shipped on a dense site and its wide GEMM, beside Q8_0 (gpu, placement, model: BF16 dense weights in their words, picked per tensor (Qwen3.8) #592)
• A checkpoint prepared with every part in its shipped format by default: BF16 stays BF16; the Q8_0 rewrite stays as a named opt-in lever with its measured KL (prepare, levers: a checkpoint's BF16 matrices stay BF16 by default; their Q8_0 rewrite behind BLOOMERY_PREPARE_Q8 #607)
• Per-rank slices from a safetensors folder: scales decoded first, each format cut on its own block boundary (gguf, placement, hf, model, model-hf, gpu, gpu-glm5next: a TP rank reads its parts from a GGUF file or a checkpoint folder #618)
• The other FP8 E4M3 scale layouts on the same FP8 kernels: one scale per output channel and one per tensor (hf: FP8 weights of one scale a row, a tensor or a 32 by 32 E8M0 block read onto FP8_B128 and MXFP8 #579)
• One card type table: card columns on each type's QROWS row (crates/gguf/src/qtype.rs), which the card kernels and admission read instead of their own lists (gguf, placement, gpu: a type's card columns in its QROWS row, read by every card operation and layout #588)
• bloomery's private value types named in GgmlType with their row sizes, so they pass the same checks as GGUF types (gguf, gpu, placement: MXFP8 weights (E4M3 blocks of 32 with an E8M0 scale) in a gemv and the F32 tile, bloomery's own types named with rows #521, hf, model: prepare reads Qwen3.8-Flash-Next-FP4-Local, its compressed expert scales decoded, its PLE table in NVFP4 rows and its MXFP8 weights as stored #543)

Round 1 results

Measured on main after the last round 1 PR merges.

DGX Spark TP=1: Qwen3.8-Flash-Next

NVFP4

5. Round 2: the models

The two FP4-Local models and their MTP drafts on the round 1 base.

• A safetensors checkpoint folder served directly on one DGX Spark, NVFP4 and FP4-Local as downloaded, with no prepare step (bloomery-serve, hf, gpu, model-hf: -m <folder> and --hf serve a Qwen3.8 NVFP4 checkpoint folder as downloaded, with no prepare step #635)
• Qwen3.8-Flash-Next-FP4-Local served on one DGX Spark, every part as shipped, against its pinned reference
• MTP draft state after a prime: fix the draft-state defect before the MTP drafts ship (app, gpu, gpu-gates: a window no verify takes leaves no stale draft state #631)
• MTP draft from the FP4-Local checkpoint: W4A16_NVFP4 experts and BF16 dense parts
• MTP draft from the NVFP4 checkpoint: FP8 block experts and BF16 dense parts
• GLM card paths for the FP4-Local formats: NVFP4 experts, MXFP8, FP8 block and BF16 projections
• GLM-5.3-Flash-FP4-Local loaded on TP=2 from the folder and checked against its reference
• TP=2 decode on two DGX Sparks
• TP=2 prompt batches
• The GLM server on TP=2
• GLM beyond 16,384 positions: run the full context on two Sparks
• NextN (MTP) layer of the GLM FP4-Local checkpoint on TP=2, MTP on against MTP off

Round 2 results

Measured on main after the last round 2 PR merges.

DGX Spark TP=1: Qwen3.8-Flash-Next

FP4-Local, MTP off

FP4-Local, MTP on

DGX Spark TP=2: GLM-5.3-Flash

FP4-Local, MTP off

FP4-Local, MTP on

6. Round 3: speed

Optimization against the round 2 numbers, one pull request at a time, each with its own A/B under section 3. The round 2 measurement sets the order. Candidates:

• NVFP4 routed prompt GEMM latency: a deeper tile pipeline and wide tiles
• Shared expert and GDN projections on a side stream beside the routed experts, on Qwen3.8
• MTP verify reads each routed expert once for all its rows
• Expert-once reads in every format's verify bodies
• GLM verify past two rows
• Prompt GEMM on tensor cores for MXFP8, FP8 and BF16 weights, with the weights read in their stored format
• MXFP4 routed prompt GEMM on the GB10's block-scaled tensor cores
• Routed NVFP4 scales kept compressed on the card and decoded inside the expert kernels, for both models
• NVFP4 prompt GEMM on the GB10's FP4 tensor cores, as an opt-in lever with its measured KL (4-bit activations)
• Qwen3.8 FP4-Local's BF16 hyper-connections, router and head written as Q8_0, as an opt-in lever with its measured KL; the default stays BF16
• GLM FP4-Local's BF16 projections written as Q8_0 per rank slice, as an opt-in lever with its measured KL; the default stays BF16
• 8-bit KV cache as an opt-in lever, with its KLD
• Early send of TP partial rows: start the network send while the producing GEMM still runs
• Hyper-connection kernels at the plain read rate: fuse norm and mix, and read the multi-row path at the single-row rate
• GDN delta rule on tensor cores in prefill
• Prefill kernels folded into their producers: activation quantize, accumulate, SwiGLU and shared-expert add
• No host union on layers without host experts: skip it in prefill
• Decode attention at one query row (low priority, only if long context shows a share)
• Programmatic dependent launch in the decode graph (lowest priority, only if idle time is above 2 %)
• TP=2 gather wait in one shot on the GB10
• L2 prefetch of the next weights while a gather waits
• Prompt-size gathers striped over both ports of the cable
• MTP draft stops at the chosen width instead of walking its full width
• Draft-only vocabulary head for MTP proposals; the verify keeps the shipped head, so output tokens do not change
• Mixed prompt and decode rows in one graph pass, for the first token of waiting requests
• Prompt cache checkpoints at message ends with the full state, on one DGX Spark
• Prompt cache on TP=2, each node saving its own half of the state

Round 3 results

Measured on main after the last round 3 PR merges.

DGX Spark TP=1: Qwen3.8-Flash-Next

FP4-Local, MTP off

FP4-Local, MTP on

DGX Spark TP=2: GLM-5.3-Flash

FP4-Local, MTP off

FP4-Local, MTP on

7. Round rules

PRs inside a round merge in a fixed order, and PRs that share files rebase once onto the one before. A round ends when every PR in it is merged and main is measured once. That measurement is the next round's reference, so the next round starts after it.

8. Speed PRs merged outside the rounds

• Lookup (n-gram) draft on Qwen3.8 (serve seats, generate_qwen3moe: the n-gram lookup draft on Qwen3.8 through the seats' one lookup owner #546): MTP serve decode on one DGX Spark, median of 3, 28.15 tok/s on main against 28.747 tok/s, +2.12 %.
• Dense Q8_0 prompt GEMM at the int8 tensor-core rate on GB10 (gpu: gemm_q8_0p_slab reads the Q8_0 planes a row's step at a time for a full card of tiles that share their slabs #549): prefill at 2,048 prompt tokens on one DGX Spark, median of 4, 1188.63 tok/s on main against 1296.56 tok/s, +9.1 %.
• The m-column F32 walk of the MTP verify (f32 gemv: the m-column walk keeps eight chunks' loads in flight a column #511): MTP decode at 4,096 tokens +7.2 % on code and +7.8 % on English prose against main e8abdc89, with decoded ids equal to main's.

9. Accuracy and related work

• Formats: every decoder is compared byte for byte with a pinned Python reference written from the format's published reference code, and a structural error is refused by name.
• Models: the reference is a Hugging Face Transformers forward of each checkpoint in f32, with every weight dequantized by its own rule and input_scale ignored. The served model is judged by the existing numerical gates with unchanged bands.
• TP: all ranks are bit-equal at every step, and the TP body run on one node matches the existing body bit for bit.
• A pull request that changes shared code shows with a test that the existing card, host and NVMe paths behave as before.

10. Other pull requests from this work

Fixes and tool changes that came up while doing the work above. None of them is a plan item, and each one stands on its own.

Engine:

• A dense call on a resident weight goes through one site by name, and the card type table picks its launch (gpu: a dense call on a resident weight goes through one site by name, and the card type table picks its launch #639)
• Q8 quantizers refuse non-finite values (quant: refuse non-finite values in Q8 quantizers #597)
• A misaligned vector view is refused at launch (gpu: refuse misaligned vector views at launch #520)
• Windows over raw memory give back their context handle on every path (gpu, gpu-deepseek41, gpu-gates: windows over raw memory give back their context handle on every path #513)
• A tier card on another host plans in its own node's memory pool (placement, gpu-gates: a tier card on another host plans in its own node's pool #503)
• Qwen3.8's placement is workstation::placed_on with the route scratch on top (model: Qwen3.8's machine is workstation::placed_on with the route scratch on top #512)
• An --hf vision projector outside the plan stays only when its media plan is free (vision: an --hf projector off the plan stays only when its media plan is free #485)

Gates and tools:

• gate_nvtier loads the all-RAM arm before the paged arm and declares the arm's real host need (gate_nvtier: load the all-RAM arm before the paged arm, and declare its real host need #633)
• Stale recorded fixture budgets are refused before planning (gpu-gates: refuse stale recorded fixture budgets before planning #606)
• The Qwen3-VL tower and V4.1 vision gates refuse a set without its full-tap image by name (gpu-gates: the Qwen3-VL tower and V4.1 vision gates refuse a set without their full-tap image by name #515)
• tools/box.sh checks the cargo-oxide pin against the executable Cargo runs (tools/box.sh: check the cargo-oxide pin against the executable Cargo runs #560)
• Test isolation: sj-vs-clef gets its own data directory, and gate-bound's FIFO self-test and executable fixtures are written in a child (tests: sj-vs-clef's own data directory, gate-bound's FIFO self-test and executable fixtures written in a child, so main's PR check passes #557)
• mac-static keeps each full run's results in its own directory (mac-static: keep each full run's results in its own directory #522)
• ptx-spill-check fails when a count or a comparison cannot be read (ptx-spill-check: a failed count or comparison read fails the binary #486)
• lds-scan fails when a line count fails (lds-scan: a failed line count fails the scan #483)
• The nsys-ds41 self-test's stub tree carries gate-bound.sh (nsys-ds41: the self-test's stub tree carries gate-bound.sh #482)

#493 and #494 together replaced #458. This issue replaces #493, #494 and #495.

AI help

AI 코딩 에이전트가 문서를 작성한 후 이슈를 올렸습니다. 문서 리뷰와 이슈 리뷰는 제가 직접 했습니다.