[juliankang4]

bloomery-serve가 받은 NVIDIA Qwen3.8 NVFP4 체크포인트 폴더를 prepare 없이 그대로 한 대의 DGX Spark에서 띄우게 합니다.

Part of #565.

What and why

bloomery-serve now serves NVIDIA's Qwen3.8-Flash-Next-NVFP4 checkpoint folder exactly as downloaded, on one DGX Spark. -m <folder> takes the folder, and --hf <repo> takes a safetensors repository and resolves it to its folder. main refuses such a repository as holding no GGUF model, and the folder has to be converted by prepare into a GGUF split of 129211441440 B first. The prepared split is still accepted and its plan is unchanged.

Every part is loaded in its stored format, with no Q8_0 rewrite. The folder load serves only what this checkpoint needs on one card. Every other case is refused by name (see "What I ran into").

• Part load (crates/gpu/src/parts.rs): parts::load is the one owner of loading a source's parts. It plans and checks every part first, then stages them through one pinned ring with one synchronize. GLM's tensor-parallel load (crates/gpu-glm5next/src/tp/mod.rs) now calls it and passes only its rank rule. The folder path uses the same source seam at one rank.
• Folder reading (crates/model-hf/src/arch/qwen35moe/folder.rs): folder::open opens the folder for a load and for prepare, which now calls it. crates/safetensors, crates/gguf, crates/tokenizer and crates/engram gain a header-only split, a tokenizer built from header keys, and a run table that reads the PLE rows (the per-layer embedding rows) where the folder stores them.
• Qwen3.8 load (crates/gpu/src/arch/qwen3moe/body38.rs): Model38 and Folder38 take a folder next to a split. The PLE table is read from the folder on the NVMe tier.
• --hf (crates/hf): a repository that holds a safetensors checkpoint and no GGUF set resolves to the checkpoint's folder, and the new hf_checkpoint record names it.
• Serve (crates/gpu-gates/src/bin/shared/serve_seats/): qwen38_folder.rs is the one place that opens a folder, and the seat is chosen from the path's header.
• checkpoint cargo feature (crates/gpu-gates/Cargo.toml): it carries bloomery-model-hf for the folder path, so the recipes that do not reach that code keep their key and reach pin at zero gap. The release feature list and the recipes that build bloomery-serve turn it on. A build without it refuses a folder by name. The per-model bloomery-serve-qwen38 recipe stays on hf, so that binary refuses a folder until it is built with checkpoint.
• Safety fix (crates/model/src/placement/host_lock.rs): releasing a file's pages is refused for a shard that holds only a header, because a page release must never madvise a heap copy. The test is a_header_alone_releases_no_page.
• Docs: README.md, docs/serving.md, docs/BUILD.md and docs/contrib/dgx-spark.md.

Release notes: Added: bloomery-serve serves NVIDIA's Qwen3.8 NVFP4 checkpoint folder as downloaded on one DGX Spark, from -m <folder> or --hf <repo>, with no prepare step.

Code shape

• No second copy. Loading a source's parts has one owner, parts::load, and GLM uses it. The check "this load carries no draft" has one owner in the serve seat, draft_off_for, which the media path and the folder path share. Finding a shard by name has one function, Folder::shard in crates/safetensors/src/lib.rs, which file_range, mapped and upstream's raw all use. prepare and the folder load share folder::open, served_as_hf and plan_tensors, which moved out of prepare.rs with their refusals unchanged.
• Model code holds only architecture facts. The source trait and parts::load carry no model name. The folder reader names only Qwen3.8's tensors and its PLE rows.
• Dead code is gone. The loop in GLM's tp::load and prepare's own copies of the folder checks are replaced by the owners above.
• No silent failure. Each of these is a named error: a folder in a build without checkpoint, a repository of another quant, a PLE row that is not F8, a PLE table on the host tier, a tensor that does not sit whole on the card, a folder with residency on, BLOOMERY_DRAFT=mtp on a folder, a run table with no rows, an empty row or a run past the end of its file, and a page release on a header-only shard.
• Comments state what is true now.
• No relaxed gate. No band or tolerance moved. The unsafe ratchet rises by one block in engram, one block and one function in model, and one block in model-hf, each with a dated PIN line and a SAFETY line at its use.
• cargo fmt and cargo clippy are clean on what I touched; the lint count did not go up.

Duplication found or removed (path:line): the duplicate scan reported three candidates. crates/engram/src/hash.rs:660 and :670 read different types and share only a one-line lookup. crates/engram/src/rows.rs:457 and :461 are two trait methods that each delegate to their own method. crates/safetensors/src/lib.rs:475 and :485 share Folder::shard, as above.

What I ran

The PR head is 1bb657e852cd5e8a844c8496e91c44cd6719c374, on main 58356c0039f1a707142e4ac2bb15677f2e1e7afe. It is c56ca5349517db0c9525255efe38436310ae681f rebased onto that main with every commit unchanged; the only file both touch is README.md, where main changed other lines. The GPU, CPU and gate runs below ran on a commit whose non-test code is the same as c56ca534; the two differ only in two #[cfg(test)] test modules.

The model is nvidia/Qwen3.8-Flash-Next-NVFP4 at revision fc694b54fb0174e0913e6adf86691ef85a4ead47, 25 files, 132,734,505,212 bytes. The prepared split of the same checkpoint is the reference.

Plan, folder against split (bloomery-serve --plan, draft off): the planned bytes are equal.

-m <folder> --ctx-size 131328:                   need bytes=92093887040
-m <split>  --ctx-size 131328:                   need bytes=92093887040
-m <folder> --parallel 6 --ctx-size 25344:       need bytes=89578782100
-m <split>  --parallel 6 --ctx-size 25344:       need bytes=89578782100

Serving, folder against split (context 8192, one slot, draft off): need bytes is 84788613696 and resident_bytes is 81854312844 for both, and the captured graph has 1295 nodes for both. Greedy decoding gives identical ids and text for both prompts ("The capital of France is", 32 ids, and a Fibonacci function prompt, 64 ids), and the chat reply to "Name three primary colors in one short sentence." (64 tokens) is identical. The comparison is JSON equality over the saved responses.

The correctness gate on the folder is under "GPU Evidence".

Residency and draft on a folder: BLOOMERY_DRAFT=mtp bloomery-serve --plan -m <folder> exits 1 with a named refusal, and an unset draft is turned off by itself.

Tests, on the PR head, on an x86_64 CPU host:

cargo test ... placement::host_lock::
test result: ok. 6 passed; 0 failed

The fail-first run is under "GPU Evidence". The host crates (model-hf, hf, safetensors, engram, gguf, tokenizer, model) pass 735 tests with 0 failures, and the gpu-gates library tests for hf, record:: and model_file:: pass 32.

GPU Evidence

The card is a DGX Spark (GB10) in the dev container image, driver 595.84, CUDA 13.3.

NVIDIA GB10, 595.84, CUDA 13.3
| NVIDIA-SMI 595.84                 Driver Version: 595.84         CUDA Version: 13.3     |

The serving build used --features glm5next,clef,mimo2,solar2,minimaxm3,vision,gemma-embedding2,hf,checkpoint. The gate is built with --features gpu,checkpoint.

Correctness gate, on the folder:

BLOOMERY_REF_MODEL=<folder> just gate-gpu-qwen4exp-nvfp4
gate rc=0
(p) argmax ours=11751 oracle=11751 PASS
(g) argmax ours=11751 oracle=11751 PASS
(t) position 3000: argmax ours=423 oracle=423 PASS

gate_qwen4exp_nvfp4 uses the Hugging Face NVFP4 checkpoint itself as its oracle. gate_qwen4exp_e2e is not run on the folder because it compares against the Q8_0 split with routed experts on the host tier, and the folder load requires every tensor whole on the card.

Reference	Result
prepared split of the same checkpoint	folder plan bytes, greedy ids and chat reply equal, PASS

The folder and the prepared split have the same planned bytes, the same greedy ids and the same chat reply.

Fail-first, on the PR head on the same card: each of four mutants turned its test red (exit 101), and with the source restored all four tests passed.

on_card_takes_the_card_s_tensors_and_refuses_the_rest        (the "whole on the card" check removed)
a_folder_load_refuses_residency_and_a_draft                  (the residency refusal removed)
a_folder_s_stored_rows_decode_as_the_split_s                 (the host-tier refusal of a folder's PLE table removed)
a_folder_load_refuses_a_set_mtp_draft_and_turns_an_unset_one_off   (the mtp refusal removed)
restored: 3 passed, 1 passed

The change adds no kernel and puts no launch, copy or synchronize on the decode path. The load path uses one pinned ring and one synchronize for the whole load, so no timed A/B is attached.

What I ran into

This does not block the PR. The folder load covers the NVFP4 checkpoint on one card and refuses the rest by name.

• A folder with the MTP draft, with residency on, or with FP4-Local's NVFP4 PLE rows is refused.
• Loading the folder took longer than loading the split in the same sitting. It is observed, not a timed comparison, so no number is claimed.
• just contrib-check --tier t0 was not run on this head.
• The checks ran on one DGX Spark. The rank source is used here at one rank, and GLM's two- and four-rank loads are covered by its existing tests.

Hardware

A DGX Spark (GB10), driver 595.84, CUDA 13.3 in the dev container, for the serving runs, the gate and the fail-first run. An x86_64 CPU host ran the unit tests.

AI help

AI 코딩 에이전트가 코드를 작성한 후 설명을 썼고, 테스트를 마쳤습니다. diff 리뷰와 PR 초안 리뷰는 제가 직접 했습니다.

[github-actions]

Verdict: CLEAN: No must-fix behavior or quality defect survived the code checks.

Optimization headroom

• E. Less work: crates/engram/src/rows.rs:417. Iterate run spans without materializing nested vectors. On folder decoding with FillMode::Touch and classification off, prefetch and copy each rebuild these groups. Removing that materialization saves at least 2 × (S + 1) heap allocations per token for S > 0 run stretches, before vector growth [derived].

Checked

Duplicates: 92 added/changed definitions checked, including moved helpers; no proven duplicate. Dryer reported 0 pairs. Its excluded gpu and gpu-gates crates were checked by Grep.
Complexity: ple_from_keys (23) and plan_tensors (20) rename/move existing logic; load_placed_from (17) relocates orchestration. Serve run (+6), open_with (+6), resolve_drafts (+4), PleHost::new (+3), and Q38::open (+2) add source selection/refusals. Draft38Off::fmt (+1) adds one match arm.
Silent failure: 5 fallback sites followed. Inverted ranges return an error; absent format overrides select the type's format; cache index paths have a parent; scale stems retain their names; absent optional BOS/EOS flags mean false.
Tests: Static mutation checks covered run-boundary comparisons, header-only tensor reads, whole-card/draft/residency refusals, PLE scale/width/row-count checks, online/offline checkpoint selection, and the file-mapping guard; their assertions reject those mutants. Renamed hash and tokenizer tests retain their reference assertions. The release/reread assertion checks byte preservation; a no-op release still passes it.
Gates: No tolerance moved. Three unsafe-count increases carry dated PIN reasons. No changed kernel or float reduction.
Adversarial checks: Boundary row ids, partial expert placement, missing stored PLE rows, and header-only page release reach named refusals.
Escape catalog: C1–C4 have no new matches.

Notes

This was a static review. Builds, tests, and GPU gates were not rerun here. The PR includes a GB10 gate run with the required evidence.

_{Card check: gpu-evidence. This pull request changes crates/gpu-gates/Cargo.toml, crates/gpu-gates/src/bin/bloomery_serve.rs, crates/gpu-gates/src/bin/bloomery_serve_qwen38.rs … and carries a gate run on the author's card, which counts as verified.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[juliankang4]

The review is done and the contribution check passed, so I'm merging this.