[void-mckenzie]

Adds the llama.cpp deepseek4 GGUF reader on the GGUF quant kind from #690, serving every DeepSeek-V4 text release: Flash, Flash-0731, Pro and Pro-0813. Vision-Exp trunks serve text too - their vision towers ship as mmproj sidecars, which stay out of scope here (#671's vision item). Part of #671. Review fixes from #774 folded in: the clamped SwiGLU reaches the routed experts, the embed exemption matches through its checkpoint root, and a missing yarn block defaults to the family values.

What it does

• parse_gguf_config builds the family's own DeepseekV4Args from the GGUF KVs, then a model_config_from_args extracted from the HF parse_config - one ModelConfig construction for both flows, no drift.
• Compressor layout derived from the tensor table (indexer.* -> 4; compressor without indexer -> 128, or 4 when its wkv has the CSA overlap's double rows). Every released file's attention.compress_ratios array (43 trunk entries + MTP zeros on Flash, 61 on Pro - all layers compressed, 30 with indexers) agrees with the derivation element for element.
• Rope: a full rope.scaling block is used as written; a wholly missing one falls back to the family values (yarn, factor 16, original 65536, betas 32 / 1 - identical in all four official configs) with a warning naming the file and values; a partly present one refuses. llama.cpp serves these files unscaled, which is wrong for them.
• iter_gguf_weights yields the same names the HF loader yields:
  • dense, dequantized: compressor wkv/wgate (read through a bf16 einsum, not the quant method; registry exempts them like the HF spec), wo_a, mHC mixers, router, attn_sinks, tid2eid (I32 -> int64)
  • packed at the checkpoint ggml type: attention linears, indexer, shared experts, embed, head
• embed resolves dense on the HF row through a new ("model.embed", "embed") checkpoint root so the bf16 exemption matches (it ships bf16 although the fp8 config claims it, the head pattern); the GGUF row's to_gguf maps it itself and packs token_embd.
• Routed experts load as GGUF expert banks at their shipped types (UD-Q2_K_XL: IQ2_XS gate/up, IQ3_XXS / Q6_K / MXFP4 down, mixed per layer) and run the clamped SwiGLU (activation="swiglu_clamp") - plain silu would silently drop the +-limit clamp on the GGUF path; the ds_fp4 kernels read only the limit, so the HF path is unchanged.
• Tokenizer: joyai-llm is DeepSeek-V4's pre-tokenizer spelling; transformers 5.18 maps no split for it, so load_gguf_tokenizer installs the model's own three splits + ByteLevel.
• Refusals, each matching llama.cpp where it also refuses:
  • expert_gating_func != sqrtsoftplus (deepseek4.cpp: no default)
  • non-uniform swiglu clamps, rope scaling != yarn, kv heads > 1, partly present rope.scaling
  • vision tensors in the trunk
  • REAP per-layer expert prunes (llama.cpp builds from one scalar n_expert; uniform prunes like the 132-expert REAM/REAP exports load fine)

tok_compare

File	Before fix	After fix
unsloth UD-Q2_K_XL vs HF 0731 tokenizer	6/13 edge cases differ	0/13 edge, 0/126,527 vocab texts

Check sweep (scripts/gguf_check)

1,992 header stubs, 812 files checked, kernels probed on the RTX 5090.

Set	Result
deepseek4	344/502 (430/626 served archs overall)
Flash GA (unsloth, ggml-org, bullerwins, tarruda)	109 standard quants pass
Flash-0731 (unsloth + mirrors)	206 pass
Vision-Exp trunks (unsloth, ggml-org, tarruda, huihui, prometheusAIR, antirez, BahamutRU)	27/28 - text; the one failure is an IQ1_M quant (kernel gap); mmproj sidecars out of scope
deepseek-ai V4-Pro (batiai mirror, Q4_K_M / Q3_K_M)	config, dry load, banks and kernels all pass - the files ship no chat template (base-style, #721's ignore class); 61 layers, hidden 7168, 384 experts, all-compressed layout the Flash files never exercise
unsloth 0731 / Flash / Pro-0813	36/38 - only UD-IQ1_M fails (kernel gap)
REAM/REAP uniform 132-expert exports	16/2 - the 2 are IQ1_M
no-yarn files (bartowski MXFP4, antirez Pro mixes, ...)	pass with the family-yarn warning
mirror farms (AtomicChat, Stk88, ...)	~12/1 each
unserved archs (dflash, deepseek41, dspark, controlvector, vits)	186, correctly skipped

Every failure is a clean refusal or a broken file. Not covered:

Count	Class
69	no token_embd (LoRA exports, per-layer streams, vision sidecars)
35	custom ggml types (ROCmFP 101, TQ3_4S, dsx 143, EXTENSOR, Q4RMFP4)
21	IQ1_M / IQ1_S experts and dense linears (no MMQ kernel)
19	REAP per-layer expert prunes
16	no chat template (base / abliterated line; includes the plain-Pro files)
14	missing expert_gating_func (llama.cpp refuses too)
11	missing-tensor refusals (MTP-only drafts, zero-expert REAP-K layers)
4	mixed gate/up expert bank types
4	broken uploads (no arch, truncated, bad split naming)
3	incomplete split sets (sweep artifact, not the loader)
1	missing swiglu_clamp_exp (custom EXTENSOR file)

Real-file serving (the dry load never runs a forward)

unsloth Flash-0731 UD-Q2_K_XL (3 shards, 90.18 GiB), RTX 5090 32 GB, Ryzen 9 7950X3D, 187 GiB RAM, driver 580.95.05:

ft serve --gpu 0 --model-path .../DeepSeek-V4-Flash-0731-UD-Q2_K_XL-00001-of-00003.gguf \
  --max-seq-len-override 32768 --moe-cache-mem 6GiB --memory-ratio 0.82

Serving caught one bug the dry load could not: the KV compressors read wkv.weight directly through a bf16 einsum, so packed Q8_0 rows must dequantize (the crash was a [1024 x 4352] matmul; 4352 = Q8_0 row bytes for k=4096). Fixed before the runs below.

Test	Result
Chat / coding / math	coherent; complete LRU-cache implementation; 17*23 + 45/9 = 396 with steps
4,424-token prefill	accurate two-sentence summary
12 concurrent greedy requests	identical correct answers, none empty or corrupted
Decode, bs=1 steady	26-28 tok/s (expert-fetch bound)
Errors in log	0

Tests

pytest tests/models (not slow): 328 passed, 1 failed - test_minimax_m3.py::test_nvfp4_experts_restricted_to_triton_for_swigluoai, which fails identically on current main 127aa0a (verified side by side). Registry and AOT-claim tests pick up the new entries. Review-fix checks: scheme_for("model.embed") resolves to None on the HF row, and the no-yarn files parse to the family values with the warning. No per-family GGUF test file, per the #721 review direction.

[jason-fxz]

Thanks, I checked your code and found the issues below.

Must fix

1. The routed experts lose the SwiGLU clamp on GGUF. deepseek_v4/moe.py:100 still builds the expert layer with activation="silu" and limit=args.swiglu_limit. The native ds_fp4 path reads only limit, but the GGUF MoE path keys on the activation name:

fused_glu(activation="silu", limit=10.0)          -> None
fused_glu(activation="swiglu_clamp", limit=10.0)  -> ('swiglu_clamp', 10.0)

With None it falls back to gated_act_and_mul("silu", ...), which is silu_and_mul and ignores limit (layers/activation.py:76-77). DeepSeek-V4 clamps (inference/model.py:655-657: up to +-10, gate to max 10) and so does llama.cpp (llama-graph.cpp:2389-2393, ggml_swiglu_clamp with swiglu_clamp_exp[il]). Fix: activation="swiglu_clamp" in DSV4OffloadMoELayer. The ds_fp4 GPU / CPU kernels read only limit, so the HF path does not change (greedy output on the HF checkpoint is identical with the change), and the Triton, CPU and ggml swiglu_clamp epilogues are exactly silu(clamp(g, max=L)) * clamp(u, -L, L) at alpha 1.

Should fix

2. The HF row's "embed" exemption never matches. register.py:215 adds "embed", but the HF row has no model.embed root, so the name stays model.embed and name_set misses it: Fp8BlockConfig.scheme_for("model.embed") returns fp8_block. HF loading works today only because fp8 has no embedding method and the layer falls back to UnquantizedEmbeddingMethod. Add ("model.embed", "embed") to the HF row's checkpoint_roots; then embed matches and resolves to scheme None. The GGUF row has no roots and its to_gguf maps model.embed itself, so it stays as is and still packs token_embd.

Your open question: files without rope.scaling

Default them to the family values. The rope settings are identical in all four official DeepSeek-V4 configs (V4-Flash, V4-Flash-0731, V4-Pro, V4-Pro-0813): rope_theta 10000, yarn, factor 16, original_max_position_embeddings 65536, beta_fast 32, beta_slow 1 (and compress_rope_theta 160000). The 31 files lost the block in conversion, they are not unscaled models, so llama.cpp's unscaled rope is the wrong answer for them. Please make it:

• full rope.scaling block present: use the file's values (as now)
• whole block missing: use the values above and log a warning that names the file and the values used
• block only partly present: refuse, the file is inconsistent

Follow-up (not blocking)

Nothing to change in this PR; keep wo_a dequantized to bf16 (285) here. Keeping it quantized will be a separate PR, opened after this one merges: today it costs +1.26 GiB over Q8_0 and ~1.26 of ~10 GiB more weight bytes per bs=1 decode step, and llama.cpp runs it quantized as one 3-D ggml_mul_mat over the 8 groups. That PR adds a batched op_mul_mat to ggml_ops.cu and switches Attention._wo to it.

Verified

• parse_gguf_config against an independent parser on 45 published V4-Flash / 0731 files: identical DeepseekV4Args on 44, including the tensor-derived compress_ratios; the 45th is the no-yarn file both refuse.
• gguf_check on 196 stubs from 13 repos: 56 / 61 deepseek4 pass (failures: 8 ik_llama types, 2 IQ1_M kernel gap, 2 mtp-only, 1 no-yarn).
• Tokenizer: 0 / 308 texts differ from the HF tokenizer.json on the ggml-org V4-Flash, ggml-org 0731 and unsloth 0731 headers.
• pytest tests/models tests/server tests/tokenizer tests/dsv4 on CPU: 999 passed, 4 failed, all pre-existing on main (test_quant_config cases that need checkpoints not in the local HF cache).
• F16 hc / ape (huihui files) are cast by the engine before the dtype assert; indexer.weights_proj packed is fine (called through .forward); ft checkpoint is unaffected by the embedding change (it writes the reader's tensors as is).
• Merges cleanly onto current main fe5a5ec (its three new commits do not touch these files).

[void-mckenzie]

All three addressed in 39f6b7e, rebased onto current main (cc4aa08, clean).

1. swiglu_clamp. DSV4OffloadMoELayer now builds with activation="swiglu_clamp"; the routed experts keep the +-limit clamp on the GGUF path. The ds_fp4 path reads only limit, so the HF flow is byte-identical.

2. embed exemption. Added ("model.embed", "embed") to the HF row's checkpoint_roots; scheme_for("model.embed") now resolves to None instead of fp8_block (verified against the local 0731 config). The GGUF row is untouched and still packs token_embd.

3. rope.scaling, per your ruling. Full block -> used as written. Wholly missing -> family values (yarn, factor 16, original 65536, betas 32 / 1) with a warning naming the file and values. Partly present -> refused as inconsistent.

Re-ran the sweep rows that were failing on the missing block: 26 of the 31 now pass the dry load (bartowski MXFP4, the antirez Pro mixes, ...); the other 5 parse past config and surface their next problem, all in existing classes (2 MTP-ish missing attn_q_a, 1 no chat template, 1 sidecar without token_embd, 1 mixed gate/up). deepseek4 is now 344/502 in the sweep, 430/626 served archs overall; the PR body's tables are updated.

LEt me know if you have any other changes :)

[void-mckenzie]

Retitling and updating the body: this loads the text trunks of the whole V4 family : Flash (GA + 0731), Pro (GA+ 0813), and Vision-Exp, which has the mmproj for vision. Plain Pro is verified in the sweep now too (batiai Q4_K_M / Q3_K_M: config, dry load, banks and kernels pass; the files just carry no chat template).

On the mmproj follow-up: I am happy to pick it up. I have a local branch with vision GGUF support already working for GLM-5.3-Flash and DeepSeek-Vision-Exp, so if you have pointers on how you want it structured (mmproj discovery, vision tower mount, where the mm processor hooks in), I can adapt it to the gguf pattern.