feat(MODEL-MM-GLM53-FLASH): W7a — the glm5_next GGUF converter, gated byte-for-byte against llama.cpp b10451 (#2011)

[localai-bot]

W7a of MODEL-MM-GLM53-FLASH: the safetensors→GGUF converter for glm5_next, and the gate that holds its k-quant encoders to the byte.

This pull request is STACKED on #2001. Its base is row/MODEL-MM-GLM53-FLASH, not main, because the spec it implements is committed there and not yet on main — the split-pull-request case AGENTS.md names, where a helper dispatch needs a base-reachable committed spec. Merge #2001 first, and do not delete its branch before this one is merged: deleting a stacked base auto-closes the child. The base moved once while this was in flight and was merged in, not rebased, so #2001's spec commit stays the shared merge base.

Every GPU gate on this row is blocked behind an artifact that does not exist. Measured live 2026-08-26 against ~119.63 GiB usable on dgx:gpu0: FP8 305.78 GiB, BF16 598.53, the smallest published NVFP4 181.32, and all four repositories named *-GGUF contain zero .gguf files. No upstream tool can make one — llama.cpp has no glm5_next at any revision — so this authors it.

What lands

scripts/convert-glm5-next-gguf.py reads a safetensors checkpoint and writes a GGUF at arch glm5next. It streams: headers for the plan, mmap slices for the data, so peak resident memory is one tensor rather than one shard, which is what makes a 305.78 GiB source tractable. FP8 e4m3 is decoded against the weight_scale_inv grid the checkpoint declares (weight_block_size: [128, 128]); per-expert tensors are stacked into ffn_{gate,up,down}_exps; the layer-45 MTP block and every shared_head.* are dropped, following the reference's own _keys_to_ignore_on_load_unexpected and the glm4_moe_lite_registry.cpp:21-26 precedent.

The metadata carries the parameters the port hinges on. glm5next.kda.gate_lower_bound is the load-bearing one: -5.0 selects -bound * sigmoid(exp(A_log) * (g + dt_bias)), a different function from the -exp(A_log) * softplus(g + dt_bias) our Kimi-Linear KDA implements, and even the sign of decay_rate differs. Writing it into the file is what lets a loader take the right branch instead of inheriting Kimi's. Beside it: the indexer geometry with index_kpool = 4 where the config class default is 16, the mHC triple keeping hc_eps 1e-6 distinct from rms_norm_eps 1e-5, and layer_types as the authoritative schedule — the reference ignores linear_attn_config.kda_layers entirely, so only one of those two lists may travel. No rope.freq_base is written; the text stack is NoPE end to end.

The encoders are byte-identical to the pinned reference, and that is the point

gguf.quants.Q2_K upstream implements dequantize_blocks and no quantize_blocks. Upstream Python cannot produce a k-quant at all, so Q2_K, Q6_K and Q8_0 are ported from ggml/src/ggml-quants.c at our pin b10451 (:891, :1869, :276, over make_qkx2_quants:799, make_qx_quants:628, nearest_int:621) and gated against a frozen golden captured from that reference compiled -ffp-contract=off. Bytes rather than a tolerance, because an encoder that is close but not exact writes a file that loads, generates fluent text, and is quietly worse than the arm it claims to be.

Two traps changed bytes during the port and are recorded in the source so nobody re-finds them:

• nearest_int is the +12582912.0 add-and-mask trick and rounds half to even, not round.
• C roundf in quantize_row_q8_0_ref rounds half away from zero where np.rint rounds half to even. Caught by a crafted [0.5, -0.5, 1.5, -1.5, 2.5, -2.5] case: reference [1,-1,2,-2,3,-3], np.rint [0,0,2,-2,2,-2].

The pin already had everything, so nothing advances

D6 said llama.cpp has no glm5_next, and it still does not — re-verified at origin/master 539f24529 (fetched 2026-08-26) and at the pin: the enumerators are LLM_ARCH_GLM4, LLM_ARCH_GLM4_MOE, LLM_ARCH_GLM_DSA (src/llama-arch.h:86-88), and src/models/glm-dsa.cpp is GLM-5.2, citing zai-org/GLM-5.2/blob/main/config.json, a different model. But "no implementation" is not "no convention", and every convention this converter needs is present at b10451: class KDA with {arch}.kda.head_dim and {arch}.kda.gate_lower_bound (gguf-py/gguf/constants.py:262-264); the KDA tensor spellings including the three separate ssm_conv1d_q/k/v this checkpoint's packing needs (src/llama-arch.cpp:465-479); the HF module paths that map onto them (gguf-py/gguf/tensor_mapping.py:896-933, Kimi-Linear's paths being GLM-5.3-Flash's verbatim); and the indexer names including the k-pool compressor (:626-636). KDA.SAFE_GATE is the only master-only member and this model declares no safe_gate. No pin advance was taken and none is owed.

The arithmetic, from the converter's own plan

Not bits-per-weight times a parameter count — the converter resolves a type per tensor, so its plan over the real topology is what will actually be written: 1719 tensors carrying 313,890,512,702 parameters, which is 321.32B less the 7.43B MTP block and therefore an independent confirmation that the skip is exactly the 2.31% the spec measured.

arm	weights	mixed bpw	breakdown
q2_k (experts Q2_K, rest Q6_K)	100.35 GiB	2.746	Q2_K 93.02, Q6_K 7.17, F32 0.08, Q8_0 0.07
q6_k	239.89 GiB	6.565	Q6_K 239.73
q8_0	310.67 GiB	8.502	Q8_0 310.58
bf16	584.67 GiB	16.000	BF16 584.67

Against ~119.63 GiB at 128K context and one sequence: KV 1.43 GiB (11 MLA layers × kv_lora_rank 512 × 2 B, plus an indexer side cache of index_head_dim 128 × 2 B / index_kpool 4 = 64 B per layer × 11, so 11,968 B/token), KDA recurrent state 0.14 GiB (64 heads × 128 × 128 × 4 B × 34 layers) plus conv states, leaving ~17.7 GiB. The Q2_K figure is 100.35 and not the spec's 102.6 because that table stated every arm including layer 45. Arithmetic, not measurement: #1963 and #1966 record this accounting being wrong by 48x, and W5 re-derives it from the runner.

The fallback ladder is want → Q8_0 → F32 rather than want → F32, and that is not tidiness: with an F32 fallback a finer arm can come out larger than a coarser one, which is not a property a size table may have. The gate asserts the ordering.

Refusals

Every unimplemented arm is refused by name with the missing part. The i-quants because an imatrix needs a forward pass, a forward pass needs 181 GiB, and the dependency is circular on this fleet — a boundary, not a to-do. q3_k/q4_k/q5_k because their encoders are unported and shipping an ungated encoder is worse than refusing. So are --keep-mtp, a non-glm5_next config, and an FP8 tensor whose weight_scale_inv companion is missing, which would otherwise write a loadable, wrong file.

Evidence

RED, with the converter absent:

  FAIL scripts/convert-glm5-next-gguf.py is absent

1 failure(s)
RED_RC=1

GREEN: python3 tests/scripts/test_convert_glm5_next_gguf.py → 55 assertions, All cases passed., rc=0. The suite builds a SYNTHETIC tiny-shape checkpoint and parses the result with an independent in-test GGUF reader that shares no code with the writer. No checkpoint, no GPU, no C++ build — no ninja target was built by this change and no build directory was touched, which is deliberate given the disk state on this box.

Two defects the gate caught during development, both of the write-a-loadable-wrong-file kind: the np.rint rounding above, and a 5-axis tensor. model.visual.patch_embed.proj.weight is a Conv3d at [1024, 3, 2, 14, 14] and ggml carries at most GGML_MAX_DIMS = 4, so writing it verbatim produces a header no reader can index. 4-D and 5-D convolution kernels are now flattened to the im2col form [out, prod(rest)]; 3-D shapes are left alone because they mean something in ggml (the depthwise conv is [ch, 1, k], the expert lane is [experts, n, m]). A hard refusal backs it up.

Mutation

Every guarantee claimed above was broken in a scratch copy, the break was verified to have LANDED, the gate was rerun, the file was restored and the restore was verified by SHA-256, and the gate was rerun again. __pycache__ is cleared before every run, because a restored file can still execute the mutant's bytecode. No ninja target exists in this change, so there is no build step to report an exit code for: the harness runs a Python gate directly, ninja was never invoked, and no build directory was created or touched.

#	mutation	applied?	gate rc	failures	first failing assertion
M1	nearest_int half-to-EVEN → half-UP	yes	1	2	Q2_K matches the pinned reference byte-for-byte (504 bytes)
M2	roundf → np.rint	yes	1	1	Q8_0 matches the pinned reference byte-for-byte (1632 bytes)
M3	Q6_K scale stride 0/2/4/6 → 0/1/2/3	yes	1	2	Q6_K matches the pinned reference byte-for-byte (1260 bytes)
M4	drop glm5next.kda.gate_lower_bound	yes	1	1	gate_lower_bound is -5.0
M5	carry the layer-45 MTP block	yes	1	6	q2_k conversion exits 0 (rc=1)
M6	stack experts in REVERSE order	yes	1	2	expert lane round-trips through Q2_K (max rel 2.0220)
M7	FP8 dequant ignores weight_scale_inv	yes	1	1	FP8 MLA projection round-trips within 2%
M8	no fallback ladder: force the requested type	yes	1	3	the k-pool embedding steps down to Q8_0
M9	stop refusing the i-quant arms	yes	1	2	--arm iq2_xxs is refused and names the imatrix
M10	write a uniform schedule instead of layer_types	yes	1	1	layer_types is the authoritative schedule

Baseline rc=0/0 failures; post-restore rc=0/0 failures; both file hashes match.

The first run of this table found a hole, and the third commit closes it. M1 and M2 came back gate rc 0, 0 failures — GATE BLIND. The assertions were right and the inputs were wrong: the golden was captured over weight-like random data, heavy-tailed data and zeros, none of which ever lands a value exactly on a rounding tie, so half-to-even and half-up agree everywhere the fixture looked. Three super-blocks were added and the golden recaptured from the same pinned reference: blocks 3 and 4 were searched on an eighth-lattice until the two rules disagree under Q2_K and Q6_K, and block 5 is eight Q8_0 sub-blocks with amax exactly 127 so x * id is ±0.5, ±1.5, ±2.5. The table above is the rerun.

M9's first form was also a non-mutation: adding iq2_xxs to ARMS changes nothing while REFUSED_ARMS is checked first. It is reported here because "a mutation that never applied reads as a passing test", and the fix was to mutate the refusal itself.

Not reached — read this before reviewing

glm5next has no entry in the general.architecture dispatch (src/vllm/entrypoints/model_loader.cpp:1000), so the file this converter writes is not loadable by this tree. That wiring is W1's, owned by row MODEL-MM-glm5-next-glm5-next-for-conditional-generation and tracked by #1998; the spec lists it under ## Owed as O9. The converter itself IS reached, as a command-line path, and the gate enters through it as a user does.

No artifact has been produced against the real checkpoint either: that needs the 300–600 GiB weights staged on local disk, explicit developer authority for the download, and a box with room for source and output at once. Owed as O7; W7b owns it. The Q3_K/Q4_K/Q5_K encoders are owed as O8.

Gate

scripts/agent-preflight.sh on the merge commit: 1 gate failed, test_cpu_x86_llamacpp_floor, and it is not attributable. Its failure is #618/#529 verbatim — AssertionError: 4 != 2 and, on the isolated re-run, NO_QUIET_WINDOW after 30s (busy=121% builders=0 load=46.91 66.29 65.22). The box carried loadavg 25–85 from other sessions throughout. This branch changes no .cpp, .cu or .h file, nothing under benchmarks/, and not the harness itself — the whole diff is one Python converter, its Python gate, a JSON fixture, records and docs. Re-run and reported rather than waved through.

Everything else is green, including test_convert_glm5_next_gguf, check-agent-record (stale=31 broken=6 rot=37, unchanged from the base), check-model-checklist, audit-live-rows, issue-index append-only, commit-trailers and commit-style.

One process note worth recording, because it produced a red that was not one: the worktree was deleted underneath a running preflight by another session's reaper, and the run reported 74 failing gates of the form python3: can't open file 'scripts/check-issue-index-append-only.py': No such file or directory. That signature is a moved tree, not a defect. The branch survived in the shared repository, the worktree was recreated at the same commit, and the verdict above is from the clean re-run.

Closes #2011.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-fable-5 [Claude Code]