From 78d2c8a61377b1e747e96711b4950c986a6f1d69 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 16:59:37 +0000 Subject: [PATCH 1/5] spec(MODEL-MM-GLM53-FLASH): GLM-5.3-Flash is an architecture no oracle implements and no device here can hold (#1998) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `zai-org/GLM-5.3-Flash` was published 2026-08-26 and declares `Glm5NextForConditionalGeneration` / `model_type: glm5_next`. Nothing in this tree matched it. This lands the row's records and its spec; no product code. The name collides with work that is already blocked, and that misreading is the expensive one. `.agents/model-matrix.md` already carries a row labelled "GLM-5": `MODEL-TEXT-deepseek-v2-glm-moe-dsa-for-causal-lm`, `BLOCKED` at 753.9B, which is DeepSeek-V3.2 verbatim. `glm5_next` is a different model_type, a different parameter count, a different attention topology, and it has a vision tower. New row, not a revival. **vLLM implements nothing, at either end of the pin.** Read live 2026-08-26: `git grep "Glm5\|glm5_next"` returns zero hits at `555967922` and zero at `origin/main` = `c71f6f8a81`, whose `vllm/models/` package holds `common`, `deepseek_v32`, `deepseek_v4`, `dots3_note`, `inkling`, `kimi_k3` and `minimax_m3`. That is absence from vLLM `main`, not staleness in our pin, so advancing the pin — 348 commits of reconciliation — reaches nothing here. What exists upstream is vllm#53906, opened the same day, open and `mergeable: false`, and an unmerged pull request is not a revision. It is inadmissible as an oracle and is cited in the spec only as evidence of upstream shape. SGLang is the same picture with one trap worth recording: its GLM-5.3-Flash *cookbook* merged while its implementation PR stayed open, so the model card links a recipe SGLang `main` cannot run. llama.cpp has no `glm5_next` either. So the only admissible oracle is transformers, and the release is bounded rather than assumed: `modeling_glm5_next.py` is HTTP 200 at `v5.16.1` and 404 at `v5.16.0` and `v5.15.1`, making 5.16.1 the first release carrying the architecture. The `MODEL-MM-QWEN4-EXP` lane pins 5.16.0 and this one pins 5.16.1, one day apart, because `Qwen4Exp` merged before the 5.16.0 cut and `Glm5Next` after it. Two lanes, two releases, which is what a lane pin is for. Most of the model is already here, and the survey moved three conclusions. **Two of the three obvious reuses are wrong in one place each, in ways a token gate cannot see.** The KDA forget gate branches on `safe_gate_lower_bound`; the checkpoint sets `gate_lower_bound: -5.0`, so it takes the bounded branch, `-5.0 * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias))`, where our `kimi_kda.cpp` implements Kimi-Linear's `-exp(A_log) * softplus(g + dt_bias)`. Different functions of the same inputs, both smooth, both negative, both fluent, and the sign of `decay_rate` differs too. And the mHC head collapse is `hidden_streams.mean(dim=2)`, an unweighted mean whose own docstring says "Unlike DeepSeek-V4", where our `HcHeadCollapse` is the V4 weighted-gated collapse — the checkpoint carries no `hc_head.*` tensors at all, which settles it. Both get a red-first wave test whose failure against the wrong reuse is the deliverable. **The text stack has no rotary embedding anywhere**, which is easy to get wrong in the other direction. `qk_rope_head_dim: 0`, `mla_use_nope: true`, no `rope_theta`, no `rope_scaling`; the reference deletes the inherited `rope_parameters` field and passes `position_embeddings=None` to every layer. `indexer_rope_interleave: true` is a vestigial flag the indexer override ignores, so implementing it because the config names one is implementing a bug. Position comes only from the KDA decay and the short convs. NoPE does buy one thing: MLA weight absorption becomes trivially valid, since there is no rope slice to keep out of it. It also blocks: `MlaBlockDims::Validate` refuses `qk_rope_head_dim == 0` today, and Kimi-Linear is not the same case — it sets `mla_use_nope` while keeping `qk_rope_head_dim: 64`, so the rope slice still occupies the cache row. **The k-pool indexer is unobservable at or below 2048 tokens.** With `index_topk: 2048` a shorter context selects every candidate, the selection is the identity, the pooling is invisible, and any short-prompt gate passes over a broken indexer. That is now a stated requirement in `## Gates` rather than a note. The pooling itself is learned and per channel — 128 independent 4-way softmaxes over a learned `[4,128]` intra-pool position embedding, pools aligned to the first valid token so the grid is batch-dependent, expanded back to raw indices and widened to 2051 by an unscored ragged tail. **Nothing fits, and no GGUF exists.** Measured by HTTP RANGE over all 62 safetensors headers, payload never fetched: 76,108 tensors, 321,322,735,872 real parameters, routed experts 311.65B = 97.0%. Against ~119.63 GiB on GB10 the published FP8 is 305.78 GiB, BF16 598.53, and the smallest, an NVFP4 requant, is 181.32 — 1.52x over. All four repositories named `*-GGUF` contain zero `.gguf` files. `gateable = no` on memory, and the consequence is the scoping result: no oracle can execute this model on any device this project reaches, so no end-to-end token gate exists or can exist here. That is recorded as visible debt rather than waived, and what replaces it is a tiny-shape CPU reference oracle instantiated from the pinned transformers — a real oracle for the numerics, and explicitly not oracle gateability for the model. The developer set the W7 target as a low GGUF quant that fits the DGX. The arithmetic, with experts at the named type and the other 3% at Q6_K and bits per weight taken from our own reader's block traits: Q8_0 318.0 GiB, Q6_K 245.5, Q4_K 170.6, IQ3_XXS 118.1 which leaves ~1.5 GiB and is not a margin, Q2_K 102.6 with ~17 GiB left, IQ2_XXS 82.3. Q2_K is the recommendation and the reason is producibility, not quality: i-quants need an importance matrix, an importance matrix needs a forward pass, a forward pass needs 181 GiB, and the dependency is circular on this fleet. K-quants need none. Since llama.cpp has no `glm5_next` the converter is ours to author, with no reference GGUF to diff against and no llama.cpp floor for the arm. The `MODEL` row ratchet moves 378 -> 379 with its justification appended in the existing log format, re-derived off the matrix rather than carried forward. One row and not three, which is the arithmetic the new test exists to hold: the open vllm#53906 would register `Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` and `Glm5NextMTPModel`, so the IndexTTS-2.5 and dots3-note two-row shape is the tempting read and it does not apply. None of the three is registered at any vLLM revision, and the only architecture a published artifact declares is `Glm5NextForConditionalGeneration`; the MTP head is `layers.45` inside the same checkpoint, which the transformers reference discards outright. The at-the-pin static invariants (324/373/356/310/261) are unchanged, because the Upstream cell carries no pinned module/class target. Two anchors in `engine-matrix.md` are repaired in the same change. Inserting the new test into `tests/scripts/test_agent_record.py` shifted `ENG-RECORD-ANCHOR-RATCHET`'s two citations of that same file by 45 lines — the rot that row exists to measure, produced by an edit to the very file it cites. Rot goes 39 -> 37. Evidence. `check-agent-record.py` OK at ENGINE=173 MODEL=379 QUANT=84 KERNEL=58 BACKEND=87 ANCHOR-ROT=37; `check-model-checklist.py` OK; `tests/scripts/test_agent_record.py` 124 passed. The new test was mutation-proved three ways, each restored byte-for-byte and re-verified green: renaming the row reds it, adding a `-for-causal-lm` sibling to "match" the upstream PR reds it, and leaving the ratchet at 378 reds it. The third mutation initially read as a false red after restore because `scripts/__pycache__` still held the mutant's bytecode; it was re-run with the caches cleared on both sides. Scoping only. No product code lands under this pull request, and the eight implementation waves are dispatchable from the committed spec — W0 to W2 and W4 CPU-gateable, W3 and W5 to W8 needing a GPU, W8 unscheduled because there is no denominator. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-fable-5 [Claude Code] --- .agents/engine-matrix.md | 2 +- .agents/issue-index.md | 1 + .agents/model-matrix.md | 8 +- .agents/specs/glm5-next-flash.md | 1221 ++++++++++++++++++++++++++++ scripts/check-agent-record.py | 19 +- tests/scripts/test_agent_record.py | 45 + 6 files changed, 1291 insertions(+), 5 deletions(-) create mode 100644 .agents/specs/glm5-next-flash.md diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index c9a62f2b9..009e8a3d0 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -227,7 +227,7 @@ claims it. | `ENG-RELEASE-WINDOWS` | Native Windows x86_64 pre-alpha release extension: one adaptive MSVC/UCRT CPU bundle with AVX2 executed in CI and one Vulkan preview bundle, both deterministic ZIPs and authenticated by the existing release handoff | T0 | vLLM has no Windows release path; runtime behavior remains pinned to vLLM `555967922`. Platform substrate reference: llama.cpp `src/llama-mmap.cpp:520-590` @ `237ad9b961f009ae19ac29dbce4cd0c1251f94b3`; Win32 API is the OS authority | W14 Win32 portability/MSVC CPU, W15 deterministic ZIP/PE packaging + Vulkan, and W16 ten-tuple prerelease workflow/version/docs implemented for one PR | Linux portability/release mutation gates are local evidence only. Native `windows-2022` MSVC `/W4 /WX`, extracted runtime/ISA smokes, merged-SHA ten-tuple dry run, `v0.0.3-pre.1` publication, attestations, and exact 32-asset audit remain pending; no Windows ZIP exists yet | [windows-binary-release.md](specs/windows-binary-release.md); [#117](https://github.com/mudler/vllm.cpp/issues/117) | `ACTIVE` | `CLAIM-ENG-RELEASE-WINDOWS` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | `docker/Dockerfile` (cpu/vulkan/cuda targets calling the release scripts); `docker/healthcheck.sh`; `release/container-matrix.json`; `scripts/check-container-matrix.py`; `scripts/check-container-workflow.py`; `scripts/validate-container-image.py`; `scripts/container_tags.py`; `.github/workflows/containers.yml`; SIGTERM handler `src/vllm/entrypoints/openai/server_main.cpp` (`SignalShutdown`, all three `listen()` sites); the pre-existing `docker/Dockerfile.arm64` is an unrelated CPU bench cross-check | issues `#170`, `#312`, `#394`; `tests/scripts/test_check_container_matrix.py` 31/31; `test_check_container_workflow.py` 29/29; `test_check_cuda_fat_gencode.py` 7+4 subtests. **GB10 2026-08-11 (`promaxgb10-4ad8`, `sm_121a`, CUDA 13.3): arm64 cuda image 1.71 GB, 673/673 objects, ten-SM gencode audit PASS, and a REAL GPU boot -- `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, `--gpus all`, host driver 580.159.03 injected.** cpu amd64 783 MB gated locally; cpu+vulkan amd64 green on hosted CI **arm64 cuda lane RUNTIME-VERIFIED on GB10 2026-08-11** -- the first accelerator-hardware evidence for any lane. Four defects were removed to get there, each found by building rather than reading: the CUDA 12.9 base could not compile `sm_110`, the BuildKit cache mount outlived its toolchain (both #366), Marlin gencode had drifted from the feature table and failed the audit on 14 correctly-compiled TUs (#394, blocking BOTH cuda tuples project-wide), and the validator could only ever produce build evidence because its boot smoke never passed `--gpus`. **NOT established: nothing is published to GHCR; amd64 cuda is unbuilt; the published arm64 image is SBSA (`targets/sbsa-linux`), so Tegra -- Thor `sm_110`, Orin `sm_87` -- is untested and NOT covered** **ORIN (Tegra) 2026-08-11: the SBSA image RUNS on Jetson AGX Orin `sm_87` (L4T R36.4.3, Docker 27.5.1) -- Qwen3-0.6B (rev `c1899de2`) loads and GENERATES via `/v1/completions`, tegrastats GR3D 95-97% during decode vs 14-15% idle.** Tegra needs `--runtime nvidia --gpus all`: `--gpus` alone is refused by the hook and `--runtime` alone mounts no driver | [container-images.md](specs/container-images.md); issues [#170](https://github.com/mudler/vllm.cpp/issues/170), [#312](https://github.com/mudler/vllm.cpp/issues/312), [#394](https://github.com/mudler/vllm.cpp/issues/394) | `ACTIVE` | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | | `ENG-DOCS-SITE` | Publish the 11 `docs/*.md` as a browsable GitHub Pages site at `https://mudler.github.io/vllm.cpp/` WITHOUT a second copy of the prose. A Hugo site at `website/` mounts `../docs` READ-ONLY and derives everything else from what is already in the files: each page title from the file's first `# H1`, the sidebar order from `website/data/nav.yaml`, and links through a Goldmark render hook (internal `.md` → site URL; the 139 `../.agents/**` and `../AGENTS.md` escapes → GitHub blob URLs, since the protocol tree is deliberately NOT published). **No file under `docs/` is modified, moved, renamed, or given front matter**, so `check-doc-checkpoint.py` and every protocol path reference keep working and there is no second surface that can drift — the whole point of the row. Custom lean layouts, NO theme and NO submodule: off-the-shelf docs themes read titles, weights and menus out of front matter this design deliberately does not have, so each would need its title partial, menu and link hook overridden anyway, and hugo-book additionally floors at Hugo 0.158 against the 0.146.3 pin CI and the local toolchain share. Hard prerequisite inside the repo: `classify_path` in `scripts/check-pr-size.py` FAILS CLOSED on `website/**` (verified: raises `ValueError: unclassified repository path`), so the classifier must learn the path or the PR cannot pass the project's own size gate. Hard prerequisite outside it: GitHub Pages must be enabled with the source set to GitHub Actions — the workflow is inert otherwise. A marketing landing page is explicitly OUT of scope (`README.md` stays the front door), as is any restructuring of `docs/`; the custom domain is parked behind the pending vLLM trademark question | T1 | NO vLLM analogue — upstream's docs are a separate mkdocs site and nothing in this row mirrors upstream *behavior*, so it carries no parity obligation. The STRUCTURAL reference is LocalAI's `.github/workflows/gh-pages.yml` (two Hugo sites merged into one Pages artifact), reduced to the docs half | read-only mount `website/hugo.toml:29`; title-from-H1 `website/layouts/partials/title.html:10`; link rewriting `website/layouts/_default/_markup/render-link.html:27`; guard `scripts/check-site.py:70`; deploy `.github/workflows/gh-pages.yml` | `tests/scripts/test_check_site.py:51,56,66,80,89,97` (6 mutation cases: clean tree, H1 stripped, doc absent from nav, nav entry with no file, duplicated entry, missing nav file); build evidence 14 pages with `docs/bench-evidence` + `docs/superpowers` absent from `public/` and no `href` ending in `.md`; 48 protocol links rewritten in `docs/status/`. NO published page is claimed: GitHub Pages is not yet enabled on the repository, which is the recorded stop condition holding this row at `GATING` | [gh-pages-docs-site.md](specs/gh-pages-docs-site.md); issue [#224](https://github.com/mudler/vllm.cpp/issues/224) | `READY` | `CLAIM-ENG-DOCS-SITE` | -| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1494` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1562`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | +| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1539` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1607`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | | `ENG-RECORD-CONFLICT-SURFACES` | Retire the shared record surfaces that make concurrent PRs conflict by construction. MEASURED at `origin/main` `d928e2c3` with `git merge-tree --write-tree` over every open PR: **16 of 29 conflict (55%), and 13 of the 16 conflict in bookkeeping files ONLY**, with no product code involved — `.agents/coordination.md` in 8, `.agents/NOW.md` in 5, `.agents/roadmap_v1.md` in 4, `scripts/check-public-doc-tables.py` in 4, `docs/STATUS.md` in 4, and any `src/`/`tests/` path in just 3. Three defects, each of which GUARANTEES rather than risks a collision. (1) `.agents/NOW.md` is a fixed-size shared buffer at EXACTLY 6000/6000 chars (`check-now-current.py:31`), so adding a row requires evicting another and every PR is a read-modify-write of one global — and the conflict is the LUCKY outcome, since a clean three-way merge would apply both evictions and both additions, silently dropping live rows and blowing the very budget the checker defends. (2) `STATUS_RATCHET = {"chars": 243245}` (`check-public-doc-tables.py:557`) is a hardcoded byte count of a DIFFERENT file that may only fall, so a PR owing `docs/STATUS.md` one lifecycle line must delete unrelated prose from another row to pay for it and edit the checker too; the checker's own comment at `:331` already records the failure (*"a ratchet pinned to the byte turns every concurrently merged row's one-line status edit into a spurious failure"*) and answered it with slack instead of removing the coupling. (3) `.agents/coordination.md`'s active-claims table is insert-at-one-anchor: the six ROCm GDN PRs (#334 #336 #341 #343 #345 #348) are ONE author's sequential stack that conflicts on nothing else, each appending a ~1,500-char row — the PR description, transcribed into a file every other claim also writes. It also contradicts the protocol it serves: `AGENTS.md` holds that *"History is git"* and *"There is no state log"*, yet both claims tables ARE state logs duplicating `gh pr list`, `row/` branch names and issue state; the argument that refuses a waiver registry applies unchanged to a claims registry. Precedent twice over — `policy.csv` retired in `0f3e44ee`, per-class line budgets retired 2026-08-10 because the gate fired on ordinary work. The exonerated surfaces share ONE property, one writer per file: `.agents/specs/.md` (one file per row, **zero conflicts** in the sample), the `*-matrix.md` inventories, and the append-only `.agents/benchmark-record.md`. SCOPE: remove `STATUS_RATCHET` and the doc-gating global counters while KEEPING the per-cell/per-paragraph caps (local, so they couple nothing); remove the active-claims table and derive claims from open PRs and branch names; drop `NOW.md`'s byte budget; order the roadmap's keyed tables by ID so distinct keys stop colliding at one anchor; and record the invariant — **no surface that every PR must write** — in `AGENTS.md`. No product source, kernel or gate semantic moves | T0 | NO vLLM analogue — this is local protocol machinery, so the mirror rule does not apply and no upstream `file:line` exists to port from. Governed instead by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | - | - (spec-before-code: the red-before suites are named in the spec's Tests section — `tests/scripts/test_check_public_doc_tables.py`, `tests/scripts/test_check_now_current.py`, a mutation case per removed rule proving the obligation survives in the retained caps and `check-doc-checkpoint.py`, and a `git merge-tree` merge-shape regression that must be RED before the `NOW.md`/roadmap work and GREEN after) | [retire-shared-record-surfaces.md](specs/retire-shared-record-surfaces.md); issue [#364](https://github.com/mudler/vllm.cpp/issues/364) | `READY` | `CLAIM-ENG-RECORD-CONFLICT-SURFACES` | | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate rejects CORRECT commits because of paragraph placement, and that is why `main` is red on `agent-record`. `check-commit-trailers.py` reads trailers via `git interpret-trailers --parse`, which treats ONLY the final paragraph as the block; GitHub appends `Co-authored-by:` as a SEPARATE trailing paragraph on a squash merge, so a complete correct block becomes invisible and the gate reports it missing. MEASURED: piping `git show -s --format=%B dbd0d51c` into `git interpret-trailers --parse` prints nothing but the co-author line, and 13 of the last 30 commits on `main` fail the check -- unnoticed only because those runs were cancelled (#274), which HID the defect rather than causing it. FIX: fuse consecutive trailing TRAILER-SHAPED paragraphs before parsing. Nothing is relaxed -- the block must still exist, the marker must still sit above it, each declaration must still appear exactly once, and an AI co-author is still forbidden; the block is merely FOUND where the merge tool left it. A prose paragraph still terminates it. REJECTED IN FLIGHT and recorded because it is the more instructive half: a first attempt also collapsed identical duplicate trailers to fix the multi-commit-squash shape, which relaxes the uniqueness rule an existing test already pins. Rewriting that assertion to suit the change is what AGENTS.md forbids, and the distinction is real -- a doubled block is genuinely malformed and fixable at source, whereas the co-author case is a correct commit defeated by the parser. Reverted in full. SCOPE LIMIT, stated rather than implied: this fixes ONE of five observed shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had been hiding and now correctly fails; `87308dea` (GitHub's `---------` separator), `b8293c88` (squash doubled the block) and `b580452d` (merge button, no trailers) stay red by design. Closing those is a merge-method change, not a checker change | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:60` (`join_trailing_trailer_paragraphs`, `_is_trailer_paragraph`, and the fused `parsed_trailers`) | `tests/scripts/test_check_commit_trailers.py:1` 21 cases -- the RED-BEFORE appended-co-author case plus four GUARDS that keep the fusion bounded (doubled block still fails, contradictory declarations still fail, a no-trailer merge message still fails, prose after the block still fails), all four green before and after; closure [parity-ledger.md#L941](parity-ledger.md#L941) | [trailer-merge-artifacts.md](specs/trailer-merge-artifacts.md); issue [#406](https://github.com/mudler/vllm.cpp/issues/406) | `DONE` | `157080c8` | | `ENG-FORGE-COAUTHOR` | The forbidden-AI-trailer rule was catching ATTRIBUTION rather than an authorship claim, which is why bot-opened PRs red `main` on merge. GitHub composes the squash message itself and appends the account that opened the PR — `Co-authored-by: localai-org-maint-bot <...@users.noreply.github.com>` — and most PRs here are opened by a bot, so nearly every squash trips the AI-identity check. Real instance `f64f2b71`, invisible until #406 repaired the parse, which is why it reads as a new failure and is not one. The rule exists so an AI cannot claim it WROTE the code, and that stays; GitHub is recording who pressed the button, and the AI-involvement claim is already carried separately by `AI-Assisted` and `Assisted-by` in the same block. FIX: accept a `Co-authored-by` at a GitHub account noreply address even when the name matches an AI identity token, keyed on the FORGE'S OWN DOMAIN rather than the name so the exemption cannot be borrowed. A hand-written `Co-authored-by: Claude ` still fails; `Signed-off-by` is excluded from the exemption entirely, because a sign-off is a legal assertion about provenance rather than attribution. `AGENTS.md` records the same distinction in the same change so prose and checker cannot drift | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:38` (`FORGE_ACCOUNT_EMAIL` and the forbidden-trailer skip) | `tests/scripts/test_check_commit_trailers.py:1` 25 cases -- the RED-BEFORE forge-bot case plus THREE guards that matter more than the relaxation because this LOOSENS a rule: a hand-written AI co-author still fails, `Signed-off-by` at the same noreply address still fails, and a human co-author still passes; all three green before and after. Real commit `f64f2b71` re-verified per commit | [forge-coauthor-attribution.md](specs/forge-coauthor-attribution.md); issue [#418](https://github.com/mudler/vllm.cpp/issues/418) | `ACTIVE` | `CLAIM-ENG-FORGE-COAUTHOR` | diff --git a/.agents/issue-index.md b/.agents/issue-index.md index ff108e7fd..b89302e6e 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -745,3 +745,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#526](https://github.com/mudler/vllm.cpp/issues/526) | `SERVE-TOOL-HISTORY-ARGS` | OpenAI multi-turn tool history reaches chat templates with string-valued arguments | bug | | [#1934](https://github.com/mudler/vllm.cpp/issues/1934) | `BACKEND-ROCM` | `RocmPlatform::needs_weight_staging()` is stale-false (a W0-era placeholder never revisited despite #523/#509/#506/ROCM_ATTN/hipGraph landing since), so `CheckDeviceWeightFit` — the #1123/#1870 load-time refusal, including the `policy_forces_full_expand` fix — never runs on ROCm: measured directly, `VT_DEVICE_WEIGHT_BUDGET_BYTES=1` produced no refusal on a real load. The actual device allocation the refusal guards is not gated on this flag, so #1870's crash stays reachable until this closes; owed, not fixed in flow, because flipping the flag also moves `DirectDeviceLoadEligible` and several GDN kernel-dispatch defaults that each need their own correctness check | bug | | [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-QWEN4-EXP` | **`Qwen/Qwen3.8-Flash-Next` declares `Qwen4ExpForConditionalGeneration` / `qwen4_exp`, a new architecture vLLM does not implement, so the port runs on a split oracle: transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH.** Released 2026-08-24, 180B total / 6B activated, image-text-to-text. The `Qwen3.8` in the name is marketing continuity: `.agents/specs/qwen38-27b-bf16-gate.md`'s "one config key differs" precedent does NOT extend here. Read live 2026-08-26 at vLLM `origin/main` = `6a5e8f5979`: no `qwen4*` path, no registry entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. That is absence from vLLM `main` rather than staleness in our pin `555967922`, so a pin advance does not reach it. What exists is transformers [#48337](https://github.com/huggingface/transformers/pull/48337) "Add Qwen4Exp model", MERGED 2026-08-26, 5211 lines, and SGLang [#36497](https://github.com/sgl-project/sglang/pull/36497), still OPEN and therefore inadmissible. **Developer direction 2026-08-26, recorded verbatim: "use transformers as oracle for algorithmic side. but use ops from vllm so we account for optimized path."** Justified rather than convenient: `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so porting it as written yields a correct model at an indefensible speed, while AGENTS.md's mirror-vLLM polarity still binds every primitive vLLM implements. `Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`), all of which this tree has; GDN is an exact match for our AOT gate (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). The delta is four things, and **exactly two have no vLLM op at all**: the PLE dilated depthwise conv (kernel 4, dilation 3; `git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the n-gram hashed embedding. **The survey's load-bearing finding, and it REVERSES this row's first reading: QSA's structural twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The original call was that QSA, being plain GQA rather than MLA, had to map onto vLLM's non-MLA block-sparse case; that reasoning rested on treating `MLAAttentionSpec` as an MLA claim, and **it is not one** — M3's own indexer cache uses it while M3 is a plain-GQA model, with the comment "Key-only: MLAAttentionSpec budgets one vector/token (not 2x for K+V)". It is a budget shape. Remove that prop and the GQA-vs-MLA argument collapses. Verified line by line at `6a5e8f5979`: **nine independent structural matches with DSv4**, `compress_ratio == 4` literally the same number — MQA index with 1 key head at dim 128; `relu(q.k)` summed over index heads vs `(score.relu() * weights).sum(dim=0)`; `1/sqrt(head_dim)`; one score set per query token with no head axis vs `topk_indices_buffer[num_tokens, topk]`; pooling boundary `(position+1) % COMPRESS_RATIO == 0`; RMSNorm on the pooled key; **RoPE at the block-start position** vs `compressed_pos = (position // CR) * CR`; candidate count `visible // compress_ratio`; and one stored state per 4 tokens via `MLAAttentionSpec(tokens_per_state=compress_ratio)`, a first-class KV field documented as "Ints > 1 compress multiple tokens into one state (DSv4 sparse MLA)" that has no M3 equivalent. **M3 is a DIFFERENT ALGORITHM**, not a worse fit: its score is `tl.max(qk, axis=1)` over 128 RAW token dots with no pooling, no relu and no head reduction, it asserts `num_idx_heads == num_kv_heads` ("no topk index reduce") so it emits one block set PER KV HEAD, and its `SPARSE_BLOCK_SIZE = 128` is welded to the KV page size ("One sparse block == one KV page") on both the score and the attend side — moving it to 4 forces a page size of 4 and breaks `tl.dot`, whose tile needs >= 16. M3 contributes exactly ONE thing and it is a wiring precedent, not an algorithm: that a plain-GQA model can own a key-only side cache through `MLAAttentionSpec` and a private indexer backend. **The genuinely new work is the CONSUMER and nothing upstream supplies it** — every DSv4 sparse consumer attends to COMPRESSED MLA KV (one state per 4 tokens) and M3's attend to raw tokens only at page granularity, while QSA attends to RAW tokens selected at ratio-4 granularity. Two silent-failure traps follow: wiring QSA's top-k into a DSv4 sparse-MLA consumer attends a POOLED key/value and still emits plausible tokens, and **a short-prompt token gate cannot catch it because at context <= `indexer_budget` 2048 every candidate is selected** — so any QSA gate must run past 2048 tokens of context, which is now a stated `## Gates` requirement; and `SparseAttnCompressNormRopeStoreC4Kernel` does NOT mean-pool despite its name — it is a learned softmax pool over an OVERLAPPING window of 8 using a score channel this checkpoint does not have, and the CuteDSL variant refuses `overlap=False` at compile, so the **Triton** `head_dim=128` variant is the correct starting point. Two structural consequences beyond the module list: the residual stream is `hc_count * hidden_size` = **4 x 2560 = 10240 wide through the whole stack** with a low-rank read gate and per-branch scalar write gate around both attention and MLP, which is a change to the per-layer loop and every residual buffer rather than a drop-in module; and `number_of_conv_states = 3` on a PLE layer (GDN conv, PLE conv, n-gram token history) plus the indexer side cache, adjacent to [#1963](https://github.com/mudler/vllm.cpp/issues/1963) and [#1966](https://github.com/mudler/vllm.cpp/issues/1966). **NOTHING PUBLISHED FITS**, read live from the HF API against ~119 GB usable on GB10: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB (NVFP4 backbone with the n-gram table left at FP8, 51.2 GB) and `unsloth/...-GGUF` is a README with ZERO weight files. No GGUF exists and no tool can make one, because llama.cpp has no `qwen4_exp` either, so the standing k-quant requirement means authoring the arch on our side AND states that the quantized arms have NO llama.cpp oracle. **The chosen arm does NOT load today, and the blocker is neither the offload nor the budget: this tree cannot keep a gather table quantized at all.** `KeepQuantKDim` returns `-1` for `GgufTensorRole::kEmbeddingTable` (`src/vllm/model_executor/model_loader/gguf_keep_quant.cpp`), and `qwen3_5_gguf_weights.cpp` asserts it by name — "the embedding table cannot keep quant blocks" — so a Q4_K or Q8_0 n-gram table EXPANDS to bf16 and 51.2B params become **102.4 GB of anonymous memory**; the arm dies before the first forward. The reason was already sitting in a header comment ("a gather, not a GEMM ... A quantized-gather op is a follow-up row") and **no such row exists**. The only non-expanding gather residency is `kKeepF16`, requiring ggml type 1 exactly (102.4 GB on disk) and CPU-ONLY, because `EmbeddingKernelCuda` refuses anything but f32/bf16. **Second blocker:** `moe_intermediate_size = 640` makes `ffn_down_exps` Q4_K-illegal on its reduction dim (640 % 256 = 128), as does `hc_lowrank = 320`; llama.cpp's substitution is believed to be Q5_0 (**UNVERIFIED, owed against the pinned llama.cpp oracle**) and the dependent fact IS verified in-tree — our reader knows ggml ids `0,1,2,8,10..14,16,18,19,22..28,30,39,40,41,66` and has **no entry for 3, 6, 7 or 20**, so a stock `llama-quantize -Q4_K_M` file fails at header parse. We author the converter, so the fix is Q4_0 (block 32, same 4.5 bpw). **`ENG-WEIGHT-OFFLOAD` will not help** — it moves zero bytes today (`ConsiderWeight` has no production callers, pinned by a test) and is documented inert on GB10; the tier that DOES work already ships and is proven by the 2.4T model serving 369.97 GiB from a 119.631 GiB box at ~62 GiB resident: mmap `MAP_PRIVATE`, borrow in place, alias the host pointer, `prefault: false`. Corrected sizing: backbone ~67.7 GiB, whole process ~73.5 GiB of 119.631 at 32K single-stream, ~46 GiB of headroom for the page cache, so the ~76 GB estimate was right within 10%. The design works because per-token demand is **<= 64 KiB of reads** (16 lookups x 160 dims over at most 16 pages) against the 2.4T expert lane's 6.95 GB/token. The architecture supplies its own lever: the per-token n-gram cost is `(ngram_size-1)*heads_per_ngram` = 16 lookups of 160 dims, so **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per token** and making it non-resident is the intended design point (RadixArk reached the same split independently). Sizing arithmetic, NOT measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB (yes, ~10 GB left for KV), Q4_K_M backbone with the table non-resident ~76 GB. GB10 is UNIFIED memory so "offload to host" is not a move there; non-resident means disk-backed, and its cost is unmeasured. **Two decisions were put to the developer as explicit accept-or-reject and BOTH are settled 2026-08-26, recorded in place rather than left open.** (1) `.agents/oracles/transformers.md` pins transformers to 5.14.1, deliberately tied to what the pinned vLLM environment resolves so the environment cannot hold two `transformers` at once, and **5.14.1 does not contain `Qwen4Exp`**; the lane-scoped second pin is **ACCEPTED**, on the argument that the invariant guards a vLLM environment against drifting from its transformers and here there is no vLLM implementation to drift from, and it expires the moment vLLM registers `qwen4_exp`. **The lane pin is a real release, not a branch SHA**, which was not the expected outcome: `Qwen4Exp` merged to `main` at 12:03:40Z on 2026-08-26 and `v5.16.0` published at 12:35:15Z, and this was BOUNDED rather than assumed by fetching `models/qwen4_exp/modeling_qwen4_exp.py` at each tag — `v5.16.0` HTTP **200**, `v5.15.0` HTTP **404** — making 5.16.0 the FIRST release carrying the architecture and therefore the tightest available pin. The version string is UNMEASURED (it is the release proven to contain the model, not a `transformers.__version__` read off a running oracle) and `gateable` stays `no`. (2) The first runnable arm is the **Q4_K_M backbone with the n-gram table NON-RESIDENT** (~76 GB). Q8_0 was raised and does not fit at ~191 GB, and no partial-Q8 split reaches 119 GB with the backbone at 8 bits; Q4_K_M-throughout fits on paper at ~109 GB but leaves ~10 GB for KV and activations on a 262144-native-context model, which is not a margin. This promotes the non-resident table from a note to a first-class W6 deliverable, and it is NOT free: GB10 is UNIFIED memory, so the existing host-pinned offload seam (`ENG-WEIGHT-OFFLOAD`, mirroring vLLM's `cpu_offload_gb`) does not by itself solve it there and the mechanism must be disk-backed or genuinely unloaded — established before it is designed around. Spec: [`specs/qwen4-exp-flash-next.md`](specs/qwen4-exp-flash-next.md). No product code lands under the spec pull request | feature | +| [#1998](https://github.com/mudler/vllm.cpp/issues/1998) | `MODEL-MM-GLM53-FLASH` | **`zai-org/GLM-5.3-Flash` declares `Glm5NextForConditionalGeneration` / `glm5_next`, an architecture vLLM implements at NO revision, whose only admissible oracle is transformers and which NOTHING on this fleet can execute.** Read live 2026-08-26: `git grep "Glm5\|glm5_next"` returns ZERO hits at the parity pin `555967922` AND at vLLM `origin/main` = `c71f6f8a81` (2026-08-26 08:42 -0700), whose `vllm/models/` package holds `common, deepseek_v32, deepseek_v4, dots3_note, inkling, kimi_k3, minimax_m3` and no `glm5next`. That is ABSENCE from vLLM `main`, not staleness in our pin, so a pin advance — 348 commits of reconciliation — reaches nothing here. [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) "[Model] add GLM-5.3-Flash support" (opened 2026-08-26T14:12:00Z, head `933876c388fb`, 85 files, +12,511/-540) would register `Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` and `Glm5NextMTPModel`, and is **OPEN, unmerged, `mergeable: false`** — an unmerged PR is not a revision and is INADMISSIBLE as an oracle, the same rule this index already applied to SGLang #36497 on the qwen4-exp row; it is cited only as evidence of upstream shape. SGLang implements nothing either (PR #36507 open; code search 0) and the trap there is that its **cookbook** PRs #36440 and #36513 MERGED at 14:00:16Z and 14:39:29Z — documentation landed ahead of the code, so the model card links a recipe SGLang `main` cannot run. llama.cpp implements nothing (code search 0; PR #27752 open; our oracle pinned at `b10451`), and vllm-omni implements nothing. **The sole admissible reference is transformers**, implementing commit `eb4d9e2a64` (2026-08-26T14:26:40Z, transformers#48342), and the first RELEASE carrying it is **`v5.16.1`** — bounded rather than assumed by fetching `models/glm5_next/modeling_glm5_next.py` at each tag: HTTP **200** at `v5.16.1`, **404** at `v5.16.0`, **404** at `v5.15.1`. Our transformers pin is 5.14.1, so this row needs a lane-scoped second pin expiring when vLLM registers `glm5_next`; note it is `5.16.1` where the `MODEL-MM-QWEN4-EXP` lane is `5.16.0`, one day apart because `Qwen4Exp` merged before the 5.16.0 cut and `Glm5Next` after it — two lanes, two releases, which is what a lane pin is for. **Not the blocked GLM-5 row:** `MODEL-TEXT-deepseek-v2-glm-moe-dsa-for-causal-lm` (`GlmMoeDsaForCausalLM`, `BLOCKED`, `CLAIM-GLM-DSA-LATEST-DEEPSEEK`) is DeepSeek-V3.2 verbatim at 753.9B; `glm5_next` is a different `model_type`, a different parameter count, a different attention topology and has a vision tower. **The architecture:** 45 layers, hidden 4096, vocab 154880, 1,048,576 context, `layer_types` = 34 `linear_attention` (KDA) + 11 `deepseek_sparse_attention` at 3,7,…,43; MoE 288 routed + 1 shared, top-8, `moe_intermediate_size` 2048, sigmoid/`noaux_tc`, `n_group=topk_group=1` (which makes the group stage a NO-OP), `routed_scaling_factor` 2.5, clamped SwiGLU `swiglu_limit` 10.0 in FIVE places; mHC `hc_mult` 4 / `hc_sinkhorn_iters` 20 / `hc_eps` 1e-6; vision 24L/1024/patch **14**/merge 2/temporal 2/out 4096. **The text stack has NO rotary anywhere.** `qk_rope_head_dim: 0`, `mla_use_nope: true`, no `rope_theta`, no `rope_scaling`; the reference *deletes* the inherited `rope_parameters` field and passes `position_embeddings=None` to every layer, and **`indexer_rope_interleave: true` is a VESTIGIAL flag** the indexer override ignores — implementing it because the config names one is implementing a bug. Position comes only from the KDA decay and short convs. **HEAVY REUSE — and the two obvious reuses are WRONG in one place each, in ways a token gate cannot see.** (1) The KDA forget gate takes the OTHER branch: `gate_lower_bound: -5.0` is not None, so it computes `-5.0 * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias))`, whereas our `src/vllm/model_executor/models/kimi_kda.cpp:60` implements Kimi-Linear's `-exp(A_log) * softplus(g + dt_bias)`. Different functions of the same inputs, both smooth, both negative, both fluent — and the sign of `decay_rate` differs too, so getting it wrong flips decay to growth. (2) The mHC head collapse is `hidden_streams.mean(dim=2)`, an UNWEIGHTED mean whose own docstring says "Unlike DeepSeek-V4", where our `src/vllm/model_executor/models/deepseek_v4_mhc.cpp:168` `HcHeadCollapse` is the V4 weighted-gated collapse; the checkpoint carries **no `hc_head.*` tensors at all**, which settles it. **(3) The k-pool indexer is unobservable at or below 2048 tokens:** with `index_topk: 2048` a shorter context selects every candidate, the selection is the identity, the pooling is invisible and any short-prompt gate passes over a broken indexer — the same shape as the trap recorded on the qwen4-exp row. The pooling is **learned, per channel, not a mean**: 128 independent 4-way softmaxes over a learned `[4,128]` intra-pool position embedding plus a cached `[128,4096]` gate score, pools aligned to the first VALID token (so the grid is batch-dependent), a pool must be COMPLETE to be a candidate and is visible iff its LAST member is, `select_k = min(index_topk/index_kpool, P) = min(512, P)`, expanded back to raw token indices and widened to `index_topk + kpool - 1` = **2051** by an unscored ragged tail. Its cache is 257 floats/token/layer, not the parent's 128. **Structural blocker:** `MlaBlockDims::Validate` REFUSES `qk_rope_head_dim == 0` (`src/vllm/model_executor/layers/attention/mla_attention.cpp:90-93` requires every dim > 0, `:95-99` requires it even); Kimi-Linear is NOT the same case — it sets `mla_use_nope` while keeping `qk_rope_head_dim: 64`, so the rope slice still occupies the cache row (`include/vllm/model_executor/models/kimi_linear.h:86,:88`). NoPE does buy one thing: MLA weight absorption becomes trivially valid, since there is no rope slice to keep out of it. **NOTHING FITS, and no GGUF exists.** Measured live by HTTP RANGE over all 62 safetensors headers, payload never fetched: 76,108 tensors, `F8_E4M3` 314,396,639,232 + `BF16` 6,926,096,640 elements = **321,322,735,872 real parameters** plus 19,484,766 F32 block scales; routed experts are **311.65B = 97.0%**, so any GGUF arm's mixed bpw is the experts' bpw to within a percent. On disk against ~119.63 GiB on GB10: `zai-org/GLM-5.3-Flash` FP8 e4m3 block-128x128 **305.78 GiB (2.56x over)**, `zai-org/GLM-5.3-Flash-BF16` **598.53 GiB (5.00x)**, `unsloth/GLM-5.3-Flash-FP8` 305.82 GiB, `LibertAIDAI/GLM-5.3-Flash-NVFP4` **181.32 GiB (1.52x)**. **All four repos named `*-GGUF` contain ZERO `.gguf` files** (`unsloth`, `AtomicChat`, `aj9o9`, `vcruz305` — READMEs and images); a repository name is not an artifact. **`gateable = no` on MEMORY, and the consequence is the scoping result: no oracle can execute this model on any device this project reaches, so NO end-to-end token gate exists or can exist here** — recorded as visible debt, not waived. What replaces it is a tiny-shape CPU reference oracle instantiated from the pinned transformers, which is a real oracle for the NUMERICS and is explicitly NOT oracle gateability for the model. Developer-set W7 target is a low GGUF quant that fits the DGX; arithmetic (not measurement) with experts at the named type and the other 3% at Q6_K, bpw taken from our own reader's block traits at `src/vllm/model_executor/model_loader/gguf_reader.cpp:200`: Q8_0 318.0 GiB, Q6_K 245.5, Q4_K 170.6, IQ3_XXS 118.1 (~1.5 GiB left — not a margin), **Q2_K 102.6 (~17 GiB left)**, IQ2_S 100.3, IQ2_XXS 82.3, IQ1_S 64.0; KV is small by design at ~11.7 KiB/token (1.5 GiB at 128K, ~11.4 GiB at the full 1M) though #1963/#1966 mean that arithmetic is re-derived against the runner, not trusted. **Q2_K is the recommended first arm and the reason is producibility, not quality: i-quants need an importance matrix, an importance matrix needs a forward pass, and a forward pass needs 181 GiB — the dependency is CIRCULAR on this fleet.** K-quants need none. And since llama.cpp has no `glm5_next`, the converter is ours to author, with no reference GGUF to diff against and no llama.cpp floor for the arm. Layer 45 is the MTP block — 888 tensors, 7.43B params, 2.31% of the model, DSA/MLA rather than KDA and carrying no `hc_*` tensors — present in the checkpoint and discarded by the reference at `modular_glm5_next.py:1235`; owed, not implemented, and skipping it in the converter is ~2.3 GiB of headroom. Live seam contention to rebase onto rather than fork: [#1971](https://github.com/mudler/vllm.cpp/pull/1971) (DSA geometry) and [#1977](https://github.com/mudler/vllm.cpp/pull/1977) (DSv4 KV multicache). Spec: [`specs/glm5-next-flash.md`](specs/glm5-next-flash.md), 8 waves, W0-W2 and W4 CPU-gateable and W3/W5-W8 needing a GPU. No product code lands under the spec pull request | feature | diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 6f955c418..37fd1d334 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -81,14 +81,15 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): | SPIKE | 9 | | BLOCKED | 5 | | DONE | 3 | -| READY | 4 | +| READY | 5 | | GATING | 1 | -| **Total** | **378** | +| **Total** | **379** | -Engaged architectures (the 54 non-`INVENTORIED` rows): +Engaged architectures (the 55 non-`INVENTORIED` rows): | Support | Architecture | Family / example | Status | Row | |---|---|---|---|---| +| 🚧 | `Glm5NextForConditionalGeneration` | GLM-5.3-Flash (321.32B total / ~18B active, natively multimodal; 34 KDA linear-attention + 11 DeepSeek-sparse MLA layers, mHC residual manifold, 288+1 expert MoE, 24-layer ViT) | **SPEC ONLY.** New architecture vLLM implements at NO revision -- absent from the pin `555967922` AND from `main` `c71f6f8a81`, so a pin advance does not reach it; vllm#53906 is OPEN and inadmissible. Sole admissible oracle is transformers, lane-pinned `v5.16.1` (the first release carrying `glm5_next`; `v5.16.0` is 404). `gateable = no` on MEMORY: the smallest published artifact is NVFP4 at 181.32 GiB against ~119.63 GiB on GB10, so no oracle can run this model on any device here and NO end-to-end token gate is reachable. Zero `.gguf` files exist in any of the four `*-GGUF` repos and llama.cpp has no `glm5_next`, so the converter is ours to author. Spec plans 8 waves; no product code has landed | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | | 🚧 | `Qwen4ExpForConditionalGeneration` | Qwen3.8-Flash-Next (180B total / 6B activated, image-text-to-text) | **SPEC ONLY.** New architecture vLLM does NOT implement at any revision; split oracle by developer direction 2026-08-26 (transformers for the algorithm, vLLM ops for the optimized path). Nothing published fits ~119 GB on GB10, so `gateable = no` and the quantized arms are load-bearing; no GGUF exists and llama.cpp has no `qwen4_exp` either. NO token, NO speed. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | | ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling scope CLOSED (#323): `60e71a0e` fixed the eager path; `DenseDecodeGraphForward` ran first and replayed against stale HOST ids, so it now declines while the mirror is live and falls back to the proven eager path. Async gate 7/7 across Qwen3-0.6B/4B + Llama/Mistral/InternLM2 | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` | | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` | @@ -523,6 +524,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-cohere-asr-cohere-asr-for-conditional-generation` | `CohereAsrForConditionalGeneration` | `registry.py:571-574`; `vllm/model_executor/models/cohere_asr.py::CohereAsrForConditionalGeneration` | conditional generation / audio+image | MM processor; encoder/merge; cross-attention; audio/ASR frontend; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-nemotron-parse-nemotron-parse-for-conditional-generation` | `NemotronParseForConditionalGeneration` | `registry.py:575-578`; `vllm/model_executor/models/nemotron_parse.py::NemotronParseForConditionalGeneration` | conditional generation / audio+image | MM processor; encoder/merge; cross-attention; audio/ASR frontend; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-whisper-whisper-for-conditional-generation` | `WhisperForConditionalGeneration` | `registry.py:579`; `vllm/model_executor/models/whisper.py::WhisperForConditionalGeneration` | conditional generation / audio+image | MM processor; encoder/merge; sliding-window attention; cross-attention; audio/ASR frontend | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | `Glm5NextForConditionalGeneration` (`model_type: glm5_next`; campaign row `MODEL-MM-GLM53-FLASH`) | **NOT IN vLLM AT ANY REVISION** -- deliberately written with no pinned module/class target, the same convention `MODEL-MM-qwen4-exp-*` follows, so the at-the-pin static invariants are untouched. Read live 2026-08-26: `git grep "Glm5\|glm5_next"` returns ZERO hits at the pin `555967922` AND at `origin/main` `c71f6f8a81` (2026-08-26 08:42 -0700), whose `vllm/models/` package holds `common, deepseek_v32, deepseek_v4, dots3_note, inkling, kimi_k3, minimax_m3` and no `glm5next`. That is ABSENCE, not pin staleness, so advancing the parity pin -- 348 commits of reconciliation -- reaches nothing here. [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) (opened 2026-08-26T14:12:00Z, head `933876c388fb`, 85 files, +12,511/-540) would register `Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` and `Glm5NextMTPModel`, and is **OPEN, unmerged, `mergeable: false`** -- an unmerged PR is not a revision and is inadmissible as an oracle, the same rule applied to SGLang #36497 on the qwen4-exp row; it is cited only as evidence of upstream shape. SGLang (PR #36507 open, cookbook #36440/#36513 MERGED ahead of the code), llama.cpp (PR #27752 open, our pin `b10451`) and vllm-omni likewise implement nothing. **ONE row and not three:** none of the three names is registered at any vLLM revision, and the only architecture a published artifact declares is `Glm5NextForConditionalGeneration`. | conditional generation / image + video | model loader/forward; GDN/linear-attention state (KDA); MLA/latent KV; DSA sparse indexer; FusedMoE/grouped GEMM; mHC residual manifold; MM processor + ViT encoder/merge; FP8 block-wise quant; GGUF k-quant | [glm5-next-flash](specs/glm5-next-flash.md) | `READY` | **SCOPING ONLY, 2026-08-26 (issue [#1998](https://github.com/mudler/vllm.cpp/issues/1998)): the spec and its records are the whole change; NO product code has landed.** Architecture read from `config.json` live: 45 layers, hidden 4096, vocab 154880, 1,048,576 context; `layer_types` = 34 `linear_attention` + 11 `deepseek_sparse_attention` (at 3,7,...,43); MLA is **fully NoPE** (`qk_rope_head_dim: 0`, `mla_use_nope: true`, `qk_nope_head_dim: 256`, `v_head_dim: 256`, `kv_lora_rank: 512`, `q_lora_rank: 1536`) and `text_config` carries NO `rope_theta` and NO `rope_scaling` -- the transformers reference *deletes* the inherited `rope_parameters` field and passes `position_embeddings=None` to every layer, and `indexer_rope_interleave: true` is a VESTIGIAL flag the indexer override ignores, so the text stack has no rotary anywhere. DSA indexer adds a **learned per-channel k-pool** (`index_kpool: 4`, `index_kpool_compress`, `index_kpool_always_select_tail`): 128 independent 4-way softmaxes over a learned `[4,128]` intra-pool position embedding, pools aligned to the first VALID token (batch-dependent), `select_k = min(index_topk/index_kpool, P) = min(512, P)`, expanded back to raw indices and widened to `index_topk + kpool - 1` = **2051** by an unscored ragged tail. mHC `hc_mult: 4`, `hc_sinkhorn_iters: 20`, `hc_eps: 1e-6`. MoE 288 routed + 1 shared, top-8, `moe_intermediate_size` 2048, sigmoid/`noaux_tc`, `n_group=topk_group=1`, `routed_scaling_factor` 2.5, clamped SwiGLU `swiglu_limit` 10.0 in FIVE places. Vision 24L/1024/patch **14**/merge 2/temporal 2/out 4096. **HEAVY REUSE, and the two obvious reuses are WRONG in one place each.** (1) The KDA forget gate takes the OTHER branch: `gate_lower_bound: -5.0` is not None, so it computes `-5.0 * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias))`, where our `src/vllm/model_executor/models/kimi_kda.cpp:60` implements Kimi-Linear's `-exp(A_log) * softplus(g + dt_bias)` -- different functions of the same inputs, both fluent, and the sign of `decay_rate` differs too. (2) The mHC head collapse is `hidden_streams.mean(dim=2)`, an UNWEIGHTED mean whose own docstring says "Unlike DeepSeek-V4", where our `deepseek_v4_mhc.cpp:168` `HcHeadCollapse` is the V4 weighted-gated collapse; the checkpoint carries no `hc_head.*` tensors at all, which settles it. **Third trap: the k-pool selection is unobservable at or below 2048 tokens** -- with `index_topk: 2048` a shorter context selects every candidate, so any short-prompt gate passes over a broken indexer. **Structural blocker:** `MlaBlockDims::Validate` REFUSES `qk_rope_head_dim == 0` (`src/vllm/model_executor/layers/attention/mla_attention.cpp:90-93` requires every dim > 0, `:95-99` requires it even); Kimi-Linear is NOT the same case, it sets `mla_use_nope` while keeping `qk_rope_head_dim: 64` so the rope slice still occupies the cache row (`include/vllm/model_executor/models/kimi_linear.h:86,:88`). **HW: NOTHING FITS.** Measured live by HTTP RANGE over all 62 safetensors headers (no payload fetched): 76,108 tensors, `F8_E4M3` 314,396,639,232 + `BF16` 6,926,096,640 elements = **321,322,735,872 real parameters**, routed experts **311.65B = 97.0%**; on disk `zai-org/GLM-5.3-Flash` FP8 305.78 GiB (2.56x GB10), `-BF16` 598.53 GiB (5.00x), `LibertAIDAI/...-NVFP4` 181.32 GiB (1.52x). **All four `*-GGUF` repos contain ZERO `.gguf` files.** `gateable = no` on MEMORY: no oracle can execute this model on any device this project reaches, so **no end-to-end token gate exists or can exist here** -- recorded as visible debt, replaced by a tiny-shape CPU reference oracle instantiated from the pinned transformers, which is an oracle for the NUMERICS and is NOT oracle gateability for the model. Layer 45 is the MTP block (888 tensors, 7.43B params, 2.31%), present in the checkpoint and discarded by the reference; owed, not implemented. | `-` | ## MODEL-AUDIO - Audio encoder components (not vLLM registry architectures) diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md new file mode 100644 index 000000000..a97a6d268 --- /dev/null +++ b/.agents/specs/glm5-next-flash.md @@ -0,0 +1,1221 @@ +# `Glm5NextForConditionalGeneration` (GLM-5.3-Flash) + +**SCOPING ONLY. NO PRODUCT CODE LANDS UNDER THIS PULL REQUEST.** This document +and its records are the whole deliverable of the pull request that introduces +it. Implementation follows in separate `row/MODEL-MM-GLM53-FLASH-W` branches +dispatched from the committed spec, per the split-pull-request case in AGENTS.md +§"Spec before code". + +- **Matrix row:** `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` + (`.agents/model-matrix.md`, §MODEL-MM). +- **Campaign row:** `MODEL-MM-GLM53-FLASH`. +- **Issue:** [#1998](https://github.com/mudler/vllm.cpp/issues/1998). +- **State on landing:** `READY` — the spec is committed and no product code has + landed. It may not become `ACTIVE` until wave 1 is claimed. +- **Model card:** , read live + 2026-08-26. + +Every fact below carries the date it was read. Nothing here is measured on +hardware: no lease was taken, no GPU ran, and no weights were downloaded. Where +a number is arithmetic over measured file sizes it says so, and where a +quantity is unknown it says "not measured" rather than being estimated. + +## Scope + +Port `Glm5NextForConditionalGeneration` — GLM-5.3-Flash, 321.32B total +parameters, ~18B activated, natively multimodal (text + image + video) — so that +it loads and generates through `ModelRegistry::Forward` and the `include/vllm.h` +ABI, with the quantized arms AGENTS.md makes a standing requirement. + +IN scope for the campaign this spec plans: + +- The text backbone: a 45-layer hybrid of 34 KDA linear-attention layers and 11 + DeepSeek-Sparse-Attention (DSA) MLA layers, with a manifold hyper-connection + (mHC) residual topology, a 288-expert + 1-shared fine-grained MoE, and a + clamped SwiGLU. +- The vision tower (24-layer ViT, patch 14, spatial merge 2, temporal patch 2) + and the image/video placeholder expansion. +- Config resolution, architecture registration, weight-name mapping, and a + forward that refuses by name until each primitive exists. +- The GGUF k-quant arm, including the converter that has to be authored because + no upstream tool can emit this architecture. + +OUT of scope, explicitly: + +- Advancing the vLLM parity pin. `555967922` does not reach this architecture + and neither does vLLM `main`; see §Oracles. Nothing in this campaign may move + `.agents/upstream-sync.md`. +- The MTP speculative head (`num_nextn_predict_layers = 1`). It is recorded + under §Owed and gets its own row when the backbone runs. +- Any claim of speed parity. There is no denominator: no oracle runs this model + on any device this project can reach. +- Reworking the shared MLA block for models other than this one. Where the + shared seam cannot represent the geometry, §Design says which seam is + extended and under which wave. + +## Why this needs a spec before code + +Three reasons, each of which has already cost this repository once. + +**The name collides with a different, blocked model.** +`.agents/model-matrix.md` already carries a row labelled "GLM-5": +`MODEL-TEXT-deepseek-v2-glm-moe-dsa-for-causal-lm` (`GlmMoeDsaForCausalLM`), +state `BLOCKED`, owned by `CLAIM-GLM-DSA-LATEST-DEEPSEEK`. That row is +DeepSeek-V3.2 verbatim (`deepseek_v2.py:1917-1918` at the pin: `class +GlmMoeDsaForCausalLM(DeepseekV2ForCausalLM): pass`) and is blocked at 753.9B / +1404 GiB. `Glm5NextForConditionalGeneration` is a **different architecture with +its own `model_type` (`glm5_next`)**, a different parameter count, a different +attention topology and a vision tower. It is a new row, not a revival of the +blocked one, and this paragraph exists so the next reader does not merge them. + +**The obvious reuse is the wrong reuse in one specific place.** Three of the +four exotic families this model needs already exist in this tree — mHC, the DSA +lightning indexer, and KDA. An implementer who sees that will reach for them +directly. Two of the three have a documented delta that a token gate cannot +see, and §Port map names them: the KDA forget gate takes a **different branch** +from the one our Kimi reference ports, and the DSA indexer adds a **k-pool +compression stage** our DeepSeek-V4 indexer does not have. + +**Nothing can gate it.** No oracle implements this architecture at a pinned +revision, and no artifact of it fits any device on this fleet. A campaign that +does not decide up front what "correct" means here will produce a model that +generates plausible text and is never checked. §Gates decides it. + +## Oracles + +Everything in this section was read live on **2026-08-26**. + +### vLLM at the parity pin `555967922` — implements NOTHING + +`git grep -n "Glm5\|glm5_next" 555967922 -- vllm/` returns zero hits. The +registry at the pin carries `GlmForCausalLM`, `Glm4ForCausalLM`, +`Glm4MoeForCausalLM`, `Glm4MoeLiteForCausalLM`, `GlmMoeDsaForCausalLM`, +`Glm4vForConditionalGeneration`, `Glm4vMoeForConditionalGeneration`, +`GlmOcrForConditionalGeneration`, `GlmAsrForConditionalGeneration` and three +GLM MTP heads — `vllm/model_executor/models/registry.py:113-117,409-413,640-642` +@ `555967922`. There is no `Glm5*` entry of any kind. + +### vLLM `main` — implements NOTHING EITHER + +Fetched to `origin/main` = `c71f6f8a81d3d3c49a045c8b88eed36366cc7d92` +(2026-08-26 08:42:52 -0700). `git grep -n "Glm5\|glm5_next" origin/main -- +vllm/` returns zero hits. The `vllm/models/` package at `main` holds `common`, +`deepseek_v32`, `deepseek_v4`, `dots3_note`, `inkling`, `kimi_k3` and +`minimax_m3` — no `glm5next`. The GLM entries in +`registry.py:114-118,401-405,655-657` @ `c71f6f8a81` are the same nine plus +`GlmMoeDsaForCausalLM` re-homed to `vllm.models.deepseek_v32`. + +**This is absence, not pin staleness.** Advancing the parity pin does not reach +this architecture, so this campaign carries no pin-advance dependency and must +not create one. What advancing the pin *would* cost, stated so the option is +priced rather than left vague: 348 commits between `555967922` and +`c71f6f8a81`, each of which AGENTS.md §"Pin vLLM" requires be reconciled +against every affected row and gate, and the last such cycle +(`.agents/specs/pin-advance.md`) re-validated the binding golden grids on GB10. +That cost buys nothing here, because the architecture is absent at both ends. + +### vLLM PR #53906 — OPEN, and therefore INADMISSIBLE + +[vllm#53906](https://github.com/vllm-project/vllm/pull/53906) "[Model] add +GLM-5.3-Flash support" was opened 2026-08-26T14:12:00Z, is **open, unmerged, and +`mergeable: false`**, one commit at head `933876c388fb129ad82590660e6506614559cb86` +over base `903a02192fca19e4c89705af7017c0f24971ea4f`, 85 files, +12,511/-540. +It would register three architectures: +`Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` and +`Glm5NextMTPModel`, all against a new `vllm/models/glm5next` package +(`vllm/model_executor/models/registry.py` hunks at `+119`, `+425`, `+672` in the +PR diff). + +An unmerged pull request is not a revision and cannot be pinned, so it is not an +oracle. This repository already recorded that rule on the `MODEL-MM-QWEN4-EXP` +row for SGLang #36497 and it applies here unchanged. **The PR is cited in this +spec only as evidence of upstream shape** — it tells us which components +upstream considers new enough to need their own file, and that is planning +information, not a port source. No `file:line` from it may appear in a port map +cell. When it merges, §Stop conditions says what happens. + +What it tells us about shape, and nothing more: upstream needed a new +`sparse_attn_indexer_kpool.py` (+1009), a `glm5next/nvidia/ops/kpool_compress.py` +(+948), a `glm5next/nvidia/kda.py` (+628), a `glm5next/nvidia/attention.py` +(+599), a `fused_eh_norm.py` (+77), and edits to `layers/mhc.py`, +`layers/mla.py` and `third_party/flash_linear_attention/ops/kda.py` (+93/-55). +That is the same four-family split this spec independently derives from the +transformers reference, which raises confidence in the decomposition without +being usable as its source. + +### transformers — the ONLY admissible reference, and it needs a lane pin + +`glm5_next` exists in `huggingface/transformers`. The implementing commit is +`eb4d9e2a64` (2026-08-26T14:26:40Z, PR +[transformers#48342](https://github.com/huggingface/transformers/pull/48342) +"[Glm 5.3 Flash] GLM 5.3 Flash Support"). The package is +`src/transformers/models/glm5_next/`: `modular_glm5_next.py` (95,314 bytes, the +source of truth), `modeling_glm5_next.py` (102,403), `configuration_glm5_next.py` +(14,202), `processing_glm5_next.py` (9,598), `image_processing_glm5_next.py`, +`image_processing_pil_glm5_next.py`, `video_processing_glm5_next.py`. + +**The first RELEASE carrying it is `v5.16.1`** (published 2026-08-26T14:50:01Z). +This was bounded rather than assumed, by fetching +`src/transformers/models/glm5_next/modeling_glm5_next.py` at each tag: + +| tag | published | `modeling_glm5_next.py` | +|---|---|---| +| `v5.15.1` | 2026-08-19T10:50:47Z | HTTP **404** | +| `v5.16.0` | 2026-08-26T12:35:15Z | HTTP **404** | +| `v5.16.1` | 2026-08-26T14:50:01Z | HTTP **200** | + +`.agents/oracles/transformers.md` pins transformers to **5.14.1**, deliberately +tied to what the pinned vLLM environment resolves so that environment cannot +hold two `transformers` at once. 5.14.1 does not contain `glm5_next`. This row +therefore needs a **lane-scoped second pin at `v5.16.1`**, on exactly the +argument the `MODEL-MM-QWEN4-EXP` row records and had accepted: the invariant +guards a *vLLM environment* against drifting from its transformers, and here +there is no vLLM implementation to drift from. It expires the moment vLLM +registers `glm5_next`. + +**Note for whoever writes that oracle file: the `MODEL-MM-QWEN4-EXP` lane pins +`5.16.0` and this lane pins `5.16.1`.** Two lanes, two different releases, one +day apart, because `Qwen4Exp` merged before the 5.16.0 cut and `Glm5Next` after +it. That is not a contradiction to reconcile; it is why a lane pin is written as +a lane pin. Do not "tidy" one onto the other. + +### llama.cpp — implements NOTHING + +Code search `glm5_next repo:ggml-org/llama.cpp` returns **0**. PR +[ggml-org/llama.cpp#27752](https://github.com/ggml-org/llama.cpp/pull/27752) +"model : add GLM-5.3-Flash (glm5next)" is **open**. Our llama.cpp oracle is +pinned at tag `b10451` (`.agents/oracles/llama-cpp.md`), which predates the +model entirely. **Consequence: `convert_hf_to_gguf.py` cannot emit this +architecture, `llama-quantize` cannot quantize it, and the GGUF arms of this +model have no llama.cpp oracle and no llama.cpp floor.** §Hardware says what +that forces. + +### SGLang — implements NOTHING + +`glm5_next repo:sgl-project/sglang` code search returns **0**, and +`python/sglang/srt/models/` at `main` carries `glm4*`, `glm_ocr*`, `glm_image_vl`, +`glmasr`, `chatglm` and no `glm5`. PR +[sgl-project/sglang#36507](https://github.com/sgl-project/sglang/pull/36507) +"GLM-5.3-Flash support" is **open**. The GLM-5.3-Flash *cookbook* PRs #36440 and +#36513 **did** merge (2026-08-26T14:00:16Z, 14:39:29Z) — documentation landed +ahead of the implementation, which is a trap: the model card links a cookbook +that describes a code path SGLang `main` does not yet contain. SGLang is +inadmissible here. + +### vLLM-Omni — implements NOTHING + +`glm5 repo:vllm-project/vllm-omni` code search returns **0**. This is not an +omni-only architecture and there is no reason to reach for that repository. + +### Gateability + +`gateable = no` for every oracle, on **memory**, and this is the load-bearing +verdict of the whole scoping exercise. AGENTS.md admits `gateable = yes` only +after an oracle demonstrably builds and runs the model. The only admissible +oracle is transformers `v5.16.1`, and running it needs the FP8 checkpoint +(305.78 GiB) or the BF16 one (598.5 GiB) resident. The largest device this +project can reach is `dgx:gpu0` at ~119.63 GiB of unified memory. **There is no +device on this fleet, and no combination of them, on which the reference +implementation of this model can be executed.** No token-exact end-to-end gate +against an oracle is reachable, now or after any amount of implementation work. + +That is a statement about the full model. It is NOT a statement that this +campaign is ungateable; §Gates constructs the gate that is actually reachable. + +## Upstream chain + +`transformers` `v5.16.1`, `src/transformers/models/glm5_next/`. The modular file +is the source of truth and every port-map cell below cites +`modular_glm5_next.py:` at that tag. The expanded +`modeling_glm5_next.py` is generated from it and is the file to read when the +modular inheritance is ambiguous. + +The inheritance graph, which is most of the port plan (`modular_glm5_next.py` +@ `v5.16.1`): + +| class | line | inherits | +|---|---|---| +| `Glm5NextTextConfig` | `:92` | `GlmMoeDsaConfig` | +| `Glm5NextVisionConfig` | `:248` | `GlmOcrVisionConfig` | +| `Glm5NextConfig` | `:265` | `PreTrainedConfig` | +| `Glm5NextTextMLP` | `:321` | `Qwen2MoeMLP` (+ `swiglu_limit` clamp) | +| `Glm5NextTextExperts` | `:335` | `MiniMaxM3VLExperts` | +| `Glm5NextTextTopkRouter` | `:350` | `DeepseekV3TopkRouter` (`pass`) | +| `Glm5NextTextMoE` | `:354` | `DeepseekV3MoE` | +| `Glm5NextTextHyperConnection` | `:364` | `DeepseekV4HyperConnection` (`pass`) | +| `Glm5NextTextHyperHead` | `:368` | `nn.Module` — **unweighted mean** | +| `Glm5NextTextForgetGate` | `:375` | `nn.Module` — **NEW branch** | +| `Glm5NextTextRMSNormGated` | `:409` | `Qwen3_5RMSNormGated`, activation `sigmoid` | +| `l2norm` | `:429` | free function, `sqrt(sum + eps)` not `max(.., eps)` | +| `recurrent_kimi_delta_attention` | `:441` | free function (decode) | +| `chunk_kimi_delta_attention` | `:495` | free function (prefill) | +| `Glm5NextTextLinearAttention` | `:597` | `nn.Module` (KDA arm) | +| `Glm5NextTextIndexer` | `:749` | `GlmMoeDsaIndexer` (+ k-pool) | +| `Glm5NextTextAttention` | `:1025` | `GlmMoeDsaAttention` (MLA, NoPE) | +| `Glm5NextTextDecoderLayer` | `:1142` | `GlmMoeDsaDecoderLayer` | +| `Glm5NextTextModel` | `:1285` | `Glm5NextPreTrainedModel` | +| `Glm5NextVisionMLP` / `PatchMerger` / `Block` / `Model` | `:1373`/`:1387`/`:1403`/`:1410` | `GlmOcrVision*` | +| `Glm5NextModel` | `:1422` | `Exaone4_5_Model` | +| `Glm5NextForConditionalGeneration` | `:1563` | `Glm46VForConditionalGeneration` | +| `Glm5NextProcessor` | `:1704` | `Glm46VProcessor` | +| `Glm5NextImageProcessor` | `:1792` | `GlmgaImageProcessor` | +| `Glm5NextVideoProcessor` | `:2121` | `GlmgaVideoProcessor` | + +Read that table as: **the model is assembled almost entirely from families this +tree has already engaged**, and the work is in the handful of classes that are +not a `pass`. + +## Our baseline + +What this tree already has, with the maturity stated rather than implied. Every +anchor below was read at `21fe11cf1`. + +### DSA lightning indexer — host reference plus host-vector CUDA, no fused pipeline + +- CPU reference: `src/vllm/model_executor/models/deepseek_v4_dsa.cpp:14` + (`DsaIndexerWeightFold`), `:31` (`DsaIndexerLogits`), `:72` (`DsaTopkSelect`), + `:116` (`SoftmaxWithSink`), `:136` (`GroupedOutputLora`). +- CUDA kernels: `src/vt/cuda/cuda_deepseek_v4.cu:592`, `:598`, `:624`, `:705`, + `:722`; registered `:2094`. +- **Maturity caveat:** the device entry points take and return host + `std::vector` (`include/vllm/model_executor/models/deepseek_v4_device.h:82-102`), + so each call is upload → launch → download → sync. The one in-place device + kernel is `decode_attn` (`deepseek_v4_device.h:112`), called from + `src/vllm/model_executor/models/deepseek_v4.cpp:853`. The V4 forward is + DERIVED and structurally gated at a synthetic shape, not real-checkpoint + token-gated — its own header says so at + `src/vllm/model_executor/models/deepseek_v4.cpp:1-17`. + +### MLA — present, and it REFUSES this model's geometry + +`MlaBlockDims` is `include/vllm/model_executor/models/mla_attention.h:108-166`; +`Validate()` is `src/vllm/model_executor/layers/attention/mla_attention.cpp:89`. + +- `qk_rope_head_dim == 0` is **refused**: `mla_attention.cpp:90-93` requires + every dimension `> 0`, and `:95-99` additionally requires it even. GLM-5.3-Flash + sets `qk_rope_head_dim: 0`. This is the single hardest structural item in the + port. +- Kimi-Linear is the closest analogue and is **not the same thing**: it sets + `mla_use_nope = true` (`include/vllm/model_executor/models/kimi_linear.h:88`) + while keeping `qk_rope_head_dim = 64` (`:86`) — the rope slice still exists in + the cache row, only the rotation is skipped. Here the slice does not exist and + `head_size()` (`mla_attention.h:162`) becomes `512`, not `576`. +- `v_head_dim <= qk_head_dim()` (`mla_attention.cpp:100-104`) passes at 256/256 + once the rope check is relaxed. +- `q_lora_rank = 1536` is the supported DeepSeek-V3 branch + (`mla_attention.h:117`, `:164`); the loader must fuse `q_a_proj` + + `kv_a_proj_with_mqa` (`mla_attention.h:238-244`). +- Prefill FA-2 compiles head dims `{128, 192, 256}` + (`src/vt/cuda/cuda_mla_prefill.cu:194-208`), refusing `> 256` by name at + `:208`; `qk_head_dim == 256` is already live (GLM-4.7-Flash). Decode reads + `head_size` dynamically (`src/vt/cuda/cuda_mla_attn.cu:487`) and branches on + `v_head_dim <= 512` at `:564`, with a shared-memory guard at `:545-546`; a + 512/256 pair should fit and is **untested**. + +### mHC — host and CUDA, configurable, DeepSeek-V4-exclusive + +`src/vllm/model_executor/models/deepseek_v4_mhc.cpp:23` (`MhcSinkhorn`), `:72` +(`MhcPre`), `:149` (`MhcPost`), `:168` (`HcHeadCollapse`); all take `int64_t hc` +as a runtime argument. CUDA at `src/vt/cuda/cuda_deepseek_v4.cu:143`, `:208`, +`:270`, `:285`, `:534`, `:552`; registered `:2092`. Reached only from +`src/vllm/model_executor/models/deepseek_v4.cpp:255,269,284,287,298,301,1815`. + +### KDA — the most mature of the three, and default-OFF on device + +Kimi-Linear-48B-A3B runs end to end with a real gate (`docs/FEATURES.md:151`: +engine==CLI 128/128 byte-identical, 122/128 vs golden, ~19.0 tok/s vs vLLM ~21). + +- `vt::KdaGatedDeltaRule`: `include/vt/ops.h:105`, `:2867`; CUDA + `src/vt/cuda/cuda_gdn.cu:3207`, registered `:6648`. +- `vt::KdaChunkPrefill`: `include/vt/ops.h:2871`, vendored FLA Triton-AOT cubins. +- **Both are opt-in.** `VT_KIMI_DEVICE_KDA` and `VT_KIMI_DEVICE_KDA_CHUNK` are + default OFF (`docs/ENVIRONMENT.md:166`, `:171`); the default production path is + the host reference compose (`src/vllm/model_executor/models/kimi_linear.cpp:13-21`). +- KDA-specific host references: `src/vllm/model_executor/models/kimi_kda.cpp:24` + (`KdaLowRankDecay`), `:60` (`KdaDecayGate`). +- **AOT specialization limits:** `src/vt/cuda/cuda_gdn.cu:5217-5220` and + `:5277-5278` require `dk == dv == 128`, `hk_n == 16`, `hv_n ∈ {48, 32}`. + GLM-5.3-Flash asks for **64 heads**, which falls off every specialization onto + the hand C++ kernel (`:5195-5199`). Not a refusal; a speed cliff. +- `gate_lower_bound` is **not implemented anywhere in the tree** — it appears + only in `.agents/specs/kimi-k3.md:48,:123` and + `.agents/specs/kda-kernel-delta.md:101`. GLM-5.3-Flash sets it to `-5.0` and + §Port map shows it selects a *different formula*, not a clamp. + +### MoE — the routing rule this model needs is already live + +`vt::MoeRouterTopK` args at `include/vt/ops.h:1391-1410` carry `top_k`, +`renormalize`, `scoring_func`, `num_expert_group`, `topk_group`, +`routed_scaling_factor`. Scoring funcs are exactly `kSoftmax` and `kSigmoid` +(`include/vt/ops.h:1378-1381`). Grouped `noaux_tc` is a separate kernel +(`:1385-1390`) with `e_score_correction_bias` as the trailing bias +(`:3132`, `:3143`). CUDA `src/vt/cuda/cuda_moe.cu:64`, `:238`, grouped `:350`, +order `renormalize` **then** `routed_scaling_factor` at `:519-526`, registered +`:935`. `n_group = topk_group = 1` is live and gated on GLM-4.7-Flash. + +Clamped SwiGLU already exists and is already fused into the keep-quant grouped +epilogue: semantics `include/vllm/model_executor/models/deepseek_v4_moe.h:148-166`, +host `src/vllm/model_executor/models/deepseek_v4_moe.cpp:105`, CUDA +`src/vt/cuda/cuda_deepseek_v4.cu:1053`, fused form `include/vt/ops.h:1846-1859`. +**Read `:588-592` before using it:** a zero limit means "clamp to zero" in one +form and "no clamp" in the other, a recorded semantic divergence. + +Expert GEMM ops: `kMoeGroupedGemmBf16` (`include/vt/ops.h:202`), +`kMoeGroupedGemmBf16GateUpSilu` (`:335`), `kMoeGateUpSwiGLUGrouped` (`:305`), +`kMoeCombine` / `kMoeCombineGate` (`:108`, `:160`), `kSharedExpertGate` (`:159`). +No expert-count ceiling was found; 288 is a config value. + +### Shared seams + +| Seam | Definition | Example router | +|---|---|---| +| `vt::FusedChain` | `include/vt/ops.h:2517`, `:2531`, `:2544`, `:2551` | `src/vllm/model_executor/models/kimi_linear.cpp:31-33` | +| `layers::MlpGateUpMethodBase` | `include/vllm/model_executor/layers/linear.h:82` | `src/vllm/model_executor/models/gemma.cpp:131` | +| `vt::MergedGemmGroup` | `include/vt/merged_gemm.h:44`, `:64`, `:94` | `include/vllm/model_executor/models/dense_fp8_block_gemm.h:490` | +| `ModelRegistry::Forward` | `src/vllm/model_executor/models/model_registry.cpp:376` | `src/vllm/v1/worker/gpu/runner.cpp:1827` | +| `dense_attn::AttnBlock` | `include/vllm/model_executor/models/dense_attn_block.h:361` | `src/vllm/model_executor/models/qwen3.cpp:185` | +| on-device sampling | `include/vllm/v1/sample/sampler.h:72`, `:96` | `ForwardLogits::on_device()`, `include/vllm/model_executor/models/qwen3_5.h:117` | +| ABI | `include/vllm.h` (v23 at `:297`) | `mmproj_path` `:273`, mm limits `:170-185` | + +Note before planning: most models deliberately do **not** route attention +through `dense_attn::AttnBlock` and each says why in a header comment — GLM-4 +`src/vllm/model_executor/models/glm4.cpp:48` is the family precedent. A GLM-5.3 +attention block will be its own function, and that is the established pattern +rather than a seam bypass. + +### Vision — one production ViT, and it is the wrong patch size + +`src/vllm/model_executor/models/qwen3_vl_vision.cpp` is the only production +ViT-style encoder for a text LLM (patch embed `:306`, `:385`; merger `:206`, +`:262`). Its config +(`include/vllm/model_executor/models/qwen3_vl_vision.h:35-49`) already matches +GLM-5.3-Flash on `hidden_size 1024`, `num_heads 16`, `depth 24`, +`temporal_patch_size 2`, `spatial_merge_size 2` — and differs on `patch_size`, +which defaults to **16** there and is **14** here +(`include/vllm/multimodal/qwen3vl_processor.h:31`). Both are fields, not +constants, so 14 is a config change; nothing in the tree exercises it. mm +preprocessing lives in `src/vllm/multimodal/qwen3vl_processor.cpp` (image +contract `qwen3vl_processor.h:82`, video `:93-103`, smart_resize `:58`). + +### MTP — supported, and this model's head is out of scope + +`docs/SPECULATIVE-DECODING.md:41` accepts `mtp`; implementation +`src/vllm/model_executor/models/qwen3_5_mtp.cpp`. The GLM-4.7-Flash precedent is +to **skip** the nextn tail rather than run it +(`src/vllm/model_executor/models/glm4_moe_lite_registry.cpp:21-26`, +`allow_mtp_tail = true`). This campaign does the same and records the head as +owed. + +### GGUF — reader only, no converter anywhere in the tree + +The ggml type table is `src/vllm/model_executor/model_loader/gguf_reader.cpp:200` +(`FindGgmlTraits`), one `case` per id. Ids with traits: 0 F32, 1 F16, 2 Q4_0, +8 Q8_0, 10 Q2_K, 11 Q3_K, 12 Q4_K, 13 Q5_K, 14 Q6_K, 16 IQ2_XXS, 18 IQ3_XXS, +19 IQ1_S, 21 IQ1_XXXS, 22 IQ2_S, 23 IQ4_XS, 24-28 I8/I16/I32/I64/F64, 30 BF16, +39 MXFP4, 40 NVFP4, 4x Q1_0. **Traits are not dequant:** the dequant dispatch +(`gguf_dequant.cpp:89-160`) covers 0, 1, 2, 8, 10, 11, 12, 13, 14, 16, 18, 19, +22, 26, 30, 39, 40, 66 — so **IQ4_XS (23) and IQ1_XXXS (21) have traits and no +dequant path**, and a plan that assumes "it's in the table so it loads" is +wrong. Verify per type before choosing an arm. + +`scripts/` contains no safetensors→GGUF converter: `gen-gguf-nvfp4-goldens.py` +only reads GGUF, and the `check-*-gguf-namemap.py` / `gen-*-gguf-manifest.py` +scripts consume existing files. + +### Registration + +`REGISTER_VLLM_MODEL` at `include/vllm/model_executor/models/model_registry.h:532` +(seam doc `:502-523`). The closest structural template is +`src/vllm/model_executor/models/glm4_moe_lite_registry.cpp`, which registers +GLM-4.7-Flash by composing DeepSeek-V2's forward and loader over the same +weights and changing three things (`:18-38`). The canonical refuse-by-name +forward is `src/vllm/model_executor/models/kimi_k3.cpp:44-51`, used at `:54`: +name the architecture, name each missing primitive, name the owning row, cite +the spec. + +## Design + +### What the model IS + +Read from `config.json` at `zai-org/GLM-5.3-Flash`, live 2026-08-26. +`architectures: ["Glm5NextForConditionalGeneration"]`, `model_type: glm5_next`, +`transformers_version: 5.16.0`. + +Text (`text_config`, `model_type: glm5_next_text`): + +| key | value | +|---|---| +| `hidden_size` | 4096 | +| `num_hidden_layers` | 45 | +| `intermediate_size` (dense MLP) | 12288 | +| `num_attention_heads` / `num_key_value_heads` | 64 / 64 | +| `vocab_size` | 154880 | +| `max_position_embeddings` | 1048576 | +| `rms_norm_eps` | 1e-05 | +| `hidden_act` | silu, with `swiglu_limit` 10.0 | +| `tie_word_embeddings` | false | +| `layer_types` | 45 entries: 34 `linear_attention`, 11 `deepseek_sparse_attention` at layers 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43 | +| `mlp_layer_types` | 45 entries: `dense` for layers 0-2, `sparse` thereafter (`first_k_dense_replace: 3`) | +| **MLA** | `q_lora_rank` 1536, `kv_lora_rank` 512, `qk_nope_head_dim` 256, `qk_rope_head_dim` **0**, `qk_head_dim` 256, `v_head_dim` 256, `mla_use_nope` true, `head_dim` 0 | +| **DSA indexer** | `index_head_dim` 128, `index_n_heads` 32, `index_topk` 2048, `index_kpool` 4, `index_kpool_compress` true, `index_kpool_always_select_tail` true, `indexer_rope_interleave` true, `index_share_for_mtp_iteration` true, `indexer_types` all `full` | +| **KDA** | `linear_attn_config`: `num_heads` 64, `head_dim` 128, `short_conv_kernel_size` 4, `gate_lower_bound` -5.0, plus explicit `kda_layers` (34) and `full_attn_layers` (11) index lists | +| **mHC** | `mhc` true, `hc_mult` 4, `hc_sinkhorn_iters` 20, `hc_eps` 1e-06 | +| **MoE** | `n_routed_experts` 288, `n_shared_experts` 1, `num_experts_per_tok` 8, `moe_intermediate_size` 2048, `scoring_func` sigmoid, `topk_method` noaux_tc, `n_group` 1, `topk_group` 1, `routed_scaling_factor` 2.5, `norm_topk_prob` true, `moe_router_dtype` float32 | +| **MTP** | `num_nextn_predict_layers` 1 | + +**The text stack has NO rotary embedding anywhere — not in the MLA block and +not in the indexer.** `text_config` carries no `rope_theta` and no +`rope_scaling`; `Glm5NextTextConfig` *deletes* the inherited `rope_parameters` +field outright (`modular_glm5_next.py:167-169` sets it to the `@strict` +`AttributeError()` sentinel), `Glm5NextTextModel.forward` passes +`position_embeddings=None` to every layer (`:1361-1362`), and `position_ids` is +threaded but never read by any sublayer. `Glm5NextTextAttention` has no rotary +call and no `q_pass`/`q_rot` split. + +**`indexer_rope_interleave: true` in `config.json` is a VESTIGIAL FLAG.** The +parent `GlmMoeDsaIndexer` applies interleaved rotary +(`parent glm_moe_dsa modeling:232`), and `Glm5NextTextIndexer.forward` overrides +that method wholesale and applies no rotary at all. Implementing it because the +config asks for it is implementing a bug. Positional information in this model +comes exclusively from the 34 KDA layers' channel-wise decay and their short +convs. An implementer who wires a rotary into the MLA block because "MLA has +rope", or into the indexer because the config names one, produces a model that +decodes fluently and is wrong. + +Vision (`vision_config`, `model_type: glm5_next_vision`): depth 24, hidden 1024, +`num_heads` 16, `intermediate_size` 4096, `out_hidden_size` 4096, +`projection_intermediate_size` 10240, `patch_size` 14, `image_size` 448, +`spatial_merge_size` 2, `temporal_patch_size` 2, `swiglu_limit` 10.0, +`rms_norm_eps` 1e-05, `attention_bias` true. Placeholder ids: image 154854, +video 154855, image start/end 154830/154831, video start/end 154832/154833. + +Processor (`processor_config.json`): CLIP mean/std +`[0.48145466, 0.4578275, 0.40821073]` / `[0.26862954, 0.26130258, 0.27577711]`, +`merge_size` 2, `patch_size` 14, `temporal_patch_size` 2, `patch_expand_factor` +1, image tokens 16..8000, video tokens 16..240000, `fps` 2. Tokenizer is a +`tokenizers` backend BPE, `model_max_length` 1048576, eos ids +`[154820, 154827, 154829]`, pad 154820; `generation_config.json` carries +`temperature 1.0`, `top_p 0.95`. + +### The four families, and where each one lands + +1. **KDA linear attention, 34 layers.** Reuse the Kimi-Linear/GDN machinery. + Net-new: the forget-gate branch (below) and 64 heads falling off the AOT + specializations. +2. **DSA MLA, 11 layers.** Reuse the DeepSeek-V4 indexer numerics. Net-new: the + k-pool compression stage, and a NoPE MLA geometry the shared block currently + refuses. +3. **mHC residual topology.** Reuse `deepseek_v4_mhc.*` essentially as-is; + `Glm5NextTextHyperConnection` is `pass` over `DeepseekV4HyperConnection` + (`modular_glm5_next.py:364-366`). One documented difference, and it is in the + *head*: `Glm5NextTextHyperHead.forward` is + `hidden_streams.mean(dim=2)` — an **unweighted mean**, whose own docstring + says "Unlike DeepSeek-V4" (`modular_glm5_next.py:368-373`). Our + `HcHeadCollapse` (`deepseek_v4_mhc.cpp:168`) implements the DeepSeek-V4 + weighted collapse (weight-free RMSNorm → `hc_head_fn` → sigmoid gate → + weighted sum). Using it here is wrong and a token gate on a 4-stream mean + versus a gated sum will not obviously look wrong; it will look slightly off. +4. **MoE.** Reuse `vt::MoeRouterTopK` grouped `noaux_tc` and the clamped-SwiGLU + grouped epilogue. The router is `pass` over `DeepseekV3TopkRouter` + (`modular_glm5_next.py:350-351`). + +Plus the vision tower, which is a GLM-OCR ViT +(`Glm5NextVisionModel(GlmOcrVisionModel)`, `modular_glm5_next.py:1410`) and has +no counterpart in this tree — the nearest is Qwen3-VL's, which differs in patch +size and in whichever GLM-OCR specifics wave 6 has to establish. + +## Port map + +`OURS` is a target path in this tree; `<-` names the upstream anchor. Every +upstream anchor is `transformers v5.16.1` +`src/transformers/models/glm5_next/modular_glm5_next.py` unless stated. Anchors +were read at that tag on 2026-08-26; a wave that finds one stale re-reads it and +corrects this table in the same change. + +| OURS (planned) | <- upstream | reuse or net-new | +|---|---|---| +| `glm5_next_config.{h,cpp}` | `:92-247` (`Glm5NextTextConfig`), `:248-264` (vision), `:265-316` (top) | net-new, mechanical | +| `glm5_next_registry.cpp` | registry contract is ours | pattern from `glm4_moe_lite_registry.cpp:18-38` | +| KDA arm | `:597-748` (`Glm5NextTextLinearAttention`), `:441-494` (recurrent), `:495-596` (chunk) | REUSE `vt::KdaGatedDeltaRule` / `KdaChunkPrefill` | +| KDA forget gate | `:375-408` (`Glm5NextTextForgetGate`) | **NET-NEW BRANCH — see below** | +| KDA output norm | `:409-428` (`Glm5NextTextRMSNormGated`) | REUSE, but strict-fp32 cast | +| `l2norm` | `:429-440` | REUSE; `sqrt(sum(x*x) + eps)`, eps 1e-6, NOT `max(..., eps)` | +| DSA indexer | `:749-1024` (`Glm5NextTextIndexer`) | REUSE `deepseek_v4_dsa.cpp:14,31,72` + **NET-NEW k-pool** | +| MLA block | `:1025-1141` (`Glm5NextTextAttention`) | EXTEND `mla_attention.*` for `qk_rope_head_dim == 0` | +| mHC pre/post/sinkhorn | `:364-367` -> `DeepseekV4HyperConnection` | REUSE `deepseek_v4_mhc.cpp:23,72,149` | +| mHC head collapse | `:368-374` (`Glm5NextTextHyperHead`) | **NET-NEW — unweighted mean, NOT `HcHeadCollapse`** | +| MoE router | `:350-353` | REUSE `vt::MoeRouterTopK` grouped `noaux_tc` | +| MoE experts | `:335-349` (`Glm5NextTextExperts`) | REUSE `kMoeGateUpSwiGLUGrouped` clamped epilogue | +| dense MLP | `:321-334` (`Glm5NextTextMLP`) | REUSE clamped SwiGLU; note the asymmetry below | +| decoder layer | `:1142-1208` | net-new control flow | +| text model | `:1285-1372` | net-new | +| vision tower | `:1373-1421` | net-new, nearest `qwen3_vl_vision.cpp` | +| mm wrapper | `:1563-1703` | net-new, nearest `qwen3_vl.cpp` | +| processor | `processing_glm5_next.py`, `image_processing_glm5_next.py`, `video_processing_glm5_next.py` | net-new, nearest `qwen3vl_processor.cpp` | + +### The forget gate is a DIFFERENT FORMULA, not a clamp + +This is the finding most likely to be got wrong, so it is written out. +`Glm5NextTextForgetGate.forward` (`modular_glm5_next.py:389-407`) computes the +low-rank projection exactly as Kimi KDA does — `f_b_proj(f_a_proj(x))`, add +`dt_bias`, reshape to `[..., H, D]`, `decay_rate = exp(A_log)` — and then +**branches**: + +```python +if self.safe_gate_lower_bound is not None: + return self.safe_gate_lower_bound * torch.sigmoid(decay_rate * g) +g_softplus = torch.where(g > 20.0, g, torch.log(1.0 + torch.exp(g))) +return -decay_rate * g_softplus +``` + +`safe_gate_lower_bound` is `config.linear_lower_bound`, and the published +checkpoint sets `linear_attn_config.gate_lower_bound = -5.0`, which is **not +None**. So GLM-5.3-Flash takes the FIRST branch: +`g_out = -5.0 * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias))`. + +Our `KdaDecayGate` (`src/vllm/model_executor/models/kimi_kda.cpp:60`, softplus +at `:15`) implements the SECOND branch — `-exp(A_log) * softplus(g + dt_bias)`, +which is Kimi-Linear's. **The two are different functions of the same inputs.** +Both are smooth, both are negative, both produce a plausible decay, and both +produce fluent text. A model built on the wrong one is wrong everywhere and +looks fine. Note also the sign: `decay_rate = +exp(A_log)` in the sigmoid branch, +where the softplus branch negates it. Wave 2's red-first test is this branch. + +### The k-pool indexer is a compression stage DeepSeek-V4 does not have + +`Glm5NextTextIndexer` inherits `GlmMoeDsaIndexer` — the DeepSeek-V3.2 lightning +indexer whose numerics `deepseek_v4_dsa.cpp` already ports — and adds +`index_kpool = 4`, `index_kpool_compress = true`, +`index_kpool_always_select_tail = true`. The candidate set the top-`2048` +selection runs over is therefore **pooled keys, not raw tokens**, with the tail +always kept. Two consequences the wave that implements it must design for, both +recorded here because they are the same shape as a trap already recorded on the +`MODEL-MM-QWEN4-EXP` row: + +- Feeding raw-token candidates into the top-k, or pooled candidates into a + consumer expecting raw ones, yields plausible tokens either way. +- **A short-prompt gate cannot catch it.** With `index_topk = 2048`, any context + at or below 2048 candidate positions selects everything, so the selection is + the identity and the pooling is unobservable. Any indexer gate must run past + that threshold. §Gates makes this a requirement, not a note. + +**The pooling operator is LEARNED, per channel, and it is not a mean.** Read at +`modular_glm5_next.py:897-970` (`get_pooled_states`): + +``` +P = ceil(kv_len / kpool) # kpool = 4 +first_key[b] = argmax(valid_keys[b]) # index of the first NON-PAD token +member(b,p,j)= first_key[b] + p*kpool + j +logits[b,p,j,c] = fp32(gate_scores[member][c]) + fp32(ape[j][c]) # -inf on invalid members +prob = nan_to_num(softmax(logits, dim=j)) # 128 INDEPENDENT 4-way softmaxes +pool_key[b,p,c] = sum_j prob[b,p,j,c] * key[member(b,p,j)][c] +``` + +`index_kpool_compress_ape` is a learned `[kpool, head_dim] = [4, 128]` +intra-pool absolute-position embedding; `index_kpool_compress_gate` is a +`[128, 4096]` weight applied as `F.linear(x, W)` and its output is **cached per +token** (see the packed cache below). `index_kpool_compress: true` selects this +path and the reference implements no mean-pool fallback, so a `false` value is +not expressible. + +Four more facts that are the actual work, none of which DeepSeek-V4 has: + +- **The pool grid is batch-dependent.** Pools start at the first *valid* token, + not at slot 0 (`:938-945`), so a left-padded row groups differently from an + unpadded one. +- **A pool must be complete to be a candidate** (`pool_valid[b,p]` = all four + members valid), and a pool is *visible* iff its LAST member is visible to the + query (`:830-842`). Invalid candidates are masked with `finfo(dtype).min`, not + `-inf`. +- **The budget is `select_k = min(index_topk // index_kpool, P) = min(512, P)`**, + and `validate_architecture` enforces `index_topk % index_kpool == 0` + (`:234-235`). The selected pools are then expanded back to raw token indices — + `[B,S,512,4]` flattened to `[B,S,2048]` (`:850-862`). +- **The ragged tail is appended raw and UNSCORED** when + `index_kpool_always_select_tail` is set (`append_visible_tail`, `:972-1022`), + widening the output to `index_topk + kpool - 1 = **2051**`, not 2048. `-1` is + the invalid sentinel throughout, and duplicate indices are possible and are + absorbed downstream by `scatter_add_` + `ne(0)`. + +**The indexer cache is packed and wider than DeepSeek-V4's.** The parent caches +`k` only (128 floats/token/layer); this caches +`concat[k(128), gate_scores(128), valid(1)]` = **257 floats/token/layer** +(`:798-808`), because pools are re-formed over the whole history on every call. +The §Hardware KV arithmetic uses the compressed figure and W5 must re-derive it +against the real allocator. + +**This round trip is the genuinely new idea in the model: score coarse, attend +fine, always keep the ragged tail.** DeepSeek-V4's compressed-sparse-attention +indexer selects compressed blocks and then *attends to the compressed entries*; +this selects pools and then attends to the **raw** tokens they cover. The +pooling *kernel* itself is not novel — `DeepseekV4HCACompressor` already does a +softmax-over-window weighted pool — so the port reuses that shape and must not +reuse its `compress_rate`, its post-pool RMSNorm, or its RoPE, none of which +appear here. + +### The clamped SwiGLU is asymmetric, and we already have the asymmetry + +`Glm5NextTextMLP.forward` (`:326-333`) clamps `gate` with `max` only and `up` +with both bounds, at `swiglu_limit = 10.0`; `Glm5NextTextExperts._apply_gate` +(`:341-348`) does the same and then plain `silu(gate) * up` with no alpha or +beta. That is exactly the semantic our +`include/vllm/model_executor/models/deepseek_v4_moe.h:148-166` documents with +`alpha = 1`, `beta = 0`. Reuse it, and read `include/vt/ops.h:588-592` first for +the zero-limit divergence between the two forms. + +The same clamp appears in **five** places, and dropping it or symmetrising it in +any one of them is a silent defect: the dense MLP (layers 0-2, width 12288), the +shared expert (width 2048), the routed experts, the vision MLP, and the vision +patch merger. + +### Constants and layouts a port gets silently wrong + +Collected here because each one is a plausible default that is not this model's +value. + +| item | value here | the wrong-but-plausible value | +|---|---|---| +| text RMSNorm eps | **1e-5** (`rms_norm_eps`) | 1e-6 — the `GlmMoeDsa` parent constructs its MLA LoRA norms with the default 1e-6; `Glm5NextTextAttention.__init__` passes `eps` explicitly (`modular_glm5_next.py:1029-1032`) | +| `hc_eps` | **1e-6** | 1e-5 — it is a *different* constant from `rms_norm_eps` and is added to every Sinkhorn denominator, not used as a floor | +| `l2norm` eps | **1e-6**, inside the sqrt | `F.normalize`'s `max(norm, eps)` — the reference deliberately matches FLA's Triton `sqrt(sum + eps)` (`:429-437`) | +| KDA output-norm activation | **sigmoid** | silu — both Qwen3.5-GDN and FLA's KDA use silu here (`:409-412`) | +| KDA output-norm eps | **1e-5** (`rms_norm_eps` passed in at `:635`) | 1e-6, the constructor default | +| indexer `k_norm` | **`nn.LayerNorm(128, eps=1e-6)` WITH bias** | an RMSNorm — the checkpoint carries `indexer.k_norm.bias`, which settles it | +| `first_k_dense_replace` | the literal **3**, hardcoded at `:176` | reading the config key — the *field is deleted* from the config class, so the checkpoint's `first_k_dense_replace: 3` is an inert extra kwarg that happens to agree | +| per-layer schedule | the top-level **`layer_types`** list | `linear_attn_config.kda_layers` / `full_attn_layers` — the reference **ignores** both lists | +| `index_kpool` | **4** (from the checkpoint) | 16, the config class default | +| vision `out_hidden_size` | **4096** | 1536, the config class default | +| vision `image_size` | **448** | 336, the config class default | +| vision merger context dim | **`projection_intermediate_size` = 10240** | `out_hidden_size * in_channels` = 12288, which is the `GlmOcr` parent's rule | +| router selection vs weighting | select on **biased** scores, weight with **unbiased** ones, normaliser `+ 1e-20` | using one set for both | +| group routing | `n_group = topk_group = 1` makes the group stage a **no-op**; skip it | implementing masked group selection that cannot change the result | + +Checkpoint layout differs from the reference's packing in three places, and the +loader has to bridge each: + +- **KDA convs.** The reference uses ONE grouped `nn.Conv1d` over the + concatenated `[q; k; v]` channel axis (`in = out = groups = 3 * 8192 = 24576`, + kernel 4, **bias-free**). The checkpoint stores **three separate depthwise + convs**: `self_attn.{q,k,v}_conv1d.weight`. Concatenate in **q, k, v** order, + or run three convs. +- **Experts.** The reference packs `gate_up_proj: [288, 2*2048, 4096]` with gate + in the first half. The checkpoint stores per-expert + `mlp.experts.{0..287}.{gate,up,down}_proj.weight`, FP8 block-quantised with + `weight_scale_inv` companions. +- **Hyper-connections.** The reference's module paths are + `attn_hc.{fn,base,scale}` / `ffn_hc.*`; the checkpoint stores them **flat on + the layer** as `hc_attn_{fn,base,scale}` and `hc_ffn_{fn,base,scale}`, and + `hyper_connection` is in the FP8 `modules_to_not_convert` list so they ship + unquantised. **There are no `hc_head.*` tensors at all**, which independently + confirms the unweighted-mean head — do not allocate them. + +Two more, in the layer body: + +- **`comb` is consumed TRANSPOSED.** The mix is + `out[k] = sum_j comb[j, k] * residual[j]`, summing over the *first* hc axis + (`:1194-1203`). The Sinkhorn result is doubly stochastic but **asymmetric**, so + transposing it wrongly degrades quality silently instead of crashing. +- **`post` and `comb` are produced in fp32 and cast to the activation dtype + BEFORE the mix**, at both the attention and the FFN site. Keeping them fp32 + through the mix does not reproduce the reference. + +### MLA: cache the latent, do not cache what the reference caches + +The reference caches the **expanded** `K [B, 64, S, 256]` and `V [B, 64, S, 256]` +— 32,768 values per token per layer. Across 11 DSA layers that is not a design +this project should copy. The standard MLA absorption applies here and is +**trivially valid precisely because the model is NoPE**: with `qk_rope_head_dim +== 0` there is no rope slice that has to stay outside the absorption, so +`kv_b_proj` folds into `q_b_proj` and the cache holds the 512-wide latent. The +§Hardware KV arithmetic assumes the absorbed form, and W3 owns proving the +equivalence rather than asserting it. + +### The MTP block is in the checkpoint, and the reference throws it away + +`num_hidden_layers` is 45, and the checkpoint carries **46** layer directories. +`model.language_model.layers.45.*` is a DeepSeek-V3-style MTP block: `enorm`, +`hnorm`, `eh_proj` (`[hidden, 2*hidden]`), a full MLA layer **with its own +indexer**, a full 288-expert sparse MoE, and `shared_head.norm` — it reuses +`lm_head` and has no head of its own. The transformers reference discards it: +`_keys_to_ignore_on_load_unexpected = [r"layers\.45\.", r"layers\.\d+\.shared_head\."]` +(`modular_glm5_next.py:1235`). `num_nextn_predict_layers` and +`index_share_for_mtp_iteration` are not config fields there; they arrive as +inert kwargs. + +Two structural facts worth having on record before O2 is ever picked up: the MTP +block is **DSA/MLA, not KDA**, and it carries **no `hc_*` tensors**, so it runs +on a single residual stream rather than the four-stream manifold. + +**Consequence for W7:** those weights are a large share of the checkpoint (a +46th layer whose MoE alone is 288 experts) and the converter should skip them, +following the `glm4_moe_lite_registry.cpp:21-26` precedent of requesting only +`[0, num_hidden_layers)`. + +### Image and video share ONE token id + +`image_token_id` is 154854 and `video_token_id` is 154855, and **the processor +emits 154854 for both**. Video is written frame by frame as +`<|begin_of_image|>{<|image|> * n}<|end_of_image|>{ts:.1f} seconds`, with the +whole run wrapped in `<|begin_of_video|>` / `<|end_of_video|>` by the chat +template. Disambiguation is by SPAN, not by id: +`in_video_span = cumsum(id == video_start) > cumsum(id == video_end)`, then +`image_mask = is_mm & ~in_video_span` (`modular_glm5_next.py:1477-1485`, which +calls this out as "the core difference to other VLMs"). A port that keys off the +token id gets every video frame classified as an image. + +Also in the vision path, and both differ from Qwen3-VL: the rope is **2D (h, w) +only, with no temporal axis** — `get_video_features` rewrites `video_grid_thw` +from `[t, h, w]` into `t` rows of `[1, h, w]` so each frame is an independent +`t = 1` image — and it uses **half-split `rotate_half`**, not the interleaved +form the GLM-MoE-DSA text side uses. There is no mrope: `Glm5NextModel.__init__` +does `del self.rope_deltas` (`:1429`). + +## Dependencies + +| # | Dependency | State | Effect | +|---|---|---|---| +| D1 | MLA `qk_rope_head_dim == 0` | refused at `mla_attention.cpp:90-93` | blocks every DSA layer; wave 3 | +| D2 | KDA device path default-OFF | `docs/ENVIRONMENT.md:166`, `:171` | first arm runs the host compose; speed is not a wave-1..5 claim | +| D3 | KDA AOT specializations exclude 64 heads | `cuda_gdn.cu:5217-5220`, `:5277-5278` | hand-kernel fallback; a named speed residual, not a blocker | +| D4 | DSA device entry points are host-vector | `deepseek_v4_device.h:82-102` | upload/download per call; a named speed residual | +| D5 | No GGUF converter in tree | `scripts/` has none | wave 7 must author one | +| D6 | llama.cpp has no `glm5_next` | code search 0; PR #27752 open | no llama.cpp quant oracle and no floor for the GGUF arms | +| D7 | transformers lane pin at `v5.16.1` | `.agents/oracles/transformers.md` pins 5.14.1 | wave 0 writes the lane pin; `gateable = no` | +| D8 | Live seam contention | PRs [#1971](https://github.com/mudler/vllm.cpp/pull/1971) (DSA geometry), [#1977](https://github.com/mudler/vllm.cpp/pull/1977) (DSv4 KV multicache) | waves 3 and 5 rebase onto whichever lands first; do NOT fork the seam | + +D8 is scheduling, not a blocker, and it is why the wave order below puts config +and KDA before the DSA and KV work. + +## Work breakdown + +Eight waves. Each is a separate `row/MODEL-MM-GLM53-FLASH-W` branch, a +separate pull request, a fresh implementer and a fresh reviewer. **W0-W2 and W4 +are CPU-gateable and need no GPU at all. W3 and W5-W8 need a GPU.** Sizes are +the author's estimate of reviewable diff, not a budget. + +### W0 — records and the lane oracle pin (CPU, small) + +Write `.agents/oracles/transformers.md`'s lane-scoped `v5.16.1` pin for this row +with `gateable = no` and the issue that owes the measurement. Verify +`scripts/check-oracle-pins.py` accepts the shape. **Deliverable:** the pin and +nothing else. **Exclusion:** no model code. **Gate:** `agent-preflight.sh` green. +**Stop:** if the checker refuses a second lane pin, return `NEEDS_DECISION` +rather than editing the checker. + +### W1 — config, registration, refuse-by-name (CPU, medium) + +Resolve `glm5_next`'s nested `text_config` / `vision_config` / +`quantization_config`; register `Glm5NextForConditionalGeneration`; enumerate +the 76,108-tensor weight name map structurally; make `Forward` refuse by name +naming every unimplemented primitive and this spec. +**Scope:** `glm5_next_config.*`, `glm5_next_registry.cpp`, +`glm5_next_weights.cpp`, `include/.../glm5_next.h`. +**Exclusions:** no forward math, no kernels, no loader materialization. +**Anchors:** `modular_glm5_next.py:92-316`; pattern +`glm4_moe_lite_registry.cpp:18-38`; refusal pattern `kimi_k3.cpp:44-51`. +**Tests:** registry resolve; config descent including the +`linear_attn_config` → `linear_*` key remap and the two per-layer index lists; +the 45-entry `layer_types` / `mlp_layer_types` split; refuse-by-name message +names each missing primitive. +**Gate:** CPU build `-DVLLM_CPP_CUDA=OFF`, focused ctest, full preflight. +**Evidence:** the registry contract test's architecture count moves by exactly +one. **Reachability:** the registration is reached from +`ModelRegistry::Forward`; deleting the `REGISTER_VLLM_MODEL` line must red the +focused gate. + +### W2 — the KDA arm's numerics (CPU, medium) + +Port `Glm5NextTextForgetGate`'s **sigmoid branch**, the strict-fp32 +`RMSNormGated`, and `l2norm`, as portable host references with an independent +double-precision derivation, exactly as `kimi_kda.cpp` did for the softplus +branch. Wire the KDA layer's q/k/v short convs (`short_conv_kernel_size = 4`) +and the delta recurrence onto the existing `vt::KdaGatedDeltaRule` / +`KdaChunkPrefill` seams. +**Scope:** `glm5_next_kda.{h,cpp}` plus the seam call sites. Note three +layout facts the wave must handle: the checkpoint stores **three separate +depthwise convs** where the reference has one grouped conv over `[q; k; v]` +(concatenate in q, k, v order); `g`, `beta` and the output gate are all computed +from the **pre-conv** hidden states, so they must not be fused into the conv +path; and the KDA cache is a `[B, 24576, 4]` conv state plus a **fp32** +`[B, 64, 128, 128]` recurrent state, 4 MiB per layer per sequence, ~136 MiB +across the 34 layers. +**Exclusions:** do not touch `src/vt/cuda/cuda_gdn.cu`; do not touch +`kimi_kda.cpp` (Kimi-Linear's gate is the other branch and stays). +**Anchors:** `modular_glm5_next.py:375-408`, `:409-428`, `:429-440`, `:597-748`. +**Tests, RED FIRST:** a case whose expected value is computed from the sigmoid +branch and which FAILS against `kimi_kda.cpp:60`'s softplus branch. That failure +is the deliverable's proof. +**Gate:** CPU, focused ctest + full preflight. **CPU-gateable.** + +### W3 — NoPE MLA and the DSA k-pool indexer (GPU, large) + +Extend `MlaBlockDims::Validate` to accept `qk_rope_head_dim == 0` with +`head_size() == kv_lora_rank`, and thread it through the decode and prefill +paths; resolve and implement the k-pool compression and tail-keep in the +indexer. +**Scope:** `mla_attention.{h,cpp}`, `cuda_mla_attn.cu`, `cuda_mla_prefill.cu`, +`glm5_next_dsa.{h,cpp}`. +**Exclusions:** do not change any existing model's resolved geometry. SACRED +inertness against DeepSeek-V2/V3, Kimi-Linear and GLM-4.7-Flash goldens is a +gate, not a hope. +**Anchors:** `modular_glm5_next.py:749-1024`, `:1025-1141`. +**Tests:** NoPE geometry validation, both accept and refuse; a k-pool selection +case at a context **strictly greater than `index_topk`** so the selection is not +the identity; existing MLA goldens byte-identical. +**Needs GPU:** the decode kernel's 512/256 head pair is untested and the +shared-memory guard at `cuda_mla_attn.cu:545-546` can only be checked by +running. **Rebase note:** coordinate with PRs #1971 and #1977. + +### W4 — mHC wiring and the unweighted head (CPU, small) + +Reuse `MhcSinkhorn` / `MhcPre` / `MhcPost` at `hc_mult = 4`, +`hc_sinkhorn_iters = 20`, `hc_eps = 1e-06`; implement the **unweighted-mean** +head collapse as a GLM-5-specific function and do NOT reuse `HcHeadCollapse`. +**Anchors:** `modular_glm5_next.py:364-374`; ours `deepseek_v4_mhc.cpp:23,72,149,168`. +**Tests, RED FIRST:** a case that passes against the mean and fails against +`HcHeadCollapse`. **CPU-gateable.** + +### W5 — MoE, the decoder layer, and the assembled text forward (GPU, large) + +Wire the 288+1 expert MoE through the existing grouped `noaux_tc` router and +clamped-SwiGLU epilogue; implement the per-layer control flow (KDA vs DSA, +dense vs sparse, mHC stream threading); assemble `Glm5NextTextModel::Forward`; +build the heterogeneous KV cache (11 MLA groups + 34 KDA state groups + the +indexer side cache) through `MakeKVCache`, following +`kimi_linear_registry.cpp:135-166`. +**Anchors:** `modular_glm5_next.py:321-363`, `:1142-1208`, `:1285-1372`. +**Needs GPU.** **Rebase note:** the KV grouping overlaps PR #1977 directly. + +### W6 — vision tower, processor, mm placeholder expansion (GPU, large) + +The 24-layer GLM-OCR-style ViT at patch 14, the patch merger at +`spatial_merge_size 2`, the projection at `projection_intermediate_size 10240`, +and the image/video preprocessing including `fps 2`, `temporal_patch_size 2`, +the 16..8000 image and 16..240000 video token bounds, and the six placeholder +ids. **Anchors:** `modular_glm5_next.py:1373-1421`, `:1563-1703`, +`processing_glm5_next.py`, `image_processing_glm5_next.py`, +`video_processing_glm5_next.py`. **Nearest ours:** `qwen3_vl_vision.cpp`, +`qwen3vl_processor.cpp`. + +### W7 — the GGUF converter and the first fitting arm (GPU + large asset) + +Author a safetensors→GGUF converter for `glm5_next` (no upstream tool can do it, +D5/D6) and produce the arm §Hardware names. **This wave needs explicit developer +authority for a large-asset download** and is the only wave that does. +**Exclusions:** no i-quant arm in this wave — see §Risks R4. + +### W8 — speed, once and only once a correctness gate exists + +Not scheduled by this spec. AGENTS.md forbids accepting a performance result +before the declared correctness gate stands, and §Gates explains why the +end-to-end one cannot stand here. W8 opens when W7 closes and its scope is set +then. + +## Tests to port + +`tests/models/` in transformers `v5.16.1` is the upstream suite. What is +portable and what is not, stated rather than assumed: + +- **Config resolution and the layer-type expansions** port directly: the 45-entry + `layer_types`, `mlp_layer_types`, `indexer_types` lists and the + `linear_attn_config` → `linear_*` remap are pure functions of the config. +- **The processor tests port**: upstream's own + `tests/transformers_utils/processors/test_glm5next.py` shape (a 389-line + processor suite exists in the inadmissible vLLM PR, which is evidence that a + processor suite is the right unit, not a source to copy). Ours goes against + the transformers processor at the lane pin. +- **Component numerics do not port as tests; they port as GOLDENS.** transformers + has no numeric fixtures for these layers. §Gates says how we make them. +- **No end-to-end generation test ports**, because no side can run the model. + +Local red-first tests, one per wave, are named in each wave above. Every one of +them must fail for the intended reason before the implementation exists, and the +failure must be captured. + +## Gates + +### The end-to-end token gate is UNREACHABLE, and saying so is the gate decision + +No oracle implements this architecture at a pinned revision except transformers +`v5.16.1`, and running it needs 305.78 GiB (FP8) or 598.5 GiB (BF16). The +largest reachable device is `dgx:gpu0` at ~119.63 GiB unified. **There is no +token-exact end-to-end gate against an oracle for this model on this fleet, and +no amount of implementation work creates one.** Any campaign report claiming one +is wrong. This is recorded as visible debt under §Owed, not waived. + +### What IS reachable, and it is the campaign's spine + +**A tiny-shape reference oracle, on CPU, from the pinned transformers.** A +randomly-initialised `Glm5NextConfig` at a small shape — the shape is wave 1's +to fix, on the order of `hidden_size 128`, 4 layers alternating KDA and DSA, +8 experts, `hc_mult 4`, a 32-token vocabulary — instantiates and runs on CPU in +seconds with no weights and no GPU. Dumping per-component activations from it +gives a REAL numerical oracle for every component in waves 1-6, at f32/f64 +tolerance, hermetic and seed-reproducible. + +Two things this is and is not, stated because the distinction has been got +wrong here before: + +- It **is** an oracle for the *numerics*: the reference implementation computes + the values, not a hand derivation, so a transcription bug in our port is + caught. +- It is **not** oracle gateability for the *model*. AGENTS.md's bar is that the + oracle demonstrably builds and runs THE MODEL; constructing a config proves + nothing. `gateable` stays `no` for `transformers` on this row and the oracle + file says so. + +This is the same shape as the DeepSeek-V4 W3/W5 lane +(`deepseek_v4_dsa.cpp`, `deepseek_v4_mhc.cpp`) and it is strictly stronger, +because those gate against hand-derived cases and this gates against the +reference implementation's own output. + +### Binding gates per wave + +| wave | gate | CPU or GPU | +|---|---|---| +| W0 | `check-oracle-pins.py` accepts the lane pin; preflight green | CPU | +| W1 | registry resolve, config descent, refuse-by-name; architecture count +1; preflight | CPU | +| W2 | tiny-shape forget-gate / gated-norm / l2norm goldens; RED-first against the softplus branch | CPU | +| W3 | NoPE MLA accept+refuse; k-pool selection at context **> `index_topk` = 2048**; SACRED inertness on DeepSeek-V2/V3, Kimi-Linear, GLM-4.7-Flash goldens byte-identical | GPU | +| W4 | mHC goldens at `hc_mult 4`; RED-first against `HcHeadCollapse` | CPU | +| W5 | per-layer control-flow goldens; assembled tiny-model forward vs the tiny reference | GPU | +| W6 | processor parity vs the transformers processor at the lane pin; placeholder-expansion goldens | GPU | +| W7 | **the arm below RUNS on `dgx:gpu0` and generates coherent text** | GPU | + +### W7's verification target, set by the developer 2026-08-26 + +**A low GGUF quant that fits on `dgx:gpu0` (GB10, ~119.63 GiB usable).** The +candidates, with the arithmetic and its limits, are in §Hardware. The gate is: +the arm loads, the runner serves it through `include/vllm.h`, and it generates +coherent text on a prompt with an image. It is a **run gate, not a token gate**, +because §Gates' first paragraph holds — nothing can produce the reference +tokens. Do not report it as a correctness gate. + +## Hardware + +### The measured artifacts + +Read live 2026-08-26 from the HuggingFace API, and for the primary repo by HTTP +RANGE over all 62 safetensors headers — payload was never fetched. + +| repo | bytes | GiB | vs 119.63 GiB | +|---|---|---|---| +| `zai-org/GLM-5.3-Flash` (FP8 e4m3, block 128x128) | 328,326,771,576 | 305.78 | **2.56x over** | +| `unsloth/GLM-5.3-Flash-FP8` | 328,366,169,538 | 305.82 | 2.56x over | +| `zai-org/GLM-5.3-Flash-BF16` | 642,676,397,788 | 598.53 | 5.00x over | +| `LibertAIDAI/GLM-5.3-Flash-NVFP4` | 194,692,687,003 | 181.32 | **1.52x over** | + +**NOTHING PUBLISHED FITS.** The smallest published artifact is 1.52x the whole +GB10 pool. + +**No GGUF exists.** Four repositories are named `*-GGUF` and every one of them +contains **zero `.gguf` files**: `unsloth/GLM-5.3-Flash-GGUF` (README + +`.gitattributes`), `AtomicChat/GLM-5.3-Flash-GGUF` (README + four PNGs), +`aj9o9/GLM-5.3-Flash-GGUF` (README), `vcruz305/GLM-5.3-Flash-GGUF` (README). +A repository name is not an artifact. + +### The exact parameter split, from the shard headers + +62 shards, 76,108 tensors. By dtype: `F8_E4M3` 314,396,639,232 elements, +`BF16` 6,926,096,640, `F32` 19,484,766 (the block scales — 314.4e9 / 128 / 128 = +19.19M, which is what that count is). **Total real parameters: +321,322,735,872 (321.32B)**, consistent with the card's "320B total". + +By group: routed experts **311.65B (97.0%)** across 37,152 tensors; everything +else 6.76B; shared expert 1.08B; embedding 634M; `lm_head` 634M; vision 564M. +That 97% is the single most important number for quantization planning: the +mixed bits-per-weight of any GGUF arm is, to within a percent, the bits-per-weight +of whatever type the experts get. + +**The MTP block is inside that total.** `model.language_model.layers.45.*` is +888 tensors and **7.43B parameters, 2.31% of the model** — a 46th layer whose +MoE alone carries 288 experts. Skipping it in the converter (O2, and the +`glm4_moe_lite_registry.cpp:21-26` precedent) removes ~2.3 GiB from a Q2_K arm. +Every arm figure below is stated INCLUDING layer 45, so skipping it is headroom +this table does not already spend. + +### GGUF arms — arithmetic, NOT measurement + +Bits per weight are from our own reader's block traits +(`gguf_reader.cpp:200`), so they are exact for the type: Q2_K 84B/256 = 2.625, +IQ2_S 82/256 = 2.5625, IQ2_XXS 66/256 = 2.0625, IQ3_XXS 98/256 = 3.0625, +IQ1_S 50/256 = 1.5625, Q4_K 144/256 = 4.5, Q6_K 210/256 = 6.5625, Q8_0 +34/32 = 8.5. Mixed arms below put the experts at the named type and the +remaining 3% at Q6_K. + +| arm (experts @) | mixed bpw | weights GiB | fits 119.63? | headroom | +|---|---|---|---|---| +| Q8_0 | 8.50 | 318.0 | no | — | +| Q6_K | 6.56 | 245.5 | no | — | +| Q4_K | 4.56 | 170.6 | no | — | +| IQ3_XXS | 3.16 | 118.1 | **no, not really** | ~1.5 GiB — not a margin | +| Q2_K | 2.74 | 102.6 | yes | ~17 GiB | +| IQ2_S | 2.68 | 100.3 | yes | ~19 GiB | +| IQ2_XXS | 2.20 | 82.3 | yes | ~37 GiB | +| IQ1_S | 1.71 | 64.0 | yes | ~56 GiB | + +The KV side is small, which is the whole point of a linear-attention hybrid. +Per token: 11 MLA layers x `kv_lora_rank` 512 x 2 B = 11,264 B, plus the indexer +side cache at `index_head_dim` 128 x 2 B / `index_kpool` 4 = 64 B per layer x 11 += 704 B. **~11.7 KiB/token**, so 128K context is ~1.5 GiB and the full 1M +context is ~11.4 GiB. The 34 KDA layers hold a per-sequence recurrent state +rather than per-token KV: 64 heads x 128 x 128 x 4 B x 34 ~ 143 MiB per +sequence, plus conv states. **This arithmetic is unverified against our own +allocator** and #1963/#1966 record that the KV byte accounting has been wrong by +48x before, so W5 re-derives it from the runner rather than from this table. + +**Recommended first arm: experts at Q2_K, ~102.6 GiB, with ~17 GiB for KV, +activations and page cache at 128K context.** Not IQ2_S or below, and the reason +is not quality — it is that **i-quants need an importance matrix, and an +importance matrix needs a forward pass over the model, which needs 181 GiB +minimum.** The dependency is circular on this fleet. K-quants do not need one. +That makes Q2_K the only fitting arm that is *producible* here, and it should be +stated in W7's scope rather than discovered. + +Second-choice, if Q2_K's ~17 GiB proves thin once W5 measures the real KV: drop +the experts to Q2_K and leave the 3% at Q5_K, or accept a shorter maximum +context. Do not reach for IQ2_XXS to buy headroom without first solving the +imatrix problem. + +### Three things that make "fits in VRAM" the wrong question + +- **GB10 is unified memory.** The 119.63 GiB is the whole pool, not a VRAM + budget beside a separate host RAM. +- **`gpu_memory_utilization` does not bound host RAM on GB10.** A plan that sets + it and considers the question closed has not bounded anything. +- **The oracle side has been measured at ~103 GB of host RAM at KV profiling** + for a large model on an idle box (#1431). Here it is moot — the oracle cannot + run at all — but the same shape applies to our own loader, and W7 measures + peak RSS rather than assuming the on-disk size is the resident size. + +### Fleet verdict + +| device | can run the reference oracle | can run our Q2_K arm | +|---|---|---| +| `dgx:gpu0` (GB10, ~119.63 GiB) | **no** — 305.78 GiB needed | plausible on arithmetic; **not measured** | +| `thor:gpu0` | no | no | +| `orin:gpu0` | no | no | + +No lease was taken for this spec and no GPU ran. Every entry above is arithmetic +over measured file sizes. + +## Risks + +**R1 — the forget-gate branch.** Highest-probability silent defect in the +campaign. Mitigated by W2's RED-first test against the softplus branch. If that +test is not red before the implementation, the wave has not proven anything. + +**R2 — the mHC head collapse.** Reusing `HcHeadCollapse` produces a working +model with a wrong final projection. Mitigated by W4's RED-first test. + +**R3 — the k-pool selection is unobservable below 2048 tokens.** Any gate at or +under `index_topk` selects everything and passes regardless. W3's gate requires +a context strictly greater than 2048. This is the single most likely way for the +campaign to ship a green gate over a broken indexer. + +**R4 — the i-quant/imatrix circularity.** An importance matrix requires running +the model; running the model requires 181 GiB; therefore no i-quant arm is +producible on this fleet. Recorded as a boundary, not a task. + +**R5 — the converter is ours to write and has no oracle.** llama.cpp cannot emit +this architecture, so there is no reference GGUF to diff against and no +llama.cpp floor for the arm. W7 must state which side ran what. + +**R6 — seam contention.** PRs #1971 and #1977 are live on the DSA-geometry and +DSv4-KV seams. W3 and W5 rebase; they do not fork. + +**R7 — vLLM #53906 may merge mid-campaign.** That is the good case and it +changes the oracle. §Stop conditions says what to do; the risk is that a wave +in flight silently keeps using transformers as the mirror source after vLLM +becomes authoritative. + +**R8 — 64 KDA heads fall off every AOT specialization.** A correctness-complete +model at an indefensible decode speed. Named as a residual for W8, not +designed around now. + +**R10 — image and video share token id 154854.** The processor emits the image +id for video frames and disambiguates by `<|begin_of_video|>` / `<|end_of_video|>` +span. A port that keys off `video_token_id` classifies every frame as an image +and still produces output. W6 gates on a placeholder-expansion golden that +contains both an image and a video in one prompt. + +**R11 — the eps values are the parent classes' defaults in five places and this +model overrides them.** The table in §Port map lists them. Each one is a +plausible default that produces a slightly-off model rather than a failure. + +**R9 — the campaign can complete without ever being token-gated.** This is not a +risk to mitigate; it is the recorded state of the world (§Gates). The risk is +that a later reader forgets it. Hence §Owed. + +### Decisions taken in this spec + +**D-a. One matrix row, not three.** vLLM #53906 would register +`Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` and +`Glm5NextMTPModel`. None of the three is registered at any vLLM revision today, +and the only architecture any published artifact declares is +`Glm5NextForConditionalGeneration` (`config.json`, read 2026-08-26). One row is +what is checkable. If #53906 merges, the text-only and MTP arms get their own +rows in the change that reconciles the pin, and the `MODEL` ratchet moves then. + +**D-b. transformers is the mirror source for this row, and only until vLLM +implements it.** Not a preference — it is the only admissible oracle. Every +port-map cell cites transformers. The moment vLLM registers `glm5_next`, this +row reconciles onto vLLM per AGENTS.md §"When vLLM has no implementation", the +lane pin expires, and the spec records the change. + +**D-c. Pull-request shape: separate spec and implementation.** Not asked of the +developer. The AGENTS.md split case applies on its face — this pull request +deliberately adds a row, an issue and a spec with no product code, and the +campaign wants scope agreement before eight implementation waves start. Recorded +so it is not re-derived, and NOT written into +`.agents/developer-preferences.md`, because an inference must not be filed where +a developer answer is read. + +## Evidence required + +Per wave, before its pull request is opened: the RED capture of its named +red-first test, the focused ctest green after, a full `agent-preflight.sh` whose +**verdict line** is read (not its exit code), and for W3/W5-W7 the exact build +and run recipe, device, driver, and contention state. W7 additionally records +the artifact's sha256, the exact conversion recipe, and peak RSS. + +## Stop conditions + +- **If vLLM #53906 merges**, stop the wave in flight at its next commit + boundary, reconcile this spec onto vLLM as the mirror source, and record the + change. Do not advance the parity pin as part of that reconciliation without a + separate row: 348 commits of reconciliation is its own unit of work. +- **If a wave finds an upstream anchor stale**, correct the port map in the same + change. Do not port from memory. +- **If the tiny-shape reference cannot be constructed** (the config refuses a + small shape, or a component hard-codes a dimension), return `NEEDS_DECISION`. + That would remove the campaign's only numerical oracle and the plan must + change rather than proceed hand-derived. +- **If W7's arithmetic does not survive contact** — the Q2_K arm does not fit + once real KV and activations are measured — stop and re-plan the arm. Do not + reach for a smaller i-quant, because R4 says it is not producible. +- **No wave takes a GPU lease without `rc`**, and no wave downloads a large + asset without explicit developer authority. + +## Owed + +Debts this row carries, each visible rather than waived: + +- **O1 — no end-to-end token gate exists or can exist on this fleet** for + `Glm5NextForConditionalGeneration`. Owed against a device that can hold 306 + GiB, or against a multi-device execution path this project does not have. + Tracked by [#1998](https://github.com/mudler/vllm.cpp/issues/1998). +- **O2 — the MTP head** (`num_nextn_predict_layers: 1`, + `index_share_for_mtp_iteration: true`) is not implemented and the nextn tail is + skipped, following `glm4_moe_lite_registry.cpp:21-26`. +- **O3 — the text-only arm** `Glm5NextForCausalLM` has no row and no + implementation; it is not declared by any published artifact today. +- **O4 — no llama.cpp floor and no llama.cpp oracle** for the GGUF arms (D6). +- **O5 — no i-quant arm is producible on this fleet** (R4). +- **O6 — speed.** No number on any axis, and no denominator exists. + +## Now + +`READY`, 2026-08-26. The spec and its records are committed; no product code has +landed. The next action is to claim W0 or W1 on a fresh +`row/MODEL-MM-GLM53-FLASH-W` branch. diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index 98832fae8..ec3a2ab6e 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -146,7 +146,24 @@ # algorithm source is transformers#48337, so the at-the-pin static invariants # (324/373/356/310/261) are UNCHANGED. Bumped because one row EXISTS, never to # make a transition pass. - "MODEL": (AGENTS / "model-matrix.md", 378), + # 379 since 2026-08-26, and RE-DERIVED off the matrix rather than carried + # forward: +1 for `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` + # (`Glm5NextForConditionalGeneration`, `zai-org/GLM-5.3-Flash`), landing + # `READY` with its spec committed (#1998). ONE row and not three, which is + # the arithmetic this comment exists to justify: the OPEN vllm#53906 would + # register `Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` AND + # `Glm5NextMTPModel`, so the IndexTTS-2.5 / dots3-note two-row shape is the + # tempting read. It does not apply. None of the three names is registered at + # ANY vLLM revision, and the only architecture a published artifact declares + # is `Glm5NextForConditionalGeneration` -- the MTP head is `layers.45` inside + # the same checkpoint, which the transformers reference discards outright at + # `modular_glm5_next.py:1235`. Beyond-pin in the same strongest sense as the + # qwen4-exp row above: read live 2026-08-26, `git grep "Glm5\|glm5_next"` + # returns zero hits at `555967922` and at `origin/main` = `c71f6f8a81`. Its + # Upstream cell therefore carries no pinned module/class target and the + # at-the-pin static invariants (324/373/356/310/261) are UNCHANGED. Bumped + # because one row EXISTS, never to make a transition pass. + "MODEL": (AGENTS / "model-matrix.md", 379), # 82 since 2026-07-21: +`QUANT-NVFP4-CT-W4A16` (compressed-tensors NVFP4A16 / # W4A16 — NVFP4 weights with BF16 activations, distinct from the existing # `QUANT-NVFP4-CT-W4A4` and `QUANT-NVFP4-MO-W4A16` rows in both scheme diff --git a/tests/scripts/test_agent_record.py b/tests/scripts/test_agent_record.py index 2b94772e3..c3954bfde 100644 --- a/tests/scripts/test_agent_record.py +++ b/tests/scripts/test_agent_record.py @@ -646,6 +646,51 @@ def test_qwen4_exp_row_is_inside_the_model_ratchet(self) -> None: siblings = [row for row in rows if "qwen4-exp" in row.item_id] self.assertEqual([row.item_id for row in siblings], [item_id]) + def test_glm5_next_row_is_inside_the_model_ratchet(self) -> None: + """The #1998 row and the 378 -> 379 bump are one semantic change. + + Same contract as the qwen4-exp test above, and it guards the arithmetic + against a *stronger* pull toward two-or-three. dots3-note and + IndexTTS-2.5 each moved this pin by TWO because vLLM registers two + architectures for what prose calls one model, and the OPEN vllm#53906 + would register THREE for GLM-5.3-Flash: `Glm5NextForCausalLM`, + `Glm5NextForConditionalGeneration` and `Glm5NextMTPModel`. It still + moves by ONE, because none of the three is registered at any vLLM + revision and the only architecture a published artifact declares is + `Glm5NextForConditionalGeneration`. The MTP head is `layers.45` inside + the same checkpoint -- the transformers reference discards it at + `modular_glm5_next.py:1235` -- not a separately registered architecture, + so there is no `MODEL-SPEC-glm5-next-*` row and there must not be one + until vLLM registers one. + + What this catches that nothing else does: renaming the row, or adding a + second or third glm5_next row to "match" the upstream PR, both leave the + count reachable by a compensating edit elsewhere in the matrix while + every other check stays green. Only an assertion that names the row goes + red. + + `READY` is pinned deliberately and is the weaker half of the evidence, + stated rather than implied, for the same reason the qwen4-exp test gives: + the row is `READY` because its spec is committed and no product code has + landed, and pinning it here means a future refactor of the + structured-spec or claim-ownership rules cannot silently take this pin + with it. + """ + errors: list[str] = [] + rows, _ = agent_record.check_matrices(errors) + self.assertEqual([error for error in errors if "MODEL rows" in error], []) + + item_id = "MODEL-MM-glm5-next-glm5-next-for-conditional-generation" + found = [row for row in rows if row.item_id == item_id] + self.assertEqual(len(found), 1, item_id) + self.assertEqual(found[0].path.name, "model-matrix.md", item_id) + self.assertEqual(found[0].field("state").strip().strip("`"), "READY", item_id) + + # One row, not three: neither the text-only arm nor the MTP head has a + # sibling row, and adding one to mirror the upstream PR is the mistake. + siblings = [row for row in rows if "glm5-next" in row.item_id] + self.assertEqual([row.item_id for row in siblings], [item_id]) + def test_recipe_backfill_rows_are_inside_the_model_ratchet(self) -> None: """The #609/#610 rows and the 362 -> 369 bump are one semantic change. From 56623c4e0dabe054730b23e49042f44187fe3db6 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 20:15:05 +0000 Subject: [PATCH 2/5] spec(MODEL-MM-GLM53-FLASH): split W7, because the pin already carries every GGUF convention the converter needs (#2011) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit W7 as written was one wave that needed a GPU and a 300-600 GiB download, which made the converter -- the thing every GPU gate on this row is waiting for -- unreachable until somebody granted a large asset. The two halves have different blockers, so this splits them: W7a is the converter and its synthetic-fixture gate, CPU-only and needing no checkpoint at all; W7b is the artifact and keeps the download authority W7 always required. Issue #2011 owns W7a. Three findings move what a later wave should believe, and all three came from reading rather than guessing. **D6 was too strong.** llama.cpp still implements no `glm5_next` -- re-verified at `origin/master` `539f24529` fetched 2026-08-26 and at our pin: the enumerators are `LLM_ARCH_GLM4`, `LLM_ARCH_GLM4_MOE`, `LLM_ARCH_GLM_DSA` (`src/llama-arch.h:86-88`), and `src/models/glm-dsa.cpp` is GLM-5.2, citing `zai-org/GLM-5.2/blob/main/config.json`, a different model. But "no implementation" is not "no convention", and every convention this converter needs is present AT `b10451`: `class KDA` with `{arch}.kda.head_dim` and `{arch}.kda.gate_lower_bound` (`gguf-py/gguf/constants.py:262-264`), the KDA tensor spellings including the three separate `ssm_conv1d_q/k/v` this checkpoint's packing needs (`src/llama-arch.cpp:465-479`), the HF module paths that map onto them (`gguf-py/gguf/tensor_mapping.py:896-933`, Kimi-Linear's paths being GLM-5.3-Flash's verbatim), and the indexer names including the k-pool compressor (`:626-636`). `KDA.SAFE_GATE` is the only member that is `master`-only, and this model declares no `safe_gate`. **No pin advance is required by any part of W7, and none is owed.** **Upstream Python cannot quantize at all.** `gguf.quants.Q2_K` implements `dequantize_blocks` and no `quantize_blocks`; the encoders live only in `ggml/src/ggml-quants.c`. So W7a ports them and gates them byte-for-byte against the pinned C reference rather than against a tolerance, which is recorded here because "we could not use gguf-py" is otherwise the kind of claim a reader has to re-derive. **The arm arithmetic moved, and the table it replaces was the weaker kind.** The §Hardware table is bits-per-weight times a parameter count. The converter resolves a type per tensor, so running its own resolver over the real topology is the arithmetic that will be written: 1719 tensors carrying 313,890,512,702 parameters, which is 321.32B less the 7.43B MTP block and therefore an independent confirmation that the skip is exactly the 2.31% §Port map measured. The Q2_K arm is **100.35 GiB, not 102.6**, because the old table stated every figure including layer 45. Against ~119.63 GiB at 128K context and one sequence that leaves ~17.7 GiB after 1.43 GiB of KV and 0.14 GiB of KDA state. Still arithmetic and not measurement: #1963 and #1966 record this accounting being wrong by 48x, and W5 re-derives it from the runner. Three debts are added rather than waived. O7, no artifact exists and what producing one needs is named. O8, the Q3_K/Q4_K/Q5_K encoders are not ported and the converter refuses those arms. O9, the emitted file is not loadable by this tree because `glm5next` has no `general.architecture` dispatch entry -- that wiring is W1's, and it is written down because W7a lands a capability a production entry point does not yet reach. Spec and records only. The converter is not in this commit. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-fable-5 [Claude Code] --- .agents/claims/CLAIM-GLM53-FLASH-W7A.md | 5 + .agents/issue-index.md | 1 + .agents/specs/glm5-next-flash.md | 145 ++++++++++++++++++++++-- 3 files changed, 139 insertions(+), 12 deletions(-) create mode 100644 .agents/claims/CLAIM-GLM53-FLASH-W7A.md diff --git a/.agents/claims/CLAIM-GLM53-FLASH-W7A.md b/.agents/claims/CLAIM-GLM53-FLASH-W7A.md new file mode 100644 index 000000000..5c14ddfac --- /dev/null +++ b/.agents/claims/CLAIM-GLM53-FLASH-W7A.md @@ -0,0 +1,5 @@ +# CLAIM-GLM53-FLASH-W7A + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-GLM53-FLASH-W7A` | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` (`ACTIVE`, advanced from `READY` by this claim: the first product code on the row) | fresh implementer, helper role per `scripts/agent-role.py`, working from the committed spec `.agents/specs/glm5-next-flash.md` §W7a | isolated worktree `/home/mudler/_git/vllm.cpp-glm53w7a`; CPU only, no `rc` lease taken and no GPU touched | `row/MODEL-MM-GLM53-FLASH-W7A`, base `row/MODEL-MM-GLM53-FLASH` at `78d2c8a61` (the spec commit, PR [#2001](https://github.com/mudler/vllm.cpp/pull/2001), not yet on `main`), issue [#2011](https://github.com/mudler/vllm.cpp/issues/2011) under campaign issue [#1998](https://github.com/mudler/vllm.cpp/issues/1998) | Owns ONLY the safetensors→GGUF converter and its synthetic-fixture gate: `scripts/convert-glm5-next-gguf.py`, `tests/scripts/test_convert_glm5_next_gguf.py`, the frozen llama.cpp `b10451` k-quant golden, the `docs/USAGE.md` weights-provenance entry, and the spec/record edits those invalidate. EXCLUDES artifact production (O7 — needs the 300–600 GiB checkpoint, developer authority for the download, and a box), the loader-side `glm5next` registration and `general.architecture` dispatch (W1, O9 — the emitted file is not loadable until then), every model-side primitive (W1–W6), the Q3_K/Q4_K/Q5_K encoders (O8), any i-quant arm (R4/O5), and any pin advance: nothing in this claim moves `.agents/upstream-sync.md` or `.agents/oracles/*` | `ACTIVE` | 2026-08-26 — converter landed, gated on synthetic fixtures with the k-quant encoders byte-identical to the pinned llama.cpp reference; artifact owed as O7 | diff --git a/.agents/issue-index.md b/.agents/issue-index.md index b89302e6e..fdb277ab9 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -746,3 +746,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1934](https://github.com/mudler/vllm.cpp/issues/1934) | `BACKEND-ROCM` | `RocmPlatform::needs_weight_staging()` is stale-false (a W0-era placeholder never revisited despite #523/#509/#506/ROCM_ATTN/hipGraph landing since), so `CheckDeviceWeightFit` — the #1123/#1870 load-time refusal, including the `policy_forces_full_expand` fix — never runs on ROCm: measured directly, `VT_DEVICE_WEIGHT_BUDGET_BYTES=1` produced no refusal on a real load. The actual device allocation the refusal guards is not gated on this flag, so #1870's crash stays reachable until this closes; owed, not fixed in flow, because flipping the flag also moves `DirectDeviceLoadEligible` and several GDN kernel-dispatch defaults that each need their own correctness check | bug | | [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-QWEN4-EXP` | **`Qwen/Qwen3.8-Flash-Next` declares `Qwen4ExpForConditionalGeneration` / `qwen4_exp`, a new architecture vLLM does not implement, so the port runs on a split oracle: transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH.** Released 2026-08-24, 180B total / 6B activated, image-text-to-text. The `Qwen3.8` in the name is marketing continuity: `.agents/specs/qwen38-27b-bf16-gate.md`'s "one config key differs" precedent does NOT extend here. Read live 2026-08-26 at vLLM `origin/main` = `6a5e8f5979`: no `qwen4*` path, no registry entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. That is absence from vLLM `main` rather than staleness in our pin `555967922`, so a pin advance does not reach it. What exists is transformers [#48337](https://github.com/huggingface/transformers/pull/48337) "Add Qwen4Exp model", MERGED 2026-08-26, 5211 lines, and SGLang [#36497](https://github.com/sgl-project/sglang/pull/36497), still OPEN and therefore inadmissible. **Developer direction 2026-08-26, recorded verbatim: "use transformers as oracle for algorithmic side. but use ops from vllm so we account for optimized path."** Justified rather than convenient: `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so porting it as written yields a correct model at an indefensible speed, while AGENTS.md's mirror-vLLM polarity still binds every primitive vLLM implements. `Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`), all of which this tree has; GDN is an exact match for our AOT gate (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). The delta is four things, and **exactly two have no vLLM op at all**: the PLE dilated depthwise conv (kernel 4, dilation 3; `git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the n-gram hashed embedding. **The survey's load-bearing finding, and it REVERSES this row's first reading: QSA's structural twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The original call was that QSA, being plain GQA rather than MLA, had to map onto vLLM's non-MLA block-sparse case; that reasoning rested on treating `MLAAttentionSpec` as an MLA claim, and **it is not one** — M3's own indexer cache uses it while M3 is a plain-GQA model, with the comment "Key-only: MLAAttentionSpec budgets one vector/token (not 2x for K+V)". It is a budget shape. Remove that prop and the GQA-vs-MLA argument collapses. Verified line by line at `6a5e8f5979`: **nine independent structural matches with DSv4**, `compress_ratio == 4` literally the same number — MQA index with 1 key head at dim 128; `relu(q.k)` summed over index heads vs `(score.relu() * weights).sum(dim=0)`; `1/sqrt(head_dim)`; one score set per query token with no head axis vs `topk_indices_buffer[num_tokens, topk]`; pooling boundary `(position+1) % COMPRESS_RATIO == 0`; RMSNorm on the pooled key; **RoPE at the block-start position** vs `compressed_pos = (position // CR) * CR`; candidate count `visible // compress_ratio`; and one stored state per 4 tokens via `MLAAttentionSpec(tokens_per_state=compress_ratio)`, a first-class KV field documented as "Ints > 1 compress multiple tokens into one state (DSv4 sparse MLA)" that has no M3 equivalent. **M3 is a DIFFERENT ALGORITHM**, not a worse fit: its score is `tl.max(qk, axis=1)` over 128 RAW token dots with no pooling, no relu and no head reduction, it asserts `num_idx_heads == num_kv_heads` ("no topk index reduce") so it emits one block set PER KV HEAD, and its `SPARSE_BLOCK_SIZE = 128` is welded to the KV page size ("One sparse block == one KV page") on both the score and the attend side — moving it to 4 forces a page size of 4 and breaks `tl.dot`, whose tile needs >= 16. M3 contributes exactly ONE thing and it is a wiring precedent, not an algorithm: that a plain-GQA model can own a key-only side cache through `MLAAttentionSpec` and a private indexer backend. **The genuinely new work is the CONSUMER and nothing upstream supplies it** — every DSv4 sparse consumer attends to COMPRESSED MLA KV (one state per 4 tokens) and M3's attend to raw tokens only at page granularity, while QSA attends to RAW tokens selected at ratio-4 granularity. Two silent-failure traps follow: wiring QSA's top-k into a DSv4 sparse-MLA consumer attends a POOLED key/value and still emits plausible tokens, and **a short-prompt token gate cannot catch it because at context <= `indexer_budget` 2048 every candidate is selected** — so any QSA gate must run past 2048 tokens of context, which is now a stated `## Gates` requirement; and `SparseAttnCompressNormRopeStoreC4Kernel` does NOT mean-pool despite its name — it is a learned softmax pool over an OVERLAPPING window of 8 using a score channel this checkpoint does not have, and the CuteDSL variant refuses `overlap=False` at compile, so the **Triton** `head_dim=128` variant is the correct starting point. Two structural consequences beyond the module list: the residual stream is `hc_count * hidden_size` = **4 x 2560 = 10240 wide through the whole stack** with a low-rank read gate and per-branch scalar write gate around both attention and MLP, which is a change to the per-layer loop and every residual buffer rather than a drop-in module; and `number_of_conv_states = 3` on a PLE layer (GDN conv, PLE conv, n-gram token history) plus the indexer side cache, adjacent to [#1963](https://github.com/mudler/vllm.cpp/issues/1963) and [#1966](https://github.com/mudler/vllm.cpp/issues/1966). **NOTHING PUBLISHED FITS**, read live from the HF API against ~119 GB usable on GB10: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB (NVFP4 backbone with the n-gram table left at FP8, 51.2 GB) and `unsloth/...-GGUF` is a README with ZERO weight files. No GGUF exists and no tool can make one, because llama.cpp has no `qwen4_exp` either, so the standing k-quant requirement means authoring the arch on our side AND states that the quantized arms have NO llama.cpp oracle. **The chosen arm does NOT load today, and the blocker is neither the offload nor the budget: this tree cannot keep a gather table quantized at all.** `KeepQuantKDim` returns `-1` for `GgufTensorRole::kEmbeddingTable` (`src/vllm/model_executor/model_loader/gguf_keep_quant.cpp`), and `qwen3_5_gguf_weights.cpp` asserts it by name — "the embedding table cannot keep quant blocks" — so a Q4_K or Q8_0 n-gram table EXPANDS to bf16 and 51.2B params become **102.4 GB of anonymous memory**; the arm dies before the first forward. The reason was already sitting in a header comment ("a gather, not a GEMM ... A quantized-gather op is a follow-up row") and **no such row exists**. The only non-expanding gather residency is `kKeepF16`, requiring ggml type 1 exactly (102.4 GB on disk) and CPU-ONLY, because `EmbeddingKernelCuda` refuses anything but f32/bf16. **Second blocker:** `moe_intermediate_size = 640` makes `ffn_down_exps` Q4_K-illegal on its reduction dim (640 % 256 = 128), as does `hc_lowrank = 320`; llama.cpp's substitution is believed to be Q5_0 (**UNVERIFIED, owed against the pinned llama.cpp oracle**) and the dependent fact IS verified in-tree — our reader knows ggml ids `0,1,2,8,10..14,16,18,19,22..28,30,39,40,41,66` and has **no entry for 3, 6, 7 or 20**, so a stock `llama-quantize -Q4_K_M` file fails at header parse. We author the converter, so the fix is Q4_0 (block 32, same 4.5 bpw). **`ENG-WEIGHT-OFFLOAD` will not help** — it moves zero bytes today (`ConsiderWeight` has no production callers, pinned by a test) and is documented inert on GB10; the tier that DOES work already ships and is proven by the 2.4T model serving 369.97 GiB from a 119.631 GiB box at ~62 GiB resident: mmap `MAP_PRIVATE`, borrow in place, alias the host pointer, `prefault: false`. Corrected sizing: backbone ~67.7 GiB, whole process ~73.5 GiB of 119.631 at 32K single-stream, ~46 GiB of headroom for the page cache, so the ~76 GB estimate was right within 10%. The design works because per-token demand is **<= 64 KiB of reads** (16 lookups x 160 dims over at most 16 pages) against the 2.4T expert lane's 6.95 GB/token. The architecture supplies its own lever: the per-token n-gram cost is `(ngram_size-1)*heads_per_ngram` = 16 lookups of 160 dims, so **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per token** and making it non-resident is the intended design point (RadixArk reached the same split independently). Sizing arithmetic, NOT measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB (yes, ~10 GB left for KV), Q4_K_M backbone with the table non-resident ~76 GB. GB10 is UNIFIED memory so "offload to host" is not a move there; non-resident means disk-backed, and its cost is unmeasured. **Two decisions were put to the developer as explicit accept-or-reject and BOTH are settled 2026-08-26, recorded in place rather than left open.** (1) `.agents/oracles/transformers.md` pins transformers to 5.14.1, deliberately tied to what the pinned vLLM environment resolves so the environment cannot hold two `transformers` at once, and **5.14.1 does not contain `Qwen4Exp`**; the lane-scoped second pin is **ACCEPTED**, on the argument that the invariant guards a vLLM environment against drifting from its transformers and here there is no vLLM implementation to drift from, and it expires the moment vLLM registers `qwen4_exp`. **The lane pin is a real release, not a branch SHA**, which was not the expected outcome: `Qwen4Exp` merged to `main` at 12:03:40Z on 2026-08-26 and `v5.16.0` published at 12:35:15Z, and this was BOUNDED rather than assumed by fetching `models/qwen4_exp/modeling_qwen4_exp.py` at each tag — `v5.16.0` HTTP **200**, `v5.15.0` HTTP **404** — making 5.16.0 the FIRST release carrying the architecture and therefore the tightest available pin. The version string is UNMEASURED (it is the release proven to contain the model, not a `transformers.__version__` read off a running oracle) and `gateable` stays `no`. (2) The first runnable arm is the **Q4_K_M backbone with the n-gram table NON-RESIDENT** (~76 GB). Q8_0 was raised and does not fit at ~191 GB, and no partial-Q8 split reaches 119 GB with the backbone at 8 bits; Q4_K_M-throughout fits on paper at ~109 GB but leaves ~10 GB for KV and activations on a 262144-native-context model, which is not a margin. This promotes the non-resident table from a note to a first-class W6 deliverable, and it is NOT free: GB10 is UNIFIED memory, so the existing host-pinned offload seam (`ENG-WEIGHT-OFFLOAD`, mirroring vLLM's `cpu_offload_gb`) does not by itself solve it there and the mechanism must be disk-backed or genuinely unloaded — established before it is designed around. Spec: [`specs/qwen4-exp-flash-next.md`](specs/qwen4-exp-flash-next.md). No product code lands under the spec pull request | feature | | [#1998](https://github.com/mudler/vllm.cpp/issues/1998) | `MODEL-MM-GLM53-FLASH` | **`zai-org/GLM-5.3-Flash` declares `Glm5NextForConditionalGeneration` / `glm5_next`, an architecture vLLM implements at NO revision, whose only admissible oracle is transformers and which NOTHING on this fleet can execute.** Read live 2026-08-26: `git grep "Glm5\|glm5_next"` returns ZERO hits at the parity pin `555967922` AND at vLLM `origin/main` = `c71f6f8a81` (2026-08-26 08:42 -0700), whose `vllm/models/` package holds `common, deepseek_v32, deepseek_v4, dots3_note, inkling, kimi_k3, minimax_m3` and no `glm5next`. That is ABSENCE from vLLM `main`, not staleness in our pin, so a pin advance — 348 commits of reconciliation — reaches nothing here. [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) "[Model] add GLM-5.3-Flash support" (opened 2026-08-26T14:12:00Z, head `933876c388fb`, 85 files, +12,511/-540) would register `Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` and `Glm5NextMTPModel`, and is **OPEN, unmerged, `mergeable: false`** — an unmerged PR is not a revision and is INADMISSIBLE as an oracle, the same rule this index already applied to SGLang #36497 on the qwen4-exp row; it is cited only as evidence of upstream shape. SGLang implements nothing either (PR #36507 open; code search 0) and the trap there is that its **cookbook** PRs #36440 and #36513 MERGED at 14:00:16Z and 14:39:29Z — documentation landed ahead of the code, so the model card links a recipe SGLang `main` cannot run. llama.cpp implements nothing (code search 0; PR #27752 open; our oracle pinned at `b10451`), and vllm-omni implements nothing. **The sole admissible reference is transformers**, implementing commit `eb4d9e2a64` (2026-08-26T14:26:40Z, transformers#48342), and the first RELEASE carrying it is **`v5.16.1`** — bounded rather than assumed by fetching `models/glm5_next/modeling_glm5_next.py` at each tag: HTTP **200** at `v5.16.1`, **404** at `v5.16.0`, **404** at `v5.15.1`. Our transformers pin is 5.14.1, so this row needs a lane-scoped second pin expiring when vLLM registers `glm5_next`; note it is `5.16.1` where the `MODEL-MM-QWEN4-EXP` lane is `5.16.0`, one day apart because `Qwen4Exp` merged before the 5.16.0 cut and `Glm5Next` after it — two lanes, two releases, which is what a lane pin is for. **Not the blocked GLM-5 row:** `MODEL-TEXT-deepseek-v2-glm-moe-dsa-for-causal-lm` (`GlmMoeDsaForCausalLM`, `BLOCKED`, `CLAIM-GLM-DSA-LATEST-DEEPSEEK`) is DeepSeek-V3.2 verbatim at 753.9B; `glm5_next` is a different `model_type`, a different parameter count, a different attention topology and has a vision tower. **The architecture:** 45 layers, hidden 4096, vocab 154880, 1,048,576 context, `layer_types` = 34 `linear_attention` (KDA) + 11 `deepseek_sparse_attention` at 3,7,…,43; MoE 288 routed + 1 shared, top-8, `moe_intermediate_size` 2048, sigmoid/`noaux_tc`, `n_group=topk_group=1` (which makes the group stage a NO-OP), `routed_scaling_factor` 2.5, clamped SwiGLU `swiglu_limit` 10.0 in FIVE places; mHC `hc_mult` 4 / `hc_sinkhorn_iters` 20 / `hc_eps` 1e-6; vision 24L/1024/patch **14**/merge 2/temporal 2/out 4096. **The text stack has NO rotary anywhere.** `qk_rope_head_dim: 0`, `mla_use_nope: true`, no `rope_theta`, no `rope_scaling`; the reference *deletes* the inherited `rope_parameters` field and passes `position_embeddings=None` to every layer, and **`indexer_rope_interleave: true` is a VESTIGIAL flag** the indexer override ignores — implementing it because the config names one is implementing a bug. Position comes only from the KDA decay and short convs. **HEAVY REUSE — and the two obvious reuses are WRONG in one place each, in ways a token gate cannot see.** (1) The KDA forget gate takes the OTHER branch: `gate_lower_bound: -5.0` is not None, so it computes `-5.0 * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias))`, whereas our `src/vllm/model_executor/models/kimi_kda.cpp:60` implements Kimi-Linear's `-exp(A_log) * softplus(g + dt_bias)`. Different functions of the same inputs, both smooth, both negative, both fluent — and the sign of `decay_rate` differs too, so getting it wrong flips decay to growth. (2) The mHC head collapse is `hidden_streams.mean(dim=2)`, an UNWEIGHTED mean whose own docstring says "Unlike DeepSeek-V4", where our `src/vllm/model_executor/models/deepseek_v4_mhc.cpp:168` `HcHeadCollapse` is the V4 weighted-gated collapse; the checkpoint carries **no `hc_head.*` tensors at all**, which settles it. **(3) The k-pool indexer is unobservable at or below 2048 tokens:** with `index_topk: 2048` a shorter context selects every candidate, the selection is the identity, the pooling is invisible and any short-prompt gate passes over a broken indexer — the same shape as the trap recorded on the qwen4-exp row. The pooling is **learned, per channel, not a mean**: 128 independent 4-way softmaxes over a learned `[4,128]` intra-pool position embedding plus a cached `[128,4096]` gate score, pools aligned to the first VALID token (so the grid is batch-dependent), a pool must be COMPLETE to be a candidate and is visible iff its LAST member is, `select_k = min(index_topk/index_kpool, P) = min(512, P)`, expanded back to raw token indices and widened to `index_topk + kpool - 1` = **2051** by an unscored ragged tail. Its cache is 257 floats/token/layer, not the parent's 128. **Structural blocker:** `MlaBlockDims::Validate` REFUSES `qk_rope_head_dim == 0` (`src/vllm/model_executor/layers/attention/mla_attention.cpp:90-93` requires every dim > 0, `:95-99` requires it even); Kimi-Linear is NOT the same case — it sets `mla_use_nope` while keeping `qk_rope_head_dim: 64`, so the rope slice still occupies the cache row (`include/vllm/model_executor/models/kimi_linear.h:86,:88`). NoPE does buy one thing: MLA weight absorption becomes trivially valid, since there is no rope slice to keep out of it. **NOTHING FITS, and no GGUF exists.** Measured live by HTTP RANGE over all 62 safetensors headers, payload never fetched: 76,108 tensors, `F8_E4M3` 314,396,639,232 + `BF16` 6,926,096,640 elements = **321,322,735,872 real parameters** plus 19,484,766 F32 block scales; routed experts are **311.65B = 97.0%**, so any GGUF arm's mixed bpw is the experts' bpw to within a percent. On disk against ~119.63 GiB on GB10: `zai-org/GLM-5.3-Flash` FP8 e4m3 block-128x128 **305.78 GiB (2.56x over)**, `zai-org/GLM-5.3-Flash-BF16` **598.53 GiB (5.00x)**, `unsloth/GLM-5.3-Flash-FP8` 305.82 GiB, `LibertAIDAI/GLM-5.3-Flash-NVFP4` **181.32 GiB (1.52x)**. **All four repos named `*-GGUF` contain ZERO `.gguf` files** (`unsloth`, `AtomicChat`, `aj9o9`, `vcruz305` — READMEs and images); a repository name is not an artifact. **`gateable = no` on MEMORY, and the consequence is the scoping result: no oracle can execute this model on any device this project reaches, so NO end-to-end token gate exists or can exist here** — recorded as visible debt, not waived. What replaces it is a tiny-shape CPU reference oracle instantiated from the pinned transformers, which is a real oracle for the NUMERICS and is explicitly NOT oracle gateability for the model. Developer-set W7 target is a low GGUF quant that fits the DGX; arithmetic (not measurement) with experts at the named type and the other 3% at Q6_K, bpw taken from our own reader's block traits at `src/vllm/model_executor/model_loader/gguf_reader.cpp:200`: Q8_0 318.0 GiB, Q6_K 245.5, Q4_K 170.6, IQ3_XXS 118.1 (~1.5 GiB left — not a margin), **Q2_K 102.6 (~17 GiB left)**, IQ2_S 100.3, IQ2_XXS 82.3, IQ1_S 64.0; KV is small by design at ~11.7 KiB/token (1.5 GiB at 128K, ~11.4 GiB at the full 1M) though #1963/#1966 mean that arithmetic is re-derived against the runner, not trusted. **Q2_K is the recommended first arm and the reason is producibility, not quality: i-quants need an importance matrix, an importance matrix needs a forward pass, and a forward pass needs 181 GiB — the dependency is CIRCULAR on this fleet.** K-quants need none. And since llama.cpp has no `glm5_next`, the converter is ours to author, with no reference GGUF to diff against and no llama.cpp floor for the arm. Layer 45 is the MTP block — 888 tensors, 7.43B params, 2.31% of the model, DSA/MLA rather than KDA and carrying no `hc_*` tensors — present in the checkpoint and discarded by the reference at `modular_glm5_next.py:1235`; owed, not implemented, and skipping it in the converter is ~2.3 GiB of headroom. Live seam contention to rebase onto rather than fork: [#1971](https://github.com/mudler/vllm.cpp/pull/1971) (DSA geometry) and [#1977](https://github.com/mudler/vllm.cpp/pull/1977) (DSv4 KV multicache). Spec: [`specs/glm5-next-flash.md`](specs/glm5-next-flash.md), 8 waves, W0-W2 and W4 CPU-gateable and W3/W5-W8 needing a GPU. No product code lands under the spec pull request | feature | +| [#2011](https://github.com/mudler/vllm.cpp/issues/2011) | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | **GLM-5.3-Flash has no GGUF, no upstream tool can make one, and upstream Python cannot even quantize: author the `glm5_next` converter and gate its k-quant encoders BYTE-FOR-BYTE against llama.cpp `b10451`.** W7a of [#1998](https://github.com/mudler/vllm.cpp/issues/1998); spec [`specs/glm5-next-flash.md`](specs/glm5-next-flash.md) §W7a. Every GPU gate on this row is blocked behind an artifact that does not exist: measured live 2026-08-26 against ~119.63 GiB usable on `dgx:gpu0`, the published arms are FP8 **305.78 GiB**, BF16 **598.53 GiB** and `LibertAIDAI/...-NVFP4` **181.32 GiB**, and all four repositories named `*-GGUF` contain **ZERO `.gguf` files**. **D6 holds but was too strong, and the correction is what made this wave tractable.** llama.cpp implements no `glm5_next` — re-verified at `origin/master` `539f24529` (fetched 2026-08-26) and at our pin: the enumerators are `LLM_ARCH_GLM4`, `LLM_ARCH_GLM4_MOE`, `LLM_ARCH_GLM_DSA` (`src/llama-arch.h:86-88`) and `src/models/glm-dsa.cpp` is **GLM-5.2**, citing `zai-org/GLM-5.2/blob/main/config.json`, a different model. **But every GGUF convention this converter needs is already present AT `b10451`, so no pin advance was taken and none is owed:** `gguf-py/gguf/constants.py:262-264` carries `class KDA` with `{arch}.kda.head_dim` and **`{arch}.kda.gate_lower_bound`** (`KDA.SAFE_GATE` is `master`-only and GLM-5.3-Flash declares no `safe_gate`, so nothing reaches for it); the KDA tensor spellings `ssm_conv1d_q/k/v`, `ssm_f_a/f_b`, `ssm_g_a/g_b`, `ssm_beta`, `ssm_a`, `ssm_dt`, `ssm_norm` are at `src/llama-arch.cpp:465-479`; `gguf-py/gguf/tensor_mapping.py:896-933` maps them from Kimi-Linear's HF module paths, which are **GLM-5.3-Flash's paths verbatim**; and the indexer names, including `indexer_compressor_ape`/`indexer_compressor_gate` for the k-pool stage, are at `:626-636`. **The one thing upstream cannot supply at any revision is an ENCODER:** `gguf.quants.Q2_K` implements `dequantize_blocks` and **no** `quantize_blocks`, so Q2_K/Q6_K/Q8_0 were ported from `ggml/src/ggml-quants.c` @ `b10451` (`:891`, `:1869`, `:276`, over `make_qkx2_quants:799`, `make_qx_quants:628`, `nearest_int:621`) and gated **byte-identical** against a frozen golden captured from that reference compiled `-ffp-contract=off`. Two traps changed bytes and are recorded in the source rather than left to be re-found: `nearest_int` is the `+12582912.0` add-and-mask trick and rounds half to EVEN, not `round`; and C `roundf` in `quantize_row_q8_0_ref` rounds half AWAY FROM ZERO where `np.rint` rounds half to even, so `np.rint` mis-encodes every exact `.5` — caught by a crafted `[0.5, -0.5, 1.5, ...]` case, `[1,-1,2,-2,3,-3]` expected against `[0,0,2,-2,2,-2]` observed. **Exact arithmetic, from the converter's OWN type resolver run over the real topology** rather than bpw times a parameter count: 1719 output tensors carrying **313,890,512,702** parameters — 321.32B less the 7.43B MTP block, which independently confirms the skip is the 2.31% the spec measured. Q2_K (experts Q2_K, everything else Q6_K) **100.35 GiB** at 2.746 mixed bpw, Q6_K 239.89, Q8_0 310.67, BF16 584.67. Against 119.63 GiB at 128K context and one sequence: KV **1.43 GiB** (11 MLA layers x `kv_lora_rank` 512 x 2 B plus an 11 x 64 B indexer side cache = 11,968 B/token), KDA recurrent state **0.14 GiB** (64 x 128 x 128 x 4 B x 34) plus conv states, leaving **~17.7 GiB** — arithmetic, not measurement, and [#1963](https://github.com/mudler/vllm.cpp/issues/1963)/[#1966](https://github.com/mudler/vllm.cpp/issues/1966) mean W5 re-derives it from the runner. The Q2_K figure is **100.35 and not the spec's 102.6** because that table stated every arm including layer 45. The fallback ladder is `want -> Q8_0 -> F32` rather than `want -> F32`, because with an F32 fallback a FINER arm can come out LARGER than a coarser one and a size table must not have that property; on the real model the F32 sliver is 0.08 GiB (the depthwise conv kernels at `ne0=4`, the 4-D downsample and 5-D patch-embed). The tensor inventory is EXACT, not inferred: the real 8.4 MB `model.safetensors.index.json` (76,108 entries) and shard headers 2/32/62 were read by HTTP RANGE, payload never fetched, confirming the three separate `{q,k,v}_conv1d` convs, `hc_{attn,ffn}_{fn,base,scale}` flat on the layer with **no `hc_head.*` at any layer**, `indexer.k_norm.bias` present (settling LayerNorm-with-bias over RMSNorm), and `index_kpool_compress_{ape,gate}` on 12 layers. Refusals by name for every unimplemented arm: the i-quants because an imatrix needs a forward pass, a forward pass needs 181 GiB, and the dependency is **circular on this fleet** (O5); Q3_K/Q4_K/Q5_K because their encoders are unported and shipping an ungated encoder is worse than refusing (O8); `--keep-mtp` because nothing on the loader side reads an MTP tail (O2); a non-`glm5_next` config; and an FP8 tensor whose `weight_scale_inv` companion is missing, which would otherwise produce a loadable, wrong file. Gate: 50 assertions on a SYNTHETIC tiny-shape checkpoint with an independent in-test GGUF reader — no real checkpoint, no GPU, no C++ build. **NOT REACHED and disclosed:** `glm5next` has no `general.architecture` dispatch entry (`src/vllm/entrypoints/model_loader.cpp:1000`), so the file the converter writes is not loadable by this tree; W1 owns that wiring, tracked as **O9** under #1998. **Owed on close:** O7, artifact production — needs the 300-600 GiB checkpoint staged on local disk, explicit developer authority for the download, and a box with room for source and output at once, plus the sha256, recipe and peak RSS the spec's §Evidence requires | feature | diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index a97a6d268..587b3fad6 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -778,8 +778,8 @@ does `del self.rope_deltas` (`:1429`). | D2 | KDA device path default-OFF | `docs/ENVIRONMENT.md:166`, `:171` | first arm runs the host compose; speed is not a wave-1..5 claim | | D3 | KDA AOT specializations exclude 64 heads | `cuda_gdn.cu:5217-5220`, `:5277-5278` | hand-kernel fallback; a named speed residual, not a blocker | | D4 | DSA device entry points are host-vector | `deepseek_v4_device.h:82-102` | upload/download per call; a named speed residual | -| D5 | No GGUF converter in tree | `scripts/` has none | wave 7 must author one | -| D6 | llama.cpp has no `glm5_next` | code search 0; PR #27752 open | no llama.cpp quant oracle and no floor for the GGUF arms | +| D5 | No GGUF converter in tree | CLOSED by W7a: `scripts/convert-glm5-next-gguf.py` | — | +| D6 | llama.cpp has no `glm5_next` | holds at `master` `539f24529` and at the pin; but the KDA/MLA/indexer KEYS and NAMES it needs are all present at `b10451` | no llama.cpp quant oracle and no floor for the GGUF arms; the CONVENTION is available without a pin advance | | D7 | transformers lane pin at `v5.16.1` | `.agents/oracles/transformers.md` pins 5.14.1 | wave 0 writes the lane pin; `gateable = no` | | D8 | Live seam contention | PRs [#1971](https://github.com/mudler/vllm.cpp/pull/1971) (DSA geometry), [#1977](https://github.com/mudler/vllm.cpp/pull/1977) (DSv4 KV multicache) | waves 3 and 5 rebase onto whichever lands first; do NOT fork the seam | @@ -897,12 +897,77 @@ ids. **Anchors:** `modular_glm5_next.py:1373-1421`, `:1563-1703`, `video_processing_glm5_next.py`. **Nearest ours:** `qwen3_vl_vision.cpp`, `qwen3vl_processor.cpp`. -### W7 — the GGUF converter and the first fitting arm (GPU + large asset) - -Author a safetensors→GGUF converter for `glm5_next` (no upstream tool can do it, -D5/D6) and produce the arm §Hardware names. **This wave needs explicit developer -authority for a large-asset download** and is the only wave that does. -**Exclusions:** no i-quant arm in this wave — see §Risks R4. +### W7 — the GGUF converter and the first fitting arm + +Split into two, because the two halves have different blockers and only one of +them needs a box. The split is recorded here rather than derived again: W7a is +CPU-only and needs no checkpoint, W7b needs 300–600 GiB of staged weights and +explicit developer authority for a large-asset download. + +#### W7a — the converter and its synthetic gate (CPU, large) + +Issue: [#2011](https://github.com/mudler/vllm.cpp/issues/2011). + +Author `scripts/convert-glm5-next-gguf.py` (no upstream tool can do it, D5/D6) +and gate it on synthetic tiny-shape fixtures. **Deliverable:** the converter, the +metadata schema, the tensor-name map, the FP8 e4m3 block dequant, the expert +stacking, the arm table with refusals, and +`tests/scripts/test_convert_glm5_next_gguf.py`. **Exclusions:** no artifact +(O7), no i-quant arm (R4), and no Q3_K/Q4_K/Q5_K encoder (O8). + +Three findings this wave moved, each of which changes what a later wave should +believe: + +**D6 was too strong, and the correction is in our favour.** llama.cpp has no +`glm5_next` — that part holds, verified at `origin/master` = `539f24529` +(fetched 2026-08-26) and at our pin: the arch enumerators are `LLM_ARCH_GLM4`, +`LLM_ARCH_GLM4_MOE` and `LLM_ARCH_GLM_DSA` (`src/llama-arch.h:86-88`), and +`src/models/glm-dsa.cpp` is GLM-5.2, citing `zai-org/GLM-5.2/blob/main/config.json`. +But **everything the converter needs by way of convention already exists AT THE +PIN `b10451`**, and no pin advance is required by any part of this wave: + +- `gguf-py/gguf/constants.py:262-264` @ b10451 already carries `class KDA` with + `{arch}.kda.head_dim` and **`{arch}.kda.gate_lower_bound`**. `KDA.SAFE_GATE` + is the only member that is `master`-only, and GLM-5.3-Flash declares no + `safe_gate` key, so nothing here reaches for it. +- The KDA tensor spellings are at `src/llama-arch.cpp:465-479` @ b10451 — + `ssm_conv1d_q/k/v` (the three separate depthwise convs, exactly the packing + this checkpoint uses), `ssm_f_a`, `ssm_f_b`, `ssm_g_a`, `ssm_g_b`, `ssm_beta`, + `ssm_a`, `ssm_dt`, `ssm_norm`. +- `gguf-py/gguf/tensor_mapping.py:896-933` @ b10451 maps them from Kimi-Linear's + HF module paths, which are **GLM-5.3-Flash's paths verbatim**. +- The indexer names, including `indexer_compressor_ape` and + `indexer_compressor_gate` for the k-pool stage, are at + `src/llama-arch.cpp:626-636` @ b10451. + +**Upstream Python cannot quantize at all.** `gguf.quants.Q2_K` implements +`dequantize_blocks` and **no** `quantize_blocks`; the Q2_K, Q6_K and Q8_0 +encoders exist only in `ggml/src/ggml-quants.c`. They are therefore ported, and +gated byte-for-byte against the pinned C reference over a frozen golden +(`tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json`) rather than +against a tolerance. Two traps changed bytes during the port and are recorded in +the source so the next reader does not re-find them: `nearest_int` is the +`+12582912.0` add-and-mask trick at `:621` and rounds half to EVEN, not +`round`; and C `roundf` in `quantize_row_q8_0_ref` rounds half AWAY FROM ZERO +where `np.rint` rounds half to even, which mis-encodes every exact `.5`. + +**The tensor inventory is exact, not inferred.** The real +`model.safetensors.index.json` (8.4 MB, 76,108 entries) and three shard headers +(shards 2, 32 and 62) were read by HTTP RANGE on 2026-08-26, payload never +fetched. They confirm four things §Port map asserted: the three separate +`{q,k,v}_conv1d` convs; `hc_{attn,ffn}_{fn,base,scale}` flat on the layer with +**no `hc_head.*` tensor at any layer**; `indexer.k_norm.bias` present, which +settles LayerNorm-with-bias over RMSNorm; and `index_kpool_compress_ape` / +`index_kpool_compress_gate` present on 12 layers — the 11 DSA layers plus the +MTP block. + +#### W7b — the first fitting artifact (GPU + large asset). NOT STARTED + +Produce the Q2_K arm and run it. **This wave needs explicit developer authority +for a large-asset download** and is the only wave that does. Owed as O7 with +what it needs named. Its gate is §Gates' W7 row unchanged: the arm loads, the +runner serves it through `include/vllm.h`, and it generates coherent text on a +prompt with an image — a RUN gate, not a token gate. ### W8 — speed, once and only once a correctness gate exists @@ -1064,7 +1129,40 @@ sequence, plus conv states. **This arithmetic is unverified against our own allocator** and #1963/#1966 record that the KV byte accounting has been wrong by 48x before, so W5 re-derives it from the runner rather than from this table. -**Recommended first arm: experts at Q2_K, ~102.6 GiB, with ~17 GiB for KV, +**W7a superseded this table with the converter's own plan, and the numbers +moved in the right direction.** The table above is bits-per-weight times a +parameter count. The converter resolves a TYPE PER TENSOR, so its plan is the +arithmetic that will actually be written, and running its type resolver over the +real topology gives 1719 output tensors and **313,890,512,702 parameters +carried** — 321.32B minus the 7.43B MTP block, which is the independent check +that the skip is exactly the 2.31% §Port map measured: + +| arm | weights | mixed bpw | breakdown | +|---|---|---|---| +| `q2_k` (experts Q2_K, everything else Q6_K) | **100.35 GiB** | 2.746 | Q2_K 93.02, Q6_K 7.17, F32 0.08, Q8_0 0.07 | +| `q6_k` | 239.89 GiB | 6.565 | Q6_K 239.73 | +| `q8_0` | 310.67 GiB | 8.502 | Q8_0 310.58 | +| `bf16` | 584.67 GiB | 16.000 | BF16 584.67 | + +The Q2_K arm is **100.35 GiB, not 102.6**, because the table above stated every +figure including layer 45 and the converter drops it. The F32 and Q8_0 slivers +are the fallback ladder: a k-quant needs `ne0 % 256 == 0`, so a row that does not +divide steps down to Q8_0 (block 32) and only falls to F32 when even 32 does not +divide (the depthwise conv kernels at 4) or the tensor is 4-D or 5-D (the +patch-embed and downsample kernels). Stepping down rather than jumping to F32 +matters for more than tidiness: with an F32 fallback a FINER arm can come out +LARGER than a coarser one, which is not a property a size table may have. + +Against ~119.63 GiB at 128K context and one sequence: weights 100.35, KV **1.43 +GiB** (11 MLA layers x `kv_lora_rank` 512 x 2 B, plus an indexer side cache of +`index_head_dim` 128 x 2 B / `index_kpool` 4 = 64 B per layer x 11, so 11,968 +B/token), KDA recurrent state **0.14 GiB** (64 heads x 128 x 128 x 4 B x 34 +layers) plus conv states. **~17.7 GiB is left** for activations, allocator +overhead and page cache. Still arithmetic and still not measurement: #1963 and +#1966 record this accounting being wrong by 48x, and W5 re-derives it from the +runner. + +**Recommended first arm: experts at Q2_K, ~100.35 GiB, with ~17.7 GiB for KV, activations and page cache at 128K context.** Not IQ2_S or below, and the reason is not quality — it is that **i-quants need an importance matrix, and an importance matrix needs a forward pass over the model, which needs 181 GiB @@ -1213,9 +1311,32 @@ Debts this row carries, each visible rather than waived: - **O4 — no llama.cpp floor and no llama.cpp oracle** for the GGUF arms (D6). - **O5 — no i-quant arm is producible on this fleet** (R4). - **O6 — speed.** No number on any axis, and no denominator exists. +- **O7 — no artifact of this model exists.** W7a authored the converter and + gated it on synthetic fixtures; it has never been run against the real + checkpoint. Producing the Q2_K arm needs the 300–600 GiB checkpoint staged on + local disk (not CIFS), explicit developer authority for the download, and a + box with room for the source and the ~100.35 GiB output at once. Until then + every GPU gate on this row — W3, W5, W6 and W7b — has nothing to load, and + §Evidence's sha256, conversion recipe and peak RSS are unpaid. + W7b/[#2011](https://github.com/mudler/vllm.cpp/issues/2011) owns it. +- **O8 — the Q3_K, Q4_K and Q5_K encoders are not ported** and the converter + refuses those arms by name. Only Q2_K, Q6_K and Q8_0 are ported from the + pinned llama.cpp reference and gated byte-for-byte against it. No arm this row + needs uses them today; the §Hardware second-choice line that mentions Q5_K for + the non-expert 3% would need this first. + [#2011](https://github.com/mudler/vllm.cpp/issues/2011) records it. +- **O9 — the converter's output is not loadable by this tree.** `glm5next` has + no entry in the `general.architecture` dispatch + (`src/vllm/entrypoints/model_loader.cpp:1000`), so the file W7a can write is a + file nothing here can read. The wiring is **W1's**, owned by row + `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` and tracked by + [#1998](https://github.com/mudler/vllm.cpp/issues/1998). Named here because + W7a lands a capability that a production entry point does not yet reach, which + AGENTS.md §"Nothing lands dead" allows only when it is written down. ## Now -`READY`, 2026-08-26. The spec and its records are committed; no product code has -landed. The next action is to claim W0 or W1 on a fresh -`row/MODEL-MM-GLM53-FLASH-W` branch. +`READY`, 2026-08-26. The spec, its records and the W7a scope are committed; the +converter is not in this commit. The next action is W7a +([#2011](https://github.com/mudler/vllm.cpp/issues/2011), +`CLAIM-GLM53-FLASH-W7A`), then W1. From 417e00d05175b6ac9de7320ced300a0ae6131a32 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 20:15:24 +0000 Subject: [PATCH 3/5] =?UTF-8?q?feat(MODEL-MM-GLM53-FLASH):=20W7a=20?= =?UTF-8?q?=E2=80=94=20the=20glm5=5Fnext=20GGUF=20converter,=20gated=20byt?= =?UTF-8?q?e-for-byte=20against=20llama.cpp=20b10451=20(#2011)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every GPU gate on this row is blocked behind an artifact that does not exist. Measured live 2026-08-26 against ~119.63 GiB usable on `dgx:gpu0`: FP8 305.78 GiB, BF16 598.53, the smallest published NVFP4 181.32, and all four repositories named `*-GGUF` contain zero `.gguf` files. No upstream tool can make one, so this authors it. `scripts/convert-glm5-next-gguf.py` reads a safetensors checkpoint and writes a GGUF at arch `glm5next`. It streams -- headers for the plan, `mmap` slices for the data -- so peak resident memory is one tensor rather than one shard, which is what makes a 305.78 GiB source tractable at all. FP8 e4m3 is decoded against the `weight_scale_inv` grid the checkpoint declares (`weight_block_size: [128, 128]`); per-expert tensors are stacked into `ffn_{gate,up,down}_exps`; the layer-45 MTP block and every `shared_head.*` are dropped, following the reference's own `_keys_to_ignore_on_load_unexpected` and the `glm4_moe_lite_registry.cpp:21-26` precedent. The metadata carries the parameters the port hinges on, spelled at the pin. `glm5next.kda.gate_lower_bound` is the load-bearing one: -5.0 selects `-bound * sigmoid(exp(A_log) * (g + dt_bias))`, a different function from the `-exp(A_log) * softplus(g + dt_bias)` our Kimi-Linear KDA implements, and even the sign of `decay_rate` differs. Writing it into the file is what lets a loader take the right branch instead of inheriting Kimi's. Alongside it: the indexer geometry with `index_kpool` = 4 where the config class default is 16, the mHC triple keeping `hc_eps` 1e-6 distinct from `rms_norm_eps` 1e-5, and `layer_types` as the authoritative schedule -- the reference ignores `linear_attn_config.kda_layers` entirely, so only one of those two lists may travel. No `rope.freq_base` is written; the text stack is NoPE end to end. **Q2_K, Q6_K and Q8_0 are ported from `ggml/src/ggml-quants.c` at our pin `b10451` and are byte-identical to it.** They had to be ported rather than called: `gguf.quants.Q2_K` upstream implements `dequantize_blocks` and no `quantize_blocks`, so no upstream Python can produce a k-quant. The gate is a frozen golden captured from that reference compiled `-ffp-contract=off`, and it is bytes rather than a tolerance because an encoder that is close but not exact writes a file that loads, generates fluent text, and is quietly worse than the arm it claims to be. Two traps changed bytes during the port and are recorded in the source so nobody re-finds them: `nearest_int` is the `+12582912.0` add-and-mask trick at `:621` and rounds half to EVEN, not `round`; and C `roundf` in `quantize_row_q8_0_ref` rounds half AWAY FROM ZERO where `np.rint` rounds half to even. The second was caught by a crafted `[0.5, -0.5, 1.5, ...]` case: reference `[1,-1,2,-2,3,-3]`, `np.rint` `[0,0,2,-2,2,-2]`. The arms, from the converter's own per-tensor plan over the real topology (1719 tensors, 313.89B parameters after the MTP block is dropped from 321.32B): q2_k 100.35 GiB at 2.746 mixed bpw, q6_k 239.89, q8_0 310.67, bf16 584.67. Only q2_k fits. Experts are 97% of the model, so the arm name is the expert type and the other 3% rides at Q6_K almost for free. The fallback ladder is `want -> Q8_0 -> F32` rather than `want -> F32`, and that is not tidiness: with an F32 fallback a finer arm can come out LARGER than a coarser one, which is not a property a size table may have. Unimplemented arms are refused by name with the missing part. The i-quants are refused because an imatrix needs a forward pass, a forward pass needs 181 GiB, and the dependency is circular on this fleet -- a boundary, not a to-do. Q3_K/Q4_K/Q5_K are refused because their encoders are unported and shipping an ungated encoder is worse than refusing. So are `--keep-mtp`, a non-`glm5_next` config, and an FP8 tensor whose `weight_scale_inv` companion is missing, which would otherwise write a loadable, wrong file. `tests/scripts/test_convert_glm5_next_gguf.py` is 50 assertions over a SYNTHETIC tiny-shape checkpoint and an independent in-test GGUF reader that shares no code with the writer. No checkpoint, no GPU, no C++ build. RED before this commit was `FAIL scripts/convert-glm5-next-gguf.py is absent`, rc=1. **NOT REACHED, and this is the part to read before reviewing.** `glm5next` has no entry in the `general.architecture` dispatch (`src/vllm/entrypoints/model_loader.cpp:1000`), so the file this converter writes is not loadable by this tree. That wiring is W1's, owned by row `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` and tracked by #1998; the spec lists it under `## Owed` as O9. The converter itself IS reached, as a command-line path, and the gate enters through it as a user does. No artifact has been produced against the real checkpoint either -- that needs staged weights, disk and a box, and is owed as O7. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-fable-5 [Claude Code] --- .agents/model-matrix.md | 4 +- .agents/specs/glm5-next-flash.md | 15 +- .github/workflows/ci.yml | 7 + docs/USAGE.md | 52 + scripts/agent-preflight.sh | 17 + scripts/convert-glm5-next-gguf.py | 1246 +++++++++++++++++ .../glm5_next_kquant_golden_b10451.json | 22 + tests/scripts/test_convert_glm5_next_gguf.py | 865 ++++++++++++ 8 files changed, 2222 insertions(+), 6 deletions(-) create mode 100755 scripts/convert-glm5-next-gguf.py create mode 100644 tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json create mode 100644 tests/scripts/test_convert_glm5_next_gguf.py diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 37fd1d334..a83cb6335 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -89,7 +89,7 @@ Engaged architectures (the 55 non-`INVENTORIED` rows): | Support | Architecture | Family / example | Status | Row | |---|---|---|---|---| -| 🚧 | `Glm5NextForConditionalGeneration` | GLM-5.3-Flash (321.32B total / ~18B active, natively multimodal; 34 KDA linear-attention + 11 DeepSeek-sparse MLA layers, mHC residual manifold, 288+1 expert MoE, 24-layer ViT) | **SPEC ONLY.** New architecture vLLM implements at NO revision -- absent from the pin `555967922` AND from `main` `c71f6f8a81`, so a pin advance does not reach it; vllm#53906 is OPEN and inadmissible. Sole admissible oracle is transformers, lane-pinned `v5.16.1` (the first release carrying `glm5_next`; `v5.16.0` is 404). `gateable = no` on MEMORY: the smallest published artifact is NVFP4 at 181.32 GiB against ~119.63 GiB on GB10, so no oracle can run this model on any device here and NO end-to-end token gate is reachable. Zero `.gguf` files exist in any of the four `*-GGUF` repos and llama.cpp has no `glm5_next`, so the converter is ours to author. Spec plans 8 waves; no product code has landed | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | +| 🚧 | `Glm5NextForConditionalGeneration` | GLM-5.3-Flash (321.32B total / ~18B active, natively multimodal; 34 KDA linear-attention + 11 DeepSeek-sparse MLA layers, mHC residual manifold, 288+1 expert MoE, 24-layer ViT) | **CONVERTER LANDED, MODEL NOT STARTED.** New architecture vLLM implements at NO revision -- absent from the pin `555967922` AND from `main` `c71f6f8a81`, so a pin advance does not reach it; vllm#53906 is OPEN and inadmissible. Sole admissible oracle is transformers, lane-pinned `v5.16.1` (the first release carrying `glm5_next`; `v5.16.0` is 404). `gateable = no` on MEMORY: the smallest published artifact is NVFP4 at 181.32 GiB against ~119.63 GiB on GB10, so no oracle can run this model on any device here and NO end-to-end token gate is reachable. Zero `.gguf` files exist in any of the four `*-GGUF` repos and llama.cpp has no `glm5_next`, so the converter was ours to author and W7a ([#2011](https://github.com/mudler/vllm.cpp/issues/2011)) authored it: `scripts/convert-glm5-next-gguf.py`, Q2_K arm 100.35 GiB against ~119.63 GiB, k-quant encoders byte-identical to the pinned llama.cpp `b10451`. The MODEL is still not started -- no `glm5next` registration, so the file it writes is not loadable here (O9), and no artifact has been produced (O7). Spec plans 8 waves; W7a is the only one landed | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | | 🚧 | `Qwen4ExpForConditionalGeneration` | Qwen3.8-Flash-Next (180B total / 6B activated, image-text-to-text) | **SPEC ONLY.** New architecture vLLM does NOT implement at any revision; split oracle by developer direction 2026-08-26 (transformers for the algorithm, vLLM ops for the optimized path). Nothing published fits ~119 GB on GB10, so `gateable = no` and the quantized arms are load-bearing; no GGUF exists and llama.cpp has no `qwen4_exp` either. NO token, NO speed. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | | ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling scope CLOSED (#323): `60e71a0e` fixed the eager path; `DenseDecodeGraphForward` ran first and replayed against stale HOST ids, so it now declines while the mirror is live and falls back to the proven eager path. Async gate 7/7 across Qwen3-0.6B/4B + Llama/Mistral/InternLM2 | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` | | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` | @@ -524,7 +524,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-cohere-asr-cohere-asr-for-conditional-generation` | `CohereAsrForConditionalGeneration` | `registry.py:571-574`; `vllm/model_executor/models/cohere_asr.py::CohereAsrForConditionalGeneration` | conditional generation / audio+image | MM processor; encoder/merge; cross-attention; audio/ASR frontend; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-nemotron-parse-nemotron-parse-for-conditional-generation` | `NemotronParseForConditionalGeneration` | `registry.py:575-578`; `vllm/model_executor/models/nemotron_parse.py::NemotronParseForConditionalGeneration` | conditional generation / audio+image | MM processor; encoder/merge; cross-attention; audio/ASR frontend; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-whisper-whisper-for-conditional-generation` | `WhisperForConditionalGeneration` | `registry.py:579`; `vllm/model_executor/models/whisper.py::WhisperForConditionalGeneration` | conditional generation / audio+image | MM processor; encoder/merge; sliding-window attention; cross-attention; audio/ASR frontend | ☐ required | `INVENTORIED` | none | unassigned | -| `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | `Glm5NextForConditionalGeneration` (`model_type: glm5_next`; campaign row `MODEL-MM-GLM53-FLASH`) | **NOT IN vLLM AT ANY REVISION** -- deliberately written with no pinned module/class target, the same convention `MODEL-MM-qwen4-exp-*` follows, so the at-the-pin static invariants are untouched. Read live 2026-08-26: `git grep "Glm5\|glm5_next"` returns ZERO hits at the pin `555967922` AND at `origin/main` `c71f6f8a81` (2026-08-26 08:42 -0700), whose `vllm/models/` package holds `common, deepseek_v32, deepseek_v4, dots3_note, inkling, kimi_k3, minimax_m3` and no `glm5next`. That is ABSENCE, not pin staleness, so advancing the parity pin -- 348 commits of reconciliation -- reaches nothing here. [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) (opened 2026-08-26T14:12:00Z, head `933876c388fb`, 85 files, +12,511/-540) would register `Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` and `Glm5NextMTPModel`, and is **OPEN, unmerged, `mergeable: false`** -- an unmerged PR is not a revision and is inadmissible as an oracle, the same rule applied to SGLang #36497 on the qwen4-exp row; it is cited only as evidence of upstream shape. SGLang (PR #36507 open, cookbook #36440/#36513 MERGED ahead of the code), llama.cpp (PR #27752 open, our pin `b10451`) and vllm-omni likewise implement nothing. **ONE row and not three:** none of the three names is registered at any vLLM revision, and the only architecture a published artifact declares is `Glm5NextForConditionalGeneration`. | conditional generation / image + video | model loader/forward; GDN/linear-attention state (KDA); MLA/latent KV; DSA sparse indexer; FusedMoE/grouped GEMM; mHC residual manifold; MM processor + ViT encoder/merge; FP8 block-wise quant; GGUF k-quant | [glm5-next-flash](specs/glm5-next-flash.md) | `READY` | **SCOPING ONLY, 2026-08-26 (issue [#1998](https://github.com/mudler/vllm.cpp/issues/1998)): the spec and its records are the whole change; NO product code has landed.** Architecture read from `config.json` live: 45 layers, hidden 4096, vocab 154880, 1,048,576 context; `layer_types` = 34 `linear_attention` + 11 `deepseek_sparse_attention` (at 3,7,...,43); MLA is **fully NoPE** (`qk_rope_head_dim: 0`, `mla_use_nope: true`, `qk_nope_head_dim: 256`, `v_head_dim: 256`, `kv_lora_rank: 512`, `q_lora_rank: 1536`) and `text_config` carries NO `rope_theta` and NO `rope_scaling` -- the transformers reference *deletes* the inherited `rope_parameters` field and passes `position_embeddings=None` to every layer, and `indexer_rope_interleave: true` is a VESTIGIAL flag the indexer override ignores, so the text stack has no rotary anywhere. DSA indexer adds a **learned per-channel k-pool** (`index_kpool: 4`, `index_kpool_compress`, `index_kpool_always_select_tail`): 128 independent 4-way softmaxes over a learned `[4,128]` intra-pool position embedding, pools aligned to the first VALID token (batch-dependent), `select_k = min(index_topk/index_kpool, P) = min(512, P)`, expanded back to raw indices and widened to `index_topk + kpool - 1` = **2051** by an unscored ragged tail. mHC `hc_mult: 4`, `hc_sinkhorn_iters: 20`, `hc_eps: 1e-6`. MoE 288 routed + 1 shared, top-8, `moe_intermediate_size` 2048, sigmoid/`noaux_tc`, `n_group=topk_group=1`, `routed_scaling_factor` 2.5, clamped SwiGLU `swiglu_limit` 10.0 in FIVE places. Vision 24L/1024/patch **14**/merge 2/temporal 2/out 4096. **HEAVY REUSE, and the two obvious reuses are WRONG in one place each.** (1) The KDA forget gate takes the OTHER branch: `gate_lower_bound: -5.0` is not None, so it computes `-5.0 * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias))`, where our `src/vllm/model_executor/models/kimi_kda.cpp:60` implements Kimi-Linear's `-exp(A_log) * softplus(g + dt_bias)` -- different functions of the same inputs, both fluent, and the sign of `decay_rate` differs too. (2) The mHC head collapse is `hidden_streams.mean(dim=2)`, an UNWEIGHTED mean whose own docstring says "Unlike DeepSeek-V4", where our `deepseek_v4_mhc.cpp:168` `HcHeadCollapse` is the V4 weighted-gated collapse; the checkpoint carries no `hc_head.*` tensors at all, which settles it. **Third trap: the k-pool selection is unobservable at or below 2048 tokens** -- with `index_topk: 2048` a shorter context selects every candidate, so any short-prompt gate passes over a broken indexer. **Structural blocker:** `MlaBlockDims::Validate` REFUSES `qk_rope_head_dim == 0` (`src/vllm/model_executor/layers/attention/mla_attention.cpp:90-93` requires every dim > 0, `:95-99` requires it even); Kimi-Linear is NOT the same case, it sets `mla_use_nope` while keeping `qk_rope_head_dim: 64` so the rope slice still occupies the cache row (`include/vllm/model_executor/models/kimi_linear.h:86,:88`). **HW: NOTHING FITS.** Measured live by HTTP RANGE over all 62 safetensors headers (no payload fetched): 76,108 tensors, `F8_E4M3` 314,396,639,232 + `BF16` 6,926,096,640 elements = **321,322,735,872 real parameters**, routed experts **311.65B = 97.0%**; on disk `zai-org/GLM-5.3-Flash` FP8 305.78 GiB (2.56x GB10), `-BF16` 598.53 GiB (5.00x), `LibertAIDAI/...-NVFP4` 181.32 GiB (1.52x). **All four `*-GGUF` repos contain ZERO `.gguf` files.** `gateable = no` on MEMORY: no oracle can execute this model on any device this project reaches, so **no end-to-end token gate exists or can exist here** -- recorded as visible debt, replaced by a tiny-shape CPU reference oracle instantiated from the pinned transformers, which is an oracle for the NUMERICS and is NOT oracle gateability for the model. Layer 45 is the MTP block (888 tensors, 7.43B params, 2.31%), present in the checkpoint and discarded by the reference; owed, not implemented. | `-` | +| `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | `Glm5NextForConditionalGeneration` (`model_type: glm5_next`; campaign row `MODEL-MM-GLM53-FLASH`) | **NOT IN vLLM AT ANY REVISION** -- deliberately written with no pinned module/class target, the same convention `MODEL-MM-qwen4-exp-*` follows, so the at-the-pin static invariants are untouched. Read live 2026-08-26: `git grep "Glm5\|glm5_next"` returns ZERO hits at the pin `555967922` AND at `origin/main` `c71f6f8a81` (2026-08-26 08:42 -0700), whose `vllm/models/` package holds `common, deepseek_v32, deepseek_v4, dots3_note, inkling, kimi_k3, minimax_m3` and no `glm5next`. That is ABSENCE, not pin staleness, so advancing the parity pin -- 348 commits of reconciliation -- reaches nothing here. [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) (opened 2026-08-26T14:12:00Z, head `933876c388fb`, 85 files, +12,511/-540) would register `Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` and `Glm5NextMTPModel`, and is **OPEN, unmerged, `mergeable: false`** -- an unmerged PR is not a revision and is inadmissible as an oracle, the same rule applied to SGLang #36497 on the qwen4-exp row; it is cited only as evidence of upstream shape. SGLang (PR #36507 open, cookbook #36440/#36513 MERGED ahead of the code), llama.cpp (PR #27752 open, our pin `b10451`) and vllm-omni likewise implement nothing. **ONE row and not three:** none of the three names is registered at any vLLM revision, and the only architecture a published artifact declares is `Glm5NextForConditionalGeneration`. | conditional generation / image + video | model loader/forward; GDN/linear-attention state (KDA); MLA/latent KV; DSA sparse indexer; FusedMoE/grouped GEMM; mHC residual manifold; MM processor + ViT encoder/merge; FP8 block-wise quant; GGUF k-quant | [glm5-next-flash](specs/glm5-next-flash.md) | `ACTIVE` | **SCOPING ONLY, 2026-08-26 (issue [#1998](https://github.com/mudler/vllm.cpp/issues/1998)): the spec and its records are the whole change; NO product code has landed.** Architecture read from `config.json` live: 45 layers, hidden 4096, vocab 154880, 1,048,576 context; `layer_types` = 34 `linear_attention` + 11 `deepseek_sparse_attention` (at 3,7,...,43); MLA is **fully NoPE** (`qk_rope_head_dim: 0`, `mla_use_nope: true`, `qk_nope_head_dim: 256`, `v_head_dim: 256`, `kv_lora_rank: 512`, `q_lora_rank: 1536`) and `text_config` carries NO `rope_theta` and NO `rope_scaling` -- the transformers reference *deletes* the inherited `rope_parameters` field and passes `position_embeddings=None` to every layer, and `indexer_rope_interleave: true` is a VESTIGIAL flag the indexer override ignores, so the text stack has no rotary anywhere. DSA indexer adds a **learned per-channel k-pool** (`index_kpool: 4`, `index_kpool_compress`, `index_kpool_always_select_tail`): 128 independent 4-way softmaxes over a learned `[4,128]` intra-pool position embedding, pools aligned to the first VALID token (batch-dependent), `select_k = min(index_topk/index_kpool, P) = min(512, P)`, expanded back to raw indices and widened to `index_topk + kpool - 1` = **2051** by an unscored ragged tail. mHC `hc_mult: 4`, `hc_sinkhorn_iters: 20`, `hc_eps: 1e-6`. MoE 288 routed + 1 shared, top-8, `moe_intermediate_size` 2048, sigmoid/`noaux_tc`, `n_group=topk_group=1`, `routed_scaling_factor` 2.5, clamped SwiGLU `swiglu_limit` 10.0 in FIVE places. Vision 24L/1024/patch **14**/merge 2/temporal 2/out 4096. **HEAVY REUSE, and the two obvious reuses are WRONG in one place each.** (1) The KDA forget gate takes the OTHER branch: `gate_lower_bound: -5.0` is not None, so it computes `-5.0 * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias))`, where our `src/vllm/model_executor/models/kimi_kda.cpp:60` implements Kimi-Linear's `-exp(A_log) * softplus(g + dt_bias)` -- different functions of the same inputs, both fluent, and the sign of `decay_rate` differs too. (2) The mHC head collapse is `hidden_streams.mean(dim=2)`, an UNWEIGHTED mean whose own docstring says "Unlike DeepSeek-V4", where our `deepseek_v4_mhc.cpp:168` `HcHeadCollapse` is the V4 weighted-gated collapse; the checkpoint carries no `hc_head.*` tensors at all, which settles it. **Third trap: the k-pool selection is unobservable at or below 2048 tokens** -- with `index_topk: 2048` a shorter context selects every candidate, so any short-prompt gate passes over a broken indexer. **Structural blocker:** `MlaBlockDims::Validate` REFUSES `qk_rope_head_dim == 0` (`src/vllm/model_executor/layers/attention/mla_attention.cpp:90-93` requires every dim > 0, `:95-99` requires it even); Kimi-Linear is NOT the same case, it sets `mla_use_nope` while keeping `qk_rope_head_dim: 64` so the rope slice still occupies the cache row (`include/vllm/model_executor/models/kimi_linear.h:86,:88`). **HW: NOTHING FITS.** Measured live by HTTP RANGE over all 62 safetensors headers (no payload fetched): 76,108 tensors, `F8_E4M3` 314,396,639,232 + `BF16` 6,926,096,640 elements = **321,322,735,872 real parameters**, routed experts **311.65B = 97.0%**; on disk `zai-org/GLM-5.3-Flash` FP8 305.78 GiB (2.56x GB10), `-BF16` 598.53 GiB (5.00x), `LibertAIDAI/...-NVFP4` 181.32 GiB (1.52x). **All four `*-GGUF` repos contain ZERO `.gguf` files.** `gateable = no` on MEMORY: no oracle can execute this model on any device this project reaches, so **no end-to-end token gate exists or can exist here** -- recorded as visible debt, replaced by a tiny-shape CPU reference oracle instantiated from the pinned transformers, which is an oracle for the NUMERICS and is NOT oracle gateability for the model. Layer 45 is the MTP block (888 tensors, 7.43B params, 2.31%), present in the checkpoint and discarded by the reference; owed, not implemented. **W7a landed 2026-08-26 ([#2011](https://github.com/mudler/vllm.cpp/issues/2011)), advancing the row to `ACTIVE`: `scripts/convert-glm5-next-gguf.py` is the safetensors->GGUF converter no upstream tool can supply, with the Q2_K/Q6_K/Q8_0 encoders ported from `ggml/src/ggml-quants.c` @ our pin `b10451` and gated BYTE-IDENTICAL against it over a frozen golden -- `gguf.quants.Q2_K` upstream implements `dequantize_blocks` and NO `quantize_blocks`, so no upstream Python can produce a k-quant at all. Its plan over the real topology emits 1719 tensors carrying 313.89B parameters (321.32B less the dropped 7.43B MTP block): Q2_K arm 100.35 GiB, Q6_K 239.89, Q8_0 310.67, BF16 584.67, so Q2_K fits ~119.63 GiB with ~17.7 GiB left after 1.43 GiB of KV at 128K and 0.14 GiB of KDA state. Everything the converter needed by way of GGUF convention was already AT THE PIN -- `class KDA` with `{arch}.kda.gate_lower_bound` at `gguf-py/gguf/constants.py:262-264`, the `ssm_conv1d_q/k/v` KDA tensor names at `src/llama-arch.cpp:465-479`, the indexer names at `:626-636` -- so no pin advance was taken. NOT REACHED: `glm5next` has no `general.architecture` dispatch entry, so the emitted file is not loadable here (O9, W1 owns it); and no artifact has been produced (O7, needs staged weights, disk and a box).** | `CLAIM-GLM53-FLASH-W7A` | ## MODEL-AUDIO - Audio encoder components (not vLLM registry architectures) diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index 587b3fad6..067757c11 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -904,7 +904,7 @@ them needs a box. The split is recorded here rather than derived again: W7a is CPU-only and needs no checkpoint, W7b needs 300–600 GiB of staged weights and explicit developer authority for a large-asset download. -#### W7a — the converter and its synthetic gate (CPU, large) +#### W7a — the converter and its synthetic gate (CPU, large). LANDED Issue: [#2011](https://github.com/mudler/vllm.cpp/issues/2011). @@ -1336,7 +1336,14 @@ Debts this row carries, each visible rather than waived: ## Now -`READY`, 2026-08-26. The spec, its records and the W7a scope are committed; the -converter is not in this commit. The next action is W7a +`ACTIVE`, 2026-08-26. Advanced from `READY` by W7a ([#2011](https://github.com/mudler/vllm.cpp/issues/2011), -`CLAIM-GLM53-FLASH-W7A`), then W1. +`CLAIM-GLM53-FLASH-W7A`), which lands the first product code on the row: the +safetensors→GGUF converter and its synthetic-fixture gate, with the Q2_K, Q6_K +and Q8_0 encoders byte-identical to the pinned llama.cpp `b10451` reference. + +**No artifact exists** (O7) and **nothing in this tree can read what the +converter writes** (O9), so no GPU gate has moved and no correctness claim about +the MODEL has been made. The next actions are W1 — config, registration and the +`general.architecture` dispatch entry that discharges O9 — and, whenever the +developer grants a large-asset download, W7b. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2a2029c14..5da6ee182 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -495,6 +495,13 @@ jobs: sudo apt-get update -qq sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends python3-numpy python3 tests/scripts/test_ltx25_render_compare.py + # Same numpy-only lane, same reason (#2011): the GLM-5.3-Flash + # GGUF converter avoids gguf-py on purpose -- upstream has no + # `glm5_next`, and `gguf.quants.Q2_K` has a dequantizer and NO + # quantizer -- so numpy is its only dependency. Preflight SKIPS + # this suite when numpy is absent; this lane is where it must + # never be silent. + python3 tests/scripts/test_convert_glm5_next_gguf.py python3 tests/scripts/test_ltx25_pixel_ab_harness.py - name: Every LTX-2.5 A/B arm sets a knob value the dispatch parses (#1751) # The `VLLM_LTX2_DIT_FLASH_ATTN` values a harness exports and the values diff --git a/docs/USAGE.md b/docs/USAGE.md index fd419b670..8ec26ab9f 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -573,8 +573,60 @@ repository in this project's history. | Qwen3.8-2.4T-A95B | `UD-Q1_0` ten-file GGUF split | about 370 GiB | `unsloth/Qwen3.8-2.4T-A95B-GGUF` @ `567d3e6ac26c5474b18311e619c04350fb9a5556` | `b7770552b2ac24e7334c917bc92e90e218e87cfe29484db65e62e8ef2a60334d` (shard 1); `2765517f833c736338d3ab34354e1c10eb8d79e62325f998285b435e5cf03dcd` (shard 2) | CPU expert streaming from disk | CUDA refuses a checkpoint that exceeds device capacity | | DeepSeek-V4-Flash EXL3 trellis shard 1 of 172 | `exl3-layer-000-tp4-rank0.safetensors` | 515,850,920 bytes | `0xSero/deepseek-v4-flash-0731-spark` @ `22f28d32b9b29b4352eaa380ff8c2c170b2847ab` | `2ed7ae798a794019810b027fe2609e2cf4ad78d70b49c47b2970d03a0a7aaadf` | The rank-sliced EXL3 routed-expert tower LOADS (TP4 coalesced to TP1) and its experts EXECUTE through `vt::Exl3Gemm` on a CPU queue | The CUDA arm has passed no compiler and no GPU, so no device arm is claimed. A SYNTHETIC rank-sliced checkpoint now loads and emits logits end to end; THIS artifact still does not, because its DSA compressor and indexer tensors are stored at twice the width the host forward indexes (`compressor.wgate` `[2*head_dim, H]`) and the loader refuses them BY NAME, and because its tokenizer is not read ([#1924](https://github.com/mudler/vllm.cpp/issues/1924)) | | DeepSeek-V4-Flash EXL3 carried tower shard 1 of 5 | `carried-001.safetensors` | 4,288,630,252 bytes | `0xSero/deepseek-v4-flash-0731-spark` @ `22f28d32b9b29b4352eaa380ff8c2c170b2847ab` | `3b67ae29f1e75c2ecadfcafd3b0eecec640b06fd60b832f77e6bd3c2a8c85ccf` | The un-requantized `deepseek_v4_fp8` attention, router, shared-expert, compressor and embedding tensors, MATERIALIZED at load into the host-float tower the forward composes with — block-wise FP8 (`F8_E4M3` + `F8_E8M0` over 128x128 blocks) decoded to f32, BF16 norms and embeddings widened, I64 `tid2eid` narrowed to int32 | The DSA compressor and indexer tensors of this artifact are `2 * head_dim` / `2 * index_head_dim` wide and the loader refuses them by name (41 of its 43 layers carry a compressor); the 3,985 `mtp.*` NVFP4 draft tensors are skipped and counted, never silently dropped | +| GLM-5.3-Flash FP8 source | `model-000{01..62}-of-00062.safetensors` | 328,326,771,576 bytes total (305.78 GiB) | `zai-org/GLM-5.3-Flash` @ `main`, read 2026-08-26 | Owed: no byte of payload has been fetched, so no local hash exists to state, and an unauthenticated tree hash is not a pin here | Declared source of `scripts/convert-glm5-next-gguf.py`. Only the safetensors HEADERS were read, by HTTP RANGE over all 62 shards: 76,108 tensors, `F8_E4M3` block-quantized at `weight_block_size: [128, 128]` with `weight_scale_inv` companions, plus BF16 and F32 scales | **Nothing has been converted.** The download needs explicit developer authority and a box with room for 305.78 GiB of source and ~100.35 GiB of output at once; owed as O7 on [#2011](https://github.com/mudler/vllm.cpp/issues/2011). The revision is a branch name and not a commit, which is NOT a pin: it is what was read, and W7b re-reads and records the commit when it stages the bytes | +| GLM-5.3-Flash GGUF | none exists | n/a | `unsloth/GLM-5.3-Flash-GGUF`, `AtomicChat/GLM-5.3-Flash-GGUF`, `aj9o9/GLM-5.3-Flash-GGUF`, `vcruz305/GLM-5.3-Flash-GGUF`, all read 2026-08-26 | n/a | none | **All four repositories named `*-GGUF` contain ZERO `.gguf` files** — READMEs, a `.gitattributes` and four PNGs between them. A repository name is not an artifact, and this row exists so the next reader does not go looking again. llama.cpp cannot produce one either: no `glm5_next` at `origin/master` `539f24529` or at our pin `b10451` | +### Convert a GLM-5.3-Flash checkpoint to GGUF + +`zai-org/GLM-5.3-Flash` (`Glm5NextForConditionalGeneration` / `glm5_next`) +publishes no arm that fits any device this project reaches, and no upstream tool +can make one: llama.cpp has no `glm5_next` at our pin `b10451` or at its +`master`, and `gguf-py`'s `Q2_K` has a dequantizer and **no** quantizer. So the +converter ships here. + +```sh +# Read the headers and print the plan, without writing a byte. +scripts/convert-glm5-next-gguf.py --src /path/to/GLM-5.3-Flash --arm q2_k --dry-run + +# Write the arm. +scripts/convert-glm5-next-gguf.py --src /path/to/GLM-5.3-Flash \ + --dst GLM-5.3-Flash-Q2_K.gguf --arm q2_k +``` + +numpy is its only dependency. It streams shard by shard, so peak resident memory +is one tensor rather than one shard, but the output is written in one pass and +needs its full size free on the destination. + +| `--arm` | what the experts get | everything else | weights on the real model | +|---|---|---|---| +| `q2_k` | Q2_K | Q6_K | **100.35 GiB** — the only arm that fits ~119.63 GiB | +| `q6_k` | Q6_K | Q6_K | 239.89 GiB | +| `q8_0` | Q8_0 | Q8_0 | 310.67 GiB | +| `bf16`, `f16`, `f32` | passthrough | passthrough | 584.67 GiB at bf16 | + +Routed and shared experts are 97% of this model, so the arm name is the expert +type and the remaining 3% rides at a finer one almost for free. Figures are the +converter's own per-tensor plan over the real topology (1719 tensors, 313.89B +parameters after the layer-45 MTP block is dropped), not bits-per-weight times a +parameter count. + +**Refused by name, each with the missing part.** Every i-quant — `iq1_s`, +`iq2_xxs`, `iq2_s`, `iq3_xxs`, `iq4_xs`, `iq1_xxxs` — needs an importance +matrix, an importance matrix needs a forward pass over the model, and the +smallest published artifact is 181.32 GiB, so the dependency is circular on this +fleet. `q3_k`, `q4_k` and `q5_k` are refused because those encoders are not +ported: only Q2_K, Q6_K and Q8_0 are ported from `ggml/src/ggml-quants.c` at the +pinned llama.cpp `b10451` and gated byte-for-byte against it. +`--keep-mtp` is refused because nothing here reads an MTP tail. + +**The file it writes is not loadable by this tree yet.** `glm5next` has no +`general.architecture` dispatch entry, so the converter runs ahead of the model +port. That is tracked on +[#1998](https://github.com/mudler/vllm.cpp/issues/1998), and no artifact has +been produced against the real checkpoint either +([#2011](https://github.com/mudler/vllm.cpp/issues/2011)). + ### The distilled NVFP4 DiT was re-quantized under an unchanged name `Lightricks/LTX-2.5` published two different files at diff --git a/scripts/agent-preflight.sh b/scripts/agent-preflight.sh index 645657d3b..fe7ad5ae1 100755 --- a/scripts/agent-preflight.sh +++ b/scripts/agent-preflight.sh @@ -399,6 +399,23 @@ else "numpy is not importable here, and the tool this suite exercises needs it." \ "CI installs python3-numpy and runs the same suite." fi +# THE GLM-5.3-Flash GGUF CONVERTER (#2011). Same shape and the same one +# dependency: `scripts/convert-glm5-next-gguf.py` deliberately does not use +# gguf-py -- upstream has no `glm5_next` and, decisively, `gguf.quants.Q2_K` +# implements `dequantize_blocks` and NO `quantize_blocks` -- so numpy is all it +# needs and all this suite needs. The suite builds a SYNTHETIC tiny-shape +# checkpoint: it needs no weights, no GPU and no C++ build, which is the whole +# reason the converter can be gated at all while the real 305.78 GiB artifact +# stays out of reach. +# +# A missing numpy is a SKIP and never an `ok`, for the reason above. +if python3 -c 'import numpy' >/dev/null 2>&1; then + run "test_convert_glm5_next_gguf" python3 tests/scripts/test_convert_glm5_next_gguf.py +else + skip "test_convert_glm5_next_gguf" \ + "numpy is not importable here, and the converter this suite exercises needs it." \ + "CI installs python3-numpy and runs the same suite." +fi # THE WINDOWS SOURCE CONTRACT, which ran on NO lane until #1829 (#646, #680). # `main` failed to COMPILE under MSVC while every POSIX lane was green, because # `[[noreturn]]` on a non-void return type is C4646 -> C2220 there and a silent diff --git a/scripts/convert-glm5-next-gguf.py b/scripts/convert-glm5-next-gguf.py new file mode 100755 index 000000000..e0cbde946 --- /dev/null +++ b/scripts/convert-glm5-next-gguf.py @@ -0,0 +1,1246 @@ +#!/usr/bin/env python3 +"""safetensors -> GGUF converter for `glm5_next` (GLM-5.3-Flash). + +W7a of `MODEL-MM-GLM53-FLASH` ([#1998](https://github.com/mudler/vllm.cpp/issues/1998), +wave issue in the pull request body). Spec: `.agents/specs/glm5-next-flash.md`. + +**Why this file exists at all.** `zai-org/GLM-5.3-Flash` publishes FP8 (305.78 +GiB), BF16 (598.53 GiB) and NVFP4 (181.32 GiB) artifacts, and the largest device +this project reaches is `dgx:gpu0` at ~119.63 GiB unified. Nothing published +fits. All four HuggingFace repositories named `*-GGUF` contain zero `.gguf` +files, and no upstream tool can emit this architecture: llama.cpp at our pin +`b10451` knows `glm4`, `glm4moe` and `glm-dsa` and has no `glm5_next` (verified +2026-08-26, `src/llama-arch.h` and a repository-wide search). So the converter is +ours to write, and until it exists there is no artifact of this model that any +gate on this fleet can load. + +**What this file is NOT.** It has never been run against the real checkpoint. +W7a deliberately produces no artifact -- the checkpoint is 300-600 GiB and +downloading it needs authority and disk this wave has neither of. Everything +below is gated on synthetic tiny-shape fixtures, and artifact production is owed +(spec `## Owed`, O7). The k-quant encoders are the exception: they are gated +byte-for-byte against the pinned llama.cpp reference, so the one part that +cannot be checked by inspection is checked by an oracle. + +**Where the bytes come from.** llama.cpp is never the mirror source -- AGENTS.md +settles behaviour against vLLM, and where vLLM implements nothing, against the +declared secondary oracle. What llama.cpp supplies here is *convention*: the +GGUF container layout, the metadata key spellings, the tensor-name spellings and +the k-quant block encodings. Every one of those is read at our pin `b10451` and +not at llama.cpp `master`: + + - metadata keys `gguf-py/gguf/constants.py` @ b10451, including + `class KDA` (:262) with `{arch}.kda.head_dim` (:263) and + `{arch}.kda.gate_lower_bound` (:264). `KDA.SAFE_GATE` + exists only at `master` and is NOT used here, because + GLM-5.3-Flash's `linear_attn_config` declares no + `safe_gate` key. Nothing in this file requires a pin + advance. + - tensor names `src/llama-arch.cpp` @ b10451: the KDA set at :465-479 + (`ssm_conv1d_q/k/v`, `ssm_f_a`, `ssm_f_b`, `ssm_g_a`, + `ssm_g_b`, `ssm_beta`, `ssm_a`, `ssm_dt`, `ssm_norm`), + the MLA set at :481-486 and the indexer set at :626-636. + - HF module paths `gguf-py/gguf/tensor_mapping.py` @ b10451 :896-933, whose + Kimi-Linear KDA paths (`self_attn.q_conv1d`, + `self_attn.f_a_proj`, ...) are GLM-5.3-Flash's paths + verbatim. + - k-quant encoders `ggml/src/ggml-quants.c` @ b10451, ported function by + function with the line anchors on each one. + +The in-tree precedent for the per-layer spellings is +`scripts/check-dsv4-gguf-namemap.py`, which already carries `hc_attn_fn.weight`, +`ffn_gate_exps.weight` and the DeepSeek-V4 MLA names. + +**The arch string `glm5next` is OURS.** llama.cpp has no `glm5_next` enumerator +at any revision and its PR #27752 is open, so no upstream spelling exists to +mirror. If llama.cpp lands a different string, this converter and the loader +reconcile onto it in the same change. + +Self-contained on purpose: numpy is the only import beyond the standard library, +matching `scripts/gen-minimax-h3-gguf-manifest.py` and +`scripts/gen-gguf-nvfp4-goldens.py`. gguf-py is deliberately not used -- it has +no `glm5_next` and, more to the point, `gguf.quants.Q2_K` implements +`dequantize_blocks` and **no** `quantize_blocks`, so upstream Python cannot +produce a Q2_K artifact at all. + +Usage: + + scripts/convert-glm5-next-gguf.py --src --dst out.gguf \\ + --arm q2_k [--keep-mtp] [--no-vision] [--dry-run] + +`--dry-run` reads only the safetensors headers and prints the plan and the exact +output size, which is how an arm's arithmetic is checked before 100+ GiB of +writing starts. +""" + +import argparse +import json +import os +import struct +import sys + +try: + import numpy as np +except ImportError: # pragma: no cover - reported, never guessed around + sys.stderr.write( + "convert-glm5-next-gguf: numpy is required and is not importable.\n") + raise SystemExit(4) + +f32 = np.float32 + +# --------------------------------------------------------------------------- +# ggml type ids. Only the ids this converter can EMIT are named. Every one of +# them has both traits and a dequant path in our own reader -- traits alone are +# not enough, and `gguf_dequant.cpp:89-160` is the list that decides: +# IQ4_XS (23) and IQ1_XXXS (21) have traits and NO dequant, which is why no arm +# below reaches for them. +GGML_F32 = 0 +GGML_F16 = 1 +GGML_Q8_0 = 8 +GGML_Q2_K = 10 +GGML_Q6_K = 14 +GGML_BF16 = 30 + +TYPE_NAME = {GGML_F32: "F32", GGML_F16: "F16", GGML_Q8_0: "Q8_0", + GGML_Q2_K: "Q2_K", GGML_Q6_K: "Q6_K", GGML_BF16: "BF16"} +# (block elements, bytes per block) +TYPE_BLOCK = {GGML_F32: (1, 4), GGML_F16: (1, 2), GGML_BF16: (1, 2), + GGML_Q8_0: (32, 34), GGML_Q2_K: (256, 84), GGML_Q6_K: (256, 210)} + +QK_K = 256 + +# GGUF metadata value type ids (gguf-py/gguf/constants.py `GGUFValueType`). +GV_UINT8, GV_INT8, GV_UINT16, GV_INT16, GV_UINT32, GV_INT32 = 0, 1, 2, 3, 4, 5 +GV_FLOAT32, GV_BOOL, GV_STRING, GV_ARRAY, GV_UINT64 = 6, 7, 8, 9, 10 +GV_INT64, GV_FLOAT64 = 11, 12 + +ARCH = "glm5next" +GGUF_MAGIC = b"GGUF" +GGUF_VERSION = 3 +GGUF_ALIGNMENT = 32 + + +# --------------------------------------------------------------------------- +# k-quant encoders, ported from llama.cpp `ggml/src/ggml-quants.c` @ b10451. +# +# These are ported rather than called because no upstream Python can produce +# them (see the module docstring) and because building llama.cpp is not a +# dependency this converter may take. They are gated byte-for-byte against the +# pinned C reference by `tests/scripts/test_convert_glm5_next_gguf.py` over a +# frozen golden captured from that reference; a divergence in the last ulp shows +# up as a byte difference, not as a tolerance argument. +# +# Two traps the port had to reproduce exactly, both of which changed bytes: +# - `nearest_int` is NOT `round`. It is the 12582912.0 add-and-mask trick at +# :621, which rounds half to EVEN. +# - C `roundf` in `quantize_row_q8_0_ref` rounds half AWAY FROM ZERO, where +# `np.rint` rounds half to even. Using `np.rint` mis-encodes every exact .5. +# Float32 accumulation order is mirrored lane by lane rather than handed to +# `ndarray.sum`, whose pairwise summation is a different order. + +def _nearest_int(v): + """`ggml-quants.c:621` @ b10451, vectorized.""" + v = np.asarray(v, dtype=f32) + f32(12582912.0) + i = v.view(np.int32) + return (i & 0x007FFFFF) - 0x00400000 + + +def _fp16(x): + return np.asarray(x, dtype=f32).astype(np.float16) + + +GROUP_MAX_EPS = f32(1e-15) + + +def _make_qkx2_quants(x, nmax=3, rmin=f32(-0.5), rdelta=f32(0.1), nstep=15): + """`ggml-quants.c:799` @ b10451, vectorized over rows of `x` (M, n). + + `min` is state that UPDATES inside the search loop and the next step's + `nearest_int(iscale*(x - min))` reads the updated value, so the 16 steps stay + sequential; only the rows are vectorized. + """ + M, n = x.shape + w = np.abs(x).astype(f32) + mn = x.min(axis=1).astype(f32) + mx = x.max(axis=1).astype(f32) + sum_w = w[:, 0].copy() + sum_x = (sum_w * x[:, 0]).astype(f32) + for i in range(1, n): + sum_w = (sum_w + w[:, i]).astype(f32) + sum_x = (sum_x + (w[:, i] * x[:, i]).astype(f32)).astype(f32) + mn = np.where(mn > f32(0), f32(0), mn).astype(f32) + degen = mx == mn + span = (mx - mn).astype(f32) + safe = np.where(degen, f32(1), span).astype(f32) + iscale = (f32(nmax) / safe).astype(f32) + scale = (f32(1) / iscale).astype(f32) + L = np.clip(_nearest_int((iscale[:, None] * (x - mn[:, None]).astype(f32)).astype(f32)), + 0, nmax).astype(np.uint8) + best = np.zeros(M, dtype=f32) + for i in range(n): + diff = np.abs((scale * L[:, i].astype(f32) + mn - x[:, i]).astype(f32)).astype(f32) + best = (best + (w[:, i] * diff).astype(f32)).astype(f32) + for istep in range(nstep + 1): + denom = np.where(degen, f32(1), (mx - mn).astype(f32)).astype(f32) + isc = ((rmin + rdelta * f32(istep) + f32(nmax)) / denom).astype(f32) + isc = np.where(degen, f32(0), isc).astype(f32) + Laux = np.clip(_nearest_int((isc[:, None] * (x - mn[:, None]).astype(f32)).astype(f32)), + 0, nmax).astype(np.uint8) + sum_l = np.zeros(M, dtype=f32) + sum_l2 = np.zeros(M, dtype=f32) + sum_xl = np.zeros(M, dtype=f32) + for i in range(n): + li = Laux[:, i].astype(f32) + wl = (w[:, i] * li).astype(f32) + sum_l = (sum_l + wl).astype(f32) + sum_l2 = (sum_l2 + (wl * li).astype(f32)).astype(f32) + sum_xl = (sum_xl + (wl * x[:, i]).astype(f32)).astype(f32) + D = ((sum_w * sum_l2).astype(f32) - (sum_l * sum_l).astype(f32)).astype(f32) + okD = D > f32(0) + Dsafe = np.where(okD, D, f32(1)).astype(f32) + this_scale = ((((sum_w * sum_xl).astype(f32) - (sum_x * sum_l).astype(f32)).astype(f32)) + / Dsafe).astype(f32) + this_min = ((((sum_l2 * sum_x).astype(f32) - (sum_l * sum_xl).astype(f32)).astype(f32)) + / Dsafe).astype(f32) + pos = this_min > f32(0) + l2safe = np.where(sum_l2 == f32(0), f32(1), sum_l2).astype(f32) + this_scale = np.where(pos, (sum_xl / l2safe).astype(f32), this_scale).astype(f32) + this_min = np.where(pos, f32(0), this_min).astype(f32) + cur = np.zeros(M, dtype=f32) + for i in range(n): + diff = np.abs(((this_scale * Laux[:, i].astype(f32)).astype(f32) + + this_min - x[:, i]).astype(f32)).astype(f32) + cur = (cur + (w[:, i] * diff).astype(f32)).astype(f32) + upd = okD & (cur < best) + L = np.where(upd[:, None], Laux, L) + best = np.where(upd, cur, best).astype(f32) + scale = np.where(upd, this_scale, scale).astype(f32) + mn = np.where(upd, this_min, mn).astype(f32) + scale = np.where(degen, f32(0), scale).astype(f32) + L = np.where(degen[:, None], np.uint8(0), L) + return scale, (-mn).astype(f32), L + + +def quantize_q2_k(x): + """(nb, 256) f32 -> (nb, 84) u8. `ggml-quants.c:891` @ b10451.""" + x = np.ascontiguousarray(x, dtype=f32) + nb = x.shape[0] + scales, mins, L = _make_qkx2_quants(x.reshape(nb * 16, 16)) + scales = scales.reshape(nb, 16) + mins = mins.reshape(nb, 16) + L = L.reshape(nb, 256) + q4scale = f32(15.0) + max_scale = np.maximum(scales.max(axis=1), f32(0)).astype(f32) + max_min = np.maximum(mins.max(axis=1), f32(0)).astype(f32) + pos_s = max_scale > f32(0) + isc = np.where(pos_s, (q4scale / np.where(pos_s, max_scale, f32(1))).astype(f32), + f32(0)).astype(f32) + out_scales = np.where(pos_s[:, None], + _nearest_int((isc[:, None] * scales).astype(f32)).astype(np.uint8), + np.uint8(0)) + d = np.where(pos_s, (max_scale / q4scale).astype(f32), f32(0)).astype(f32) + pos_m = max_min > f32(0) + iscm = np.where(pos_m, (q4scale / np.where(pos_m, max_min, f32(1))).astype(f32), + f32(0)).astype(f32) + lm = _nearest_int((iscm[:, None] * mins).astype(f32)).astype(np.uint8) + out_scales = np.where(pos_m[:, None], out_scales | (lm << 4), out_scales) + dmin = np.where(pos_m, (max_min / q4scale).astype(f32), f32(0)).astype(f32) + dh = _fp16(d) + dminh = _fp16(dmin) + dl = (dh.astype(f32)[:, None] * (out_scales & 0xF).astype(f32)).astype(f32) + dm = (dminh.astype(f32)[:, None] * (out_scales >> 4).astype(f32)).astype(f32) + dl_e = np.repeat(dl, 16, axis=1) + dm_e = np.repeat(dm, 16, axis=1) + nz = dl_e != f32(0) + num = (x + dm_e).astype(f32) + Lq = np.clip(_nearest_int(np.divide(num, np.where(nz, dl_e, f32(1))).astype(f32)), + 0, 3).astype(np.uint8) + L = np.where(nz, Lq, L) + qs = np.zeros((nb, 64), dtype=np.uint8) + for j in range(0, 256, 128): + base = j // 4 + seg = L[:, j:j + 128] + qs[:, base:base + 32] = (seg[:, 0:32] | (seg[:, 32:64] << 2) + | (seg[:, 64:96] << 4) | (seg[:, 96:128] << 6)) + out = np.zeros((nb, 84), dtype=np.uint8) + out[:, 0:16] = out_scales + out[:, 16:80] = qs + out[:, 80:82] = dh.view(np.uint8).reshape(nb, 2) + out[:, 82:84] = dminh.view(np.uint8).reshape(nb, 2) + return out + + +def _make_qx_quants(x, nmax=32): + """`ggml-quants.c:628` @ b10451 with `rmse_type=1`, `qw=NULL`.""" + M, n = x.shape + ax = np.abs(x).astype(f32) + idx = np.argmax(ax, axis=1) + rows = np.arange(M) + amax = ax[rows, idx] + mx = x[rows, idx].astype(f32) + allzero = amax < GROUP_MAX_EPS + mxs = np.where(allzero, f32(1), mx).astype(f32) + w = (x * x).astype(f32) + iscale = (f32(-nmax) / mxs).astype(f32) + lq = np.clip(_nearest_int((iscale[:, None] * x).astype(f32)), -nmax, nmax - 1).astype(np.int32) + L = (lq + nmax).astype(np.uint8) + sumlx = np.zeros(M, dtype=f32) + suml2 = np.zeros(M, dtype=f32) + for i in range(n): + li = lq[:, i].astype(f32) + sumlx = (sumlx + ((w[:, i] * x[:, i]).astype(f32) * li).astype(f32)).astype(f32) + suml2 = (suml2 + ((w[:, i] * li).astype(f32) * li).astype(f32)).astype(f32) + l2safe = np.where(suml2 == f32(0), f32(1), suml2).astype(f32) + scale = np.where(suml2 != f32(0), (sumlx / l2safe).astype(f32), f32(0)).astype(f32) + best = (scale * sumlx).astype(f32) + for istep in list(range(-9, 0)) + list(range(1, 10)): + isc = ((-(f32(nmax) + (f32(0.1) * f32(istep)).astype(f32)).astype(f32)) / mxs).astype(f32) + lq2 = np.clip(_nearest_int((isc[:, None] * x).astype(f32)), + -nmax, nmax - 1).astype(np.int32) + sx = np.zeros(M, dtype=f32) + s2 = np.zeros(M, dtype=f32) + for i in range(n): + li = lq2[:, i].astype(f32) + sx = (sx + ((w[:, i] * x[:, i]).astype(f32) * li).astype(f32)).astype(f32) + s2 = (s2 + ((w[:, i] * li).astype(f32) * li).astype(f32)).astype(f32) + upd = (s2 > f32(0)) & ((sx * sx).astype(f32) > (best * s2).astype(f32)) + L = np.where(upd[:, None], (lq2 + nmax).astype(np.uint8), L) + news = (sx / np.where(s2 == f32(0), f32(1), s2)).astype(f32) + best = np.where(upd, (news * sx).astype(f32), best).astype(f32) + scale = np.where(upd, news, scale).astype(f32) + scale = np.where(allzero, f32(0), scale).astype(f32) + L = np.where(allzero[:, None], np.uint8(0), L) + return scale, L + + +def quantize_q6_k(x): + """(nb, 256) f32 -> (nb, 210) u8. `ggml-quants.c:1869` @ b10451.""" + x = np.ascontiguousarray(x, dtype=f32) + nb = x.shape[0] + scales, L8 = _make_qx_quants(x.reshape(nb * 16, 16)) + scales = scales.reshape(nb, 16) + L = L8.reshape(nb, 256).astype(np.int32) + absc = np.abs(scales).astype(f32) + j = np.argmax(absc, axis=1) + rows = np.arange(nb) + dead = absc[rows, j] < GROUP_MAX_EPS + ms = np.where(dead, f32(1), scales[rows, j].astype(f32)).astype(f32) + iscale = (f32(-128.0) / ms).astype(f32) + dh = _fp16((f32(1) / iscale).astype(f32)) + sc = np.minimum(_nearest_int((iscale[:, None] * scales).astype(f32)), 127).astype(np.int8) + dl = (dh.astype(f32)[:, None] * sc.astype(f32)).astype(f32) + dl_e = np.repeat(dl, 16, axis=1) + nz = dl_e != f32(0) + Lq = np.clip(_nearest_int(np.divide(x, np.where(nz, dl_e, f32(1))).astype(f32)), + -32, 31) + 32 + L = np.where(nz, Lq.astype(np.int32), L) + ql = np.zeros((nb, 128), dtype=np.uint8) + qh = np.zeros((nb, 64), dtype=np.uint8) + for jj in range(0, 256, 128): + o_ql = (jj // 128) * 64 + o_qh = (jj // 128) * 32 + s = L[:, jj:jj + 128].astype(np.uint8) + ql[:, o_ql:o_ql + 32] = (s[:, 0:32] & 0xF) | ((s[:, 64:96] & 0xF) << 4) + ql[:, o_ql + 32:o_ql + 64] = (s[:, 32:64] & 0xF) | ((s[:, 96:128] & 0xF) << 4) + qh[:, o_qh:o_qh + 32] = ((s[:, 0:32] >> 4) | ((s[:, 32:64] >> 4) << 2) + | ((s[:, 64:96] >> 4) << 4) | ((s[:, 96:128] >> 4) << 6)) + out = np.zeros((nb, 210), dtype=np.uint8) + out[:, 0:128] = ql + out[:, 128:192] = qh + out[:, 192:208] = sc.view(np.uint8) + out[:, 208:210] = dh.view(np.uint8).reshape(nb, 2) + if dead.any(): + out[dead] = 0 + return out + + +def quantize_q8_0(x): + """(nb, 32) f32 -> (nb, 34) u8. `ggml-quants.c:276` @ b10451.""" + x = np.ascontiguousarray(x, dtype=f32) + nb = x.shape[0] + amax = np.abs(x).max(axis=1).astype(f32) + d = (amax / f32(127.0)).astype(f32) + idv = np.where(d != f32(0), (f32(1.0) / np.where(d == f32(0), f32(1), d)).astype(f32), + f32(0)).astype(f32) + dh = _fp16(d) + v = (x * idv[:, None]).astype(f32) + # C `roundf` rounds half AWAY FROM ZERO; `np.rint` rounds half to EVEN. + qs = np.where(v >= f32(0), np.floor((v + f32(0.5)).astype(f32)), + np.ceil((v - f32(0.5)).astype(f32))).astype(np.int8) + out = np.zeros((nb, 34), dtype=np.uint8) + out[:, 0:2] = dh.view(np.uint8).reshape(nb, 2) + out[:, 2:34] = qs.view(np.uint8) + return out + + +def encode(rows, ggml_type): + """Encode a 2-D f32 array (rows x ne0) into the ggml block layout.""" + rows = np.ascontiguousarray(rows, dtype=f32) + if ggml_type == GGML_F32: + return rows.tobytes() + if ggml_type == GGML_F16: + return rows.astype(np.float16).tobytes() + if ggml_type == GGML_BF16: + u = rows.view(np.uint32) + # round-to-nearest-even bf16, the same rounding the loader's reader + # expects; a plain truncation biases every weight toward zero. + rounded = ((u + np.uint32(0x7FFF) + ((u >> np.uint32(16)) & np.uint32(1))) + >> np.uint32(16)).astype(np.uint16) + return rounded.tobytes() + n_elem, _ = TYPE_BLOCK[ggml_type] + flat = rows.reshape(-1, n_elem) + if ggml_type == GGML_Q8_0: + return quantize_q8_0(flat).tobytes() + if ggml_type == GGML_Q2_K: + return quantize_q2_k(flat).tobytes() + if ggml_type == GGML_Q6_K: + return quantize_q6_k(flat).tobytes() + raise ValueError("no encoder for ggml type %d" % ggml_type) + + +def type_nbytes(shape, ggml_type): + n = 1 + for d in shape: + n *= d + n_elem, blk_bytes = TYPE_BLOCK[ggml_type] + if n % n_elem: + raise ValueError("element count %d is not a multiple of the %s block %d" + % (n, TYPE_NAME[ggml_type], n_elem)) + return (n // n_elem) * blk_bytes + + +# --------------------------------------------------------------------------- +# safetensors reading. Header only for the plan pass; mmap slices for the data +# pass, so the peak resident set is one tensor rather than one shard. + +_ST_DTYPE = { + "F64": np.float64, "F32": np.float32, "F16": np.float16, + "BF16": np.uint16, "F8_E4M3": np.uint8, "F8_E5M2": np.uint8, + "I64": np.int64, "I32": np.int32, "I16": np.int16, "I8": np.int8, + "U8": np.uint8, "BOOL": np.bool_, +} + + +class Safetensors: + """Lazy multi-shard reader: headers up front, payload on demand.""" + + def __init__(self, src): + self.src = src + index = os.path.join(src, "model.safetensors.index.json") + if os.path.exists(index): + with open(index) as fh: + self.weight_map = json.load(fh)["weight_map"] + else: + single = os.path.join(src, "model.safetensors") + if not os.path.exists(single): + raise SystemExit( + "convert-glm5-next-gguf: %s has neither " + "model.safetensors.index.json nor model.safetensors" % src) + self.weight_map = None + self._headers = {} + self._maps = {} + if self.weight_map is None: + self.weight_map = {k: "model.safetensors" + for k in self._header("model.safetensors") + if k != "__metadata__"} + + def _header(self, shard): + if shard not in self._headers: + path = os.path.join(self.src, shard) + with open(path, "rb") as fh: + n = struct.unpack("> 7) else 1.0 + e = (b >> 3) & 0xF + m = b & 0x7 + if e == 0xF and m == 0x7: + out[b] = np.float32("nan") + elif e == 0: + out[b] = np.float32(s * (m / 8.0) * (2.0 ** -6)) + else: + out[b] = np.float32(s * (1.0 + m / 8.0) * (2.0 ** (e - 7))) + _E4M3_TABLE = out + return _E4M3_TABLE + + +def dequant_fp8_block(q_u8, scale_inv, block): + """FP8 e4m3 with a `weight_scale_inv` block grid -> f32. + + The checkpoint stores `[128, 128]` block scales and `w = q * scale_inv`, + which is the DeepSeek/vLLM block-wise FP8 convention this repository already + carries. Trailing partial blocks take the last block's scale, which is what + the ceil-division grid in the shard headers encodes (a `[2048, 4096]` expert + has a `[16, 32]` scale grid). + """ + q = _e4m3_table()[q_u8] + rows, cols = q.shape + br, bc = block + sr, sc = scale_inv.shape + ri = np.minimum(np.arange(rows) // br, sr - 1) + ci = np.minimum(np.arange(cols) // bc, sc - 1) + return (q * scale_inv[np.ix_(ri, ci)].astype(f32)).astype(f32) + + +# --------------------------------------------------------------------------- +# GGUF writing. + +def _pack_str(s): + b = s.encode("utf-8") + return struct.pack(" GGUF tensor name. +# +# Spellings are llama.cpp's at b10451 (`src/llama-arch.cpp`, anchors in the +# module docstring); the HF side is what the real checkpoint's +# `model.safetensors.index.json` carries, read live 2026-08-26 over 76,108 +# tensors. `model.language_model.` is stripped because `Glm5NextForConditional +# Generation` nests the text model under the multimodal wrapper and the GGUF +# block namespace is flat. + +KDA_MAP = { + "self_attn.q_proj.weight": "attn_q.weight", + "self_attn.k_proj.weight": "attn_k.weight", + "self_attn.v_proj.weight": "attn_v.weight", + "self_attn.o_proj.weight": "attn_output.weight", + "self_attn.q_conv1d.weight": "ssm_conv1d_q.weight", + "self_attn.k_conv1d.weight": "ssm_conv1d_k.weight", + "self_attn.v_conv1d.weight": "ssm_conv1d_v.weight", + "self_attn.f_a_proj.weight": "ssm_f_a.weight", + "self_attn.f_b_proj.weight": "ssm_f_b.weight", + "self_attn.g_a_proj.weight": "ssm_g_a.weight", + "self_attn.g_b_proj.weight": "ssm_g_b.weight", + "self_attn.b_proj.weight": "ssm_beta.weight", + "self_attn.A_log": "ssm_a", + "self_attn.dt_bias": "ssm_dt", + "self_attn.o_norm.weight": "ssm_norm.weight", +} + +DSA_MAP = { + "self_attn.q_a_proj.weight": "attn_q_a.weight", + "self_attn.q_a_layernorm.weight": "attn_q_a_norm.weight", + "self_attn.q_b_proj.weight": "attn_q_b.weight", + "self_attn.kv_a_proj_with_mqa.weight": "attn_kv_a_mqa.weight", + "self_attn.kv_a_layernorm.weight": "attn_kv_a_norm.weight", + "self_attn.kv_b_proj.weight": "attn_kv_b.weight", + "self_attn.o_proj.weight": "attn_output.weight", + "self_attn.indexer.wq_b.weight": "indexer.attn_q_b.weight", + "self_attn.indexer.wk.weight": "indexer.attn_k.weight", + "self_attn.indexer.k_norm.weight": "indexer.k_norm.weight", + "self_attn.indexer.k_norm.bias": "indexer.k_norm.bias", + "self_attn.indexer.weights_proj.weight": "indexer.proj.weight", + "self_attn.indexer.index_kpool_compress_ape": "indexer_compressor_ape.weight", + "self_attn.indexer.index_kpool_compress_gate": "indexer_compressor_gate.weight", +} + +COMMON_MAP = { + "input_layernorm.weight": "attn_norm.weight", + "post_attention_layernorm.weight": "ffn_norm.weight", + # mHC lives FLAT on the layer in the checkpoint, not under `attn_hc.*`. + # `hc_head.*` does not exist at any layer, which is what settles the + # unweighted-mean head collapse; nothing here allocates one. + "hc_attn_fn": "hc_attn_fn.weight", + "hc_attn_base": "hc_attn_base.weight", + "hc_attn_scale": "hc_attn_scale.weight", + "hc_ffn_fn": "hc_ffn_fn.weight", + "hc_ffn_base": "hc_ffn_base.weight", + "hc_ffn_scale": "hc_ffn_scale.weight", +} + +DENSE_MLP_MAP = { + "mlp.gate_proj.weight": "ffn_gate.weight", + "mlp.up_proj.weight": "ffn_up.weight", + "mlp.down_proj.weight": "ffn_down.weight", +} + +SPARSE_MLP_MAP = { + "mlp.gate.weight": "ffn_gate_inp.weight", + "mlp.gate.e_score_correction_bias": "exp_probs_b.bias", + "mlp.shared_experts.gate_proj.weight": "ffn_gate_shexp.weight", + "mlp.shared_experts.up_proj.weight": "ffn_up_shexp.weight", + "mlp.shared_experts.down_proj.weight": "ffn_down_shexp.weight", +} + +# Per-expert tensors are STACKED: the checkpoint stores +# `mlp.experts.{0..287}.{gate,up,down}_proj.weight` and the GGUF carries one +# 3-D tensor per projection, which is llama.cpp's `ffn_*_exps` convention and +# our own reader's `kStackedExpertWeight` role. +EXPERT_STACK = { + "gate_proj": "ffn_gate_exps.weight", + "up_proj": "ffn_up_exps.weight", + "down_proj": "ffn_down_exps.weight", +} + +VISION_MAP = { + "patch_embed.proj.weight": "v.patch_embd.weight", + "patch_embed.proj.bias": "v.patch_embd.bias", + "post_layernorm.weight": "v.post_ln.weight", + "downsample.weight": "v.downsample.weight", + "downsample.bias": "v.downsample.bias", + "merger.proj.weight": "v.merger.proj.weight", + "merger.gate_proj.weight": "v.merger.gate.weight", + "merger.up_proj.weight": "v.merger.up.weight", + "merger.down_proj.weight": "v.merger.down.weight", + "merger.post_projection_norm.weight": "v.merger.norm.weight", + "merger.post_projection_norm.bias": "v.merger.norm.bias", +} + +VISION_BLOCK_MAP = { + "norm1.weight": "ln1.weight", + "norm2.weight": "ln2.weight", + "attn.qkv.weight": "attn_qkv.weight", + "attn.qkv.bias": "attn_qkv.bias", + "attn.proj.weight": "attn_out.weight", + "attn.proj.bias": "attn_out.bias", + "attn.q_norm.weight": "attn_q_norm.weight", + "attn.k_norm.weight": "attn_k_norm.weight", + "mlp.gate_proj.weight": "ffn_gate.weight", + "mlp.gate_proj.bias": "ffn_gate.bias", + "mlp.up_proj.weight": "ffn_up.weight", + "mlp.up_proj.bias": "ffn_up.bias", + "mlp.down_proj.weight": "ffn_down.weight", + "mlp.down_proj.bias": "ffn_down.bias", +} + + +# --------------------------------------------------------------------------- + +def pick_type(gguf_name, shape, arm): + """Resolve the ggml type for one output tensor. + + Two rules, and both are refusals to guess rather than preferences: + + - A 1-D tensor (norm, bias, `A_log`, `dt_bias`, the mHC scalars) stays + F32. It is a vector, not a GEMM operand; quantizing it saves nothing and + costs accuracy on every layer. + - A quantized type needs `ne0 % block == 0`. `ne0` is the ROW length, the + contraction dimension. Where a k-quant's 256-element super-block does not + divide the row -- the k-pool positional embedding at 128, the vision + head norms -- the tensor steps DOWN to Q8_0, whose block is 32, and only + falls all the way to F32 when even 32 does not divide (the depthwise conv + kernels at 4) or the tensor is 4-D or 5-D (the patch-embed and downsample + kernels). Stepping down rather than jumping to F32 is what keeps the arms + ordered by size; an arm whose fallback is 32 bits can be LARGER than a + coarser one, which is not a property a size table should have. Nothing is + reshaped to make a row divide, and every fallback shows in the plan. + """ + types = ARMS[arm] + want = types["experts"] if gguf_name.endswith("_exps.weight") else types["other"] + quantized = (GGML_Q8_0, GGML_Q2_K, GGML_Q6_K) + if want not in quantized: + return want + if len(shape) < 2 or len(shape) > 3: + return GGML_F32 + ne0 = shape[-1] + for candidate in (want, GGML_Q8_0): + n_elem, _ = TYPE_BLOCK[candidate] + if ne0 % n_elem == 0: + return candidate + return GGML_F32 + + +class Plan: + """One output tensor: where its bytes come from and what they become.""" + + def __init__(self, gguf_name, sources, shape, ggml_type, kind): + self.gguf_name = gguf_name + self.sources = sources # list of HF tensor names, stacked in order + self.shape = shape # logical shape, numpy/torch order + self.ggml_type = ggml_type + self.kind = kind # "plain" | "stacked" + self.nbytes = type_nbytes(shape, ggml_type) + + @property + def dims_ggml(self): + return list(reversed(self.shape)) + + +def _layer_types(text_cfg): + lt = text_cfg.get("layer_types") + if not lt: + raise SystemExit( + "convert-glm5-next-gguf: text_config carries no `layer_types`. That " + "list is the AUTHORITATIVE per-layer schedule for this architecture " + "-- the reference ignores `linear_attn_config.kda_layers` and " + "`full_attn_layers` entirely -- so there is nothing to fall back to.") + return lt + + +def build_plan(st, cfg, arm, keep_mtp, want_vision): + text = cfg["text_config"] + n_layers = int(text["num_hidden_layers"]) + layer_types = _layer_types(text) + if len(layer_types) != n_layers: + raise SystemExit("convert-glm5-next-gguf: layer_types has %d entries and " + "num_hidden_layers is %d" % (len(layer_types), n_layers)) + mlp_types = text.get("mlp_layer_types") or [] + n_experts = int(text["n_routed_experts"]) + prefix = "model.language_model." + plans = [] + skipped = [] + + def add(gguf_name, hf_names, shape, kind="plain"): + t = pick_type(gguf_name, shape, arm) + plans.append(Plan(gguf_name, hf_names, shape, t, kind)) + + def logical(name): + # The FP8 form stores one byte per element, so the safetensors shape IS + # the logical shape; the `weight_scale_inv` companion is metadata and is + # never emitted as a tensor of its own. + # + # ggml carries at most GGML_MAX_DIMS = 4 axes, and this checkpoint has + # two tensors that exceed it or sit awkwardly inside it: the vision + # patch embed is a Conv3d at `[1024, 3, 2, 14, 14]` (five axes, which + # simply cannot be written) and the downsample is a Conv2d at + # `[4096, 1024, 2, 2]`. Both are consumed as a matmul over flattened + # patches, so both are flattened to `[out, prod(rest)]` here -- the + # im2col form llama.cpp's own vision converters write. Flattening is + # NOT free and is not applied blindly: it is confined to 4-D and 5-D + # convolution kernels. The 3-D shapes are left alone, because they mean + # something in ggml -- the depthwise KDA conv is `[ch, 1, k]` and the + # stacked expert lane is `[experts, n, m]`, and reshaping either would + # change how the loader indexes it. + shape = st.shape(name) + if len(shape) > 3: + n = 1 + for d in shape[1:]: + n *= d + return [shape[0], n] + return shape + + add("token_embd.weight", [prefix + "embed_tokens.weight"], + logical(prefix + "embed_tokens.weight")) + add("output_norm.weight", [prefix + "norm.weight"], logical(prefix + "norm.weight")) + if st.has("lm_head.weight"): + add("output.weight", ["lm_head.weight"], logical("lm_head.weight")) + + for L in range(n_layers): + base = "%slayers.%d." % (prefix, L) + table = dict(COMMON_MAP) + table.update(KDA_MAP if layer_types[L] == "linear_attention" else DSA_MAP) + for hf_suffix, gg_suffix in sorted(table.items()): + hf = base + hf_suffix + if not st.has(hf): + continue + add("blk.%d.%s" % (L, gg_suffix), [hf], logical(hf)) + sparse = (mlp_types[L] if L < len(mlp_types) + else ("sparse" if st.has(base + "mlp.gate.weight") else "dense")) + if sparse == "dense": + for hf_suffix, gg_suffix in sorted(DENSE_MLP_MAP.items()): + hf = base + hf_suffix + if st.has(hf): + add("blk.%d.%s" % (L, gg_suffix), [hf], logical(hf)) + else: + for hf_suffix, gg_suffix in sorted(SPARSE_MLP_MAP.items()): + hf = base + hf_suffix + if st.has(hf): + add("blk.%d.%s" % (L, gg_suffix), [hf], logical(hf)) + for proj, gg_suffix in sorted(EXPERT_STACK.items()): + names = ["%smlp.experts.%d.%s.weight" % (base, e, proj) + for e in range(n_experts)] + missing = [n for n in names if not st.has(n)] + if missing: + raise SystemExit( + "convert-glm5-next-gguf: layer %d is sparse but %d of its " + "%d `%s` expert tensors are absent (first: %s)" + % (L, len(missing), n_experts, proj, missing[0])) + per = logical(names[0]) + add("blk.%d.%s" % (L, gg_suffix), names, [n_experts] + per, "stacked") + + # The MTP block. `num_hidden_layers` is 45 and the checkpoint carries 46 + # layer directories; `model.language_model.layers.45.*` is a DeepSeek-V3 + # style multi-token-prediction block that the transformers reference + # DISCARDS (`_keys_to_ignore_on_load_unexpected` names `layers\.45\.` and + # `shared_head\.`). We follow the reference and the in-tree + # `glm4_moe_lite_registry.cpp:21-26` precedent and request only + # `[0, num_hidden_layers)`. It is 888 tensors and 7.43B parameters -- 2.31% + # of the model, ~2.3 GiB of a Q2_K arm. + for name in st.weight_map: + if name.startswith("%slayers.%d." % (prefix, n_layers)) or ".shared_head." in name: + skipped.append(name) + if keep_mtp and skipped: + raise SystemExit( + "convert-glm5-next-gguf: --keep-mtp is REFUSED. Missing part: the " + "multi-token-prediction head is not implemented on the loader side " + "either, so a GGUF carrying %d layer-%d tensors would be %d tensors " + "nothing can read. Owed as O2 in .agents/specs/glm5-next-flash.md." + % (len(skipped), n_layers, len(skipped))) + + vision_plans = [] + if want_vision: + vcfg = cfg.get("vision_config") or {} + depth = int(vcfg.get("depth", 0)) + for hf_suffix, gg_name in sorted(VISION_MAP.items()): + hf = "model.visual." + hf_suffix + if st.has(hf): + t = pick_type(gg_name, logical(hf), arm) + vision_plans.append(Plan(gg_name, [hf], logical(hf), t, "plain")) + for b in range(depth): + for hf_suffix, gg_suffix in sorted(VISION_BLOCK_MAP.items()): + hf = "model.visual.blocks.%d.%s" % (b, hf_suffix) + if st.has(hf): + gg = "v.blk.%d.%s" % (b, gg_suffix) + t = pick_type(gg, logical(hf), arm) + vision_plans.append(Plan(gg, [hf], logical(hf), t, "plain")) + plans.extend(vision_plans) + return plans, skipped + + +def write_metadata(w, cfg, arm, n_layers, name): + text = cfg["text_config"] + vis = cfg.get("vision_config") or {} + lin = text.get("linear_attn_config") or {} + layer_types = _layer_types(text) + + def u32(k, v): + w.add(k, GV_UINT32, int(v)) + + def f(k, v): + w.add(k, GV_FLOAT32, float(v)) + + def s(k, v): + w.add(k, GV_STRING, str(v)) + + def b(k, v): + w.add(k, GV_BOOL, bool(v)) + + s("general.architecture", ARCH) + s("general.type", "model") + s("general.name", name) + s("general.basename", "GLM-5.3-Flash") + s("general.size_label", "%dA%dB" % ( + round(int(text["n_routed_experts"]) * int(text["moe_intermediate_size"]) + * 3 * int(text["hidden_size"]) / 1e9), + round(int(text["num_experts_per_tok"]) * int(text["moe_intermediate_size"]) + * 3 * int(text["hidden_size"]) / 1e9))) + # `general.file_type` is a UINT32 enum upstream, not a string: + # `llama_ftype` in `include/llama.h:117-149` @ b10451. A mixed arm + # reports the type its BULK carries, which is what "MOSTLY" means + # there, so the q2_k arm reports Q2_K even though the other 3% is + # Q6_K. The arm name travels beside it as a string, because "q2_k" + # names a policy of ours that no upstream enum has a value for. + u32("general.file_type", FTYPE[arm]) + s("general.quantization_arm", arm) + u32("general.quantization_version", 2) + u32("general.alignment", GGUF_ALIGNMENT) + + k = ARCH + u32("%s.vocab_size" % k, text["vocab_size"]) + u32("%s.context_length" % k, text["max_position_embeddings"]) + u32("%s.embedding_length" % k, text["hidden_size"]) + u32("%s.block_count" % k, n_layers) + u32("%s.feed_forward_length" % k, text["intermediate_size"]) + u32("%s.expert_feed_forward_length" % k, text["moe_intermediate_size"]) + u32("%s.expert_shared_feed_forward_length" % k, text["moe_intermediate_size"]) + u32("%s.leading_dense_block_count" % k, + sum(1 for t in (text.get("mlp_layer_types") or []) if t == "dense")) + u32("%s.expert_count" % k, text["n_routed_experts"]) + u32("%s.expert_used_count" % k, text["num_experts_per_tok"]) + u32("%s.expert_shared_count" % k, text["n_shared_experts"]) + u32("%s.expert_group_count" % k, text["n_group"]) + u32("%s.expert_group_used_count" % k, text["topk_group"]) + f("%s.expert_weights_scale" % k, text["routed_scaling_factor"]) + b("%s.expert_weights_norm" % k, text["norm_topk_prob"]) + # `scoring_func: sigmoid` -> gating func 2, llama.cpp's + # `LLAMA_EXPERT_GATING_FUNC_TYPE_SIGMOID`. + u32("%s.expert_gating_func" % k, 2 if text["scoring_func"] == "sigmoid" else 1) + u32("%s.attention.head_count" % k, text["num_attention_heads"]) + u32("%s.attention.head_count_kv" % k, text["num_key_value_heads"]) + f("%s.attention.layer_norm_rms_epsilon" % k, text["rms_norm_eps"]) + u32("%s.attention.q_lora_rank" % k, text["q_lora_rank"]) + u32("%s.attention.kv_lora_rank" % k, text["kv_lora_rank"]) + u32("%s.attention.key_length_mla" % k, text["qk_head_dim"]) + u32("%s.attention.value_length_mla" % k, text["v_head_dim"]) + u32("%s.attention.key_length" % k, text["qk_nope_head_dim"]) + u32("%s.attention.value_length" % k, text["v_head_dim"]) + f("%s.swiglu_clamp_exp" % k, text["swiglu_limit"]) + f("%s.swiglu_clamp_shexp" % k, text["swiglu_limit"]) + u32("%s.nextn_predict_layers" % k, 0) # the MTP tail is not carried; O2. + + # NO ROPE. `qk_rope_head_dim` is 0 and `mla_use_nope` is true; the reference + # deletes the inherited rope parameters and passes `position_embeddings=None` + # to every layer. `indexer_rope_interleave` is a VESTIGIAL config flag the + # indexer override ignores, so no `{arch}.rope.*` key is written and a reader + # that finds one here is reading a different file. + u32("%s.rope.dimension_count" % k, 0) + + # DSA indexer geometry. Keys: llama.cpp b10451 + # `src/llama-arch.cpp:272-277`. + u32("%s.attention.indexer.head_count" % k, text["index_n_heads"]) + u32("%s.attention.indexer.key_length" % k, text["index_head_dim"]) + u32("%s.attention.indexer.top_k" % k, text["index_topk"]) + w.add("%s.attention.indexer.types" % k, GV_ARRAY, + (GV_STRING, list(text.get("indexer_types") or []))) + # The k-pool compression stage has NO upstream key at any revision, because + # no upstream implements this indexer. These three are OURS, namespaced under + # the indexer they belong to. `index_kpool` is 4 HERE and the config class + # default is 16, so a reader that defaults instead of reading is wrong. + u32("%s.attention.indexer.kpool" % k, text["index_kpool"]) + b("%s.attention.indexer.kpool_compress" % k, text["index_kpool_compress"]) + b("%s.attention.indexer.kpool_always_select_tail" % k, + text["index_kpool_always_select_tail"]) + + # KDA. Keys: llama.cpp b10451 `gguf-py/gguf/constants.py:262-264` / + # `src/llama-arch.cpp:320-321`. `gate_lower_bound` is the parameter this port + # hinges on: it is -5.0 here and NOT None, which selects + # `-bound * sigmoid(exp(A_log) * (g + dt_bias))`, a DIFFERENT function from + # the `-exp(A_log) * softplus(g + dt_bias)` our Kimi-Linear KDA implements -- + # different formula, and the sign of `decay_rate` differs too. Writing the + # value into the file is what lets the loader take the right branch instead + # of inheriting Kimi's. + u32("%s.kda.head_dim" % k, lin["head_dim"]) + f("%s.kda.gate_lower_bound" % k, lin["gate_lower_bound"]) + u32("%s.attention.linear_head_count" % k, lin["num_heads"]) + u32("%s.ssm.conv_kernel" % k, lin["short_conv_kernel_size"]) + + # mHC. Keys: llama.cpp b10451 `HyperConnection` in constants.py. + # `hc_eps` is a DIFFERENT constant from `rms_norm_eps` (1e-6 vs 1e-5) and is + # added to every Sinkhorn denominator rather than used as a floor. + u32("%s.hyper_connection.count" % k, text["hc_mult"]) + u32("%s.hyper_connection.sinkhorn_iterations" % k, text["hc_sinkhorn_iters"]) + f("%s.hyper_connection.epsilon" % k, text["hc_eps"]) + + # The per-layer schedule. The top-level `layer_types` list is authoritative; + # `linear_attn_config.kda_layers` and `full_attn_layers` are IGNORED by the + # reference and are not carried here, so nothing downstream can read the + # wrong one. + w.add("%s.layer_types" % k, GV_ARRAY, (GV_STRING, list(layer_types))) + if text.get("mlp_layer_types"): + w.add("%s.mlp_layer_types" % k, GV_ARRAY, + (GV_STRING, list(text["mlp_layer_types"]))) + + # Multimodal. Image and video share ONE token id: the processor emits + # `image_token_id` for video frames too and disambiguates by the + # begin/end-of-video SPAN, so all six ids travel together or a reader + # classifies every frame as an image. + for key in ("image_token_id", "video_token_id", "image_start_token_id", + "image_end_token_id", "video_start_token_id", "video_end_token_id"): + if key in cfg: + u32("%s.%s" % (k, key), cfg[key]) + if vis: + u32("%s.vision.block_count" % k, vis["depth"]) + u32("%s.vision.embedding_length" % k, vis["hidden_size"]) + u32("%s.vision.feed_forward_length" % k, vis["intermediate_size"]) + u32("%s.vision.head_count" % k, vis["num_heads"]) + u32("%s.vision.patch_size" % k, vis["patch_size"]) + u32("%s.vision.image_size" % k, vis["image_size"]) + u32("%s.vision.spatial_merge_size" % k, vis["spatial_merge_size"]) + u32("%s.vision.temporal_patch_size" % k, vis["temporal_patch_size"]) + u32("%s.vision.out_embedding_length" % k, vis["out_hidden_size"]) + u32("%s.vision.projection_intermediate_size" % k, + vis["projection_intermediate_size"]) + f("%s.vision.attention.layer_norm_rms_epsilon" % k, vis["rms_norm_eps"]) + f("%s.vision.swiglu_clamp" % k, vis["swiglu_limit"]) + + +def materialize(st, plan, block): + """Read a plan's sources and return its rows as f32 (rows x ne0).""" + ne0 = plan.shape[-1] + chunks = [] + for name in plan.sources: + raw, dt = st.raw(name) + if dt == "F8_E4M3": + scale_name = name[:-len(".weight")] + ".weight_scale_inv" \ + if name.endswith(".weight") else name + ".weight_scale_inv" + if not st.has(scale_name): + raise SystemExit( + "convert-glm5-next-gguf: %s is F8_E4M3 and has no " + "`weight_scale_inv` companion. A block-quantized FP8 tensor " + "without its scales cannot be dequantized, and treating the " + "bytes as values would produce a loadable, wrong file." + % name) + scale, _ = st.raw(scale_name) + vals = dequant_fp8_block(raw, np.asarray(scale, dtype=f32), block) + elif dt == "BF16": + vals = _bf16_to_f32(raw) + else: + vals = np.asarray(raw, dtype=f32) + chunks.append(np.ascontiguousarray(vals, dtype=f32).reshape(-1, ne0)) + return np.concatenate(chunks, axis=0) if len(chunks) > 1 else chunks[0] + + +def main(argv=None): + ap = argparse.ArgumentParser( + description="Convert a GLM-5.3-Flash (`glm5_next`) checkpoint to GGUF.") + ap.add_argument("--src", required=True, help="checkpoint directory") + ap.add_argument("--dst", help="output .gguf (omit with --dry-run)") + ap.add_argument("--arm", default="q2_k", + help="quantization arm: " + ", ".join(sorted(ARMS)) + + " (refused: " + ", ".join(sorted(REFUSED_ARMS)) + ")") + ap.add_argument("--keep-mtp", action="store_true", + help="carry the layer-45 MTP block (refused; see O2)") + ap.add_argument("--no-vision", action="store_true", + help="omit the vision tower") + ap.add_argument("--dry-run", action="store_true", + help="read headers only; print the plan and the exact output size") + ap.add_argument("--name", default="GLM-5.3-Flash", help="general.name") + args = ap.parse_args(argv) + + arm = args.arm.lower() + if arm in REFUSED_ARMS: + sys.stderr.write(refuse_arm(arm) + "\n") + return 3 + if arm not in ARMS: + sys.stderr.write("convert-glm5-next-gguf: unknown arm %r. Producible: %s. " + "Refused by name: %s.\n" + % (args.arm, ", ".join(sorted(ARMS)), + ", ".join(sorted(REFUSED_ARMS)))) + return 2 + if not args.dry_run and not args.dst: + sys.stderr.write("convert-glm5-next-gguf: --dst is required without --dry-run\n") + return 2 + + cfg_path = os.path.join(args.src, "config.json") + if not os.path.exists(cfg_path): + sys.stderr.write("convert-glm5-next-gguf: %s has no config.json\n" % args.src) + return 2 + with open(cfg_path) as fh: + cfg = json.load(fh) + + arch_list = cfg.get("architectures") or [] + if cfg.get("model_type") != "glm5_next" or \ + "Glm5NextForConditionalGeneration" not in arch_list: + sys.stderr.write( + "convert-glm5-next-gguf: this converter handles `model_type: " + "glm5_next` / `Glm5NextForConditionalGeneration` only. This " + "checkpoint declares model_type=%r architectures=%r. Converting it " + "anyway would emit a file whose metadata describes a model the " + "weights are not.\n" % (cfg.get("model_type"), arch_list)) + return 2 + + qcfg = cfg.get("quantization_config") or {} + block = tuple(qcfg.get("weight_block_size") or (128, 128)) + if qcfg and qcfg.get("quant_method") not in (None, "fp8"): + sys.stderr.write( + "convert-glm5-next-gguf: quantization_config.quant_method=%r is not " + "handled. Missing part: only the published `fp8` e4m3 block form and " + "unquantized bf16 checkpoints have a dequant path here.\n" + % qcfg.get("quant_method")) + return 3 + if qcfg and qcfg.get("fmt") not in (None, "e4m3"): + sys.stderr.write("convert-glm5-next-gguf: quantization_config.fmt=%r is " + "not handled; only e4m3.\n" % qcfg.get("fmt")) + return 3 + + st = Safetensors(args.src) + text = cfg["text_config"] + n_layers = int(text["num_hidden_layers"]) + plans, skipped = build_plan(st, cfg, arm, args.keep_mtp, not args.no_vision) + + w = GgufWriter() + write_metadata(w, cfg, arm, n_layers, args.name) + for p in plans: + if len(p.dims_ggml) > 4: + sys.stderr.write( + "convert-glm5-next-gguf: %s has %d axes and ggml carries at most " + "4 (GGML_MAX_DIMS). Writing it would produce a header no reader " + "can index, so nothing is written.\n" + % (p.gguf_name, len(p.dims_ggml))) + return 3 + w.add_tensor(p.gguf_name, p.dims_ggml, p.ggml_type, p.nbytes) + head, data_bytes = w.header_bytes() + + counts = {} + for p in plans: + counts[TYPE_NAME[p.ggml_type]] = counts.get(TYPE_NAME[p.ggml_type], 0) + 1 + total = len(head) + data_bytes + sys.stderr.write( + "convert-glm5-next-gguf: arch=%s arm=%s layers=%d tensors=%d " + "skipped_mtp=%d header=%d data=%d total=%d (%.2f GiB)\n" + % (ARCH, arm, n_layers, len(plans), len(skipped), len(head), data_bytes, + total, total / float(1 << 30))) + sys.stderr.write("convert-glm5-next-gguf: types " + " ".join( + "%s=%d" % (t, c) for t, c in sorted(counts.items())) + "\n") + if args.dry_run: + for p in plans: + sys.stdout.write("%s\t%s\t%s\t%d\n" + % (p.gguf_name, TYPE_NAME[p.ggml_type], + ",".join(str(d) for d in p.dims_ggml), p.nbytes)) + return 0 + + with open(args.dst, "wb") as out: + out.write(head) + for p in plans: + rows = materialize(st, p, block) + blob = encode(rows, p.ggml_type) + if len(blob) != p.nbytes: + raise SystemExit( + "convert-glm5-next-gguf: %s encoded to %d bytes, the header " + "reserved %d. The file would be unreadable, so nothing is " + "written past this point." % (p.gguf_name, len(blob), p.nbytes)) + out.write(blob) + pad = (-len(blob)) % GGUF_ALIGNMENT + if pad: + out.write(b"\x00" * pad) + sys.stderr.write("convert-glm5-next-gguf: wrote %s\n" % args.dst) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json b/tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json new file mode 100644 index 000000000..051facdb3 --- /dev/null +++ b/tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json @@ -0,0 +1,22 @@ +{ + "provenance": { + "oracle": "llama.cpp", + "pin": "b10451", + "object": "10bf611e533d81f739128304991c5e133c6aebd8", + "source": "ggml/src/ggml-quants.c", + "anchors": { + "nearest_int": 621, + "make_qx_quants": 628, + "make_qkx2_quants": 799, + "quantize_row_q2_K_ref": 891, + "quantize_row_q6_K_ref": 1869, + "quantize_row_q8_0_ref": 276 + }, + "harness": "the five functions above spliced verbatim into a standalone C file, gcc -O2 -ffp-contract=off -std=c11, _Float16 for GGML_FP32_TO_FP16", + "captured": "2026-08-26" + }, + "input_f32_le_b64": "6BopPNiX4Lo9TQy9THamvGh7YTy+TrU8vojXN0e9Rb2fizy8dH5QOAfnObvKJpQ87CD0PEWiMTxupj48UJ+9vAgQ/7oZzhC9yX2DN3l2Jzu97vc8mrPaPMolAj3rPRc8gwIbvSSbmDx2uNY83u/1O3ai2DwshFC6UvWZO4hKlzxxVTw8tzeIvMYAG7w96iQ8aIPbPHn6oruSm5A7CrJPPJaFejqEZaY8o/eGu9uvWbzdZK68DjCXvMqvGbxrFja4b/5hvbHKj7sgVlu7zGdRvcP7UbsQvNE7s4QNvI6Wd7yMf2O9fh8UvCJKmjwsI6S8re6AupGMKTyfwjw8rihhPD9rKrxL0oE800rHPNDyYjwFtS+8SQXNuW5+n7zkz1U82fF/vIKYGDzVb2c8astNvdMhGT01YOQ6Ptn6vLJVPrwReda3bCZOvDD6CrxqIF68mCfZO+AAlzx/RFo8KYH0PG8mPDtsJLs7WL/wO/vgibwjucY8Kj9IPFNsoDwszFa8G/gTPLsos7x4oAu9KkAXPPE8fDxqkfk80Jc2POiIgjsE/X+5AvucO2A7y7wSbiS8oSf5vAKi7Du1X0+93dOTvOxeDjyQ+2O9J0WFvP36EbuWWsU593eQvM3PqjyZX8s87Zn8PBEIk7zoIjq8ZAfvvJWv37zuzuo78KzOO+lK7jwePtU8bdexO6ssjTzwz2E8xl7fvJL9QTzIwx67ma7avO7ejbyY+aq7CUzWO7XGkjztzno8U4UIvUzFZTwaEw68UbbePHqMR7tTYgM9GQn+O9SswTyyd4q84l+EvNSDxbyYCnq8IAW1O8WLgzupVYm6ilGNuseVCDqmete7SjwcPGPs1zoiPxC5OXpEu8Gd+rxJaLw8DLHHO+ucRzzEYxm7lsCxvP4XtbuOcqC8G2mCOvz9krutQ+W8eQojPR280jzHa0o6GRolPF5Cjrt9M2Y8s6USvODLTD365Eo9vLFUPIQV57klm8u7ZSi0PKRBlzwsSoW8kBemPFT2WDzCZqI8lzOFPJrCWzuyi508Gou/O/UHFT3fAOA7NY6OvK+atbxllN+8BuMCuyKYKjt1+DS98E05vKDJsLwjKyK8UrM2Pe4F0Lut+1m8IL2ZvOLAv7spmoa7qODlPMhcTbxqAIG8fzABvUBarLwrn0A831YBul/s1jyDWCC7R5dCvN5vuTviVoU7J5OuPIB2xDrE14G50cxKPK2YQzzfWuM7uUpbPdUeDT3t++w8W4mXvAAnqjyfbKg5/ygPvWbc/LuL6z28lSi3OzfcaDshr8E5UcjQOy7k0zw0LEI7imWqPA+9pDxM76g81H/NO1Dt9jzJdvK7wXNkPGVnMj0y++e75+QAOwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADjkiC8O9Q/PCVXHL0Uf9W77CmevY/vkjxhDmE9r3jbPLeBnbw2nB695iqmPDO2cL25U9a7PpeYvJX1Gb6DkAs+AqQivQAAYED0kZ292NDEPNkdyDwbWaw8l/AMPc/M8jtd9kw+MhqJPTs/27y/MkS8lvswPWPtqTxoHzE9VkQXPc61g7ilmOu8LJdKPWhTmb0d4zC9untlvUWAgb3bBCs73DwhvTaSyzywBj884DxhPRpvOD1V8Bi9zzxwPnJjbr0bOJe8ORWgvBFclL1W5Wi8joXzvLp4Vzy/l9y9cvwHvqDGkbu1Upc9fFwFPBHuLbz4FPo9hc43vQI2Ijz5vKm8FPgYvYjr9bsPtdm80jbjux9U87yApFY8AKskPkKjwb2l5Jc7Vg63Pc+BcT1y6Du98R2aPPpzQj25YXm9gGVdPXiJxDyjXbK9VcfTu0xTuD3pWAi8MY7nvMSr6zzhz1Q+LNtiPVhbhD2934o9bNGSPT/r8jxFOqI7sR/wvL6D273d9we+FBKWu7KiMb2SViI9MOmWvfuMlTzhb4W9NG8ovQm+nT3ASFM9gkc5vYADrrzej487boPqvHAKaT2VXWe9dkH0vB1Pgj3oXP28iM6TPVWxaLvjNYS9+xwVPTKh67uq7IA9pO+HvE4pur2M3/A9uUEPvGXfGb4YL6u7yNPnOUZBRDo6FKm9t2ekvRCZZL66yy2925erPa4sg7ybpIO9/tNevYvRobxf5YM+kJ8AvNiR5b5GhOU8mHmeuxdFoL3TOBO9JAgbPRLeCD3O2KG8EXMgvDIpTj4yRgc9ifQyPUc+czxpCT29c/5cvFYvBbtv8IQ9sg97PavdnzxTuiy9AUREvSzmT76TkBI7f84mvVqyiD3hG0+9L5KHvQNQKz0PstY89GYNPcXUgTwIHWA9nzeuPcNsFDuEObE9sqUTPcrZ4ryhEHU7i8epO1xbHL6FF4S9AN4CPSsonTxl5s871t5mPaCPBD1YoFQ9PWdcO9HHCD7/V4K9mDhGvd1P2j18j7E8MIuJvXAeSz04zxU8xQatPRw28LziKmS9nJNNPQAAMMBGByW8iEuZvffxGD0fBbu9CPkevSLiYL0n3Z2+SRMuvSD32LxZmLM9KNNAvW16CT7wrKC91w2fO2ALyzy/orA7dk4vvTOaZL5gNLw7E+Q2vHd+CL4gEdU8Rip7vbk2tjs1suK70ltPvlcp0L4pMTY9TLdYvLbbCj+JYIa8pdH/vH4m8buKHUM+TAeqveo5+Tze3gi9PWu2Pc+Ybz38z6a7pqLuPefYYT3xnSK+r7OxPTljAbtD4ha9bNhzuhkMyL3VZSI+G7KSPTiCpT37d8a7xxIAPtbD1zwGJ1o9", + "q2_k_b64": "26pX+9837f6IV3s/zoqeKJtiZanf55qolq5mg3OjF1WSCEqCm2/r+tBbn0prrq7PZ6ZnY7inWtSRlWKHVxBfeVNVE1aTeJiUgEKmFXHVFWY9GJkbAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEQ8BEREREREiARER+xEjERGiUaJRonJSoaFiUaJhmGNklmSkpFiksKhoaORsJJiUZmEpJWlm5mlJaatqaGpqaZTYqBC0mmioaNiIqJ3FpJh3L94x", + "q6_k_b64": "p6/6k5kOQIFpYJ4ccxfXkQ7jkPKZxioI0a+FJmY/tO9iNLtkkDZ7IQ84FaDZRhtQUB1+8g5ky+cQa6sEH0ZHCFjlMM13u873MBvm8Y7xfRilM9CoyMFACB+6zAGh70dm70DyCJ6bxhLB9L9PBEZBfMDaA7fW4BAYELSiVGoDWPOWbSilkuqWpJWiaU4/rgpZoSRmYpdLl4bQVlt2R5mK6ldaW55AVmUoPSmad6rsooFYVohVGHNTV0pJJaZ6FZZlkak8hY1CjICzR1tvZrh7YeyAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAfhJZH2HTGyX8+ESVz3zDW8DQETAQQMAgjn9gYMDwQAAQ9Gqa9kcYYCXNXrkaldE4rMuv7fkzd/HP0eJ+7TXyCwD6GgAdBv8bDA/zD/AC8MqmytqTQlAaSR6oAhDl1k13CQLAitoZBhYkBVJY7CC91ftxsRITBpQB2/bqAfqEV6UpamWpKZpKetmFVmWVWaBbVqRilmZqkkKVXRrWXhZqtqaRqeVpphWpialbqViqWllW2akS9phpqSnaiupdxaWeBoD3Bfr4BQYQ+QgGZQjsBgAT", + "q8_0_b64": "Og4b+6bLJDoAgeIA+TBOHR/D+6MAB1BGVBicMUUURv4MMSoPGtrqFz31Ch0CLvfiz9brAIL2+Iv5D+zdgesr0v4YGh97DuYoPSPl/88h2RgkgV8Es+MA4OveES8iSwcOE9U9HzLfLg8VzrIVI0YZCf8Lx+m7EIzXFIHb+wHYMDlG1+a9whAOQk0MYxVCNZgt95q+7BlEOoE132j0eh5awMKkxhUP/PwC5yRzDgQA+LI6Dx/6yfLOA/W5ZUECGvUk6X9+If/wOC/XMyIywQ0uCjcRaBPOwbL6B4Lgw+R/7trL7/RQ3NOmxCH/S/neEOgOCjME/x0cEH9SRdQxAa3u5Q0IAQ89BzEwMQ9I7iFn7wUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA4nAAD/AP0BAgH//wH+AP/7Bf9//QEBAQEABwL/AAIBAgGRFwDwG9fp4t4B6w0GHhjsf+D29dn48AfGuP4oBPpC6AX1tBbp+/D87ghixgM3JOQLHdshD8v8N/vvEn8iJyksEgPuv9kUkPzbIcIPyd1BLNruBOgw0Oc25j39yR/6NfKzY/mB/AA7GwDp6cH0GPvu8fpJ/oEI/+r2Cwn6/TkJDATz/P8SEQb0jBbigQHnKuDXGhAWCiI1ATYX7wIDoNgUDAQjFCACVNjiQ4slAf0CAAT//QKBAP0C/P798v7/BP4G/AABAP72AP/6Af1gHAH+0aEK/X/8+f4t7Qf4FQ7/Gw3bFAD3AOklERP/HQYM" +} \ No newline at end of file diff --git a/tests/scripts/test_convert_glm5_next_gguf.py b/tests/scripts/test_convert_glm5_next_gguf.py new file mode 100644 index 000000000..b6d667cc7 --- /dev/null +++ b/tests/scripts/test_convert_glm5_next_gguf.py @@ -0,0 +1,865 @@ +#!/usr/bin/env python3 +"""Gate for `scripts/convert-glm5-next-gguf.py` (W7a of MODEL-MM-GLM53-FLASH). + +The converter is the only path to a GLM-5.3-Flash artifact that fits any device +this project reaches, and it has never been run against the real checkpoint -- +that is 300-600 GiB and is owed, not done (spec `## Owed` O7). So the gate is +built to need no checkpoint at all: + + 1. A SYNTHETIC tiny-shape checkpoint, written here, with the real + architecture's topology at ~1/16 the width: the 45-layer KDA/DSA hybrid + becomes 4 layers, 288 experts become 4, and the FP8 e4m3 block form with + `weight_scale_inv` companions is reproduced exactly, because the dequant + path is the one part of the plumbing that silently produces a loadable + wrong file. + 2. An INDEPENDENT GGUF reader, written in this file, that shares no code with + the converter. A parser that reused the writer's structs would prove the + two agree, not that either is right. + 3. A FROZEN k-quant golden captured from the pinned llama.cpp reference + (`tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json`, provenance in + the fixture). Q2_K, Q6_K and Q8_0 must come back BYTE-IDENTICAL. This is the + gate that matters most: a k-quant encoder that is close but not exact + produces a file that loads, generates fluent text, and is quietly worse than + the arm it claims to be. Bytes or nothing. + +The converter's OUTPUT is not yet loadable by this tree -- `glm5next` is not +registered, which W1 owns -- so this suite gates the converter as a command-line +path, entering through `main()` the way a user does. That gap is named in the +commit body, the pull request body, and the spec's `## Owed`. + +Exit 0 iff every case passes. +""" + +import base64 +import json +import os +import struct +import subprocess +import sys +import tempfile + +ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +CONVERTER = os.path.join(ROOT, "scripts", "convert-glm5-next-gguf.py") +FIXTURE = os.path.join(ROOT, "tests", "scripts", "fixtures", + "glm5_next_kquant_golden_b10451.json") + +try: + import numpy as np +except ImportError: + sys.stderr.write( + "SKIP: numpy is not importable here, so nothing about the converter was " + "verified. CI installs python3-numpy and runs this suite.\n") + raise SystemExit(0) + +FAILURES = [] + + +def check(cond, msg): + if cond: + print(" ok %s" % msg) + else: + print(" FAIL %s" % msg) + FAILURES.append(msg) + + +# --------------------------------------------------------------------------- +# The synthetic checkpoint. Shapes are the real config's relations at a small +# width, so a hard-coded dimension in the converter shows up as a shape error +# rather than passing by luck. + +HIDDEN = 256 +N_LAYERS = 4 +LAYER_TYPES = ["linear_attention", "linear_attention", + "deepseek_sparse_attention", "linear_attention"] +MLP_TYPES = ["dense", "sparse", "sparse", "sparse"] +N_EXPERTS = 4 +MOE_INTER = 256 +DENSE_INTER = 512 +VOCAB = 512 +HC_MULT = 4 +KDA_HEADS = 2 +KDA_HEAD_DIM = 128 +KDA_INNER = KDA_HEADS * KDA_HEAD_DIM +CONV_K = 4 +Q_LORA = 128 +KV_LORA = 64 +QK_NOPE = 64 +V_HEAD = 64 +N_HEADS = 4 +IDX_HEADS = 2 +IDX_HEAD_DIM = 64 +IDX_KPOOL = 4 +VIS_DEPTH = 2 +VIS_HIDDEN = 128 +VIS_INTER = 256 + + +def tiny_config(): + return { + "architectures": ["Glm5NextForConditionalGeneration"], + "model_type": "glm5_next", + "image_token_id": 154854, "video_token_id": 154855, + "image_start_token_id": 154830, "image_end_token_id": 154831, + "video_start_token_id": 154832, "video_end_token_id": 154833, + "tie_word_embeddings": False, + "quantization_config": {"activation_scheme": "dynamic", "fmt": "e4m3", + "quant_method": "fp8", + "weight_block_size": [128, 128], + "modules_to_not_convert": ["hyper_connection"]}, + "text_config": { + "model_type": "glm5_next_text", + "hidden_size": HIDDEN, "num_hidden_layers": N_LAYERS, + "intermediate_size": DENSE_INTER, "moe_intermediate_size": MOE_INTER, + "vocab_size": VOCAB, "max_position_embeddings": 4096, + "layer_types": LAYER_TYPES, "mlp_layer_types": MLP_TYPES, + "indexer_types": ["full"] * N_LAYERS, + "num_attention_heads": N_HEADS, "num_key_value_heads": N_HEADS, + "n_routed_experts": N_EXPERTS, "n_shared_experts": 1, + "num_experts_per_tok": 2, "n_group": 1, "topk_group": 1, + "norm_topk_prob": True, "scoring_func": "sigmoid", + "topk_method": "noaux_tc", "routed_scaling_factor": 2.5, + "first_k_dense_replace": 1, + "rms_norm_eps": 1e-05, "hc_eps": 1e-06, "hc_mult": HC_MULT, + "hc_sinkhorn_iters": 20, "mhc": True, "mla_use_nope": True, + "swiglu_limit": 10.0, "hidden_act": "silu", + "q_lora_rank": Q_LORA, "kv_lora_rank": KV_LORA, + "qk_head_dim": QK_NOPE, "qk_nope_head_dim": QK_NOPE, + "qk_rope_head_dim": 0, "v_head_dim": V_HEAD, "head_dim": 0, + "index_head_dim": IDX_HEAD_DIM, "index_n_heads": IDX_HEADS, + "index_topk": 2048, "index_kpool": IDX_KPOOL, + "index_kpool_compress": True, "index_kpool_always_select_tail": True, + "index_share_for_mtp_iteration": True, "indexer_rope_interleave": True, + "num_nextn_predict_layers": 1, "moe_router_dtype": "float32", + "linear_attn_config": {"num_heads": KDA_HEADS, + "gate_lower_bound": -5.0, + "head_dim": KDA_HEAD_DIM, + "short_conv_kernel_size": CONV_K, + "kda_layers": [0, 1, 3], + "full_attn_layers": [2]}, + }, + "vision_config": { + "model_type": "glm5_next_vision", "depth": VIS_DEPTH, + "hidden_size": VIS_HIDDEN, "intermediate_size": VIS_INTER, + "num_heads": 2, "patch_size": 14, "image_size": 448, + "spatial_merge_size": 2, "temporal_patch_size": 2, + "out_hidden_size": HIDDEN, "projection_intermediate_size": 512, + "in_channels": 3, "rms_norm_eps": 1e-05, "swiglu_limit": 10.0, + "attention_bias": True, + }, + } + + +def det(shape, seed): + """Deterministic values in a weight-like range, no RNG dependency.""" + n = int(np.prod(shape)) + i = np.arange(n, dtype=np.float64) + v = np.sin(i * 0.7391 + seed * 1.1) * 0.05 + np.cos(i * 0.113 + seed) * 0.01 + return v.astype(np.float32).reshape(shape) + + +def f32_to_e4m3(x): + """Encode f32 -> e4m3 bytes by nearest-value search over the 256 codes. + + Deliberately brute force and written here rather than shared with the + converter: the converter owns DECODE, this owns ENCODE, and the round trip + is only evidence when the two sides are independent. + """ + codes = np.zeros(256, np.float32) + for b in range(256): + s = -1.0 if (b >> 7) else 1.0 + e = (b >> 3) & 0xF + m = b & 0x7 + if e == 0xF and m == 0x7: + codes[b] = np.nan + elif e == 0: + codes[b] = np.float32(s * (m / 8.0) * (2.0 ** -6)) + else: + codes[b] = np.float32(s * (1.0 + m / 8.0) * (2.0 ** (e - 7))) + finite = np.where(np.isfinite(codes))[0] + tbl = codes[finite] + flat = np.asarray(x, np.float32).reshape(-1, 1) + idx = np.argmin(np.abs(flat - tbl.reshape(1, -1)), axis=1) + return finite[idx].astype(np.uint8).reshape(np.shape(x)), tbl[idx].astype( + np.float32).reshape(np.shape(x)) + + +def st_write(path, tensors): + """Minimal safetensors writer. `tensors` maps name -> (dtype, ndarray).""" + header = {} + blobs = [] + off = 0 + for name in sorted(tensors): + dt, arr = tensors[name] + raw = arr.tobytes() + header[name] = {"dtype": dt, "shape": list(arr.shape), + "data_offsets": [off, off + len(raw)]} + blobs.append(raw) + off += len(raw) + hb = json.dumps(header).encode("utf-8") + hb += b" " * ((-len(hb)) % 8) + with open(path, "wb") as fh: + fh.write(struct.pack("> np.uint32(16)) & np.uint32(1))) + >> np.uint32(16)).astype(np.uint16) + return r.reshape(arr.shape) + + +def fp8_pair(shape, seed, block=(128, 128)): + """An FP8 e4m3 block-quantized tensor plus its `weight_scale_inv` grid.""" + ref = det(shape, seed) + rows, cols = shape + sr = (rows + block[0] - 1) // block[0] + sc = (cols + block[1] - 1) // block[1] + scale = (det((sr, sc), seed + 3) * 0.5 + 1.0).astype(np.float32) + ri = np.minimum(np.arange(rows) // block[0], sr - 1) + ci = np.minimum(np.arange(cols) // block[1], sc - 1) + grid = scale[np.ix_(ri, ci)] + q, decoded = f32_to_e4m3(ref / grid) + return q, scale, (decoded * grid).astype(np.float32) + + +def build_checkpoint(dirpath, with_mtp=True): + """Write config.json and one safetensors shard. Returns the exact values the + converter must reproduce, keyed by HF tensor name.""" + cfg = tiny_config() + with open(os.path.join(dirpath, "config.json"), "w") as fh: + json.dump(cfg, fh) + t = {} + expect = {} + seed = [0] + + def add_bf16(name, shape): + seed[0] += 1 + v = det(shape, seed[0]) + b = bf16(v) + t[name] = (BF16, b) + expect[name] = (b.astype(np.uint32) << np.uint32(16)).view(np.float32) + + def add_f32(name, shape): + seed[0] += 1 + v = det(shape, seed[0]) + t[name] = (F32, v) + expect[name] = v + + def add_fp8(name, shape): + seed[0] += 1 + q, scale, ref = fp8_pair(shape, seed[0]) + t[name] = (FP8, q) + t[name[:-len(".weight")] + ".weight_scale_inv"] = (F32, scale) + expect[name] = ref + + p = "model.language_model." + add_bf16(p + "embed_tokens.weight", (VOCAB, HIDDEN)) + add_bf16(p + "norm.weight", (HIDDEN,)) + add_bf16("lm_head.weight", (VOCAB, HIDDEN)) + + layers = list(range(N_LAYERS)) + ([N_LAYERS] if with_mtp else []) + for L in layers: + b = "%slayers.%d." % (p, L) + add_bf16(b + "input_layernorm.weight", (HIDDEN,)) + add_bf16(b + "post_attention_layernorm.weight", (HIDDEN,)) + is_mtp = L == N_LAYERS + if not is_mtp: + # mHC lives FLAT on the layer and the MTP block carries NONE of it. + add_bf16(b + "hc_attn_fn", (HC_MULT * (HC_MULT + 2), HC_MULT * HIDDEN)) + add_f32(b + "hc_attn_base", (HC_MULT * (HC_MULT + 2),)) + add_f32(b + "hc_attn_scale", (HC_MULT - 1,)) + add_bf16(b + "hc_ffn_fn", (HC_MULT * (HC_MULT + 2), HC_MULT * HIDDEN)) + add_f32(b + "hc_ffn_base", (HC_MULT * (HC_MULT + 2),)) + add_f32(b + "hc_ffn_scale", (HC_MULT - 1,)) + kind = "deepseek_sparse_attention" if is_mtp else LAYER_TYPES[L] + if kind == "linear_attention": + for nm in ("q_proj", "k_proj", "v_proj"): + add_bf16(b + "self_attn.%s.weight" % nm, (KDA_INNER, HIDDEN)) + for nm in ("q_conv1d", "k_conv1d", "v_conv1d"): + add_bf16(b + "self_attn.%s.weight" % nm, (KDA_INNER, 1, CONV_K)) + add_bf16(b + "self_attn.f_a_proj.weight", (KDA_HEAD_DIM, HIDDEN)) + add_bf16(b + "self_attn.f_b_proj.weight", (KDA_INNER, KDA_HEAD_DIM)) + add_bf16(b + "self_attn.g_a_proj.weight", (KDA_HEAD_DIM, HIDDEN)) + add_bf16(b + "self_attn.g_b_proj.weight", (KDA_INNER, KDA_HEAD_DIM)) + add_bf16(b + "self_attn.b_proj.weight", (KDA_HEADS, HIDDEN)) + add_f32(b + "self_attn.A_log", (KDA_HEADS,)) + add_f32(b + "self_attn.dt_bias", (KDA_INNER,)) + add_bf16(b + "self_attn.o_norm.weight", (KDA_HEAD_DIM,)) + add_bf16(b + "self_attn.o_proj.weight", (HIDDEN, KDA_INNER)) + else: + add_fp8(b + "self_attn.q_a_proj.weight", (Q_LORA, HIDDEN)) + add_bf16(b + "self_attn.q_a_layernorm.weight", (Q_LORA,)) + add_fp8(b + "self_attn.q_b_proj.weight", (N_HEADS * QK_NOPE, Q_LORA)) + add_fp8(b + "self_attn.kv_a_proj_with_mqa.weight", (KV_LORA, HIDDEN)) + add_bf16(b + "self_attn.kv_a_layernorm.weight", (KV_LORA,)) + add_bf16(b + "self_attn.kv_b_proj.weight", + (N_HEADS * (QK_NOPE + V_HEAD), KV_LORA)) + add_fp8(b + "self_attn.o_proj.weight", (HIDDEN, N_HEADS * V_HEAD)) + add_bf16(b + "self_attn.indexer.wq_b.weight", + (IDX_HEADS * IDX_HEAD_DIM, Q_LORA)) + add_bf16(b + "self_attn.indexer.wk.weight", (IDX_HEAD_DIM, HIDDEN)) + add_bf16(b + "self_attn.indexer.k_norm.weight", (IDX_HEAD_DIM,)) + add_bf16(b + "self_attn.indexer.k_norm.bias", (IDX_HEAD_DIM,)) + add_bf16(b + "self_attn.indexer.weights_proj.weight", (IDX_HEADS, HIDDEN)) + add_bf16(b + "self_attn.indexer.index_kpool_compress_ape", + (IDX_KPOOL, IDX_HEAD_DIM)) + add_bf16(b + "self_attn.indexer.index_kpool_compress_gate", + (IDX_HEAD_DIM, HIDDEN)) + sparse = True if is_mtp else (MLP_TYPES[L] == "sparse") + if not sparse: + add_fp8(b + "mlp.gate_proj.weight", (DENSE_INTER, HIDDEN)) + add_fp8(b + "mlp.up_proj.weight", (DENSE_INTER, HIDDEN)) + add_fp8(b + "mlp.down_proj.weight", (HIDDEN, DENSE_INTER)) + else: + add_bf16(b + "mlp.gate.weight", (N_EXPERTS, HIDDEN)) + add_f32(b + "mlp.gate.e_score_correction_bias", (N_EXPERTS,)) + for e in range(N_EXPERTS): + add_fp8(b + "mlp.experts.%d.gate_proj.weight" % e, (MOE_INTER, HIDDEN)) + add_fp8(b + "mlp.experts.%d.up_proj.weight" % e, (MOE_INTER, HIDDEN)) + add_fp8(b + "mlp.experts.%d.down_proj.weight" % e, (HIDDEN, MOE_INTER)) + add_fp8(b + "mlp.shared_experts.gate_proj.weight", (MOE_INTER, HIDDEN)) + add_fp8(b + "mlp.shared_experts.up_proj.weight", (MOE_INTER, HIDDEN)) + add_fp8(b + "mlp.shared_experts.down_proj.weight", (HIDDEN, MOE_INTER)) + if is_mtp: + add_bf16(b + "enorm.weight", (HIDDEN,)) + add_bf16(b + "hnorm.weight", (HIDDEN,)) + add_bf16(b + "eh_proj.weight", (HIDDEN, 2 * HIDDEN)) + add_bf16(b + "shared_head.norm.weight", (HIDDEN,)) + + v = "model.visual." + add_bf16(v + "patch_embed.proj.weight", (VIS_HIDDEN, 3, 2, 14, 14)) + add_bf16(v + "patch_embed.proj.bias", (VIS_HIDDEN,)) + add_bf16(v + "post_layernorm.weight", (VIS_HIDDEN,)) + add_bf16(v + "downsample.weight", (HIDDEN, VIS_HIDDEN, 2, 2)) + add_bf16(v + "downsample.bias", (HIDDEN,)) + add_bf16(v + "merger.proj.weight", (HIDDEN, HIDDEN)) + add_bf16(v + "merger.gate_proj.weight", (512, HIDDEN)) + add_bf16(v + "merger.up_proj.weight", (512, HIDDEN)) + add_bf16(v + "merger.down_proj.weight", (HIDDEN, 512)) + add_bf16(v + "merger.post_projection_norm.weight", (HIDDEN,)) + add_bf16(v + "merger.post_projection_norm.bias", (HIDDEN,)) + for blk in range(VIS_DEPTH): + vb = "%sblocks.%d." % (v, blk) + add_bf16(vb + "norm1.weight", (VIS_HIDDEN,)) + add_bf16(vb + "norm2.weight", (VIS_HIDDEN,)) + add_bf16(vb + "attn.qkv.weight", (3 * VIS_HIDDEN, VIS_HIDDEN)) + add_bf16(vb + "attn.qkv.bias", (3 * VIS_HIDDEN,)) + add_bf16(vb + "attn.proj.weight", (VIS_HIDDEN, VIS_HIDDEN)) + add_bf16(vb + "attn.proj.bias", (VIS_HIDDEN,)) + add_bf16(vb + "attn.q_norm.weight", (VIS_HIDDEN // 2,)) + add_bf16(vb + "attn.k_norm.weight", (VIS_HIDDEN // 2,)) + add_bf16(vb + "mlp.gate_proj.weight", (VIS_INTER, VIS_HIDDEN)) + add_bf16(vb + "mlp.gate_proj.bias", (VIS_INTER,)) + add_bf16(vb + "mlp.up_proj.weight", (VIS_INTER, VIS_HIDDEN)) + add_bf16(vb + "mlp.up_proj.bias", (VIS_INTER,)) + add_bf16(vb + "mlp.down_proj.weight", (VIS_HIDDEN, VIS_INTER)) + add_bf16(vb + "mlp.down_proj.bias", (VIS_HIDDEN,)) + + names = st_write(os.path.join(dirpath, "model.safetensors"), t) + with open(os.path.join(dirpath, "model.safetensors.index.json"), "w") as fh: + json.dump({"metadata": {}, "weight_map": + {n: "model.safetensors" for n in names}}, fh) + return expect + + +# --------------------------------------------------------------------------- +# An independent GGUF reader. No code is shared with the converter. + +class Gguf: + _SC = {0: ("> 4).astype(np.float32)) + shift = np.array([0, 2, 4, 6], np.uint8).reshape(1, 1, 4, 1) + q = (qs.reshape(-1, 2, 1, 32) >> shift) & np.uint8(3) + q = q.reshape(-1, 16, 16).astype(np.float32) + out = dl.reshape(-1, 16, 1) * q - ml.reshape(-1, 16, 1) + return out.reshape(-1)[:n] + if ttype == 14: + # Transcribed from `dequantize_row_q6_K` (ggml-quants.c @ b10451): the + # scale index inside a 128-chunk is `l/16` plus 0/2/4/6 for the four + # quarters, and `sc` advances by 8 per chunk. Getting that stride wrong + # decodes a plausible tensor with the wrong scales. + b = np.frombuffer(blob, np.uint8).reshape(-1, 210) + nb = b.shape[0] + ql = b[:, 0:128].reshape(nb, 2, 64).astype(np.int32) + qh = b[:, 128:192].reshape(nb, 2, 32).astype(np.int32) + sc = b[:, 192:208].copy().view(np.int8).astype(np.float32).reshape(nb, 2, 8) + d = b[:, 208:210].copy().view(np.float16).astype(np.float32).reshape(nb, 1) + isx = (np.arange(32) // 16) + out = np.zeros((nb, 256), np.float32) + for half in range(2): + lo = ql[:, half, 0:32] + hi = ql[:, half, 32:64] + h = qh[:, half, :] + quarters = ( + ((lo & 0xF) | (((h >> 0) & 3) << 4), 0, 0), + ((hi & 0xF) | (((h >> 2) & 3) << 4), 32, 2), + ((lo >> 4) | (((h >> 4) & 3) << 4), 64, 4), + ((hi >> 4) | (((h >> 6) & 3) << 4), 96, 6), + ) + for q, dst, sofs in quarters: + s_l = sc[:, half, :][:, isx + sofs] + o = half * 128 + dst + out[:, o:o + 32] = (q.astype(np.float32) - 32.0) * s_l + return (out * d).reshape(-1)[:n] + + raise AssertionError("no decoder for ggml type %d" % ttype) + + +def run(args, cwd=None): + return subprocess.run([sys.executable, CONVERTER] + args, + capture_output=True, text=True, cwd=cwd) + + +# --------------------------------------------------------------------------- + +def case_kquant_golden(): + print("case: k-quant encoders are BYTE-IDENTICAL to llama.cpp b10451") + if not os.path.exists(FIXTURE): + check(False, "fixture %s is present" % FIXTURE) + return + fx = json.load(open(FIXTURE)) + sys.path.insert(0, os.path.join(ROOT, "scripts")) + import importlib.util + spec = importlib.util.spec_from_file_location("conv", CONVERTER) + conv = importlib.util.module_from_spec(spec) + spec.loader.exec_module(conv) + x = np.frombuffer(base64.b64decode(fx["input_f32_le_b64"]), np.float32) + for key, fn, width in (("q2_k", conv.quantize_q2_k, 256), + ("q6_k", conv.quantize_q6_k, 256), + ("q8_0", conv.quantize_q8_0, 32)): + want = np.frombuffer(base64.b64decode(fx[key + "_b64"]), np.uint8) + got = fn(x.reshape(-1, width)).reshape(-1) + check(want.shape == got.shape and bool((want == got).all()), + "%s matches the pinned reference byte-for-byte (%d bytes)" + % (key.upper(), want.size)) + check(fx["provenance"]["pin"] == "b10451", + "the golden names the pin it was captured at (b10451), not `master`") + + +def case_convert(tmp): + print("case: the converter emits a well-formed glm5next GGUF") + src = os.path.join(tmp, "ckpt") + os.makedirs(src, exist_ok=True) + expect = build_checkpoint(src) + dst = os.path.join(tmp, "out-q2k.gguf") + r = run(["--src", src, "--dst", dst, "--arm", "q2_k"]) + check(r.returncode == 0, "q2_k conversion exits 0 (rc=%d)\n%s" + % (r.returncode, r.stderr[-2000:] if r.returncode else "")) + if r.returncode != 0: + return None, expect + g = Gguf(dst) + check(g.version == 3, "GGUF version is 3") + check(g.kv.get("general.architecture") == "glm5next", + "general.architecture is `glm5next`") + check(g.kv.get("glm5next.block_count") == N_LAYERS, + "block_count is %d" % N_LAYERS) + check(g.kv.get("general.file_type") == 10 + and g.kv.get("general.quantization_arm") == "q2_k", + "general.file_type is the UINT32 `llama_ftype` MOSTLY_Q2_K = 10, not a " + "string, and the arm name travels beside it") + + print("case: the metadata carries the parameters the port hinges on") + check(abs(g.kv.get("glm5next.kda.gate_lower_bound", 0.0) - (-5.0)) < 1e-6, + "`glm5next.kda.gate_lower_bound` is -5.0 -- the value that selects the " + "sigmoid forget-gate branch over Kimi-Linear's softplus one") + check(g.kv.get("glm5next.kda.head_dim") == KDA_HEAD_DIM, + "`glm5next.kda.head_dim` is carried") + check(g.kv.get("glm5next.ssm.conv_kernel") == CONV_K, + "the short-conv kernel size is carried") + check(g.kv.get("glm5next.attention.indexer.kpool") == IDX_KPOOL, + "`indexer.kpool` is %d, the CHECKPOINT value, not the config class " + "default of 16" % IDX_KPOOL) + check(g.kv.get("glm5next.attention.indexer.top_k") == 2048, + "`indexer.top_k` is 2048") + check(g.kv.get("glm5next.hyper_connection.count") == HC_MULT, + "mHC stream count is carried") + check(abs(g.kv.get("glm5next.hyper_connection.epsilon", 0.0) - 1e-6) < 1e-12, + "`hc_eps` is 1e-6 and is a DIFFERENT constant from rms_norm_eps 1e-5") + check(abs(g.kv.get("glm5next.attention.layer_norm_rms_epsilon", 0.0) - 1e-5) + < 1e-11, "`rms_norm_eps` is 1e-5") + check(g.kv.get("glm5next.layer_types") == LAYER_TYPES, + "the AUTHORITATIVE per-layer schedule is carried as `layer_types`") + check("glm5next.rope.freq_base" not in g.kv + and g.kv.get("glm5next.rope.dimension_count") == 0, + "no rope frequency is written: the text stack is NoPE end to end") + check(g.kv.get("glm5next.expert_count") == N_EXPERTS + and g.kv.get("glm5next.expert_used_count") == 2, + "MoE routing width is carried") + check(abs(g.kv.get("glm5next.swiglu_clamp_exp", 0.0) - 10.0) < 1e-6, + "the clamped-SwiGLU limit is carried") + check(g.kv.get("glm5next.image_token_id") == 154854 + and g.kv.get("glm5next.video_token_id") == 154855, + "both multimodal token ids travel together") + return g, expect + + +def case_tensors(g): + print("case: the tensor set is exactly what the topology generates") + want = set() + want |= {"token_embd.weight", "output_norm.weight", "output.weight"} + for L in range(N_LAYERS): + want |= {"blk.%d.attn_norm.weight" % L, "blk.%d.ffn_norm.weight" % L} + for s in ("hc_attn_fn", "hc_attn_base", "hc_attn_scale", + "hc_ffn_fn", "hc_ffn_base", "hc_ffn_scale"): + want.add("blk.%d.%s.weight" % (L, s)) + if LAYER_TYPES[L] == "linear_attention": + for s in ("attn_q", "attn_k", "attn_v", "attn_output", + "ssm_conv1d_q", "ssm_conv1d_k", "ssm_conv1d_v", + "ssm_f_a", "ssm_f_b", "ssm_g_a", "ssm_g_b", + "ssm_beta", "ssm_norm"): + want.add("blk.%d.%s.weight" % (L, s)) + want |= {"blk.%d.ssm_a" % L, "blk.%d.ssm_dt" % L} + else: + for s in ("attn_q_a", "attn_q_a_norm", "attn_q_b", "attn_kv_a_mqa", + "attn_kv_a_norm", "attn_kv_b", "attn_output", + "indexer.attn_q_b", "indexer.attn_k", "indexer.k_norm", + "indexer.proj", "indexer_compressor_ape", + "indexer_compressor_gate"): + want.add("blk.%d.%s.weight" % (L, s)) + want.add("blk.%d.indexer.k_norm.bias" % L) + if MLP_TYPES[L] == "dense": + for s in ("ffn_gate", "ffn_up", "ffn_down"): + want.add("blk.%d.%s.weight" % (L, s)) + else: + for s in ("ffn_gate_inp", "ffn_gate_exps", "ffn_up_exps", + "ffn_down_exps", "ffn_gate_shexp", "ffn_up_shexp", + "ffn_down_shexp"): + want.add("blk.%d.%s.weight" % (L, s)) + want.add("blk.%d.exp_probs_b.bias" % L) + for s in ("v.patch_embd.weight", "v.patch_embd.bias", "v.post_ln.weight", + "v.downsample.weight", "v.downsample.bias", "v.merger.proj.weight", + "v.merger.gate.weight", "v.merger.up.weight", "v.merger.down.weight", + "v.merger.norm.weight", "v.merger.norm.bias"): + want.add(s) + for b in range(VIS_DEPTH): + for s in ("ln1.weight", "ln2.weight", "attn_qkv.weight", "attn_qkv.bias", + "attn_out.weight", "attn_out.bias", "attn_q_norm.weight", + "attn_k_norm.weight", "ffn_gate.weight", "ffn_gate.bias", + "ffn_up.weight", "ffn_up.bias", "ffn_down.weight", + "ffn_down.bias"): + want.add("v.blk.%d.%s" % (b, s)) + got = set(g.tensors) + check(got == want, "tensor set matches (%d expected). missing=%s extra=%s" + % (len(want), sorted(want - got)[:6], sorted(got - want)[:6])) + + print("case: the MTP block is dropped, following the reference") + mtp = [n for n in got if n.startswith("blk.%d." % N_LAYERS)] + check(not mtp, "no `blk.%d.*` tensor survives (found %s)" % (N_LAYERS, mtp[:4])) + check(not any("shared_head" in n or "enorm" in n or "eh_proj" in n for n in got), + "no MTP-only tensor (`enorm`, `hnorm`, `eh_proj`, `shared_head`) survives") + + print("case: the 288-expert lane is STACKED, not one tensor per expert") + for L in range(N_LAYERS): + if MLP_TYPES[L] != "sparse": + continue + info = g.tensors["blk.%d.ffn_gate_exps.weight" % L] + # ggml dims are reversed against the logical shape. + check(info["dims"] == [HIDDEN, MOE_INTER, N_EXPERTS], + "layer %d ffn_gate_exps is [ne0=%d, ne1=%d, ne2=%d experts]" + % (L, HIDDEN, MOE_INTER, N_EXPERTS)) + break + check(not any(".experts." in n for n in got), + "no per-expert tensor name leaks into the file") + + +def case_types(g): + print("case: the q2_k arm puts the EXPERTS at Q2_K and the rest at Q6_K") + exps = [n for n in g.tensors if n.endswith("_exps.weight")] + check(exps and all(g.tensors[n]["type"] == 10 for n in exps), + "all %d stacked expert tensors are Q2_K" % len(exps)) + check(g.tensors["blk.0.attn_q.weight"]["type"] == 14, + "a non-expert 2-D weight is Q6_K") + check(g.tensors["blk.0.attn_norm.weight"]["type"] == 0, + "a 1-D norm stays F32 rather than being quantized") + check(g.tensors["blk.0.ssm_a"]["type"] == 0, "`ssm_a` (A_log) stays F32") + print("case: a row that does not divide the block STEPS DOWN rather than " + "being reshaped") + check(g.tensors["blk.2.indexer_compressor_ape.weight"]["type"] == 8, + "the k-pool positional embedding (ne0=%d, not a multiple of 256) steps " + "down to Q8_0, whose block is 32" % IDX_HEAD_DIM) + check(g.tensors["blk.0.ssm_conv1d_q.weight"]["type"] == 0, + "the depthwise conv kernel (ne0=%d divides neither 256 nor 32) falls " + "all the way to F32" % CONV_K) + print("case: a convolution kernel with more axes than ggml carries is " + "flattened, not written") + # ggml has GGML_MAX_DIMS = 4. The patch embed is a Conv3d at + # `[hidden, 3, 2, 14, 14]` -- five axes -- and writing it verbatim would + # produce a header no reader can index. It is flattened to + # `[hidden, 3*2*14*14]`, whose row (1176) divides neither 256 nor 32, so it + # lands at F32. + check(len(g.tensors["v.patch_embd.weight"]["dims"]) == 2 + and g.tensors["v.patch_embd.weight"]["dims"] == [3 * 2 * 14 * 14, + VIS_HIDDEN], + "the 5-D patch-embed kernel is flattened to [%d, %d] rather than " + "written with five axes" % (3 * 2 * 14 * 14, VIS_HIDDEN)) + check(g.tensors["v.patch_embd.weight"]["type"] == 0, + "the flattened patch-embed row (1176) divides neither 256 nor 32, so " + "it lands at F32") + check(g.tensors["v.downsample.weight"]["dims"] == [VIS_HIDDEN * 2 * 2, HIDDEN] + and g.tensors["v.downsample.weight"]["type"] == 14, + "the 4-D downsample kernel is flattened to a row that DOES divide 256 " + "and is quantized rather than dumped at F32") + check(all(len(i["dims"]) <= 4 for i in g.tensors.values()), + "no tensor in the file carries more than ggml's 4 axes") + check(g.tensors["blk.0.ssm_conv1d_q.weight"]["dims"] == [CONV_K, 1, KDA_INNER] + and g.tensors["blk.1.ffn_gate_exps.weight"]["dims"] + == [HIDDEN, MOE_INTER, N_EXPERTS], + "3-D shapes are NOT flattened: the depthwise conv stays [k, 1, ch] and " + "the expert lane stays [m, n, experts], because ggml indexes both") + + +def case_values(g, expect): + print("case: FP8 e4m3 block dequant and bf16 widening reach the file intact") + p = "model.language_model." + name = "blk.0.attn_norm.weight" + ref = expect[p + "layers.0.input_layernorm.weight"].reshape(-1) + got = dequant(g.blob(name, nbytes_of(g.tensors[name])), 0, ref.size) + check(np.array_equal(got, ref), "an F32-carried bf16 norm is bit-exact") + + name = "blk.2.attn_q_a.weight" + info = g.tensors[name] + ref = expect[p + "layers.2.self_attn.q_a_proj.weight"].reshape(-1) + got = dequant(g.blob(name, nbytes_of(info)), info["type"], ref.size) + rel = np.abs(got - ref).max() / max(np.abs(ref).max(), 1e-30) + check(rel < 0.02, "an FP8 e4m3 block-quantized MLA projection round-trips " + "through Q6_K within 2%% (max rel %.4f)" % rel) + + name = "blk.1.ffn_gate_exps.weight" + info = g.tensors[name] + ref = np.concatenate([ + expect[p + "layers.1.mlp.experts.%d.gate_proj.weight" % e].reshape(-1) + for e in range(N_EXPERTS)]) + got = dequant(g.blob(name, nbytes_of(info)), info["type"], ref.size) + rel = np.abs(got - ref).max() / max(np.abs(ref).max(), 1e-30) + check(rel < 0.60, "the stacked expert lane round-trips through Q2_K " + "(max rel %.4f)" % rel) + # Expert ORDER, checked slab by slab rather than by a correlation over the + # whole tensor: a correlation stays high under a permutation of the slabs, + # and it cannot see a scale error at all. Every slab must be nearer its own + # expert than any other, which a swap breaks and a correlation does not. + per = ref.size // N_EXPERTS + refs = [expect[p + "layers.1.mlp.experts.%d.gate_proj.weight" % e].reshape(-1) + for e in range(N_EXPERTS)] + order_ok = True + for e in range(N_EXPERTS): + slab = got[e * per:(e + 1) * per] + errs = [float(np.abs(slab - r).mean()) for r in refs] + if int(np.argmin(errs)) != e: + order_ok = False + check(order_ok, "every expert slab is nearest its OWN expert: the lane is in " + "expert order, not permuted") + + +def case_refusals(tmp): + print("case: an unimplemented arm is REFUSED by name") + src = os.path.join(tmp, "ckpt") + for arm in ("iq2_xxs", "iq1_s", "iq2_s"): + r = run(["--src", src, "--dst", os.path.join(tmp, "x.gguf"), "--arm", arm]) + ok = (r.returncode != 0 and arm.upper() in r.stderr + and "importance matrix" in r.stderr and "circular" in r.stderr) + check(ok, "--arm %s is refused and names the missing part (imatrix) " + "and why it is unreachable here" % arm) + for arm in ("q4_k", "q3_k"): + r = run(["--src", src, "--dst", os.path.join(tmp, "x.gguf"), "--arm", arm]) + check(r.returncode != 0 and arm.upper() in r.stderr + and "encoder is not ported" in r.stderr, + "--arm %s is refused and names the unported encoder" % arm) + r = run(["--src", src, "--dst", os.path.join(tmp, "x.gguf"), + "--arm", "q2_k", "--keep-mtp"]) + check(r.returncode != 0 and "MTP" in r.stderr.upper() + and "O2" in r.stderr, + "--keep-mtp is refused and names the owed item that would carry it") + + print("case: a checkpoint of another architecture is REFUSED, not converted") + other = os.path.join(tmp, "other") + os.makedirs(other, exist_ok=True) + cfg = tiny_config() + cfg["model_type"] = "glm4_moe" + cfg["architectures"] = ["Glm4MoeForCausalLM"] + with open(os.path.join(other, "config.json"), "w") as fh: + json.dump(cfg, fh) + r = run(["--src", other, "--dst", os.path.join(tmp, "x.gguf"), "--arm", "q2_k"]) + check(r.returncode != 0 and "glm5_next" in r.stderr, + "a non-glm5_next config is refused by name") + + print("case: an FP8 tensor with no scale companion is REFUSED, not guessed") + broken = os.path.join(tmp, "broken") + os.makedirs(broken, exist_ok=True) + build_checkpoint(broken) + hdr_path = os.path.join(broken, "model.safetensors") + with open(hdr_path, "rb") as fh: + n = struct.unpack(" Date: Wed, 26 Aug 2026 20:22:50 +0000 Subject: [PATCH 4/5] test(MODEL-MM-GLM53-FLASH): the k-quant golden was blind to both rounding rules it claims to hold (#2011) Mutation found it, which is what mutation is for. Two guarantees the converter states in its own comments -- that `nearest_int` rounds half to EVEN and that C `roundf` rounds half AWAY FROM ZERO -- were not held by the gate. Replacing either rule left the golden green: M1 nearest_int half-to-EVEN -> half-UP gate rc 0, 0 failures GATE BLIND M2 roundf -> np.rint gate rc 0, 0 failures GATE BLIND The cause is the inputs, not the assertion. The golden was captured over weight-like random data, heavy-tailed data and zeros, and none of those ever lands a value exactly on a rounding tie, so the two rules agree everywhere the fixture looks. A mis-encoded `.5` is not a hypothetical here: it is how the `np.rint` defect was found in the first place, by a hand-crafted case that the frozen fixture then failed to preserve. Three super-blocks are added and the golden is recaptured from the same pinned reference. Blocks 3 and 4 were SEARCHED on an eighth-lattice until half-to-even and half-up disagree under Q2_K and under Q6_K respectively -- they are not hand-derived, they are the first blocks found that discriminate. Block 5 is eight Q8_0 sub-blocks whose `amax` is exactly 127, so `x * id` is 0.5, -0.5, 1.5, -1.5, 2.5, -2.5 and the reference emits [1,-1,2,-2,3,-3] where `np.rint` emits [0,0,2,-2,2,-2]. Both mutations now fail, and the other eight in the table were already caught. Also corrects the `.agents/model-matrix.md` rollup, which the row's move to `ACTIVE` invalidated: ACTIVE 10 -> 11, READY 5 -> 4. `scripts/check-model-checklist.py` reports OK. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-fable-5 [Claude Code] --- .agents/model-matrix.md | 4 ++-- .../glm5_next_kquant_golden_b10451.json | 19 ++++++++++++++----- tests/scripts/test_convert_glm5_next_gguf.py | 11 +++++++++++ 3 files changed, 27 insertions(+), 7 deletions(-) diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index a83cb6335..557802c43 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -77,11 +77,11 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): |---|---| | INVENTORIED | 324 | | PARTIAL | 22 | -| ACTIVE | 10 | +| ACTIVE | 11 | | SPIKE | 9 | | BLOCKED | 5 | | DONE | 3 | -| READY | 5 | +| READY | 4 | | GATING | 1 | | **Total** | **379** | diff --git a/tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json b/tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json index 051facdb3..6490b8067 100644 --- a/tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json +++ b/tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json @@ -13,10 +13,19 @@ "quantize_row_q8_0_ref": 276 }, "harness": "the five functions above spliced verbatim into a standalone C file, gcc -O2 -ffp-contract=off -std=c11, _Float16 for GGML_FP32_TO_FP16", - "captured": "2026-08-26" + "captured": "2026-08-26", + "blocks": [ + "0: ordinary weight-like spread", + "1: all zeros (the degenerate max==min and amax Date: Wed, 26 Aug 2026 20:28:53 +0000 Subject: [PATCH 5/5] record(MODEL-MM-GLM53-FLASH): move the glm5_next state pin to ACTIVE, and repair the two anchors that edit shifted (#2011) `test_glm5_next_row_is_inside_the_model_ratchet` pins the row's lifecycle state as a literal, and its own docstring gives the premise: `READY` because the spec was committed and no product code had landed. W7a landed product code, so the premise expired and the pin moves with it, in the same change that moves the matrix row. The assertion is not weakened -- it still names one exact state, and a pin that followed the row automatically would assert nothing. What it stops catching is the one transition it was updated for; a rename, a second glm5_next row, and any later state change made without touching this file still red it. Editing that docstring added nine lines to `tests/scripts/test_agent_record.py` and shifted `RecordAnchorRatchet` from :1539 to :1548 and `test_one_good_link_does_not_cover_a_rotted_bare_citation` from :1607 to :1616. `ENG-RECORD-ANCHOR-RATCHET` cites both, so the anchor ratchet went 31 -> 33 and the gate reported a regression. The anchors are repaired rather than the baseline raised, which is what the ratchet's own message demands. It is also the failure that row exists to measure, produced the way that row's record says it gets produced: by an edit to the very file the row cites, inside the pull request that makes it. `scripts/check-agent-record.py --report` now reports stale=31 broken=6 rot=37, unchanged from the base, and `tests/scripts/test_agent_record.py` runs 118 tests OK. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-fable-5 [Claude Code] --- .agents/engine-matrix.md | 2 +- tests/scripts/test_agent_record.py | 21 +++++++++++++++------ 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 009e8a3d0..17eec4748 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -227,7 +227,7 @@ claims it. | `ENG-RELEASE-WINDOWS` | Native Windows x86_64 pre-alpha release extension: one adaptive MSVC/UCRT CPU bundle with AVX2 executed in CI and one Vulkan preview bundle, both deterministic ZIPs and authenticated by the existing release handoff | T0 | vLLM has no Windows release path; runtime behavior remains pinned to vLLM `555967922`. Platform substrate reference: llama.cpp `src/llama-mmap.cpp:520-590` @ `237ad9b961f009ae19ac29dbce4cd0c1251f94b3`; Win32 API is the OS authority | W14 Win32 portability/MSVC CPU, W15 deterministic ZIP/PE packaging + Vulkan, and W16 ten-tuple prerelease workflow/version/docs implemented for one PR | Linux portability/release mutation gates are local evidence only. Native `windows-2022` MSVC `/W4 /WX`, extracted runtime/ISA smokes, merged-SHA ten-tuple dry run, `v0.0.3-pre.1` publication, attestations, and exact 32-asset audit remain pending; no Windows ZIP exists yet | [windows-binary-release.md](specs/windows-binary-release.md); [#117](https://github.com/mudler/vllm.cpp/issues/117) | `ACTIVE` | `CLAIM-ENG-RELEASE-WINDOWS` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | `docker/Dockerfile` (cpu/vulkan/cuda targets calling the release scripts); `docker/healthcheck.sh`; `release/container-matrix.json`; `scripts/check-container-matrix.py`; `scripts/check-container-workflow.py`; `scripts/validate-container-image.py`; `scripts/container_tags.py`; `.github/workflows/containers.yml`; SIGTERM handler `src/vllm/entrypoints/openai/server_main.cpp` (`SignalShutdown`, all three `listen()` sites); the pre-existing `docker/Dockerfile.arm64` is an unrelated CPU bench cross-check | issues `#170`, `#312`, `#394`; `tests/scripts/test_check_container_matrix.py` 31/31; `test_check_container_workflow.py` 29/29; `test_check_cuda_fat_gencode.py` 7+4 subtests. **GB10 2026-08-11 (`promaxgb10-4ad8`, `sm_121a`, CUDA 13.3): arm64 cuda image 1.71 GB, 673/673 objects, ten-SM gencode audit PASS, and a REAL GPU boot -- `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, `--gpus all`, host driver 580.159.03 injected.** cpu amd64 783 MB gated locally; cpu+vulkan amd64 green on hosted CI **arm64 cuda lane RUNTIME-VERIFIED on GB10 2026-08-11** -- the first accelerator-hardware evidence for any lane. Four defects were removed to get there, each found by building rather than reading: the CUDA 12.9 base could not compile `sm_110`, the BuildKit cache mount outlived its toolchain (both #366), Marlin gencode had drifted from the feature table and failed the audit on 14 correctly-compiled TUs (#394, blocking BOTH cuda tuples project-wide), and the validator could only ever produce build evidence because its boot smoke never passed `--gpus`. **NOT established: nothing is published to GHCR; amd64 cuda is unbuilt; the published arm64 image is SBSA (`targets/sbsa-linux`), so Tegra -- Thor `sm_110`, Orin `sm_87` -- is untested and NOT covered** **ORIN (Tegra) 2026-08-11: the SBSA image RUNS on Jetson AGX Orin `sm_87` (L4T R36.4.3, Docker 27.5.1) -- Qwen3-0.6B (rev `c1899de2`) loads and GENERATES via `/v1/completions`, tegrastats GR3D 95-97% during decode vs 14-15% idle.** Tegra needs `--runtime nvidia --gpus all`: `--gpus` alone is refused by the hook and `--runtime` alone mounts no driver | [container-images.md](specs/container-images.md); issues [#170](https://github.com/mudler/vllm.cpp/issues/170), [#312](https://github.com/mudler/vllm.cpp/issues/312), [#394](https://github.com/mudler/vllm.cpp/issues/394) | `ACTIVE` | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | | `ENG-DOCS-SITE` | Publish the 11 `docs/*.md` as a browsable GitHub Pages site at `https://mudler.github.io/vllm.cpp/` WITHOUT a second copy of the prose. A Hugo site at `website/` mounts `../docs` READ-ONLY and derives everything else from what is already in the files: each page title from the file's first `# H1`, the sidebar order from `website/data/nav.yaml`, and links through a Goldmark render hook (internal `.md` → site URL; the 139 `../.agents/**` and `../AGENTS.md` escapes → GitHub blob URLs, since the protocol tree is deliberately NOT published). **No file under `docs/` is modified, moved, renamed, or given front matter**, so `check-doc-checkpoint.py` and every protocol path reference keep working and there is no second surface that can drift — the whole point of the row. Custom lean layouts, NO theme and NO submodule: off-the-shelf docs themes read titles, weights and menus out of front matter this design deliberately does not have, so each would need its title partial, menu and link hook overridden anyway, and hugo-book additionally floors at Hugo 0.158 against the 0.146.3 pin CI and the local toolchain share. Hard prerequisite inside the repo: `classify_path` in `scripts/check-pr-size.py` FAILS CLOSED on `website/**` (verified: raises `ValueError: unclassified repository path`), so the classifier must learn the path or the PR cannot pass the project's own size gate. Hard prerequisite outside it: GitHub Pages must be enabled with the source set to GitHub Actions — the workflow is inert otherwise. A marketing landing page is explicitly OUT of scope (`README.md` stays the front door), as is any restructuring of `docs/`; the custom domain is parked behind the pending vLLM trademark question | T1 | NO vLLM analogue — upstream's docs are a separate mkdocs site and nothing in this row mirrors upstream *behavior*, so it carries no parity obligation. The STRUCTURAL reference is LocalAI's `.github/workflows/gh-pages.yml` (two Hugo sites merged into one Pages artifact), reduced to the docs half | read-only mount `website/hugo.toml:29`; title-from-H1 `website/layouts/partials/title.html:10`; link rewriting `website/layouts/_default/_markup/render-link.html:27`; guard `scripts/check-site.py:70`; deploy `.github/workflows/gh-pages.yml` | `tests/scripts/test_check_site.py:51,56,66,80,89,97` (6 mutation cases: clean tree, H1 stripped, doc absent from nav, nav entry with no file, duplicated entry, missing nav file); build evidence 14 pages with `docs/bench-evidence` + `docs/superpowers` absent from `public/` and no `href` ending in `.md`; 48 protocol links rewritten in `docs/status/`. NO published page is claimed: GitHub Pages is not yet enabled on the repository, which is the recorded stop condition holding this row at `GATING` | [gh-pages-docs-site.md](specs/gh-pages-docs-site.md); issue [#224](https://github.com/mudler/vllm.cpp/issues/224) | `READY` | `CLAIM-ENG-DOCS-SITE` | -| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1539` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1607`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | +| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1548` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1616`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | | `ENG-RECORD-CONFLICT-SURFACES` | Retire the shared record surfaces that make concurrent PRs conflict by construction. MEASURED at `origin/main` `d928e2c3` with `git merge-tree --write-tree` over every open PR: **16 of 29 conflict (55%), and 13 of the 16 conflict in bookkeeping files ONLY**, with no product code involved — `.agents/coordination.md` in 8, `.agents/NOW.md` in 5, `.agents/roadmap_v1.md` in 4, `scripts/check-public-doc-tables.py` in 4, `docs/STATUS.md` in 4, and any `src/`/`tests/` path in just 3. Three defects, each of which GUARANTEES rather than risks a collision. (1) `.agents/NOW.md` is a fixed-size shared buffer at EXACTLY 6000/6000 chars (`check-now-current.py:31`), so adding a row requires evicting another and every PR is a read-modify-write of one global — and the conflict is the LUCKY outcome, since a clean three-way merge would apply both evictions and both additions, silently dropping live rows and blowing the very budget the checker defends. (2) `STATUS_RATCHET = {"chars": 243245}` (`check-public-doc-tables.py:557`) is a hardcoded byte count of a DIFFERENT file that may only fall, so a PR owing `docs/STATUS.md` one lifecycle line must delete unrelated prose from another row to pay for it and edit the checker too; the checker's own comment at `:331` already records the failure (*"a ratchet pinned to the byte turns every concurrently merged row's one-line status edit into a spurious failure"*) and answered it with slack instead of removing the coupling. (3) `.agents/coordination.md`'s active-claims table is insert-at-one-anchor: the six ROCm GDN PRs (#334 #336 #341 #343 #345 #348) are ONE author's sequential stack that conflicts on nothing else, each appending a ~1,500-char row — the PR description, transcribed into a file every other claim also writes. It also contradicts the protocol it serves: `AGENTS.md` holds that *"History is git"* and *"There is no state log"*, yet both claims tables ARE state logs duplicating `gh pr list`, `row/` branch names and issue state; the argument that refuses a waiver registry applies unchanged to a claims registry. Precedent twice over — `policy.csv` retired in `0f3e44ee`, per-class line budgets retired 2026-08-10 because the gate fired on ordinary work. The exonerated surfaces share ONE property, one writer per file: `.agents/specs/.md` (one file per row, **zero conflicts** in the sample), the `*-matrix.md` inventories, and the append-only `.agents/benchmark-record.md`. SCOPE: remove `STATUS_RATCHET` and the doc-gating global counters while KEEPING the per-cell/per-paragraph caps (local, so they couple nothing); remove the active-claims table and derive claims from open PRs and branch names; drop `NOW.md`'s byte budget; order the roadmap's keyed tables by ID so distinct keys stop colliding at one anchor; and record the invariant — **no surface that every PR must write** — in `AGENTS.md`. No product source, kernel or gate semantic moves | T0 | NO vLLM analogue — this is local protocol machinery, so the mirror rule does not apply and no upstream `file:line` exists to port from. Governed instead by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | - | - (spec-before-code: the red-before suites are named in the spec's Tests section — `tests/scripts/test_check_public_doc_tables.py`, `tests/scripts/test_check_now_current.py`, a mutation case per removed rule proving the obligation survives in the retained caps and `check-doc-checkpoint.py`, and a `git merge-tree` merge-shape regression that must be RED before the `NOW.md`/roadmap work and GREEN after) | [retire-shared-record-surfaces.md](specs/retire-shared-record-surfaces.md); issue [#364](https://github.com/mudler/vllm.cpp/issues/364) | `READY` | `CLAIM-ENG-RECORD-CONFLICT-SURFACES` | | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate rejects CORRECT commits because of paragraph placement, and that is why `main` is red on `agent-record`. `check-commit-trailers.py` reads trailers via `git interpret-trailers --parse`, which treats ONLY the final paragraph as the block; GitHub appends `Co-authored-by:` as a SEPARATE trailing paragraph on a squash merge, so a complete correct block becomes invisible and the gate reports it missing. MEASURED: piping `git show -s --format=%B dbd0d51c` into `git interpret-trailers --parse` prints nothing but the co-author line, and 13 of the last 30 commits on `main` fail the check -- unnoticed only because those runs were cancelled (#274), which HID the defect rather than causing it. FIX: fuse consecutive trailing TRAILER-SHAPED paragraphs before parsing. Nothing is relaxed -- the block must still exist, the marker must still sit above it, each declaration must still appear exactly once, and an AI co-author is still forbidden; the block is merely FOUND where the merge tool left it. A prose paragraph still terminates it. REJECTED IN FLIGHT and recorded because it is the more instructive half: a first attempt also collapsed identical duplicate trailers to fix the multi-commit-squash shape, which relaxes the uniqueness rule an existing test already pins. Rewriting that assertion to suit the change is what AGENTS.md forbids, and the distinction is real -- a doubled block is genuinely malformed and fixable at source, whereas the co-author case is a correct commit defeated by the parser. Reverted in full. SCOPE LIMIT, stated rather than implied: this fixes ONE of five observed shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had been hiding and now correctly fails; `87308dea` (GitHub's `---------` separator), `b8293c88` (squash doubled the block) and `b580452d` (merge button, no trailers) stay red by design. Closing those is a merge-method change, not a checker change | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:60` (`join_trailing_trailer_paragraphs`, `_is_trailer_paragraph`, and the fused `parsed_trailers`) | `tests/scripts/test_check_commit_trailers.py:1` 21 cases -- the RED-BEFORE appended-co-author case plus four GUARDS that keep the fusion bounded (doubled block still fails, contradictory declarations still fail, a no-trailer merge message still fails, prose after the block still fails), all four green before and after; closure [parity-ledger.md#L941](parity-ledger.md#L941) | [trailer-merge-artifacts.md](specs/trailer-merge-artifacts.md); issue [#406](https://github.com/mudler/vllm.cpp/issues/406) | `DONE` | `157080c8` | | `ENG-FORGE-COAUTHOR` | The forbidden-AI-trailer rule was catching ATTRIBUTION rather than an authorship claim, which is why bot-opened PRs red `main` on merge. GitHub composes the squash message itself and appends the account that opened the PR — `Co-authored-by: localai-org-maint-bot <...@users.noreply.github.com>` — and most PRs here are opened by a bot, so nearly every squash trips the AI-identity check. Real instance `f64f2b71`, invisible until #406 repaired the parse, which is why it reads as a new failure and is not one. The rule exists so an AI cannot claim it WROTE the code, and that stays; GitHub is recording who pressed the button, and the AI-involvement claim is already carried separately by `AI-Assisted` and `Assisted-by` in the same block. FIX: accept a `Co-authored-by` at a GitHub account noreply address even when the name matches an AI identity token, keyed on the FORGE'S OWN DOMAIN rather than the name so the exemption cannot be borrowed. A hand-written `Co-authored-by: Claude ` still fails; `Signed-off-by` is excluded from the exemption entirely, because a sign-off is a legal assertion about provenance rather than attribution. `AGENTS.md` records the same distinction in the same change so prose and checker cannot drift | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:38` (`FORGE_ACCOUNT_EMAIL` and the forbidden-trailer skip) | `tests/scripts/test_check_commit_trailers.py:1` 25 cases -- the RED-BEFORE forge-bot case plus THREE guards that matter more than the relaxation because this LOOSENS a rule: a hand-written AI co-author still fails, `Signed-off-by` at the same noreply address still fails, and a human co-author still passes; all three green before and after. Real commit `f64f2b71` re-verified per commit | [forge-coauthor-attribution.md](specs/forge-coauthor-attribution.md); issue [#418](https://github.com/mudler/vllm.cpp/issues/418) | `ACTIVE` | `CLAIM-ENG-FORGE-COAUTHOR` | diff --git a/tests/scripts/test_agent_record.py b/tests/scripts/test_agent_record.py index c3954bfde..72b213640 100644 --- a/tests/scripts/test_agent_record.py +++ b/tests/scripts/test_agent_record.py @@ -669,12 +669,21 @@ def test_glm5_next_row_is_inside_the_model_ratchet(self) -> None: every other check stays green. Only an assertion that names the row goes red. - `READY` is pinned deliberately and is the weaker half of the evidence, + The state is pinned deliberately and is the weaker half of the evidence, stated rather than implied, for the same reason the qwen4-exp test gives: - the row is `READY` because its spec is committed and no product code has - landed, and pinning it here means a future refactor of the - structured-spec or claim-ownership rules cannot silently take this pin - with it. + pinning it here means a future refactor of the structured-spec or + claim-ownership rules cannot silently take this pin with it. + + It was `READY` when this test was written, on the stated premise that the + spec was committed and no product code had landed. W7a (#2011) landed + product code -- `scripts/convert-glm5-next-gguf.py`, the first thing on + this row that is not a record -- so the premise expired and the pin moves + with it to `ACTIVE`, in the same change that moves the matrix row. The + assertion is NOT weakened: it still names one exact state, and a pin that + followed the row automatically would assert nothing at all. What it stops + catching is only the one transition it was updated for; it still goes red + on a rename, on a second glm5_next row, and on any later state change + made without touching this file. """ errors: list[str] = [] rows, _ = agent_record.check_matrices(errors) @@ -684,7 +693,7 @@ def test_glm5_next_row_is_inside_the_model_ratchet(self) -> None: found = [row for row in rows if row.item_id == item_id] self.assertEqual(len(found), 1, item_id) self.assertEqual(found[0].path.name, "model-matrix.md", item_id) - self.assertEqual(found[0].field("state").strip().strip("`"), "READY", item_id) + self.assertEqual(found[0].field("state").strip().strip("`"), "ACTIVE", item_id) # One row, not three: neither the text-only arm nor the MTP head has a # sibling row, and adding one to mirror the upstream PR is the mistake.