diff --git a/.agents/claims/CLAIM-GLM53-FLASH-W7A.md b/.agents/claims/CLAIM-GLM53-FLASH-W7A.md new file mode 100644 index 000000000..5c14ddfac --- /dev/null +++ b/.agents/claims/CLAIM-GLM53-FLASH-W7A.md @@ -0,0 +1,5 @@ +# CLAIM-GLM53-FLASH-W7A + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-GLM53-FLASH-W7A` | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` (`ACTIVE`, advanced from `READY` by this claim: the first product code on the row) | fresh implementer, helper role per `scripts/agent-role.py`, working from the committed spec `.agents/specs/glm5-next-flash.md` §W7a | isolated worktree `/home/mudler/_git/vllm.cpp-glm53w7a`; CPU only, no `rc` lease taken and no GPU touched | `row/MODEL-MM-GLM53-FLASH-W7A`, base `row/MODEL-MM-GLM53-FLASH` at `78d2c8a61` (the spec commit, PR [#2001](https://github.com/mudler/vllm.cpp/pull/2001), not yet on `main`), issue [#2011](https://github.com/mudler/vllm.cpp/issues/2011) under campaign issue [#1998](https://github.com/mudler/vllm.cpp/issues/1998) | Owns ONLY the safetensors→GGUF converter and its synthetic-fixture gate: `scripts/convert-glm5-next-gguf.py`, `tests/scripts/test_convert_glm5_next_gguf.py`, the frozen llama.cpp `b10451` k-quant golden, the `docs/USAGE.md` weights-provenance entry, and the spec/record edits those invalidate. EXCLUDES artifact production (O7 — needs the 300–600 GiB checkpoint, developer authority for the download, and a box), the loader-side `glm5next` registration and `general.architecture` dispatch (W1, O9 — the emitted file is not loadable until then), every model-side primitive (W1–W6), the Q3_K/Q4_K/Q5_K encoders (O8), any i-quant arm (R4/O5), and any pin advance: nothing in this claim moves `.agents/upstream-sync.md` or `.agents/oracles/*` | `ACTIVE` | 2026-08-26 — converter landed, gated on synthetic fixtures with the k-quant encoders byte-identical to the pinned llama.cpp reference; artifact owed as O7 | diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 462f4ebe1..fdd368837 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -227,7 +227,7 @@ claims it. | `ENG-RELEASE-WINDOWS` | Native Windows x86_64 pre-alpha release extension: one adaptive MSVC/UCRT CPU bundle with AVX2 executed in CI and one Vulkan preview bundle, both deterministic ZIPs and authenticated by the existing release handoff | T0 | vLLM has no Windows release path; runtime behavior remains pinned to vLLM `555967922`. Platform substrate reference: llama.cpp `src/llama-mmap.cpp:520-590` @ `237ad9b961f009ae19ac29dbce4cd0c1251f94b3`; Win32 API is the OS authority | W14 Win32 portability/MSVC CPU, W15 deterministic ZIP/PE packaging + Vulkan, and W16 ten-tuple prerelease workflow/version/docs implemented for one PR | Linux portability/release mutation gates are local evidence only. Native `windows-2022` MSVC `/W4 /WX`, extracted runtime/ISA smokes, merged-SHA ten-tuple dry run, `v0.0.3-pre.1` publication, attestations, and exact 32-asset audit remain pending; no Windows ZIP exists yet | [windows-binary-release.md](specs/windows-binary-release.md); [#117](https://github.com/mudler/vllm.cpp/issues/117) | `ACTIVE` | `CLAIM-ENG-RELEASE-WINDOWS` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | `docker/Dockerfile` (cpu/vulkan/cuda targets calling the release scripts); `docker/healthcheck.sh`; `release/container-matrix.json`; `scripts/check-container-matrix.py`; `scripts/check-container-workflow.py`; `scripts/validate-container-image.py`; `scripts/container_tags.py`; `.github/workflows/containers.yml`; SIGTERM handler `src/vllm/entrypoints/openai/server_main.cpp` (`SignalShutdown`, all three `listen()` sites); the pre-existing `docker/Dockerfile.arm64` is an unrelated CPU bench cross-check | issues `#170`, `#312`, `#394`; `tests/scripts/test_check_container_matrix.py` 31/31; `test_check_container_workflow.py` 29/29; `test_check_cuda_fat_gencode.py` 7+4 subtests. **GB10 2026-08-11 (`promaxgb10-4ad8`, `sm_121a`, CUDA 13.3): arm64 cuda image 1.71 GB, 673/673 objects, ten-SM gencode audit PASS, and a REAL GPU boot -- `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, `--gpus all`, host driver 580.159.03 injected.** cpu amd64 783 MB gated locally; cpu+vulkan amd64 green on hosted CI **arm64 cuda lane RUNTIME-VERIFIED on GB10 2026-08-11** -- the first accelerator-hardware evidence for any lane. Four defects were removed to get there, each found by building rather than reading: the CUDA 12.9 base could not compile `sm_110`, the BuildKit cache mount outlived its toolchain (both #366), Marlin gencode had drifted from the feature table and failed the audit on 14 correctly-compiled TUs (#394, blocking BOTH cuda tuples project-wide), and the validator could only ever produce build evidence because its boot smoke never passed `--gpus`. **NOT established: nothing is published to GHCR; amd64 cuda is unbuilt; the published arm64 image is SBSA (`targets/sbsa-linux`), so Tegra -- Thor `sm_110`, Orin `sm_87` -- is untested and NOT covered** **ORIN (Tegra) 2026-08-11: the SBSA image RUNS on Jetson AGX Orin `sm_87` (L4T R36.4.3, Docker 27.5.1) -- Qwen3-0.6B (rev `c1899de2`) loads and GENERATES via `/v1/completions`, tegrastats GR3D 95-97% during decode vs 14-15% idle.** Tegra needs `--runtime nvidia --gpus all`: `--gpus` alone is refused by the hook and `--runtime` alone mounts no driver | [container-images.md](specs/container-images.md); issues [#170](https://github.com/mudler/vllm.cpp/issues/170), [#312](https://github.com/mudler/vllm.cpp/issues/312), [#394](https://github.com/mudler/vllm.cpp/issues/394) | `ACTIVE` | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | | `ENG-DOCS-SITE` | Publish the 11 `docs/*.md` as a browsable GitHub Pages site at `https://mudler.github.io/vllm.cpp/` WITHOUT a second copy of the prose. A Hugo site at `website/` mounts `../docs` READ-ONLY and derives everything else from what is already in the files: each page title from the file's first `# H1`, the sidebar order from `website/data/nav.yaml`, and links through a Goldmark render hook (internal `.md` → site URL; the 139 `../.agents/**` and `../AGENTS.md` escapes → GitHub blob URLs, since the protocol tree is deliberately NOT published). **No file under `docs/` is modified, moved, renamed, or given front matter**, so `check-doc-checkpoint.py` and every protocol path reference keep working and there is no second surface that can drift — the whole point of the row. Custom lean layouts, NO theme and NO submodule: off-the-shelf docs themes read titles, weights and menus out of front matter this design deliberately does not have, so each would need its title partial, menu and link hook overridden anyway, and hugo-book additionally floors at Hugo 0.158 against the 0.146.3 pin CI and the local toolchain share. Hard prerequisite inside the repo: `classify_path` in `scripts/check-pr-size.py` FAILS CLOSED on `website/**` (verified: raises `ValueError: unclassified repository path`), so the classifier must learn the path or the PR cannot pass the project's own size gate. Hard prerequisite outside it: GitHub Pages must be enabled with the source set to GitHub Actions — the workflow is inert otherwise. A marketing landing page is explicitly OUT of scope (`README.md` stays the front door), as is any restructuring of `docs/`; the custom domain is parked behind the pending vLLM trademark question | T1 | NO vLLM analogue — upstream's docs are a separate mkdocs site and nothing in this row mirrors upstream *behavior*, so it carries no parity obligation. The STRUCTURAL reference is LocalAI's `.github/workflows/gh-pages.yml` (two Hugo sites merged into one Pages artifact), reduced to the docs half | read-only mount `website/hugo.toml:29`; title-from-H1 `website/layouts/partials/title.html:10`; link rewriting `website/layouts/_default/_markup/render-link.html:27`; guard `scripts/check-site.py:70`; deploy `.github/workflows/gh-pages.yml` | `tests/scripts/test_check_site.py:51,56,66,80,89,97` (6 mutation cases: clean tree, H1 stripped, doc absent from nav, nav entry with no file, duplicated entry, missing nav file); build evidence 14 pages with `docs/bench-evidence` + `docs/superpowers` absent from `public/` and no `href` ending in `.md`; 48 protocol links rewritten in `docs/status/`. NO published page is claimed: GitHub Pages is not yet enabled on the repository, which is the recorded stop condition holding this row at `GATING` | [gh-pages-docs-site.md](specs/gh-pages-docs-site.md); issue [#224](https://github.com/mudler/vllm.cpp/issues/224) | `READY` | `CLAIM-ENG-DOCS-SITE` | -| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1559` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1627`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | +| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1568` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1636`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | | `ENG-RECORD-CONFLICT-SURFACES` | Retire the shared record surfaces that make concurrent PRs conflict by construction. MEASURED at `origin/main` `d928e2c3` with `git merge-tree --write-tree` over every open PR: **16 of 29 conflict (55%), and 13 of the 16 conflict in bookkeeping files ONLY**, with no product code involved — `.agents/coordination.md` in 8, `.agents/NOW.md` in 5, `.agents/roadmap_v1.md` in 4, `scripts/check-public-doc-tables.py` in 4, `docs/STATUS.md` in 4, and any `src/`/`tests/` path in just 3. Three defects, each of which GUARANTEES rather than risks a collision. (1) `.agents/NOW.md` is a fixed-size shared buffer at EXACTLY 6000/6000 chars (`check-now-current.py:31`), so adding a row requires evicting another and every PR is a read-modify-write of one global — and the conflict is the LUCKY outcome, since a clean three-way merge would apply both evictions and both additions, silently dropping live rows and blowing the very budget the checker defends. (2) `STATUS_RATCHET = {"chars": 243245}` (`check-public-doc-tables.py:557`) is a hardcoded byte count of a DIFFERENT file that may only fall, so a PR owing `docs/STATUS.md` one lifecycle line must delete unrelated prose from another row to pay for it and edit the checker too; the checker's own comment at `:331` already records the failure (*"a ratchet pinned to the byte turns every concurrently merged row's one-line status edit into a spurious failure"*) and answered it with slack instead of removing the coupling. (3) `.agents/coordination.md`'s active-claims table is insert-at-one-anchor: the six ROCm GDN PRs (#334 #336 #341 #343 #345 #348) are ONE author's sequential stack that conflicts on nothing else, each appending a ~1,500-char row — the PR description, transcribed into a file every other claim also writes. It also contradicts the protocol it serves: `AGENTS.md` holds that *"History is git"* and *"There is no state log"*, yet both claims tables ARE state logs duplicating `gh pr list`, `row/` branch names and issue state; the argument that refuses a waiver registry applies unchanged to a claims registry. Precedent twice over — `policy.csv` retired in `0f3e44ee`, per-class line budgets retired 2026-08-10 because the gate fired on ordinary work. The exonerated surfaces share ONE property, one writer per file: `.agents/specs/.md` (one file per row, **zero conflicts** in the sample), the `*-matrix.md` inventories, and the append-only `.agents/benchmark-record.md`. SCOPE: remove `STATUS_RATCHET` and the doc-gating global counters while KEEPING the per-cell/per-paragraph caps (local, so they couple nothing); remove the active-claims table and derive claims from open PRs and branch names; drop `NOW.md`'s byte budget; order the roadmap's keyed tables by ID so distinct keys stop colliding at one anchor; and record the invariant — **no surface that every PR must write** — in `AGENTS.md`. No product source, kernel or gate semantic moves | T0 | NO vLLM analogue — this is local protocol machinery, so the mirror rule does not apply and no upstream `file:line` exists to port from. Governed instead by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | - | - (spec-before-code: the red-before suites are named in the spec's Tests section — `tests/scripts/test_check_public_doc_tables.py`, `tests/scripts/test_check_now_current.py`, a mutation case per removed rule proving the obligation survives in the retained caps and `check-doc-checkpoint.py`, and a `git merge-tree` merge-shape regression that must be RED before the `NOW.md`/roadmap work and GREEN after) | [retire-shared-record-surfaces.md](specs/retire-shared-record-surfaces.md); issue [#364](https://github.com/mudler/vllm.cpp/issues/364) | `READY` | `CLAIM-ENG-RECORD-CONFLICT-SURFACES` | | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate rejects CORRECT commits because of paragraph placement, and that is why `main` is red on `agent-record`. `check-commit-trailers.py` reads trailers via `git interpret-trailers --parse`, which treats ONLY the final paragraph as the block; GitHub appends `Co-authored-by:` as a SEPARATE trailing paragraph on a squash merge, so a complete correct block becomes invisible and the gate reports it missing. MEASURED: piping `git show -s --format=%B dbd0d51c` into `git interpret-trailers --parse` prints nothing but the co-author line, and 13 of the last 30 commits on `main` fail the check -- unnoticed only because those runs were cancelled (#274), which HID the defect rather than causing it. FIX: fuse consecutive trailing TRAILER-SHAPED paragraphs before parsing. Nothing is relaxed -- the block must still exist, the marker must still sit above it, each declaration must still appear exactly once, and an AI co-author is still forbidden; the block is merely FOUND where the merge tool left it. A prose paragraph still terminates it. REJECTED IN FLIGHT and recorded because it is the more instructive half: a first attempt also collapsed identical duplicate trailers to fix the multi-commit-squash shape, which relaxes the uniqueness rule an existing test already pins. Rewriting that assertion to suit the change is what AGENTS.md forbids, and the distinction is real -- a doubled block is genuinely malformed and fixable at source, whereas the co-author case is a correct commit defeated by the parser. Reverted in full. SCOPE LIMIT, stated rather than implied: this fixes ONE of five observed shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had been hiding and now correctly fails; `87308dea` (GitHub's `---------` separator), `b8293c88` (squash doubled the block) and `b580452d` (merge button, no trailers) stay red by design. Closing those is a merge-method change, not a checker change | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:60` (`join_trailing_trailer_paragraphs`, `_is_trailer_paragraph`, and the fused `parsed_trailers`) | `tests/scripts/test_check_commit_trailers.py:1` 21 cases -- the RED-BEFORE appended-co-author case plus four GUARDS that keep the fusion bounded (doubled block still fails, contradictory declarations still fail, a no-trailer merge message still fails, prose after the block still fails), all four green before and after; closure [parity-ledger.md#L941](parity-ledger.md#L941) | [trailer-merge-artifacts.md](specs/trailer-merge-artifacts.md); issue [#406](https://github.com/mudler/vllm.cpp/issues/406) | `DONE` | `157080c8` | | `ENG-FORGE-COAUTHOR` | The forbidden-AI-trailer rule was catching ATTRIBUTION rather than an authorship claim, which is why bot-opened PRs red `main` on merge. GitHub composes the squash message itself and appends the account that opened the PR — `Co-authored-by: localai-org-maint-bot <...@users.noreply.github.com>` — and most PRs here are opened by a bot, so nearly every squash trips the AI-identity check. Real instance `f64f2b71`, invisible until #406 repaired the parse, which is why it reads as a new failure and is not one. The rule exists so an AI cannot claim it WROTE the code, and that stays; GitHub is recording who pressed the button, and the AI-involvement claim is already carried separately by `AI-Assisted` and `Assisted-by` in the same block. FIX: accept a `Co-authored-by` at a GitHub account noreply address even when the name matches an AI identity token, keyed on the FORGE'S OWN DOMAIN rather than the name so the exemption cannot be borrowed. A hand-written `Co-authored-by: Claude ` still fails; `Signed-off-by` is excluded from the exemption entirely, because a sign-off is a legal assertion about provenance rather than attribution. `AGENTS.md` records the same distinction in the same change so prose and checker cannot drift | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:38` (`FORGE_ACCOUNT_EMAIL` and the forbidden-trailer skip) | `tests/scripts/test_check_commit_trailers.py:1` 25 cases -- the RED-BEFORE forge-bot case plus THREE guards that matter more than the relaxation because this LOOSENS a rule: a hand-written AI co-author still fails, `Signed-off-by` at the same noreply address still fails, and a human co-author still passes; all three green before and after. Real commit `f64f2b71` re-verified per commit | [forge-coauthor-attribution.md](specs/forge-coauthor-attribution.md); issue [#418](https://github.com/mudler/vllm.cpp/issues/418) | `ACTIVE` | `CLAIM-ENG-FORGE-COAUTHOR` | diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 289dc59b9..adaa0c8c9 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -779,3 +779,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1979](https://github.com/mudler/vllm.cpp/issues/1979) | `ENG-RELEASE-WINDOWS` | **`windows-msvc-vulkan` is red on every open PR: `test_openai_api_server.exe` fail-fasts with `0xC0000409`, and the job has no `main` baseline to bisect against.** Found while gating [#1973](https://github.com/mudler/vllm.cpp/issues/1973) (PR [#1977](https://github.com/mudler/vllm.cpp/pull/1977)); PRE-EXISTING and not that row's defect. `scripts/build-windows-release.ps1:31` throws on `build-pr-windows-vulkan\tests\Release\test_openai_api_server.exe exited with status -1073740791`, which is `0xC0000409` `STATUS_STACK_BUFFER_OVERRUN`, the MSVC fail-fast code. It is a CRASH and not a doctest failure: the log carries no `[doctest]` summary and no assertion line for that binary, so nothing reports which case was running; the last lines before it are ordinary chat-completions request logs. MEASURED 2026-08-26 by reading four job logs directly -- [#1977](https://github.com/mudler/vllm.cpp/pull/1977) job 98175568298, [#1975](https://github.com/mudler/vllm.cpp/pull/1975) job 98169254199, [#1969](https://github.com/mudler/vllm.cpp/pull/1969) job 98152969359 and [#1967](https://github.com/mudler/vllm.cpp/pull/1967) job 98144800167 -- all four stopping on the SAME binary. Those four pull requests touch the KV-cache interface, the runner, the LTX-2.5 lane and the record surfaces, and the intersection of their diffs contains no file under `src/vllm/entrypoints/`, so the common factor is the job and not the change. **The second half of the defect is that `windows-msvc-cpu` and `windows-msvc-vulkan` are PR-ONLY jobs that never run on `main`**, so there is no green reference commit to bisect against and no signal for when this started -- which makes a repository-wide red something every author must independently re-attribute, and this issue exists so that stops happening a fifth time. NOT ESTABLISHED and stated rather than implied: no Windows host was available, so nothing was reproduced locally, the crashing case is not identified, the first bad commit is not identified and no cause is proposed; `httplib`'s `stop()` being a no-op on a never-listened server so `SO_REUSEPORT` splits the port (`.agents/environment.md`) is named as an ADJACENT shape to check first, NOT as an attribution. NOT FIXED IN FLOW: it needs a Windows host to reproduce, it sits in a server path unrelated to the KV-cache row that found it, and identifying the crashing case is its own investigation | bug | | [#1998](https://github.com/mudler/vllm.cpp/issues/1998) | `MODEL-MM-GLM53-FLASH` | **`zai-org/GLM-5.3-Flash` declares `Glm5NextForConditionalGeneration` / `glm5_next`, an architecture vLLM implements at NO revision, whose only admissible oracle is transformers and which NOTHING on this fleet can execute.** Read live 2026-08-26: `git grep "Glm5\|glm5_next"` returns ZERO hits at the parity pin `555967922` AND at vLLM `origin/main` = `c71f6f8a81` (2026-08-26 08:42 -0700), whose `vllm/models/` package holds `common, deepseek_v32, deepseek_v4, dots3_note, inkling, kimi_k3, minimax_m3` and no `glm5next`. That is ABSENCE from vLLM `main`, not staleness in our pin, so a pin advance — 348 commits of reconciliation — reaches nothing here. [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) "[Model] add GLM-5.3-Flash support" (opened 2026-08-26T14:12:00Z, head `933876c388fb`, 85 files, +12,511/-540) would register `Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` and `Glm5NextMTPModel`, and is **OPEN, unmerged, `mergeable: false`** — an unmerged PR is not a revision and is INADMISSIBLE as an oracle, the same rule this index already applied to SGLang #36497 on the qwen4-exp row; it is cited only as evidence of upstream shape. SGLang implements nothing either (PR #36507 open; code search 0) and the trap there is that its **cookbook** PRs #36440 and #36513 MERGED at 14:00:16Z and 14:39:29Z — documentation landed ahead of the code, so the model card links a recipe SGLang `main` cannot run. llama.cpp implements nothing (code search 0; PR #27752 open; our oracle pinned at `b10451`), and vllm-omni implements nothing. **The sole admissible reference is transformers**, implementing commit `eb4d9e2a64` (2026-08-26T14:26:40Z, transformers#48342), and the first RELEASE carrying it is **`v5.16.1`** — bounded rather than assumed by fetching `models/glm5_next/modeling_glm5_next.py` at each tag: HTTP **200** at `v5.16.1`, **404** at `v5.16.0`, **404** at `v5.15.1`. Our transformers pin is 5.14.1, so this row needs a lane-scoped second pin expiring when vLLM registers `glm5_next`; note it is `5.16.1` where the `MODEL-MM-QWEN4-EXP` lane is `5.16.0`, one day apart because `Qwen4Exp` merged before the 5.16.0 cut and `Glm5Next` after it — two lanes, two releases, which is what a lane pin is for. **Not the blocked GLM-5 row:** `MODEL-TEXT-deepseek-v2-glm-moe-dsa-for-causal-lm` (`GlmMoeDsaForCausalLM`, `BLOCKED`, `CLAIM-GLM-DSA-LATEST-DEEPSEEK`) is DeepSeek-V3.2 verbatim at 753.9B; `glm5_next` is a different `model_type`, a different parameter count, a different attention topology and has a vision tower. **The architecture:** 45 layers, hidden 4096, vocab 154880, 1,048,576 context, `layer_types` = 34 `linear_attention` (KDA) + 11 `deepseek_sparse_attention` at 3,7,…,43; MoE 288 routed + 1 shared, top-8, `moe_intermediate_size` 2048, sigmoid/`noaux_tc`, `n_group=topk_group=1` (which makes the group stage a NO-OP), `routed_scaling_factor` 2.5, clamped SwiGLU `swiglu_limit` 10.0 in FIVE places; mHC `hc_mult` 4 / `hc_sinkhorn_iters` 20 / `hc_eps` 1e-6; vision 24L/1024/patch **14**/merge 2/temporal 2/out 4096. **The text stack has NO rotary anywhere.** `qk_rope_head_dim: 0`, `mla_use_nope: true`, no `rope_theta`, no `rope_scaling`; the reference *deletes* the inherited `rope_parameters` field and passes `position_embeddings=None` to every layer, and **`indexer_rope_interleave: true` is a VESTIGIAL flag** the indexer override ignores — implementing it because the config names one is implementing a bug. Position comes only from the KDA decay and short convs. **HEAVY REUSE — and the two obvious reuses are WRONG in one place each, in ways a token gate cannot see.** (1) The KDA forget gate takes the OTHER branch: `gate_lower_bound: -5.0` is not None, so it computes `-5.0 * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias))`, whereas our `src/vllm/model_executor/models/kimi_kda.cpp:60` implements Kimi-Linear's `-exp(A_log) * softplus(g + dt_bias)`. Different functions of the same inputs, both smooth, both negative, both fluent — and the sign of `decay_rate` differs too, so getting it wrong flips decay to growth. (2) The mHC head collapse is `hidden_streams.mean(dim=2)`, an UNWEIGHTED mean whose own docstring says "Unlike DeepSeek-V4", where our `src/vllm/model_executor/models/deepseek_v4_mhc.cpp:168` `HcHeadCollapse` is the V4 weighted-gated collapse; the checkpoint carries **no `hc_head.*` tensors at all**, which settles it. **(3) The k-pool indexer is unobservable at or below 2048 tokens:** with `index_topk: 2048` a shorter context selects every candidate, the selection is the identity, the pooling is invisible and any short-prompt gate passes over a broken indexer — the same shape as the trap recorded on the qwen4-exp row. The pooling is **learned, per channel, not a mean**: 128 independent 4-way softmaxes over a learned `[4,128]` intra-pool position embedding plus a cached `[128,4096]` gate score, pools aligned to the first VALID token (so the grid is batch-dependent), a pool must be COMPLETE to be a candidate and is visible iff its LAST member is, `select_k = min(index_topk/index_kpool, P) = min(512, P)`, expanded back to raw token indices and widened to `index_topk + kpool - 1` = **2051** by an unscored ragged tail. Its cache is 257 floats/token/layer, not the parent's 128. **Structural blocker:** `MlaBlockDims::Validate` REFUSES `qk_rope_head_dim == 0` (`src/vllm/model_executor/layers/attention/mla_attention.cpp:90-93` requires every dim > 0, `:95-99` requires it even); Kimi-Linear is NOT the same case — it sets `mla_use_nope` while keeping `qk_rope_head_dim: 64`, so the rope slice still occupies the cache row (`include/vllm/model_executor/models/kimi_linear.h:86,:88`). NoPE does buy one thing: MLA weight absorption becomes trivially valid, since there is no rope slice to keep out of it. **NOTHING FITS, and no GGUF exists.** Measured live by HTTP RANGE over all 62 safetensors headers, payload never fetched: 76,108 tensors, `F8_E4M3` 314,396,639,232 + `BF16` 6,926,096,640 elements = **321,322,735,872 real parameters** plus 19,484,766 F32 block scales; routed experts are **311.65B = 97.0%**, so any GGUF arm's mixed bpw is the experts' bpw to within a percent. On disk against ~119.63 GiB on GB10: `zai-org/GLM-5.3-Flash` FP8 e4m3 block-128x128 **305.78 GiB (2.56x over)**, `zai-org/GLM-5.3-Flash-BF16` **598.53 GiB (5.00x)**, `unsloth/GLM-5.3-Flash-FP8` 305.82 GiB, `LibertAIDAI/GLM-5.3-Flash-NVFP4` **181.32 GiB (1.52x)**. **All four repos named `*-GGUF` contain ZERO `.gguf` files** (`unsloth`, `AtomicChat`, `aj9o9`, `vcruz305` — READMEs and images); a repository name is not an artifact. **`gateable = no` on MEMORY, and the consequence is the scoping result: no oracle can execute this model on any device this project reaches, so NO end-to-end token gate exists or can exist here** — recorded as visible debt, not waived. What replaces it is a tiny-shape CPU reference oracle instantiated from the pinned transformers, which is a real oracle for the NUMERICS and is explicitly NOT oracle gateability for the model. Developer-set W7 target is a low GGUF quant that fits the DGX; arithmetic (not measurement) with experts at the named type and the other 3% at Q6_K, bpw taken from our own reader's block traits at `src/vllm/model_executor/model_loader/gguf_reader.cpp:200`: Q8_0 318.0 GiB, Q6_K 245.5, Q4_K 170.6, IQ3_XXS 118.1 (~1.5 GiB left — not a margin), **Q2_K 102.6 (~17 GiB left)**, IQ2_S 100.3, IQ2_XXS 82.3, IQ1_S 64.0; KV is small by design at ~11.7 KiB/token (1.5 GiB at 128K, ~11.4 GiB at the full 1M) though #1963/#1966 mean that arithmetic is re-derived against the runner, not trusted. **Q2_K is the recommended first arm and the reason is producibility, not quality: i-quants need an importance matrix, an importance matrix needs a forward pass, and a forward pass needs 181 GiB — the dependency is CIRCULAR on this fleet.** K-quants need none. And since llama.cpp has no `glm5_next`, the converter is ours to author, with no reference GGUF to diff against and no llama.cpp floor for the arm. Layer 45 is the MTP block — 888 tensors, 7.43B params, 2.31% of the model, DSA/MLA rather than KDA and carrying no `hc_*` tensors — present in the checkpoint and discarded by the reference at `modular_glm5_next.py:1235`; owed, not implemented, and skipping it in the converter is ~2.3 GiB of headroom. Live seam contention to rebase onto rather than fork: [#1971](https://github.com/mudler/vllm.cpp/pull/1971) (DSA geometry) and [#1977](https://github.com/mudler/vllm.cpp/pull/1977) (DSv4 KV multicache). Spec: [`specs/glm5-next-flash.md`](specs/glm5-next-flash.md), 8 waves, W0-W2 and W4 CPU-gateable and W3/W5-W8 needing a GPU. No product code lands under the spec pull request | feature | | [#2061](https://github.com/mudler/vllm.cpp/issues/2061) | `MODEL-TEXT-laguna-laguna-for-causal-lm` | Laguna's grouped MoE issues two `LqGemmGrouped` calls over the same activation, so it quantizes to Q8_K twice where `vt::MoeGateUpSwiGLUGrouped` quantizes once — W11 measured `QuantizeQ8KKernel` at 12.4% of decode GPU. Also a shared-seam obligation, since AGENTS.md routes mergeable MLP projections through the fused group. Bounded by whether a DYNAMIC UD quant gives both expert towers the same block-quant dtype, which W1 measures before any code | perf | +| [#2011](https://github.com/mudler/vllm.cpp/issues/2011) | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | **GLM-5.3-Flash has no GGUF, no upstream tool can make one, and upstream Python cannot even quantize: author the `glm5_next` converter and gate its k-quant encoders BYTE-FOR-BYTE against llama.cpp `b10451`.** W7a of [#1998](https://github.com/mudler/vllm.cpp/issues/1998); spec [`specs/glm5-next-flash.md`](specs/glm5-next-flash.md) §W7a. Every GPU gate on this row is blocked behind an artifact that does not exist: measured live 2026-08-26 against ~119.63 GiB usable on `dgx:gpu0`, the published arms are FP8 **305.78 GiB**, BF16 **598.53 GiB** and `LibertAIDAI/...-NVFP4` **181.32 GiB**, and all four repositories named `*-GGUF` contain **ZERO `.gguf` files**. **D6 holds but was too strong, and the correction is what made this wave tractable.** llama.cpp implements no `glm5_next` — re-verified at `origin/master` `539f24529` (fetched 2026-08-26) and at our pin: the enumerators are `LLM_ARCH_GLM4`, `LLM_ARCH_GLM4_MOE`, `LLM_ARCH_GLM_DSA` (`src/llama-arch.h:86-88`) and `src/models/glm-dsa.cpp` is **GLM-5.2**, citing `zai-org/GLM-5.2/blob/main/config.json`, a different model. **But every GGUF convention this converter needs is already present AT `b10451`, so no pin advance was taken and none is owed:** `gguf-py/gguf/constants.py:262-264` carries `class KDA` with `{arch}.kda.head_dim` and **`{arch}.kda.gate_lower_bound`** (`KDA.SAFE_GATE` is `master`-only and GLM-5.3-Flash declares no `safe_gate`, so nothing reaches for it); the KDA tensor spellings `ssm_conv1d_q/k/v`, `ssm_f_a/f_b`, `ssm_g_a/g_b`, `ssm_beta`, `ssm_a`, `ssm_dt`, `ssm_norm` are at `src/llama-arch.cpp:465-479`; `gguf-py/gguf/tensor_mapping.py:896-933` maps them from Kimi-Linear's HF module paths, which are **GLM-5.3-Flash's paths verbatim**; and the indexer names, including `indexer_compressor_ape`/`indexer_compressor_gate` for the k-pool stage, are at `:626-636`. **The one thing upstream cannot supply at any revision is an ENCODER:** `gguf.quants.Q2_K` implements `dequantize_blocks` and **no** `quantize_blocks`, so Q2_K/Q6_K/Q8_0 were ported from `ggml/src/ggml-quants.c` @ `b10451` (`:891`, `:1869`, `:276`, over `make_qkx2_quants:799`, `make_qx_quants:628`, `nearest_int:621`) and gated **byte-identical** against a frozen golden captured from that reference compiled `-ffp-contract=off`. Two traps changed bytes and are recorded in the source rather than left to be re-found: `nearest_int` is the `+12582912.0` add-and-mask trick and rounds half to EVEN, not `round`; and C `roundf` in `quantize_row_q8_0_ref` rounds half AWAY FROM ZERO where `np.rint` rounds half to even, so `np.rint` mis-encodes every exact `.5` — caught by a crafted `[0.5, -0.5, 1.5, ...]` case, `[1,-1,2,-2,3,-3]` expected against `[0,0,2,-2,2,-2]` observed. **Exact arithmetic, from the converter's OWN type resolver run over the real topology** rather than bpw times a parameter count: 1719 output tensors carrying **313,890,512,702** parameters — 321.32B less the 7.43B MTP block, which independently confirms the skip is the 2.31% the spec measured. Q2_K (experts Q2_K, everything else Q6_K) **100.35 GiB** at 2.746 mixed bpw, Q6_K 239.89, Q8_0 310.67, BF16 584.67. Against 119.63 GiB at 128K context and one sequence: KV **1.43 GiB** (11 MLA layers x `kv_lora_rank` 512 x 2 B plus an 11 x 64 B indexer side cache = 11,968 B/token), KDA recurrent state **0.14 GiB** (64 x 128 x 128 x 4 B x 34) plus conv states, leaving **~17.7 GiB** — arithmetic, not measurement, and [#1963](https://github.com/mudler/vllm.cpp/issues/1963)/[#1966](https://github.com/mudler/vllm.cpp/issues/1966) mean W5 re-derives it from the runner. The Q2_K figure is **100.35 and not the spec's 102.6** because that table stated every arm including layer 45. The fallback ladder is `want -> Q8_0 -> F32` rather than `want -> F32`, because with an F32 fallback a FINER arm can come out LARGER than a coarser one and a size table must not have that property; on the real model the F32 sliver is 0.08 GiB (the depthwise conv kernels at `ne0=4`, the 4-D downsample and 5-D patch-embed). The tensor inventory is EXACT, not inferred: the real 8.4 MB `model.safetensors.index.json` (76,108 entries) and shard headers 2/32/62 were read by HTTP RANGE, payload never fetched, confirming the three separate `{q,k,v}_conv1d` convs, `hc_{attn,ffn}_{fn,base,scale}` flat on the layer with **no `hc_head.*` at any layer**, `indexer.k_norm.bias` present (settling LayerNorm-with-bias over RMSNorm), and `index_kpool_compress_{ape,gate}` on 12 layers. Refusals by name for every unimplemented arm: the i-quants because an imatrix needs a forward pass, a forward pass needs 181 GiB, and the dependency is **circular on this fleet** (O5); Q3_K/Q4_K/Q5_K because their encoders are unported and shipping an ungated encoder is worse than refusing (O8); `--keep-mtp` because nothing on the loader side reads an MTP tail (O2); a non-`glm5_next` config; and an FP8 tensor whose `weight_scale_inv` companion is missing, which would otherwise produce a loadable, wrong file. Gate: 50 assertions on a SYNTHETIC tiny-shape checkpoint with an independent in-test GGUF reader — no real checkpoint, no GPU, no C++ build. **NOT REACHED and disclosed:** `glm5next` has no `general.architecture` dispatch entry (`src/vllm/entrypoints/model_loader.cpp:1000`), so the file the converter writes is not loadable by this tree; W1 owns that wiring, tracked as **O9** under #1998. **Owed on close:** O7, artifact production — needs the 300-600 GiB checkpoint staged on local disk, explicit developer authority for the download, and a box with room for source and output at once, plus the sha256, recipe and peak RSS the spec's §Evidence requires | feature | diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 00ef7e5ca..91b2f35ee 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -77,11 +77,11 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): |---|---| | INVENTORIED | 324 | | PARTIAL | 22 | -| ACTIVE | 11 | +| ACTIVE | 12 | | SPIKE | 9 | | BLOCKED | 5 | | DONE | 3 | -| READY | 4 | +| READY | 3 | | GATING | 1 | | **Total** | **379** | @@ -90,7 +90,7 @@ Engaged architectures (the 55 non-`INVENTORIED` rows): | Support | Architecture | Family / example | Status | Row | |---|---|---|---|---| | 🚧 | `Qwen4ExpForConditionalGeneration` | Qwen3.8-Flash-Next (180B total / 6B activated, image-text-to-text) | **ALL SIX WAVES LANDED; STILL NOT REACHABLE.** New architecture vLLM does NOT implement at any revision; split oracle by developer direction 2026-08-26 (transformers for the algorithm, vLLM ops for the optimized path). Nothing published fits ~119 GB on GB10, so `gateable = no` and the quantized arms are load-bearing; no GGUF exists and llama.cpp has no `qwen4_exp` either. NO token, NO speed. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | -| 🚧 | `Glm5NextForConditionalGeneration` | GLM-5.3-Flash (321.32B total / ~18B active, natively multimodal; 34 KDA linear-attention + 11 DeepSeek-sparse MLA layers, mHC residual manifold, 288+1 expert MoE, 24-layer ViT) | **SPEC ONLY.** New architecture vLLM implements at NO revision -- absent from the pin `555967922` AND from `main` `c71f6f8a81`, so a pin advance does not reach it; vllm#53906 is OPEN and inadmissible. Sole admissible oracle is transformers, lane-pinned `v5.16.1` (the first release carrying `glm5_next`; `v5.16.0` is 404). `gateable = no` on MEMORY: the smallest published artifact is NVFP4 at 181.32 GiB against ~119.63 GiB on GB10, so no oracle can run this model on any device here and NO end-to-end token gate is reachable. Zero `.gguf` files exist in any of the four `*-GGUF` repos and llama.cpp has no `glm5_next`, so the converter is ours to author. Spec plans 8 waves; no product code has landed | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | +| 🚧 | `Glm5NextForConditionalGeneration` | GLM-5.3-Flash (321.32B total / ~18B active, natively multimodal; 34 KDA linear-attention + 11 DeepSeek-sparse MLA layers, mHC residual manifold, 288+1 expert MoE, 24-layer ViT) | **CONVERTER LANDED, MODEL NOT STARTED.** New architecture vLLM implements at NO revision -- absent from the pin `555967922` AND from `main` `c71f6f8a81`, so a pin advance does not reach it; vllm#53906 is OPEN and inadmissible. Sole admissible oracle is transformers, lane-pinned `v5.16.1` (the first release carrying `glm5_next`; `v5.16.0` is 404). `gateable = no` on MEMORY: the smallest published artifact is NVFP4 at 181.32 GiB against ~119.63 GiB on GB10, so no oracle can run this model on any device here and NO end-to-end token gate is reachable. Zero `.gguf` files exist in any of the four `*-GGUF` repos and llama.cpp has no `glm5_next`, so the converter was ours to author and W7a ([#2011](https://github.com/mudler/vllm.cpp/issues/2011)) authored it: `scripts/convert-glm5-next-gguf.py`, Q2_K arm 100.35 GiB against ~119.63 GiB, k-quant encoders byte-identical to the pinned llama.cpp `b10451`. The MODEL is still not started -- no `glm5next` registration, so the file it writes is not loadable here (O9), and no artifact has been produced (O7). Spec plans 8 waves; W7a is the only one landed | `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | | ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling scope CLOSED (#323): `60e71a0e` fixed the eager path; `DenseDecodeGraphForward` ran first and replayed against stale HOST ids, so it now declines while the mirror is live and falls back to the proven eager path. Async gate 7/7 across Qwen3-0.6B/4B + Llama/Mistral/InternLM2 | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` | | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` | | ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving. **SECOND CHECKPOINT TOKEN-GATED 2026-08-15 (`Qwen/Qwen3.8-27B`@`1d4bf0f2`, bf16, [#915](https://github.com/mudler/vllm.cpp/issues/915), [spec](specs/qwen38-27b-bf16-gate.md)): 4/7 prompts STRICT 16/16 vs the pinned oracle `555967922`, and all THREE first-divergence positions are EXACT fp32 TIES** — oracle-minus-ours and top-2 gap both **0.000 mnats**, our token at rank **3 / 2 / 2** in the oracle top-20, so `ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. Every one is the [#910](https://github.com/mudler/vllm.cpp/issues/910) tie-break signature and nothing else: vLLM's pick carries the LOWER token id (1814/11/16309) and ours the HIGHER (22960/13/27180) at a bit-identical logprob. Only the first divergence per prompt is adjudicable, so this is three numbers; a raw position count over the grid is NOT a quality score and is not recorded as one. Adjudicated twice on the pinned oracle's fp32 logprobs — a greedy re-decode and an independent TEACHER-FORCED probe that asserts the echoed prefix — because the earlier `transformers` bf16 CPU probe could not resolve below one bf16 ULP (every runner-up gap it printed was a multiple of 0.125) and so could not have reported anything but a tie. **SPEED on the same checkpoint, vs vLLM's PRODUCTION graphed config at the pin, clocks 2184 MHz: 1 of 3 concurrency cells established.** c4 is the only cell where both arms completed every request — **0.963x** output throughput, **1.008x** median ITL. c1 and c8 throughput WERE withheld on 2026-08-15 (superseded, below): our server failed 1/6 in all three reps and 12/11/12 of 48 where vLLM failed none in nine legs ([#931](https://github.com/mudler/vllm.cpp/issues/931)), and `output_throughput` divides tokens by a duration still containing the dead request, so c1 read 0.677x while median TPOT in the SAME file read 1.014x in our favour. **SUPERSEDED 2026-08-19 by the c1/c8 RE-MEASURE ([#915](https://github.com/mudler/vllm.cpp/issues/915), [#979](https://github.com/mudler/vllm.cpp/issues/979), `.agents/benchmark-record.md` `BENCH-QWEN38-27B-BF16 c1/c8 RE-MEASURE`):** #931 landed, and with `VT_SERVER_SSE_PING_S=0` our arm completed **162 of 162** requests, `failed=0` on every leg — c1 **4.4040 tok/s** (CV 0.039%), c8 **22.6402 tok/s** (CV 0.205%). **Our half of the withholding is discharged; NEITHER cell became a ratio and the two halves are blocked differently.** At c1 vLLM also completed everything (**4.2835 tok/s**, CV 0.033%) and `gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three pairings — the cross-arm rule PASSED (same boot, both arms 2489 MHz median, 0.0% offset) and the WITHIN-RUN rule failed on both against the 5% ceiling ([#1354](https://github.com/mudler/vllm.cpp/issues/1354): clocks cannot be pinned inside an `rc` lease), so the c1 ratio is OWED, not withheld for being unflattering. At c8 the vLLM denominator is **NOT MEASURABLE on this box at the recorded configuration** — that is the answer, not a gap, and not a claim that vLLM is defective. Read the two output-throughput absolutes with [#1355](https://github.com/mudler/vllm.cpp/issues/1355): our `usage.prompt_tokens` reports 5,942 where vLLM reports 6,144 on identical prompts, which corrupts total-token throughput outright and biases output throughput up by more than its own CV. Cold start **53 s vs 780 s = 14.7x**; host memory after warmup **42.5 vs 110.1 GiB = 2.59x**, caveated because vLLM's is set by `--gpu-memory-utilization 0.85` pre-reserving KV | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | @@ -512,7 +512,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | `Qwen3_5ForConditionalGeneration` | `registry.py:556`; `vllm/model_executor/models/qwen3_5.py::Qwen3_5ForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; GDN/linear-attention state; vision encoder; video path | 🚧 [family scoping](specs/mm-tools-scoping-2026-07-10.md); [plain-BF16 loader leaf](specs/qwen35-plain-bf16-direct-load.md); **[multimodal-track W-plan](specs/multimodal-track.md)**; full target spec required | `PARTIAL` (text-only) | text-only: `include/vllm/model_executor/models/qwen3_5_dense.h:40-105,146-171`; plain BF16/F32 + stacked/tied load `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:52-133,187-246,334-472`; plain execution `src/vllm/model_executor/models/qwen3_5.cpp:1993-2003,4579-4585,5249-5255,5533-5545`; loader route/queue reuse `src/vllm/entrypoints/model_loader.cpp:364-400`; real 4B gate `tests/vllm/models/test_qwen35_plain_weights.cpp:80-196`: CPU topology/load **1656/1656**, AOT CUDA direct OFF/ON full-engine token equivalence **1664/1664**. Existing W3-G immutable `ae9e8ff` default/fallback each pass **235/235 + 16/16** with the frozen 64 plans. Corrected root `/tmp/qwen35-transplant-4b-aot-557ab41d` proves ON=OFF 128/128 and records ON/OFF/vLLM total **6155.10/6064.06/6730.46 tok/s**, peak PSS **2.405/8.571/7.569 GiB**; current ON is 0.9316x historical AOT ON. Current-v0.25 oracle, sanitizer, vision, strict VRAM and external 27B/35B regressions remain unverified, with no new support claim. **MM-completion plan ([multimodal-track.md](specs/multimodal-track.md), `CLAIM-MULTIMODAL-TRACK`, 2026-07-25):** modalities = image + video (NO audio); reuses the landed GDN-hybrid text path — the mm half is the shared `Qwen3_VisionTransformer` (DeepStack, `qwen3_vl.py:519`) stood up on Qwen3-VL-4B first (M2) then attached to this wrapper (M3). Oracle-runnable (0.25.0 ships `qwen3_5.py`+`qwen3_vl.py`); NOT HW/oracle-blocked but **CHECKPOINT-gated** — the cached `unsloth/Qwen3.6-27B-NVFP4` quant is TEXT-ONLY (2111 tensors, ZERO `visual.*`; `vision_config` declared but weights absent), so a vision-inclusive checkpoint download is required (M0). Tower ~0.5-0.7 B params (~1-1.4 GiB bf16) fits GB10 trivially alongside the 27B. Plan owner `CLAIM-MULTIMODAL-TRACK` (row stays PARTIAL/narrative-only; the mm work re-claims it at M3). **M3-b LANDED 2026-07-25 (`CLAIM-MULTIMODAL-M3B`): IMAGE e2e WORKING** — `Qwen3_5VLGenerateGreedy` (`src/vllm/model_executor/models/qwen3_5.cpp`) forks the GDN-hybrid forward on inputs_embeds(scatter tower merger `[196,5120]` into image_token 248056 rows, no deepstack) + 3-section MRoPE `[11,11,10]` interleaved in the 16 full-attn layers (host `BuildMropeCosSinHost` → the `mrope_cos_sin` param on `DenseForwardLayers`, nullptr on text ⇒ byte-identical); vision loader `LoadQwen3VLVisionWeights` (`src/vllm/model_executor/models/qwen3_vl.cpp`, 27B config) + M2a tower + `LoadQwen3_5Dense` bf16 LLM. STRICT gate `tests/vllm/multimodal/test_qwen3_5_vl_e2e.cpp` **32/32 token-exact vs vLLM 0.25.0** (sha256 `ead4b484…`); text-inertness re-run cutlass-ON 27B/35B/Coder **235/315/138**. **M3d LANDED 2026-07-25 (`CLAIM-MULTIMODAL-M3D`): VIDEO e2e WORKING** — `Qwen3_5VLGenerateGreedyVideo` (`src/vllm/model_executor/models/qwen3_5.cpp`) reuses the M3-b image driver via a shared `VLGenerateCoreGdn` (video merge mask on video_token 248057 + `Qwen3VLGetRopeIndexVideo` per-frame temporal MRoPE; M3c processor/windowed-tower reused verbatim; no deepstack). STRICT gate `tests/vllm/multimodal/test_qwen3_5_vl_video_e2e.cpp` **32/32 token-exact vs vLLM 0.25.0** (oracle `scripts/mm/m3d_video_oracle_capture.py`, K=5 deterministic, near-tie gaps 0.0000); image e2e re-run STRICT 32/32 (refactor-safe); text SACRED byte-identical by construction (shared forward untouched). **Qwen video modalities COMPLETE (image+video e2e; audio N/A); speed still pending** (row stays PARTIAL). Speed lever #2 CLOSED 2026-07-27 (`CLAIM-MULTIMODAL-SPEED-DECODE`, multimodal-speed.md §8): on-GPU greedy argmax + decode embed round-trip removed on the shared `VLGenerateCoreGdn`; bit-exact (image+video STRICT 32/32 held, goldens md5-identical); 27B decode TPOT NEUTRAL (223 ms, ~222 ms bandwidth floor, at vLLM parity). **Speed lever #3 FIRST BRICK 2026-07-27 (`CLAIM-MULTIMODAL-SPEED-GRAPH`, multimodal-speed.md §9): the shared `VLGenerateCoreGdn` decode step now routes through the production `Qwen3_5DenseDecodeGraph` (cold→warm→replay captured decode) — mm decode is GRAPH-CAPTURABLE (was eager per-step). S==B==1 bit-identical rebuild; the decode-time 1-D device RoPE at p reproduces the degenerate MRoPE {p,p,p} → token-exact HELD (image+video STRICT 32/32, 30 graph replays confirmed); A/B graphed 232.5 vs eager 233.4 ms/tok = NEUTRAL at the 27B bandwidth floor. Structural gap closed; W-plan = Voxtral decode-graph (audio 1.52× gap-closer) + batched c2+ + serving ingestion. Row stays PARTIAL/speed-pending.** | unassigned | | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` | `Qwen3_5MoeForConditionalGeneration` | `registry.py:557-560`; `vllm/model_executor/models/qwen3_5.py::Qwen3_5MoeForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; GDN/linear-attention state; vision encoder; video path | 🚧 [family scoping](specs/mm-tools-scoping-2026-07-10.md); **[multimodal-track W-plan](specs/multimodal-track.md)**; target spec required | `PARTIAL` (text gated, vision NOT gated) | text-only: `include/vllm/model_executor/models/qwen3_5.h:1-17,97`; direct registry `src/vllm/model_executor/models/registry.cpp:10-20`; gate `tests/parity/test_qwen36_paged_engine.cpp:78,140`. W3-G immutable `ae9e8ff` correctness-only ratio-8 inertness passes **2/2 + 315/315**; no 35B performance claim; vision not implemented AT THAT DATE (it landed later, see M2/M3 below, and is still NOT gated). Disk load now DEFERS the routed-expert host copies and streams+frees them per layer during `PrepareMarlinResident` to bound load-phase peak PSS (`ENG-MOE-LOADSTREAM`, engine-matrix; CPU-gated, DGX pending) — device residents byte-identical. **MM-completion plan ([multimodal-track.md](specs/multimodal-track.md), `CLAIM-MULTIMODAL-TRACK`, 2026-07-25):** image + video (NO audio); same shared `Qwen3_VisionTransformer` as the 27B row, attached to the landed MoE GDN-hybrid text path (M3). Oracle-runnable (0.25.0); **CHECKPOINT-gated** — the cached `nvidia/Qwen3.6-35B-A3B-NVFP4` quant is TEXT-ONLY (`vision_config` declared, `visual.*` weights absent); vision-inclusive download required (M0). Tower fits GB10 alongside the 35B MoE per the landed text run. Plan owner `CLAIM-MULTIMODAL-TRACK` (row stays PARTIAL/narrative-only; the mm work re-claims it at M3). **M2/M3 LANDED (#891, `.agents/specs/moe-vision-tower.md`):** the loader no longer drops the checkpoint's 333 `model.visual.*` tensors (`LoadQwen3_5MoeVision` -> the SHARED `LoadQwen3VLVisionWeights` the dense arm is gated on; their ABSENCE is refused by name), and `Qwen3_5MoeVLGenerateGreedy[Video]` forks the forward gated on mm input over a greedy core now TEMPLATED on the weights arm rather than copied. Evidence: CPU suite 479/479 serial; the new `test_qwen3_5_moe_vision` proves the forked forward reduces EXACTLY to the text forward over the tower row (one visual token, 1x1x1 LLM grid) and that MRoPE is applied (8x8 grid must DIFFER from the 1-D run), with 4 mutations driven RED and restored byte-exact; on Thor (sm_110, FALLBACK attention) `test_qwen3_5_moe_vision_hw` loads the real 333 tensors and runs the tower on the fixture image. **OWED: the binding image and video token-exact gates vs the pinned oracle at 35B.** Not runnable on Thor -- vLLM cannot import there (`libcuda.so.1` absent on the host, `torch.cuda.is_available()` False) and the bf16 35B is ~67 GiB against this box's documented 25 GB single-model reboot ceiling; dgx.casa was off-limits mid-run for a sibling row. **TEXT ARM ORACLE-GATED ON THE PUBLISHED BF16 REPO 2026-08-15 ([#740](https://github.com/mudler/vllm.cpp/issues/740) + [#864](https://github.com/mudler/vllm.cpp/issues/864)), and this changes NOTHING about the vision claim:** greedy 7 prompts x 3 repeats x 16 tokens on `Qwen/Qwen3.6-35B-A3B` bf16 @`995ad96eacd98c81ed38be0c5b274b04031597b0` vs the pinned oracle gave **6/7 prompts STRICT 16/16**, the seventh one exact logit tie (`top2_gap_mnats = 0.0`) our on-device argmax breaks toward the higher id ([#910](https://github.com/mudler/vllm.cpp/issues/910)); only the FIRST divergence per prompt is adjudicable, so the raw 108/112 position count is NOT a quality score. SACRED inertness 3/3, goldens byte-identical (27B 235/235, 35B 315/315, Coder 138/138). NO throughput, latency or memory number exists for this checkpoint. The row therefore stays PARTIAL: **the binding image and video token-exact gates at 35B are still OWED**, the vision claim remains "the tower loads and computes" rather than "produces correct tokens", and the sm_110 run that proved it used the FALLBACK attention path, which is not coverage of the shipped GB10 path. [#908](https://github.com/mudler/vllm.cpp/issues/908)'s dense regression check is PARTIAL too: dense TEXT is 235/235 at `2f2bce926`, a true before/after (binary md5 `db889909d4…` vs `49ded1ece8…`, 500 TUs recompiled), while dense image/video stays UNVERIFIED (network-blocked) | unassigned | | `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | `Qwen4ExpForConditionalGeneration` (`model_type: qwen4_exp`; campaign row `MODEL-MM-QWEN4-EXP`) | **NOT IN vLLM AT ANY REVISION** — deliberately written with no pinned module/class target, the convention `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` follows for a beyond-pin arm, and stronger here: this is absence from vLLM `main` rather than staleness in `555967922`. Read live 2026-08-26 at `origin/main` = `6a5e8f5979`: no `qwen4*` path, no `registry.py` entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. Algorithm source is [transformers#48337](https://github.com/huggingface/transformers/pull/48337) `models/qwen4_exp/modular_qwen4_exp.py`, MERGED 2026-08-26 | conditional generation / text + image + video | MM processor; vision encoder (UNCHANGED from `Qwen3_5MoeVisionModel`); GDN/linear-attention state; block-sparse attention + indexer side cache; FusedMoE/grouped GEMM; hyper-connection residual streams; hashed n-gram embedding; dilated depthwise conv; MTP | ✅ [Qwen3.8-Flash-Next](specs/qwen4-exp-flash-next.md) | `ACTIVE` | **PRODUCT CODE LANDING, NO TOKEN, NO SPEED.** `Qwen/Qwen3.8-Flash-Next` (2026-08-24, 180B total / 6B activated). W6a ([#1989](https://github.com/mudler/vllm.cpp/issues/1989)) is the first wave to reach `main` and moves this row `READY` -> `ACTIVE`; W1/W2/W3/W4 are implemented, reviewed and repaired on their branches. **Nothing is REACHABLE yet** — no `ModelRegistry` entry resolves `Qwen4ExpForConditionalGeneration`, so every landed slice is unreached by design and named as such under the spec's `## Owed`. Split oracle by developer direction 2026-08-26: **transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH**, because `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so the reference is semantics and not a serving path. `Qwen4ExpTextModel` inherits `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`); GDN matches our AOT gate exactly (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). **Exactly two components have NO vLLM op**: the PLE dilated depthwise conv (`git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the hashed n-gram embedding. **CORRECTED after the component deep-dives: QSA's twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The first reading rested on `MLAAttentionSpec` being an MLA claim; it is a per-state BUDGET shape, and M3 — itself plain GQA — uses it. Nine structural matches with DSv4 including `compress_ratio == 4`, relu-summed MQA scoring, RoPE at the block-start position and `tokens_per_state=compress_ratio`. M3 scores by `max` over 128 RAW dots, emits one set per KV head, and welds `SPARSE_BLOCK_SIZE = 128` to the KV page size. The genuinely NEW work is the consumer: DSv4 attends COMPRESSED KV, M3 attends raw at page granularity, QSA attends RAW at ratio-4 granularity and nothing upstream does that. Any QSA gate must exceed 2048 tokens of context or it cannot distinguish a correct port from one attending pooled keys. Residual stream is `hc_count * hidden_size` = 4 x 2560 = **10240 wide through the whole stack**; `number_of_conv_states = 3` on a PLE layer plus the indexer side cache. **CORRECTED 2026-08-26 — ONE PUBLISHED ARTIFACT FITS.** `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S is **67.56 GiB** in 3 shards (`general.architecture = qwen4exp`, 1224 tensors) against ~119.6 GiB usable on GB10, roughly 52 GiB of headroom; the earlier "README with ZERO weight files" reading was true when written and false hours later. Every safetensors artifact still does not fit: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB. llama.cpp has no MERGED `qwen4_exp`, so arms nobody publishes are still ours to author and still have no llama.cpp oracle. Sizing ARITHMETIC and not measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB, Q4_K_M backbone with the 51 GB n-gram table non-resident ~76 GB — that table is 28% of the model and is touched 16 times per token, which is the offload the card itself argues for. `gateable = no` until an arm runs. **W6a CLEARED BOTH LOAD BLOCKERS**: IQ4_NL (20) and Q5_0 (6) are in the reader with geometry read from the pinned llama.cpp oracle at `b10451` and decoders gated bit-exactly against its compiled output on real file bytes, so the three shards OPEN; and `kEmbeddingTable` is keep-quant eligible behind a dequantizing gather, so the n-gram table no longer expands to 102.4 GB. `moe_intermediate_size = 640` / `hc_lowrank = 320` remain Q4_K-illegal on their reduction dims — the published file uses IQ4_NL there, and a `-Q4_K_M` build would land on Q5_0, which is why both were added. Types 3 and 7 are still unsupported and recorded as owed. `ENG-WEIGHT-OFFLOAD` is inert and does not help; the shipping mmap-borrow tier does. Both blocking decisions SETTLED 2026-08-26: the transformers lane pin is **ACCEPTED at 5.16.0** ([`oracles/transformers.md`](oracles/transformers.md)), because the registry pin 5.14.1 does not contain `Qwen4Exp` — and 5.16.0 is a real release rather than a SHA, bounded by fetching the model file at each tag (`v5.16.0` HTTP 200, `v5.15.0` HTTP 404), with the version string UNMEASURED until an oracle stands up; and the first runnable arm is the **Q4_K_M backbone with a NON-RESIDENT n-gram table** (~76 GB), Q8_0 having been raised and rejected on ~191 GB. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `CLAIM-MODEL-MM-QWEN4-EXP` | -| `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | `Glm5NextForConditionalGeneration` (`model_type: glm5_next`; campaign row `MODEL-MM-GLM53-FLASH`) | **NOT IN vLLM AT ANY REVISION** -- deliberately written with no pinned module/class target, the same convention `MODEL-MM-qwen4-exp-*` follows, so the at-the-pin static invariants are untouched. Read live 2026-08-26: `git grep "Glm5\|glm5_next"` returns ZERO hits at the pin `555967922` AND at `origin/main` `c71f6f8a81` (2026-08-26 08:42 -0700), whose `vllm/models/` package holds `common, deepseek_v32, deepseek_v4, dots3_note, inkling, kimi_k3, minimax_m3` and no `glm5next`. That is ABSENCE, not pin staleness, so advancing the parity pin -- 348 commits of reconciliation -- reaches nothing here. [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) (opened 2026-08-26T14:12:00Z, head `933876c388fb`, 85 files, +12,511/-540) would register `Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` and `Glm5NextMTPModel`, and is **OPEN, unmerged, `mergeable: false`** -- an unmerged PR is not a revision and is inadmissible as an oracle, the same rule applied to SGLang #36497 on the qwen4-exp row; it is cited only as evidence of upstream shape. SGLang (PR #36507 open, cookbook #36440/#36513 MERGED ahead of the code), llama.cpp (PR #27752 open, our pin `b10451`) and vllm-omni likewise implement nothing. **ONE row and not three:** none of the three names is registered at any vLLM revision, and the only architecture a published artifact declares is `Glm5NextForConditionalGeneration`. | conditional generation / image + video | model loader/forward; GDN/linear-attention state (KDA); MLA/latent KV; DSA sparse indexer; FusedMoE/grouped GEMM; mHC residual manifold; MM processor + ViT encoder/merge; FP8 block-wise quant; GGUF k-quant | [glm5-next-flash](specs/glm5-next-flash.md) | `READY` | **SCOPING ONLY, 2026-08-26 (issue [#1998](https://github.com/mudler/vllm.cpp/issues/1998)): the spec and its records are the whole change; NO product code has landed.** Architecture read from `config.json` live: 45 layers, hidden 4096, vocab 154880, 1,048,576 context; `layer_types` = 34 `linear_attention` + 11 `deepseek_sparse_attention` (at 3,7,...,43); MLA is **fully NoPE** (`qk_rope_head_dim: 0`, `mla_use_nope: true`, `qk_nope_head_dim: 256`, `v_head_dim: 256`, `kv_lora_rank: 512`, `q_lora_rank: 1536`) and `text_config` carries NO `rope_theta` and NO `rope_scaling` -- the transformers reference *deletes* the inherited `rope_parameters` field and passes `position_embeddings=None` to every layer, and `indexer_rope_interleave: true` is a VESTIGIAL flag the indexer override ignores, so the text stack has no rotary anywhere. DSA indexer adds a **learned per-channel k-pool** (`index_kpool: 4`, `index_kpool_compress`, `index_kpool_always_select_tail`): 128 independent 4-way softmaxes over a learned `[4,128]` intra-pool position embedding, pools aligned to the first VALID token (batch-dependent), `select_k = min(index_topk/index_kpool, P) = min(512, P)`, expanded back to raw indices and widened to `index_topk + kpool - 1` = **2051** by an unscored ragged tail. mHC `hc_mult: 4`, `hc_sinkhorn_iters: 20`, `hc_eps: 1e-6`. MoE 288 routed + 1 shared, top-8, `moe_intermediate_size` 2048, sigmoid/`noaux_tc`, `n_group=topk_group=1`, `routed_scaling_factor` 2.5, clamped SwiGLU `swiglu_limit` 10.0 in FIVE places. Vision 24L/1024/patch **14**/merge 2/temporal 2/out 4096. **HEAVY REUSE, and the two obvious reuses are WRONG in one place each.** (1) The KDA forget gate takes the OTHER branch: `gate_lower_bound: -5.0` is not None, so it computes `-5.0 * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias))`, where our `src/vllm/model_executor/models/kimi_kda.cpp:60` implements Kimi-Linear's `-exp(A_log) * softplus(g + dt_bias)` -- different functions of the same inputs, both fluent, and the sign of `decay_rate` differs too. (2) The mHC head collapse is `hidden_streams.mean(dim=2)`, an UNWEIGHTED mean whose own docstring says "Unlike DeepSeek-V4", where our `deepseek_v4_mhc.cpp:168` `HcHeadCollapse` is the V4 weighted-gated collapse; the checkpoint carries no `hc_head.*` tensors at all, which settles it. **Third trap: the k-pool selection is unobservable at or below 2048 tokens** -- with `index_topk: 2048` a shorter context selects every candidate, so any short-prompt gate passes over a broken indexer. **Structural blocker:** `MlaBlockDims::Validate` REFUSES `qk_rope_head_dim == 0` (`src/vllm/model_executor/layers/attention/mla_attention.cpp:90-93` requires every dim > 0, `:95-99` requires it even); Kimi-Linear is NOT the same case, it sets `mla_use_nope` while keeping `qk_rope_head_dim: 64` so the rope slice still occupies the cache row (`include/vllm/model_executor/models/kimi_linear.h:86,:88`). **HW: NOTHING FITS.** Measured live by HTTP RANGE over all 62 safetensors headers (no payload fetched): 76,108 tensors, `F8_E4M3` 314,396,639,232 + `BF16` 6,926,096,640 elements = **321,322,735,872 real parameters**, routed experts **311.65B = 97.0%**; on disk `zai-org/GLM-5.3-Flash` FP8 305.78 GiB (2.56x GB10), `-BF16` 598.53 GiB (5.00x), `LibertAIDAI/...-NVFP4` 181.32 GiB (1.52x). **All four `*-GGUF` repos contain ZERO `.gguf` files.** `gateable = no` on MEMORY: no oracle can execute this model on any device this project reaches, so **no end-to-end token gate exists or can exist here** -- recorded as visible debt, replaced by a tiny-shape CPU reference oracle instantiated from the pinned transformers, which is an oracle for the NUMERICS and is NOT oracle gateability for the model. Layer 45 is the MTP block (888 tensors, 7.43B params, 2.31%), present in the checkpoint and discarded by the reference; owed, not implemented. | `-` | +| `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` | `Glm5NextForConditionalGeneration` (`model_type: glm5_next`; campaign row `MODEL-MM-GLM53-FLASH`) | **NOT IN vLLM AT ANY REVISION** -- deliberately written with no pinned module/class target, the same convention `MODEL-MM-qwen4-exp-*` follows, so the at-the-pin static invariants are untouched. Read live 2026-08-26: `git grep "Glm5\|glm5_next"` returns ZERO hits at the pin `555967922` AND at `origin/main` `c71f6f8a81` (2026-08-26 08:42 -0700), whose `vllm/models/` package holds `common, deepseek_v32, deepseek_v4, dots3_note, inkling, kimi_k3, minimax_m3` and no `glm5next`. That is ABSENCE, not pin staleness, so advancing the parity pin -- 348 commits of reconciliation -- reaches nothing here. [vllm#53906](https://github.com/vllm-project/vllm/pull/53906) (opened 2026-08-26T14:12:00Z, head `933876c388fb`, 85 files, +12,511/-540) would register `Glm5NextForCausalLM`, `Glm5NextForConditionalGeneration` and `Glm5NextMTPModel`, and is **OPEN, unmerged, `mergeable: false`** -- an unmerged PR is not a revision and is inadmissible as an oracle, the same rule applied to SGLang #36497 on the qwen4-exp row; it is cited only as evidence of upstream shape. SGLang (PR #36507 open, cookbook #36440/#36513 MERGED ahead of the code), llama.cpp (PR #27752 open, our pin `b10451`) and vllm-omni likewise implement nothing. **ONE row and not three:** none of the three names is registered at any vLLM revision, and the only architecture a published artifact declares is `Glm5NextForConditionalGeneration`. | conditional generation / image + video | model loader/forward; GDN/linear-attention state (KDA); MLA/latent KV; DSA sparse indexer; FusedMoE/grouped GEMM; mHC residual manifold; MM processor + ViT encoder/merge; FP8 block-wise quant; GGUF k-quant | [glm5-next-flash](specs/glm5-next-flash.md) | `ACTIVE` | **SCOPING ONLY, 2026-08-26 (issue [#1998](https://github.com/mudler/vllm.cpp/issues/1998)): the spec and its records are the whole change; NO product code has landed.** Architecture read from `config.json` live: 45 layers, hidden 4096, vocab 154880, 1,048,576 context; `layer_types` = 34 `linear_attention` + 11 `deepseek_sparse_attention` (at 3,7,...,43); MLA is **fully NoPE** (`qk_rope_head_dim: 0`, `mla_use_nope: true`, `qk_nope_head_dim: 256`, `v_head_dim: 256`, `kv_lora_rank: 512`, `q_lora_rank: 1536`) and `text_config` carries NO `rope_theta` and NO `rope_scaling` -- the transformers reference *deletes* the inherited `rope_parameters` field and passes `position_embeddings=None` to every layer, and `indexer_rope_interleave: true` is a VESTIGIAL flag the indexer override ignores, so the text stack has no rotary anywhere. DSA indexer adds a **learned per-channel k-pool** (`index_kpool: 4`, `index_kpool_compress`, `index_kpool_always_select_tail`): 128 independent 4-way softmaxes over a learned `[4,128]` intra-pool position embedding, pools aligned to the first VALID token (batch-dependent), `select_k = min(index_topk/index_kpool, P) = min(512, P)`, expanded back to raw indices and widened to `index_topk + kpool - 1` = **2051** by an unscored ragged tail. mHC `hc_mult: 4`, `hc_sinkhorn_iters: 20`, `hc_eps: 1e-6`. MoE 288 routed + 1 shared, top-8, `moe_intermediate_size` 2048, sigmoid/`noaux_tc`, `n_group=topk_group=1`, `routed_scaling_factor` 2.5, clamped SwiGLU `swiglu_limit` 10.0 in FIVE places. Vision 24L/1024/patch **14**/merge 2/temporal 2/out 4096. **HEAVY REUSE, and the two obvious reuses are WRONG in one place each.** (1) The KDA forget gate takes the OTHER branch: `gate_lower_bound: -5.0` is not None, so it computes `-5.0 * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias))`, where our `src/vllm/model_executor/models/kimi_kda.cpp:60` implements Kimi-Linear's `-exp(A_log) * softplus(g + dt_bias)` -- different functions of the same inputs, both fluent, and the sign of `decay_rate` differs too. (2) The mHC head collapse is `hidden_streams.mean(dim=2)`, an UNWEIGHTED mean whose own docstring says "Unlike DeepSeek-V4", where our `deepseek_v4_mhc.cpp:168` `HcHeadCollapse` is the V4 weighted-gated collapse; the checkpoint carries no `hc_head.*` tensors at all, which settles it. **Third trap: the k-pool selection is unobservable at or below 2048 tokens** -- with `index_topk: 2048` a shorter context selects every candidate, so any short-prompt gate passes over a broken indexer. **Structural blocker:** `MlaBlockDims::Validate` REFUSES `qk_rope_head_dim == 0` (`src/vllm/model_executor/layers/attention/mla_attention.cpp:90-93` requires every dim > 0, `:95-99` requires it even); Kimi-Linear is NOT the same case, it sets `mla_use_nope` while keeping `qk_rope_head_dim: 64` so the rope slice still occupies the cache row (`include/vllm/model_executor/models/kimi_linear.h:86,:88`). **HW: NOTHING FITS.** Measured live by HTTP RANGE over all 62 safetensors headers (no payload fetched): 76,108 tensors, `F8_E4M3` 314,396,639,232 + `BF16` 6,926,096,640 elements = **321,322,735,872 real parameters**, routed experts **311.65B = 97.0%**; on disk `zai-org/GLM-5.3-Flash` FP8 305.78 GiB (2.56x GB10), `-BF16` 598.53 GiB (5.00x), `LibertAIDAI/...-NVFP4` 181.32 GiB (1.52x). **All four `*-GGUF` repos contain ZERO `.gguf` files.** `gateable = no` on MEMORY: no oracle can execute this model on any device this project reaches, so **no end-to-end token gate exists or can exist here** -- recorded as visible debt, replaced by a tiny-shape CPU reference oracle instantiated from the pinned transformers, which is an oracle for the NUMERICS and is NOT oracle gateability for the model. Layer 45 is the MTP block (888 tensors, 7.43B params, 2.31%), present in the checkpoint and discarded by the reference; owed, not implemented. **W7a landed 2026-08-26 ([#2011](https://github.com/mudler/vllm.cpp/issues/2011)), advancing the row to `ACTIVE`: `scripts/convert-glm5-next-gguf.py` is the safetensors->GGUF converter no upstream tool can supply, with the Q2_K/Q6_K/Q8_0 encoders ported from `ggml/src/ggml-quants.c` @ our pin `b10451` and gated BYTE-IDENTICAL against it over a frozen golden -- `gguf.quants.Q2_K` upstream implements `dequantize_blocks` and NO `quantize_blocks`, so no upstream Python can produce a k-quant at all. Its plan over the real topology emits 1719 tensors carrying 313.89B parameters (321.32B less the dropped 7.43B MTP block): Q2_K arm 100.35 GiB, Q6_K 239.89, Q8_0 310.67, BF16 584.67, so Q2_K fits ~119.63 GiB with ~17.7 GiB left after 1.43 GiB of KV at 128K and 0.14 GiB of KDA state. Everything the converter needed by way of GGUF convention was already AT THE PIN -- `class KDA` with `{arch}.kda.gate_lower_bound` at `gguf-py/gguf/constants.py:262-264`, the `ssm_conv1d_q/k/v` KDA tensor names at `src/llama-arch.cpp:465-479`, the indexer names at `:626-636` -- so no pin advance was taken. NOT REACHED: `glm5next` has no `general.architecture` dispatch entry, so the emitted file is not loadable here (O9, W1 owns it); and no artifact has been produced (O7, needs staged weights, disk and a box).** | `CLAIM-GLM53-FLASH-W7A` | | `MODEL-MM-rvl-rfor-conditional-generation` | `RForConditionalGeneration` | `registry.py:561`; `vllm/model_executor/models/rvl.py::RForConditionalGeneration` | conditional generation / video+image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-skyworkr1v-skywork-r1-vchat-model` | `SkyworkR1VChatModel` | `registry.py:562`; `vllm/model_executor/models/skyworkr1v.py::SkyworkR1VChatModel` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-smolvlm-smol-vlmfor-conditional-generation` | `SmolVLMForConditionalGeneration` | `registry.py:563`; `vllm/model_executor/models/smolvlm.py::SmolVLMForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/specs/glm5-next-flash.md b/.agents/specs/glm5-next-flash.md index a97a6d268..067757c11 100644 --- a/.agents/specs/glm5-next-flash.md +++ b/.agents/specs/glm5-next-flash.md @@ -778,8 +778,8 @@ does `del self.rope_deltas` (`:1429`). | D2 | KDA device path default-OFF | `docs/ENVIRONMENT.md:166`, `:171` | first arm runs the host compose; speed is not a wave-1..5 claim | | D3 | KDA AOT specializations exclude 64 heads | `cuda_gdn.cu:5217-5220`, `:5277-5278` | hand-kernel fallback; a named speed residual, not a blocker | | D4 | DSA device entry points are host-vector | `deepseek_v4_device.h:82-102` | upload/download per call; a named speed residual | -| D5 | No GGUF converter in tree | `scripts/` has none | wave 7 must author one | -| D6 | llama.cpp has no `glm5_next` | code search 0; PR #27752 open | no llama.cpp quant oracle and no floor for the GGUF arms | +| D5 | No GGUF converter in tree | CLOSED by W7a: `scripts/convert-glm5-next-gguf.py` | — | +| D6 | llama.cpp has no `glm5_next` | holds at `master` `539f24529` and at the pin; but the KDA/MLA/indexer KEYS and NAMES it needs are all present at `b10451` | no llama.cpp quant oracle and no floor for the GGUF arms; the CONVENTION is available without a pin advance | | D7 | transformers lane pin at `v5.16.1` | `.agents/oracles/transformers.md` pins 5.14.1 | wave 0 writes the lane pin; `gateable = no` | | D8 | Live seam contention | PRs [#1971](https://github.com/mudler/vllm.cpp/pull/1971) (DSA geometry), [#1977](https://github.com/mudler/vllm.cpp/pull/1977) (DSv4 KV multicache) | waves 3 and 5 rebase onto whichever lands first; do NOT fork the seam | @@ -897,12 +897,77 @@ ids. **Anchors:** `modular_glm5_next.py:1373-1421`, `:1563-1703`, `video_processing_glm5_next.py`. **Nearest ours:** `qwen3_vl_vision.cpp`, `qwen3vl_processor.cpp`. -### W7 — the GGUF converter and the first fitting arm (GPU + large asset) - -Author a safetensors→GGUF converter for `glm5_next` (no upstream tool can do it, -D5/D6) and produce the arm §Hardware names. **This wave needs explicit developer -authority for a large-asset download** and is the only wave that does. -**Exclusions:** no i-quant arm in this wave — see §Risks R4. +### W7 — the GGUF converter and the first fitting arm + +Split into two, because the two halves have different blockers and only one of +them needs a box. The split is recorded here rather than derived again: W7a is +CPU-only and needs no checkpoint, W7b needs 300–600 GiB of staged weights and +explicit developer authority for a large-asset download. + +#### W7a — the converter and its synthetic gate (CPU, large). LANDED + +Issue: [#2011](https://github.com/mudler/vllm.cpp/issues/2011). + +Author `scripts/convert-glm5-next-gguf.py` (no upstream tool can do it, D5/D6) +and gate it on synthetic tiny-shape fixtures. **Deliverable:** the converter, the +metadata schema, the tensor-name map, the FP8 e4m3 block dequant, the expert +stacking, the arm table with refusals, and +`tests/scripts/test_convert_glm5_next_gguf.py`. **Exclusions:** no artifact +(O7), no i-quant arm (R4), and no Q3_K/Q4_K/Q5_K encoder (O8). + +Three findings this wave moved, each of which changes what a later wave should +believe: + +**D6 was too strong, and the correction is in our favour.** llama.cpp has no +`glm5_next` — that part holds, verified at `origin/master` = `539f24529` +(fetched 2026-08-26) and at our pin: the arch enumerators are `LLM_ARCH_GLM4`, +`LLM_ARCH_GLM4_MOE` and `LLM_ARCH_GLM_DSA` (`src/llama-arch.h:86-88`), and +`src/models/glm-dsa.cpp` is GLM-5.2, citing `zai-org/GLM-5.2/blob/main/config.json`. +But **everything the converter needs by way of convention already exists AT THE +PIN `b10451`**, and no pin advance is required by any part of this wave: + +- `gguf-py/gguf/constants.py:262-264` @ b10451 already carries `class KDA` with + `{arch}.kda.head_dim` and **`{arch}.kda.gate_lower_bound`**. `KDA.SAFE_GATE` + is the only member that is `master`-only, and GLM-5.3-Flash declares no + `safe_gate` key, so nothing here reaches for it. +- The KDA tensor spellings are at `src/llama-arch.cpp:465-479` @ b10451 — + `ssm_conv1d_q/k/v` (the three separate depthwise convs, exactly the packing + this checkpoint uses), `ssm_f_a`, `ssm_f_b`, `ssm_g_a`, `ssm_g_b`, `ssm_beta`, + `ssm_a`, `ssm_dt`, `ssm_norm`. +- `gguf-py/gguf/tensor_mapping.py:896-933` @ b10451 maps them from Kimi-Linear's + HF module paths, which are **GLM-5.3-Flash's paths verbatim**. +- The indexer names, including `indexer_compressor_ape` and + `indexer_compressor_gate` for the k-pool stage, are at + `src/llama-arch.cpp:626-636` @ b10451. + +**Upstream Python cannot quantize at all.** `gguf.quants.Q2_K` implements +`dequantize_blocks` and **no** `quantize_blocks`; the Q2_K, Q6_K and Q8_0 +encoders exist only in `ggml/src/ggml-quants.c`. They are therefore ported, and +gated byte-for-byte against the pinned C reference over a frozen golden +(`tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json`) rather than +against a tolerance. Two traps changed bytes during the port and are recorded in +the source so the next reader does not re-find them: `nearest_int` is the +`+12582912.0` add-and-mask trick at `:621` and rounds half to EVEN, not +`round`; and C `roundf` in `quantize_row_q8_0_ref` rounds half AWAY FROM ZERO +where `np.rint` rounds half to even, which mis-encodes every exact `.5`. + +**The tensor inventory is exact, not inferred.** The real +`model.safetensors.index.json` (8.4 MB, 76,108 entries) and three shard headers +(shards 2, 32 and 62) were read by HTTP RANGE on 2026-08-26, payload never +fetched. They confirm four things §Port map asserted: the three separate +`{q,k,v}_conv1d` convs; `hc_{attn,ffn}_{fn,base,scale}` flat on the layer with +**no `hc_head.*` tensor at any layer**; `indexer.k_norm.bias` present, which +settles LayerNorm-with-bias over RMSNorm; and `index_kpool_compress_ape` / +`index_kpool_compress_gate` present on 12 layers — the 11 DSA layers plus the +MTP block. + +#### W7b — the first fitting artifact (GPU + large asset). NOT STARTED + +Produce the Q2_K arm and run it. **This wave needs explicit developer authority +for a large-asset download** and is the only wave that does. Owed as O7 with +what it needs named. Its gate is §Gates' W7 row unchanged: the arm loads, the +runner serves it through `include/vllm.h`, and it generates coherent text on a +prompt with an image — a RUN gate, not a token gate. ### W8 — speed, once and only once a correctness gate exists @@ -1064,7 +1129,40 @@ sequence, plus conv states. **This arithmetic is unverified against our own allocator** and #1963/#1966 record that the KV byte accounting has been wrong by 48x before, so W5 re-derives it from the runner rather than from this table. -**Recommended first arm: experts at Q2_K, ~102.6 GiB, with ~17 GiB for KV, +**W7a superseded this table with the converter's own plan, and the numbers +moved in the right direction.** The table above is bits-per-weight times a +parameter count. The converter resolves a TYPE PER TENSOR, so its plan is the +arithmetic that will actually be written, and running its type resolver over the +real topology gives 1719 output tensors and **313,890,512,702 parameters +carried** — 321.32B minus the 7.43B MTP block, which is the independent check +that the skip is exactly the 2.31% §Port map measured: + +| arm | weights | mixed bpw | breakdown | +|---|---|---|---| +| `q2_k` (experts Q2_K, everything else Q6_K) | **100.35 GiB** | 2.746 | Q2_K 93.02, Q6_K 7.17, F32 0.08, Q8_0 0.07 | +| `q6_k` | 239.89 GiB | 6.565 | Q6_K 239.73 | +| `q8_0` | 310.67 GiB | 8.502 | Q8_0 310.58 | +| `bf16` | 584.67 GiB | 16.000 | BF16 584.67 | + +The Q2_K arm is **100.35 GiB, not 102.6**, because the table above stated every +figure including layer 45 and the converter drops it. The F32 and Q8_0 slivers +are the fallback ladder: a k-quant needs `ne0 % 256 == 0`, so a row that does not +divide steps down to Q8_0 (block 32) and only falls to F32 when even 32 does not +divide (the depthwise conv kernels at 4) or the tensor is 4-D or 5-D (the +patch-embed and downsample kernels). Stepping down rather than jumping to F32 +matters for more than tidiness: with an F32 fallback a FINER arm can come out +LARGER than a coarser one, which is not a property a size table may have. + +Against ~119.63 GiB at 128K context and one sequence: weights 100.35, KV **1.43 +GiB** (11 MLA layers x `kv_lora_rank` 512 x 2 B, plus an indexer side cache of +`index_head_dim` 128 x 2 B / `index_kpool` 4 = 64 B per layer x 11, so 11,968 +B/token), KDA recurrent state **0.14 GiB** (64 heads x 128 x 128 x 4 B x 34 +layers) plus conv states. **~17.7 GiB is left** for activations, allocator +overhead and page cache. Still arithmetic and still not measurement: #1963 and +#1966 record this accounting being wrong by 48x, and W5 re-derives it from the +runner. + +**Recommended first arm: experts at Q2_K, ~100.35 GiB, with ~17.7 GiB for KV, activations and page cache at 128K context.** Not IQ2_S or below, and the reason is not quality — it is that **i-quants need an importance matrix, and an importance matrix needs a forward pass over the model, which needs 181 GiB @@ -1213,9 +1311,39 @@ Debts this row carries, each visible rather than waived: - **O4 — no llama.cpp floor and no llama.cpp oracle** for the GGUF arms (D6). - **O5 — no i-quant arm is producible on this fleet** (R4). - **O6 — speed.** No number on any axis, and no denominator exists. +- **O7 — no artifact of this model exists.** W7a authored the converter and + gated it on synthetic fixtures; it has never been run against the real + checkpoint. Producing the Q2_K arm needs the 300–600 GiB checkpoint staged on + local disk (not CIFS), explicit developer authority for the download, and a + box with room for the source and the ~100.35 GiB output at once. Until then + every GPU gate on this row — W3, W5, W6 and W7b — has nothing to load, and + §Evidence's sha256, conversion recipe and peak RSS are unpaid. + W7b/[#2011](https://github.com/mudler/vllm.cpp/issues/2011) owns it. +- **O8 — the Q3_K, Q4_K and Q5_K encoders are not ported** and the converter + refuses those arms by name. Only Q2_K, Q6_K and Q8_0 are ported from the + pinned llama.cpp reference and gated byte-for-byte against it. No arm this row + needs uses them today; the §Hardware second-choice line that mentions Q5_K for + the non-expert 3% would need this first. + [#2011](https://github.com/mudler/vllm.cpp/issues/2011) records it. +- **O9 — the converter's output is not loadable by this tree.** `glm5next` has + no entry in the `general.architecture` dispatch + (`src/vllm/entrypoints/model_loader.cpp:1000`), so the file W7a can write is a + file nothing here can read. The wiring is **W1's**, owned by row + `MODEL-MM-glm5-next-glm5-next-for-conditional-generation` and tracked by + [#1998](https://github.com/mudler/vllm.cpp/issues/1998). Named here because + W7a lands a capability that a production entry point does not yet reach, which + AGENTS.md §"Nothing lands dead" allows only when it is written down. ## Now -`READY`, 2026-08-26. The spec and its records are committed; no product code has -landed. The next action is to claim W0 or W1 on a fresh -`row/MODEL-MM-GLM53-FLASH-W` branch. +`ACTIVE`, 2026-08-26. Advanced from `READY` by W7a +([#2011](https://github.com/mudler/vllm.cpp/issues/2011), +`CLAIM-GLM53-FLASH-W7A`), which lands the first product code on the row: the +safetensors→GGUF converter and its synthetic-fixture gate, with the Q2_K, Q6_K +and Q8_0 encoders byte-identical to the pinned llama.cpp `b10451` reference. + +**No artifact exists** (O7) and **nothing in this tree can read what the +converter writes** (O9), so no GPU gate has moved and no correctness claim about +the MODEL has been made. The next actions are W1 — config, registration and the +`general.architecture` dispatch entry that discharges O9 — and, whenever the +developer grants a large-asset download, W7b. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2a2029c14..5da6ee182 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -495,6 +495,13 @@ jobs: sudo apt-get update -qq sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends python3-numpy python3 tests/scripts/test_ltx25_render_compare.py + # Same numpy-only lane, same reason (#2011): the GLM-5.3-Flash + # GGUF converter avoids gguf-py on purpose -- upstream has no + # `glm5_next`, and `gguf.quants.Q2_K` has a dequantizer and NO + # quantizer -- so numpy is its only dependency. Preflight SKIPS + # this suite when numpy is absent; this lane is where it must + # never be silent. + python3 tests/scripts/test_convert_glm5_next_gguf.py python3 tests/scripts/test_ltx25_pixel_ab_harness.py - name: Every LTX-2.5 A/B arm sets a knob value the dispatch parses (#1751) # The `VLLM_LTX2_DIT_FLASH_ATTN` values a harness exports and the values diff --git a/docs/USAGE.md b/docs/USAGE.md index c9c0fa564..8f53fe0f5 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -611,8 +611,60 @@ repository in this project's history. | Qwen3.8-2.4T-A95B | `UD-Q1_0` ten-file GGUF split | about 370 GiB | `unsloth/Qwen3.8-2.4T-A95B-GGUF` @ `567d3e6ac26c5474b18311e619c04350fb9a5556` | `b7770552b2ac24e7334c917bc92e90e218e87cfe29484db65e62e8ef2a60334d` (shard 1); `2765517f833c736338d3ab34354e1c10eb8d79e62325f998285b435e5cf03dcd` (shard 2) | CPU expert streaming from disk | CUDA refuses a checkpoint that exceeds device capacity | | DeepSeek-V4-Flash EXL3 trellis shard 1 of 172 | `exl3-layer-000-tp4-rank0.safetensors` | 515,850,920 bytes | `0xSero/deepseek-v4-flash-0731-spark` @ `22f28d32b9b29b4352eaa380ff8c2c170b2847ab` | `2ed7ae798a794019810b027fe2609e2cf4ad78d70b49c47b2970d03a0a7aaadf` | The rank-sliced EXL3 routed-expert tower LOADS (TP4 coalesced to TP1) and its experts EXECUTE through `vt::Exl3Gemm` on a CPU queue | The CUDA arm has passed no compiler and no GPU, so no device arm is claimed. A SYNTHETIC rank-sliced checkpoint now loads and emits logits end to end; THIS artifact still does not, because its DSA compressor and indexer tensors are stored at twice the width the host forward indexes (`compressor.wgate` `[2*head_dim, H]`) and the loader refuses them BY NAME, and because its tokenizer is not read ([#1924](https://github.com/mudler/vllm.cpp/issues/1924)) | | DeepSeek-V4-Flash EXL3 carried tower shard 1 of 5 | `carried-001.safetensors` | 4,288,630,252 bytes | `0xSero/deepseek-v4-flash-0731-spark` @ `22f28d32b9b29b4352eaa380ff8c2c170b2847ab` | `3b67ae29f1e75c2ecadfcafd3b0eecec640b06fd60b832f77e6bd3c2a8c85ccf` | The un-requantized `deepseek_v4_fp8` attention, router, shared-expert, compressor and embedding tensors, MATERIALIZED at load into the host-float tower the forward composes with — block-wise FP8 (`F8_E4M3` + `F8_E8M0` over 128x128 blocks) decoded to f32, BF16 norms and embeddings widened, I64 `tid2eid` narrowed to int32 | The DSA compressor and indexer tensors of this artifact are `2 * head_dim` / `2 * index_head_dim` wide and the loader refuses them by name (41 of its 43 layers carry a compressor); the 3,985 `mtp.*` NVFP4 draft tensors are skipped and counted, never silently dropped | +| GLM-5.3-Flash FP8 source | `model-000{01..62}-of-00062.safetensors` | 328,326,771,576 bytes total (305.78 GiB) | `zai-org/GLM-5.3-Flash` @ `main`, read 2026-08-26 | Owed: no byte of payload has been fetched, so no local hash exists to state, and an unauthenticated tree hash is not a pin here | Declared source of `scripts/convert-glm5-next-gguf.py`. Only the safetensors HEADERS were read, by HTTP RANGE over all 62 shards: 76,108 tensors, `F8_E4M3` block-quantized at `weight_block_size: [128, 128]` with `weight_scale_inv` companions, plus BF16 and F32 scales | **Nothing has been converted.** The download needs explicit developer authority and a box with room for 305.78 GiB of source and ~100.35 GiB of output at once; owed as O7 on [#2011](https://github.com/mudler/vllm.cpp/issues/2011). The revision is a branch name and not a commit, which is NOT a pin: it is what was read, and W7b re-reads and records the commit when it stages the bytes | +| GLM-5.3-Flash GGUF | none exists | n/a | `unsloth/GLM-5.3-Flash-GGUF`, `AtomicChat/GLM-5.3-Flash-GGUF`, `aj9o9/GLM-5.3-Flash-GGUF`, `vcruz305/GLM-5.3-Flash-GGUF`, all read 2026-08-26 | n/a | none | **All four repositories named `*-GGUF` contain ZERO `.gguf` files** — READMEs, a `.gitattributes` and four PNGs between them. A repository name is not an artifact, and this row exists so the next reader does not go looking again. llama.cpp cannot produce one either: no `glm5_next` at `origin/master` `539f24529` or at our pin `b10451` | +### Convert a GLM-5.3-Flash checkpoint to GGUF + +`zai-org/GLM-5.3-Flash` (`Glm5NextForConditionalGeneration` / `glm5_next`) +publishes no arm that fits any device this project reaches, and no upstream tool +can make one: llama.cpp has no `glm5_next` at our pin `b10451` or at its +`master`, and `gguf-py`'s `Q2_K` has a dequantizer and **no** quantizer. So the +converter ships here. + +```sh +# Read the headers and print the plan, without writing a byte. +scripts/convert-glm5-next-gguf.py --src /path/to/GLM-5.3-Flash --arm q2_k --dry-run + +# Write the arm. +scripts/convert-glm5-next-gguf.py --src /path/to/GLM-5.3-Flash \ + --dst GLM-5.3-Flash-Q2_K.gguf --arm q2_k +``` + +numpy is its only dependency. It streams shard by shard, so peak resident memory +is one tensor rather than one shard, but the output is written in one pass and +needs its full size free on the destination. + +| `--arm` | what the experts get | everything else | weights on the real model | +|---|---|---|---| +| `q2_k` | Q2_K | Q6_K | **100.35 GiB** — the only arm that fits ~119.63 GiB | +| `q6_k` | Q6_K | Q6_K | 239.89 GiB | +| `q8_0` | Q8_0 | Q8_0 | 310.67 GiB | +| `bf16`, `f16`, `f32` | passthrough | passthrough | 584.67 GiB at bf16 | + +Routed and shared experts are 97% of this model, so the arm name is the expert +type and the remaining 3% rides at a finer one almost for free. Figures are the +converter's own per-tensor plan over the real topology (1719 tensors, 313.89B +parameters after the layer-45 MTP block is dropped), not bits-per-weight times a +parameter count. + +**Refused by name, each with the missing part.** Every i-quant — `iq1_s`, +`iq2_xxs`, `iq2_s`, `iq3_xxs`, `iq4_xs`, `iq1_xxxs` — needs an importance +matrix, an importance matrix needs a forward pass over the model, and the +smallest published artifact is 181.32 GiB, so the dependency is circular on this +fleet. `q3_k`, `q4_k` and `q5_k` are refused because those encoders are not +ported: only Q2_K, Q6_K and Q8_0 are ported from `ggml/src/ggml-quants.c` at the +pinned llama.cpp `b10451` and gated byte-for-byte against it. +`--keep-mtp` is refused because nothing here reads an MTP tail. + +**The file it writes is not loadable by this tree yet.** `glm5next` has no +`general.architecture` dispatch entry, so the converter runs ahead of the model +port. That is tracked on +[#1998](https://github.com/mudler/vllm.cpp/issues/1998), and no artifact has +been produced against the real checkpoint either +([#2011](https://github.com/mudler/vllm.cpp/issues/2011)). + ### The distilled NVFP4 DiT was re-quantized under an unchanged name `Lightricks/LTX-2.5` published two different files at diff --git a/scripts/agent-preflight.sh b/scripts/agent-preflight.sh index 224270f93..d04891729 100755 --- a/scripts/agent-preflight.sh +++ b/scripts/agent-preflight.sh @@ -400,6 +400,23 @@ else "numpy is not importable here, and the tool this suite exercises needs it." \ "CI installs python3-numpy and runs the same suite." fi +# THE GLM-5.3-Flash GGUF CONVERTER (#2011). Same shape and the same one +# dependency: `scripts/convert-glm5-next-gguf.py` deliberately does not use +# gguf-py -- upstream has no `glm5_next` and, decisively, `gguf.quants.Q2_K` +# implements `dequantize_blocks` and NO `quantize_blocks` -- so numpy is all it +# needs and all this suite needs. The suite builds a SYNTHETIC tiny-shape +# checkpoint: it needs no weights, no GPU and no C++ build, which is the whole +# reason the converter can be gated at all while the real 305.78 GiB artifact +# stays out of reach. +# +# A missing numpy is a SKIP and never an `ok`, for the reason above. +if python3 -c 'import numpy' >/dev/null 2>&1; then + run "test_convert_glm5_next_gguf" python3 tests/scripts/test_convert_glm5_next_gguf.py +else + skip "test_convert_glm5_next_gguf" \ + "numpy is not importable here, and the converter this suite exercises needs it." \ + "CI installs python3-numpy and runs the same suite." +fi # THE WINDOWS SOURCE CONTRACT, which ran on NO lane until #1829 (#646, #680). # `main` failed to COMPILE under MSVC while every POSIX lane was green, because # `[[noreturn]]` on a non-void return type is C4646 -> C2220 there and a silent diff --git a/scripts/convert-glm5-next-gguf.py b/scripts/convert-glm5-next-gguf.py new file mode 100755 index 000000000..e0cbde946 --- /dev/null +++ b/scripts/convert-glm5-next-gguf.py @@ -0,0 +1,1246 @@ +#!/usr/bin/env python3 +"""safetensors -> GGUF converter for `glm5_next` (GLM-5.3-Flash). + +W7a of `MODEL-MM-GLM53-FLASH` ([#1998](https://github.com/mudler/vllm.cpp/issues/1998), +wave issue in the pull request body). Spec: `.agents/specs/glm5-next-flash.md`. + +**Why this file exists at all.** `zai-org/GLM-5.3-Flash` publishes FP8 (305.78 +GiB), BF16 (598.53 GiB) and NVFP4 (181.32 GiB) artifacts, and the largest device +this project reaches is `dgx:gpu0` at ~119.63 GiB unified. Nothing published +fits. All four HuggingFace repositories named `*-GGUF` contain zero `.gguf` +files, and no upstream tool can emit this architecture: llama.cpp at our pin +`b10451` knows `glm4`, `glm4moe` and `glm-dsa` and has no `glm5_next` (verified +2026-08-26, `src/llama-arch.h` and a repository-wide search). So the converter is +ours to write, and until it exists there is no artifact of this model that any +gate on this fleet can load. + +**What this file is NOT.** It has never been run against the real checkpoint. +W7a deliberately produces no artifact -- the checkpoint is 300-600 GiB and +downloading it needs authority and disk this wave has neither of. Everything +below is gated on synthetic tiny-shape fixtures, and artifact production is owed +(spec `## Owed`, O7). The k-quant encoders are the exception: they are gated +byte-for-byte against the pinned llama.cpp reference, so the one part that +cannot be checked by inspection is checked by an oracle. + +**Where the bytes come from.** llama.cpp is never the mirror source -- AGENTS.md +settles behaviour against vLLM, and where vLLM implements nothing, against the +declared secondary oracle. What llama.cpp supplies here is *convention*: the +GGUF container layout, the metadata key spellings, the tensor-name spellings and +the k-quant block encodings. Every one of those is read at our pin `b10451` and +not at llama.cpp `master`: + + - metadata keys `gguf-py/gguf/constants.py` @ b10451, including + `class KDA` (:262) with `{arch}.kda.head_dim` (:263) and + `{arch}.kda.gate_lower_bound` (:264). `KDA.SAFE_GATE` + exists only at `master` and is NOT used here, because + GLM-5.3-Flash's `linear_attn_config` declares no + `safe_gate` key. Nothing in this file requires a pin + advance. + - tensor names `src/llama-arch.cpp` @ b10451: the KDA set at :465-479 + (`ssm_conv1d_q/k/v`, `ssm_f_a`, `ssm_f_b`, `ssm_g_a`, + `ssm_g_b`, `ssm_beta`, `ssm_a`, `ssm_dt`, `ssm_norm`), + the MLA set at :481-486 and the indexer set at :626-636. + - HF module paths `gguf-py/gguf/tensor_mapping.py` @ b10451 :896-933, whose + Kimi-Linear KDA paths (`self_attn.q_conv1d`, + `self_attn.f_a_proj`, ...) are GLM-5.3-Flash's paths + verbatim. + - k-quant encoders `ggml/src/ggml-quants.c` @ b10451, ported function by + function with the line anchors on each one. + +The in-tree precedent for the per-layer spellings is +`scripts/check-dsv4-gguf-namemap.py`, which already carries `hc_attn_fn.weight`, +`ffn_gate_exps.weight` and the DeepSeek-V4 MLA names. + +**The arch string `glm5next` is OURS.** llama.cpp has no `glm5_next` enumerator +at any revision and its PR #27752 is open, so no upstream spelling exists to +mirror. If llama.cpp lands a different string, this converter and the loader +reconcile onto it in the same change. + +Self-contained on purpose: numpy is the only import beyond the standard library, +matching `scripts/gen-minimax-h3-gguf-manifest.py` and +`scripts/gen-gguf-nvfp4-goldens.py`. gguf-py is deliberately not used -- it has +no `glm5_next` and, more to the point, `gguf.quants.Q2_K` implements +`dequantize_blocks` and **no** `quantize_blocks`, so upstream Python cannot +produce a Q2_K artifact at all. + +Usage: + + scripts/convert-glm5-next-gguf.py --src --dst out.gguf \\ + --arm q2_k [--keep-mtp] [--no-vision] [--dry-run] + +`--dry-run` reads only the safetensors headers and prints the plan and the exact +output size, which is how an arm's arithmetic is checked before 100+ GiB of +writing starts. +""" + +import argparse +import json +import os +import struct +import sys + +try: + import numpy as np +except ImportError: # pragma: no cover - reported, never guessed around + sys.stderr.write( + "convert-glm5-next-gguf: numpy is required and is not importable.\n") + raise SystemExit(4) + +f32 = np.float32 + +# --------------------------------------------------------------------------- +# ggml type ids. Only the ids this converter can EMIT are named. Every one of +# them has both traits and a dequant path in our own reader -- traits alone are +# not enough, and `gguf_dequant.cpp:89-160` is the list that decides: +# IQ4_XS (23) and IQ1_XXXS (21) have traits and NO dequant, which is why no arm +# below reaches for them. +GGML_F32 = 0 +GGML_F16 = 1 +GGML_Q8_0 = 8 +GGML_Q2_K = 10 +GGML_Q6_K = 14 +GGML_BF16 = 30 + +TYPE_NAME = {GGML_F32: "F32", GGML_F16: "F16", GGML_Q8_0: "Q8_0", + GGML_Q2_K: "Q2_K", GGML_Q6_K: "Q6_K", GGML_BF16: "BF16"} +# (block elements, bytes per block) +TYPE_BLOCK = {GGML_F32: (1, 4), GGML_F16: (1, 2), GGML_BF16: (1, 2), + GGML_Q8_0: (32, 34), GGML_Q2_K: (256, 84), GGML_Q6_K: (256, 210)} + +QK_K = 256 + +# GGUF metadata value type ids (gguf-py/gguf/constants.py `GGUFValueType`). +GV_UINT8, GV_INT8, GV_UINT16, GV_INT16, GV_UINT32, GV_INT32 = 0, 1, 2, 3, 4, 5 +GV_FLOAT32, GV_BOOL, GV_STRING, GV_ARRAY, GV_UINT64 = 6, 7, 8, 9, 10 +GV_INT64, GV_FLOAT64 = 11, 12 + +ARCH = "glm5next" +GGUF_MAGIC = b"GGUF" +GGUF_VERSION = 3 +GGUF_ALIGNMENT = 32 + + +# --------------------------------------------------------------------------- +# k-quant encoders, ported from llama.cpp `ggml/src/ggml-quants.c` @ b10451. +# +# These are ported rather than called because no upstream Python can produce +# them (see the module docstring) and because building llama.cpp is not a +# dependency this converter may take. They are gated byte-for-byte against the +# pinned C reference by `tests/scripts/test_convert_glm5_next_gguf.py` over a +# frozen golden captured from that reference; a divergence in the last ulp shows +# up as a byte difference, not as a tolerance argument. +# +# Two traps the port had to reproduce exactly, both of which changed bytes: +# - `nearest_int` is NOT `round`. It is the 12582912.0 add-and-mask trick at +# :621, which rounds half to EVEN. +# - C `roundf` in `quantize_row_q8_0_ref` rounds half AWAY FROM ZERO, where +# `np.rint` rounds half to even. Using `np.rint` mis-encodes every exact .5. +# Float32 accumulation order is mirrored lane by lane rather than handed to +# `ndarray.sum`, whose pairwise summation is a different order. + +def _nearest_int(v): + """`ggml-quants.c:621` @ b10451, vectorized.""" + v = np.asarray(v, dtype=f32) + f32(12582912.0) + i = v.view(np.int32) + return (i & 0x007FFFFF) - 0x00400000 + + +def _fp16(x): + return np.asarray(x, dtype=f32).astype(np.float16) + + +GROUP_MAX_EPS = f32(1e-15) + + +def _make_qkx2_quants(x, nmax=3, rmin=f32(-0.5), rdelta=f32(0.1), nstep=15): + """`ggml-quants.c:799` @ b10451, vectorized over rows of `x` (M, n). + + `min` is state that UPDATES inside the search loop and the next step's + `nearest_int(iscale*(x - min))` reads the updated value, so the 16 steps stay + sequential; only the rows are vectorized. + """ + M, n = x.shape + w = np.abs(x).astype(f32) + mn = x.min(axis=1).astype(f32) + mx = x.max(axis=1).astype(f32) + sum_w = w[:, 0].copy() + sum_x = (sum_w * x[:, 0]).astype(f32) + for i in range(1, n): + sum_w = (sum_w + w[:, i]).astype(f32) + sum_x = (sum_x + (w[:, i] * x[:, i]).astype(f32)).astype(f32) + mn = np.where(mn > f32(0), f32(0), mn).astype(f32) + degen = mx == mn + span = (mx - mn).astype(f32) + safe = np.where(degen, f32(1), span).astype(f32) + iscale = (f32(nmax) / safe).astype(f32) + scale = (f32(1) / iscale).astype(f32) + L = np.clip(_nearest_int((iscale[:, None] * (x - mn[:, None]).astype(f32)).astype(f32)), + 0, nmax).astype(np.uint8) + best = np.zeros(M, dtype=f32) + for i in range(n): + diff = np.abs((scale * L[:, i].astype(f32) + mn - x[:, i]).astype(f32)).astype(f32) + best = (best + (w[:, i] * diff).astype(f32)).astype(f32) + for istep in range(nstep + 1): + denom = np.where(degen, f32(1), (mx - mn).astype(f32)).astype(f32) + isc = ((rmin + rdelta * f32(istep) + f32(nmax)) / denom).astype(f32) + isc = np.where(degen, f32(0), isc).astype(f32) + Laux = np.clip(_nearest_int((isc[:, None] * (x - mn[:, None]).astype(f32)).astype(f32)), + 0, nmax).astype(np.uint8) + sum_l = np.zeros(M, dtype=f32) + sum_l2 = np.zeros(M, dtype=f32) + sum_xl = np.zeros(M, dtype=f32) + for i in range(n): + li = Laux[:, i].astype(f32) + wl = (w[:, i] * li).astype(f32) + sum_l = (sum_l + wl).astype(f32) + sum_l2 = (sum_l2 + (wl * li).astype(f32)).astype(f32) + sum_xl = (sum_xl + (wl * x[:, i]).astype(f32)).astype(f32) + D = ((sum_w * sum_l2).astype(f32) - (sum_l * sum_l).astype(f32)).astype(f32) + okD = D > f32(0) + Dsafe = np.where(okD, D, f32(1)).astype(f32) + this_scale = ((((sum_w * sum_xl).astype(f32) - (sum_x * sum_l).astype(f32)).astype(f32)) + / Dsafe).astype(f32) + this_min = ((((sum_l2 * sum_x).astype(f32) - (sum_l * sum_xl).astype(f32)).astype(f32)) + / Dsafe).astype(f32) + pos = this_min > f32(0) + l2safe = np.where(sum_l2 == f32(0), f32(1), sum_l2).astype(f32) + this_scale = np.where(pos, (sum_xl / l2safe).astype(f32), this_scale).astype(f32) + this_min = np.where(pos, f32(0), this_min).astype(f32) + cur = np.zeros(M, dtype=f32) + for i in range(n): + diff = np.abs(((this_scale * Laux[:, i].astype(f32)).astype(f32) + + this_min - x[:, i]).astype(f32)).astype(f32) + cur = (cur + (w[:, i] * diff).astype(f32)).astype(f32) + upd = okD & (cur < best) + L = np.where(upd[:, None], Laux, L) + best = np.where(upd, cur, best).astype(f32) + scale = np.where(upd, this_scale, scale).astype(f32) + mn = np.where(upd, this_min, mn).astype(f32) + scale = np.where(degen, f32(0), scale).astype(f32) + L = np.where(degen[:, None], np.uint8(0), L) + return scale, (-mn).astype(f32), L + + +def quantize_q2_k(x): + """(nb, 256) f32 -> (nb, 84) u8. `ggml-quants.c:891` @ b10451.""" + x = np.ascontiguousarray(x, dtype=f32) + nb = x.shape[0] + scales, mins, L = _make_qkx2_quants(x.reshape(nb * 16, 16)) + scales = scales.reshape(nb, 16) + mins = mins.reshape(nb, 16) + L = L.reshape(nb, 256) + q4scale = f32(15.0) + max_scale = np.maximum(scales.max(axis=1), f32(0)).astype(f32) + max_min = np.maximum(mins.max(axis=1), f32(0)).astype(f32) + pos_s = max_scale > f32(0) + isc = np.where(pos_s, (q4scale / np.where(pos_s, max_scale, f32(1))).astype(f32), + f32(0)).astype(f32) + out_scales = np.where(pos_s[:, None], + _nearest_int((isc[:, None] * scales).astype(f32)).astype(np.uint8), + np.uint8(0)) + d = np.where(pos_s, (max_scale / q4scale).astype(f32), f32(0)).astype(f32) + pos_m = max_min > f32(0) + iscm = np.where(pos_m, (q4scale / np.where(pos_m, max_min, f32(1))).astype(f32), + f32(0)).astype(f32) + lm = _nearest_int((iscm[:, None] * mins).astype(f32)).astype(np.uint8) + out_scales = np.where(pos_m[:, None], out_scales | (lm << 4), out_scales) + dmin = np.where(pos_m, (max_min / q4scale).astype(f32), f32(0)).astype(f32) + dh = _fp16(d) + dminh = _fp16(dmin) + dl = (dh.astype(f32)[:, None] * (out_scales & 0xF).astype(f32)).astype(f32) + dm = (dminh.astype(f32)[:, None] * (out_scales >> 4).astype(f32)).astype(f32) + dl_e = np.repeat(dl, 16, axis=1) + dm_e = np.repeat(dm, 16, axis=1) + nz = dl_e != f32(0) + num = (x + dm_e).astype(f32) + Lq = np.clip(_nearest_int(np.divide(num, np.where(nz, dl_e, f32(1))).astype(f32)), + 0, 3).astype(np.uint8) + L = np.where(nz, Lq, L) + qs = np.zeros((nb, 64), dtype=np.uint8) + for j in range(0, 256, 128): + base = j // 4 + seg = L[:, j:j + 128] + qs[:, base:base + 32] = (seg[:, 0:32] | (seg[:, 32:64] << 2) + | (seg[:, 64:96] << 4) | (seg[:, 96:128] << 6)) + out = np.zeros((nb, 84), dtype=np.uint8) + out[:, 0:16] = out_scales + out[:, 16:80] = qs + out[:, 80:82] = dh.view(np.uint8).reshape(nb, 2) + out[:, 82:84] = dminh.view(np.uint8).reshape(nb, 2) + return out + + +def _make_qx_quants(x, nmax=32): + """`ggml-quants.c:628` @ b10451 with `rmse_type=1`, `qw=NULL`.""" + M, n = x.shape + ax = np.abs(x).astype(f32) + idx = np.argmax(ax, axis=1) + rows = np.arange(M) + amax = ax[rows, idx] + mx = x[rows, idx].astype(f32) + allzero = amax < GROUP_MAX_EPS + mxs = np.where(allzero, f32(1), mx).astype(f32) + w = (x * x).astype(f32) + iscale = (f32(-nmax) / mxs).astype(f32) + lq = np.clip(_nearest_int((iscale[:, None] * x).astype(f32)), -nmax, nmax - 1).astype(np.int32) + L = (lq + nmax).astype(np.uint8) + sumlx = np.zeros(M, dtype=f32) + suml2 = np.zeros(M, dtype=f32) + for i in range(n): + li = lq[:, i].astype(f32) + sumlx = (sumlx + ((w[:, i] * x[:, i]).astype(f32) * li).astype(f32)).astype(f32) + suml2 = (suml2 + ((w[:, i] * li).astype(f32) * li).astype(f32)).astype(f32) + l2safe = np.where(suml2 == f32(0), f32(1), suml2).astype(f32) + scale = np.where(suml2 != f32(0), (sumlx / l2safe).astype(f32), f32(0)).astype(f32) + best = (scale * sumlx).astype(f32) + for istep in list(range(-9, 0)) + list(range(1, 10)): + isc = ((-(f32(nmax) + (f32(0.1) * f32(istep)).astype(f32)).astype(f32)) / mxs).astype(f32) + lq2 = np.clip(_nearest_int((isc[:, None] * x).astype(f32)), + -nmax, nmax - 1).astype(np.int32) + sx = np.zeros(M, dtype=f32) + s2 = np.zeros(M, dtype=f32) + for i in range(n): + li = lq2[:, i].astype(f32) + sx = (sx + ((w[:, i] * x[:, i]).astype(f32) * li).astype(f32)).astype(f32) + s2 = (s2 + ((w[:, i] * li).astype(f32) * li).astype(f32)).astype(f32) + upd = (s2 > f32(0)) & ((sx * sx).astype(f32) > (best * s2).astype(f32)) + L = np.where(upd[:, None], (lq2 + nmax).astype(np.uint8), L) + news = (sx / np.where(s2 == f32(0), f32(1), s2)).astype(f32) + best = np.where(upd, (news * sx).astype(f32), best).astype(f32) + scale = np.where(upd, news, scale).astype(f32) + scale = np.where(allzero, f32(0), scale).astype(f32) + L = np.where(allzero[:, None], np.uint8(0), L) + return scale, L + + +def quantize_q6_k(x): + """(nb, 256) f32 -> (nb, 210) u8. `ggml-quants.c:1869` @ b10451.""" + x = np.ascontiguousarray(x, dtype=f32) + nb = x.shape[0] + scales, L8 = _make_qx_quants(x.reshape(nb * 16, 16)) + scales = scales.reshape(nb, 16) + L = L8.reshape(nb, 256).astype(np.int32) + absc = np.abs(scales).astype(f32) + j = np.argmax(absc, axis=1) + rows = np.arange(nb) + dead = absc[rows, j] < GROUP_MAX_EPS + ms = np.where(dead, f32(1), scales[rows, j].astype(f32)).astype(f32) + iscale = (f32(-128.0) / ms).astype(f32) + dh = _fp16((f32(1) / iscale).astype(f32)) + sc = np.minimum(_nearest_int((iscale[:, None] * scales).astype(f32)), 127).astype(np.int8) + dl = (dh.astype(f32)[:, None] * sc.astype(f32)).astype(f32) + dl_e = np.repeat(dl, 16, axis=1) + nz = dl_e != f32(0) + Lq = np.clip(_nearest_int(np.divide(x, np.where(nz, dl_e, f32(1))).astype(f32)), + -32, 31) + 32 + L = np.where(nz, Lq.astype(np.int32), L) + ql = np.zeros((nb, 128), dtype=np.uint8) + qh = np.zeros((nb, 64), dtype=np.uint8) + for jj in range(0, 256, 128): + o_ql = (jj // 128) * 64 + o_qh = (jj // 128) * 32 + s = L[:, jj:jj + 128].astype(np.uint8) + ql[:, o_ql:o_ql + 32] = (s[:, 0:32] & 0xF) | ((s[:, 64:96] & 0xF) << 4) + ql[:, o_ql + 32:o_ql + 64] = (s[:, 32:64] & 0xF) | ((s[:, 96:128] & 0xF) << 4) + qh[:, o_qh:o_qh + 32] = ((s[:, 0:32] >> 4) | ((s[:, 32:64] >> 4) << 2) + | ((s[:, 64:96] >> 4) << 4) | ((s[:, 96:128] >> 4) << 6)) + out = np.zeros((nb, 210), dtype=np.uint8) + out[:, 0:128] = ql + out[:, 128:192] = qh + out[:, 192:208] = sc.view(np.uint8) + out[:, 208:210] = dh.view(np.uint8).reshape(nb, 2) + if dead.any(): + out[dead] = 0 + return out + + +def quantize_q8_0(x): + """(nb, 32) f32 -> (nb, 34) u8. `ggml-quants.c:276` @ b10451.""" + x = np.ascontiguousarray(x, dtype=f32) + nb = x.shape[0] + amax = np.abs(x).max(axis=1).astype(f32) + d = (amax / f32(127.0)).astype(f32) + idv = np.where(d != f32(0), (f32(1.0) / np.where(d == f32(0), f32(1), d)).astype(f32), + f32(0)).astype(f32) + dh = _fp16(d) + v = (x * idv[:, None]).astype(f32) + # C `roundf` rounds half AWAY FROM ZERO; `np.rint` rounds half to EVEN. + qs = np.where(v >= f32(0), np.floor((v + f32(0.5)).astype(f32)), + np.ceil((v - f32(0.5)).astype(f32))).astype(np.int8) + out = np.zeros((nb, 34), dtype=np.uint8) + out[:, 0:2] = dh.view(np.uint8).reshape(nb, 2) + out[:, 2:34] = qs.view(np.uint8) + return out + + +def encode(rows, ggml_type): + """Encode a 2-D f32 array (rows x ne0) into the ggml block layout.""" + rows = np.ascontiguousarray(rows, dtype=f32) + if ggml_type == GGML_F32: + return rows.tobytes() + if ggml_type == GGML_F16: + return rows.astype(np.float16).tobytes() + if ggml_type == GGML_BF16: + u = rows.view(np.uint32) + # round-to-nearest-even bf16, the same rounding the loader's reader + # expects; a plain truncation biases every weight toward zero. + rounded = ((u + np.uint32(0x7FFF) + ((u >> np.uint32(16)) & np.uint32(1))) + >> np.uint32(16)).astype(np.uint16) + return rounded.tobytes() + n_elem, _ = TYPE_BLOCK[ggml_type] + flat = rows.reshape(-1, n_elem) + if ggml_type == GGML_Q8_0: + return quantize_q8_0(flat).tobytes() + if ggml_type == GGML_Q2_K: + return quantize_q2_k(flat).tobytes() + if ggml_type == GGML_Q6_K: + return quantize_q6_k(flat).tobytes() + raise ValueError("no encoder for ggml type %d" % ggml_type) + + +def type_nbytes(shape, ggml_type): + n = 1 + for d in shape: + n *= d + n_elem, blk_bytes = TYPE_BLOCK[ggml_type] + if n % n_elem: + raise ValueError("element count %d is not a multiple of the %s block %d" + % (n, TYPE_NAME[ggml_type], n_elem)) + return (n // n_elem) * blk_bytes + + +# --------------------------------------------------------------------------- +# safetensors reading. Header only for the plan pass; mmap slices for the data +# pass, so the peak resident set is one tensor rather than one shard. + +_ST_DTYPE = { + "F64": np.float64, "F32": np.float32, "F16": np.float16, + "BF16": np.uint16, "F8_E4M3": np.uint8, "F8_E5M2": np.uint8, + "I64": np.int64, "I32": np.int32, "I16": np.int16, "I8": np.int8, + "U8": np.uint8, "BOOL": np.bool_, +} + + +class Safetensors: + """Lazy multi-shard reader: headers up front, payload on demand.""" + + def __init__(self, src): + self.src = src + index = os.path.join(src, "model.safetensors.index.json") + if os.path.exists(index): + with open(index) as fh: + self.weight_map = json.load(fh)["weight_map"] + else: + single = os.path.join(src, "model.safetensors") + if not os.path.exists(single): + raise SystemExit( + "convert-glm5-next-gguf: %s has neither " + "model.safetensors.index.json nor model.safetensors" % src) + self.weight_map = None + self._headers = {} + self._maps = {} + if self.weight_map is None: + self.weight_map = {k: "model.safetensors" + for k in self._header("model.safetensors") + if k != "__metadata__"} + + def _header(self, shard): + if shard not in self._headers: + path = os.path.join(self.src, shard) + with open(path, "rb") as fh: + n = struct.unpack("> 7) else 1.0 + e = (b >> 3) & 0xF + m = b & 0x7 + if e == 0xF and m == 0x7: + out[b] = np.float32("nan") + elif e == 0: + out[b] = np.float32(s * (m / 8.0) * (2.0 ** -6)) + else: + out[b] = np.float32(s * (1.0 + m / 8.0) * (2.0 ** (e - 7))) + _E4M3_TABLE = out + return _E4M3_TABLE + + +def dequant_fp8_block(q_u8, scale_inv, block): + """FP8 e4m3 with a `weight_scale_inv` block grid -> f32. + + The checkpoint stores `[128, 128]` block scales and `w = q * scale_inv`, + which is the DeepSeek/vLLM block-wise FP8 convention this repository already + carries. Trailing partial blocks take the last block's scale, which is what + the ceil-division grid in the shard headers encodes (a `[2048, 4096]` expert + has a `[16, 32]` scale grid). + """ + q = _e4m3_table()[q_u8] + rows, cols = q.shape + br, bc = block + sr, sc = scale_inv.shape + ri = np.minimum(np.arange(rows) // br, sr - 1) + ci = np.minimum(np.arange(cols) // bc, sc - 1) + return (q * scale_inv[np.ix_(ri, ci)].astype(f32)).astype(f32) + + +# --------------------------------------------------------------------------- +# GGUF writing. + +def _pack_str(s): + b = s.encode("utf-8") + return struct.pack(" GGUF tensor name. +# +# Spellings are llama.cpp's at b10451 (`src/llama-arch.cpp`, anchors in the +# module docstring); the HF side is what the real checkpoint's +# `model.safetensors.index.json` carries, read live 2026-08-26 over 76,108 +# tensors. `model.language_model.` is stripped because `Glm5NextForConditional +# Generation` nests the text model under the multimodal wrapper and the GGUF +# block namespace is flat. + +KDA_MAP = { + "self_attn.q_proj.weight": "attn_q.weight", + "self_attn.k_proj.weight": "attn_k.weight", + "self_attn.v_proj.weight": "attn_v.weight", + "self_attn.o_proj.weight": "attn_output.weight", + "self_attn.q_conv1d.weight": "ssm_conv1d_q.weight", + "self_attn.k_conv1d.weight": "ssm_conv1d_k.weight", + "self_attn.v_conv1d.weight": "ssm_conv1d_v.weight", + "self_attn.f_a_proj.weight": "ssm_f_a.weight", + "self_attn.f_b_proj.weight": "ssm_f_b.weight", + "self_attn.g_a_proj.weight": "ssm_g_a.weight", + "self_attn.g_b_proj.weight": "ssm_g_b.weight", + "self_attn.b_proj.weight": "ssm_beta.weight", + "self_attn.A_log": "ssm_a", + "self_attn.dt_bias": "ssm_dt", + "self_attn.o_norm.weight": "ssm_norm.weight", +} + +DSA_MAP = { + "self_attn.q_a_proj.weight": "attn_q_a.weight", + "self_attn.q_a_layernorm.weight": "attn_q_a_norm.weight", + "self_attn.q_b_proj.weight": "attn_q_b.weight", + "self_attn.kv_a_proj_with_mqa.weight": "attn_kv_a_mqa.weight", + "self_attn.kv_a_layernorm.weight": "attn_kv_a_norm.weight", + "self_attn.kv_b_proj.weight": "attn_kv_b.weight", + "self_attn.o_proj.weight": "attn_output.weight", + "self_attn.indexer.wq_b.weight": "indexer.attn_q_b.weight", + "self_attn.indexer.wk.weight": "indexer.attn_k.weight", + "self_attn.indexer.k_norm.weight": "indexer.k_norm.weight", + "self_attn.indexer.k_norm.bias": "indexer.k_norm.bias", + "self_attn.indexer.weights_proj.weight": "indexer.proj.weight", + "self_attn.indexer.index_kpool_compress_ape": "indexer_compressor_ape.weight", + "self_attn.indexer.index_kpool_compress_gate": "indexer_compressor_gate.weight", +} + +COMMON_MAP = { + "input_layernorm.weight": "attn_norm.weight", + "post_attention_layernorm.weight": "ffn_norm.weight", + # mHC lives FLAT on the layer in the checkpoint, not under `attn_hc.*`. + # `hc_head.*` does not exist at any layer, which is what settles the + # unweighted-mean head collapse; nothing here allocates one. + "hc_attn_fn": "hc_attn_fn.weight", + "hc_attn_base": "hc_attn_base.weight", + "hc_attn_scale": "hc_attn_scale.weight", + "hc_ffn_fn": "hc_ffn_fn.weight", + "hc_ffn_base": "hc_ffn_base.weight", + "hc_ffn_scale": "hc_ffn_scale.weight", +} + +DENSE_MLP_MAP = { + "mlp.gate_proj.weight": "ffn_gate.weight", + "mlp.up_proj.weight": "ffn_up.weight", + "mlp.down_proj.weight": "ffn_down.weight", +} + +SPARSE_MLP_MAP = { + "mlp.gate.weight": "ffn_gate_inp.weight", + "mlp.gate.e_score_correction_bias": "exp_probs_b.bias", + "mlp.shared_experts.gate_proj.weight": "ffn_gate_shexp.weight", + "mlp.shared_experts.up_proj.weight": "ffn_up_shexp.weight", + "mlp.shared_experts.down_proj.weight": "ffn_down_shexp.weight", +} + +# Per-expert tensors are STACKED: the checkpoint stores +# `mlp.experts.{0..287}.{gate,up,down}_proj.weight` and the GGUF carries one +# 3-D tensor per projection, which is llama.cpp's `ffn_*_exps` convention and +# our own reader's `kStackedExpertWeight` role. +EXPERT_STACK = { + "gate_proj": "ffn_gate_exps.weight", + "up_proj": "ffn_up_exps.weight", + "down_proj": "ffn_down_exps.weight", +} + +VISION_MAP = { + "patch_embed.proj.weight": "v.patch_embd.weight", + "patch_embed.proj.bias": "v.patch_embd.bias", + "post_layernorm.weight": "v.post_ln.weight", + "downsample.weight": "v.downsample.weight", + "downsample.bias": "v.downsample.bias", + "merger.proj.weight": "v.merger.proj.weight", + "merger.gate_proj.weight": "v.merger.gate.weight", + "merger.up_proj.weight": "v.merger.up.weight", + "merger.down_proj.weight": "v.merger.down.weight", + "merger.post_projection_norm.weight": "v.merger.norm.weight", + "merger.post_projection_norm.bias": "v.merger.norm.bias", +} + +VISION_BLOCK_MAP = { + "norm1.weight": "ln1.weight", + "norm2.weight": "ln2.weight", + "attn.qkv.weight": "attn_qkv.weight", + "attn.qkv.bias": "attn_qkv.bias", + "attn.proj.weight": "attn_out.weight", + "attn.proj.bias": "attn_out.bias", + "attn.q_norm.weight": "attn_q_norm.weight", + "attn.k_norm.weight": "attn_k_norm.weight", + "mlp.gate_proj.weight": "ffn_gate.weight", + "mlp.gate_proj.bias": "ffn_gate.bias", + "mlp.up_proj.weight": "ffn_up.weight", + "mlp.up_proj.bias": "ffn_up.bias", + "mlp.down_proj.weight": "ffn_down.weight", + "mlp.down_proj.bias": "ffn_down.bias", +} + + +# --------------------------------------------------------------------------- + +def pick_type(gguf_name, shape, arm): + """Resolve the ggml type for one output tensor. + + Two rules, and both are refusals to guess rather than preferences: + + - A 1-D tensor (norm, bias, `A_log`, `dt_bias`, the mHC scalars) stays + F32. It is a vector, not a GEMM operand; quantizing it saves nothing and + costs accuracy on every layer. + - A quantized type needs `ne0 % block == 0`. `ne0` is the ROW length, the + contraction dimension. Where a k-quant's 256-element super-block does not + divide the row -- the k-pool positional embedding at 128, the vision + head norms -- the tensor steps DOWN to Q8_0, whose block is 32, and only + falls all the way to F32 when even 32 does not divide (the depthwise conv + kernels at 4) or the tensor is 4-D or 5-D (the patch-embed and downsample + kernels). Stepping down rather than jumping to F32 is what keeps the arms + ordered by size; an arm whose fallback is 32 bits can be LARGER than a + coarser one, which is not a property a size table should have. Nothing is + reshaped to make a row divide, and every fallback shows in the plan. + """ + types = ARMS[arm] + want = types["experts"] if gguf_name.endswith("_exps.weight") else types["other"] + quantized = (GGML_Q8_0, GGML_Q2_K, GGML_Q6_K) + if want not in quantized: + return want + if len(shape) < 2 or len(shape) > 3: + return GGML_F32 + ne0 = shape[-1] + for candidate in (want, GGML_Q8_0): + n_elem, _ = TYPE_BLOCK[candidate] + if ne0 % n_elem == 0: + return candidate + return GGML_F32 + + +class Plan: + """One output tensor: where its bytes come from and what they become.""" + + def __init__(self, gguf_name, sources, shape, ggml_type, kind): + self.gguf_name = gguf_name + self.sources = sources # list of HF tensor names, stacked in order + self.shape = shape # logical shape, numpy/torch order + self.ggml_type = ggml_type + self.kind = kind # "plain" | "stacked" + self.nbytes = type_nbytes(shape, ggml_type) + + @property + def dims_ggml(self): + return list(reversed(self.shape)) + + +def _layer_types(text_cfg): + lt = text_cfg.get("layer_types") + if not lt: + raise SystemExit( + "convert-glm5-next-gguf: text_config carries no `layer_types`. That " + "list is the AUTHORITATIVE per-layer schedule for this architecture " + "-- the reference ignores `linear_attn_config.kda_layers` and " + "`full_attn_layers` entirely -- so there is nothing to fall back to.") + return lt + + +def build_plan(st, cfg, arm, keep_mtp, want_vision): + text = cfg["text_config"] + n_layers = int(text["num_hidden_layers"]) + layer_types = _layer_types(text) + if len(layer_types) != n_layers: + raise SystemExit("convert-glm5-next-gguf: layer_types has %d entries and " + "num_hidden_layers is %d" % (len(layer_types), n_layers)) + mlp_types = text.get("mlp_layer_types") or [] + n_experts = int(text["n_routed_experts"]) + prefix = "model.language_model." + plans = [] + skipped = [] + + def add(gguf_name, hf_names, shape, kind="plain"): + t = pick_type(gguf_name, shape, arm) + plans.append(Plan(gguf_name, hf_names, shape, t, kind)) + + def logical(name): + # The FP8 form stores one byte per element, so the safetensors shape IS + # the logical shape; the `weight_scale_inv` companion is metadata and is + # never emitted as a tensor of its own. + # + # ggml carries at most GGML_MAX_DIMS = 4 axes, and this checkpoint has + # two tensors that exceed it or sit awkwardly inside it: the vision + # patch embed is a Conv3d at `[1024, 3, 2, 14, 14]` (five axes, which + # simply cannot be written) and the downsample is a Conv2d at + # `[4096, 1024, 2, 2]`. Both are consumed as a matmul over flattened + # patches, so both are flattened to `[out, prod(rest)]` here -- the + # im2col form llama.cpp's own vision converters write. Flattening is + # NOT free and is not applied blindly: it is confined to 4-D and 5-D + # convolution kernels. The 3-D shapes are left alone, because they mean + # something in ggml -- the depthwise KDA conv is `[ch, 1, k]` and the + # stacked expert lane is `[experts, n, m]`, and reshaping either would + # change how the loader indexes it. + shape = st.shape(name) + if len(shape) > 3: + n = 1 + for d in shape[1:]: + n *= d + return [shape[0], n] + return shape + + add("token_embd.weight", [prefix + "embed_tokens.weight"], + logical(prefix + "embed_tokens.weight")) + add("output_norm.weight", [prefix + "norm.weight"], logical(prefix + "norm.weight")) + if st.has("lm_head.weight"): + add("output.weight", ["lm_head.weight"], logical("lm_head.weight")) + + for L in range(n_layers): + base = "%slayers.%d." % (prefix, L) + table = dict(COMMON_MAP) + table.update(KDA_MAP if layer_types[L] == "linear_attention" else DSA_MAP) + for hf_suffix, gg_suffix in sorted(table.items()): + hf = base + hf_suffix + if not st.has(hf): + continue + add("blk.%d.%s" % (L, gg_suffix), [hf], logical(hf)) + sparse = (mlp_types[L] if L < len(mlp_types) + else ("sparse" if st.has(base + "mlp.gate.weight") else "dense")) + if sparse == "dense": + for hf_suffix, gg_suffix in sorted(DENSE_MLP_MAP.items()): + hf = base + hf_suffix + if st.has(hf): + add("blk.%d.%s" % (L, gg_suffix), [hf], logical(hf)) + else: + for hf_suffix, gg_suffix in sorted(SPARSE_MLP_MAP.items()): + hf = base + hf_suffix + if st.has(hf): + add("blk.%d.%s" % (L, gg_suffix), [hf], logical(hf)) + for proj, gg_suffix in sorted(EXPERT_STACK.items()): + names = ["%smlp.experts.%d.%s.weight" % (base, e, proj) + for e in range(n_experts)] + missing = [n for n in names if not st.has(n)] + if missing: + raise SystemExit( + "convert-glm5-next-gguf: layer %d is sparse but %d of its " + "%d `%s` expert tensors are absent (first: %s)" + % (L, len(missing), n_experts, proj, missing[0])) + per = logical(names[0]) + add("blk.%d.%s" % (L, gg_suffix), names, [n_experts] + per, "stacked") + + # The MTP block. `num_hidden_layers` is 45 and the checkpoint carries 46 + # layer directories; `model.language_model.layers.45.*` is a DeepSeek-V3 + # style multi-token-prediction block that the transformers reference + # DISCARDS (`_keys_to_ignore_on_load_unexpected` names `layers\.45\.` and + # `shared_head\.`). We follow the reference and the in-tree + # `glm4_moe_lite_registry.cpp:21-26` precedent and request only + # `[0, num_hidden_layers)`. It is 888 tensors and 7.43B parameters -- 2.31% + # of the model, ~2.3 GiB of a Q2_K arm. + for name in st.weight_map: + if name.startswith("%slayers.%d." % (prefix, n_layers)) or ".shared_head." in name: + skipped.append(name) + if keep_mtp and skipped: + raise SystemExit( + "convert-glm5-next-gguf: --keep-mtp is REFUSED. Missing part: the " + "multi-token-prediction head is not implemented on the loader side " + "either, so a GGUF carrying %d layer-%d tensors would be %d tensors " + "nothing can read. Owed as O2 in .agents/specs/glm5-next-flash.md." + % (len(skipped), n_layers, len(skipped))) + + vision_plans = [] + if want_vision: + vcfg = cfg.get("vision_config") or {} + depth = int(vcfg.get("depth", 0)) + for hf_suffix, gg_name in sorted(VISION_MAP.items()): + hf = "model.visual." + hf_suffix + if st.has(hf): + t = pick_type(gg_name, logical(hf), arm) + vision_plans.append(Plan(gg_name, [hf], logical(hf), t, "plain")) + for b in range(depth): + for hf_suffix, gg_suffix in sorted(VISION_BLOCK_MAP.items()): + hf = "model.visual.blocks.%d.%s" % (b, hf_suffix) + if st.has(hf): + gg = "v.blk.%d.%s" % (b, gg_suffix) + t = pick_type(gg, logical(hf), arm) + vision_plans.append(Plan(gg, [hf], logical(hf), t, "plain")) + plans.extend(vision_plans) + return plans, skipped + + +def write_metadata(w, cfg, arm, n_layers, name): + text = cfg["text_config"] + vis = cfg.get("vision_config") or {} + lin = text.get("linear_attn_config") or {} + layer_types = _layer_types(text) + + def u32(k, v): + w.add(k, GV_UINT32, int(v)) + + def f(k, v): + w.add(k, GV_FLOAT32, float(v)) + + def s(k, v): + w.add(k, GV_STRING, str(v)) + + def b(k, v): + w.add(k, GV_BOOL, bool(v)) + + s("general.architecture", ARCH) + s("general.type", "model") + s("general.name", name) + s("general.basename", "GLM-5.3-Flash") + s("general.size_label", "%dA%dB" % ( + round(int(text["n_routed_experts"]) * int(text["moe_intermediate_size"]) + * 3 * int(text["hidden_size"]) / 1e9), + round(int(text["num_experts_per_tok"]) * int(text["moe_intermediate_size"]) + * 3 * int(text["hidden_size"]) / 1e9))) + # `general.file_type` is a UINT32 enum upstream, not a string: + # `llama_ftype` in `include/llama.h:117-149` @ b10451. A mixed arm + # reports the type its BULK carries, which is what "MOSTLY" means + # there, so the q2_k arm reports Q2_K even though the other 3% is + # Q6_K. The arm name travels beside it as a string, because "q2_k" + # names a policy of ours that no upstream enum has a value for. + u32("general.file_type", FTYPE[arm]) + s("general.quantization_arm", arm) + u32("general.quantization_version", 2) + u32("general.alignment", GGUF_ALIGNMENT) + + k = ARCH + u32("%s.vocab_size" % k, text["vocab_size"]) + u32("%s.context_length" % k, text["max_position_embeddings"]) + u32("%s.embedding_length" % k, text["hidden_size"]) + u32("%s.block_count" % k, n_layers) + u32("%s.feed_forward_length" % k, text["intermediate_size"]) + u32("%s.expert_feed_forward_length" % k, text["moe_intermediate_size"]) + u32("%s.expert_shared_feed_forward_length" % k, text["moe_intermediate_size"]) + u32("%s.leading_dense_block_count" % k, + sum(1 for t in (text.get("mlp_layer_types") or []) if t == "dense")) + u32("%s.expert_count" % k, text["n_routed_experts"]) + u32("%s.expert_used_count" % k, text["num_experts_per_tok"]) + u32("%s.expert_shared_count" % k, text["n_shared_experts"]) + u32("%s.expert_group_count" % k, text["n_group"]) + u32("%s.expert_group_used_count" % k, text["topk_group"]) + f("%s.expert_weights_scale" % k, text["routed_scaling_factor"]) + b("%s.expert_weights_norm" % k, text["norm_topk_prob"]) + # `scoring_func: sigmoid` -> gating func 2, llama.cpp's + # `LLAMA_EXPERT_GATING_FUNC_TYPE_SIGMOID`. + u32("%s.expert_gating_func" % k, 2 if text["scoring_func"] == "sigmoid" else 1) + u32("%s.attention.head_count" % k, text["num_attention_heads"]) + u32("%s.attention.head_count_kv" % k, text["num_key_value_heads"]) + f("%s.attention.layer_norm_rms_epsilon" % k, text["rms_norm_eps"]) + u32("%s.attention.q_lora_rank" % k, text["q_lora_rank"]) + u32("%s.attention.kv_lora_rank" % k, text["kv_lora_rank"]) + u32("%s.attention.key_length_mla" % k, text["qk_head_dim"]) + u32("%s.attention.value_length_mla" % k, text["v_head_dim"]) + u32("%s.attention.key_length" % k, text["qk_nope_head_dim"]) + u32("%s.attention.value_length" % k, text["v_head_dim"]) + f("%s.swiglu_clamp_exp" % k, text["swiglu_limit"]) + f("%s.swiglu_clamp_shexp" % k, text["swiglu_limit"]) + u32("%s.nextn_predict_layers" % k, 0) # the MTP tail is not carried; O2. + + # NO ROPE. `qk_rope_head_dim` is 0 and `mla_use_nope` is true; the reference + # deletes the inherited rope parameters and passes `position_embeddings=None` + # to every layer. `indexer_rope_interleave` is a VESTIGIAL config flag the + # indexer override ignores, so no `{arch}.rope.*` key is written and a reader + # that finds one here is reading a different file. + u32("%s.rope.dimension_count" % k, 0) + + # DSA indexer geometry. Keys: llama.cpp b10451 + # `src/llama-arch.cpp:272-277`. + u32("%s.attention.indexer.head_count" % k, text["index_n_heads"]) + u32("%s.attention.indexer.key_length" % k, text["index_head_dim"]) + u32("%s.attention.indexer.top_k" % k, text["index_topk"]) + w.add("%s.attention.indexer.types" % k, GV_ARRAY, + (GV_STRING, list(text.get("indexer_types") or []))) + # The k-pool compression stage has NO upstream key at any revision, because + # no upstream implements this indexer. These three are OURS, namespaced under + # the indexer they belong to. `index_kpool` is 4 HERE and the config class + # default is 16, so a reader that defaults instead of reading is wrong. + u32("%s.attention.indexer.kpool" % k, text["index_kpool"]) + b("%s.attention.indexer.kpool_compress" % k, text["index_kpool_compress"]) + b("%s.attention.indexer.kpool_always_select_tail" % k, + text["index_kpool_always_select_tail"]) + + # KDA. Keys: llama.cpp b10451 `gguf-py/gguf/constants.py:262-264` / + # `src/llama-arch.cpp:320-321`. `gate_lower_bound` is the parameter this port + # hinges on: it is -5.0 here and NOT None, which selects + # `-bound * sigmoid(exp(A_log) * (g + dt_bias))`, a DIFFERENT function from + # the `-exp(A_log) * softplus(g + dt_bias)` our Kimi-Linear KDA implements -- + # different formula, and the sign of `decay_rate` differs too. Writing the + # value into the file is what lets the loader take the right branch instead + # of inheriting Kimi's. + u32("%s.kda.head_dim" % k, lin["head_dim"]) + f("%s.kda.gate_lower_bound" % k, lin["gate_lower_bound"]) + u32("%s.attention.linear_head_count" % k, lin["num_heads"]) + u32("%s.ssm.conv_kernel" % k, lin["short_conv_kernel_size"]) + + # mHC. Keys: llama.cpp b10451 `HyperConnection` in constants.py. + # `hc_eps` is a DIFFERENT constant from `rms_norm_eps` (1e-6 vs 1e-5) and is + # added to every Sinkhorn denominator rather than used as a floor. + u32("%s.hyper_connection.count" % k, text["hc_mult"]) + u32("%s.hyper_connection.sinkhorn_iterations" % k, text["hc_sinkhorn_iters"]) + f("%s.hyper_connection.epsilon" % k, text["hc_eps"]) + + # The per-layer schedule. The top-level `layer_types` list is authoritative; + # `linear_attn_config.kda_layers` and `full_attn_layers` are IGNORED by the + # reference and are not carried here, so nothing downstream can read the + # wrong one. + w.add("%s.layer_types" % k, GV_ARRAY, (GV_STRING, list(layer_types))) + if text.get("mlp_layer_types"): + w.add("%s.mlp_layer_types" % k, GV_ARRAY, + (GV_STRING, list(text["mlp_layer_types"]))) + + # Multimodal. Image and video share ONE token id: the processor emits + # `image_token_id` for video frames too and disambiguates by the + # begin/end-of-video SPAN, so all six ids travel together or a reader + # classifies every frame as an image. + for key in ("image_token_id", "video_token_id", "image_start_token_id", + "image_end_token_id", "video_start_token_id", "video_end_token_id"): + if key in cfg: + u32("%s.%s" % (k, key), cfg[key]) + if vis: + u32("%s.vision.block_count" % k, vis["depth"]) + u32("%s.vision.embedding_length" % k, vis["hidden_size"]) + u32("%s.vision.feed_forward_length" % k, vis["intermediate_size"]) + u32("%s.vision.head_count" % k, vis["num_heads"]) + u32("%s.vision.patch_size" % k, vis["patch_size"]) + u32("%s.vision.image_size" % k, vis["image_size"]) + u32("%s.vision.spatial_merge_size" % k, vis["spatial_merge_size"]) + u32("%s.vision.temporal_patch_size" % k, vis["temporal_patch_size"]) + u32("%s.vision.out_embedding_length" % k, vis["out_hidden_size"]) + u32("%s.vision.projection_intermediate_size" % k, + vis["projection_intermediate_size"]) + f("%s.vision.attention.layer_norm_rms_epsilon" % k, vis["rms_norm_eps"]) + f("%s.vision.swiglu_clamp" % k, vis["swiglu_limit"]) + + +def materialize(st, plan, block): + """Read a plan's sources and return its rows as f32 (rows x ne0).""" + ne0 = plan.shape[-1] + chunks = [] + for name in plan.sources: + raw, dt = st.raw(name) + if dt == "F8_E4M3": + scale_name = name[:-len(".weight")] + ".weight_scale_inv" \ + if name.endswith(".weight") else name + ".weight_scale_inv" + if not st.has(scale_name): + raise SystemExit( + "convert-glm5-next-gguf: %s is F8_E4M3 and has no " + "`weight_scale_inv` companion. A block-quantized FP8 tensor " + "without its scales cannot be dequantized, and treating the " + "bytes as values would produce a loadable, wrong file." + % name) + scale, _ = st.raw(scale_name) + vals = dequant_fp8_block(raw, np.asarray(scale, dtype=f32), block) + elif dt == "BF16": + vals = _bf16_to_f32(raw) + else: + vals = np.asarray(raw, dtype=f32) + chunks.append(np.ascontiguousarray(vals, dtype=f32).reshape(-1, ne0)) + return np.concatenate(chunks, axis=0) if len(chunks) > 1 else chunks[0] + + +def main(argv=None): + ap = argparse.ArgumentParser( + description="Convert a GLM-5.3-Flash (`glm5_next`) checkpoint to GGUF.") + ap.add_argument("--src", required=True, help="checkpoint directory") + ap.add_argument("--dst", help="output .gguf (omit with --dry-run)") + ap.add_argument("--arm", default="q2_k", + help="quantization arm: " + ", ".join(sorted(ARMS)) + + " (refused: " + ", ".join(sorted(REFUSED_ARMS)) + ")") + ap.add_argument("--keep-mtp", action="store_true", + help="carry the layer-45 MTP block (refused; see O2)") + ap.add_argument("--no-vision", action="store_true", + help="omit the vision tower") + ap.add_argument("--dry-run", action="store_true", + help="read headers only; print the plan and the exact output size") + ap.add_argument("--name", default="GLM-5.3-Flash", help="general.name") + args = ap.parse_args(argv) + + arm = args.arm.lower() + if arm in REFUSED_ARMS: + sys.stderr.write(refuse_arm(arm) + "\n") + return 3 + if arm not in ARMS: + sys.stderr.write("convert-glm5-next-gguf: unknown arm %r. Producible: %s. " + "Refused by name: %s.\n" + % (args.arm, ", ".join(sorted(ARMS)), + ", ".join(sorted(REFUSED_ARMS)))) + return 2 + if not args.dry_run and not args.dst: + sys.stderr.write("convert-glm5-next-gguf: --dst is required without --dry-run\n") + return 2 + + cfg_path = os.path.join(args.src, "config.json") + if not os.path.exists(cfg_path): + sys.stderr.write("convert-glm5-next-gguf: %s has no config.json\n" % args.src) + return 2 + with open(cfg_path) as fh: + cfg = json.load(fh) + + arch_list = cfg.get("architectures") or [] + if cfg.get("model_type") != "glm5_next" or \ + "Glm5NextForConditionalGeneration" not in arch_list: + sys.stderr.write( + "convert-glm5-next-gguf: this converter handles `model_type: " + "glm5_next` / `Glm5NextForConditionalGeneration` only. This " + "checkpoint declares model_type=%r architectures=%r. Converting it " + "anyway would emit a file whose metadata describes a model the " + "weights are not.\n" % (cfg.get("model_type"), arch_list)) + return 2 + + qcfg = cfg.get("quantization_config") or {} + block = tuple(qcfg.get("weight_block_size") or (128, 128)) + if qcfg and qcfg.get("quant_method") not in (None, "fp8"): + sys.stderr.write( + "convert-glm5-next-gguf: quantization_config.quant_method=%r is not " + "handled. Missing part: only the published `fp8` e4m3 block form and " + "unquantized bf16 checkpoints have a dequant path here.\n" + % qcfg.get("quant_method")) + return 3 + if qcfg and qcfg.get("fmt") not in (None, "e4m3"): + sys.stderr.write("convert-glm5-next-gguf: quantization_config.fmt=%r is " + "not handled; only e4m3.\n" % qcfg.get("fmt")) + return 3 + + st = Safetensors(args.src) + text = cfg["text_config"] + n_layers = int(text["num_hidden_layers"]) + plans, skipped = build_plan(st, cfg, arm, args.keep_mtp, not args.no_vision) + + w = GgufWriter() + write_metadata(w, cfg, arm, n_layers, args.name) + for p in plans: + if len(p.dims_ggml) > 4: + sys.stderr.write( + "convert-glm5-next-gguf: %s has %d axes and ggml carries at most " + "4 (GGML_MAX_DIMS). Writing it would produce a header no reader " + "can index, so nothing is written.\n" + % (p.gguf_name, len(p.dims_ggml))) + return 3 + w.add_tensor(p.gguf_name, p.dims_ggml, p.ggml_type, p.nbytes) + head, data_bytes = w.header_bytes() + + counts = {} + for p in plans: + counts[TYPE_NAME[p.ggml_type]] = counts.get(TYPE_NAME[p.ggml_type], 0) + 1 + total = len(head) + data_bytes + sys.stderr.write( + "convert-glm5-next-gguf: arch=%s arm=%s layers=%d tensors=%d " + "skipped_mtp=%d header=%d data=%d total=%d (%.2f GiB)\n" + % (ARCH, arm, n_layers, len(plans), len(skipped), len(head), data_bytes, + total, total / float(1 << 30))) + sys.stderr.write("convert-glm5-next-gguf: types " + " ".join( + "%s=%d" % (t, c) for t, c in sorted(counts.items())) + "\n") + if args.dry_run: + for p in plans: + sys.stdout.write("%s\t%s\t%s\t%d\n" + % (p.gguf_name, TYPE_NAME[p.ggml_type], + ",".join(str(d) for d in p.dims_ggml), p.nbytes)) + return 0 + + with open(args.dst, "wb") as out: + out.write(head) + for p in plans: + rows = materialize(st, p, block) + blob = encode(rows, p.ggml_type) + if len(blob) != p.nbytes: + raise SystemExit( + "convert-glm5-next-gguf: %s encoded to %d bytes, the header " + "reserved %d. The file would be unreadable, so nothing is " + "written past this point." % (p.gguf_name, len(blob), p.nbytes)) + out.write(blob) + pad = (-len(blob)) % GGUF_ALIGNMENT + if pad: + out.write(b"\x00" * pad) + sys.stderr.write("convert-glm5-next-gguf: wrote %s\n" % args.dst) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json b/tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json new file mode 100644 index 000000000..6490b8067 --- /dev/null +++ b/tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json @@ -0,0 +1,31 @@ +{ + "provenance": { + "oracle": "llama.cpp", + "pin": "b10451", + "object": "10bf611e533d81f739128304991c5e133c6aebd8", + "source": "ggml/src/ggml-quants.c", + "anchors": { + "nearest_int": 621, + "make_qx_quants": 628, + "make_qkx2_quants": 799, + "quantize_row_q2_K_ref": 891, + "quantize_row_q6_K_ref": 1869, + "quantize_row_q8_0_ref": 276 + }, + "harness": "the five functions above spliced verbatim into a standalone C file, gcc -O2 -ffp-contract=off -std=c11, _Float16 for GGML_FP32_TO_FP16", + "captured": "2026-08-26", + "blocks": [ + "0: ordinary weight-like spread", + "1: all zeros (the degenerate max==min and amax None: every other check stays green. Only an assertion that names the row goes red. - `READY` is pinned deliberately and is the weaker half of the evidence, + The state is pinned deliberately and is the weaker half of the evidence, stated rather than implied, for the same reason the qwen4-exp test gives: - the row is `READY` because its spec is committed and no product code has - landed, and pinning it here means a future refactor of the - structured-spec or claim-ownership rules cannot silently take this pin - with it. + pinning it here means a future refactor of the structured-spec or + claim-ownership rules cannot silently take this pin with it. + + It was `READY` when this test was written, on the stated premise that the + spec was committed and no product code had landed. W7a (#2011) landed + product code -- `scripts/convert-glm5-next-gguf.py`, the first thing on + this row that is not a record -- so the premise expired and the pin moves + with it to `ACTIVE`, in the same change that moves the matrix row. The + assertion is NOT weakened: it still names one exact state, and a pin that + followed the row automatically would assert nothing at all. What it stops + catching is only the one transition it was updated for; it still goes red + on a rename, on a second glm5_next row, and on any later state change + made without touching this file. """ errors: list[str] = [] rows, _ = agent_record.check_matrices(errors) @@ -704,7 +713,7 @@ def test_glm5_next_row_is_inside_the_model_ratchet(self) -> None: found = [row for row in rows if row.item_id == item_id] self.assertEqual(len(found), 1, item_id) self.assertEqual(found[0].path.name, "model-matrix.md", item_id) - self.assertEqual(found[0].field("state").strip().strip("`"), "READY", item_id) + self.assertEqual(found[0].field("state").strip().strip("`"), "ACTIVE", item_id) # One row, not three: neither the text-only arm nor the MTP head has a # sibling row, and adding one to mirror the upstream PR is the mistake. diff --git a/tests/scripts/test_convert_glm5_next_gguf.py b/tests/scripts/test_convert_glm5_next_gguf.py new file mode 100644 index 000000000..808318d09 --- /dev/null +++ b/tests/scripts/test_convert_glm5_next_gguf.py @@ -0,0 +1,876 @@ +#!/usr/bin/env python3 +"""Gate for `scripts/convert-glm5-next-gguf.py` (W7a of MODEL-MM-GLM53-FLASH). + +The converter is the only path to a GLM-5.3-Flash artifact that fits any device +this project reaches, and it has never been run against the real checkpoint -- +that is 300-600 GiB and is owed, not done (spec `## Owed` O7). So the gate is +built to need no checkpoint at all: + + 1. A SYNTHETIC tiny-shape checkpoint, written here, with the real + architecture's topology at ~1/16 the width: the 45-layer KDA/DSA hybrid + becomes 4 layers, 288 experts become 4, and the FP8 e4m3 block form with + `weight_scale_inv` companions is reproduced exactly, because the dequant + path is the one part of the plumbing that silently produces a loadable + wrong file. + 2. An INDEPENDENT GGUF reader, written in this file, that shares no code with + the converter. A parser that reused the writer's structs would prove the + two agree, not that either is right. + 3. A FROZEN k-quant golden captured from the pinned llama.cpp reference + (`tests/scripts/fixtures/glm5_next_kquant_golden_b10451.json`, provenance in + the fixture). Q2_K, Q6_K and Q8_0 must come back BYTE-IDENTICAL. This is the + gate that matters most: a k-quant encoder that is close but not exact + produces a file that loads, generates fluent text, and is quietly worse than + the arm it claims to be. Bytes or nothing. + + **Three of the golden's six super-blocks are there because the first three + were BLIND.** Random weight-like data never lands on a rounding tie, so a + mutation replacing `nearest_int`'s round-half-to-EVEN with round-half-up + passed, and so did one replacing C `roundf`'s round-half-away-from-zero with + `np.rint`. Both are real defects and both were invisible. Blocks 3 and 4 + were SEARCHED on an eighth-lattice until the two rules disagreed under Q2_K + and Q6_K; block 5 is eight Q8_0 sub-blocks with `amax` exactly 127 so that + `x * id` lands on `.5`. The mutations fail now. A golden is only as strong + as the inputs it was captured over, and this one had to be measured rather + than assumed. + +The converter's OUTPUT is not yet loadable by this tree -- `glm5next` is not +registered, which W1 owns -- so this suite gates the converter as a command-line +path, entering through `main()` the way a user does. That gap is named in the +commit body, the pull request body, and the spec's `## Owed`. + +Exit 0 iff every case passes. +""" + +import base64 +import json +import os +import struct +import subprocess +import sys +import tempfile + +ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +CONVERTER = os.path.join(ROOT, "scripts", "convert-glm5-next-gguf.py") +FIXTURE = os.path.join(ROOT, "tests", "scripts", "fixtures", + "glm5_next_kquant_golden_b10451.json") + +try: + import numpy as np +except ImportError: + sys.stderr.write( + "SKIP: numpy is not importable here, so nothing about the converter was " + "verified. CI installs python3-numpy and runs this suite.\n") + raise SystemExit(0) + +FAILURES = [] + + +def check(cond, msg): + if cond: + print(" ok %s" % msg) + else: + print(" FAIL %s" % msg) + FAILURES.append(msg) + + +# --------------------------------------------------------------------------- +# The synthetic checkpoint. Shapes are the real config's relations at a small +# width, so a hard-coded dimension in the converter shows up as a shape error +# rather than passing by luck. + +HIDDEN = 256 +N_LAYERS = 4 +LAYER_TYPES = ["linear_attention", "linear_attention", + "deepseek_sparse_attention", "linear_attention"] +MLP_TYPES = ["dense", "sparse", "sparse", "sparse"] +N_EXPERTS = 4 +MOE_INTER = 256 +DENSE_INTER = 512 +VOCAB = 512 +HC_MULT = 4 +KDA_HEADS = 2 +KDA_HEAD_DIM = 128 +KDA_INNER = KDA_HEADS * KDA_HEAD_DIM +CONV_K = 4 +Q_LORA = 128 +KV_LORA = 64 +QK_NOPE = 64 +V_HEAD = 64 +N_HEADS = 4 +IDX_HEADS = 2 +IDX_HEAD_DIM = 64 +IDX_KPOOL = 4 +VIS_DEPTH = 2 +VIS_HIDDEN = 128 +VIS_INTER = 256 + + +def tiny_config(): + return { + "architectures": ["Glm5NextForConditionalGeneration"], + "model_type": "glm5_next", + "image_token_id": 154854, "video_token_id": 154855, + "image_start_token_id": 154830, "image_end_token_id": 154831, + "video_start_token_id": 154832, "video_end_token_id": 154833, + "tie_word_embeddings": False, + "quantization_config": {"activation_scheme": "dynamic", "fmt": "e4m3", + "quant_method": "fp8", + "weight_block_size": [128, 128], + "modules_to_not_convert": ["hyper_connection"]}, + "text_config": { + "model_type": "glm5_next_text", + "hidden_size": HIDDEN, "num_hidden_layers": N_LAYERS, + "intermediate_size": DENSE_INTER, "moe_intermediate_size": MOE_INTER, + "vocab_size": VOCAB, "max_position_embeddings": 4096, + "layer_types": LAYER_TYPES, "mlp_layer_types": MLP_TYPES, + "indexer_types": ["full"] * N_LAYERS, + "num_attention_heads": N_HEADS, "num_key_value_heads": N_HEADS, + "n_routed_experts": N_EXPERTS, "n_shared_experts": 1, + "num_experts_per_tok": 2, "n_group": 1, "topk_group": 1, + "norm_topk_prob": True, "scoring_func": "sigmoid", + "topk_method": "noaux_tc", "routed_scaling_factor": 2.5, + "first_k_dense_replace": 1, + "rms_norm_eps": 1e-05, "hc_eps": 1e-06, "hc_mult": HC_MULT, + "hc_sinkhorn_iters": 20, "mhc": True, "mla_use_nope": True, + "swiglu_limit": 10.0, "hidden_act": "silu", + "q_lora_rank": Q_LORA, "kv_lora_rank": KV_LORA, + "qk_head_dim": QK_NOPE, "qk_nope_head_dim": QK_NOPE, + "qk_rope_head_dim": 0, "v_head_dim": V_HEAD, "head_dim": 0, + "index_head_dim": IDX_HEAD_DIM, "index_n_heads": IDX_HEADS, + "index_topk": 2048, "index_kpool": IDX_KPOOL, + "index_kpool_compress": True, "index_kpool_always_select_tail": True, + "index_share_for_mtp_iteration": True, "indexer_rope_interleave": True, + "num_nextn_predict_layers": 1, "moe_router_dtype": "float32", + "linear_attn_config": {"num_heads": KDA_HEADS, + "gate_lower_bound": -5.0, + "head_dim": KDA_HEAD_DIM, + "short_conv_kernel_size": CONV_K, + "kda_layers": [0, 1, 3], + "full_attn_layers": [2]}, + }, + "vision_config": { + "model_type": "glm5_next_vision", "depth": VIS_DEPTH, + "hidden_size": VIS_HIDDEN, "intermediate_size": VIS_INTER, + "num_heads": 2, "patch_size": 14, "image_size": 448, + "spatial_merge_size": 2, "temporal_patch_size": 2, + "out_hidden_size": HIDDEN, "projection_intermediate_size": 512, + "in_channels": 3, "rms_norm_eps": 1e-05, "swiglu_limit": 10.0, + "attention_bias": True, + }, + } + + +def det(shape, seed): + """Deterministic values in a weight-like range, no RNG dependency.""" + n = int(np.prod(shape)) + i = np.arange(n, dtype=np.float64) + v = np.sin(i * 0.7391 + seed * 1.1) * 0.05 + np.cos(i * 0.113 + seed) * 0.01 + return v.astype(np.float32).reshape(shape) + + +def f32_to_e4m3(x): + """Encode f32 -> e4m3 bytes by nearest-value search over the 256 codes. + + Deliberately brute force and written here rather than shared with the + converter: the converter owns DECODE, this owns ENCODE, and the round trip + is only evidence when the two sides are independent. + """ + codes = np.zeros(256, np.float32) + for b in range(256): + s = -1.0 if (b >> 7) else 1.0 + e = (b >> 3) & 0xF + m = b & 0x7 + if e == 0xF and m == 0x7: + codes[b] = np.nan + elif e == 0: + codes[b] = np.float32(s * (m / 8.0) * (2.0 ** -6)) + else: + codes[b] = np.float32(s * (1.0 + m / 8.0) * (2.0 ** (e - 7))) + finite = np.where(np.isfinite(codes))[0] + tbl = codes[finite] + flat = np.asarray(x, np.float32).reshape(-1, 1) + idx = np.argmin(np.abs(flat - tbl.reshape(1, -1)), axis=1) + return finite[idx].astype(np.uint8).reshape(np.shape(x)), tbl[idx].astype( + np.float32).reshape(np.shape(x)) + + +def st_write(path, tensors): + """Minimal safetensors writer. `tensors` maps name -> (dtype, ndarray).""" + header = {} + blobs = [] + off = 0 + for name in sorted(tensors): + dt, arr = tensors[name] + raw = arr.tobytes() + header[name] = {"dtype": dt, "shape": list(arr.shape), + "data_offsets": [off, off + len(raw)]} + blobs.append(raw) + off += len(raw) + hb = json.dumps(header).encode("utf-8") + hb += b" " * ((-len(hb)) % 8) + with open(path, "wb") as fh: + fh.write(struct.pack("> np.uint32(16)) & np.uint32(1))) + >> np.uint32(16)).astype(np.uint16) + return r.reshape(arr.shape) + + +def fp8_pair(shape, seed, block=(128, 128)): + """An FP8 e4m3 block-quantized tensor plus its `weight_scale_inv` grid.""" + ref = det(shape, seed) + rows, cols = shape + sr = (rows + block[0] - 1) // block[0] + sc = (cols + block[1] - 1) // block[1] + scale = (det((sr, sc), seed + 3) * 0.5 + 1.0).astype(np.float32) + ri = np.minimum(np.arange(rows) // block[0], sr - 1) + ci = np.minimum(np.arange(cols) // block[1], sc - 1) + grid = scale[np.ix_(ri, ci)] + q, decoded = f32_to_e4m3(ref / grid) + return q, scale, (decoded * grid).astype(np.float32) + + +def build_checkpoint(dirpath, with_mtp=True): + """Write config.json and one safetensors shard. Returns the exact values the + converter must reproduce, keyed by HF tensor name.""" + cfg = tiny_config() + with open(os.path.join(dirpath, "config.json"), "w") as fh: + json.dump(cfg, fh) + t = {} + expect = {} + seed = [0] + + def add_bf16(name, shape): + seed[0] += 1 + v = det(shape, seed[0]) + b = bf16(v) + t[name] = (BF16, b) + expect[name] = (b.astype(np.uint32) << np.uint32(16)).view(np.float32) + + def add_f32(name, shape): + seed[0] += 1 + v = det(shape, seed[0]) + t[name] = (F32, v) + expect[name] = v + + def add_fp8(name, shape): + seed[0] += 1 + q, scale, ref = fp8_pair(shape, seed[0]) + t[name] = (FP8, q) + t[name[:-len(".weight")] + ".weight_scale_inv"] = (F32, scale) + expect[name] = ref + + p = "model.language_model." + add_bf16(p + "embed_tokens.weight", (VOCAB, HIDDEN)) + add_bf16(p + "norm.weight", (HIDDEN,)) + add_bf16("lm_head.weight", (VOCAB, HIDDEN)) + + layers = list(range(N_LAYERS)) + ([N_LAYERS] if with_mtp else []) + for L in layers: + b = "%slayers.%d." % (p, L) + add_bf16(b + "input_layernorm.weight", (HIDDEN,)) + add_bf16(b + "post_attention_layernorm.weight", (HIDDEN,)) + is_mtp = L == N_LAYERS + if not is_mtp: + # mHC lives FLAT on the layer and the MTP block carries NONE of it. + add_bf16(b + "hc_attn_fn", (HC_MULT * (HC_MULT + 2), HC_MULT * HIDDEN)) + add_f32(b + "hc_attn_base", (HC_MULT * (HC_MULT + 2),)) + add_f32(b + "hc_attn_scale", (HC_MULT - 1,)) + add_bf16(b + "hc_ffn_fn", (HC_MULT * (HC_MULT + 2), HC_MULT * HIDDEN)) + add_f32(b + "hc_ffn_base", (HC_MULT * (HC_MULT + 2),)) + add_f32(b + "hc_ffn_scale", (HC_MULT - 1,)) + kind = "deepseek_sparse_attention" if is_mtp else LAYER_TYPES[L] + if kind == "linear_attention": + for nm in ("q_proj", "k_proj", "v_proj"): + add_bf16(b + "self_attn.%s.weight" % nm, (KDA_INNER, HIDDEN)) + for nm in ("q_conv1d", "k_conv1d", "v_conv1d"): + add_bf16(b + "self_attn.%s.weight" % nm, (KDA_INNER, 1, CONV_K)) + add_bf16(b + "self_attn.f_a_proj.weight", (KDA_HEAD_DIM, HIDDEN)) + add_bf16(b + "self_attn.f_b_proj.weight", (KDA_INNER, KDA_HEAD_DIM)) + add_bf16(b + "self_attn.g_a_proj.weight", (KDA_HEAD_DIM, HIDDEN)) + add_bf16(b + "self_attn.g_b_proj.weight", (KDA_INNER, KDA_HEAD_DIM)) + add_bf16(b + "self_attn.b_proj.weight", (KDA_HEADS, HIDDEN)) + add_f32(b + "self_attn.A_log", (KDA_HEADS,)) + add_f32(b + "self_attn.dt_bias", (KDA_INNER,)) + add_bf16(b + "self_attn.o_norm.weight", (KDA_HEAD_DIM,)) + add_bf16(b + "self_attn.o_proj.weight", (HIDDEN, KDA_INNER)) + else: + add_fp8(b + "self_attn.q_a_proj.weight", (Q_LORA, HIDDEN)) + add_bf16(b + "self_attn.q_a_layernorm.weight", (Q_LORA,)) + add_fp8(b + "self_attn.q_b_proj.weight", (N_HEADS * QK_NOPE, Q_LORA)) + add_fp8(b + "self_attn.kv_a_proj_with_mqa.weight", (KV_LORA, HIDDEN)) + add_bf16(b + "self_attn.kv_a_layernorm.weight", (KV_LORA,)) + add_bf16(b + "self_attn.kv_b_proj.weight", + (N_HEADS * (QK_NOPE + V_HEAD), KV_LORA)) + add_fp8(b + "self_attn.o_proj.weight", (HIDDEN, N_HEADS * V_HEAD)) + add_bf16(b + "self_attn.indexer.wq_b.weight", + (IDX_HEADS * IDX_HEAD_DIM, Q_LORA)) + add_bf16(b + "self_attn.indexer.wk.weight", (IDX_HEAD_DIM, HIDDEN)) + add_bf16(b + "self_attn.indexer.k_norm.weight", (IDX_HEAD_DIM,)) + add_bf16(b + "self_attn.indexer.k_norm.bias", (IDX_HEAD_DIM,)) + add_bf16(b + "self_attn.indexer.weights_proj.weight", (IDX_HEADS, HIDDEN)) + add_bf16(b + "self_attn.indexer.index_kpool_compress_ape", + (IDX_KPOOL, IDX_HEAD_DIM)) + add_bf16(b + "self_attn.indexer.index_kpool_compress_gate", + (IDX_HEAD_DIM, HIDDEN)) + sparse = True if is_mtp else (MLP_TYPES[L] == "sparse") + if not sparse: + add_fp8(b + "mlp.gate_proj.weight", (DENSE_INTER, HIDDEN)) + add_fp8(b + "mlp.up_proj.weight", (DENSE_INTER, HIDDEN)) + add_fp8(b + "mlp.down_proj.weight", (HIDDEN, DENSE_INTER)) + else: + add_bf16(b + "mlp.gate.weight", (N_EXPERTS, HIDDEN)) + add_f32(b + "mlp.gate.e_score_correction_bias", (N_EXPERTS,)) + for e in range(N_EXPERTS): + add_fp8(b + "mlp.experts.%d.gate_proj.weight" % e, (MOE_INTER, HIDDEN)) + add_fp8(b + "mlp.experts.%d.up_proj.weight" % e, (MOE_INTER, HIDDEN)) + add_fp8(b + "mlp.experts.%d.down_proj.weight" % e, (HIDDEN, MOE_INTER)) + add_fp8(b + "mlp.shared_experts.gate_proj.weight", (MOE_INTER, HIDDEN)) + add_fp8(b + "mlp.shared_experts.up_proj.weight", (MOE_INTER, HIDDEN)) + add_fp8(b + "mlp.shared_experts.down_proj.weight", (HIDDEN, MOE_INTER)) + if is_mtp: + add_bf16(b + "enorm.weight", (HIDDEN,)) + add_bf16(b + "hnorm.weight", (HIDDEN,)) + add_bf16(b + "eh_proj.weight", (HIDDEN, 2 * HIDDEN)) + add_bf16(b + "shared_head.norm.weight", (HIDDEN,)) + + v = "model.visual." + add_bf16(v + "patch_embed.proj.weight", (VIS_HIDDEN, 3, 2, 14, 14)) + add_bf16(v + "patch_embed.proj.bias", (VIS_HIDDEN,)) + add_bf16(v + "post_layernorm.weight", (VIS_HIDDEN,)) + add_bf16(v + "downsample.weight", (HIDDEN, VIS_HIDDEN, 2, 2)) + add_bf16(v + "downsample.bias", (HIDDEN,)) + add_bf16(v + "merger.proj.weight", (HIDDEN, HIDDEN)) + add_bf16(v + "merger.gate_proj.weight", (512, HIDDEN)) + add_bf16(v + "merger.up_proj.weight", (512, HIDDEN)) + add_bf16(v + "merger.down_proj.weight", (HIDDEN, 512)) + add_bf16(v + "merger.post_projection_norm.weight", (HIDDEN,)) + add_bf16(v + "merger.post_projection_norm.bias", (HIDDEN,)) + for blk in range(VIS_DEPTH): + vb = "%sblocks.%d." % (v, blk) + add_bf16(vb + "norm1.weight", (VIS_HIDDEN,)) + add_bf16(vb + "norm2.weight", (VIS_HIDDEN,)) + add_bf16(vb + "attn.qkv.weight", (3 * VIS_HIDDEN, VIS_HIDDEN)) + add_bf16(vb + "attn.qkv.bias", (3 * VIS_HIDDEN,)) + add_bf16(vb + "attn.proj.weight", (VIS_HIDDEN, VIS_HIDDEN)) + add_bf16(vb + "attn.proj.bias", (VIS_HIDDEN,)) + add_bf16(vb + "attn.q_norm.weight", (VIS_HIDDEN // 2,)) + add_bf16(vb + "attn.k_norm.weight", (VIS_HIDDEN // 2,)) + add_bf16(vb + "mlp.gate_proj.weight", (VIS_INTER, VIS_HIDDEN)) + add_bf16(vb + "mlp.gate_proj.bias", (VIS_INTER,)) + add_bf16(vb + "mlp.up_proj.weight", (VIS_INTER, VIS_HIDDEN)) + add_bf16(vb + "mlp.up_proj.bias", (VIS_INTER,)) + add_bf16(vb + "mlp.down_proj.weight", (VIS_HIDDEN, VIS_INTER)) + add_bf16(vb + "mlp.down_proj.bias", (VIS_HIDDEN,)) + + names = st_write(os.path.join(dirpath, "model.safetensors"), t) + with open(os.path.join(dirpath, "model.safetensors.index.json"), "w") as fh: + json.dump({"metadata": {}, "weight_map": + {n: "model.safetensors" for n in names}}, fh) + return expect + + +# --------------------------------------------------------------------------- +# An independent GGUF reader. No code is shared with the converter. + +class Gguf: + _SC = {0: ("> 4).astype(np.float32)) + shift = np.array([0, 2, 4, 6], np.uint8).reshape(1, 1, 4, 1) + q = (qs.reshape(-1, 2, 1, 32) >> shift) & np.uint8(3) + q = q.reshape(-1, 16, 16).astype(np.float32) + out = dl.reshape(-1, 16, 1) * q - ml.reshape(-1, 16, 1) + return out.reshape(-1)[:n] + if ttype == 14: + # Transcribed from `dequantize_row_q6_K` (ggml-quants.c @ b10451): the + # scale index inside a 128-chunk is `l/16` plus 0/2/4/6 for the four + # quarters, and `sc` advances by 8 per chunk. Getting that stride wrong + # decodes a plausible tensor with the wrong scales. + b = np.frombuffer(blob, np.uint8).reshape(-1, 210) + nb = b.shape[0] + ql = b[:, 0:128].reshape(nb, 2, 64).astype(np.int32) + qh = b[:, 128:192].reshape(nb, 2, 32).astype(np.int32) + sc = b[:, 192:208].copy().view(np.int8).astype(np.float32).reshape(nb, 2, 8) + d = b[:, 208:210].copy().view(np.float16).astype(np.float32).reshape(nb, 1) + isx = (np.arange(32) // 16) + out = np.zeros((nb, 256), np.float32) + for half in range(2): + lo = ql[:, half, 0:32] + hi = ql[:, half, 32:64] + h = qh[:, half, :] + quarters = ( + ((lo & 0xF) | (((h >> 0) & 3) << 4), 0, 0), + ((hi & 0xF) | (((h >> 2) & 3) << 4), 32, 2), + ((lo >> 4) | (((h >> 4) & 3) << 4), 64, 4), + ((hi >> 4) | (((h >> 6) & 3) << 4), 96, 6), + ) + for q, dst, sofs in quarters: + s_l = sc[:, half, :][:, isx + sofs] + o = half * 128 + dst + out[:, o:o + 32] = (q.astype(np.float32) - 32.0) * s_l + return (out * d).reshape(-1)[:n] + + raise AssertionError("no decoder for ggml type %d" % ttype) + + +def run(args, cwd=None): + return subprocess.run([sys.executable, CONVERTER] + args, + capture_output=True, text=True, cwd=cwd) + + +# --------------------------------------------------------------------------- + +def case_kquant_golden(): + print("case: k-quant encoders are BYTE-IDENTICAL to llama.cpp b10451") + if not os.path.exists(FIXTURE): + check(False, "fixture %s is present" % FIXTURE) + return + fx = json.load(open(FIXTURE)) + sys.path.insert(0, os.path.join(ROOT, "scripts")) + import importlib.util + spec = importlib.util.spec_from_file_location("conv", CONVERTER) + conv = importlib.util.module_from_spec(spec) + spec.loader.exec_module(conv) + x = np.frombuffer(base64.b64decode(fx["input_f32_le_b64"]), np.float32) + for key, fn, width in (("q2_k", conv.quantize_q2_k, 256), + ("q6_k", conv.quantize_q6_k, 256), + ("q8_0", conv.quantize_q8_0, 32)): + want = np.frombuffer(base64.b64decode(fx[key + "_b64"]), np.uint8) + got = fn(x.reshape(-1, width)).reshape(-1) + check(want.shape == got.shape and bool((want == got).all()), + "%s matches the pinned reference byte-for-byte (%d bytes)" + % (key.upper(), want.size)) + check(fx["provenance"]["pin"] == "b10451", + "the golden names the pin it was captured at (b10451), not `master`") + + +def case_convert(tmp): + print("case: the converter emits a well-formed glm5next GGUF") + src = os.path.join(tmp, "ckpt") + os.makedirs(src, exist_ok=True) + expect = build_checkpoint(src) + dst = os.path.join(tmp, "out-q2k.gguf") + r = run(["--src", src, "--dst", dst, "--arm", "q2_k"]) + check(r.returncode == 0, "q2_k conversion exits 0 (rc=%d)\n%s" + % (r.returncode, r.stderr[-2000:] if r.returncode else "")) + if r.returncode != 0: + return None, expect + g = Gguf(dst) + check(g.version == 3, "GGUF version is 3") + check(g.kv.get("general.architecture") == "glm5next", + "general.architecture is `glm5next`") + check(g.kv.get("glm5next.block_count") == N_LAYERS, + "block_count is %d" % N_LAYERS) + check(g.kv.get("general.file_type") == 10 + and g.kv.get("general.quantization_arm") == "q2_k", + "general.file_type is the UINT32 `llama_ftype` MOSTLY_Q2_K = 10, not a " + "string, and the arm name travels beside it") + + print("case: the metadata carries the parameters the port hinges on") + check(abs(g.kv.get("glm5next.kda.gate_lower_bound", 0.0) - (-5.0)) < 1e-6, + "`glm5next.kda.gate_lower_bound` is -5.0 -- the value that selects the " + "sigmoid forget-gate branch over Kimi-Linear's softplus one") + check(g.kv.get("glm5next.kda.head_dim") == KDA_HEAD_DIM, + "`glm5next.kda.head_dim` is carried") + check(g.kv.get("glm5next.ssm.conv_kernel") == CONV_K, + "the short-conv kernel size is carried") + check(g.kv.get("glm5next.attention.indexer.kpool") == IDX_KPOOL, + "`indexer.kpool` is %d, the CHECKPOINT value, not the config class " + "default of 16" % IDX_KPOOL) + check(g.kv.get("glm5next.attention.indexer.top_k") == 2048, + "`indexer.top_k` is 2048") + check(g.kv.get("glm5next.hyper_connection.count") == HC_MULT, + "mHC stream count is carried") + check(abs(g.kv.get("glm5next.hyper_connection.epsilon", 0.0) - 1e-6) < 1e-12, + "`hc_eps` is 1e-6 and is a DIFFERENT constant from rms_norm_eps 1e-5") + check(abs(g.kv.get("glm5next.attention.layer_norm_rms_epsilon", 0.0) - 1e-5) + < 1e-11, "`rms_norm_eps` is 1e-5") + check(g.kv.get("glm5next.layer_types") == LAYER_TYPES, + "the AUTHORITATIVE per-layer schedule is carried as `layer_types`") + check("glm5next.rope.freq_base" not in g.kv + and g.kv.get("glm5next.rope.dimension_count") == 0, + "no rope frequency is written: the text stack is NoPE end to end") + check(g.kv.get("glm5next.expert_count") == N_EXPERTS + and g.kv.get("glm5next.expert_used_count") == 2, + "MoE routing width is carried") + check(abs(g.kv.get("glm5next.swiglu_clamp_exp", 0.0) - 10.0) < 1e-6, + "the clamped-SwiGLU limit is carried") + check(g.kv.get("glm5next.image_token_id") == 154854 + and g.kv.get("glm5next.video_token_id") == 154855, + "both multimodal token ids travel together") + return g, expect + + +def case_tensors(g): + print("case: the tensor set is exactly what the topology generates") + want = set() + want |= {"token_embd.weight", "output_norm.weight", "output.weight"} + for L in range(N_LAYERS): + want |= {"blk.%d.attn_norm.weight" % L, "blk.%d.ffn_norm.weight" % L} + for s in ("hc_attn_fn", "hc_attn_base", "hc_attn_scale", + "hc_ffn_fn", "hc_ffn_base", "hc_ffn_scale"): + want.add("blk.%d.%s.weight" % (L, s)) + if LAYER_TYPES[L] == "linear_attention": + for s in ("attn_q", "attn_k", "attn_v", "attn_output", + "ssm_conv1d_q", "ssm_conv1d_k", "ssm_conv1d_v", + "ssm_f_a", "ssm_f_b", "ssm_g_a", "ssm_g_b", + "ssm_beta", "ssm_norm"): + want.add("blk.%d.%s.weight" % (L, s)) + want |= {"blk.%d.ssm_a" % L, "blk.%d.ssm_dt" % L} + else: + for s in ("attn_q_a", "attn_q_a_norm", "attn_q_b", "attn_kv_a_mqa", + "attn_kv_a_norm", "attn_kv_b", "attn_output", + "indexer.attn_q_b", "indexer.attn_k", "indexer.k_norm", + "indexer.proj", "indexer_compressor_ape", + "indexer_compressor_gate"): + want.add("blk.%d.%s.weight" % (L, s)) + want.add("blk.%d.indexer.k_norm.bias" % L) + if MLP_TYPES[L] == "dense": + for s in ("ffn_gate", "ffn_up", "ffn_down"): + want.add("blk.%d.%s.weight" % (L, s)) + else: + for s in ("ffn_gate_inp", "ffn_gate_exps", "ffn_up_exps", + "ffn_down_exps", "ffn_gate_shexp", "ffn_up_shexp", + "ffn_down_shexp"): + want.add("blk.%d.%s.weight" % (L, s)) + want.add("blk.%d.exp_probs_b.bias" % L) + for s in ("v.patch_embd.weight", "v.patch_embd.bias", "v.post_ln.weight", + "v.downsample.weight", "v.downsample.bias", "v.merger.proj.weight", + "v.merger.gate.weight", "v.merger.up.weight", "v.merger.down.weight", + "v.merger.norm.weight", "v.merger.norm.bias"): + want.add(s) + for b in range(VIS_DEPTH): + for s in ("ln1.weight", "ln2.weight", "attn_qkv.weight", "attn_qkv.bias", + "attn_out.weight", "attn_out.bias", "attn_q_norm.weight", + "attn_k_norm.weight", "ffn_gate.weight", "ffn_gate.bias", + "ffn_up.weight", "ffn_up.bias", "ffn_down.weight", + "ffn_down.bias"): + want.add("v.blk.%d.%s" % (b, s)) + got = set(g.tensors) + check(got == want, "tensor set matches (%d expected). missing=%s extra=%s" + % (len(want), sorted(want - got)[:6], sorted(got - want)[:6])) + + print("case: the MTP block is dropped, following the reference") + mtp = [n for n in got if n.startswith("blk.%d." % N_LAYERS)] + check(not mtp, "no `blk.%d.*` tensor survives (found %s)" % (N_LAYERS, mtp[:4])) + check(not any("shared_head" in n or "enorm" in n or "eh_proj" in n for n in got), + "no MTP-only tensor (`enorm`, `hnorm`, `eh_proj`, `shared_head`) survives") + + print("case: the 288-expert lane is STACKED, not one tensor per expert") + for L in range(N_LAYERS): + if MLP_TYPES[L] != "sparse": + continue + info = g.tensors["blk.%d.ffn_gate_exps.weight" % L] + # ggml dims are reversed against the logical shape. + check(info["dims"] == [HIDDEN, MOE_INTER, N_EXPERTS], + "layer %d ffn_gate_exps is [ne0=%d, ne1=%d, ne2=%d experts]" + % (L, HIDDEN, MOE_INTER, N_EXPERTS)) + break + check(not any(".experts." in n for n in got), + "no per-expert tensor name leaks into the file") + + +def case_types(g): + print("case: the q2_k arm puts the EXPERTS at Q2_K and the rest at Q6_K") + exps = [n for n in g.tensors if n.endswith("_exps.weight")] + check(exps and all(g.tensors[n]["type"] == 10 for n in exps), + "all %d stacked expert tensors are Q2_K" % len(exps)) + check(g.tensors["blk.0.attn_q.weight"]["type"] == 14, + "a non-expert 2-D weight is Q6_K") + check(g.tensors["blk.0.attn_norm.weight"]["type"] == 0, + "a 1-D norm stays F32 rather than being quantized") + check(g.tensors["blk.0.ssm_a"]["type"] == 0, "`ssm_a` (A_log) stays F32") + print("case: a row that does not divide the block STEPS DOWN rather than " + "being reshaped") + check(g.tensors["blk.2.indexer_compressor_ape.weight"]["type"] == 8, + "the k-pool positional embedding (ne0=%d, not a multiple of 256) steps " + "down to Q8_0, whose block is 32" % IDX_HEAD_DIM) + check(g.tensors["blk.0.ssm_conv1d_q.weight"]["type"] == 0, + "the depthwise conv kernel (ne0=%d divides neither 256 nor 32) falls " + "all the way to F32" % CONV_K) + print("case: a convolution kernel with more axes than ggml carries is " + "flattened, not written") + # ggml has GGML_MAX_DIMS = 4. The patch embed is a Conv3d at + # `[hidden, 3, 2, 14, 14]` -- five axes -- and writing it verbatim would + # produce a header no reader can index. It is flattened to + # `[hidden, 3*2*14*14]`, whose row (1176) divides neither 256 nor 32, so it + # lands at F32. + check(len(g.tensors["v.patch_embd.weight"]["dims"]) == 2 + and g.tensors["v.patch_embd.weight"]["dims"] == [3 * 2 * 14 * 14, + VIS_HIDDEN], + "the 5-D patch-embed kernel is flattened to [%d, %d] rather than " + "written with five axes" % (3 * 2 * 14 * 14, VIS_HIDDEN)) + check(g.tensors["v.patch_embd.weight"]["type"] == 0, + "the flattened patch-embed row (1176) divides neither 256 nor 32, so " + "it lands at F32") + check(g.tensors["v.downsample.weight"]["dims"] == [VIS_HIDDEN * 2 * 2, HIDDEN] + and g.tensors["v.downsample.weight"]["type"] == 14, + "the 4-D downsample kernel is flattened to a row that DOES divide 256 " + "and is quantized rather than dumped at F32") + check(all(len(i["dims"]) <= 4 for i in g.tensors.values()), + "no tensor in the file carries more than ggml's 4 axes") + check(g.tensors["blk.0.ssm_conv1d_q.weight"]["dims"] == [CONV_K, 1, KDA_INNER] + and g.tensors["blk.1.ffn_gate_exps.weight"]["dims"] + == [HIDDEN, MOE_INTER, N_EXPERTS], + "3-D shapes are NOT flattened: the depthwise conv stays [k, 1, ch] and " + "the expert lane stays [m, n, experts], because ggml indexes both") + + +def case_values(g, expect): + print("case: FP8 e4m3 block dequant and bf16 widening reach the file intact") + p = "model.language_model." + name = "blk.0.attn_norm.weight" + ref = expect[p + "layers.0.input_layernorm.weight"].reshape(-1) + got = dequant(g.blob(name, nbytes_of(g.tensors[name])), 0, ref.size) + check(np.array_equal(got, ref), "an F32-carried bf16 norm is bit-exact") + + name = "blk.2.attn_q_a.weight" + info = g.tensors[name] + ref = expect[p + "layers.2.self_attn.q_a_proj.weight"].reshape(-1) + got = dequant(g.blob(name, nbytes_of(info)), info["type"], ref.size) + rel = np.abs(got - ref).max() / max(np.abs(ref).max(), 1e-30) + check(rel < 0.02, "an FP8 e4m3 block-quantized MLA projection round-trips " + "through Q6_K within 2%% (max rel %.4f)" % rel) + + name = "blk.1.ffn_gate_exps.weight" + info = g.tensors[name] + ref = np.concatenate([ + expect[p + "layers.1.mlp.experts.%d.gate_proj.weight" % e].reshape(-1) + for e in range(N_EXPERTS)]) + got = dequant(g.blob(name, nbytes_of(info)), info["type"], ref.size) + rel = np.abs(got - ref).max() / max(np.abs(ref).max(), 1e-30) + check(rel < 0.60, "the stacked expert lane round-trips through Q2_K " + "(max rel %.4f)" % rel) + # Expert ORDER, checked slab by slab rather than by a correlation over the + # whole tensor: a correlation stays high under a permutation of the slabs, + # and it cannot see a scale error at all. Every slab must be nearer its own + # expert than any other, which a swap breaks and a correlation does not. + per = ref.size // N_EXPERTS + refs = [expect[p + "layers.1.mlp.experts.%d.gate_proj.weight" % e].reshape(-1) + for e in range(N_EXPERTS)] + order_ok = True + for e in range(N_EXPERTS): + slab = got[e * per:(e + 1) * per] + errs = [float(np.abs(slab - r).mean()) for r in refs] + if int(np.argmin(errs)) != e: + order_ok = False + check(order_ok, "every expert slab is nearest its OWN expert: the lane is in " + "expert order, not permuted") + + +def case_refusals(tmp): + print("case: an unimplemented arm is REFUSED by name") + src = os.path.join(tmp, "ckpt") + for arm in ("iq2_xxs", "iq1_s", "iq2_s"): + r = run(["--src", src, "--dst", os.path.join(tmp, "x.gguf"), "--arm", arm]) + ok = (r.returncode != 0 and arm.upper() in r.stderr + and "importance matrix" in r.stderr and "circular" in r.stderr) + check(ok, "--arm %s is refused and names the missing part (imatrix) " + "and why it is unreachable here" % arm) + for arm in ("q4_k", "q3_k"): + r = run(["--src", src, "--dst", os.path.join(tmp, "x.gguf"), "--arm", arm]) + check(r.returncode != 0 and arm.upper() in r.stderr + and "encoder is not ported" in r.stderr, + "--arm %s is refused and names the unported encoder" % arm) + r = run(["--src", src, "--dst", os.path.join(tmp, "x.gguf"), + "--arm", "q2_k", "--keep-mtp"]) + check(r.returncode != 0 and "MTP" in r.stderr.upper() + and "O2" in r.stderr, + "--keep-mtp is refused and names the owed item that would carry it") + + print("case: a checkpoint of another architecture is REFUSED, not converted") + other = os.path.join(tmp, "other") + os.makedirs(other, exist_ok=True) + cfg = tiny_config() + cfg["model_type"] = "glm4_moe" + cfg["architectures"] = ["Glm4MoeForCausalLM"] + with open(os.path.join(other, "config.json"), "w") as fh: + json.dump(cfg, fh) + r = run(["--src", other, "--dst", os.path.join(tmp, "x.gguf"), "--arm", "q2_k"]) + check(r.returncode != 0 and "glm5_next" in r.stderr, + "a non-glm5_next config is refused by name") + + print("case: an FP8 tensor with no scale companion is REFUSED, not guessed") + broken = os.path.join(tmp, "broken") + os.makedirs(broken, exist_ok=True) + build_checkpoint(broken) + hdr_path = os.path.join(broken, "model.safetensors") + with open(hdr_path, "rb") as fh: + n = struct.unpack("