From 47b19b89b1a46def215e8c9fdca3e8dbc38acf05 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 13:11:11 +0000 Subject: [PATCH 1/3] spec(MODEL-MM-QWEN4-EXP): Qwen3.8-Flash-Next is a new architecture vLLM does not implement, so the port splits its oracle MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Qwen/Qwen3.8-Flash-Next` was released on 2026-08-24 and declares `Qwen4ExpForConditionalGeneration` / `model_type: qwen4_exp`. Nothing in this tree matched it. This lands the row's records and its spec; no product code. The name is misleading and the misreading is the expensive one. `Qwen3.8-27B` is, per `specs/qwen38-27b-bf16-gate.md`, the Qwen3.6-27B shape retrained with exactly one config key changed. That precedent does not extend here: the card calls this "the architecture that will underpin Qwen4", and it diverges from `qwen3_5` in four load-bearing places. **vLLM implements nothing.** Read live 2026-08-26 at `origin/main` = `6a5e8f5979`: no `qwen4*` path, no `registry.py` entry, and a repository-wide GitHub search for `qwen4` returns zero results. `vllm-omni` likewise. That is absence from vLLM `main`, not staleness in `555967922`, so advancing the parity pin does not reach it. What exists is transformers#48337 "Add Qwen4Exp model", merged 2026-08-26, and SGLang #36497, still open and therefore inadmissible. So the row runs a split oracle, on developer direction of 2026-08-26: transformers for the algorithm, vLLM ops for the optimized path. That is not a compromise. The transformers reference states its own limits in code — `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and carries the comment "we only allow eager and sdpa" — so porting it as written yields a correct model at an indefensible speed, while AGENTS.md's mirror-vLLM polarity still binds every primitive vLLM does implement. The spec resolves each component against exactly one of the two, and an implementer who cannot name the oracle for the line they are writing has found a gap in the spec rather than a licence to choose. The op survey is the substance of this change, and it moved two conclusions. **Most of the model is already here.** `Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the entire vision tower unchanged — `class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`. GDN is an exact hit on our Triton-AOT gate: the config's `128 / 128 / 16 / 48` against the specializations `cuda_gdn.cu` pins to `K=V=128, Hg=16, H in {48,32}`. Exactly two components have no vLLM op at all, and they are the two we must author ourselves: the PLE dilated depthwise conv (`git grep dilation` over vLLM's `layers/mamba/` returns nothing) and the hashed n-gram embedding. **QSA's twin in vLLM is MiniMax-M3, not DeepSeek-V4.** DSA is an MLA indexer; QSA is plain GQA, 24 Q heads over 2 KV, `head_dim` 256. `models/minimax_m3/common/indexer.py` is vLLM's non-MLA block-sparse case and its own docstring describes QSA's shape: scores KV blocks with index heads, selects top-k blocks, owns a side cache of one index-key vector per token. `common/ops/index_topk.py` supplies the block-score kernel and a bitonic top-k, and the pooled-key build is `deepseek_v4/nvidia/ops/sparse_attn_compress_cutedsl.py`'s `SparseAttnCompressNormRopeStoreC4Kernel`, which already carries `compress_ratio`. This tree has a working DSA indexer, which makes DSA the path an implementer reaches for first, and it is the wrong one. Recording that is most of why this spec exists. Two structural consequences beyond the module list, both of which touch code this row does not otherwise go near. The residual stream is `hc_count * hidden_size` = 4 x 2560 = 10240 wide through the whole stack, with a low-rank read gate and per-branch scalar write gate around both attention and MLP, so the per-layer loop and every residual buffer change. And `number_of_conv_states` is 3 on a PLE layer, plus the indexer side cache, which lands on the same KV-cache seam #1963 and #1966 are moving. **Nothing published fits.** Against roughly 119 GB usable on GB10: BF16 is ~360 GB, the official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB, and `unsloth/...-GGUF` is a README with zero weight files. No GGUF exists and no tool can produce one, because llama.cpp has no `qwen4_exp` either, so the standing k-quant requirement means authoring the architecture here and stating plainly that those arms have no llama.cpp oracle. The architecture supplies its own lever: the per-token n-gram cost is 16 lookups of 160 dims, so 51 GB of the 180 GB — 28% of the model — is a table touched 16 times per token, and making it non-resident is the intended design point rather than a workaround. RadixArk reached the same split independently. `gateable` is therefore `no`, on memory rather than software. Two decisions were put to the developer as explicit accept-or-reject, and both are settled and recorded in place rather than left open. The first is the oracle. `oracles/transformers.md` pins transformers to 5.14.1, tied to whatever the pinned vLLM environment resolves so that environment cannot hold two `transformers` at once, and 5.14.1 does not contain `Qwen4Exp`. The lane-scoped second pin is accepted, on the argument that the invariant guards a vLLM environment against drifting from its transformers and here there is no vLLM implementation to drift from, with expiry the moment vLLM registers `qwen4_exp`. It resolves to a real release rather than a branch SHA, which was not the expected outcome: the merge landed at 12:03:40Z on 2026-08-26 and `v5.16.0` published at 12:35:15Z. That was bounded rather than assumed — fetching `models/qwen4_exp/modeling_qwen4_exp.py` at each tag gives HTTP 200 at `v5.16.0` and HTTP 404 at `v5.15.0`, so 5.16.0 is the first release carrying the architecture and therefore the tightest pin available. The version string stays UNMEASURED: it is the release proven to contain the model, not a `transformers.__version__` read off a running oracle, and `gateable` stays `no`. The second is the first runnable arm: a Q4_K_M backbone with the n-gram table non-resident, about 76 GB. Q8_0 was raised and does not fit at ~191 GB, and no partial split reaches 119 GB while keeping the backbone at 8 bits. Q4_K_M throughout fits on paper at ~109 GB but leaves roughly 10 GB for KV and activations on a model with 262144 native context, which is not a margin. This promotes the non-resident table from a note to a first-class W6 deliverable and splits that wave in two, because it is not free: GB10 is unified memory, so the existing host-pinned offload seam does not by itself solve it there and the mechanism has to be disk-backed or genuinely unloaded. W6b is the unknown, and the spec says to spike it before scheduling rather than design around an unmeasured cost. The `MODEL` row ratchet moves 377 -> 378 with its justification appended in the existing log format, re-derived off the matrix rather than carried forward. One row and not two: the MTP head is a `mtp` block inside the same text config, not a separately registered architecture, so this is not the IndexTTS-2.5 or dots3-note shape that moved the pin by two. The at-the-pin static invariants (324/373/356/310/261) are unchanged, because the Upstream cell carries no pinned module/class target. Tracked by #1978. Gates: `check-agent-record` ok, `check-model-checklist` ok, `tests/scripts/test_agent_record.py` + `test_check_model_checklist.py` 130 passed, `agent-preflight.sh --staged` ok. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/issue-index.md | 1 + .agents/model-matrix.md | 8 +- .agents/oracles/transformers.md | 88 +++++ .agents/specs/qwen4-exp-flash-next.md | 465 ++++++++++++++++++++++++++ scripts/check-agent-record.py | 18 +- 5 files changed, 576 insertions(+), 4 deletions(-) create mode 100644 .agents/specs/qwen4-exp-flash-next.md diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 3ebd1c7e7..c5c4b9c82 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -744,3 +744,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1762](https://github.com/mudler/vllm.cpp/issues/1762) | `GEMMA4-FP8-WMMA-EXPERT-GEMM` | KEEP Gemma-4 FP8 T>1 expert path still dequantizes to BF16 and calls hipBLAS Tensile; a gated gfx1201 FP8 WMMA expert GEMM is the unblocked L2 lever (31.5% prefill). Spec-first, default-OFF, no GPU on this filing | perf | | [#526](https://github.com/mudler/vllm.cpp/issues/526) | `SERVE-TOOL-HISTORY-ARGS` | OpenAI multi-turn tool history reaches chat templates with string-valued arguments | bug | | [#1934](https://github.com/mudler/vllm.cpp/issues/1934) | `BACKEND-ROCM` | `RocmPlatform::needs_weight_staging()` is stale-false (a W0-era placeholder never revisited despite #523/#509/#506/ROCM_ATTN/hipGraph landing since), so `CheckDeviceWeightFit` — the #1123/#1870 load-time refusal, including the `policy_forces_full_expand` fix — never runs on ROCm: measured directly, `VT_DEVICE_WEIGHT_BUDGET_BYTES=1` produced no refusal on a real load. The actual device allocation the refusal guards is not gated on this flag, so #1870's crash stays reachable until this closes; owed, not fixed in flow, because flipping the flag also moves `DirectDeviceLoadEligible` and several GDN kernel-dispatch defaults that each need their own correctness check | bug | +| [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-QWEN4-EXP` | **`Qwen/Qwen3.8-Flash-Next` declares `Qwen4ExpForConditionalGeneration` / `qwen4_exp`, a new architecture vLLM does not implement, so the port runs on a split oracle: transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH.** Released 2026-08-24, 180B total / 6B activated, image-text-to-text. The `Qwen3.8` in the name is marketing continuity: `.agents/specs/qwen38-27b-bf16-gate.md`'s "one config key differs" precedent does NOT extend here. Read live 2026-08-26 at vLLM `origin/main` = `6a5e8f5979`: no `qwen4*` path, no registry entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. That is absence from vLLM `main` rather than staleness in our pin `555967922`, so a pin advance does not reach it. What exists is transformers [#48337](https://github.com/huggingface/transformers/pull/48337) "Add Qwen4Exp model", MERGED 2026-08-26, 5211 lines, and SGLang [#36497](https://github.com/sgl-project/sglang/pull/36497), still OPEN and therefore inadmissible. **Developer direction 2026-08-26, recorded verbatim: "use transformers as oracle for algorithmic side. but use ops from vllm so we account for optimized path."** Justified rather than convenient: `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so porting it as written yields a correct model at an indefensible speed, while AGENTS.md's mirror-vLLM polarity still binds every primitive vLLM implements. `Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`), all of which this tree has; GDN is an exact match for our AOT gate (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). The delta is four things, and **exactly two have no vLLM op at all**: the PLE dilated depthwise conv (kernel 4, dilation 3; `git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the n-gram hashed embedding. **The survey's load-bearing finding: QSA's structural twin in vLLM is MiniMax-M3, NOT DeepSeek-V4.** DSA is an MLA indexer; QSA is plain GQA (24 Q / 2 KV, `head_dim` 256, `partial_rotary_factor` 0.25). `models/minimax_m3/common/indexer.py` is vLLM's non-MLA block-sparse case and its own docstring describes QSA's shape — scores KV blocks with index heads, selects top-k blocks, owns a side cache of one index-key vector per token — with `common/ops/index_topk.py` supplying `_index_block_score_kernel` and a bitonic `_topk_index_kernel`; the pooled-key build (mean-pool, `k_layernorm`, RoPE at block start) is `deepseek_v4/nvidia/ops/sparse_attn_compress_cutedsl.py`'s `SparseAttnCompressNormRopeStoreC4Kernel`, which carries `compress_ratio`. Building QSA on the DSA/MLA path is the wrong port and is the one this tree reaches for first, because it already has DSA. Two structural consequences beyond the module list: the residual stream is `hc_count * hidden_size` = **4 x 2560 = 10240 wide through the whole stack** with a low-rank read gate and per-branch scalar write gate around both attention and MLP, which is a change to the per-layer loop and every residual buffer rather than a drop-in module; and `number_of_conv_states = 3` on a PLE layer (GDN conv, PLE conv, n-gram token history) plus the indexer side cache, adjacent to [#1963](https://github.com/mudler/vllm.cpp/issues/1963) and [#1966](https://github.com/mudler/vllm.cpp/issues/1966). **NOTHING PUBLISHED FITS**, read live from the HF API against ~119 GB usable on GB10: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB (NVFP4 backbone with the n-gram table left at FP8, 51.2 GB) and `unsloth/...-GGUF` is a README with ZERO weight files. No GGUF exists and no tool can make one, because llama.cpp has no `qwen4_exp` either, so the standing k-quant requirement means authoring the arch on our side AND states that the quantized arms have NO llama.cpp oracle. The architecture supplies its own lever: the per-token n-gram cost is `(ngram_size-1)*heads_per_ngram` = 16 lookups of 160 dims, so **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per token** and making it non-resident is the intended design point (RadixArk reached the same split independently). Sizing arithmetic, NOT measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB (yes, ~10 GB left for KV), Q4_K_M backbone with the table non-resident ~76 GB. GB10 is UNIFIED memory so "offload to host" is not a move there; non-resident means disk-backed, and its cost is unmeasured. **Two decisions were put to the developer as explicit accept-or-reject and BOTH are settled 2026-08-26, recorded in place rather than left open.** (1) `.agents/oracles/transformers.md` pins transformers to 5.14.1, deliberately tied to what the pinned vLLM environment resolves so the environment cannot hold two `transformers` at once, and **5.14.1 does not contain `Qwen4Exp`**; the lane-scoped second pin is **ACCEPTED**, on the argument that the invariant guards a vLLM environment against drifting from its transformers and here there is no vLLM implementation to drift from, and it expires the moment vLLM registers `qwen4_exp`. **The lane pin is a real release, not a branch SHA**, which was not the expected outcome: `Qwen4Exp` merged to `main` at 12:03:40Z on 2026-08-26 and `v5.16.0` published at 12:35:15Z, and this was BOUNDED rather than assumed by fetching `models/qwen4_exp/modeling_qwen4_exp.py` at each tag — `v5.16.0` HTTP **200**, `v5.15.0` HTTP **404** — making 5.16.0 the FIRST release carrying the architecture and therefore the tightest available pin. The version string is UNMEASURED (it is the release proven to contain the model, not a `transformers.__version__` read off a running oracle) and `gateable` stays `no`. (2) The first runnable arm is the **Q4_K_M backbone with the n-gram table NON-RESIDENT** (~76 GB). Q8_0 was raised and does not fit at ~191 GB, and no partial-Q8 split reaches 119 GB with the backbone at 8 bits; Q4_K_M-throughout fits on paper at ~109 GB but leaves ~10 GB for KV and activations on a 262144-native-context model, which is not a margin. This promotes the non-resident table from a note to a first-class W6 deliverable, and it is NOT free: GB10 is UNIFIED memory, so the existing host-pinned offload seam (`ENG-WEIGHT-OFFLOAD`, mirroring vLLM's `cpu_offload_gb`) does not by itself solve it there and the mechanism must be disk-backed or genuinely unloaded — established before it is designed around. Spec: [`specs/qwen4-exp-flash-next.md`](specs/qwen4-exp-flash-next.md). No product code lands under the spec pull request | feature | diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 294d16930..766a3bed7 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -81,14 +81,15 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): | SPIKE | 9 | | BLOCKED | 5 | | DONE | 3 | -| READY | 3 | +| READY | 4 | | GATING | 1 | -| **Total** | **377** | +| **Total** | **378** | -Engaged architectures (the 53 non-`INVENTORIED` rows): +Engaged architectures (the 54 non-`INVENTORIED` rows): | Support | Architecture | Family / example | Status | Row | |---|---|---|---|---| +| 🚧 | `Qwen4ExpForConditionalGeneration` | Qwen3.8-Flash-Next (180B total / 6B activated, image-text-to-text) | **SPEC ONLY.** New architecture vLLM does NOT implement at any revision; split oracle by developer direction 2026-08-26 (transformers for the algorithm, vLLM ops for the optimized path). Nothing published fits ~119 GB on GB10, so `gateable = no` and the quantized arms are load-bearing; no GGUF exists and llama.cpp has no `qwen4_exp` either. NO token, NO speed. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | | ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling scope CLOSED (#323): `60e71a0e` fixed the eager path; `DenseDecodeGraphForward` ran first and replayed against stale HOST ids, so it now declines while the mirror is live and falls back to the proven eager path. Async gate 7/7 across Qwen3-0.6B/4B + Llama/Mistral/InternLM2 | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` | | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` | | ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving. **SECOND CHECKPOINT TOKEN-GATED 2026-08-15 (`Qwen/Qwen3.8-27B`@`1d4bf0f2`, bf16, [#915](https://github.com/mudler/vllm.cpp/issues/915), [spec](specs/qwen38-27b-bf16-gate.md)): 4/7 prompts STRICT 16/16 vs the pinned oracle `555967922`, and all THREE first-divergence positions are EXACT fp32 TIES** — oracle-minus-ours and top-2 gap both **0.000 mnats**, our token at rank **3 / 2 / 2** in the oracle top-20, so `ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. Every one is the [#910](https://github.com/mudler/vllm.cpp/issues/910) tie-break signature and nothing else: vLLM's pick carries the LOWER token id (1814/11/16309) and ours the HIGHER (22960/13/27180) at a bit-identical logprob. Only the first divergence per prompt is adjudicable, so this is three numbers; a raw position count over the grid is NOT a quality score and is not recorded as one. Adjudicated twice on the pinned oracle's fp32 logprobs — a greedy re-decode and an independent TEACHER-FORCED probe that asserts the echoed prefix — because the earlier `transformers` bf16 CPU probe could not resolve below one bf16 ULP (every runner-up gap it printed was a multiple of 0.125) and so could not have reported anything but a tie. **SPEED on the same checkpoint, vs vLLM's PRODUCTION graphed config at the pin, clocks 2184 MHz: 1 of 3 concurrency cells established.** c4 is the only cell where both arms completed every request — **0.963x** output throughput, **1.008x** median ITL. c1 and c8 throughput WERE withheld on 2026-08-15 (superseded, below): our server failed 1/6 in all three reps and 12/11/12 of 48 where vLLM failed none in nine legs ([#931](https://github.com/mudler/vllm.cpp/issues/931)), and `output_throughput` divides tokens by a duration still containing the dead request, so c1 read 0.677x while median TPOT in the SAME file read 1.014x in our favour. **SUPERSEDED 2026-08-19 by the c1/c8 RE-MEASURE ([#915](https://github.com/mudler/vllm.cpp/issues/915), [#979](https://github.com/mudler/vllm.cpp/issues/979), `.agents/benchmark-record.md` `BENCH-QWEN38-27B-BF16 c1/c8 RE-MEASURE`):** #931 landed, and with `VT_SERVER_SSE_PING_S=0` our arm completed **162 of 162** requests, `failed=0` on every leg — c1 **4.4040 tok/s** (CV 0.039%), c8 **22.6402 tok/s** (CV 0.205%). **Our half of the withholding is discharged; NEITHER cell became a ratio and the two halves are blocked differently.** At c1 vLLM also completed everything (**4.2835 tok/s**, CV 0.033%) and `gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three pairings — the cross-arm rule PASSED (same boot, both arms 2489 MHz median, 0.0% offset) and the WITHIN-RUN rule failed on both against the 5% ceiling ([#1354](https://github.com/mudler/vllm.cpp/issues/1354): clocks cannot be pinned inside an `rc` lease), so the c1 ratio is OWED, not withheld for being unflattering. At c8 the vLLM denominator is **NOT MEASURABLE on this box at the recorded configuration** — that is the answer, not a gap, and not a claim that vLLM is defective. Read the two output-throughput absolutes with [#1355](https://github.com/mudler/vllm.cpp/issues/1355): our `usage.prompt_tokens` reports 5,942 where vLLM reports 6,144 on identical prompts, which corrupts total-token throughput outright and biases output throughput up by more than its own CV. Cold start **53 s vs 780 s = 14.7x**; host memory after warmup **42.5 vs 110.1 GiB = 2.59x**, caveated because vLLM's is set by `--gpu-memory-utilization 0.85` pre-reserving KV | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | @@ -509,6 +510,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-qwen3-vl-moe-qwen3-vlmoe-for-conditional-generation` | `Qwen3VLMoeForConditionalGeneration` | `registry.py:552-555`; `vllm/model_executor/models/qwen3_vl_moe.py::Qwen3VLMoeForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder; video path | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | `Qwen3_5ForConditionalGeneration` | `registry.py:556`; `vllm/model_executor/models/qwen3_5.py::Qwen3_5ForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; GDN/linear-attention state; vision encoder; video path | 🚧 [family scoping](specs/mm-tools-scoping-2026-07-10.md); [plain-BF16 loader leaf](specs/qwen35-plain-bf16-direct-load.md); **[multimodal-track W-plan](specs/multimodal-track.md)**; full target spec required | `PARTIAL` (text-only) | text-only: `include/vllm/model_executor/models/qwen3_5_dense.h:40-105,146-171`; plain BF16/F32 + stacked/tied load `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:52-133,187-246,334-472`; plain execution `src/vllm/model_executor/models/qwen3_5.cpp:1993-2003,4579-4585,5249-5255,5533-5545`; loader route/queue reuse `src/vllm/entrypoints/model_loader.cpp:364-400`; real 4B gate `tests/vllm/models/test_qwen35_plain_weights.cpp:80-196`: CPU topology/load **1656/1656**, AOT CUDA direct OFF/ON full-engine token equivalence **1664/1664**. Existing W3-G immutable `ae9e8ff` default/fallback each pass **235/235 + 16/16** with the frozen 64 plans. Corrected root `/tmp/qwen35-transplant-4b-aot-557ab41d` proves ON=OFF 128/128 and records ON/OFF/vLLM total **6155.10/6064.06/6730.46 tok/s**, peak PSS **2.405/8.571/7.569 GiB**; current ON is 0.9316x historical AOT ON. Current-v0.25 oracle, sanitizer, vision, strict VRAM and external 27B/35B regressions remain unverified, with no new support claim. **MM-completion plan ([multimodal-track.md](specs/multimodal-track.md), `CLAIM-MULTIMODAL-TRACK`, 2026-07-25):** modalities = image + video (NO audio); reuses the landed GDN-hybrid text path — the mm half is the shared `Qwen3_VisionTransformer` (DeepStack, `qwen3_vl.py:519`) stood up on Qwen3-VL-4B first (M2) then attached to this wrapper (M3). Oracle-runnable (0.25.0 ships `qwen3_5.py`+`qwen3_vl.py`); NOT HW/oracle-blocked but **CHECKPOINT-gated** — the cached `unsloth/Qwen3.6-27B-NVFP4` quant is TEXT-ONLY (2111 tensors, ZERO `visual.*`; `vision_config` declared but weights absent), so a vision-inclusive checkpoint download is required (M0). Tower ~0.5-0.7 B params (~1-1.4 GiB bf16) fits GB10 trivially alongside the 27B. Plan owner `CLAIM-MULTIMODAL-TRACK` (row stays PARTIAL/narrative-only; the mm work re-claims it at M3). **M3-b LANDED 2026-07-25 (`CLAIM-MULTIMODAL-M3B`): IMAGE e2e WORKING** — `Qwen3_5VLGenerateGreedy` (`src/vllm/model_executor/models/qwen3_5.cpp`) forks the GDN-hybrid forward on inputs_embeds(scatter tower merger `[196,5120]` into image_token 248056 rows, no deepstack) + 3-section MRoPE `[11,11,10]` interleaved in the 16 full-attn layers (host `BuildMropeCosSinHost` → the `mrope_cos_sin` param on `DenseForwardLayers`, nullptr on text ⇒ byte-identical); vision loader `LoadQwen3VLVisionWeights` (`src/vllm/model_executor/models/qwen3_vl.cpp`, 27B config) + M2a tower + `LoadQwen3_5Dense` bf16 LLM. STRICT gate `tests/vllm/multimodal/test_qwen3_5_vl_e2e.cpp` **32/32 token-exact vs vLLM 0.25.0** (sha256 `ead4b484…`); text-inertness re-run cutlass-ON 27B/35B/Coder **235/315/138**. **M3d LANDED 2026-07-25 (`CLAIM-MULTIMODAL-M3D`): VIDEO e2e WORKING** — `Qwen3_5VLGenerateGreedyVideo` (`src/vllm/model_executor/models/qwen3_5.cpp`) reuses the M3-b image driver via a shared `VLGenerateCoreGdn` (video merge mask on video_token 248057 + `Qwen3VLGetRopeIndexVideo` per-frame temporal MRoPE; M3c processor/windowed-tower reused verbatim; no deepstack). STRICT gate `tests/vllm/multimodal/test_qwen3_5_vl_video_e2e.cpp` **32/32 token-exact vs vLLM 0.25.0** (oracle `scripts/mm/m3d_video_oracle_capture.py`, K=5 deterministic, near-tie gaps 0.0000); image e2e re-run STRICT 32/32 (refactor-safe); text SACRED byte-identical by construction (shared forward untouched). **Qwen video modalities COMPLETE (image+video e2e; audio N/A); speed still pending** (row stays PARTIAL). Speed lever #2 CLOSED 2026-07-27 (`CLAIM-MULTIMODAL-SPEED-DECODE`, multimodal-speed.md §8): on-GPU greedy argmax + decode embed round-trip removed on the shared `VLGenerateCoreGdn`; bit-exact (image+video STRICT 32/32 held, goldens md5-identical); 27B decode TPOT NEUTRAL (223 ms, ~222 ms bandwidth floor, at vLLM parity). **Speed lever #3 FIRST BRICK 2026-07-27 (`CLAIM-MULTIMODAL-SPEED-GRAPH`, multimodal-speed.md §9): the shared `VLGenerateCoreGdn` decode step now routes through the production `Qwen3_5DenseDecodeGraph` (cold→warm→replay captured decode) — mm decode is GRAPH-CAPTURABLE (was eager per-step). S==B==1 bit-identical rebuild; the decode-time 1-D device RoPE at p reproduces the degenerate MRoPE {p,p,p} → token-exact HELD (image+video STRICT 32/32, 30 graph replays confirmed); A/B graphed 232.5 vs eager 233.4 ms/tok = NEUTRAL at the 27B bandwidth floor. Structural gap closed; W-plan = Voxtral decode-graph (audio 1.52× gap-closer) + batched c2+ + serving ingestion. Row stays PARTIAL/speed-pending.** | unassigned | | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` | `Qwen3_5MoeForConditionalGeneration` | `registry.py:557-560`; `vllm/model_executor/models/qwen3_5.py::Qwen3_5MoeForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; GDN/linear-attention state; vision encoder; video path | 🚧 [family scoping](specs/mm-tools-scoping-2026-07-10.md); **[multimodal-track W-plan](specs/multimodal-track.md)**; target spec required | `PARTIAL` (text gated, vision NOT gated) | text-only: `include/vllm/model_executor/models/qwen3_5.h:1-17,97`; direct registry `src/vllm/model_executor/models/registry.cpp:10-20`; gate `tests/parity/test_qwen36_paged_engine.cpp:78,140`. W3-G immutable `ae9e8ff` correctness-only ratio-8 inertness passes **2/2 + 315/315**; no 35B performance claim; vision not implemented AT THAT DATE (it landed later, see M2/M3 below, and is still NOT gated). Disk load now DEFERS the routed-expert host copies and streams+frees them per layer during `PrepareMarlinResident` to bound load-phase peak PSS (`ENG-MOE-LOADSTREAM`, engine-matrix; CPU-gated, DGX pending) — device residents byte-identical. **MM-completion plan ([multimodal-track.md](specs/multimodal-track.md), `CLAIM-MULTIMODAL-TRACK`, 2026-07-25):** image + video (NO audio); same shared `Qwen3_VisionTransformer` as the 27B row, attached to the landed MoE GDN-hybrid text path (M3). Oracle-runnable (0.25.0); **CHECKPOINT-gated** — the cached `nvidia/Qwen3.6-35B-A3B-NVFP4` quant is TEXT-ONLY (`vision_config` declared, `visual.*` weights absent); vision-inclusive download required (M0). Tower fits GB10 alongside the 35B MoE per the landed text run. Plan owner `CLAIM-MULTIMODAL-TRACK` (row stays PARTIAL/narrative-only; the mm work re-claims it at M3). **M2/M3 LANDED (#891, `.agents/specs/moe-vision-tower.md`):** the loader no longer drops the checkpoint's 333 `model.visual.*` tensors (`LoadQwen3_5MoeVision` -> the SHARED `LoadQwen3VLVisionWeights` the dense arm is gated on; their ABSENCE is refused by name), and `Qwen3_5MoeVLGenerateGreedy[Video]` forks the forward gated on mm input over a greedy core now TEMPLATED on the weights arm rather than copied. Evidence: CPU suite 479/479 serial; the new `test_qwen3_5_moe_vision` proves the forked forward reduces EXACTLY to the text forward over the tower row (one visual token, 1x1x1 LLM grid) and that MRoPE is applied (8x8 grid must DIFFER from the 1-D run), with 4 mutations driven RED and restored byte-exact; on Thor (sm_110, FALLBACK attention) `test_qwen3_5_moe_vision_hw` loads the real 333 tensors and runs the tower on the fixture image. **OWED: the binding image and video token-exact gates vs the pinned oracle at 35B.** Not runnable on Thor -- vLLM cannot import there (`libcuda.so.1` absent on the host, `torch.cuda.is_available()` False) and the bf16 35B is ~67 GiB against this box's documented 25 GB single-model reboot ceiling; dgx.casa was off-limits mid-run for a sibling row. **TEXT ARM ORACLE-GATED ON THE PUBLISHED BF16 REPO 2026-08-15 ([#740](https://github.com/mudler/vllm.cpp/issues/740) + [#864](https://github.com/mudler/vllm.cpp/issues/864)), and this changes NOTHING about the vision claim:** greedy 7 prompts x 3 repeats x 16 tokens on `Qwen/Qwen3.6-35B-A3B` bf16 @`995ad96eacd98c81ed38be0c5b274b04031597b0` vs the pinned oracle gave **6/7 prompts STRICT 16/16**, the seventh one exact logit tie (`top2_gap_mnats = 0.0`) our on-device argmax breaks toward the higher id ([#910](https://github.com/mudler/vllm.cpp/issues/910)); only the FIRST divergence per prompt is adjudicable, so the raw 108/112 position count is NOT a quality score. SACRED inertness 3/3, goldens byte-identical (27B 235/235, 35B 315/315, Coder 138/138). NO throughput, latency or memory number exists for this checkpoint. The row therefore stays PARTIAL: **the binding image and video token-exact gates at 35B are still OWED**, the vision claim remains "the tower loads and computes" rather than "produces correct tokens", and the sm_110 run that proved it used the FALLBACK attention path, which is not coverage of the shipped GB10 path. [#908](https://github.com/mudler/vllm.cpp/issues/908)'s dense regression check is PARTIAL too: dense TEXT is 235/235 at `2f2bce926`, a true before/after (binary md5 `db889909d4…` vs `49ded1ece8…`, 500 TUs recompiled), while dense image/video stays UNVERIFIED (network-blocked) | unassigned | +| `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | `Qwen4ExpForConditionalGeneration` (`model_type: qwen4_exp`; campaign row `MODEL-MM-QWEN4-EXP`) | **NOT IN vLLM AT ANY REVISION** — deliberately written with no pinned module/class target, the convention `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` follows for a beyond-pin arm, and stronger here: this is absence from vLLM `main` rather than staleness in `555967922`. Read live 2026-08-26 at `origin/main` = `6a5e8f5979`: no `qwen4*` path, no `registry.py` entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. Algorithm source is [transformers#48337](https://github.com/huggingface/transformers/pull/48337) `models/qwen4_exp/modular_qwen4_exp.py`, MERGED 2026-08-26 | conditional generation / text + image + video | MM processor; vision encoder (UNCHANGED from `Qwen3_5MoeVisionModel`); GDN/linear-attention state; block-sparse attention + indexer side cache; FusedMoE/grouped GEMM; hyper-connection residual streams; hashed n-gram embedding; dilated depthwise conv; MTP | ✅ [Qwen3.8-Flash-Next](specs/qwen4-exp-flash-next.md) | `READY` | **SPEC ONLY, NO PRODUCT CODE, NO TOKEN, NO SPEED.** `Qwen/Qwen3.8-Flash-Next` (2026-08-24, 180B total / 6B activated). Split oracle by developer direction 2026-08-26: **transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH**, because `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so the reference is semantics and not a serving path. `Qwen4ExpTextModel` inherits `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`); GDN matches our AOT gate exactly (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). **Exactly two components have NO vLLM op**: the PLE dilated depthwise conv (`git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the hashed n-gram embedding. **QSA's twin in vLLM is MiniMax-M3, NOT DeepSeek-V4** — DSA is MLA, QSA is plain GQA (24 Q / 2 KV, `head_dim` 256); `models/minimax_m3/common/indexer.py` is the non-MLA block-sparse case with `common/ops/index_topk.py`'s `_index_block_score_kernel` + bitonic `_topk_index_kernel`, and the pooled-key build is `deepseek_v4/nvidia/ops/sparse_attn_compress_cutedsl.py`'s `SparseAttnCompressNormRopeStoreC4Kernel`. Residual stream is `hc_count * hidden_size` = 4 x 2560 = **10240 wide through the whole stack**; `number_of_conv_states = 3` on a PLE layer plus the indexer side cache. **NOTHING PUBLISHED FITS ~119 GB on GB10**: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB, `unsloth/...-GGUF` is a README with ZERO weight files and llama.cpp has no `qwen4_exp` either, so the standing k-quant arms must be authored here and have NO llama.cpp oracle. Sizing ARITHMETIC and not measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB, Q4_K_M backbone with the 51 GB n-gram table non-resident ~76 GB — that table is 28% of the model and is touched 16 times per token, which is the offload the card itself argues for. `gateable = no` until an arm runs. Both blocking decisions SETTLED 2026-08-26: the transformers lane pin is **ACCEPTED at 5.16.0** ([`oracles/transformers.md`](oracles/transformers.md)), because the registry pin 5.14.1 does not contain `Qwen4Exp` — and 5.16.0 is a real release rather than a SHA, bounded by fetching the model file at each tag (`v5.16.0` HTTP 200, `v5.15.0` HTTP 404), with the version string UNMEASURED until an oracle stands up; and the first runnable arm is the **Q4_K_M backbone with a NON-RESIDENT n-gram table** (~76 GB), Q8_0 having been raised and rejected on ~191 GB. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | unassigned | | `MODEL-MM-rvl-rfor-conditional-generation` | `RForConditionalGeneration` | `registry.py:561`; `vllm/model_executor/models/rvl.py::RForConditionalGeneration` | conditional generation / video+image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-skyworkr1v-skywork-r1-vchat-model` | `SkyworkR1VChatModel` | `registry.py:562`; `vllm/model_executor/models/skyworkr1v.py::SkyworkR1VChatModel` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-smolvlm-smol-vlmfor-conditional-generation` | `SmolVLMForConditionalGeneration` | `registry.py:563`; `vllm/model_executor/models/smolvlm.py::SmolVLMForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/oracles/transformers.md b/.agents/oracles/transformers.md index c280ae84a..d2d85d055 100644 --- a/.agents/oracles/transformers.md +++ b/.agents/oracles/transformers.md @@ -43,3 +43,91 @@ pinned_on = 2026-07-26 gateable = yes evidence = .agents/specs/audio-track.md ``` + +## ACCEPTED lane exception: `qwen4_exp` (`MODEL-MM-QWEN4-EXP`, [#1978](https://github.com/mudler/vllm.cpp/issues/1978)) + +**ACCEPTED by the developer, 2026-08-26.** The `oracle-pin` block above is unchanged +and remains the pin for every other consumer; this lane pin is additional and +narrower. It was put as an explicit accept-or-reject because it changes the +semantics of the invariant this file exists to hold, and it was accepted on that +footing rather than passed as housekeeping. + +`Qwen/Qwen3.8-Flash-Next` declares `Qwen4ExpForConditionalGeneration` / +`model_type: qwen4_exp`. vLLM does not implement it: read live 2026-08-26 at +`origin/main` = `6a5e8f5979`, there is no `qwen4*` path and no registry entry, and a +repository-wide search for `qwen4` returns zero results. `vllm-omni` likewise. The +reference implementation is +[transformers#48337](https://github.com/huggingface/transformers/pull/48337), +merged 2026-08-26. + +**The pin above cannot serve that row: 5.14.1 does not contain `Qwen4Exp`.** + +The rule this would except is stated above as "Pinning it separately would let the +oracle environment hold two different `transformers` at once, which is the drift +this registry exists to stop." The argued exception is narrow, and its narrowness is +the whole case: that invariant guards a **vLLM environment** against drifting from +the `transformers` it resolves. For `qwen4_exp` there is no vLLM implementation, so +no such environment exists and nothing can drift from it. A lane pin here cannot +produce the inconsistency the rule prevents. + +Its scope and expiry, both binding if accepted: + +- It covers `model_type: qwen4_exp` and nothing else. Every other model, processor, + feature extractor and tokenizer continues to resolve against 5.14.1. +- It supplies the **algorithm** only. Per the row's direction, the optimized form of + each primitive still mirrors vLLM, which is the polarity AGENTS.md sets and which + a missing model registration does not suspend. +- **It expires the moment vLLM registers `qwen4_exp`.** At that point the row + reconciles onto vLLM and `transformers` demotes to the preprocessing role it holds + everywhere else in this file. That is a stop condition in the row's spec, not a + reminder. +- `gateable` for the lane is **no** until an arm runs, which is currently blocked on + memory rather than software: nothing published fits any fleet device. Constructing + a config proves nothing, and this file's own precedent applies. + +### The lane pin, and how it was bounded + +`transformers` **5.16.0**, and it is a real release rather than a branch SHA, which +was not the expected outcome. `Qwen4Exp` merged to `main` at 2026-08-26T12:03:40Z and +`v5.16.0` was published at 2026-08-26T12:35:15Z, 32 minutes later, so the release +carries it by a margin of half an hour. + +Bounded rather than assumed, because "the release is newer than the merge" is an +argument and not a check. Measured 2026-08-26 by fetching the model file at each tag: + +| Revision | `src/transformers/models/qwen4_exp/modeling_qwen4_exp.py` | +|---|---| +| `v5.16.0` | HTTP **200** (present) | +| `v5.15.0` | HTTP **404** (absent) | + +`v5.16.0`'s `models/auto/auto_mappings.py` carries 5 occurrences of `qwen4_exp`, so +the registration landed with the model rather than trailing it. 5.16.0 is therefore +the FIRST release containing this architecture, which is the tightest pin available +and the one a lane exception should take. + +```oracle-pin-lane +id = transformers +lane = qwen4_exp +role = secondary +scope = the algorithm for model_type qwen4_exp ONLY; every other model, processor, feature extractor and tokenizer stays on the pin above +pin = 5.16.0 +pin_label = 5.16.0 +pinned_on = 2026-08-26 +accepted_by = developer, 2026-08-26 +expires = when vLLM registers qwen4_exp +gateable = no +gateable_reason = no published artifact fits any fleet device; blocked on memory, not software +owner_row = MODEL-MM-QWEN4-EXP +issue = https://github.com/mudler/vllm.cpp/issues/1978 +evidence = .agents/specs/qwen4-exp-flash-next.md +``` + +**`gateable = no` and the version string is UNMEASURED.** The value above is the +release that provably contains the model, established by fetching its source. It is +NOT a `transformers.__version__` read off a running oracle, and this file's own +precedent says an oracle is gateable only once it demonstrably builds and runs the +model. Resolving the runtime string is owed to the first wave that stands an oracle +up. Do not promote this pin to `gateable = yes` by editing the line. + +See [`../specs/qwen4-exp-flash-next.md`](../specs/qwen4-exp-flash-next.md) +`## Oracles`. diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md new file mode 100644 index 000000000..b089d3449 --- /dev/null +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -0,0 +1,465 @@ +# `Qwen4ExpForConditionalGeneration` (Qwen3.8-Flash-Next) + +**Campaign row:** `MODEL-MM-QWEN4-EXP` (the ID carried by the branch, the issue and +the append-only index row) +**Model-matrix target row:** `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation`, +the deterministic ID the row contract requires. Both name the same work; the index +row is append-only and cannot be re-keyed, so both are recorded rather than one +silently replaced. +**Issue:** [#1978](https://github.com/mudler/vllm.cpp/issues/1978) +**State:** `READY` (spec only; no product code lands under this row's first pull request) +**Motivating checkpoint:** `Qwen/Qwen3.8-Flash-Next`, released 2026-08-24, read live 2026-08-26 + +## Scope + +Port `Qwen4ExpForConditionalGeneration` / `model_type: qwen4_exp`. The card calls it +"this experimental preview of the architecture that will underpin Qwen4"; the +`Qwen3.8` in the name is marketing continuity and not a shape relationship. It is a +180B-total / 6B-activated multimodal (image-text-to-text) hybrid: 48 layers in a +repeating `3 x linear_attention -> 1 x qwen_sparse_attention` pattern, 512-expert MoE +at top-10 plus one shared expert, a 20M-entry n-gram embedding table injected at +layer 2, a 4-branch gated residual stream, and a 1-layer MTP head. + +In scope: text generation and the image/video path, every published quantized arm, +and the GGUF k-quant arms this repository requires of any model port. + +Out of scope for the first implementation wave, each named under `## Owed` rather +than dropped: MTP depth > 1, the 1M-token RoPE extension the card advertises above +the native 262144, and any throughput claim. + +## Why this needs a spec before code + +Three of this row's decisions are expensive to reverse and cheap to get wrong, and +all three have already been made incorrectly once by an agent reading a related +record. They are settled here so a fresh implementer does not re-derive them. + +1. **This is not a Qwen3.8 row.** `.agents/specs/qwen38-27b-bf16-gate.md` records + `Qwen/Qwen3.8-27B` as the Qwen3.6-27B shape retrained, differing in exactly one + config key. That precedent does not extend here. `qwen4_exp` shares an ancestor + with `qwen3_5` and diverges in four load-bearing places. +2. **QSA's twin in vLLM is MiniMax-M3, not DeepSeek-V4.** See `## Design`. Building + it on the DSA/MLA path is the wrong port, and DSA is the path an agent reaches + for first because this tree already has it. +3. **The oracle split is a direction, not a default.** See below. + +## Oracles + +**vLLM implements nothing here.** Read live at `origin/main` = `6a5e8f5979`, +2026-08-26: no `qwen4*` path, no registry entry, and a repository-wide GitHub search +for `qwen4` returns zero results. `vllm-omni` likewise. This is absence from vLLM +`main`, not staleness in our pin (`555967922`), so a pin advance does not reach it. + +**Developer direction, 2026-08-26: transformers is the oracle for the ALGORITHM, +vLLM supplies the OPS.** Recorded verbatim because it is the axis the whole row +hangs on: "use transformers as oracle for algorithmic side. but use ops from vllm so +we account for optimized path." + +This is the correct reading of what each upstream is, and not a split of +convenience. transformers [#48337](https://github.com/huggingface/transformers/pull/48337) +(MERGED 2026-08-26, 5211 lines) is a semantics reference that says so in its own +code: `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` +and carries the comment "we only allow eager and sdpa". Ported as written it yields +a correct model at an indefensible speed. AGENTS.md's "Mirror vLLM" polarity +continues to bind every primitive vLLM implements, even though vLLM has never +assembled this particular model from them. + +Therefore: **every component resolves against exactly one oracle, named in the +`## Design` table. An implementer who cannot name the oracle for the line they are +writing has found a gap in this spec and returns `NEEDS_CONTEXT`.** + +SGLang [#36497](https://github.com/sgl-project/sglang/pull/36497) is OPEN and is not +admissible while it stays open. Re-check it at each wave; if it merges it becomes a +second op source under the `sglang` registry id, still ranked below vLLM. + +### The transformers lane pin (ACCEPTED 2026-08-26) + +`.agents/oracles/transformers.md` pins transformers to **5.14.1**, deliberately tied +to what the pinned vLLM environment resolves, on the stated ground that an +independent pin "would let the oracle environment hold two different `transformers` +at once, which is the drift this registry exists to stop". + +**5.14.1 does not contain `Qwen4Exp`**, so this row cannot run its algorithmic +oracle under the existing pin. + +The exception argued here is narrow: the invariant guards against a vLLM environment +and its transformers drifting apart, and for `qwen4_exp` there is no vLLM +implementation to drift from. A lane-scoped second pin therefore cannot create the +inconsistency the rule exists to prevent. It is recorded in the oracle file as a +lane exception naming this row and this issue, and it expires the moment vLLM +registers `qwen4_exp`, at which point the row reconciles onto vLLM and transformers +demotes to the preprocessing role it holds everywhere else. + +**Accepted by the developer on 2026-08-26**, having been put as an explicit +accept-or-reject rather than passed as housekeeping, because it changes the +semantics of a registry invariant. + +**The lane pin is `transformers` 5.16.0, and it is a real release, not a branch +SHA.** That was not the expected outcome and it is better than one. `Qwen4Exp` +merged to `main` at 12:03:40Z on 2026-08-26 and `v5.16.0` was published at +12:35:15Z, 32 minutes later. Bounded rather than assumed, by fetching the model +source at each tag on 2026-08-26: `v5.16.0` returns HTTP 200 and `v5.15.0` returns +HTTP 404, so 5.16.0 is the FIRST release containing the architecture, which is the +tightest pin available. Its `auto_mappings.py` carries 5 `qwen4_exp` occurrences, so +the registration landed with the model rather than trailing it. + +The version string is **unmeasured**: it is the release that provably contains the +model, not a `transformers.__version__` read off a running oracle. Resolving the +runtime string is owed to the first wave that stands one up. Full record and the +`oracle-pin-lane` block: [`../oracles/transformers.md`](../oracles/transformers.md). + +### Gateability + +`gateable = no` at the time of writing, and the reason is memory rather than +software: see `## Hardware`. The oracle must demonstrably build **and run the +model**, and no published artifact fits any fleet device. The first wave's real +deliverable is the arm that makes an oracle run possible at all. + +## Upstream chain + +| Source | Revision | Role | +|---|---|---| +| `huggingface/transformers` | **`v5.16.0`** (lane pin; first release containing `qwen4_exp`, landed by `#48337` merged 2026-08-26) | algorithm; `models/qwen4_exp/modular_qwen4_exp.py` is the authored delta, `modeling_qwen4_exp.py` the generated expansion | +| `vllm-project/vllm` | `origin/main` `6a5e8f5979` (survey only; the parity pin stays `555967922`) | ops | +| `Qwen/Qwen3.8-Flash-Next` | HF `main`, read 2026-08-26 | config and weights | + +Read the **modular** file, not the generated one. It is 1186 lines against 2707 and +it is the file that states what is inherited unchanged, which is most of the model. + +## Our baseline + +What this tree already has, and therefore what the port does NOT rebuild. Stated +first because the delta only means something against it, and because the size of +this list is the reason the row is tractable at all. + +- **The Qwen3.5 family end to end.** `src/vllm/model_executor/models/qwen3_5*.cpp` + carries the dense and MoE backbones, the GGUF weights path, the MTP draft + (`Qwen3_5MTPModel`) and the runner integration. `Qwen4ExpTextModel` inherits from + `Qwen3_5MoeTextModel`, so this is the base the upstream delta is written against. +- **GDN linear attention with a Triton-AOT fast path.** `src/vt/cuda/cuda_gdn.cu`. + The AOT specializations are pinned to `K=V=128, Hg=16, H in {48,32}` and this + model's `linear_key_head_dim` / `linear_value_head_dim` / `linear_num_key_heads` / + `linear_num_value_heads` are `128 / 128 / 16 / 48`. An exact hit, not a near miss. +- **A working sparse-attention indexer**, `deepseek_v4_dsa.cpp` + + `deepseek_v4_compressor.h` + `src/vt/cuda/cuda_deepseek_v4.cu`. Useful for its + compressor and its cache plumbing; **not** the right base for QSA's selection + path, see `## Port map`. +- **Hyper-connection residual streams**, `deepseek_v4_mhc.cpp`, ported 1:1 from + vLLM's `kernels/mhc/`. Different math from Gated Residual, same fused shape. +- **Interleaved mRoPE**, `layers/rotary_embedding/mrope.cpp`, which this model needs + (`mrope_section [11, 11, 10]`, `partial_rotary_factor` 0.25 over `head_dim` 256). +- **The Qwen3.5-Moe vision tower**, which upstream reuses here **unchanged**. +- MoE with grouped GEMM, and the GGUF k-quant loader stack. + +## Design + +`Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves the rotary +embedding, MLP, experts, TopK router and the **entire vision tower** unchanged +(`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`). This tree already has all +of that. The port is the delta below. + +## Port map + +| Component | Algorithm oracle | Op oracle (vLLM) | This tree | +|---|---|---|---| +| GDN linear attention | `Qwen4ExpTextGatedDeltaNet` | `layers/mamba/gdn/qwen_gdn_linear_attn.py` | **HAVE.** `K=V=128, Hg=16, Hv=48` is an exact match for the AOT gate in `TryTritonPackedDecode` / the delta_h dispatch (`src/vt/cuda/cuda_gdn.cu`, pinned to `K=V=128, Hg=16, H in {48,32}`) | +| Grouped RMSNorm | `Qwen4ExpTextRMSNorm(group_size=)` | `layers/layernorm.py` `group_size` | new, small; mirror vLLM's form | +| QSA block scoring + top-k | `Qwen4ExpTextQSAIndexer` | `models/minimax_m3/common/indexer.py`, `common/ops/index_topk.py`, `common/sparse_attention.py` | new | +| QSA pooled-key build | indexer forward | `models/deepseek_v4/nvidia/ops/sparse_attn_compress_cutedsl.py` (`SparseAttnCompressNormRopeStoreC4Kernel`, carries `compress_ratio`) | partial: `deepseek_v4_compressor.h` | +| Indexer side cache | `Cache.update_indexer` | `MiniMaxM3IndexerCache`, `v1/attention/backends/mla/indexer.py` | new KV spec | +| Gated Residual | `Qwen4ExpTextGatedResidual` | `layers/mhc.py`, `kernels/mhc/*` (**different math**, same fused shape) | partial: `deepseek_v4_mhc.cpp` | +| MoE 512 / top-10 + 1 shared / intermediate 640 | `Qwen4ExpTextSparseMoeBlock` | FusedMoE, grouped GEMM | HAVE, shape change only | +| MTP, 1 layer, `hybrid: true` | config `mtp` | `qwen3_5_mtp.py` | HAVE, needs extension | +| PLE dilated depthwise conv | `Qwen4ExpTextPLELayer._short_conv` | **NONE** | new, no vLLM op | +| N-gram hashed embedding | `Qwen4ExpTextNGramEmbedding` | **NONE** | new, no vLLM op | +| Vision tower | `Qwen4ExpVisionModel` = `Qwen3_5MoeVisionModel` | qwen3_5 vision | HAVE. `deepstack_visual_indexes: []`, so no deepstack | + +**Exactly two components have no vLLM op**, and they are the two where transformers +is the sole source and we author the kernel ourselves. Everything else has an +optimized vLLM form to mirror, and mirroring it is mandatory rather than optional. + +### QSA maps to MiniMax-M3 + +DSA is an **MLA** indexer. QSA is not MLA: plain GQA, 24 Q heads, 2 KV heads, +`head_dim` 256, `partial_rotary_factor` 0.25 giving 64 rotary dims. MiniMax-M3's +lightning indexer is vLLM's **non-MLA** block-sparse case, and its docstring +describes QSA's shape exactly: it "scores KV blocks with the index heads and selects +the top-k blocks ... that the main block-sparse attention then attends to", owning +"its own side cache (one index-key vector per token)". `common/ops/index_topk.py` +supplies `_index_block_score_kernel` and a bitonic `_topk_index_kernel`. + +Reconciliation the implementer owes, not hand-waved here: M3's sparse block size is +128 where QSA's `indexer_compress_ratio` is 4 with `block_topk = indexer_budget / +compress_ratio = 512`; and QSA's per-block key is a **mean pool over the block**, +then `k_layernorm`, then RoPE at the block-start position, scored as +`relu(q . k).sum(over 4 index heads) / sqrt(128)`. The pooled-key construction is +what `SparseAttnCompressNormRopeStoreC4Kernel` already does under a different name. + +`indexer_kv_heads` must be 1; the upstream config validator rejects anything else. + +### Two structural consequences beyond the module list + +- **The residual stream is `hc_count * hidden_size` = 4 x 2560 = 10240 wide through + the whole stack.** `Qwen4ExpTextGatedResidual` reads it through a grouped RMSNorm + and a low-rank (`hc_lowrank` = 320) SiLU-then-sigmoid gate, collapses to 2560 for + the block, and writes back with a per-branch scalar gate + (`2 * sigmoid(block_inject_weight(x) / hc_count)`). This is a change to the + per-layer loop and to every residual buffer, not a drop-in module. The + `Qwen4ExpTextModel` also holds one `use_combine=False` mixer that collapses the + stream at the end. +- **`number_of_conv_states = 3` on a PLE layer** (GDN conv, PLE conv, and the n-gram + token history, which upstream stores as a third conv state precisely because the + manipulations are identical). The KV-cache spec grows a third conv stream plus the + indexer side cache. Adjacent to [#1963](https://github.com/mudler/vllm.cpp/issues/1963) + and [#1966](https://github.com/mudler/vllm.cpp/issues/1966). + +### The n-gram embedding is integer-exact or it is silently wrong + +Head vocab sizes are successive **primes** found after `ngram_vocab_size_base - 1` +(20,000,000), indexed by a global head index; IDs are built by XOR-mixing +splitmix64-seeded per-position multipliers over shifted token windows, then reduced +modulo each head's prime. An off-by-one in the prime search, or a 32-bit truncation +anywhere in the mix, yields a valid-looking lookup into the wrong row. Nothing +downstream detects it, and a token gate against a model this size will not localise +it. Gate the ID construction directly, against transformers, before any weight is +loaded. + +## Dependencies + +Shared seams this row must route through rather than around, per AGENTS.md +"Shared seams". Each is named so a reviewer can check the routing instead of +inferring it. + +- `ModelRegistry::Forward` and `dense_attn::AttnBlock` for decode. +- `vt::FusedChain` for model fusion; `layers::MlpGateUpMethodBase` and + `vt::MergedGemmGroup` for the mergeable MLP projections. +- `include/vllm.h` for every shipped capability. Examples and servers stay thin + ABI clients and never include an internal header. +- `vllm::HfConfigFromGguf` and the `qwen3_5` GGUF builder, which currently + hard-asserts its own architecture and will refuse `qwen4_exp` by name until this + row extends it. +- `src/vllm/v1/kv_cache_interface.*` for the third conv stream and the indexer side + cache. This is the seam [#1963](https://github.com/mudler/vllm.cpp/issues/1963) + and [#1966](https://github.com/mudler/vllm.cpp/issues/1966) are moving; coordinate + rather than fork. + +New files go beside their vLLM counterparts and mirror the upstream file structure, +per AGENTS.md. Where the upstream counterpart is MiniMax-M3 rather than a Qwen file, +mirror the op's home and say so in the file header. + +## Work breakdown + +Waves are separable and each is independently reviewable. Every wave lands reachable +from a production entry point, or names what is unreached with its owning row and +issue per AGENTS.md "Nothing lands dead". + +- **W0, this pull request.** Spec, records, oracle exception proposal. No product + code. +- **W1, config and registration.** `qwen4_exp` config resolution including the + `full_attention` -> `qwen_sparse_attention` rewrite upstream performs in + `__post_init__`, every `validate_architecture` rejection, and a refusal naming any + unimplemented arm. Reachable through the loader. +- **W2, the two components with no vLLM op.** N-gram hashed embedding and the PLE + layer with its dilated depthwise conv. Gated against transformers goldens on + integer equality for the ID construction. First because they are the highest + silent-wrongness risk and because they are independent of the attention work. +- **W3, gated residual.** The 10240-wide stream through the per-layer loop, both + `use_combine` arms, and the final mixer. +- **W4, QSA.** Indexer side cache and KV spec, pooled-key build, block scoring and + top-k, block-sparse consumer. Mirrors MiniMax-M3's op shape. +- **W5, assembly and the load plan.** Full model forward, vision path, MTP. +- **W6, the first runnable arm** and the row's real unblock: a Q4_K_M backbone with + the n-gram table non-resident, per the developer decision in `## Hardware`. Two + separable halves. **W6a** authors the `qwen4_exp` GGUF architecture on our side, + because llama.cpp has none, and states in its result that these arms therefore + have no llama.cpp oracle. **W6b** makes the 51 GB table non-resident, which on + unified memory cannot be the existing host-pinned offload and needs its mechanism + established first. W6b is the one with unknown cost and should be spiked before it + is scheduled. + +Waves W2 through W4 have no ordering dependency on each other and can be dispatched +in parallel to separate worktrees. W5 is a barrier. + +## Hardware + +Usable budget on GB10 is about 119 GB. Read live from the HF API, 2026-08-26: + +| Artifact | On disk | Verdict | +|---|---|---| +| `Qwen/Qwen3.8-Flash-Next` BF16 | ~360 GB (`BF16 = 179,999,981,424` params) | no | +| `Qwen/Qwen3.8-Flash-Next-FP8` (official) | ~180 GB | no | +| `RadixArk/Qwen3.8-Flash-Next-NVFP4` | ~128 GB; NVFP4 backbone with the n-gram table kept at **FP8, 51.2 GB** | no, over budget before KV | +| `unsloth/Qwen3.8-Flash-Next-GGUF` | **README only, zero weight files** | does not exist | + +No GGUF exists and no existing tool can produce one, because llama.cpp has no +`qwen4_exp` architecture either. Per AGENTS.md the quantized arms are a standing +requirement, so this row owes them and owes authoring the arch on our side. + +**The architecture hands us the lever.** Its card argues n-gram embedding is "more +amenable to offloading than MoE", and the arithmetic agrees: the per-token cost is +`(ngram_size - 1) * heads_per_ngram` = 16 lookups of `ple_embed_dim / ngram_heads` = +160 dims. **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per +token.** Making it non-resident is the intended design point. RadixArk reached the +same split independently. + +| Arm | Backbone (125B) | N-gram (51B) | Resident | Fits | +|---|---|---|---|---| +| Q8_0 throughout | ~133 GB | ~54 GB | ~191 GB | no | +| Q4_K_M throughout | ~76 GB | ~31 GB | ~109 GB | yes, ~10 GB for KV and activations | +| **Q4_K_M backbone, n-gram table non-resident** | ~76 GB | 0 | **~76 GB** | yes, with room | + +**Developer decision, 2026-08-26: the first runnable arm is the third row — a +Q4_K_M backbone with the n-gram table non-resident.** Q8_0 was raised and does not +fit: at ~191 GB it exceeds the budget by a wider margin than BF16 exceeds it on a +box half this size, and no partial-Q8 split reaches 119 GB while keeping the +backbone at 8 bits. Q4_K_M-throughout fits on paper at ~109 GB but leaves about +10 GB for KV and activations on a model whose native context is 262144, which is +not a margin. The chosen arm is also the only one that matches what the +architecture was built for, so the offload is a design point rather than a +concession. + +This promotes the non-resident table from a note to a **first-class deliverable** +of W6. It is not free: GB10 is unified memory, so the existing host-pinned offload +seam (`ENG-WEIGHT-OFFLOAD`, mirroring vLLM's `cpu_offload_gb`) does not by itself +solve this there, and the mechanism has to be disk-backed or genuinely unloaded. +Establish that before designing around it. + +These are sizing estimates from published parameter counts, not measurements. They +decide which arm to attempt first and nothing else. GB10 is **unified** memory, so +"offload to host" is not a move there; non-resident means disk-backed and page-cached, +and its cost is unmeasured. Establish it before it is designed around. + +## Risks + +- **Porting the eager reference as written.** The stated risk of the oracle split. + The QSA indexer and the n-gram ID construction are both written as scalar Python + in transformers. A reviewer should mutate for this: an implementation whose QSA + path has no block-level kernel is a correctness result, not a port. +- **Reaching for DSA.** This tree has a working DSA indexer, and it is the wrong + base. See above. +- **Silent n-gram mis-indexing.** See above. +- **Sizing estimates hardening into measurements.** The `## Hardware` table is + arithmetic on published counts. `.agents/` already records this failure mode + (a quoted number becoming a measured one); do not let the 76 GB row be cited as + an observation. +- **A GGUF arm with no oracle.** llama.cpp does not implement `qwen4_exp`, so the + usual quant-arm cross-check against a quant-matched llama.cpp does not exist. A + k-quant arm here can only be gated against our own higher-precision path, which + is a weaker gate, and the spec must say so rather than imply parity. +- **`transformers_version: 5.8.0.dev0`** in the published config is older than both + our pin and the branch that merged `Qwen4Exp`. It records the branch the config + was authored on and is not a usable pin. Do not resolve the oracle from it. + +## Tests to port + +AGENTS.md requires the upstream tests in the same change, preserving parameters, +modes, fixtures, tolerances, failure cases and the revision anchor. The upstream +suite is `tests/models/qwen4_exp/test_modeling_qwen4_exp.py` at transformers #48337, +707 lines, two classes. Inventory, read live 2026-08-26: + +**`Qwen4ExpTextModelTest`** (`Qwen4ExpTextModelTester`, a `CausalLMModelTester`): + +| Upstream case | Ports to | Note | +|---|---|---| +| `test_ple_layers_must_use_linear_attention` | W1 | a config invariant; cheap and load-bearing | +| `test_ple_padding_and_static_cache_match_unpadded_sequence` | W2 | the padding/EOS-segment semantics of the n-gram history | +| `test_all_layer_types_cached_forward_match_full_forward` | W4/W5 | cached vs full forward across BOTH layer types; this is the incremental-decode gate | +| `test_ple_beam_generation` | W5 | PLE under beam search, where the conv and n-gram states must follow the beam | +| `test_ple_sharded_checkpoint_loads_and_forwards` | W5 | 131 shards here, so sharded load is not optional | +| `test_generate_with_ple_and_inputs_embeds` | W5 | drives `reverse_embedding`, the inputs-embeds path | +| `test_reverse_loading_mapping` | W1 | weight-name mapping both directions | +| `test_attention_outputs`, `test_hidden_states_output` | W3/W4 | both are OVERRIDDEN upstream because the hyper-connection stream changes the shapes; port the override, not the base | +| `test_tp_plan_matches_params` | not ported | tensor-parallel plan; no TP surface in this row | +| `test_generate_compile_model_forward_fullgraph`, `test_generate_compilation_all_outputs`, `test_multi_gpu_data_parallel_forward`, `test_generate_with_quant_cache` | not ported | torch.compile / multi-GPU / torch quant-cache harness, no counterpart here | + +**`Qwen4ExpCompositeModelTest`** (`Qwen4ExpVisionText2TextModelTester`, a +`VLMModelTester`): `test_mismatching_num_image_tokens`, `test_video_forward`, +`test_composite_checkpoint_loads_as_causal_lm`, +`test_base_model_checkpoint_loads_as_conditional_generation`, +`test_generate_with_ple_and_inputs_embeds`, plus its own `test_attention_outputs` / +`test_hidden_states_output` overrides. All port to W5. The remaining cases in that +class are the same harness-only skips as above. + +Adaptations must be documented per AGENTS.md, and only where genuinely unavoidable. +"Our harness differs" is not one; "upstream asserts against a `torch.compile` +fullgraph we do not have" is. + +### Local red-first tests + +Red-first, smallest failing test per slice, each entering through a production entry +point per AGENTS.md "Nothing lands dead". A unit test that constructs the type by +hand does not discharge this. + +1. N-gram ID construction against transformers goldens: prime head vocab sizes, the + splitmix64 multipliers, the shift-and-XOR mix, EOS segment handling. Integer + equality, no tolerance. +2. Grouped RMSNorm against vLLM's `group_size` form. +3. Gated Residual forward against transformers, both `use_combine` arms. +4. QSA block selection: selected token index sets equal to transformers on the same + inputs, including the ragged tail beyond the last complete block. +5. PLE layer end to end, including the dilated depthwise conv and its state. +6. Config resolution: the `full_attention` -> `qwen_sparse_attention` rewrite that + upstream `__post_init__` performs, and every rejection in `validate_architecture`. +7. Loader coverage against the published index, with the refusal path naming any + unimplemented arm. +8. Inertness: existing Qwen3.5/3.6/3.8 goldens byte-identical. + +## Gates + +No token gate is claimable until an arm runs. In order: + +1. **G0, component goldens.** Tests 1-6 above against transformers at the lane pin. + This is the only gate reachable today, and it is reachable without the weights. +2. **G1, load plan.** Every published tensor accounted against a committed manifest, + per arm, with refusals naming what is missing. +3. **G2, token-exact greedy** vs transformers at the lane pin, on whichever arm + `## Hardware` makes runnable first. Strict token equality; the near-tie + distributional doctrine applies only if the oracle's greedy decode is shown + non-deterministic, which is not assumed here. +4. **G3, quantized arms.** Per arm, with the lower-bound requirement this repository + places on quantized gates, and with the missing-llama.cpp-oracle limitation stated + in the result rather than omitted. +5. **Speed: nothing.** No throughput, latency or memory number is admissible from + this row until G2 passes. There is no vLLM denominator for this model, so when a + speed axis does open, the spec must first say what the denominator is. + +## Evidence required + +Per gate: the exact build and run recipe, the lane transformers revision, the +checkpoint repo **and revision** plus sha256 for any quantized artifact, the device, +and the contention state. `docs/USAGE.md` gains the checkpoint pins in the same +change that makes any arm reachable, not later. + +## Stop conditions + +- vLLM registers `qwen4_exp`: **stop and reconcile onto vLLM** before continuing. + This is the designed end of the transformers exception. +- SGLang #36497 merges: re-survey the op mapping; it does not displace vLLM. +- The transformers lane pin is rejected in review: the row holds at `READY` and the + gate stays `PENDING`. Do not proceed on an unpinned oracle. +- No arm is made to fit any fleet device: the row holds with G0 passed and G1-G3 + `PENDING` on hardware, recorded as visible debt, and no token claim is made. + +## Owed + +- [#1978](https://github.com/mudler/vllm.cpp/issues/1978): this port. No product + code lands under the spec pull request. +- GGUF k-quant arms, including authoring the `qwen4_exp` architecture on our side, + and the statement that no llama.cpp oracle exists for them. +- MTP depth > 1. +- The 1M-token RoPE extension above the native 262144. +- The non-resident n-gram table: its mechanism, and a measurement of its cost. +- A speed denominator, once one exists. + +## Now + +`READY`. Spec committed, no implementation. + +Both decisions this spec was blocked on are **settled** (developer, 2026-08-26) and +recorded in place rather than left as proposals: the transformers lane pin is +ACCEPTED at 5.16.0 (`## Oracles`), and the first runnable arm is the Q4_K_M backbone +with a non-resident n-gram table (`## Hardware`). + +Next actions, in order: W0 lands this spec; W1 through W3 are reachable today +against the lane pin with tiny random configs and need neither a checkpoint nor a +GPU lease; W6b's mechanism is the unknown that decides whether the chosen arm is +schedulable, and it should be spiked before W6 is planned. diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index 136d68ce5..1bebb068c 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -130,7 +130,23 @@ # vllm#51255, still being patched), carries no pinned-registry target, and # leaves the at-the-pin inventory (324/373/356/310/261) unchanged. Bumped # because two rows EXIST, never to make a transition pass. - "MODEL": (AGENTS / "model-matrix.md", 377), + # 378 since 2026-08-26, and RE-DERIVED off the matrix rather than carried + # forward: +1 for `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` + # (`Qwen4ExpForConditionalGeneration`, `Qwen/Qwen3.8-Flash-Next`), landing + # `READY` with its spec committed (#1978). ONE row and not two: the MTP head + # is a `mtp` block inside the same text config, not a separately registered + # architecture, so this is not the IndexTTS-2.5 / dots3-note shape that moved + # this pin by two. Beyond-pin in the strongest sense yet recorded here -- the + # Muse Glimmer, Qwen3.5-text-only and dots3-note entries above are all + # architectures vLLM registers on `main` AFTER `555967922`, whereas this one + # vLLM does not implement at ANY revision: read live 2026-08-26 at + # `origin/main` = `6a5e8f5979`, there is no `qwen4*` path, no `registry.py` + # entry, and a repository-wide search for `qwen4` returns zero results. Its + # Upstream cell therefore carries no pinned module/class target and its + # algorithm source is transformers#48337, so the at-the-pin static invariants + # (324/373/356/310/261) are UNCHANGED. Bumped because one row EXISTS, never to + # make a transition pass. + "MODEL": (AGENTS / "model-matrix.md", 378), # 82 since 2026-07-21: +`QUANT-NVFP4-CT-W4A16` (compressed-tensors NVFP4A16 / # W4A16 — NVFP4 weights with BF16 activations, distinct from the existing # `QUANT-NVFP4-CT-W4A4` and `QUANT-NVFP4-MO-W4A16` rows in both scheme From d70d3925e9c2d02568f1d577d037f285e356b3a8 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 13:33:11 +0000 Subject: [PATCH 2/3] spec(MODEL-MM-QWEN4-EXP): five component deep-dives, and QSA maps to DeepSeek-V4 rather than MiniMax-M3 Folds five parallel component investigations into the row spec and corrects two claims the first draft got wrong. Every load-bearing finding below was re-verified against the source before it was written down; the agents' reports are inputs, not evidence. The headline correction. The spec claimed "QSA's structural twin in vLLM is MiniMax-M3, NOT DeepSeek-V4", reasoning that QSA is plain GQA rather than MLA and so had to map onto the non-MLA block-sparse case. That reasoning rested on reading `MLAAttentionSpec` as an MLA claim. It is not one: M3's own indexer cache uses it while M3 is plain GQA, and the comment beside it says why, "Key-only: MLAAttentionSpec budgets one vector/token (not 2x for K+V)". It is a budget shape. With that prop removed the argument collapses, and nine independent structural matches with DeepSeek-V4 remain, `compress_ratio == 4` literally the same number: MQA index with one key head at dim 128, `relu(q.k)` summed over index heads, `1/sqrt(head_dim)`, one score set per query token with no head axis, the `(position+1) % COMPRESS_RATIO == 0` boundary, RMSNorm on the pooled key, RoPE at the block-start position, candidates counted as `visible // compress_ratio`, and one stored state per four tokens through `MLAAttentionSpec(tokens_per_state=compress_ratio)`. M3 is a different algorithm, not a worse fit. Verified: its score is `tl.max(qk, axis=1)` over 128 raw token dots with no pooling, no relu and no head reduction; it asserts `num_idx_heads == num_kv_heads` with the comment "no topk index reduce", so it emits one block set per KV head where QSA emits one per token; and `SPARSE_BLOCK_SIZE = 128` is welded to the KV page size on both the score and the attend side. It contributes exactly one thing, a wiring precedent: a plain-GQA model can own a key-only side cache and a private indexer backend. The genuinely new work is the consumer, and nothing upstream supplies it. Every DSv4 sparse consumer attends to compressed KV; M3's attend to raw tokens only at page granularity; QSA attends to raw tokens selected at ratio-4 granularity. Two silent failures follow. Wiring QSA's top-k into a DSv4 sparse-MLA consumer attends a pooled key and value and still emits plausible tokens, and a short-prompt gate cannot catch it, because below `indexer_budget` every candidate is selected. `## Gates` now requires at least one prompt past 2048 tokens of context for exactly that reason. And `SparseAttnCompressNormRopeStoreC4Kernel` does not mean-pool despite the name: it is a learned softmax pool over an overlapping window of eight, driven by a score channel this checkpoint does not have. The second correction is smaller and would have cost an afternoon: vLLM's grouped RMSNorm is on `RMSNormGated`, not the plain `RMSNorm`, whose only related knob is `var_hidden_size`, a prefix reduction that cannot express per-group norms. The chosen arm does not load today, and the blocker is neither the offload nor the budget. This tree cannot keep a gather table quantized at all: `KeepQuantKDim` returns -1 for `kEmbeddingTable` and the qwen3_5 loader asserts it by name, so a quantized n-gram table expands to bf16 and 51.2B parameters become 102.4 GB of anonymous memory. The reason was already recorded in a header comment, "a gather, not a GEMM ... A quantized-gather op is a follow-up row", and no such row exists. Separately, `moe_intermediate_size = 640` and `hc_lowrank = 320` are Q4_K-illegal on their reduction dims, and our GGUF reader has no entry for ggml type ids 3, 6, 7 or 20, so a stock `llama-quantize -Q4_K_M` file would fail at header parse. We author the converter, so the fix is Q4_0. `ENG-WEIGHT-OFFLOAD` does not help and must not be budgeted for: it moves zero bytes today and is documented inert on unified memory. The tier that works already ships and the 2.4T model proves it. W6 is split into a/b/c accordingly. Sizing survives: backbone ~67.7 GiB, whole process ~73.5 GiB of 119.631 at 32K single stream. The design works because per-token demand is at most 64 KiB of reads against the 2.4T expert lane's 6.95 GB/token. The n-gram derivation was verified against the published checkpoint rather than trusted: reconstructing the splitmix64 chain at the default seed 1234 reproduces the `layer_multipliers` buffer read out of the safetensors payload, and the head vocab sizes and offsets match entry for entry. Three silent C++ divergence sites are now named, the worst being that the shard reassembly must be numeric, since a lexicographic key sort gives shard_0, shard_1, shard_10 and permutes a 95 GiB table. Confirmed from the index: 128 contiguous shards, and the PLE sits on decoder layer 1. Also recorded: vLLM has no dilated 1-D convolution anywhere, a confirmed negative with zero hits in mamba/, csrc/ and tests/; the PLE conv is strided history at lags 9/6/3/0; its signed-sqrt gate clamps before the square root, so the output floor is 1e-3 and tiny scores are amplified rather than squashed; and `Qwen4ExpTextModel` has no final RMSNorm, which the natural copy of our DeepSeek-V4 tail would wrongly add. Tracked by #1978. Gates: `check-agent-record` ok, `check-model-checklist` ok, `agent-preflight.sh --staged` ok. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/issue-index.md | 2 +- .agents/model-matrix.md | 2 +- .agents/specs/qwen4-exp-flash-next.md | 380 +++++++++++++++++++++++--- 3 files changed, 340 insertions(+), 44 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index c5c4b9c82..ff108e7fd 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -744,4 +744,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1762](https://github.com/mudler/vllm.cpp/issues/1762) | `GEMMA4-FP8-WMMA-EXPERT-GEMM` | KEEP Gemma-4 FP8 T>1 expert path still dequantizes to BF16 and calls hipBLAS Tensile; a gated gfx1201 FP8 WMMA expert GEMM is the unblocked L2 lever (31.5% prefill). Spec-first, default-OFF, no GPU on this filing | perf | | [#526](https://github.com/mudler/vllm.cpp/issues/526) | `SERVE-TOOL-HISTORY-ARGS` | OpenAI multi-turn tool history reaches chat templates with string-valued arguments | bug | | [#1934](https://github.com/mudler/vllm.cpp/issues/1934) | `BACKEND-ROCM` | `RocmPlatform::needs_weight_staging()` is stale-false (a W0-era placeholder never revisited despite #523/#509/#506/ROCM_ATTN/hipGraph landing since), so `CheckDeviceWeightFit` — the #1123/#1870 load-time refusal, including the `policy_forces_full_expand` fix — never runs on ROCm: measured directly, `VT_DEVICE_WEIGHT_BUDGET_BYTES=1` produced no refusal on a real load. The actual device allocation the refusal guards is not gated on this flag, so #1870's crash stays reachable until this closes; owed, not fixed in flow, because flipping the flag also moves `DirectDeviceLoadEligible` and several GDN kernel-dispatch defaults that each need their own correctness check | bug | -| [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-QWEN4-EXP` | **`Qwen/Qwen3.8-Flash-Next` declares `Qwen4ExpForConditionalGeneration` / `qwen4_exp`, a new architecture vLLM does not implement, so the port runs on a split oracle: transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH.** Released 2026-08-24, 180B total / 6B activated, image-text-to-text. The `Qwen3.8` in the name is marketing continuity: `.agents/specs/qwen38-27b-bf16-gate.md`'s "one config key differs" precedent does NOT extend here. Read live 2026-08-26 at vLLM `origin/main` = `6a5e8f5979`: no `qwen4*` path, no registry entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. That is absence from vLLM `main` rather than staleness in our pin `555967922`, so a pin advance does not reach it. What exists is transformers [#48337](https://github.com/huggingface/transformers/pull/48337) "Add Qwen4Exp model", MERGED 2026-08-26, 5211 lines, and SGLang [#36497](https://github.com/sgl-project/sglang/pull/36497), still OPEN and therefore inadmissible. **Developer direction 2026-08-26, recorded verbatim: "use transformers as oracle for algorithmic side. but use ops from vllm so we account for optimized path."** Justified rather than convenient: `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so porting it as written yields a correct model at an indefensible speed, while AGENTS.md's mirror-vLLM polarity still binds every primitive vLLM implements. `Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`), all of which this tree has; GDN is an exact match for our AOT gate (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). The delta is four things, and **exactly two have no vLLM op at all**: the PLE dilated depthwise conv (kernel 4, dilation 3; `git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the n-gram hashed embedding. **The survey's load-bearing finding: QSA's structural twin in vLLM is MiniMax-M3, NOT DeepSeek-V4.** DSA is an MLA indexer; QSA is plain GQA (24 Q / 2 KV, `head_dim` 256, `partial_rotary_factor` 0.25). `models/minimax_m3/common/indexer.py` is vLLM's non-MLA block-sparse case and its own docstring describes QSA's shape — scores KV blocks with index heads, selects top-k blocks, owns a side cache of one index-key vector per token — with `common/ops/index_topk.py` supplying `_index_block_score_kernel` and a bitonic `_topk_index_kernel`; the pooled-key build (mean-pool, `k_layernorm`, RoPE at block start) is `deepseek_v4/nvidia/ops/sparse_attn_compress_cutedsl.py`'s `SparseAttnCompressNormRopeStoreC4Kernel`, which carries `compress_ratio`. Building QSA on the DSA/MLA path is the wrong port and is the one this tree reaches for first, because it already has DSA. Two structural consequences beyond the module list: the residual stream is `hc_count * hidden_size` = **4 x 2560 = 10240 wide through the whole stack** with a low-rank read gate and per-branch scalar write gate around both attention and MLP, which is a change to the per-layer loop and every residual buffer rather than a drop-in module; and `number_of_conv_states = 3` on a PLE layer (GDN conv, PLE conv, n-gram token history) plus the indexer side cache, adjacent to [#1963](https://github.com/mudler/vllm.cpp/issues/1963) and [#1966](https://github.com/mudler/vllm.cpp/issues/1966). **NOTHING PUBLISHED FITS**, read live from the HF API against ~119 GB usable on GB10: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB (NVFP4 backbone with the n-gram table left at FP8, 51.2 GB) and `unsloth/...-GGUF` is a README with ZERO weight files. No GGUF exists and no tool can make one, because llama.cpp has no `qwen4_exp` either, so the standing k-quant requirement means authoring the arch on our side AND states that the quantized arms have NO llama.cpp oracle. The architecture supplies its own lever: the per-token n-gram cost is `(ngram_size-1)*heads_per_ngram` = 16 lookups of 160 dims, so **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per token** and making it non-resident is the intended design point (RadixArk reached the same split independently). Sizing arithmetic, NOT measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB (yes, ~10 GB left for KV), Q4_K_M backbone with the table non-resident ~76 GB. GB10 is UNIFIED memory so "offload to host" is not a move there; non-resident means disk-backed, and its cost is unmeasured. **Two decisions were put to the developer as explicit accept-or-reject and BOTH are settled 2026-08-26, recorded in place rather than left open.** (1) `.agents/oracles/transformers.md` pins transformers to 5.14.1, deliberately tied to what the pinned vLLM environment resolves so the environment cannot hold two `transformers` at once, and **5.14.1 does not contain `Qwen4Exp`**; the lane-scoped second pin is **ACCEPTED**, on the argument that the invariant guards a vLLM environment against drifting from its transformers and here there is no vLLM implementation to drift from, and it expires the moment vLLM registers `qwen4_exp`. **The lane pin is a real release, not a branch SHA**, which was not the expected outcome: `Qwen4Exp` merged to `main` at 12:03:40Z on 2026-08-26 and `v5.16.0` published at 12:35:15Z, and this was BOUNDED rather than assumed by fetching `models/qwen4_exp/modeling_qwen4_exp.py` at each tag — `v5.16.0` HTTP **200**, `v5.15.0` HTTP **404** — making 5.16.0 the FIRST release carrying the architecture and therefore the tightest available pin. The version string is UNMEASURED (it is the release proven to contain the model, not a `transformers.__version__` read off a running oracle) and `gateable` stays `no`. (2) The first runnable arm is the **Q4_K_M backbone with the n-gram table NON-RESIDENT** (~76 GB). Q8_0 was raised and does not fit at ~191 GB, and no partial-Q8 split reaches 119 GB with the backbone at 8 bits; Q4_K_M-throughout fits on paper at ~109 GB but leaves ~10 GB for KV and activations on a 262144-native-context model, which is not a margin. This promotes the non-resident table from a note to a first-class W6 deliverable, and it is NOT free: GB10 is UNIFIED memory, so the existing host-pinned offload seam (`ENG-WEIGHT-OFFLOAD`, mirroring vLLM's `cpu_offload_gb`) does not by itself solve it there and the mechanism must be disk-backed or genuinely unloaded — established before it is designed around. Spec: [`specs/qwen4-exp-flash-next.md`](specs/qwen4-exp-flash-next.md). No product code lands under the spec pull request | feature | +| [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-QWEN4-EXP` | **`Qwen/Qwen3.8-Flash-Next` declares `Qwen4ExpForConditionalGeneration` / `qwen4_exp`, a new architecture vLLM does not implement, so the port runs on a split oracle: transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH.** Released 2026-08-24, 180B total / 6B activated, image-text-to-text. The `Qwen3.8` in the name is marketing continuity: `.agents/specs/qwen38-27b-bf16-gate.md`'s "one config key differs" precedent does NOT extend here. Read live 2026-08-26 at vLLM `origin/main` = `6a5e8f5979`: no `qwen4*` path, no registry entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. That is absence from vLLM `main` rather than staleness in our pin `555967922`, so a pin advance does not reach it. What exists is transformers [#48337](https://github.com/huggingface/transformers/pull/48337) "Add Qwen4Exp model", MERGED 2026-08-26, 5211 lines, and SGLang [#36497](https://github.com/sgl-project/sglang/pull/36497), still OPEN and therefore inadmissible. **Developer direction 2026-08-26, recorded verbatim: "use transformers as oracle for algorithmic side. but use ops from vllm so we account for optimized path."** Justified rather than convenient: `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so porting it as written yields a correct model at an indefensible speed, while AGENTS.md's mirror-vLLM polarity still binds every primitive vLLM implements. `Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`), all of which this tree has; GDN is an exact match for our AOT gate (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). The delta is four things, and **exactly two have no vLLM op at all**: the PLE dilated depthwise conv (kernel 4, dilation 3; `git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the n-gram hashed embedding. **The survey's load-bearing finding, and it REVERSES this row's first reading: QSA's structural twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The original call was that QSA, being plain GQA rather than MLA, had to map onto vLLM's non-MLA block-sparse case; that reasoning rested on treating `MLAAttentionSpec` as an MLA claim, and **it is not one** — M3's own indexer cache uses it while M3 is a plain-GQA model, with the comment "Key-only: MLAAttentionSpec budgets one vector/token (not 2x for K+V)". It is a budget shape. Remove that prop and the GQA-vs-MLA argument collapses. Verified line by line at `6a5e8f5979`: **nine independent structural matches with DSv4**, `compress_ratio == 4` literally the same number — MQA index with 1 key head at dim 128; `relu(q.k)` summed over index heads vs `(score.relu() * weights).sum(dim=0)`; `1/sqrt(head_dim)`; one score set per query token with no head axis vs `topk_indices_buffer[num_tokens, topk]`; pooling boundary `(position+1) % COMPRESS_RATIO == 0`; RMSNorm on the pooled key; **RoPE at the block-start position** vs `compressed_pos = (position // CR) * CR`; candidate count `visible // compress_ratio`; and one stored state per 4 tokens via `MLAAttentionSpec(tokens_per_state=compress_ratio)`, a first-class KV field documented as "Ints > 1 compress multiple tokens into one state (DSv4 sparse MLA)" that has no M3 equivalent. **M3 is a DIFFERENT ALGORITHM**, not a worse fit: its score is `tl.max(qk, axis=1)` over 128 RAW token dots with no pooling, no relu and no head reduction, it asserts `num_idx_heads == num_kv_heads` ("no topk index reduce") so it emits one block set PER KV HEAD, and its `SPARSE_BLOCK_SIZE = 128` is welded to the KV page size ("One sparse block == one KV page") on both the score and the attend side — moving it to 4 forces a page size of 4 and breaks `tl.dot`, whose tile needs >= 16. M3 contributes exactly ONE thing and it is a wiring precedent, not an algorithm: that a plain-GQA model can own a key-only side cache through `MLAAttentionSpec` and a private indexer backend. **The genuinely new work is the CONSUMER and nothing upstream supplies it** — every DSv4 sparse consumer attends to COMPRESSED MLA KV (one state per 4 tokens) and M3's attend to raw tokens only at page granularity, while QSA attends to RAW tokens selected at ratio-4 granularity. Two silent-failure traps follow: wiring QSA's top-k into a DSv4 sparse-MLA consumer attends a POOLED key/value and still emits plausible tokens, and **a short-prompt token gate cannot catch it because at context <= `indexer_budget` 2048 every candidate is selected** — so any QSA gate must run past 2048 tokens of context, which is now a stated `## Gates` requirement; and `SparseAttnCompressNormRopeStoreC4Kernel` does NOT mean-pool despite its name — it is a learned softmax pool over an OVERLAPPING window of 8 using a score channel this checkpoint does not have, and the CuteDSL variant refuses `overlap=False` at compile, so the **Triton** `head_dim=128` variant is the correct starting point. Two structural consequences beyond the module list: the residual stream is `hc_count * hidden_size` = **4 x 2560 = 10240 wide through the whole stack** with a low-rank read gate and per-branch scalar write gate around both attention and MLP, which is a change to the per-layer loop and every residual buffer rather than a drop-in module; and `number_of_conv_states = 3` on a PLE layer (GDN conv, PLE conv, n-gram token history) plus the indexer side cache, adjacent to [#1963](https://github.com/mudler/vllm.cpp/issues/1963) and [#1966](https://github.com/mudler/vllm.cpp/issues/1966). **NOTHING PUBLISHED FITS**, read live from the HF API against ~119 GB usable on GB10: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB (NVFP4 backbone with the n-gram table left at FP8, 51.2 GB) and `unsloth/...-GGUF` is a README with ZERO weight files. No GGUF exists and no tool can make one, because llama.cpp has no `qwen4_exp` either, so the standing k-quant requirement means authoring the arch on our side AND states that the quantized arms have NO llama.cpp oracle. **The chosen arm does NOT load today, and the blocker is neither the offload nor the budget: this tree cannot keep a gather table quantized at all.** `KeepQuantKDim` returns `-1` for `GgufTensorRole::kEmbeddingTable` (`src/vllm/model_executor/model_loader/gguf_keep_quant.cpp`), and `qwen3_5_gguf_weights.cpp` asserts it by name — "the embedding table cannot keep quant blocks" — so a Q4_K or Q8_0 n-gram table EXPANDS to bf16 and 51.2B params become **102.4 GB of anonymous memory**; the arm dies before the first forward. The reason was already sitting in a header comment ("a gather, not a GEMM ... A quantized-gather op is a follow-up row") and **no such row exists**. The only non-expanding gather residency is `kKeepF16`, requiring ggml type 1 exactly (102.4 GB on disk) and CPU-ONLY, because `EmbeddingKernelCuda` refuses anything but f32/bf16. **Second blocker:** `moe_intermediate_size = 640` makes `ffn_down_exps` Q4_K-illegal on its reduction dim (640 % 256 = 128), as does `hc_lowrank = 320`; llama.cpp's substitution is believed to be Q5_0 (**UNVERIFIED, owed against the pinned llama.cpp oracle**) and the dependent fact IS verified in-tree — our reader knows ggml ids `0,1,2,8,10..14,16,18,19,22..28,30,39,40,41,66` and has **no entry for 3, 6, 7 or 20**, so a stock `llama-quantize -Q4_K_M` file fails at header parse. We author the converter, so the fix is Q4_0 (block 32, same 4.5 bpw). **`ENG-WEIGHT-OFFLOAD` will not help** — it moves zero bytes today (`ConsiderWeight` has no production callers, pinned by a test) and is documented inert on GB10; the tier that DOES work already ships and is proven by the 2.4T model serving 369.97 GiB from a 119.631 GiB box at ~62 GiB resident: mmap `MAP_PRIVATE`, borrow in place, alias the host pointer, `prefault: false`. Corrected sizing: backbone ~67.7 GiB, whole process ~73.5 GiB of 119.631 at 32K single-stream, ~46 GiB of headroom for the page cache, so the ~76 GB estimate was right within 10%. The design works because per-token demand is **<= 64 KiB of reads** (16 lookups x 160 dims over at most 16 pages) against the 2.4T expert lane's 6.95 GB/token. The architecture supplies its own lever: the per-token n-gram cost is `(ngram_size-1)*heads_per_ngram` = 16 lookups of 160 dims, so **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per token** and making it non-resident is the intended design point (RadixArk reached the same split independently). Sizing arithmetic, NOT measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB (yes, ~10 GB left for KV), Q4_K_M backbone with the table non-resident ~76 GB. GB10 is UNIFIED memory so "offload to host" is not a move there; non-resident means disk-backed, and its cost is unmeasured. **Two decisions were put to the developer as explicit accept-or-reject and BOTH are settled 2026-08-26, recorded in place rather than left open.** (1) `.agents/oracles/transformers.md` pins transformers to 5.14.1, deliberately tied to what the pinned vLLM environment resolves so the environment cannot hold two `transformers` at once, and **5.14.1 does not contain `Qwen4Exp`**; the lane-scoped second pin is **ACCEPTED**, on the argument that the invariant guards a vLLM environment against drifting from its transformers and here there is no vLLM implementation to drift from, and it expires the moment vLLM registers `qwen4_exp`. **The lane pin is a real release, not a branch SHA**, which was not the expected outcome: `Qwen4Exp` merged to `main` at 12:03:40Z on 2026-08-26 and `v5.16.0` published at 12:35:15Z, and this was BOUNDED rather than assumed by fetching `models/qwen4_exp/modeling_qwen4_exp.py` at each tag — `v5.16.0` HTTP **200**, `v5.15.0` HTTP **404** — making 5.16.0 the FIRST release carrying the architecture and therefore the tightest available pin. The version string is UNMEASURED (it is the release proven to contain the model, not a `transformers.__version__` read off a running oracle) and `gateable` stays `no`. (2) The first runnable arm is the **Q4_K_M backbone with the n-gram table NON-RESIDENT** (~76 GB). Q8_0 was raised and does not fit at ~191 GB, and no partial-Q8 split reaches 119 GB with the backbone at 8 bits; Q4_K_M-throughout fits on paper at ~109 GB but leaves ~10 GB for KV and activations on a 262144-native-context model, which is not a margin. This promotes the non-resident table from a note to a first-class W6 deliverable, and it is NOT free: GB10 is UNIFIED memory, so the existing host-pinned offload seam (`ENG-WEIGHT-OFFLOAD`, mirroring vLLM's `cpu_offload_gb`) does not by itself solve it there and the mechanism must be disk-backed or genuinely unloaded — established before it is designed around. Spec: [`specs/qwen4-exp-flash-next.md`](specs/qwen4-exp-flash-next.md). No product code lands under the spec pull request | feature | diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 766a3bed7..6f955c418 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -510,7 +510,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-qwen3-vl-moe-qwen3-vlmoe-for-conditional-generation` | `Qwen3VLMoeForConditionalGeneration` | `registry.py:552-555`; `vllm/model_executor/models/qwen3_vl_moe.py::Qwen3VLMoeForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder; video path | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | `Qwen3_5ForConditionalGeneration` | `registry.py:556`; `vllm/model_executor/models/qwen3_5.py::Qwen3_5ForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; GDN/linear-attention state; vision encoder; video path | 🚧 [family scoping](specs/mm-tools-scoping-2026-07-10.md); [plain-BF16 loader leaf](specs/qwen35-plain-bf16-direct-load.md); **[multimodal-track W-plan](specs/multimodal-track.md)**; full target spec required | `PARTIAL` (text-only) | text-only: `include/vllm/model_executor/models/qwen3_5_dense.h:40-105,146-171`; plain BF16/F32 + stacked/tied load `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:52-133,187-246,334-472`; plain execution `src/vllm/model_executor/models/qwen3_5.cpp:1993-2003,4579-4585,5249-5255,5533-5545`; loader route/queue reuse `src/vllm/entrypoints/model_loader.cpp:364-400`; real 4B gate `tests/vllm/models/test_qwen35_plain_weights.cpp:80-196`: CPU topology/load **1656/1656**, AOT CUDA direct OFF/ON full-engine token equivalence **1664/1664**. Existing W3-G immutable `ae9e8ff` default/fallback each pass **235/235 + 16/16** with the frozen 64 plans. Corrected root `/tmp/qwen35-transplant-4b-aot-557ab41d` proves ON=OFF 128/128 and records ON/OFF/vLLM total **6155.10/6064.06/6730.46 tok/s**, peak PSS **2.405/8.571/7.569 GiB**; current ON is 0.9316x historical AOT ON. Current-v0.25 oracle, sanitizer, vision, strict VRAM and external 27B/35B regressions remain unverified, with no new support claim. **MM-completion plan ([multimodal-track.md](specs/multimodal-track.md), `CLAIM-MULTIMODAL-TRACK`, 2026-07-25):** modalities = image + video (NO audio); reuses the landed GDN-hybrid text path — the mm half is the shared `Qwen3_VisionTransformer` (DeepStack, `qwen3_vl.py:519`) stood up on Qwen3-VL-4B first (M2) then attached to this wrapper (M3). Oracle-runnable (0.25.0 ships `qwen3_5.py`+`qwen3_vl.py`); NOT HW/oracle-blocked but **CHECKPOINT-gated** — the cached `unsloth/Qwen3.6-27B-NVFP4` quant is TEXT-ONLY (2111 tensors, ZERO `visual.*`; `vision_config` declared but weights absent), so a vision-inclusive checkpoint download is required (M0). Tower ~0.5-0.7 B params (~1-1.4 GiB bf16) fits GB10 trivially alongside the 27B. Plan owner `CLAIM-MULTIMODAL-TRACK` (row stays PARTIAL/narrative-only; the mm work re-claims it at M3). **M3-b LANDED 2026-07-25 (`CLAIM-MULTIMODAL-M3B`): IMAGE e2e WORKING** — `Qwen3_5VLGenerateGreedy` (`src/vllm/model_executor/models/qwen3_5.cpp`) forks the GDN-hybrid forward on inputs_embeds(scatter tower merger `[196,5120]` into image_token 248056 rows, no deepstack) + 3-section MRoPE `[11,11,10]` interleaved in the 16 full-attn layers (host `BuildMropeCosSinHost` → the `mrope_cos_sin` param on `DenseForwardLayers`, nullptr on text ⇒ byte-identical); vision loader `LoadQwen3VLVisionWeights` (`src/vllm/model_executor/models/qwen3_vl.cpp`, 27B config) + M2a tower + `LoadQwen3_5Dense` bf16 LLM. STRICT gate `tests/vllm/multimodal/test_qwen3_5_vl_e2e.cpp` **32/32 token-exact vs vLLM 0.25.0** (sha256 `ead4b484…`); text-inertness re-run cutlass-ON 27B/35B/Coder **235/315/138**. **M3d LANDED 2026-07-25 (`CLAIM-MULTIMODAL-M3D`): VIDEO e2e WORKING** — `Qwen3_5VLGenerateGreedyVideo` (`src/vllm/model_executor/models/qwen3_5.cpp`) reuses the M3-b image driver via a shared `VLGenerateCoreGdn` (video merge mask on video_token 248057 + `Qwen3VLGetRopeIndexVideo` per-frame temporal MRoPE; M3c processor/windowed-tower reused verbatim; no deepstack). STRICT gate `tests/vllm/multimodal/test_qwen3_5_vl_video_e2e.cpp` **32/32 token-exact vs vLLM 0.25.0** (oracle `scripts/mm/m3d_video_oracle_capture.py`, K=5 deterministic, near-tie gaps 0.0000); image e2e re-run STRICT 32/32 (refactor-safe); text SACRED byte-identical by construction (shared forward untouched). **Qwen video modalities COMPLETE (image+video e2e; audio N/A); speed still pending** (row stays PARTIAL). Speed lever #2 CLOSED 2026-07-27 (`CLAIM-MULTIMODAL-SPEED-DECODE`, multimodal-speed.md §8): on-GPU greedy argmax + decode embed round-trip removed on the shared `VLGenerateCoreGdn`; bit-exact (image+video STRICT 32/32 held, goldens md5-identical); 27B decode TPOT NEUTRAL (223 ms, ~222 ms bandwidth floor, at vLLM parity). **Speed lever #3 FIRST BRICK 2026-07-27 (`CLAIM-MULTIMODAL-SPEED-GRAPH`, multimodal-speed.md §9): the shared `VLGenerateCoreGdn` decode step now routes through the production `Qwen3_5DenseDecodeGraph` (cold→warm→replay captured decode) — mm decode is GRAPH-CAPTURABLE (was eager per-step). S==B==1 bit-identical rebuild; the decode-time 1-D device RoPE at p reproduces the degenerate MRoPE {p,p,p} → token-exact HELD (image+video STRICT 32/32, 30 graph replays confirmed); A/B graphed 232.5 vs eager 233.4 ms/tok = NEUTRAL at the 27B bandwidth floor. Structural gap closed; W-plan = Voxtral decode-graph (audio 1.52× gap-closer) + batched c2+ + serving ingestion. Row stays PARTIAL/speed-pending.** | unassigned | | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` | `Qwen3_5MoeForConditionalGeneration` | `registry.py:557-560`; `vllm/model_executor/models/qwen3_5.py::Qwen3_5MoeForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; GDN/linear-attention state; vision encoder; video path | 🚧 [family scoping](specs/mm-tools-scoping-2026-07-10.md); **[multimodal-track W-plan](specs/multimodal-track.md)**; target spec required | `PARTIAL` (text gated, vision NOT gated) | text-only: `include/vllm/model_executor/models/qwen3_5.h:1-17,97`; direct registry `src/vllm/model_executor/models/registry.cpp:10-20`; gate `tests/parity/test_qwen36_paged_engine.cpp:78,140`. W3-G immutable `ae9e8ff` correctness-only ratio-8 inertness passes **2/2 + 315/315**; no 35B performance claim; vision not implemented AT THAT DATE (it landed later, see M2/M3 below, and is still NOT gated). Disk load now DEFERS the routed-expert host copies and streams+frees them per layer during `PrepareMarlinResident` to bound load-phase peak PSS (`ENG-MOE-LOADSTREAM`, engine-matrix; CPU-gated, DGX pending) — device residents byte-identical. **MM-completion plan ([multimodal-track.md](specs/multimodal-track.md), `CLAIM-MULTIMODAL-TRACK`, 2026-07-25):** image + video (NO audio); same shared `Qwen3_VisionTransformer` as the 27B row, attached to the landed MoE GDN-hybrid text path (M3). Oracle-runnable (0.25.0); **CHECKPOINT-gated** — the cached `nvidia/Qwen3.6-35B-A3B-NVFP4` quant is TEXT-ONLY (`vision_config` declared, `visual.*` weights absent); vision-inclusive download required (M0). Tower fits GB10 alongside the 35B MoE per the landed text run. Plan owner `CLAIM-MULTIMODAL-TRACK` (row stays PARTIAL/narrative-only; the mm work re-claims it at M3). **M2/M3 LANDED (#891, `.agents/specs/moe-vision-tower.md`):** the loader no longer drops the checkpoint's 333 `model.visual.*` tensors (`LoadQwen3_5MoeVision` -> the SHARED `LoadQwen3VLVisionWeights` the dense arm is gated on; their ABSENCE is refused by name), and `Qwen3_5MoeVLGenerateGreedy[Video]` forks the forward gated on mm input over a greedy core now TEMPLATED on the weights arm rather than copied. Evidence: CPU suite 479/479 serial; the new `test_qwen3_5_moe_vision` proves the forked forward reduces EXACTLY to the text forward over the tower row (one visual token, 1x1x1 LLM grid) and that MRoPE is applied (8x8 grid must DIFFER from the 1-D run), with 4 mutations driven RED and restored byte-exact; on Thor (sm_110, FALLBACK attention) `test_qwen3_5_moe_vision_hw` loads the real 333 tensors and runs the tower on the fixture image. **OWED: the binding image and video token-exact gates vs the pinned oracle at 35B.** Not runnable on Thor -- vLLM cannot import there (`libcuda.so.1` absent on the host, `torch.cuda.is_available()` False) and the bf16 35B is ~67 GiB against this box's documented 25 GB single-model reboot ceiling; dgx.casa was off-limits mid-run for a sibling row. **TEXT ARM ORACLE-GATED ON THE PUBLISHED BF16 REPO 2026-08-15 ([#740](https://github.com/mudler/vllm.cpp/issues/740) + [#864](https://github.com/mudler/vllm.cpp/issues/864)), and this changes NOTHING about the vision claim:** greedy 7 prompts x 3 repeats x 16 tokens on `Qwen/Qwen3.6-35B-A3B` bf16 @`995ad96eacd98c81ed38be0c5b274b04031597b0` vs the pinned oracle gave **6/7 prompts STRICT 16/16**, the seventh one exact logit tie (`top2_gap_mnats = 0.0`) our on-device argmax breaks toward the higher id ([#910](https://github.com/mudler/vllm.cpp/issues/910)); only the FIRST divergence per prompt is adjudicable, so the raw 108/112 position count is NOT a quality score. SACRED inertness 3/3, goldens byte-identical (27B 235/235, 35B 315/315, Coder 138/138). NO throughput, latency or memory number exists for this checkpoint. The row therefore stays PARTIAL: **the binding image and video token-exact gates at 35B are still OWED**, the vision claim remains "the tower loads and computes" rather than "produces correct tokens", and the sm_110 run that proved it used the FALLBACK attention path, which is not coverage of the shipped GB10 path. [#908](https://github.com/mudler/vllm.cpp/issues/908)'s dense regression check is PARTIAL too: dense TEXT is 235/235 at `2f2bce926`, a true before/after (binary md5 `db889909d4…` vs `49ded1ece8…`, 500 TUs recompiled), while dense image/video stays UNVERIFIED (network-blocked) | unassigned | -| `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | `Qwen4ExpForConditionalGeneration` (`model_type: qwen4_exp`; campaign row `MODEL-MM-QWEN4-EXP`) | **NOT IN vLLM AT ANY REVISION** — deliberately written with no pinned module/class target, the convention `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` follows for a beyond-pin arm, and stronger here: this is absence from vLLM `main` rather than staleness in `555967922`. Read live 2026-08-26 at `origin/main` = `6a5e8f5979`: no `qwen4*` path, no `registry.py` entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. Algorithm source is [transformers#48337](https://github.com/huggingface/transformers/pull/48337) `models/qwen4_exp/modular_qwen4_exp.py`, MERGED 2026-08-26 | conditional generation / text + image + video | MM processor; vision encoder (UNCHANGED from `Qwen3_5MoeVisionModel`); GDN/linear-attention state; block-sparse attention + indexer side cache; FusedMoE/grouped GEMM; hyper-connection residual streams; hashed n-gram embedding; dilated depthwise conv; MTP | ✅ [Qwen3.8-Flash-Next](specs/qwen4-exp-flash-next.md) | `READY` | **SPEC ONLY, NO PRODUCT CODE, NO TOKEN, NO SPEED.** `Qwen/Qwen3.8-Flash-Next` (2026-08-24, 180B total / 6B activated). Split oracle by developer direction 2026-08-26: **transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH**, because `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so the reference is semantics and not a serving path. `Qwen4ExpTextModel` inherits `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`); GDN matches our AOT gate exactly (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). **Exactly two components have NO vLLM op**: the PLE dilated depthwise conv (`git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the hashed n-gram embedding. **QSA's twin in vLLM is MiniMax-M3, NOT DeepSeek-V4** — DSA is MLA, QSA is plain GQA (24 Q / 2 KV, `head_dim` 256); `models/minimax_m3/common/indexer.py` is the non-MLA block-sparse case with `common/ops/index_topk.py`'s `_index_block_score_kernel` + bitonic `_topk_index_kernel`, and the pooled-key build is `deepseek_v4/nvidia/ops/sparse_attn_compress_cutedsl.py`'s `SparseAttnCompressNormRopeStoreC4Kernel`. Residual stream is `hc_count * hidden_size` = 4 x 2560 = **10240 wide through the whole stack**; `number_of_conv_states = 3` on a PLE layer plus the indexer side cache. **NOTHING PUBLISHED FITS ~119 GB on GB10**: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB, `unsloth/...-GGUF` is a README with ZERO weight files and llama.cpp has no `qwen4_exp` either, so the standing k-quant arms must be authored here and have NO llama.cpp oracle. Sizing ARITHMETIC and not measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB, Q4_K_M backbone with the 51 GB n-gram table non-resident ~76 GB — that table is 28% of the model and is touched 16 times per token, which is the offload the card itself argues for. `gateable = no` until an arm runs. Both blocking decisions SETTLED 2026-08-26: the transformers lane pin is **ACCEPTED at 5.16.0** ([`oracles/transformers.md`](oracles/transformers.md)), because the registry pin 5.14.1 does not contain `Qwen4Exp` — and 5.16.0 is a real release rather than a SHA, bounded by fetching the model file at each tag (`v5.16.0` HTTP 200, `v5.15.0` HTTP 404), with the version string UNMEASURED until an oracle stands up; and the first runnable arm is the **Q4_K_M backbone with a NON-RESIDENT n-gram table** (~76 GB), Q8_0 having been raised and rejected on ~191 GB. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | unassigned | +| `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | `Qwen4ExpForConditionalGeneration` (`model_type: qwen4_exp`; campaign row `MODEL-MM-QWEN4-EXP`) | **NOT IN vLLM AT ANY REVISION** — deliberately written with no pinned module/class target, the convention `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` follows for a beyond-pin arm, and stronger here: this is absence from vLLM `main` rather than staleness in `555967922`. Read live 2026-08-26 at `origin/main` = `6a5e8f5979`: no `qwen4*` path, no `registry.py` entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. Algorithm source is [transformers#48337](https://github.com/huggingface/transformers/pull/48337) `models/qwen4_exp/modular_qwen4_exp.py`, MERGED 2026-08-26 | conditional generation / text + image + video | MM processor; vision encoder (UNCHANGED from `Qwen3_5MoeVisionModel`); GDN/linear-attention state; block-sparse attention + indexer side cache; FusedMoE/grouped GEMM; hyper-connection residual streams; hashed n-gram embedding; dilated depthwise conv; MTP | ✅ [Qwen3.8-Flash-Next](specs/qwen4-exp-flash-next.md) | `READY` | **SPEC ONLY, NO PRODUCT CODE, NO TOKEN, NO SPEED.** `Qwen/Qwen3.8-Flash-Next` (2026-08-24, 180B total / 6B activated). Split oracle by developer direction 2026-08-26: **transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH**, because `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so the reference is semantics and not a serving path. `Qwen4ExpTextModel` inherits `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`); GDN matches our AOT gate exactly (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). **Exactly two components have NO vLLM op**: the PLE dilated depthwise conv (`git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the hashed n-gram embedding. **CORRECTED after the component deep-dives: QSA's twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The first reading rested on `MLAAttentionSpec` being an MLA claim; it is a per-state BUDGET shape, and M3 — itself plain GQA — uses it. Nine structural matches with DSv4 including `compress_ratio == 4`, relu-summed MQA scoring, RoPE at the block-start position and `tokens_per_state=compress_ratio`. M3 scores by `max` over 128 RAW dots, emits one set per KV head, and welds `SPARSE_BLOCK_SIZE = 128` to the KV page size. The genuinely NEW work is the consumer: DSv4 attends COMPRESSED KV, M3 attends raw at page granularity, QSA attends RAW at ratio-4 granularity and nothing upstream does that. Any QSA gate must exceed 2048 tokens of context or it cannot distinguish a correct port from one attending pooled keys. Residual stream is `hc_count * hidden_size` = 4 x 2560 = **10240 wide through the whole stack**; `number_of_conv_states = 3` on a PLE layer plus the indexer side cache. **NOTHING PUBLISHED FITS ~119 GB on GB10**: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB, `unsloth/...-GGUF` is a README with ZERO weight files and llama.cpp has no `qwen4_exp` either, so the standing k-quant arms must be authored here and have NO llama.cpp oracle. Sizing ARITHMETIC and not measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB, Q4_K_M backbone with the 51 GB n-gram table non-resident ~76 GB — that table is 28% of the model and is touched 16 times per token, which is the offload the card itself argues for. `gateable = no` until an arm runs, and **the chosen arm does not load today**: `KeepQuantKDim` returns `-1` for `kEmbeddingTable`, so a quantized n-gram table expands to bf16 (102.4 GB) and dies at load; `kKeepF16` is the only non-expanding gather residency and is CPU-only. Also `moe_intermediate_size = 640` / `hc_lowrank = 320` are Q4_K-illegal on their reduction dims and our reader cannot open ggml types 3/6/7/20, so the converter must emit Q4_0. `ENG-WEIGHT-OFFLOAD` is inert and does not help; the shipping mmap-borrow tier does. Both blocking decisions SETTLED 2026-08-26: the transformers lane pin is **ACCEPTED at 5.16.0** ([`oracles/transformers.md`](oracles/transformers.md)), because the registry pin 5.14.1 does not contain `Qwen4Exp` — and 5.16.0 is a real release rather than a SHA, bounded by fetching the model file at each tag (`v5.16.0` HTTP 200, `v5.15.0` HTTP 404), with the version string UNMEASURED until an oracle stands up; and the first runnable arm is the **Q4_K_M backbone with a NON-RESIDENT n-gram table** (~76 GB), Q8_0 having been raised and rejected on ~191 GB. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | unassigned | | `MODEL-MM-rvl-rfor-conditional-generation` | `RForConditionalGeneration` | `registry.py:561`; `vllm/model_executor/models/rvl.py::RForConditionalGeneration` | conditional generation / video+image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-skyworkr1v-skywork-r1-vchat-model` | `SkyworkR1VChatModel` | `registry.py:562`; `vllm/model_executor/models/skyworkr1v.py::SkyworkR1VChatModel` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-smolvlm-smol-vlmfor-conditional-generation` | `SmolVLMForConditionalGeneration` | `registry.py:563`; `vllm/model_executor/models/smolvlm.py::SmolVLMForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index b089d3449..02aadc606 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -162,10 +162,10 @@ of that. The port is the delta below. | Component | Algorithm oracle | Op oracle (vLLM) | This tree | |---|---|---|---| | GDN linear attention | `Qwen4ExpTextGatedDeltaNet` | `layers/mamba/gdn/qwen_gdn_linear_attn.py` | **HAVE.** `K=V=128, Hg=16, Hv=48` is an exact match for the AOT gate in `TryTritonPackedDecode` / the delta_h dispatch (`src/vt/cuda/cuda_gdn.cu`, pinned to `K=V=128, Hg=16, H in {48,32}`) | -| Grouped RMSNorm | `Qwen4ExpTextRMSNorm(group_size=)` | `layers/layernorm.py` `group_size` | new, small; mirror vLLM's form | -| QSA block scoring + top-k | `Qwen4ExpTextQSAIndexer` | `models/minimax_m3/common/indexer.py`, `common/ops/index_topk.py`, `common/sparse_attention.py` | new | -| QSA pooled-key build | indexer forward | `models/deepseek_v4/nvidia/ops/sparse_attn_compress_cutedsl.py` (`SparseAttnCompressNormRopeStoreC4Kernel`, carries `compress_ratio`) | partial: `deepseek_v4_compressor.h` | -| Indexer side cache | `Cache.update_indexer` | `MiniMaxM3IndexerCache`, `v1/attention/backends/mla/indexer.py` | new KV spec | +| Grouped RMSNorm | `Qwen4ExpTextRMSNorm(group_size=)` | `layers/layernorm.py` **`RMSNormGated`** (`group_size`), NOT the plain `RMSNorm` | new, small; see the correction below | +| QSA block scoring + top-k | `Qwen4ExpTextQSAIndexer` | **DeepSeek-V4 C4 indexer lane**: `fp8_mqa_logits` / `top_k_per_row`, `v1/attention/backends/mla/indexer.py`. NOT MiniMax-M3, see below | new | +| QSA pooled-key build | indexer forward | the **Triton** `head_dim=128` compress/norm/RoPE/store kernel with `OVERLAP=False` and the pool replaced by a mean. NOT the CuteDSL `SparseAttnCompressNormRopeStoreC4Kernel`, which refuses `overlap=False` and pools by learned softmax over 8 | partial: `deepseek_v4_compressor.h` | +| Indexer side cache | `Cache.update_indexer` | `MLAAttentionSpec(num_kv_heads=1, head_size=128, tokens_per_state=4)` + `get_compressed_slot_mapping`, as-is; M3 supplies only the registration precedent | new KV spec | | Gated Residual | `Qwen4ExpTextGatedResidual` | `layers/mhc.py`, `kernels/mhc/*` (**different math**, same fused shape) | partial: `deepseek_v4_mhc.cpp` | | MoE 512 / top-10 + 1 shared / intermediate 640 | `Qwen4ExpTextSparseMoeBlock` | FusedMoE, grouped GEMM | HAVE, shape change only | | MTP, 1 layer, `hybrid: true` | config `mtp` | `qwen3_5_mtp.py` | HAVE, needs extension | @@ -177,24 +177,83 @@ of that. The port is the delta below. is the sole source and we author the kernel ourselves. Everything else has an optimized vLLM form to mirror, and mirroring it is mandatory rather than optional. -### QSA maps to MiniMax-M3 - -DSA is an **MLA** indexer. QSA is not MLA: plain GQA, 24 Q heads, 2 KV heads, -`head_dim` 256, `partial_rotary_factor` 0.25 giving 64 rotary dims. MiniMax-M3's -lightning indexer is vLLM's **non-MLA** block-sparse case, and its docstring -describes QSA's shape exactly: it "scores KV blocks with the index heads and selects -the top-k blocks ... that the main block-sparse attention then attends to", owning -"its own side cache (one index-key vector per token)". `common/ops/index_topk.py` -supplies `_index_block_score_kernel` and a bitonic `_topk_index_kernel`. - -Reconciliation the implementer owes, not hand-waved here: M3's sparse block size is -128 where QSA's `indexer_compress_ratio` is 4 with `block_topk = indexer_budget / -compress_ratio = 512`; and QSA's per-block key is a **mean pool over the block**, -then `k_layernorm`, then RoPE at the block-start position, scored as -`relu(q . k).sum(over 4 index heads) / sqrt(128)`. The pooled-key construction is -what `SparseAttnCompressNormRopeStoreC4Kernel` already does under a different name. - -`indexer_kv_heads` must be 1; the upstream config validator rejects anything else. +### QSA maps to DeepSeek-V4's C4 indexer lane, NOT to MiniMax-M3 + +**This reverses the call this spec was first written with, and the reversal is the +most important thing in the document.** The original reading was that QSA, being plain +GQA rather than MLA, had to map onto vLLM's non-MLA block-sparse case (MiniMax-M3) and +not onto DeepSeek's DSA. That reasoning was wrong, and it was wrong for a specific, +checkable reason recorded here so it is not repeated: **`MLAAttentionSpec` is not an +MLA claim.** M3's own indexer cache uses it while being a plain-GQA model, and the +comment beside it says why -- "Key-only: MLAAttentionSpec budgets one vector/token (not +2x for K+V)". It is a budget shape, not an architecture assertion. Once that prop is +removed, the GQA-versus-MLA argument for preferring M3 collapses entirely. + +Verified at vLLM `origin/main` = `6a5e8f5979`. + +**Nine independent structural matches with DeepSeek-V4**, and `compress_ratio == 4` is +literally the same number: + +| QSA (transformers v5.16.0) | DeepSeek-V4 (vLLM) | +|---|---| +| index MQA, 1 key head, dim 128 | index MQA, 1 key head, dim 128 | +| score `relu(q.k)` summed over index heads | `(score.relu() * weights).sum(dim=0)` | +| scale `1/sqrt(indexer_head_dim)` | `softmax_scale = head_dim ** -0.5` | +| one score set per query token, no head axis | `topk_indices_buffer[num_tokens, topk]` | +| pool `compress_ratio` tokens into one key | boundary `(position + 1) % COMPRESS_RATIO == 0` | +| `k_layernorm` on the pooled key | RMSNorm on the compressed key | +| RoPE at the **block-start** position | `compressed_pos = (position // CR) * CR` | +| candidates = `visible // compress_ratio` | `len_per_token = (start_pos + 1 + offset) // CR` | +| one stored state per 4 tokens | `MLAAttentionSpec(tokens_per_state=compress_ratio)` | + +`tokens_per_state` is a first-class KV-cache field upstream, documented as "Ints > 1 +compress multiple tokens into one state (DSv4 sparse MLA)". It is exactly what QSA's +side cache needs, and it does not exist on the M3 path. + +**Why M3 is not merely a worse fit but a different algorithm.** Its score is +`tl.max(qk, axis=1)` over 128 **raw** token dots -- no pooling stage, no relu, no head +reduction -- and it asserts `num_idx_heads == num_kv_heads` with the comment "no topk +index reduce", so it produces one independent block set **per KV head** where QSA +produces one set per token. Its `SPARSE_BLOCK_SIZE = 128` is not a tunable: the file +states "One sparse block == one KV page", and both the score and the attend index +`block_table[blk]` on that identity. Moving it to 4 would force a KV page size of 4 and +break `tl.dot`, whose tile needs at least 16. + +**M3 still contributes exactly one thing, and it is a wiring precedent rather than an +algorithm:** the demonstration that a plain-GQA model can own a key-only side cache +through `MLAAttentionSpec` and a private indexer backend registered into +`static_forward_context`. Take that pattern; take no kernel. + +**The genuinely new work is the CONSUMER, and nothing upstream supplies it.** Every +DSv4 sparse consumer attends to the **compressed** MLA KV, one state per four tokens. +M3's consumers attend to raw tokens but only at page granularity. QSA attends to **raw +tokens selected at ratio-4 granularity**, which no vLLM consumer does. The port has to +expand block id `b` into tokens `[4b, 4b+4)`, append the ragged tail, and run dense GQA +(24 query heads over 2 KV heads, `head_dim` 256) across the gathered set. + +**Two silent-failure traps follow, and both would pass a naive gate.** + +1. Wiring QSA's top-k straight into a DSv4 sparse-MLA consumer attends to a **pooled** + key and value instead of the four real tokens. It still produces plausible output. + A short-prompt token gate cannot catch it, because at context <= `indexer_budget` + every candidate is selected and the only remaining difference is the value pooling. + Any QSA gate must therefore run past 2048 tokens of context to be worth anything -- + a requirement this spec did not previously state and which changes what `## Gates` + has to demand. +2. `SparseAttnCompressNormRopeStoreC4Kernel` does **not** mean-pool, despite being the + closest-named kernel. It is a learned softmax-weighted pool over an **overlapping + window of 8** driven by a score channel QSA's checkpoint does not have, and the + CuteDSL variant refuses `overlap=False` at compile time. Its scaffolding is a direct + match -- boundary predicate, block-start RoPE, paged store -- but the pooling + operator must be replaced with an unweighted mean over a non-overlapping window of + 4. The **Triton** `head_dim=128` variant, where `OVERLAP` is a plain `constexpr`, is + the correct starting point; the CuteDSL C4 one is not. + +Also reconcile, and do not inherit: DSv4 has a `weights_proj` producing per-head logit +weights that QSA has no tensor for (QSA's weight is the constant `1/sqrt(128)`), and +its RoPE is GPT-J-style over a trailing contiguous span, whereas QSA uses interleaved +mRoPE over the **leading** 64 dims with the NoPE dims trailing -- the halves are +swapped end for end. ### Two structural consequences beyond the module list @@ -214,14 +273,166 @@ what `SparseAttnCompressNormRopeStoreC4Kernel` already does under a different na ### The n-gram embedding is integer-exact or it is silently wrong -Head vocab sizes are successive **primes** found after `ngram_vocab_size_base - 1` -(20,000,000), indexed by a global head index; IDs are built by XOR-mixing -splitmix64-seeded per-position multipliers over shifted token windows, then reduced -modulo each head's prime. An off-by-one in the prime search, or a 32-bit truncation -anywhere in the mix, yields a valid-looking lookup into the wrong row. Nothing -downstream detects it, and a token gate against a model this size will not localise -it. Gate the ID construction directly, against transformers, before any weight is -loaded. +Derived from the lane pin and then **verified against the published checkpoint** by +range-reading the safetensors payload, so these are read values and not predictions. + +`config.seed` is absent from the published config, so the dataclass default **1234** +applies. That was confirmed rather than assumed: reconstructing the splitmix64 chain +at seed 1234 gives `layer_multipliers = [23703573157769, 20109073645365, +8052911324071]`, and a range read of that buffer out of +`model-00005-of-00131.safetensors` returns those three values exactly. + +Head vocab sizes are the successive primes after `ngram_vocab_size_base - 1`, so head +0 is 20000003 and head 15 is 20000171; `total_vocab_size = 320001446`, padded to +**320001536** (90 unaddressable rows), giving `320001536 x 160 = 51,200,245,760` +parameters. `ngram_heads_vocab_sizes` and `ngram_heads_offsets` were range-read from +the checkpoint and match the derivation entry for entry. + +**The three C++ divergence sites, ranked.** All are silent. + +1. **`_splitmix64` must be `uint64_t` throughout.** Its `>> 30 / 27 / 31` are logical + shifts on a non-negative Python int; on a signed `int64_t` they become arithmetic + shifts and the multiplier is wrong. The value has its top bit set about half the + time, so this fires immediately. +2. **`_splitmix64(value) % half_bound` must be an unsigned modulo.** The dividend + routinely exceeds 2^63; a signed modulo yields a negative residue. +3. **Shard reassembly must be NUMERIC, not lexicographic.** The table ships as 128 + shards, `shard_0 .. shard_127`, each bf16 `[2500012, 160]`. Sorting the key strings + gives `shard_0, shard_1, shard_10, shard_100, ...` and silently permutes a 95 GiB + table. Verified against the checkpoint index: 128 keys, contiguous 0..127, and a + lexicographic sort does produce that wrong order. + +The forward itself is int64-exact and does NOT depend on Python bignum: +`layer_multipliers[i]` is a 0-dim int64 tensor, so the product is int64 arithmetic, +and it is bounded below 2^63 by construction because `multiplier_max * vocab_size <= +2^63 - 1`. **That bound holds only while every token id is below `vocab_size`.** An +out-of-range id overflows int64 and diverges silently, so the loader must not admit +one. Because `mixed` is therefore always non-negative, Python's `%` and C's truncating +`%` agree, and a port may use `int64_t %` without a sign correction. + +Two further traps found in the cache path. The history of the previous +`ngram_size - 1` token ids is stored in the linear-attention cache as conv state 2 and +its dtype is **int64**, taken from the first tensor written; a port that stores it as +a float rounds token ids. And upstream's `update_conv_state` pads with **0**, which is +a valid token id, so the model works around it with an explicit EOS left-pad. Pad with +EOS, never with zero. + +`split_ngram_parts` is **not used in the forward at all**. It is a checkpoint-layout +parameter consumed only by the weight conversion mapping, and saying so here stops the +next reader hunting for it in the model code. + +**The PLE sits on decoder layer 1, not layer 2.** `ple_layer_ids` is 1-indexed and the +lookup is `config.ple_layer_ids.index(layer_idx + 1)`, so `[2]` selects 0-based layer +1. Confirmed from the checkpoint index: every PLE tensor is under +`model.language_model.layers.1.ple.`, and no other layer has one. + +### PLE: a strided-history conv with no vLLM op, confirmed + +**The dilated depthwise conv has no counterpart anywhere in vLLM, and the search is a +confirmed negative rather than an unfound one.** At `origin/main` = `6a5e8f5979`, +`git grep -in dilat` returns 17 lines tree-wide and **zero** in +`vllm/model_executor/layers/mamba/`, **zero** in `csrc/`, and **zero** in `tests/`. +`layers/conv.py` defines only `Conv2dLayer` and `Conv3dLayer`; there is no +`Conv1dLayer`, and the Transformers-backend auto-replacement maps only `nn.Conv2d` and +`nn.Conv3d`, leaving any `nn.Conv1d` as a bare PyTorch module. Upstream reached the +same conclusion from the other side and hand-rolled it, with the comment "We cannot use +the usual functions/kernels here for the short conv as the conv1d has dilation". + +`causal_conv1d_fn` / `causal_conv1d_update` are disqualified on four independent +counts: they take no dilation argument; their Triton state loads unroll the taps at +unit stride in the kernel source; `state_len` is `width - 1` throughout the shape +plumbing where PLE needs `(width - 1) * dilation`; and vLLM has no `state_idx` concept, +so one layer cannot own three independently addressed conv states. + +**The conv is strided history, not a local window.** `kernel_size=4`, `dilation=3`, +so output position `t` reads tokens at lags **{9, 6, 3, 0}** — a span of 10 tokens for +4 multiply-accumulates per channel, and the lag-0 tap makes it causal. The state is +therefore a genuine 9-deep ring buffer read at stride 3, and it cannot be compressed to +3 columns even though any single step touches only three of them. + +Cost: 9 columns x 10240 channels = **~180 KiB per sequence at bf16 for this one +layer**. That is a real KV-budget line item, not a rounding error, and it belongs in +the `## Hardware` accounting once measured. + +What the state holds is the **normed** conv input (`norm_conv`'s output), not the raw +hidden state and not the conv output. The layer forks: the skip term is the +**un-normed** `gated_value`, and only the normed copy enters the conv. + +**The signed-sqrt gate has a trap in the clamp order.** It is +`gate.abs().clamp_min(1e-6).sqrt() * gate.sign()`, so the clamp applies **before** the +square root and the floor on the output magnitude is `sqrt(1e-6) = 1e-3`, not `1e-6`. +Tiny scores are **amplified** to +/-1e-3 rather than squashed. Exactly zero maps to +zero, because `sign(0) = 0`, so the function is genuinely discontinuous at the origin +and that is reachable on a fully masked row. Mirror it; do not tidy it. A port that +clamps after the sqrt is wrong by three orders of magnitude in that band. + +**A GDN state-length disagreement to reconcile at the seam.** Upstream sizes the GDN +conv state as `linear_conv_kernel_dim` = **4**, where vLLM's shape calculators use +`conv_kernel - 1`. The two conventions differ by one column, and nothing will announce +the mismatch. + +**`ple_layer_ids` is one-indexed by design, not by accident.** The docstring says +"One-indexed", the config validator rejects ids outside `[1, num_hidden_layers]` and +resolves the layer type as `layer_types[layer_id - 1]`, and +`test_ple_layers_must_use_linear_attention` pins it. Do not "fix" it. + +Padding is a paired obligation: the activations are masked **and** `ple_input_ids` has +its padded positions overwritten with EOS before the layer runs, because the n-gram +hash reads token ids rather than activations. Masking only the activations leaks +padding into the hash. `conv_mask` is `None` in steady-state decode, so the masking +lines are prefill-only. + +One item is **AMBIGUOUS and must not be resolved from upstream**: the conv state +written during a chunked prefill whose first chunk is shorter than 9. Upstream +zero-pads on the left, which is arithmetically identical to what a single-shot prefill +would do, and its cache never reuses a prefix from another sequence. A prefix-caching +scheduler has to decide whether a cache hit restores the true 9-column state or re-pads +with zeros. That is our design question, not upstream's. + +### Gated Residual: what our MHC actually gives us + +The reuse verdict is sharper than "same shape, different math". Buffer plumbing is +largely reusable: the `[T, hc, H]` manifold, the layer-0 broadcast widen (upstream's +`hidden_states.repeat(1, 1, hc_count)` is exactly our broadcast), the per-token loop, +and the read/collapse/write-back cadence twice per layer. Three specifics: + +- **`MhcPost` is bit-exact reusable for Qwen's write-back with the comb matrix set to + identity.** The sum collapses to one non-zero term, so there is no reduction-order + difference. It is 3x wasteful on the residual read and must not ship that way, but + it gives a bring-up bridge from a kernel that is already gated, which is a free + mutation target for the fresh reviewer. +- **`MhcSinkhorn` is dead here.** Qwen has no doubly-stochastic mixing and no comb + matrix at all, so the carried `res_mix [T, hc, hc]` buffer goes with it. Qwen's + streams couple only on the READ path, through the shared low-rank projection. +- `MhcPre` and `HcHeadCollapse` share a skeleton and no arithmetic: their norm is + weight-free and global where Qwen's is grouped and weighted, their projection is one + dense matrix where Qwen's is a two-stage low-rank `10240 -> 320 -> 10240`, their gate + is per-stream scalar where Qwen's is per-element, and their reduce is a **sum** where + Qwen's is a **mean**. Qwen also needs no separate head-collapse op: the final + collapse is the same class with the injection branch switched off. + +**`Qwen4ExpTextModel` has no final RMSNorm.** The mixer's own `hc_norm` is the last +normalization before `lm_head`. A port that copies our DeepSeek-V4 tail will insert one +that does not exist. Stated because that tail is the natural thing to copy. + +**Weight parameterization differs from vLLM's op form.** Upstream applies +`output * (1.0 + weight)` with `weight` zero-initialized; vLLM's grouped norm applies +`out * weight` with `weight` ones-initialized. They coincide under a load-time +`w_vllm = 1.0 + w_hf`. Miss it and every `hc_norm` gets a near-zero scale, which reads +as a checkpoint bug rather than a port bug. + +**Correction to the port map above.** vLLM's grouped RMSNorm is on **`RMSNormGated`**, +not the plain `RMSNorm`, whose only related knob is `var_hidden_size` -- a prefix +reduction that cannot express per-group norms. Verified directly: `RMSNorm` opens at +`layernorm.py:37` and `RMSNormGated` at `:172`, and the `group_size` parameter is at +`:187`. A porter reaching for `RMSNorm` finds nothing. Separately, `RMSNormGated`'s +`forward_cuda` dispatches to a flash-linear-attention Triton kernel rather than the +native reference, and that kernel's grouped numerics are **unverified**; whoever writes +the device arm owes that check. + +The hyper-connection tower is **~640 M dense parameters** at this config (two modules +per layer x 48, plus the mixer), unquantized in the published scheme and read twice per +layer. That is a memory and bandwidth line item, not only a correctness one. ## Dependencies @@ -267,14 +478,15 @@ issue per AGENTS.md "Nothing lands dead". - **W4, QSA.** Indexer side cache and KV spec, pooled-key build, block scoring and top-k, block-sparse consumer. Mirrors MiniMax-M3's op shape. - **W5, assembly and the load plan.** Full model forward, vision path, MTP. -- **W6, the first runnable arm** and the row's real unblock: a Q4_K_M backbone with - the n-gram table non-resident, per the developer decision in `## Hardware`. Two - separable halves. **W6a** authors the `qwen4_exp` GGUF architecture on our side, - because llama.cpp has none, and states in its result that these arms therefore - have no llama.cpp oracle. **W6b** makes the 51 GB table non-resident, which on - unified memory cannot be the existing host-pinned offload and needs its mechanism - established first. W6b is the one with unknown cost and should be spiked before it - is scheduled. +- **W6, the first runnable arm**, and the row's real unblock. Split by the blocker + analysis above rather than by guesswork. **W6a** authors the `qwen4_exp` GGUF + architecture -- one dispatch row plus its own config builder TU, never reusing + `HfConfigFromGguf`, which asserts its own architecture by name -- and emits Q4_0 on + every K=640 / K=320 reduction dim so the file can be opened at all. **W6b** is Route A: + F16 table, mmap borrow, prefault off, CPU device, producing the token baseline. + **W6c** is Route B: the dequantizing gather op plus the `kEmbeddingTable` keep-quant + policy change, in that order, which is what makes the arm the developer actually chose + reachable on CUDA. Waves W2 through W4 have no ordering dependency on each other and can be dispatched in parallel to separate worktrees. W5 is a barrier. @@ -297,8 +509,8 @@ requirement, so this row owes them and owes authoring the arch on our side. **The architecture hands us the lever.** Its card argues n-gram embedding is "more amenable to offloading than MoE", and the arithmetic agrees: the per-token cost is `(ngram_size - 1) * heads_per_ngram` = 16 lookups of `ple_embed_dim / ngram_heads` = -160 dims. **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per -token.** Making it non-resident is the intended design point. RadixArk reached the +160 dims. **51.2B of the 180B parameters, 28% of the model, is a table touched 16 times per +token** (51.2 GB at FP8, 102.4 GB at bf16, ~31 GB at Q4_K_M). Making it non-resident is the intended design point. RadixArk reached the same split independently. | Arm | Backbone (125B) | N-gram (51B) | Resident | Fits | @@ -328,6 +540,82 @@ decide which arm to attempt first and nothing else. GB10 is **unified** memory, "offload to host" is not a move there; non-resident means disk-backed and page-cached, and its cost is unmeasured. Establish it before it is designed around. +### The chosen arm has a hard blocker, and it is not the offload + +Verified in this tree, 2026-08-26. The developer chose the Q4_K_M backbone with a +non-resident n-gram table, and that arm **does not load today**. The reason is not the +offload machinery and not the memory budget. + +**This tree cannot keep a gather table quantized, by construction.** `KeepQuantKDim` +returns `-1` for `GgufTensorRole::kEmbeddingTable`, so the keep-quant branch is +unreachable for a gather table regardless of shape or encoding, and the qwen3_5 loader +asserts it by name: "the embedding table cannot keep quant blocks". A Q4_K or Q8_0 +n-gram table therefore **expands to bf16 at load: 51.2B params become 102.4 GB of +anonymous memory** on a ~119 GiB box. The arm dies before the first forward. The reason +is already recorded in a header comment upstream of both -- "a gather, not a GEMM ... A +quantized-gather op is a follow-up row" -- and **no such row exists**. That sentence is +the whole blocker and it has been sitting in a comment. + +The only non-expanding residency for a gather table is `kKeepF16`, which requires the +file to store ggml type **1 (F16) exactly**. That makes the table 102.4 GB on disk, and +it is **CPU-only**, because `EmbeddingKernelCuda` refuses anything but f32/bf16. + +**Second blocker, cheap to avoid because we author the converter.** +`moe_intermediate_size = 640` makes `ffn_down_exps` Q4_K-illegal on its reduction dim +(`640 % 256 = 128`), and `hc_lowrank = 320` is the same class. llama.cpp's substitution +for a ragged-K Q4_K tensor is believed to be Q5_0 -- **flagged as UNVERIFIED, and owed +a check against the pinned llama.cpp oracle before it becomes an assertion.** The +dependent fact IS verified in-tree and is the one that bites: this repository's GGUF +reader knows ggml type ids `0,1,2,8,10..14,16,18,19,22..28,30,39,40,41,66` and **has no +entry for 3 (Q4_1), 6 (Q5_0), 7 (Q5_1) or 20 (IQ4_NL)**, so such a file fails at header +parse with "unknown ggml type id". A stock `llama-quantize -Q4_K_M` output for this +model would not open at all. The fix is ours: emit **Q4_0** on every K=640 and K=320 +reduction dim -- block 32, the same 4.5 bpw as Q4_K, and keep-quant capable. + +**`ENG-WEIGHT-OFFLOAD` will not help, now or later.** It moves zero bytes today +(`ConsiderWeight` has no production callers, `supports_weight_offload` is false +everywhere and a test pins that), and it is separately documented inert on GB10 because +it moves bytes inside one physical pool. **Do not budget for it.** + +**The tier that does work already ships**, and the 2.4T model is the proof: mmap the +GGUF `MAP_PRIVATE`, borrow tensors in place, and alias the host pointer into the kernel +on a host-addressable device. That serves 369.97 GiB from a 119.631 GiB box at ~62 GiB +resident. Set `vllm_cpp.mmap.prefault: false`, or `PrefaultBorrowedSpan` touches every +page of the table at load and OOM-reboots the box. + +**Two routes, and the recommendation is to do both in order.** + +- **Route A, runs on today's code, CPU only.** F16 n-gram table, Q4_0 on the ragged + reduction dims, Q4_K elsewhere, mmap borrow with prefault off. Delivers a correct + first run and the token baseline Route B needs. No shared-kernel changes. +- **Route B, the arm actually chosen.** Add a dequantizing gather to `vt::Embedding` + across CPU and CUDA, then make `kEmbeddingTable` keep-quant eligible gated on that + op's availability. Order matters: the assertion above is CORRECT today and only + becomes wrong once the op lands. Then the table is Q4_K at **28.8 GB** on disk, + borrowed, device-aliased, gathered on device -- smaller than the 51 GB the arm was + scoped at. Roughly 400 lines. + +**Corrected sizing.** Backbone ~67.7 GiB resident in the expected arm; whole process +~73.5 GiB of 119.631 at 32K context single stream, leaving ~46 GiB of headroom that is +exactly what pays for the table's page cache. The original ~76 GB estimate was right to +within 10%. The design works because the per-token demand is tiny: 16 lookups x 160 +dims x 2 B = 5120 B/token over at most 16 distinct pages, so **<= 64 KiB of reads per +token**, against the 2.4T expert lane's 6.95 GB/token. That contrast is the whole +argument for this arm, and it is why the table is offloadable where MoE experts are not. + +Two further hazards, both with escapes: the `--device cuda` load-time device-fit +refusal counts every tensor in the file including the table, and a misaligned mmap +borrow is silently STAGED into device memory with a full `Alloc` -- pad the n-gram +tensor's data offset to 256 in our writer, since `kDeviceAliasAlignment` is 256 while +GGUF guarantees only 32. + +Finally, **there is no GGUF writer in this repository.** Authoring the conversion means +authoring it outside this tree; what this repo controls is only what it will accept. +And a latent trap for exactly that writer: the parse-time and dequant-time divisibility +checks test `numel % block_elems`, not `K % block_elems`, so a hand-rolled ragged-K +K-quant tensor decodes across row boundaries into structurally wrong values with **no +error**. Assert K-divisibility in the converter. + ## Risks - **Porting the eager reference as written.** The stated risk of the oracle split. @@ -411,7 +699,10 @@ No token gate is claimable until an arm runs. In order: This is the only gate reachable today, and it is reachable without the weights. 2. **G1, load plan.** Every published tensor accounted against a committed manifest, per arm, with refusals naming what is missing. -3. **G2, token-exact greedy** vs transformers at the lane pin, on whichever arm +3. **G2, token-exact greedy** with **at least one prompt past `indexer_budget` = 2048 + tokens of context**, because below that QSA selects every candidate and the gate + cannot distinguish a correct implementation from one attending pooled keys. Vs + transformers at the lane pin, on whichever arm `## Hardware` makes runnable first. Strict token equality; the near-tie distributional doctrine applies only if the oracle's greedy decode is shown non-deterministic, which is not assumed here. @@ -447,7 +738,12 @@ change that makes any arm reachable, not later. and the statement that no llama.cpp oracle exists for them. - MTP depth > 1. - The 1M-token RoPE extension above the native 262144. -- The non-resident n-gram table: its mechanism, and a measurement of its cost. +- The non-resident n-gram table on CUDA: the dequantizing gather op and the + `kEmbeddingTable` keep-quant policy change (Route B), and a measurement of the + page-cache cost that the <= 64 KiB/token arithmetic only bounds. +- **UNVERIFIED and owed a check against the pinned llama.cpp oracle:** llama.cpp's exact + substitution for a ragged-K Q4_K tensor, asserted here as Q5_0. +- A K-divisibility assertion in whatever writes our GGUF files. - A speed denominator, once one exists. ## Now From 2358e2b186681d05fa613bba183614fea4fe4fe2 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 14:08:11 +0000 Subject: [PATCH 3/3] spec(MODEL-MM-QWEN4-EXP): name the qwen4_exp row in the MODEL ratchet, and repair the anchors that naming it shifted The 377 -> 378 bump changed `scripts/check-agent-record.py` with no evidence in `tests/scripts/test_agent_record.py`, which `check-pr-size.py` refuses by name: a governance-checker change owes semantic mutation evidence. The gate was right. `test_qwen4_exp_row_is_inside_the_model_ratchet` follows the convention the dots3-note and recipe-backfill tests set, naming the row the bump was taken for so a count raised to silence a broken parse cannot look identical to a count raised because a row landed. It also pins that there is exactly ONE qwen4_exp row: the MTP head is an `mtp` block inside the same text config, not a separately registered architecture, so this bump is not the by-two shape that IndexTTS-2.5 and dots3-note each took. Adding it shifted `tests/scripts/test_agent_record.py` by 45 lines and staled the two anchors `ENG-RECORD-ANCHOR-RATCHET` keeps into that same file, taking the rot from 31 to 33 STALE. Repaired to the true lines, 1449 -> 1494 and 1517 -> 1562, rather than raising the budget, which the baseline file forbids in terms: "never to be raised to make a failing check pass". That row's own record already describes this exact failure - an edit to the very file the row cites - which is what the row exists to measure. Isolated rather than assumed: the anchor ratchet passes on pristine `origin/main`, so the two new stale entries were this branch's. Tracked by #1978. Gates: `tests/scripts/test_agent_record.py` 121 passed, rot back to the committed baseline 31 STALE + 6 BROKEN. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/engine-matrix.md | 2 +- tests/scripts/test_agent_record.py | 45 ++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 1 deletion(-) diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 13444fb8a..c9a62f2b9 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -227,7 +227,7 @@ claims it. | `ENG-RELEASE-WINDOWS` | Native Windows x86_64 pre-alpha release extension: one adaptive MSVC/UCRT CPU bundle with AVX2 executed in CI and one Vulkan preview bundle, both deterministic ZIPs and authenticated by the existing release handoff | T0 | vLLM has no Windows release path; runtime behavior remains pinned to vLLM `555967922`. Platform substrate reference: llama.cpp `src/llama-mmap.cpp:520-590` @ `237ad9b961f009ae19ac29dbce4cd0c1251f94b3`; Win32 API is the OS authority | W14 Win32 portability/MSVC CPU, W15 deterministic ZIP/PE packaging + Vulkan, and W16 ten-tuple prerelease workflow/version/docs implemented for one PR | Linux portability/release mutation gates are local evidence only. Native `windows-2022` MSVC `/W4 /WX`, extracted runtime/ISA smokes, merged-SHA ten-tuple dry run, `v0.0.3-pre.1` publication, attestations, and exact 32-asset audit remain pending; no Windows ZIP exists yet | [windows-binary-release.md](specs/windows-binary-release.md); [#117](https://github.com/mudler/vllm.cpp/issues/117) | `ACTIVE` | `CLAIM-ENG-RELEASE-WINDOWS` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | `docker/Dockerfile` (cpu/vulkan/cuda targets calling the release scripts); `docker/healthcheck.sh`; `release/container-matrix.json`; `scripts/check-container-matrix.py`; `scripts/check-container-workflow.py`; `scripts/validate-container-image.py`; `scripts/container_tags.py`; `.github/workflows/containers.yml`; SIGTERM handler `src/vllm/entrypoints/openai/server_main.cpp` (`SignalShutdown`, all three `listen()` sites); the pre-existing `docker/Dockerfile.arm64` is an unrelated CPU bench cross-check | issues `#170`, `#312`, `#394`; `tests/scripts/test_check_container_matrix.py` 31/31; `test_check_container_workflow.py` 29/29; `test_check_cuda_fat_gencode.py` 7+4 subtests. **GB10 2026-08-11 (`promaxgb10-4ad8`, `sm_121a`, CUDA 13.3): arm64 cuda image 1.71 GB, 673/673 objects, ten-SM gencode audit PASS, and a REAL GPU boot -- `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, `--gpus all`, host driver 580.159.03 injected.** cpu amd64 783 MB gated locally; cpu+vulkan amd64 green on hosted CI **arm64 cuda lane RUNTIME-VERIFIED on GB10 2026-08-11** -- the first accelerator-hardware evidence for any lane. Four defects were removed to get there, each found by building rather than reading: the CUDA 12.9 base could not compile `sm_110`, the BuildKit cache mount outlived its toolchain (both #366), Marlin gencode had drifted from the feature table and failed the audit on 14 correctly-compiled TUs (#394, blocking BOTH cuda tuples project-wide), and the validator could only ever produce build evidence because its boot smoke never passed `--gpus`. **NOT established: nothing is published to GHCR; amd64 cuda is unbuilt; the published arm64 image is SBSA (`targets/sbsa-linux`), so Tegra -- Thor `sm_110`, Orin `sm_87` -- is untested and NOT covered** **ORIN (Tegra) 2026-08-11: the SBSA image RUNS on Jetson AGX Orin `sm_87` (L4T R36.4.3, Docker 27.5.1) -- Qwen3-0.6B (rev `c1899de2`) loads and GENERATES via `/v1/completions`, tegrastats GR3D 95-97% during decode vs 14-15% idle.** Tegra needs `--runtime nvidia --gpus all`: `--gpus` alone is refused by the hook and `--runtime` alone mounts no driver | [container-images.md](specs/container-images.md); issues [#170](https://github.com/mudler/vllm.cpp/issues/170), [#312](https://github.com/mudler/vllm.cpp/issues/312), [#394](https://github.com/mudler/vllm.cpp/issues/394) | `ACTIVE` | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | | `ENG-DOCS-SITE` | Publish the 11 `docs/*.md` as a browsable GitHub Pages site at `https://mudler.github.io/vllm.cpp/` WITHOUT a second copy of the prose. A Hugo site at `website/` mounts `../docs` READ-ONLY and derives everything else from what is already in the files: each page title from the file's first `# H1`, the sidebar order from `website/data/nav.yaml`, and links through a Goldmark render hook (internal `.md` → site URL; the 139 `../.agents/**` and `../AGENTS.md` escapes → GitHub blob URLs, since the protocol tree is deliberately NOT published). **No file under `docs/` is modified, moved, renamed, or given front matter**, so `check-doc-checkpoint.py` and every protocol path reference keep working and there is no second surface that can drift — the whole point of the row. Custom lean layouts, NO theme and NO submodule: off-the-shelf docs themes read titles, weights and menus out of front matter this design deliberately does not have, so each would need its title partial, menu and link hook overridden anyway, and hugo-book additionally floors at Hugo 0.158 against the 0.146.3 pin CI and the local toolchain share. Hard prerequisite inside the repo: `classify_path` in `scripts/check-pr-size.py` FAILS CLOSED on `website/**` (verified: raises `ValueError: unclassified repository path`), so the classifier must learn the path or the PR cannot pass the project's own size gate. Hard prerequisite outside it: GitHub Pages must be enabled with the source set to GitHub Actions — the workflow is inert otherwise. A marketing landing page is explicitly OUT of scope (`README.md` stays the front door), as is any restructuring of `docs/`; the custom domain is parked behind the pending vLLM trademark question | T1 | NO vLLM analogue — upstream's docs are a separate mkdocs site and nothing in this row mirrors upstream *behavior*, so it carries no parity obligation. The STRUCTURAL reference is LocalAI's `.github/workflows/gh-pages.yml` (two Hugo sites merged into one Pages artifact), reduced to the docs half | read-only mount `website/hugo.toml:29`; title-from-H1 `website/layouts/partials/title.html:10`; link rewriting `website/layouts/_default/_markup/render-link.html:27`; guard `scripts/check-site.py:70`; deploy `.github/workflows/gh-pages.yml` | `tests/scripts/test_check_site.py:51,56,66,80,89,97` (6 mutation cases: clean tree, H1 stripped, doc absent from nav, nav entry with no file, duplicated entry, missing nav file); build evidence 14 pages with `docs/bench-evidence` + `docs/superpowers` absent from `public/` and no `href` ending in `.md`; 48 protocol links rewritten in `docs/status/`. NO published page is claimed: GitHub Pages is not yet enabled on the repository, which is the recorded stop condition holding this row at `GATING` | [gh-pages-docs-site.md](specs/gh-pages-docs-site.md); issue [#224](https://github.com/mudler/vllm.cpp/issues/224) | `READY` | `CLAIM-ENG-DOCS-SITE` | -| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1449` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1517`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | +| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1494` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1562`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | | `ENG-RECORD-CONFLICT-SURFACES` | Retire the shared record surfaces that make concurrent PRs conflict by construction. MEASURED at `origin/main` `d928e2c3` with `git merge-tree --write-tree` over every open PR: **16 of 29 conflict (55%), and 13 of the 16 conflict in bookkeeping files ONLY**, with no product code involved — `.agents/coordination.md` in 8, `.agents/NOW.md` in 5, `.agents/roadmap_v1.md` in 4, `scripts/check-public-doc-tables.py` in 4, `docs/STATUS.md` in 4, and any `src/`/`tests/` path in just 3. Three defects, each of which GUARANTEES rather than risks a collision. (1) `.agents/NOW.md` is a fixed-size shared buffer at EXACTLY 6000/6000 chars (`check-now-current.py:31`), so adding a row requires evicting another and every PR is a read-modify-write of one global — and the conflict is the LUCKY outcome, since a clean three-way merge would apply both evictions and both additions, silently dropping live rows and blowing the very budget the checker defends. (2) `STATUS_RATCHET = {"chars": 243245}` (`check-public-doc-tables.py:557`) is a hardcoded byte count of a DIFFERENT file that may only fall, so a PR owing `docs/STATUS.md` one lifecycle line must delete unrelated prose from another row to pay for it and edit the checker too; the checker's own comment at `:331` already records the failure (*"a ratchet pinned to the byte turns every concurrently merged row's one-line status edit into a spurious failure"*) and answered it with slack instead of removing the coupling. (3) `.agents/coordination.md`'s active-claims table is insert-at-one-anchor: the six ROCm GDN PRs (#334 #336 #341 #343 #345 #348) are ONE author's sequential stack that conflicts on nothing else, each appending a ~1,500-char row — the PR description, transcribed into a file every other claim also writes. It also contradicts the protocol it serves: `AGENTS.md` holds that *"History is git"* and *"There is no state log"*, yet both claims tables ARE state logs duplicating `gh pr list`, `row/` branch names and issue state; the argument that refuses a waiver registry applies unchanged to a claims registry. Precedent twice over — `policy.csv` retired in `0f3e44ee`, per-class line budgets retired 2026-08-10 because the gate fired on ordinary work. The exonerated surfaces share ONE property, one writer per file: `.agents/specs/.md` (one file per row, **zero conflicts** in the sample), the `*-matrix.md` inventories, and the append-only `.agents/benchmark-record.md`. SCOPE: remove `STATUS_RATCHET` and the doc-gating global counters while KEEPING the per-cell/per-paragraph caps (local, so they couple nothing); remove the active-claims table and derive claims from open PRs and branch names; drop `NOW.md`'s byte budget; order the roadmap's keyed tables by ID so distinct keys stop colliding at one anchor; and record the invariant — **no surface that every PR must write** — in `AGENTS.md`. No product source, kernel or gate semantic moves | T0 | NO vLLM analogue — this is local protocol machinery, so the mirror rule does not apply and no upstream `file:line` exists to port from. Governed instead by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | - | - (spec-before-code: the red-before suites are named in the spec's Tests section — `tests/scripts/test_check_public_doc_tables.py`, `tests/scripts/test_check_now_current.py`, a mutation case per removed rule proving the obligation survives in the retained caps and `check-doc-checkpoint.py`, and a `git merge-tree` merge-shape regression that must be RED before the `NOW.md`/roadmap work and GREEN after) | [retire-shared-record-surfaces.md](specs/retire-shared-record-surfaces.md); issue [#364](https://github.com/mudler/vllm.cpp/issues/364) | `READY` | `CLAIM-ENG-RECORD-CONFLICT-SURFACES` | | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate rejects CORRECT commits because of paragraph placement, and that is why `main` is red on `agent-record`. `check-commit-trailers.py` reads trailers via `git interpret-trailers --parse`, which treats ONLY the final paragraph as the block; GitHub appends `Co-authored-by:` as a SEPARATE trailing paragraph on a squash merge, so a complete correct block becomes invisible and the gate reports it missing. MEASURED: piping `git show -s --format=%B dbd0d51c` into `git interpret-trailers --parse` prints nothing but the co-author line, and 13 of the last 30 commits on `main` fail the check -- unnoticed only because those runs were cancelled (#274), which HID the defect rather than causing it. FIX: fuse consecutive trailing TRAILER-SHAPED paragraphs before parsing. Nothing is relaxed -- the block must still exist, the marker must still sit above it, each declaration must still appear exactly once, and an AI co-author is still forbidden; the block is merely FOUND where the merge tool left it. A prose paragraph still terminates it. REJECTED IN FLIGHT and recorded because it is the more instructive half: a first attempt also collapsed identical duplicate trailers to fix the multi-commit-squash shape, which relaxes the uniqueness rule an existing test already pins. Rewriting that assertion to suit the change is what AGENTS.md forbids, and the distinction is real -- a doubled block is genuinely malformed and fixable at source, whereas the co-author case is a correct commit defeated by the parser. Reverted in full. SCOPE LIMIT, stated rather than implied: this fixes ONE of five observed shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had been hiding and now correctly fails; `87308dea` (GitHub's `---------` separator), `b8293c88` (squash doubled the block) and `b580452d` (merge button, no trailers) stay red by design. Closing those is a merge-method change, not a checker change | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:60` (`join_trailing_trailer_paragraphs`, `_is_trailer_paragraph`, and the fused `parsed_trailers`) | `tests/scripts/test_check_commit_trailers.py:1` 21 cases -- the RED-BEFORE appended-co-author case plus four GUARDS that keep the fusion bounded (doubled block still fails, contradictory declarations still fail, a no-trailer merge message still fails, prose after the block still fails), all four green before and after; closure [parity-ledger.md#L941](parity-ledger.md#L941) | [trailer-merge-artifacts.md](specs/trailer-merge-artifacts.md); issue [#406](https://github.com/mudler/vllm.cpp/issues/406) | `DONE` | `157080c8` | | `ENG-FORGE-COAUTHOR` | The forbidden-AI-trailer rule was catching ATTRIBUTION rather than an authorship claim, which is why bot-opened PRs red `main` on merge. GitHub composes the squash message itself and appends the account that opened the PR — `Co-authored-by: localai-org-maint-bot <...@users.noreply.github.com>` — and most PRs here are opened by a bot, so nearly every squash trips the AI-identity check. Real instance `f64f2b71`, invisible until #406 repaired the parse, which is why it reads as a new failure and is not one. The rule exists so an AI cannot claim it WROTE the code, and that stays; GitHub is recording who pressed the button, and the AI-involvement claim is already carried separately by `AI-Assisted` and `Assisted-by` in the same block. FIX: accept a `Co-authored-by` at a GitHub account noreply address even when the name matches an AI identity token, keyed on the FORGE'S OWN DOMAIN rather than the name so the exemption cannot be borrowed. A hand-written `Co-authored-by: Claude ` still fails; `Signed-off-by` is excluded from the exemption entirely, because a sign-off is a legal assertion about provenance rather than attribution. `AGENTS.md` records the same distinction in the same change so prose and checker cannot drift | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:38` (`FORGE_ACCOUNT_EMAIL` and the forbidden-trailer skip) | `tests/scripts/test_check_commit_trailers.py:1` 25 cases -- the RED-BEFORE forge-bot case plus THREE guards that matter more than the relaxation because this LOOSENS a rule: a hand-written AI co-author still fails, `Signed-off-by` at the same noreply address still fails, and a human co-author still passes; all three green before and after. Real commit `f64f2b71` re-verified per commit | [forge-coauthor-attribution.md](specs/forge-coauthor-attribution.md); issue [#418](https://github.com/mudler/vllm.cpp/issues/418) | `ACTIVE` | `CLAIM-ENG-FORGE-COAUTHOR` | diff --git a/tests/scripts/test_agent_record.py b/tests/scripts/test_agent_record.py index 621f3a78e..95e778ce5 100644 --- a/tests/scripts/test_agent_record.py +++ b/tests/scripts/test_agent_record.py @@ -601,6 +601,51 @@ def test_dots3_rows_are_inside_the_model_ratchet(self) -> None: self.assertEqual(found[0].path.name, "model-matrix.md", item_id) self.assertEqual(found[0].field("state").strip().strip("`"), state, item_id) + def test_qwen4_exp_row_is_inside_the_model_ratchet(self) -> None: + """The #1978 row and the 377 -> 378 bump are one semantic change. + + Same contract as `test_dots3_rows_are_inside_the_model_ratchet` above, + with the arithmetic going the other way. dots3-note and IndexTTS-2.5 + each moved this pin by TWO because vLLM registers two architectures for + what prose calls one model. `Qwen4ExpForConditionalGeneration` moves it + by ONE: its MTP head is an `mtp` block inside the same text config, not + a separately registered architecture, so there is no + `MODEL-SPEC-qwen4-exp-*` row and there must not be one. Naming the row + is what makes 378 checkable rather than plausible. + + What this catches that nothing else does: renaming the row, or adding a + second qwen4_exp row to "match" the two-row precedent, both leave the + count reachable by a compensating edit elsewhere in the matrix while + every other check stays green. Only an assertion that names the row + goes red. + + `READY` is pinned deliberately and is the weaker half of the evidence, + stated rather than implied. The row is `READY` because its spec is + committed and no product code has landed; the structured-spec rules + already catch a move to `ACTIVE`, and the claim-ownership rules already + catch `INVENTORIED`. It is pinned anyway so that a future refactor of + those rules cannot silently take this pin with it -- which is exactly + the reasoning the dots3 test records for its own asymmetry. + + The row is also beyond-pin in the strongest sense this file has carried: + vLLM does not implement `qwen4_exp` at ANY revision, not merely after + `555967922`. Its Upstream cell therefore names no pinned module or + class, and the at-the-pin static invariants are untouched. + """ + errors: list[str] = [] + rows, _ = agent_record.check_matrices(errors) + self.assertEqual([error for error in errors if "MODEL rows" in error], []) + + item_id = "MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation" + found = [row for row in rows if row.item_id == item_id] + self.assertEqual(len(found), 1, item_id) + self.assertEqual(found[0].path.name, "model-matrix.md", item_id) + self.assertEqual(found[0].field("state").strip().strip("`"), "READY", item_id) + + # One row, not two: no speculative-head sibling exists for this arch. + siblings = [row for row in rows if "qwen4-exp" in row.item_id] + self.assertEqual([row.item_id for row in siblings], [item_id]) + def test_recipe_backfill_rows_are_inside_the_model_ratchet(self) -> None: """The #609/#610 rows and the 362 -> 369 bump are one semantic change.