From a5bd41302a4cd3a2072717f99e70e8697f986c4f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 16:57:44 +0000 Subject: [PATCH 1/3] =?UTF-8?q?feat(MODEL-MM-QWEN4-EXP):=20W3=20=E2=80=94?= =?UTF-8?q?=20the=20gated-residual=20stream=20reads=20a=20GROUPED=20norm?= =?UTF-8?q?=20and=20writes=20a=20rank-1=20update=20(#1988)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Qwen4-Exp carries its residual as `hc_count * hidden_size` = 4 x 2560 = 10240 floats through all 48 layers, reads and rewrites it twice per layer, and collapses it at the end with the same class that gates it. This wave lands the host reference for that class and for the grouped RMSNorm underneath it. It is written against two oracles because the row has two, by developer direction: transformers v5.16.0 supplies the algorithm (vLLM has never registered `qwen4_exp` at any revision), vLLM supplies the op form. The grouped norm therefore mirrors `RMSNormGated(group_size=)` — not the plain `RMSNorm`, whose only related knob is `var_hidden_size` and which has no group concept at all — while its semantics come from `Qwen4ExpTextRMSNorm`. The reason this needs a gate rather than a careful reading is that four of its guarantees are one-character defects with no visible symptom: * The `1 + w` parameterization. transformers scales by `(1.0 + weight)` on a ZERO-init weight; vLLM scales by `weight` on a ONES-init one. They agree only under a load-time `w = 1.0 + w_hf`. Skip it and every `hc_norm` scales by ~zero, which reads as a broken checkpoint rather than a broken port; apply it twice and it scales by ~2x. It now lives in exactly one named function, `HcNormWeightFromHf`, so the question "has the 1 already been added?" is about one call site instead of about arithmetic inside a kernel. * The division by `hc_count` inside the SiLU sits BEFORE the activation. `silu(x/4)` is not `silu(x)/4`, and on this data the two answers differ by as little as 6.6e-3 — under a percent, and invisible to a loose comparison. * There is no such division on the up-projection sigmoid, and the reduce over the four streams is a MEAN and not a sum. * The elementwise multiply uses the NORMED stream while the write-back adds to the RAW one, so the two ends of the same function disagree about which copy they hold on purpose. The goldens are not derived, hand-computed, or transcribed. `Qwen4ExpTextRMSNorm` and `Qwen4ExpTextGatedResidual` are lifted verbatim by line range out of the lane-pinned oracle file, its sha256 re-checked at dump time, and EXECUTED; the generator is committed and reproduces the .inc byte for byte. A second, independent double-precision reference then reproduces the same numbers, and a flag-per-defect sweep proves each of the five traps opens a gap the goldens can actually see — measured, tabulated in the test, and floored at 6.6x under the smallest of them. The write-back is deliberately an in-place rank-1 update rather than the broadcast-and-multiply upstream spells. That shape is the fusion seam: both llama.cpp implementations of this architecture materialise it as `repeat_4d` + `mul`, which is 96 dense [2560, 4, T] tensors built and discarded per forward pass at 48 layers x 2 sites. Keeping the pointer form as the primitive means a device kernel replaces this one function and nothing above it moves. No speed is claimed here; no arm of this model runs on any device we own. The spec's claim that DeepSeek-V4's `MhcPost` is a bit-exact bring-up bridge for this write-back under an identity comb matrix is verified rather than repeated, by a bit-equality case against that kernel. It holds; it costs hc-squared work where hc suffices, and it is not an identity for a negative-zero or non-finite residual, neither of which this path reaches. NOT REACHED at this commit, per AGENTS.md "Nothing lands dead": W1 config registration (#1986) is still in review, so no `qwen4_exp` resolves through the loader and nothing production-side calls this yet. The wiring is owed by W5, assembly, under #1978, and the debt is recorded under the spec's `## Owed` together with the model-matrix cell that W1 owns. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/issue-index.md | 1 + .agents/specs/qwen4-exp-flash-next.md | 31 ++ CMakeLists.txt | 1 + scripts/gen-qwen4-exp-hc-goldens.py | 131 ++++++ .../model_executor/models/qwen4_exp_hc.cpp | 183 ++++++++ src/vllm/model_executor/models/qwen4_exp_hc.h | 151 +++++++ tests/CMakeLists.txt | 10 + tests/vllm/models/qwen4_exp_hc_goldens.inc | 419 +++++++++++++++++ tests/vllm/models/test_qwen4_exp_hc.cpp | 421 ++++++++++++++++++ 9 files changed, 1348 insertions(+) create mode 100755 scripts/gen-qwen4-exp-hc-goldens.py create mode 100644 src/vllm/model_executor/models/qwen4_exp_hc.cpp create mode 100644 src/vllm/model_executor/models/qwen4_exp_hc.h create mode 100644 tests/vllm/models/qwen4_exp_hc_goldens.inc create mode 100644 tests/vllm/models/test_qwen4_exp_hc.cpp diff --git a/.agents/issue-index.md b/.agents/issue-index.md index ff108e7fd..9f072867b 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -745,3 +745,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#526](https://github.com/mudler/vllm.cpp/issues/526) | `SERVE-TOOL-HISTORY-ARGS` | OpenAI multi-turn tool history reaches chat templates with string-valued arguments | bug | | [#1934](https://github.com/mudler/vllm.cpp/issues/1934) | `BACKEND-ROCM` | `RocmPlatform::needs_weight_staging()` is stale-false (a W0-era placeholder never revisited despite #523/#509/#506/ROCM_ATTN/hipGraph landing since), so `CheckDeviceWeightFit` — the #1123/#1870 load-time refusal, including the `policy_forces_full_expand` fix — never runs on ROCm: measured directly, `VT_DEVICE_WEIGHT_BUDGET_BYTES=1` produced no refusal on a real load. The actual device allocation the refusal guards is not gated on this flag, so #1870's crash stays reachable until this closes; owed, not fixed in flow, because flipping the flag also moves `DirectDeviceLoadEligible` and several GDN kernel-dispatch defaults that each need their own correctness check | bug | | [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-QWEN4-EXP` | **`Qwen/Qwen3.8-Flash-Next` declares `Qwen4ExpForConditionalGeneration` / `qwen4_exp`, a new architecture vLLM does not implement, so the port runs on a split oracle: transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH.** Released 2026-08-24, 180B total / 6B activated, image-text-to-text. The `Qwen3.8` in the name is marketing continuity: `.agents/specs/qwen38-27b-bf16-gate.md`'s "one config key differs" precedent does NOT extend here. Read live 2026-08-26 at vLLM `origin/main` = `6a5e8f5979`: no `qwen4*` path, no registry entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. That is absence from vLLM `main` rather than staleness in our pin `555967922`, so a pin advance does not reach it. What exists is transformers [#48337](https://github.com/huggingface/transformers/pull/48337) "Add Qwen4Exp model", MERGED 2026-08-26, 5211 lines, and SGLang [#36497](https://github.com/sgl-project/sglang/pull/36497), still OPEN and therefore inadmissible. **Developer direction 2026-08-26, recorded verbatim: "use transformers as oracle for algorithmic side. but use ops from vllm so we account for optimized path."** Justified rather than convenient: `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so porting it as written yields a correct model at an indefensible speed, while AGENTS.md's mirror-vLLM polarity still binds every primitive vLLM implements. `Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`), all of which this tree has; GDN is an exact match for our AOT gate (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). The delta is four things, and **exactly two have no vLLM op at all**: the PLE dilated depthwise conv (kernel 4, dilation 3; `git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the n-gram hashed embedding. **The survey's load-bearing finding, and it REVERSES this row's first reading: QSA's structural twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The original call was that QSA, being plain GQA rather than MLA, had to map onto vLLM's non-MLA block-sparse case; that reasoning rested on treating `MLAAttentionSpec` as an MLA claim, and **it is not one** — M3's own indexer cache uses it while M3 is a plain-GQA model, with the comment "Key-only: MLAAttentionSpec budgets one vector/token (not 2x for K+V)". It is a budget shape. Remove that prop and the GQA-vs-MLA argument collapses. Verified line by line at `6a5e8f5979`: **nine independent structural matches with DSv4**, `compress_ratio == 4` literally the same number — MQA index with 1 key head at dim 128; `relu(q.k)` summed over index heads vs `(score.relu() * weights).sum(dim=0)`; `1/sqrt(head_dim)`; one score set per query token with no head axis vs `topk_indices_buffer[num_tokens, topk]`; pooling boundary `(position+1) % COMPRESS_RATIO == 0`; RMSNorm on the pooled key; **RoPE at the block-start position** vs `compressed_pos = (position // CR) * CR`; candidate count `visible // compress_ratio`; and one stored state per 4 tokens via `MLAAttentionSpec(tokens_per_state=compress_ratio)`, a first-class KV field documented as "Ints > 1 compress multiple tokens into one state (DSv4 sparse MLA)" that has no M3 equivalent. **M3 is a DIFFERENT ALGORITHM**, not a worse fit: its score is `tl.max(qk, axis=1)` over 128 RAW token dots with no pooling, no relu and no head reduction, it asserts `num_idx_heads == num_kv_heads` ("no topk index reduce") so it emits one block set PER KV HEAD, and its `SPARSE_BLOCK_SIZE = 128` is welded to the KV page size ("One sparse block == one KV page") on both the score and the attend side — moving it to 4 forces a page size of 4 and breaks `tl.dot`, whose tile needs >= 16. M3 contributes exactly ONE thing and it is a wiring precedent, not an algorithm: that a plain-GQA model can own a key-only side cache through `MLAAttentionSpec` and a private indexer backend. **The genuinely new work is the CONSUMER and nothing upstream supplies it** — every DSv4 sparse consumer attends to COMPRESSED MLA KV (one state per 4 tokens) and M3's attend to raw tokens only at page granularity, while QSA attends to RAW tokens selected at ratio-4 granularity. Two silent-failure traps follow: wiring QSA's top-k into a DSv4 sparse-MLA consumer attends a POOLED key/value and still emits plausible tokens, and **a short-prompt token gate cannot catch it because at context <= `indexer_budget` 2048 every candidate is selected** — so any QSA gate must run past 2048 tokens of context, which is now a stated `## Gates` requirement; and `SparseAttnCompressNormRopeStoreC4Kernel` does NOT mean-pool despite its name — it is a learned softmax pool over an OVERLAPPING window of 8 using a score channel this checkpoint does not have, and the CuteDSL variant refuses `overlap=False` at compile, so the **Triton** `head_dim=128` variant is the correct starting point. Two structural consequences beyond the module list: the residual stream is `hc_count * hidden_size` = **4 x 2560 = 10240 wide through the whole stack** with a low-rank read gate and per-branch scalar write gate around both attention and MLP, which is a change to the per-layer loop and every residual buffer rather than a drop-in module; and `number_of_conv_states = 3` on a PLE layer (GDN conv, PLE conv, n-gram token history) plus the indexer side cache, adjacent to [#1963](https://github.com/mudler/vllm.cpp/issues/1963) and [#1966](https://github.com/mudler/vllm.cpp/issues/1966). **NOTHING PUBLISHED FITS**, read live from the HF API against ~119 GB usable on GB10: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB (NVFP4 backbone with the n-gram table left at FP8, 51.2 GB) and `unsloth/...-GGUF` is a README with ZERO weight files. No GGUF exists and no tool can make one, because llama.cpp has no `qwen4_exp` either, so the standing k-quant requirement means authoring the arch on our side AND states that the quantized arms have NO llama.cpp oracle. **The chosen arm does NOT load today, and the blocker is neither the offload nor the budget: this tree cannot keep a gather table quantized at all.** `KeepQuantKDim` returns `-1` for `GgufTensorRole::kEmbeddingTable` (`src/vllm/model_executor/model_loader/gguf_keep_quant.cpp`), and `qwen3_5_gguf_weights.cpp` asserts it by name — "the embedding table cannot keep quant blocks" — so a Q4_K or Q8_0 n-gram table EXPANDS to bf16 and 51.2B params become **102.4 GB of anonymous memory**; the arm dies before the first forward. The reason was already sitting in a header comment ("a gather, not a GEMM ... A quantized-gather op is a follow-up row") and **no such row exists**. The only non-expanding gather residency is `kKeepF16`, requiring ggml type 1 exactly (102.4 GB on disk) and CPU-ONLY, because `EmbeddingKernelCuda` refuses anything but f32/bf16. **Second blocker:** `moe_intermediate_size = 640` makes `ffn_down_exps` Q4_K-illegal on its reduction dim (640 % 256 = 128), as does `hc_lowrank = 320`; llama.cpp's substitution is believed to be Q5_0 (**UNVERIFIED, owed against the pinned llama.cpp oracle**) and the dependent fact IS verified in-tree — our reader knows ggml ids `0,1,2,8,10..14,16,18,19,22..28,30,39,40,41,66` and has **no entry for 3, 6, 7 or 20**, so a stock `llama-quantize -Q4_K_M` file fails at header parse. We author the converter, so the fix is Q4_0 (block 32, same 4.5 bpw). **`ENG-WEIGHT-OFFLOAD` will not help** — it moves zero bytes today (`ConsiderWeight` has no production callers, pinned by a test) and is documented inert on GB10; the tier that DOES work already ships and is proven by the 2.4T model serving 369.97 GiB from a 119.631 GiB box at ~62 GiB resident: mmap `MAP_PRIVATE`, borrow in place, alias the host pointer, `prefault: false`. Corrected sizing: backbone ~67.7 GiB, whole process ~73.5 GiB of 119.631 at 32K single-stream, ~46 GiB of headroom for the page cache, so the ~76 GB estimate was right within 10%. The design works because per-token demand is **<= 64 KiB of reads** (16 lookups x 160 dims over at most 16 pages) against the 2.4T expert lane's 6.95 GB/token. The architecture supplies its own lever: the per-token n-gram cost is `(ngram_size-1)*heads_per_ngram` = 16 lookups of 160 dims, so **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per token** and making it non-resident is the intended design point (RadixArk reached the same split independently). Sizing arithmetic, NOT measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB (yes, ~10 GB left for KV), Q4_K_M backbone with the table non-resident ~76 GB. GB10 is UNIFIED memory so "offload to host" is not a move there; non-resident means disk-backed, and its cost is unmeasured. **Two decisions were put to the developer as explicit accept-or-reject and BOTH are settled 2026-08-26, recorded in place rather than left open.** (1) `.agents/oracles/transformers.md` pins transformers to 5.14.1, deliberately tied to what the pinned vLLM environment resolves so the environment cannot hold two `transformers` at once, and **5.14.1 does not contain `Qwen4Exp`**; the lane-scoped second pin is **ACCEPTED**, on the argument that the invariant guards a vLLM environment against drifting from its transformers and here there is no vLLM implementation to drift from, and it expires the moment vLLM registers `qwen4_exp`. **The lane pin is a real release, not a branch SHA**, which was not the expected outcome: `Qwen4Exp` merged to `main` at 12:03:40Z on 2026-08-26 and `v5.16.0` published at 12:35:15Z, and this was BOUNDED rather than assumed by fetching `models/qwen4_exp/modeling_qwen4_exp.py` at each tag — `v5.16.0` HTTP **200**, `v5.15.0` HTTP **404** — making 5.16.0 the FIRST release carrying the architecture and therefore the tightest available pin. The version string is UNMEASURED (it is the release proven to contain the model, not a `transformers.__version__` read off a running oracle) and `gateable` stays `no`. (2) The first runnable arm is the **Q4_K_M backbone with the n-gram table NON-RESIDENT** (~76 GB). Q8_0 was raised and does not fit at ~191 GB, and no partial-Q8 split reaches 119 GB with the backbone at 8 bits; Q4_K_M-throughout fits on paper at ~109 GB but leaves ~10 GB for KV and activations on a 262144-native-context model, which is not a margin. This promotes the non-resident table from a note to a first-class W6 deliverable, and it is NOT free: GB10 is UNIFIED memory, so the existing host-pinned offload seam (`ENG-WEIGHT-OFFLOAD`, mirroring vLLM's `cpu_offload_gb`) does not by itself solve it there and the mechanism must be disk-backed or genuinely unloaded — established before it is designed around. Spec: [`specs/qwen4-exp-flash-next.md`](specs/qwen4-exp-flash-next.md). No product code lands under the spec pull request | feature | +| [#1988](https://github.com/mudler/vllm.cpp/issues/1988) | `MODEL-MM-QWEN4-EXP` | **W3 of the Qwen4-Exp port: the 4-branch GATED-RESIDUAL hyper-connection stream and the grouped RMSNorm it stands on.** The residual stream is `hc_count * hidden_size` = 4 x 2560 = 10240 wide through the whole 48-layer stack, read and written twice per layer, and collapsed at the end by the same class with its injection branch switched off — a change to every residual buffer, not a drop-in module. Landed here as a HOST reference (`src/vllm/model_executor/models/qwen4_exp_hc.{h,cpp}`) gated against goldens dumped by EXECUTING the lane-pinned oracle source: transformers `v5.16.0` `models/qwen4_exp/modeling_qwen4_exp.py` (sha256 `77fec77d…`), `Qwen4ExpTextRMSNorm` (:158-181) and `Qwen4ExpTextGatedResidual` (:941-969) lifted verbatim by line range, plus an independent double-precision reference. The grouped norm mirrors vLLM's op form — `RMSNormGated` (`layers/layernorm.py:172`, `group_size` at `:187`, grouped branch `:258-264`) with the gate disabled, NOT the plain `RMSNorm` (`:37`), whose only related knob is `var_hidden_size`, a prefix reduction that cannot express per-group norms. **Three findings the gate now pins.** (1) The `1 + w` parameterization: transformers applies `out * (1.0 + weight)` on a ZERO-init weight while vLLM applies `out * weight` on a ONES-init one, they coincide only under a load-time `w = 1.0 + w_hf`, and the published GGUF has that fold applied at CONVERT time — so it lives in exactly one named function, `HcNormWeightFromHf`, and skipping it scales every `hc_norm` by ~0 (reads as a checkpoint bug) while applying it twice scales by ~2x. (2) The two divisions by `hc_count` are different: one is INSIDE the SiLU on the `[320]` low-rank intermediate BEFORE the activation (`silu(down(x)/4)`, not `silu(down(x))/4`; SiLU is not homogeneous), the other is inside the injection sigmoid with the whole sigmoid scaled by 2 (`2*sigmoid(inject(x)/4)`, range (0,2), exactly 1.0 at a zero logit), and there is NO division on the up-projection sigmoid. (3) The elementwise multiply uses the NORMED stream, the reduce over hc is a MEAN and not a sum, and `hyper_input` is written back RAW. **The spec's `MhcPost`-with-identity-comb reuse claim is VERIFIED rather than trusted**, by a bit-equality case against our DeepSeek-V4 kernel; it holds on finite inputs and is not an identity for a negative-zero or non-finite residual, neither of which is reachable here. **Not reached at its merge commit** — W1 config registration (#1986) is still in review, so nothing loads a `qwen4_exp`; the wiring is owed by W5 (assembly) under #1978 and is listed in the spec's `## Owed`. No token claim and no speed claim: no arm of this model runs on any fleet device. | feature | diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index 02aadc606..910bee2cd 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -745,6 +745,37 @@ change that makes any arm reachable, not later. substitution for a ragged-K Q4_K tensor, asserted here as Q5_0. - A K-divisibility assertion in whatever writes our GGUF files. - A speed denominator, once one exists. +- **W3's host reference lands UNREACHED, and this is the record of it** per + AGENTS.md "Nothing lands dead". + `src/vllm/model_executor/models/qwen4_exp_hc.{h,cpp}` + ([#1988](https://github.com/mudler/vllm.cpp/issues/1988)) is reached only by + `tests/vllm/models/test_qwen4_exp_hc.cpp`. No production entry point calls it + at its merge commit: W1 config registration + ([#1986](https://github.com/mudler/vllm.cpp/issues/1986)) was still in review, + so no `qwen4_exp` resolves through the loader and there is nothing for the + gated-residual stream to hang off. The wiring is owed by **W5, assembly**, + under [#1978](https://github.com/mudler/vllm.cpp/issues/1978), which is the + wave that widens the residual buffers to `hc_count * hidden_size` and calls + the module twice per layer. +- The **model-matrix lifecycle cell** for + `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation`, which still reads + `SPEC ONLY`. Left to W1 deliberately rather than by omission: W1 is the wave + whose scope IS registration, its pull request is already open, and + `.agents/model-matrix.md` is a single shared file, so three parallel waves + editing one cell is the write-lock AGENTS.md "Records" names. Whichever of + W1/W2/W3 lands last owes the correction. +- The **device arm of the gated residual**, and with it two checks this host + wave cannot make: that `RMSNormGated.forward_cuda`'s flash-linear-attention + Triton kernel is numerically correct in its GROUPED mode (unverified upstream, + see `## Design`), and that an fp32-accumulate device reduction still lands + inside this gate's tolerance against the double-accumulated host reference. +- The **fused rank-1 write-back**. `GatedResidualWriteBackInPlace` is the seam + and is already the primitive, but no device kernel replaces it yet. Both + llama.cpp implementations of this architecture materialise the update as a + `repeat_4d` + `mul`, i.e. 96 dense `[2560, 4, T]` broadcasts built and thrown + away per forward pass at 48 layers x 2 sites, which is where a + beat-llama.cpp-at-concurrency claim would come from. Not claimed here: no arm + runs. ## Now diff --git a/CMakeLists.txt b/CMakeLists.txt index b70ed04b8..abc43d7d9 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -784,6 +784,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/qwen3_moe_registry.cpp src/vllm/model_executor/models/qwen3_moe_weights.cpp src/vllm/model_executor/models/qwen3_moe.cpp + src/vllm/model_executor/models/qwen4_exp_hc.cpp src/vllm/model_executor/models/deepseek_v2_registry.cpp src/vllm/model_executor/models/deepseek_v2_weights.cpp src/vllm/model_executor/models/deepseek_v2.cpp diff --git a/scripts/gen-qwen4-exp-hc-goldens.py b/scripts/gen-qwen4-exp-hc-goldens.py new file mode 100755 index 000000000..fae861e31 --- /dev/null +++ b/scripts/gen-qwen4-exp-hc-goldens.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""Dump Qwen4-Exp gated-residual goldens by EXECUTING the pinned oracle source. + + usage: gen-qwen4-exp-hc-goldens.py [path/to/modeling_qwen4_exp.py] + +Fetch the oracle source first (it is not vendored -- the sha256 below is the pin): + + curl -sSLO https://raw.githubusercontent.com/huggingface/transformers/\ +v5.16.0/src/transformers/models/qwen4_exp/modeling_qwen4_exp.py + + +Oracle: huggingface/transformers v5.16.0, + src/transformers/models/qwen4_exp/modeling_qwen4_exp.py + sha256 77fec77d87f2a0eb23b95fa04276fb5779698a7c7f523cf5061e49c118bcc459 + +The two classes under test are lifted VERBATIM by line range out of that file and +exec'd; nothing is retyped. The only harness adaptation is a 4-field stand-in for +`Qwen4ExpTextConfig` (the real dataclass drags in the whole transformers package, +and the installed transformers here is 5.3.0, which predates qwen4_exp entirely). +The write-back is likewise the verbatim two lines of +`Qwen4ExpTextDecoderLayer.forward`. +""" +import hashlib +import sys + +import torch +import torch.nn as nn +import torch.nn.functional as F + +SRC = sys.argv[2] if len(sys.argv) > 2 else "modeling_qwen4_exp.py" +EXPECT_SHA = "77fec77d87f2a0eb23b95fa04276fb5779698a7c7f523cf5061e49c118bcc459" + +raw = open(SRC, "rb").read() +got = hashlib.sha256(raw).hexdigest() +assert got == EXPECT_SHA, f"oracle source sha256 {got} != {EXPECT_SHA}" +lines = raw.decode().splitlines(keepends=True) + +# 1-based, inclusive, as reported by grep -n on the pinned file. +RMSNORM = (158, 181) # class Qwen4ExpTextRMSNorm +GATEDRES = (941, 969) # class Qwen4ExpTextGatedResidual + + +def lift(span): + return "".join(lines[span[0] - 1: span[1]]) + + +class Qwen4ExpTextConfig: # harness stand-in; only the 4 fields the classes read + def __init__(self, hidden_size, hc_count, hc_lowrank, rms_norm_eps): + self.hidden_size = hidden_size + self.hc_count = hc_count + self.hc_lowrank = hc_lowrank + self.rms_norm_eps = rms_norm_eps + + +ns = {"torch": torch, "nn": nn, "F": F, "Qwen4ExpTextConfig": Qwen4ExpTextConfig} +exec(lift(RMSNORM), ns) +exec(lift(GATEDRES), ns) +Qwen4ExpTextRMSNorm = ns["Qwen4ExpTextRMSNorm"] +Qwen4ExpTextGatedResidual = ns["Qwen4ExpTextGatedResidual"] + +torch.set_default_dtype(torch.float32) +out = [] + + +def emit(name, t): + flat = t.detach().reshape(-1).tolist() + out.append("const float %s[] = {" % name) + for i in range(0, len(flat), 4): + out.append(" " + ", ".join("%.9gf" % v for v in flat[i:i + 4]) + ",") + out.append("};") + + +def case(tag, hidden, hc, lowrank, eps, tokens, use_combine, seed): + g = torch.Generator().manual_seed(seed) + cfg = Qwen4ExpTextConfig(hidden, hc, lowrank, eps) + mod = Qwen4ExpTextGatedResidual(cfg, use_combine=use_combine) + hc_h = hc * hidden + with torch.no_grad(): + # hc_norm.weight is ZERO-init upstream; a zero weight makes (1 + w) == 1 + # and would hide the parameterization entirely, so it is randomized. + mod.hc_norm.weight.copy_(torch.randn(hc_h, generator=g) * 0.5) + mod.input_mix_weight_down.weight.copy_(torch.randn(lowrank, hc_h, generator=g) * 0.3) + mod.input_mix_weight_up.weight.copy_(torch.randn(hc_h, lowrank, generator=g) * 0.3) + if use_combine: + mod.block_inject_weight.weight.copy_(torch.randn(hc, hc_h, generator=g) * 0.3) + hyper = torch.randn(tokens, hc_h, generator=g) * 1.7 + block_out = torch.randn(tokens, hidden, generator=g) * 0.9 + + normed = mod.hc_norm(hyper) + res = mod(hyper) + if use_combine: + mixed, hyper_ret, inj = res + assert hyper_ret is hyper, "upstream must return hyper_input RAW" + injection = block_out.unsqueeze(-2) * inj.unsqueeze(-1) + written = hyper + injection.flatten(-2) + else: + mixed = res + + out.append("") + out.append("// ---- %s: hidden=%d hc=%d lowrank=%d eps=%g T=%d use_combine=%s seed=%d" + % (tag, hidden, hc, lowrank, eps, tokens, use_combine, seed)) + emit("k%s_norm_w_hf" % tag, mod.hc_norm.weight) + emit("k%s_down" % tag, mod.input_mix_weight_down.weight) + emit("k%s_up" % tag, mod.input_mix_weight_up.weight) + if use_combine: + emit("k%s_inject" % tag, mod.block_inject_weight.weight) + emit("k%s_hyper" % tag, hyper) + emit("k%s_normed" % tag, normed) + emit("k%s_mixed" % tag, mixed) + if use_combine: + emit("k%s_block_out" % tag, block_out) + emit("k%s_inj_w" % tag, inj) + emit("k%s_written" % tag, written) + + +out.append("// GENERATED by scripts/gen-qwen4-exp-hc-goldens.py -- do not hand-edit.") +out.append("// Oracle: transformers v5.16.0 modeling_qwen4_exp.py") +out.append("// sha256 %s" % EXPECT_SHA) +out.append("// Qwen4ExpTextRMSNorm :158-181 (grouped RMSNorm, (1.0 + weight))") +out.append("// Qwen4ExpTextGatedResidual :941-969") +out.append("// write-back: Qwen4ExpTextDecoderLayer.forward, the two lines") +out.append("// injection = hidden_states.unsqueeze(-2) * injection_weights.unsqueeze(-1)") +out.append("// hidden_states = hyper_input + injection.flatten(-2)") +out.append("// torch %s" % torch.__version__) + +case("A", hidden=6, hc=4, lowrank=5, eps=1e-6, tokens=3, use_combine=True, seed=1234) +case("B", hidden=5, hc=3, lowrank=7, eps=1e-5, tokens=2, use_combine=True, seed=99) +case("C", hidden=6, hc=4, lowrank=5, eps=1e-6, tokens=2, use_combine=False, seed=7) + +open(sys.argv[1], "w").write("\n".join(out) + "\n") +print("wrote", sys.argv[1]) diff --git a/src/vllm/model_executor/models/qwen4_exp_hc.cpp b/src/vllm/model_executor/models/qwen4_exp_hc.cpp new file mode 100644 index 000000000..56a8bff90 --- /dev/null +++ b/src/vllm/model_executor/models/qwen4_exp_hc.cpp @@ -0,0 +1,183 @@ +// Qwen4-Exp (`Qwen3.8-Flash-Next`) W3 — the 4-branch GATED-RESIDUAL +// hyper-connection stream and its grouped RMSNorm. HOST reference. +// +// Algorithm: transformers v5.16.0 (the lane pin) +// `models/qwen4_exp/modeling_qwen4_exp.py` +// ::Qwen4ExpTextRMSNorm (:158-181), ::Qwen4ExpTextGatedResidual (:941-969), +// ::Qwen4ExpTextDecoderLayer.forward write-back (:825-826, :831-832 in the +// modular file `modular_qwen4_exp.py`). +// Op form: vLLM @ origin/main 6a5e8f5979 +// `model_executor/layers/layernorm.py`::RMSNormGated (:172), `group_size` +// (:187), grouped branch in `forward_static` (:243-244, :258-264). +// See qwen4_exp_hc.h for the oracle table, the `1 + w` resolution, and the +// fusion-seam note. +#include "vllm/model_executor/models/qwen4_exp_hc.h" + +#include +#include +#include + +namespace vllm::qwen4_exp { + +namespace { + +float Sigmoid(float x) { return 1.0f / (1.0f + std::exp(-x)); } + +// y[o] = Σ_i w[o*K + i] · x[i]. PyTorch `nn.Linear(bias=False)` weight layout, +// `(out_features, in_features)` row-major. +void LinearNoBias(const float* w, const float* x, int64_t out_dim, int64_t in_dim, float* y) { + for (int64_t o = 0; o < out_dim; ++o) { + const float* row = w + o * in_dim; + float acc = 0.0f; + for (int64_t i = 0; i < in_dim; ++i) acc += row[i] * x[i]; + y[o] = acc; + } +} + +void RequireSize(size_t got, size_t want, const char* what) { + if (got != want) { + throw std::invalid_argument(std::string("qwen4_exp: ") + what + " has " + std::to_string(got) + + " elements, expected " + std::to_string(want) + "."); + } +} + +} // namespace + +std::vector HcNormWeightFromHf(const std::vector& w_hf) { + std::vector w(w_hf.size()); + for (size_t i = 0; i < w_hf.size(); ++i) w[i] = 1.0f + w_hf[i]; + return w; +} + +std::vector GroupedRmsNorm(const std::vector& x, const std::vector& weight, + int64_t group_size, float eps) { + if (group_size <= 0) { + throw std::invalid_argument("qwen4_exp: group_size must be positive, got " + + std::to_string(group_size) + "."); + } + const size_t g = static_cast(group_size); + if (x.size() % g != 0) { + // `Qwen4ExpTextRMSNorm.__init__` (:164-165) raises the same condition. + throw std::invalid_argument("qwen4_exp: hidden_size (" + std::to_string(x.size()) + + ") must be divisible by group_size (" + std::to_string(g) + ")."); + } + RequireSize(weight.size(), x.size(), "hc_norm weight"); + + std::vector out(x.size()); + for (size_t base = 0; base < x.size(); base += g) { + // The reduction is INDEPENDENT per group: this is what `RMSNormGated`'s + // `rearrange(x, "... (g d) -> ... g d")` buys and what the plain `RMSNorm` + // cannot express. Accumulated in double, per the host-reference convention. + double ss = 0.0; + for (size_t d = 0; d < g; ++d) { + const double v = x[base + d]; + ss += v * v; + } + // eps is INSIDE the rsqrt, added to the mean square, never to the norm. + const float r = 1.0f / std::sqrt(static_cast(ss / static_cast(g)) + eps); + for (size_t d = 0; d < g; ++d) { + out[base + d] = x[base + d] * r * weight[base + d]; + } + } + return out; +} + +GatedResidualResult GatedResidualForward(const std::vector& hyper_input, + const GatedResidualWeights& weights, int64_t hc, + int64_t hidden, float eps) { + if (hc <= 1 || hidden <= 0) { + // `Qwen4ExpTextConfig.__post_init__` rejects hc_count <= 1 outright. + throw std::invalid_argument("qwen4_exp: hc_count must be > 1 and hidden_size > 0, got " + + std::to_string(hc) + " and " + std::to_string(hidden) + "."); + } + const int64_t flat = hc * hidden; + if (hyper_input.size() != static_cast(flat)) { + // Mirrors `Qwen4ExpTextGatedResidual.forward` (:955-958). + throw std::invalid_argument("qwen4_exp: expected " + std::to_string(flat) + + " hyper-connection features, got " + + std::to_string(hyper_input.size()) + "."); + } + if (weights.mix_down.empty() || weights.mix_down.size() % static_cast(flat) != 0) { + throw std::invalid_argument("qwen4_exp: input_mix_weight_down is not a multiple of " + + std::to_string(flat) + " (got " + + std::to_string(weights.mix_down.size()) + ")."); + } + const int64_t rank = static_cast(weights.mix_down.size() / static_cast(flat)); + RequireSize(weights.mix_up.size(), static_cast(flat * rank), "input_mix_weight_up"); + if (!weights.block_inject.empty()) { + RequireSize(weights.block_inject.size(), static_cast(hc * flat), "block_inject_weight"); + } + + GatedResidualResult out; + out.hyper_input_normed = + GroupedRmsNorm(hyper_input, weights.hc_norm_weight, hidden, eps); + const float* normed = out.hyper_input_normed.data(); + + // DIVISION 1 — inside the SiLU, on the [rank] low-rank intermediate, BEFORE + // the activation: `F.silu(down(x) / hc_count)`. SiLU is not homogeneous, so + // `silu(a)/hc` is a different function; the placement is load-bearing. + std::vector low(static_cast(rank)); + LinearNoBias(weights.mix_down.data(), normed, rank, flat, low.data()); + // A true division, not a reciprocal multiply: upstream spells `/ self.hc_count` + // and 1/hc is inexact for any hc that is not a power of two, so the shortcut + // would put a 1-ulp wedge between this reference and the oracle at hc_count=3. + const float hc_f = static_cast(hc); + for (int64_t r = 0; r < rank; ++r) { + const float a = low[r] / hc_f; + low[r] = a * Sigmoid(a); + } + + // NO division on the up projection: `torch.sigmoid(up(...))`, full stop. + std::vector gate(static_cast(flat)); + LinearNoBias(weights.mix_up.data(), low.data(), flat, rank, gate.data()); + for (int64_t p = 0; p < flat; ++p) gate[p] = Sigmoid(gate[p]); + + // `.unflatten(-1, (hc, H))`, multiply against the NORMED stream — not the raw + // one — then `.mean(dim=-2)`. A MEAN over hc, never a sum. + out.mixed_input.assign(static_cast(hidden), 0.0f); + for (int64_t j = 0; j < hc; ++j) { + const float* g_row = gate.data() + j * hidden; + const float* s_row = normed + j * hidden; + for (int64_t h = 0; h < hidden; ++h) out.mixed_input[h] += g_row[h] * s_row[h]; + } + for (int64_t h = 0; h < hidden; ++h) out.mixed_input[h] /= hc_f; + + // `block_inject_weight is None` returns here, and that early return IS the + // model's final mixer (`use_combine=False`). + if (weights.block_inject.empty()) return out; + + // DIVISION 2 — inside the injection sigmoid, whole sigmoid scaled by 2: + // `2 * sigmoid(inject(x) / hc_count)`. Range (0, 2), exactly 1.0 at a zero + // logit, so an untrained branch is the identity rather than a half-scale. + out.injection_weights.assign(static_cast(hc), 0.0f); + LinearNoBias(weights.block_inject.data(), normed, hc, flat, out.injection_weights.data()); + for (int64_t j = 0; j < hc; ++j) { + out.injection_weights[j] = 2.0f * Sigmoid(out.injection_weights[j] / hc_f); + } + return out; +} + +void GatedResidualWriteBackInPlace(float* hyper, const float* block_out, + const float* injection_weights, int64_t hc, int64_t hidden) { + // The rank-1 update, never the materialized broadcast. See the header: this + // function is the seam a fused device kernel replaces. + for (int64_t j = 0; j < hc; ++j) { + const float w = injection_weights[j]; + float* row = hyper + j * hidden; + for (int64_t h = 0; h < hidden; ++h) row[h] += block_out[h] * w; + } +} + +std::vector GatedResidualWriteBack(const std::vector& hyper_input, + const std::vector& block_out, + const std::vector& injection_weights, int64_t hc, + int64_t hidden) { + RequireSize(hyper_input.size(), static_cast(hc * hidden), "hyper_input"); + RequireSize(block_out.size(), static_cast(hidden), "block output"); + RequireSize(injection_weights.size(), static_cast(hc), "injection_weights"); + std::vector out = hyper_input; + GatedResidualWriteBackInPlace(out.data(), block_out.data(), injection_weights.data(), hc, hidden); + return out; +} + +} // namespace vllm::qwen4_exp diff --git a/src/vllm/model_executor/models/qwen4_exp_hc.h b/src/vllm/model_executor/models/qwen4_exp_hc.h new file mode 100644 index 000000000..18683258e --- /dev/null +++ b/src/vllm/model_executor/models/qwen4_exp_hc.h @@ -0,0 +1,151 @@ +// Qwen4-Exp (`Qwen3.8-Flash-Next`) W3 — the 4-branch GATED-RESIDUAL +// hyper-connection stream and the grouped RMSNorm it stands on. HOST reference. +// Issue #1988, spec `.agents/specs/qwen4-exp-flash-next.md` ("Gated Residual: +// what our MHC actually gives us"). +// +// ─── WHAT THIS IS A PORT OF ─────────────────────────────────────────────────── +// This row splits its oracles by design (spec `## Oracles`, developer direction +// 2026-08-26): transformers supplies the ALGORITHM, vLLM supplies the OP FORM. +// vLLM has never registered `qwen4_exp` — surveyed at `origin/main` 6a5e8f5979, +// zero hits tree-wide — so there is no vLLM model to mirror, but the grouped +// RMSNorm is a vLLM op and is mirrored as one. +// +// OURS <- ALGORITHM (transformers v5.16.0, the lane pin, +// `models/qwen4_exp/modeling_qwen4_exp.py`) +// GroupedRmsNorm <- `Qwen4ExpTextRMSNorm._norm` (:167-171) +// HcNormWeightFromHf <- `Qwen4ExpTextRMSNorm.forward` (:173-178), +// `output * (1.0 + self.weight.float())` +// GatedResidualForward <- `Qwen4ExpTextGatedResidual.forward` (:952-969) +// GatedResidualWriteBack <- `Qwen4ExpTextDecoderLayer.forward`, the two +// lines `injection = hidden_states.unsqueeze(-2) +// * injection_weights.unsqueeze(-1)` / +// `hidden_states = hyper_input + +// injection.flatten(-2)` +// +// OURS <- OP FORM (vLLM `origin/main` 6a5e8f5979, +// `model_executor/layers/layernorm.py`) +// GroupedRmsNorm <- `RMSNormGated` (:172), `group_size` (:187), +// grouped branch `forward_static` (:258-264): +// x.float() (:243), weight.float() (:244), +// rearrange "... (g d) -> ... g d", +// variance = mean(x_group^2), rsqrt(var + eps), +// out = flatten * weight. +// NOT the plain `RMSNorm` (:37): its only +// related knob is `var_hidden_size`, a PREFIX +// reduction that cannot express per-group norms. +// The gate (`z`) is disabled here; Qwen's +// hc_norm has none. +// +// ─── THE `1 + w` PARAMETERIZATION, RESOLVED EXPLICITLY ──────────────────────── +// The two oracles disagree on where the identity lives and this is the single +// most expensive detail in the module (issue #1988, trap 2): +// +// transformers: weight ZERO-init, out = normed * (1.0 + w_hf) +// vLLM: weight ONES-init, out = normed * w +// +// They coincide under `w = 1.0 + w_hf`, applied ONCE, at LOAD time. We mirror +// vLLM's op form — the kernel multiplies by `w` and knows nothing about the +// offset — and give the transform one and only one home, `HcNormWeightFromHf`, +// so that "did anyone already add the 1?" is a question about a single call site +// rather than about arithmetic buried in a kernel. It matters twice over: +// * Skip it and every `hc_norm` scales by ~0. That looks like a broken +// checkpoint, not like a broken port, and it costs a day. +// * Apply it twice and the scale is ~2x. The published GGUF has the fold done +// at CONVERT time, so a GGUF reader must NOT call this function. +// +// ─── PRECISION ──────────────────────────────────────────────────────────────── +// fp32 interior throughout, with the per-group sum of squares accumulated in +// double — the `deepseek_v4_mhc.cpp` house convention for a host reference. +// Upstream runs the norm in fp32 (`self._norm(x.float())`) and vLLM likewise +// (`x = x.float()`), so nothing here is a widening of the model path: this is a +// CPU reference and the device arm is the thing that must be fp32-accumulate and +// gated against these numbers. No `f32` model-path buffer is introduced. +// +// ─── SCOPE ──────────────────────────────────────────────────────────────────── +// Host reference only, and NOT YET REACHED from a production entry point: the +// W1 config/registration wave (#1986) is still in review, so there is no +// `qwen4_exp` loader row for this to hang off. Named per AGENTS.md "Nothing +// lands dead": the wiring is owed by row `MODEL-MM-QWEN4-EXP` W5 (assembly) +// under issue #1978, and it is listed in the spec's `## Owed`. +#pragma once + +#include +#include + +namespace vllm::qwen4_exp { + +// `w_vllm = 1.0 + w_hf`, elementwise. See the parameterization note above. Call +// this exactly once, at load, on a HuggingFace `hc_norm.weight`. Never on a +// weight read out of the published GGUF, which has the fold applied already. +std::vector HcNormWeightFromHf(const std::vector& w_hf); + +// Grouped RMSNorm over one token's `x`, in vLLM's `RMSNormGated(group_size=)` +// form with the gate disabled: `x.size() / group_size` INDEPENDENT reductions, +// eps inside the rsqrt, weight applied per element after the normalize. +// +// out[g*G + d] = x[g*G + d] * rsqrt(mean_d(x[g*G + .]^2) + eps) * weight[g*G + d] +// +// Refuses an indivisible width the way `Qwen4ExpTextRMSNorm.__init__` (:164-165) +// does, and refuses a weight that is not the full flattened width. +std::vector GroupedRmsNorm(const std::vector& x, const std::vector& weight, + int64_t group_size, float eps); + +// The four `Qwen4ExpTextGatedResidual` parameters, all bias-free `nn.Linear` +// weights in PyTorch's `(out_features, in_features)` row-major layout. +struct GatedResidualWeights { + std::vector hc_norm_weight; // [hc*H] — vLLM form, i.e. 1 + w_hf + std::vector mix_down; // [R, hc*H] + std::vector mix_up; // [hc*H, R] + // EMPTY is `use_combine=False`, i.e. `block_inject_weight = None`. That arm is + // the model's FINAL mixer: `Qwen4ExpTextModel` holds one of these and there is + // NO trailing RMSNorm after it — `hc_norm` is the last normalization before + // `lm_head` (issue #1988, trap 1). + std::vector block_inject; // [hc, hc*H] or empty +}; + +struct GatedResidualResult { + std::vector mixed_input; // [H] — the block's input + std::vector hyper_input_normed; // [hc*H] — what the mean multiplies + std::vector injection_weights; // [hc] — empty on the use_combine=false arm +}; + +// One token of `Qwen4ExpTextGatedResidual.forward`. `hyper_input` is [hc*H] and +// is NOT modified: upstream returns it RAW, un-normed, and it is the raw stream +// that the write-back adds to. +// +// The two divisions by `hc` are different and both load-bearing: +// * inside the SiLU, on the [R] low-rank intermediate, BEFORE the activation — +// `silu(down(x) / hc)`, not `silu(down(x)) / hc`. SiLU is not homogeneous. +// * inside the injection sigmoid, with the whole sigmoid scaled by 2 — +// `2 * sigmoid(inject(x) / hc)`, range (0, 2), exactly 1.0 at a zero logit. +// There is NO division on the up-projection sigmoid, and the reduce over `hc` is +// a MEAN, not a sum. +GatedResidualResult GatedResidualForward(const std::vector& hyper_input, + const GatedResidualWeights& weights, int64_t hc, + int64_t hidden, float eps); + +// The rank-1 write-back, in place: +// +// hyper[j*H + h] += block_out[h] * injection_weights[j] +// +// THIS IS THE FUSION SEAM, and it is deliberately the primitive rather than the +// convenience. Upstream spells it `hidden_states.unsqueeze(-2) * +// injection_weights.unsqueeze(-1)` then `.flatten(-2)`, and both llama.cpp +// implementations of this architecture materialise it as a `repeat_4d` + `mul`, +// i.e. a dense [H, hc, T] broadcast built and thrown away 96 times per forward +// pass at 48 layers x 2 sites. It is a rank-1 update. A device arm reads +// `block_out` once per (j, h) tile and `injection_weights[j]` once per row; it +// never allocates the broadcast. Keeping the in-place pointer form as the +// primitive means the fused kernel replaces exactly this function and nothing +// above it has to change. +void GatedResidualWriteBackInPlace(float* hyper, const float* block_out, + const float* injection_weights, int64_t hc, int64_t hidden); + +// Allocating wrapper over `GatedResidualWriteBackInPlace`, for tests and for the +// host path. Bit-identical to the in-place form by construction. +std::vector GatedResidualWriteBack(const std::vector& hyper_input, + const std::vector& block_out, + const std::vector& injection_weights, int64_t hc, + int64_t hidden); + +} // namespace vllm::qwen4_exp diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 85e7624ce..ab90d7a8a 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -3075,6 +3075,16 @@ target_include_directories(test_qwen3_32b_nvfp4a16_paged_engine PRIVATE vllm_cpp_add_test(test_registry_downcast_refusal vllm/models/test_registry_downcast_refusal.cpp) +# MODEL-MM-QWEN4-EXP W3 (#1988) — the 4-branch GATED-RESIDUAL hyper-connection +# stream and its grouped RMSNorm, against goldens dumped by EXECUTING the +# lane-pinned oracle source (transformers v5.16.0 `modeling_qwen4_exp.py`, the +# two classes lifted verbatim by line range) plus an independent +# double-precision reference. Host reference; no arm of this model runs yet, so +# no token and no speed claim. `qwen4_exp_hc.h` is a MODEL-PRIVATE header under +# src/ (the `dots3_note.h` precedent), hence the explicit src/ grant. +vllm_cpp_add_test(test_qwen4_exp_hc vllm/models/test_qwen4_exp_hc.cpp) +target_include_directories(test_qwen4_exp_hc PRIVATE ${CMAKE_SOURCE_DIR}/src) + # The five suites that reach src/vllm/support/platform_compat.h, granted per # target rather than globally: the file already carries 123 explicit # ${CMAKE_SOURCE_DIR}/src grants, and a blanket one in vllm_cpp_add_test would diff --git a/tests/vllm/models/qwen4_exp_hc_goldens.inc b/tests/vllm/models/qwen4_exp_hc_goldens.inc new file mode 100644 index 000000000..3111bb205 --- /dev/null +++ b/tests/vllm/models/qwen4_exp_hc_goldens.inc @@ -0,0 +1,419 @@ +// GENERATED by scripts/gen-qwen4-exp-hc-goldens.py -- do not hand-edit. +// Oracle: transformers v5.16.0 modeling_qwen4_exp.py +// sha256 77fec77d87f2a0eb23b95fa04276fb5779698a7c7f523cf5061e49c118bcc459 +// Qwen4ExpTextRMSNorm :158-181 (grouped RMSNorm, (1.0 + weight)) +// Qwen4ExpTextGatedResidual :941-969 +// write-back: Qwen4ExpTextDecoderLayer.forward, the two lines +// injection = hidden_states.unsqueeze(-2) * injection_weights.unsqueeze(-1) +// hidden_states = hyper_input + injection.flatten(-2) +// torch 2.11.0+cu130 + +// ---- A: hidden=6 hc=4 lowrank=5 eps=1e-06 T=3 use_combine=True seed=1234 +const float kA_norm_w_hf[] = { + -0.0558592826f, -0.248295054f, 0.0815368518f, -0.44084388f, + 0.0269500967f, 0.334186852f, -0.0298288036f, -0.233748972f, + 0.318456322f, -0.357040048f, -0.5415622f, -0.277362108f, + 0.485842496f, -0.257504582f, 0.712763309f, 0.399342716f, + -1.26366758f, 0.738892138f, -0.0848118141f, -0.495928675f, + -0.728454232f, 0.128145874f, -0.201523483f, 0.209763691f, +}; +const float kA_down[] = { + 0.230009928f, 0.0057106209f, 0.00659922976f, 0.345959991f, + -0.101799741f, 0.0467645898f, 0.268991739f, -0.0890373439f, + -0.205710724f, -0.0148911141f, -0.374550611f, -0.255257607f, + -0.230697557f, -0.468171865f, -0.159271434f, 0.0653252229f, + -0.0849997327f, -0.169785336f, 0.106985554f, -0.136046484f, + -0.0891406462f, -0.461391181f, -0.307454944f, -0.113418475f, + 0.117305949f, 0.154736325f, -0.301274389f, 0.29578951f, + 0.340006769f, 0.255127072f, 0.316016972f, 0.110753559f, + -0.0165582672f, -0.183763459f, 0.225003466f, -0.220376238f, + 0.138669804f, 0.352784842f, 0.0643412024f, 0.160859257f, + 0.0409469977f, -0.699949861f, 0.459235817f, 0.0803977698f, + 0.135144517f, -0.0817496553f, -0.521964252f, 0.0389798582f, + -0.168896139f, -0.0848820359f, 0.0219309255f, -0.416413009f, + -0.0803379416f, -0.0376101807f, -0.451138169f, -0.0986020789f, + 0.130802929f, -0.443111718f, -0.366648406f, -0.0823910087f, + -0.103497915f, -0.214860126f, 0.173432574f, 0.114157476f, + -0.436142772f, -0.802202046f, 0.479510695f, 0.24064137f, + -0.105343677f, -0.0201012976f, -0.016012691f, -0.249461859f, + 0.440670997f, 0.379409522f, -0.0825821906f, -0.0397612788f, + 0.0318639055f, 0.0277184229f, -0.35462603f, 0.0638813004f, + 0.0464846902f, -0.195784643f, 0.215386584f, -0.134854406f, + -0.215582252f, 0.205178604f, -0.103728168f, 0.0557277352f, + -0.0475083292f, 0.396788061f, -0.803893566f, -0.0389022827f, + -0.207587719f, 0.355906367f, 0.0823413432f, -0.101632081f, + -0.0367469974f, 0.242349729f, -0.342644632f, 0.61518997f, + -0.0456919149f, -0.221436858f, -0.215664506f, 0.238124669f, + -0.053115081f, -0.269426376f, -0.295032978f, 0.226913646f, + 0.104232438f, 0.0856718495f, -0.0148635367f, 0.186736494f, + -0.576501429f, -0.395285994f, 0.368593544f, 0.210738167f, + -0.0050911135f, -0.0848935172f, -0.193378344f, -0.00596537627f, +}; +const float kA_up[] = { + 0.0336867087f, -0.198038742f, -0.187293917f, -0.222491011f, + 0.102670431f, -0.0861795396f, -0.201774433f, 0.265300244f, + 0.216334134f, 0.066431731f, -0.0821184516f, 0.258368105f, + -0.0182846691f, 0.632193625f, -0.297900349f, 0.422407985f, + -0.238362685f, 0.18431586f, -0.0819076896f, 0.496816367f, + 0.418209016f, 0.181785226f, 0.0662629679f, -0.247354791f, + 0.218663722f, -0.220087618f, 0.468721628f, -0.306239724f, + -0.00526024215f, -0.275821298f, -0.281670183f, -0.582625389f, + 0.0698702261f, -0.330426067f, -0.374202728f, -0.224535272f, + -0.293771446f, 0.248553723f, -0.0750395954f, 0.0480682962f, + 0.218860701f, -0.133215174f, 0.246407941f, -0.180457219f, + 0.272075921f, 0.470730275f, -0.0332507603f, -0.0771899447f, + 0.16894193f, 0.0188689008f, -0.0191753581f, 0.908752322f, + 0.0557283759f, -0.0511085875f, 0.308112562f, 0.0211206935f, + 0.107589111f, -0.707808971f, -0.418630153f, 0.260241687f, + -0.0675459132f, 0.0357870571f, -0.323149085f, 0.0754813105f, + -0.0658725873f, 0.181845143f, 0.0141932517f, 0.00372157129f, + 0.113248877f, 0.219974011f, -0.147438213f, -0.331458807f, + -0.0117366239f, 0.012483703f, -0.314042419f, 0.0100491336f, + -0.316349119f, -0.0591964051f, 0.255663157f, -0.121967025f, + -0.423375368f, 0.00599367917f, -0.123458393f, -0.101948835f, + -0.166899547f, -0.19523038f, -0.0653577447f, 0.230989307f, + -0.299673647f, 0.333176434f, 0.00934083387f, 0.470875531f, + 0.461682051f, 0.0895232856f, 0.166822284f, -0.206049591f, + -0.331997365f, 0.680676579f, -0.675302148f, -0.303457767f, + 0.114846051f, 0.370880038f, -0.31603983f, -0.282106221f, + -0.30173409f, 0.113918141f, -0.0895575508f, -0.408483177f, + -0.0999869928f, -0.375807047f, -0.292377025f, -0.464005977f, + 0.103854716f, -0.379565507f, -0.636228383f, 0.459145993f, + -0.323322773f, -0.0955618247f, 0.683090091f, 0.116701543f, +}; +const float kA_inject[] = { + -0.470525771f, 0.403277755f, 0.0111542391f, -0.107938446f, + 0.0309650674f, -0.488039285f, 0.17185615f, -0.209858194f, + -0.642993629f, -0.162584186f, -0.12766321f, 0.543742359f, + -0.409083843f, 0.0920040607f, -0.30982241f, 0.4147183f, + 0.278410882f, 0.760603249f, 0.222951218f, -0.277991802f, + 0.324819773f, -0.303619057f, -0.0792483464f, 0.0768021792f, + -0.0304893199f, -0.109005488f, 0.331583977f, -0.273273081f, + -0.196809694f, -0.211460099f, -0.355280548f, -0.0793590546f, + -0.385707438f, -0.064223744f, 0.0311175603f, -0.121160097f, + -0.147312239f, -0.295749545f, -0.161240309f, -0.511619389f, + -0.274669468f, 0.301221073f, 0.122041345f, -0.201354802f, + 0.137796208f, -0.202428162f, -0.0490248278f, -0.210767046f, + -0.201934978f, -0.172478542f, 0.277232081f, 0.218289748f, + 0.187215894f, 0.280916899f, 0.231201351f, -0.0376391932f, + -0.246289238f, -0.25061211f, -0.0999569371f, -0.405950665f, + -0.012512004f, 0.22387664f, 0.0996676609f, -0.0557701588f, + 0.392104f, 0.436399937f, -0.0631143376f, -0.102441408f, + -0.109501801f, 0.147821859f, -0.175981492f, 0.238440841f, + -0.303731024f, -0.197148591f, -0.172388151f, -0.623645246f, + -0.0135014346f, 0.398366183f, -0.188274488f, -0.112167157f, + -0.314300239f, 0.0936004147f, -0.248159006f, -0.257808775f, + -0.119168296f, 0.126265466f, 0.0301423147f, -0.197634354f, + -0.35174799f, 0.0275507923f, -0.169687241f, -0.357006609f, + -0.283018708f, -0.245096996f, 0.102671057f, -0.324268609f, +}; +const float kA_hyper[] = { + 0.533237576f, -1.20873201f, 0.107799865f, -5.22493458f, + -1.51860964f, -0.425679505f, 0.346284419f, -1.10036111f, + 2.06948781f, 1.03521371f, -1.91126668f, 1.29653561f, + 1.37460244f, -1.56329715f, -1.99296105f, 0.720400214f, + -1.09137011f, 2.1534636f, -0.135816127f, -2.71129417f, + 1.59412038f, -0.772991061f, -0.512841046f, -0.183839917f, + 1.23042381f, -0.663249493f, 1.21584845f, 2.0912149f, + -0.206895769f, 1.31602156f, -4.23796225f, -0.811664701f, + 2.19035053f, -0.668166578f, -0.414703071f, -0.655608356f, + -0.163112f, 1.60746503f, -2.16032863f, 0.502987266f, + 0.669123352f, -0.106363147f, 0.973783314f, -0.429280818f, + -0.124404483f, -0.0254335124f, 0.246692955f, 0.343353271f, + -0.996848762f, -0.0774152055f, -1.28514361f, -1.98718035f, + -1.5961374f, 1.50561988f, 2.33991623f, -1.33068836f, + -0.0533250868f, 0.758986533f, 2.51541495f, -1.77328277f, + 1.35424542f, 0.787682176f, 0.898809254f, 0.158229485f, + 2.1792016f, -1.06922519f, 1.23619056f, -2.43089294f, + -0.455594748f, 1.3866334f, 0.350418687f, -1.19509685f, +}; +const float kA_normed[] = { + 0.219569743f, -0.396271139f, 0.050848078f, -1.27417457f, + -0.680159092f, -0.247693375f, 0.237367317f, -0.59572506f, + 1.92782748f, 0.470276743f, -0.619072139f, 0.661979795f, + 1.30696869f, -0.742763638f, -2.18429613f, 0.645079553f, + 0.184138298f, 2.39621353f, -0.0926238596f, -1.01842678f, + 0.32257098f, -0.649831653f, -0.305144876f, -0.165730134f, + 0.91935569f, -0.394563079f, 1.04066956f, 0.925387979f, + -0.168148518f, 1.3895421f, -2.03614354f, -0.307999939f, + 1.4301523f, -0.212750986f, -0.0941500589f, -0.234621674f, + -0.210025251f, 1.0343039f, -3.20649099f, 0.609949589f, + -0.152888834f, -0.160279006f, 1.89472055f, -0.460050941f, + -0.0718210042f, -0.0610019937f, 0.418785185f, 0.883108616f, + -0.682352483f, -0.042190671f, -1.00771046f, -0.805588245f, + -1.18839931f, 1.45638013f, 1.33703506f, -0.600539446f, + -0.0414086804f, 0.287416756f, 0.679178834f, -0.754732132f, + 1.62680888f, 0.472836733f, 1.24460423f, 0.179010272f, + -0.464537263f, -1.50317037f, 0.831080019f, -0.900129199f, + -0.090880096f, 1.14914262f, 0.205540046f, -1.06206393f, +}; +const float kA_mixed[] = { + 0.218683407f, -0.342433631f, -0.0115971975f, -0.096745424f, + -0.165759042f, 0.317727774f, 0.0970179737f, -0.0145416558f, + -0.0814772174f, 0.160644844f, -0.00374818593f, 0.230426177f, + 0.389982581f, -0.14256005f, 0.00486969389f, 0.0945343599f, + -0.0895517245f, -0.208840132f, +}; +const float kA_block_out[] = { + 0.519128263f, -1.07747173f, -0.256617546f, -0.335337877f, + -1.49071062f, -1.27359354f, -0.170700669f, 1.6074369f, + -0.0265056677f, -0.220163777f, -0.416405171f, -1.34811532f, + -0.362151444f, -0.151884809f, -0.656513393f, -2.1005435f, + 0.512117267f, -0.332021594f, +}; +const float kA_inj_w[] = { + 1.25132978f, 1.10185194f, 0.939499736f, 1.02238166f, + 0.899588108f, 1.01564634f, 0.986539185f, 0.906849504f, + 0.666291654f, 0.847988546f, 0.959581435f, 1.20329559f, +}; +const float kA_written[] = { + 1.1828382f, -2.55700445f, -0.213313311f, -5.64455271f, + -3.38398027f, -2.01936507f, 0.91828692f, -2.28757548f, + 1.78673327f, 0.665721059f, -3.55380917f, -0.10677588f, + 1.86232328f, -2.57558155f, -2.23405313f, 0.405350357f, + -2.49189234f, 0.95692277f, 0.394931108f, -3.81288147f, + 1.33175933f, -1.11583436f, -2.03691626f, -1.48593855f, + 1.07686353f, 0.782781601f, 1.1920042f, 1.8931582f, + -0.581488907f, 0.103273034f, -4.41133356f, 0.820922732f, + 2.16343021f, -0.891775131f, -0.837623477f, -2.02481675f, + -0.331514895f, 3.19326448f, -2.18647742f, 0.285787076f, + 0.258323342f, -1.43633175f, 0.818983495f, 1.02842259f, + -0.148441136f, -0.225088924f, -0.130923867f, -0.879184484f, + -1.23814726f, -0.178614795f, -1.72257304f, -3.38675499f, + -1.25491798f, 1.28439665f, 2.03281593f, -1.45948493f, + -0.610040903f, -1.02225018f, 2.94968462f, -2.05483317f, + 1.00673163f, 0.641936302f, 0.268831193f, -1.85741317f, + 2.67061973f, -1.38782692f, 0.800415337f, -2.61365533f, + -1.24557447f, -1.14094138f, 0.966647148f, -1.59461701f, +}; + +// ---- B: hidden=5 hc=3 lowrank=7 eps=1e-05 T=2 use_combine=True seed=99 +const float kB_norm_w_hf[] = { + 0.30634293f, -0.587676823f, -0.382324636f, -0.333282828f, + 0.372182995f, -0.322658688f, -0.694513917f, -0.136483803f, + 0.470299363f, -1.30883467f, -0.475572467f, -0.109824248f, + 0.0838670358f, -0.00266977097f, -1.04641449f, +}; +const float kB_down[] = { + 0.0964841321f, 0.255086482f, 0.194116309f, 0.388963997f, + 0.33462137f, -0.302849412f, 0.186166987f, 0.136263818f, + 0.252059072f, -0.605638325f, -0.118919484f, -0.15502046f, + -0.700868011f, 0.210165933f, -0.0841370299f, 0.436674327f, + -0.627836585f, -0.21424897f, 0.141523972f, 0.584608614f, + 0.0808142722f, -0.402946323f, -0.125655219f, -0.118344508f, + 0.548009813f, 0.237366602f, -0.459965914f, -0.260818362f, + -0.0895831659f, -0.229434088f, 0.257640928f, -0.0247417837f, + 0.0624738671f, -0.0466020517f, -0.419715255f, -0.109153494f, + 0.143075585f, -0.058244545f, 0.119965412f, -0.240623817f, + 0.0215703472f, -0.403630406f, 0.435768038f, 0.236045897f, + -0.115460597f, -0.196635619f, -0.435137004f, -0.36175251f, + 0.507629991f, 0.52901876f, -0.03106484f, -0.522266626f, + -0.132427171f, -0.23122558f, 0.442002177f, -0.180983216f, + -0.101839982f, 0.354686797f, -0.458208293f, -0.381291807f, + -0.255328417f, 0.278687924f, -0.425551474f, -0.176636606f, + 0.15925926f, 0.316434145f, -0.258282274f, -0.187034011f, + -0.387811422f, -0.189466402f, -0.324946284f, 0.0134268887f, + 0.23262842f, -0.450703979f, 0.318678319f, -0.307124197f, + 0.184222028f, 0.033916451f, 0.241243362f, -0.175679237f, + 0.506585538f, -0.382096678f, 0.0134525402f, -0.308507323f, + 0.708951771f, -0.421255499f, -0.207631126f, -0.0347730294f, + -0.166764975f, -0.258377284f, -0.700190663f, -0.453181237f, + 0.208763883f, -0.171336681f, -0.0689195171f, 0.195338294f, + -0.0539493635f, -0.107879616f, -0.0958281383f, 0.467185408f, + -0.0806179717f, -0.202732861f, -0.0514242128f, -0.0472091548f, + -0.0386718884f, +}; +const float kB_up[] = { + 0.343490094f, 0.0721287504f, 0.185548693f, -0.17879653f, + -0.167254314f, -0.244209081f, 0.406540245f, 0.0789460614f, + -0.0533151589f, 0.114840999f, 0.21510464f, 0.0884370133f, + -0.0665651187f, -0.108309262f, -0.0655949786f, 0.573655665f, + -0.342038631f, 0.0259813108f, -0.0974591896f, 0.142308608f, + 0.19310008f, 0.139679015f, -0.0349669829f, -0.462970614f, + 0.0544645377f, 0.0513722971f, 0.230093256f, -0.0984883606f, + -0.347496063f, -0.255700141f, 0.191816881f, 0.036785312f, + 0.126049876f, 0.22464405f, 0.245288566f, -0.358890563f, + 0.545955479f, 0.36748749f, 0.0152934687f, 0.198187605f, + 0.367234081f, -0.281228781f, 0.0930237472f, -0.380721033f, + 0.0879324526f, -0.176725745f, -0.525668383f, 0.210080102f, + -0.748570025f, -0.0751311556f, 0.208635211f, 0.0851098001f, + 0.278763115f, -0.357726544f, -0.338402867f, 0.20990704f, + 0.495543897f, 0.255115837f, 0.0740242749f, 0.536688864f, + -0.3042005f, -0.204800203f, 0.160633355f, -0.508065045f, + -0.569782019f, -0.230778813f, -0.511960268f, -0.304472923f, + -0.166226864f, -0.0940455198f, -0.0832107216f, -0.635292888f, + 0.440075874f, -0.307712018f, -0.00636816584f, 0.216251418f, + 0.425538749f, 0.13552317f, 0.168148905f, 0.206026599f, + 0.481180012f, 0.0192144271f, 0.301048934f, 0.0327274986f, + -0.595184863f, 0.0550124943f, -0.187376693f, 0.0152243599f, + 0.325533718f, 0.233476982f, 0.236022383f, -0.0777259022f, + 0.0910165906f, 0.0486684665f, 0.197047994f, -0.228946328f, + -0.0614259019f, -0.0119054811f, -0.27952826f, -0.175241813f, + 0.392222583f, 0.537081897f, -0.0274822358f, 0.158765569f, + 0.315507412f, +}; +const float kB_inject[] = { + -0.099082835f, -0.0356387049f, -0.239228144f, -0.353755474f, + 0.0235928036f, 0.213960662f, -0.329626769f, -0.060693007f, + 0.240905926f, -0.544646084f, 0.142132103f, -0.264545947f, + 0.38622573f, 0.241344586f, -0.165297106f, 0.157298028f, + -0.00542231742f, 0.389258742f, -0.26785627f, 0.407752007f, + 0.214632183f, 0.334532082f, -0.154605836f, -0.10579019f, + -0.268874347f, 0.163597748f, -0.101839714f, 0.151244223f, + 0.265165776f, -0.169093236f, 0.243119746f, 0.181456268f, + -0.0887573957f, 0.315174669f, 0.308018744f, -0.107801668f, + -0.0438321307f, 0.198384956f, -0.422011763f, -0.301602066f, + 0.271917701f, -0.528458297f, 0.231092364f, 0.353875756f, + -0.0754795745f, +}; +const float kB_hyper[] = { + -0.0926936939f, -0.165322334f, -0.328608423f, -0.52173382f, + -2.85028839f, 0.375549883f, -2.49926162f, 1.3132093f, + -2.36222649f, 0.339655042f, 0.381446481f, -0.609766603f, + -1.00423896f, 0.0982440189f, -2.66434526f, -1.24888241f, + -0.502287209f, -1.04099572f, 0.30450502f, 0.308207721f, + 1.56670952f, 0.343868107f, 0.715105891f, -0.84744972f, + -3.48773026f, -0.86844635f, 1.05246472f, 3.19339752f, + 1.46291411f, -0.323694646f, +}; +const float kB_normed[] = { + -0.0926521868f, -0.0521575958f, -0.155305654f, -0.266157627f, + -2.99260306f, 0.15307501f, -0.459443659f, 0.682391405f, + -2.09004879f, -0.0631238073f, 0.152227014f, -0.413058937f, + -0.828296304f, 0.0745620355f, 0.0941058844f, -2.0775249f, + -0.263728648f, -0.818797827f, 0.25852561f, 0.538546264f, + 0.593845844f, 0.0587842502f, 0.345556051f, -0.697264194f, + 0.602762997f, -0.269262224f, 0.553898573f, 2.04633141f, + 0.862590671f, 0.00888251793f, +}; +const float kB_mixed[] = { + 0.0409843884f, -0.154900193f, -0.0681974888f, -0.337243527f, + -0.542721152f, -0.266406178f, 0.0658957139f, 0.320129067f, + 0.0727220923f, 0.19605504f, +}; +const float kB_block_out[] = { + 0.162516147f, 1.08877027f, 0.900735974f, -0.52850914f, + -0.586997807f, -1.7903043f, 0.215952665f, -1.42978597f, + -0.603830397f, -0.470523119f, +}; +const float kB_inj_w[] = { + 0.926605344f, 0.792011797f, 1.01729059f, 1.1219033f, + 0.991320074f, 1.04880357f, +}; +const float kB_written[] = { + 0.0578946397f, 0.843538046f, 0.5060184f, -1.01145315f, + -3.39420366f, 0.504264593f, -1.63694274f, 2.02660275f, + -2.78081203f, -0.125254154f, 0.546772599f, 0.497829199f, + -0.0879287124f, -0.439403355f, -3.26149273f, -3.25743055f, + -0.260009199f, -2.64507723f, -0.372934312f, -0.219673723f, + -0.208055019f, 0.557946324f, -0.702269673f, -1.44603896f, + -3.95416927f, -2.74612379f, 1.27895665f, 1.69383287f, + 0.829614639f, -0.817180991f, +}; + +// ---- C: hidden=6 hc=4 lowrank=5 eps=1e-06 T=2 use_combine=False seed=7 +const float kC_norm_w_hf[] = { + -0.41006726f, 0.197815582f, 0.449454248f, -0.694201946f, + -0.0834980085f, 0.142574981f, -0.320545763f, -0.44682771f, + -0.424454302f, -0.608464718f, -0.0405107923f, 0.680742562f, + -0.651608348f, 0.0437391885f, -0.692480683f, 0.25178194f, + -0.433350801f, -0.282003343f, -1.02823126f, -0.178361267f, + 0.547790229f, 0.241177082f, 0.699459195f, 0.702926457f, +}; +const float kC_down[] = { + -0.0406310707f, 0.224135473f, 0.183544725f, 0.5603351f, + 0.753480911f, -0.376435578f, 0.244936958f, -0.319619685f, + -0.491106421f, 0.0473135747f, 0.118722282f, -0.410321921f, + -0.0302202962f, 0.0711043775f, 0.189804628f, -0.0275031514f, + -0.114634782f, -0.350068241f, 0.10907276f, -0.0946923345f, + -0.159623668f, 0.0897425562f, -0.426133364f, -0.0490364656f, + 0.20703733f, -0.0432785526f, 0.483614028f, 0.470137149f, + -0.281651765f, -0.371090591f, -0.0187702198f, 0.134852827f, + 0.464885741f, 0.179670006f, -0.191300347f, -0.685745656f, + -0.110299252f, -0.264654309f, 0.163803622f, 0.0445536375f, + -0.226696074f, 0.117523648f, 0.224094257f, 0.413931698f, + 0.386315972f, 0.260531425f, -0.41465956f, -0.288968593f, + 0.123151451f, 0.530249238f, -0.0249568112f, 0.152611747f, + 0.33534354f, 0.338571399f, 0.0424795523f, -0.163746893f, + 0.346262723f, -0.46097061f, -0.167305231f, -0.131500423f, + 0.347156048f, 0.0266828649f, 0.0797621235f, -0.0572044179f, + -0.22258766f, -0.191264898f, -0.473584116f, 0.193800792f, + 0.0226887427f, -0.045759052f, 0.0587491132f, -0.470626235f, + -0.206827581f, -0.0148539627f, -0.367177248f, 0.192081794f, + 0.205183297f, 0.483970195f, 0.279949456f, -0.293148041f, + -0.115149453f, -0.0802590922f, 0.203814164f, -0.118406326f, + 0.398153782f, 0.0331896394f, 0.101221211f, -0.096043095f, + -0.389389396f, -0.236175284f, 0.522656798f, 0.176406369f, + 0.0826550126f, -0.280092716f, -0.0458217785f, -0.158988968f, + 0.190192476f, 0.301090837f, -0.656415582f, -0.297816485f, + 0.266549796f, -0.446002662f, -0.266949981f, 0.270158857f, + 0.20464319f, -0.308050931f, -0.0237137731f, 0.0431467891f, + 0.420761615f, -0.125009f, 0.0496361405f, -0.214707822f, + -0.202037096f, -0.213739291f, -0.207309023f, 0.0693440288f, + -0.226673052f, -0.0359355621f, 0.286342263f, 0.0447572842f, +}; +const float kC_up[] = { + -0.141083673f, -0.254465073f, 0.245550364f, 0.0618241392f, + 0.210331157f, -0.207286909f, 0.0876560062f, 0.438380867f, + 0.0921838433f, 0.227796912f, 0.459998161f, -0.232636064f, + -0.645309031f, -0.383894295f, -0.296743751f, -0.00131497451f, + -0.10726504f, 0.0559139587f, -0.349616796f, 0.265466481f, + 0.38530001f, -0.251042336f, -0.428089052f, -0.0804627314f, + -0.790298283f, -0.00693282904f, 0.221626207f, 0.198762387f, + 0.225792453f, 0.172736347f, 0.0203587636f, -0.0294060465f, + 0.183083028f, -0.27174899f, -0.264219195f, -0.258202434f, + -0.202312738f, -0.906178772f, 0.120427661f, -0.31809479f, + -0.339675665f, -0.550266445f, -0.157941684f, 0.344351232f, + -0.0843541771f, -0.207026422f, 0.444297135f, -0.0143096251f, + 0.141363353f, 0.194367304f, 0.399482399f, -0.0885263309f, + -0.00430287607f, -0.172600865f, 0.119755954f, -0.149922773f, + -0.386173874f, -0.0570468642f, -0.0244993381f, -0.25560981f, + -0.125611126f, 0.102468662f, -0.0725202113f, 0.386872202f, + -0.215037942f, 0.258619815f, -0.130791947f, -0.145139575f, + -0.188357875f, 0.602519155f, 0.226324424f, 0.0191508532f, + 0.187765375f, -0.078549698f, -0.123748176f, 0.0292070229f, + 0.2364171f, -0.242859244f, 0.0279372595f, -0.0901294649f, + 0.00190478773f, -0.451937228f, -0.187343627f, 0.135894462f, + 0.243344992f, 0.832985222f, 0.229062229f, -0.0537902117f, + -0.426362664f, -0.313000292f, -0.0858951062f, 0.235061765f, + 0.0489826053f, -0.253047436f, 0.22458221f, 0.0365615226f, + -0.0446157195f, 0.441168159f, 0.0601178966f, -0.20807749f, + 0.180020928f, -0.226440027f, 0.487573326f, 0.539725423f, + -0.150293678f, -0.183902428f, -0.108619519f, 0.464962065f, + -0.670442641f, 0.316658169f, 0.125152096f, 0.601781487f, + 0.195903376f, 0.800311983f, 0.210899025f, 0.30417484f, + 0.217664585f, 0.210095271f, -0.236560598f, 0.0700418428f, +}; +const float kC_hyper[] = { + -0.230136275f, -1.45792675f, -1.91336703f, 4.40088654f, + 0.175186977f, 1.52687156f, -2.71076894f, -1.03763235f, + -0.86155194f, 1.77705753f, -2.43255424f, 1.65857899f, + 0.738127708f, 3.41089416f, 1.9053688f, 0.119107477f, + 1.44350028f, -1.10254669f, -2.77361345f, 2.10040307f, + 0.59310782f, 0.300421387f, 1.25117958f, -0.693412304f, + 1.34150624f, 0.328173578f, -0.499845415f, 1.92204094f, + -0.120057032f, -0.353418171f, -1.14711487f, 2.3258779f, + -0.600307107f, -0.16371116f, 2.94823527f, 1.17002559f, + -0.981115401f, 2.0027864f, -0.965839982f, 0.683383763f, + 0.727447212f, -2.1764214f, 0.629573762f, 2.55347562f, + 0.392030865f, -2.98006725f, 0.911427855f, -1.61496496f, +}; +const float kC_normed[] = { + -0.0633359328f, -0.814682186f, -1.29379487f, 0.627823353f, + 0.074902758f, 0.813860238f, -0.984662235f, -0.306858748f, + -0.265091538f, 0.371969193f, -1.2477777f, 1.49029386f, + 0.144044727f, 1.99415255f, 0.328208327f, 0.0835152566f, + 0.458172768f, -0.443422288f, 0.0502088852f, 1.10659289f, + 0.588640451f, 0.239094153f, 1.36343586f, -0.757167161f, + 0.792009294f, 0.393394828f, -0.725062251f, 0.588210046f, + -0.110117443f, -0.404118478f, -0.460690677f, 0.760483265f, + -0.204218745f, -0.037887115f, 1.672032f, 1.1623553f, + -0.245411262f, 1.50083613f, -0.213247627f, 0.614186049f, + 0.295952767f, -1.12194622f, -0.00988700334f, 1.16707921f, + 0.337536007f, -2.05753636f, 0.861629903f, -1.5298425f, +}; +const float kC_mixed[] = { + -0.10553544f, 0.242798343f, -0.0706134364f, 0.164503202f, + 0.0861888379f, 0.155533314f, 0.0161547624f, 0.482847929f, + -0.0923302919f, -0.118207321f, 0.344788611f, -0.220340937f, +}; diff --git a/tests/vllm/models/test_qwen4_exp_hc.cpp b/tests/vllm/models/test_qwen4_exp_hc.cpp new file mode 100644 index 000000000..c9b40f149 --- /dev/null +++ b/tests/vllm/models/test_qwen4_exp_hc.cpp @@ -0,0 +1,421 @@ +// Qwen4-Exp (Qwen3.8-Flash-Next) W3 UNIT GATE — the 4-branch GATED-RESIDUAL +// hyper-connection stream and the grouped RMSNorm it stands on. +// Issue #1988, spec `.agents/specs/qwen4-exp-flash-next.md`. +// +// WHAT THE GOLDENS ARE. `qwen4_exp_hc_goldens.inc` is dumped by +// `scripts/gen-qwen4-exp-hc-goldens.py`, which lifts `Qwen4ExpTextRMSNorm` +// (:158-181) and `Qwen4ExpTextGatedResidual` (:941-969) VERBATIM by line range +// out of the lane-pinned oracle — transformers **v5.16.0** +// `src/transformers/models/qwen4_exp/modeling_qwen4_exp.py`, sha256 +// 77fec77d87f2a0eb23b95fa04276fb5779698a7c7f523cf5061e49c118bcc459 — and +// EXECUTES them under torch. Nothing was retyped, and the script re-checks that +// sha256 before it runs. The one documented harness adaptation is a four-field +// stand-in for `Qwen4ExpTextConfig`: the real dataclass drags in the whole +// transformers package, and the classes read only `hidden_size`, `hc_count`, +// `hc_lowrank` and `rms_norm_eps`. The write-back goldens replay the two +// verbatim lines of `Qwen4ExpTextDecoderLayer.forward`. +// +// WHY A DISCRIMINATION SWEEP AND NOT ONLY A GOLDEN COMPARE. Five of this +// module's guarantees are single-character defects that a plausible port makes +// silently, and a golden that cannot see them is decoration. `Variant` below is +// an INDEPENDENT double-precision reference with one flag per defect; the test +// asserts the correct spelling matches the oracle AND that each single-flag flip +// does NOT, which measures the goldens' discriminating power rather than +// assuming it (AGENTS.md "Gates", `.agents/verification.md`). +// +// SCOPE, HONESTLY. Host reference only. No token claim, no speed claim: no +// `qwen4_exp` arm runs anywhere yet (the smallest published checkpoint is +// ~128 GB against a ~119 GB budget, spec `## Hardware`), and W1 config +// registration is still in review. What this gate proves is the arithmetic. +#include "vllm/model_executor/models/qwen4_exp_hc.h" + +#include + +#include +#include +#include + +#include "support/max_abs_diff.h" +#include "vllm/model_executor/models/deepseek_v4_mhc.h" + +using namespace vllm::qwen4_exp; +using vllm_test::MaxAbsDiff; + +namespace { + +#include "qwen4_exp_hc_goldens.inc" // NOLINT — golden literals + +// The goldens are fp32 out of torch; this reference is fp32 with double-precision +// reductions (the `deepseek_v4_mhc.cpp` house convention for a host reference). +constexpr double kTol = 1e-5; + +// The floor a Variant flip has to clear. It is not a round number picked to be +// safe: it is 6.6x under the SMALLEST separation actually measured, and 100x +// over kTol, so the two bands cannot touch. Measured on case A, max|diff| of the +// flipped double reference against the oracle goldens, per token: +// +// flip t=0 t=1 t=2 affects +// silu-div-after 1.79e-2 6.63e-3 9.54e-3 mixed_input +// div-on-up 1.97e-2 1.95e-2 1.81e-2 mixed_input +// sum-not-mean 1.03e+0 6.91e-1 1.17e+0 mixed_input +// raw-not-normed 4.77e-1 3.39e-1 5.66e-1 mixed_input +// inject-no-div 5.22e-1 2.82e-1 5.49e-1 injection_weights +// +// The SiLU-division placement is the narrowest of the five, which is exactly the +// reason it is worth a gate: it is a one-token-deep edit that moves the answer +// by well under a percent at some tokens and would survive a loose eyeball. +constexpr double kFlipFloor = 1e-3; + +std::vector Load(const float* p, size_t n) { return std::vector(p, p + n); } + +// ─── The independent double-precision reference, with one flag per known trap ── +struct Variant { + bool div_before_silu = true; // silu(down(x)/hc), NOT silu(down(x))/hc + bool div_on_up = false; // there is NO division on the up-projection + bool reduce_is_mean = true; // .mean(dim=-2), NOT .sum + bool multiply_normed = true; // against hyper_input_normed, NOT hyper_input + bool inject_div = true; // 2*sigmoid(inject(x)/hc) + bool norm_is_grouped = true; // hc independent reductions, NOT one global one + bool norm_weight_plus_one = true; // (1.0 + w_hf) +}; + +double SigmoidD(double x) { return 1.0 / (1.0 + std::exp(-x)); } + +std::vector NormRefD(const std::vector& x, const std::vector& w_hf, + int64_t hc, int64_t hidden, double eps, const Variant& v) { + std::vector out(x.size()); + const int64_t groups = v.norm_is_grouped ? hc : 1; + const int64_t gsize = v.norm_is_grouped ? hidden : hc * hidden; + for (int64_t g = 0; g < groups; ++g) { + double ss = 0.0; + for (int64_t d = 0; d < gsize; ++d) { + const double t = x[g * gsize + d]; + ss += t * t; + } + const double r = 1.0 / std::sqrt(ss / static_cast(gsize) + eps); + for (int64_t d = 0; d < gsize; ++d) { + const double scale = v.norm_weight_plus_one ? 1.0 + w_hf[g * gsize + d] : w_hf[g * gsize + d]; + out[g * gsize + d] = x[g * gsize + d] * r * scale; + } + } + return out; +} + +std::vector LinearD(const float* w, const std::vector& x, int64_t out_dim, + int64_t in_dim) { + std::vector y(static_cast(out_dim), 0.0); + for (int64_t o = 0; o < out_dim; ++o) { + double acc = 0.0; + for (int64_t i = 0; i < in_dim; ++i) acc += static_cast(w[o * in_dim + i]) * x[i]; + y[o] = acc; + } + return y; +} + +struct RefOut { + std::vector normed; + std::vector mixed; + std::vector injection; +}; + +RefOut ForwardRefD(const std::vector& hyper, const std::vector& w_hf, + const float* down, const float* up, const float* inject, int64_t hc, + int64_t hidden, int64_t rank, double eps, const Variant& v) { + const int64_t flat = hc * hidden; + RefOut o; + o.normed = NormRefD(hyper, w_hf, hc, hidden, eps, v); + + std::vector lo = LinearD(down, o.normed, rank, flat); + for (int64_t r = 0; r < rank; ++r) { + const double a = v.div_before_silu ? lo[r] / static_cast(hc) : lo[r]; + const double s = a * SigmoidD(a); + lo[r] = v.div_before_silu ? s : s / static_cast(hc); + } + std::vector gate = LinearD(up, lo, flat, rank); + for (int64_t p = 0; p < flat; ++p) { + gate[p] = SigmoidD(v.div_on_up ? gate[p] / static_cast(hc) : gate[p]); + } + + o.mixed.assign(static_cast(hidden), 0.0); + for (int64_t j = 0; j < hc; ++j) { + for (int64_t h = 0; h < hidden; ++h) { + const double stream = v.multiply_normed ? o.normed[j * hidden + h] : hyper[j * hidden + h]; + o.mixed[h] += gate[j * hidden + h] * stream; + } + } + if (v.reduce_is_mean) { + for (int64_t h = 0; h < hidden; ++h) o.mixed[h] /= static_cast(hc); + } + + if (inject != nullptr) { + std::vector logits = LinearD(inject, o.normed, hc, flat); + o.injection.resize(static_cast(hc)); + for (int64_t j = 0; j < hc; ++j) { + o.injection[j] = + 2.0 * SigmoidD(v.inject_div ? logits[j] / static_cast(hc) : logits[j]); + } + } + return o; +} + +// Compare an fp32 result against the double reference directly, for the cases +// where no oracle golden exists (a deliberately huge eps). +double MaxAbsDV(const std::vector& got, const std::vector& want) { + REQUIRE(got.size() == want.size()); + double worst = 0.0; + for (size_t i = 0; i < got.size(); ++i) { + worst = std::max(worst, std::abs(static_cast(got[i]) - want[i])); + } + return worst; +} + +double MaxAbsD(const std::vector& got, const float* want) { + double worst = 0.0; + for (size_t i = 0; i < got.size(); ++i) { + worst = std::max(worst, std::abs(got[i] - static_cast(want[i]))); + } + return worst; +} + +} // namespace + +// ── 1. The grouped RMSNorm, against vLLM's `RMSNormGated(group_size=)` form ──── +// vLLM `model_executor/layers/layernorm.py` @ origin/main 6a5e8f5979: `RMSNorm` +// opens at :37 and has no group knob; `RMSNormGated` opens at :172 with +// `group_size` at :187, and its grouped branch is :258-264 — +// `rearrange(x, "... (g d) -> ... g d")`, `variance = x_group.pow(2).mean(-1)`, +// `x_normed = x_group * rsqrt(variance + eps)`, `out = flatten * weight`. That is +// the op we mirror; the SEMANTICS (which weight, which eps) come from +// transformers, and the two differ by exactly the `1 + w` transform below. +TEST_CASE("qwen4_exp grouped RMSNorm mirrors RMSNormGated(group_size) at the lane pin") { + const int64_t hc = 4, hidden = 6; + const std::vector hyper = Load(kA_hyper, static_cast(hc * hidden)); + const std::vector w_hf = Load(kA_norm_w_hf, static_cast(hc * hidden)); + + const std::vector w = HcNormWeightFromHf(w_hf); + const std::vector got = GroupedRmsNorm(hyper, w, hidden, 1e-6f); + CHECK(MaxAbsDiff(got, kA_normed, hyper.size()) < kTol); + + // A GLOBAL reduction over all 24 elements is the defect a plain RMSNorm port + // makes. It must be visibly different, or the case above proves nothing. + Variant global; + global.norm_is_grouped = false; + CHECK(MaxAbsD(NormRefD(hyper, w_hf, hc, hidden, 1e-6, global), kA_normed) > 1e-2); + + // eps lives INSIDE the rsqrt, added to the mean square. At the model's real + // 1e-6 the two placements differ by ~1e-6, which is UNDER this suite's fp32 + // tolerance — so an eps-placement defect is invisible to every golden case + // above, and a case at an eps large enough to separate them is the only thing + // that gates it. `rsqrt(m + 4)` and `1/(sqrt(m) + 4)` are far apart; the + // double reference puts eps inside, so this pins the placement rather than + // merely observing that eps does something. + const std::vector big_eps = GroupedRmsNorm(hyper, w, hidden, 4.0f); + CHECK(MaxAbsDiff(big_eps, got) > 1e-2); + CHECK(MaxAbsDV(big_eps, NormRefD(hyper, w_hf, hc, hidden, 4.0, Variant{})) < kTol); +} + +TEST_CASE("qwen4_exp hc_norm weight is 1 + w_hf, applied exactly once") { + const std::vector w_hf = Load(kA_norm_w_hf, 24); + const std::vector w = HcNormWeightFromHf(w_hf); + REQUIRE(w.size() == w_hf.size()); + for (size_t i = 0; i < w.size(); ++i) CHECK(w[i] == doctest::Approx(1.0f + w_hf[i])); + + const std::vector hyper = Load(kA_hyper, 24); + // The HF weight used raw is the "forgot the +1" defect: near-zero scale, which + // reads as a checkpoint bug rather than a port bug (spec, trap 2). + CHECK(MaxAbsDiff(GroupedRmsNorm(hyper, w_hf, 6, 1e-6f), kA_normed, 24) > 1e-1); + // Adding 1 a SECOND time is the defect a reader of the published GGUF makes, + // because the convert step has already folded it (issue #1988, trap 2). + CHECK(MaxAbsDiff(GroupedRmsNorm(hyper, HcNormWeightFromHf(w), 6, 1e-6f), kA_normed, 24) > 1e-1); +} + +TEST_CASE("qwen4_exp grouped RMSNorm refuses an indivisible width, as upstream does") { + // The MESSAGE is asserted, not merely the type. Every guard in this TU throws + // `std::invalid_argument`, so a type-only assertion passes when the guard that + // fires is a DIFFERENT one further down — which is exactly what a mutation + // sweep found: deleting a refusal left the suite green because a later bounds + // check threw the same type. Pinning the text makes each guard its own gate. + const std::vector x(10, 1.0f), w(10, 1.0f); + CHECK_THROWS_WITH_AS(GroupedRmsNorm(x, w, 4, 1e-6f), + "qwen4_exp: hidden_size (10) must be divisible by group_size (4).", + std::invalid_argument); + CHECK_THROWS_WITH_AS(GroupedRmsNorm(x, std::vector(9, 1.0f), 5, 1e-6f), + "qwen4_exp: hc_norm weight has 9 elements, expected 10.", + std::invalid_argument); +} + +// ── 2. The gated-residual READ, both arms ───────────────────────────────────── +TEST_CASE("qwen4_exp GatedResidual read matches transformers v5.16.0, use_combine=true") { + const int64_t hc = 4, hidden = 6, rank = 5, flat = hc * hidden; + GatedResidualWeights w; + w.hc_norm_weight = HcNormWeightFromHf(Load(kA_norm_w_hf, static_cast(flat))); + w.mix_down = Load(kA_down, static_cast(rank * flat)); + w.mix_up = Load(kA_up, static_cast(flat * rank)); + w.block_inject = Load(kA_inject, static_cast(hc * flat)); + + for (int64_t t = 0; t < 3; ++t) { + const std::vector hyper = Load(kA_hyper + t * flat, static_cast(flat)); + const GatedResidualResult r = GatedResidualForward(hyper, w, hc, hidden, 1e-6f); + CHECK(MaxAbsDiff(r.hyper_input_normed, kA_normed + t * flat, static_cast(flat)) < + kTol); + CHECK(MaxAbsDiff(r.mixed_input, kA_mixed + t * hidden, static_cast(hidden)) < kTol); + CHECK(MaxAbsDiff(r.injection_weights, kA_inj_w + t * hc, static_cast(hc)) < kTol); + // Range of 2*sigmoid is (0, 2) with 1.0 at a zero logit. + for (float v : r.injection_weights) CHECK((v > 0.0f && v < 2.0f)); + } +} + +TEST_CASE("qwen4_exp GatedResidual read matches transformers v5.16.0 at hc_count=3") { + // hc_count is a config value, not the constant 4. A port that folds 4 into the + // two divisions passes case A and fails here. + const int64_t hc = 3, hidden = 5, rank = 7, flat = hc * hidden; + GatedResidualWeights w; + w.hc_norm_weight = HcNormWeightFromHf(Load(kB_norm_w_hf, static_cast(flat))); + w.mix_down = Load(kB_down, static_cast(rank * flat)); + w.mix_up = Load(kB_up, static_cast(flat * rank)); + w.block_inject = Load(kB_inject, static_cast(hc * flat)); + + for (int64_t t = 0; t < 2; ++t) { + const std::vector hyper = Load(kB_hyper + t * flat, static_cast(flat)); + const GatedResidualResult r = GatedResidualForward(hyper, w, hc, hidden, 1e-5f); + CHECK(MaxAbsDiff(r.hyper_input_normed, kB_normed + t * flat, static_cast(flat)) < + kTol); + CHECK(MaxAbsDiff(r.mixed_input, kB_mixed + t * hidden, static_cast(hidden)) < kTol); + CHECK(MaxAbsDiff(r.injection_weights, kB_inj_w + t * hc, static_cast(hc)) < kTol); + + const std::vector block = Load(kB_block_out + t * hidden, static_cast(hidden)); + CHECK(MaxAbsDiff(GatedResidualWriteBack(hyper, block, r.injection_weights, hc, hidden), + kB_written + t * flat, static_cast(flat)) < kTol); + } +} + +TEST_CASE("qwen4_exp GatedResidual with use_combine=false is the final mixer") { + const int64_t hc = 4, hidden = 6, rank = 5, flat = hc * hidden; + GatedResidualWeights w; + w.hc_norm_weight = HcNormWeightFromHf(Load(kC_norm_w_hf, static_cast(flat))); + w.mix_down = Load(kC_down, static_cast(rank * flat)); + w.mix_up = Load(kC_up, static_cast(flat * rank)); + w.block_inject.clear(); // == `block_inject_weight = None` + + for (int64_t t = 0; t < 2; ++t) { + const std::vector hyper = Load(kC_hyper + t * flat, static_cast(flat)); + const GatedResidualResult r = GatedResidualForward(hyper, w, hc, hidden, 1e-6f); + CHECK(MaxAbsDiff(r.hyper_input_normed, kC_normed + t * flat, static_cast(flat)) < + kTol); + CHECK(MaxAbsDiff(r.mixed_input, kC_mixed + t * hidden, static_cast(hidden)) < kTol); + // `Qwen4ExpTextGatedResidual.forward` returns EARLY here: no injection branch + // is computed at all, and there is no trailing model norm after it either + // (`Qwen4ExpTextModel.forward` goes straight to `lm_head` — spec, trap 1). + CHECK(r.injection_weights.empty()); + CHECK(r.mixed_input.size() == static_cast(hidden)); + } +} + +TEST_CASE("qwen4_exp GatedResidual refuses a stream that is not hc_count * hidden_size") { + GatedResidualWeights w; + w.hc_norm_weight.assign(24, 1.0f); + w.mix_down.assign(5 * 24, 0.0f); + w.mix_up.assign(24 * 5, 0.0f); + // Message-pinned for the reason given on the grouped-norm refusal above: with + // a type-only assertion the 23- and 25-wide cases were caught downstream by + // GroupedRmsNorm's divisibility guard, and deleting THIS refusal changed + // nothing the suite could see. A 30-wide stream is divisible by hidden_size, + // so it walks past that guard too and reaches the weight-size check instead. + CHECK_THROWS_WITH_AS(GatedResidualForward(std::vector(23, 1.0f), w, 4, 6, 1e-6f), + "qwen4_exp: expected 24 hyper-connection features, got 23.", + std::invalid_argument); + CHECK_THROWS_WITH_AS(GatedResidualForward(std::vector(25, 1.0f), w, 4, 6, 1e-6f), + "qwen4_exp: expected 24 hyper-connection features, got 25.", + std::invalid_argument); + CHECK_THROWS_WITH_AS(GatedResidualForward(std::vector(30, 1.0f), w, 4, 6, 1e-6f), + "qwen4_exp: expected 24 hyper-connection features, got 30.", + std::invalid_argument); +} + +// ── 3. The rank-1 write-back ────────────────────────────────────────────────── +TEST_CASE("qwen4_exp write-back is hyper_input RAW plus outer(block_out, injection)") { + const int64_t hc = 4, hidden = 6, flat = hc * hidden; + for (int64_t t = 0; t < 3; ++t) { + const std::vector hyper = Load(kA_hyper + t * flat, static_cast(flat)); + const std::vector block = Load(kA_block_out + t * hidden, static_cast(hidden)); + const std::vector inj = Load(kA_inj_w + t * hc, static_cast(hc)); + const std::vector got = GatedResidualWriteBack(hyper, block, inj, hc, hidden); + CHECK(MaxAbsDiff(got, kA_written + t * flat, static_cast(flat)) < kTol); + + // The un-normed stream is what gets written. Feeding the NORMED stream here + // is the natural mistake and must be visible. + const std::vector normed = Load(kA_normed + t * flat, static_cast(flat)); + CHECK(MaxAbsDiff(GatedResidualWriteBack(normed, block, inj, hc, hidden), + kA_written + t * flat, static_cast(flat)) > 1e-2); + + // The in-place form is the fusion seam: it must agree bit-for-bit with the + // allocating one, so a device arm can replace one with the other. + std::vector inplace = hyper; + GatedResidualWriteBackInPlace(inplace.data(), block.data(), inj.data(), hc, hidden); + for (size_t i = 0; i < inplace.size(); ++i) CHECK(inplace[i] == got[i]); + } +} + +// The spec asserts our DeepSeek-V4 `MhcPost` is a bit-exact bring-up bridge for +// this write-back once its comb matrix is the identity. Verified rather than +// trusted, because it is the reviewer's free mutation target. It holds on finite +// inputs; it is NOT an identity for a negative-zero residual (0.0f + (-0.0f) +// is +0.0f) or a non-finite one (0.0f * inf is NaN), and neither case is +// reachable on this path. +TEST_CASE("qwen4_exp write-back equals MhcPost with an identity comb, bit for bit") { + const int64_t hc = 4, hidden = 6, flat = hc * hidden; + std::vector comb(static_cast(hc * hc), 0.0f); + for (int64_t j = 0; j < hc; ++j) comb[j * hc + j] = 1.0f; + + for (int64_t t = 0; t < 3; ++t) { + const std::vector hyper = Load(kA_hyper + t * flat, static_cast(flat)); + const std::vector block = Load(kA_block_out + t * hidden, static_cast(hidden)); + const std::vector inj = Load(kA_inj_w + t * hc, static_cast(hc)); + const std::vector ours = GatedResidualWriteBack(hyper, block, inj, hc, hidden); + const std::vector bridge = + vllm::deepseek_v4::MhcPost(block, hyper, inj, comb, hc, hidden); + REQUIRE(bridge.size() == ours.size()); + for (size_t i = 0; i < ours.size(); ++i) CHECK(bridge[i] == ours[i]); + } +} + +// ── 4. Discriminating power: each trap opens a gap the goldens can see ──────── +TEST_CASE("qwen4_exp goldens separate every known gated-residual trap") { + const int64_t hc = 4, hidden = 6, rank = 5, flat = hc * hidden; + const std::vector w_hf = Load(kA_norm_w_hf, static_cast(flat)); + + for (int64_t t = 0; t < 3; ++t) { + const std::vector hyper = Load(kA_hyper + t * flat, static_cast(flat)); + const RefOut ok = ForwardRefD(hyper, w_hf, kA_down, kA_up, kA_inject, hc, hidden, rank, 1e-6, + Variant{}); + // The independent double reference reproduces the oracle. Two references + // agreeing is what makes a single-flag flip attributable to the flag. + CHECK(MaxAbsD(ok.mixed, kA_mixed + t * hidden) < kTol); + CHECK(MaxAbsD(ok.injection, kA_inj_w + t * hc) < kTol); + + struct Case { + Variant v; + bool hits_mixed; + bool hits_injection; + }; + Variant a; + a.div_before_silu = false; // silu(x)/hc instead of silu(x/hc) + Variant b; + b.div_on_up = true; // a division upstream does NOT have + Variant c; + c.reduce_is_mean = false; // sum instead of mean + Variant d; + d.multiply_normed = false; // the raw stream instead of the normed one + Variant e; + e.inject_div = false; // 2*sigmoid(logits) instead of 2*sigmoid(logits/hc) + const Case cases[] = {{a, true, false}, {b, true, false}, {c, true, false}, + {d, true, false}, {e, false, true}}; + + for (const Case& k : cases) { + const RefOut bad = + ForwardRefD(hyper, w_hf, kA_down, kA_up, kA_inject, hc, hidden, rank, 1e-6, k.v); + if (k.hits_mixed) CHECK(MaxAbsD(bad.mixed, kA_mixed + t * hidden) > kFlipFloor); + if (k.hits_injection) CHECK(MaxAbsD(bad.injection, kA_inj_w + t * hc) > kFlipFloor); + } + } +} From 09ce43f760e30b107b12d778de3e2f517f56bd33 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 19:44:09 +0000 Subject: [PATCH 2/3] =?UTF-8?q?fix(MODEL-MM-QWEN4-EXP):=20W3=20=E2=80=94?= =?UTF-8?q?=20the=20gate=20could=20not=20see=20a=20NaN,=20eight=20deleted?= =?UTF-8?q?=20refusals,=20or=20the=20model's=20own=20width=20(#1988)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fresh review of a5bd41302 returned FAIL with eight findings, and its own summary is the shape of this change: "the port itself is the strongest part of this change and I could not break it ... Every finding is about the GATE, not the arithmetic." `qwen4_exp_hc.cpp` is byte-for-byte unchanged here and the header change is comments only. What changes is what the suite can detect. Issue #449, re-opened locally. `test_qwen4_exp_hc.cpp` defined two local max|diff| helpers spelled `worst = std::max(worst, std::abs(...))` — verbatim the NaN-blind "form B" that `tests/support/max_abs_diff.h` exists to prevent, in a file that already included that header. Reproduced before repairing: poisoning `GroupedRmsNorm`'s large-eps path to all-NaN left `CHECK(MaxAbsDV(big_eps, ...) < kTol)` GREEN, and the suite only went red because the sibling assertion one line up happened to use the hardened helper. The fix is not a third local copy. `MaxAbsDiffScan` and `MaxAbsDiff` are now templated on both operand types, so an fp32-result-against-double-reference comparison — the shape every "second independent reference" case takes — routes through the same hardened scan instead of being written again by hand. The same mutation now reddens the assertion itself. Eight of eleven refusals had no assertion at all. Each was deleted in turn and the suite stayed 280/280 green each time, including all three `RequireSize` calls in `GatedResidualWriteBack` — which, since `GatedResidualWriteBackInPlace` takes raw pointers and the header declares it the seam a device kernel replaces, are the only bounds check between a caller and an out-of-bounds read. That class of guard does not fail tidily: deleting the divisibility guard produces `Fatal glibc error: malloc.c:2599` and `Aborted (core dumped)` with no doctest summary at all, and deleting the `group_size` guard is `x.size() % 0` and SIGFPE. All eight are now message-pinned with `CHECK_THROWS_WITH_AS`, because every guard in the translation unit throws `std::invalid_argument` and a type-only assertion cannot tell a deleted guard from a different one firing further down. `kTol = 1e-5` was a property of the toy shapes and nothing else, and this is the part the review ranked MEDIUM that deserved the most work. At flat = 24 and 15 the implementation is bit-identical to the oracle — 2.384e-07 over every golden array — so the tolerance was unconstrained there. Measured against the pinned oracle at the model's own shape (hidden 2560, hc 4, lowrank 320), max|diff| on `mixed_input` is 2.325e-05 and 2.137e-05 over two tokens, 2.1x to 2.3x over. The reflex is to widen kTol, which AGENTS.md forbids and which would be wrong anyway. The measurement that settles it is the second row of the table now in the file: an EXACT double evaluation of the same algorithm is itself 1.360e-05 from the oracle, because torch runs this in fp32 too. No fp32 implementation meets a 1e-5 absolute bound at hidden_size 2560, and widening our own accumulator cannot rescue one — so the arithmetic is not the thing that is wrong at that width, the absolute tolerance is. kTol is unchanged and is now documented as scale-bound; a real-width case carries a RELATIVE bound derived as 6.6x the sqrt(K)*u random-walk bound for a sequential fp32 dot of K = 10240, and all three measurements sit inside it. Agreement with the oracle at model width needs a real checkpoint and is recorded as owed rather than claimed. The `double` accumulator was documented and ungated: `double ss` to `float ss` left 280/280 green, so the gate could not distinguish the convention from its opposite. It is kept, because it makes the host reference more accurate than the oracle rather than less, and it is now gated at the model's real group size of 2560 on magnitude-separated data, where the two differ by 742x (3.168430e-06 against 2.352230e-03, bound 1e-4, identical at -O0 and -O2). The consequence for the device wave is stated in the spec rather than left to be discovered: a straight fp32-accumulate device reduction will not meet that bound on that data, and that is the signal, not a defect in the gate. The GGUF fold claim was true and under-scoped in the way that causes the defect it warns about. Read at source rather than relayed: llama.cpp mainline has no `qwen4exp` at all, the converter is the still-open ggml-org/llama.cpp#27742 at 035e22731a7fd70b9854b3a2d64ec68e9b1a45d3, and it adds no `hc_norm` branch — the `+1` comes from the inherited `Qwen3NextModel` rule at `conversion/qwen.py` :302-303, `endswith("norm.weight") and not endswith("linear_attn.norm.weight")`. So the fold covers `hc_norm`, `attn_q_norm`, `attn_k_norm` and the PLE and indexer gammas, and excludes `ssm_norm`. A W5 loader author reading the old sentence would have skipped the fold for `hc_norm` and double-folded everything else. The header and the spec now state the rule, its exception, the PR and its head SHA, and that it is unmerged and may change. Three smaller ones. "Bit for bit" was asserted with `==`, which cannot see the negative-zero divergence the comment beside it names; both sites now compare bit patterns, and a case pins the helper's polarity. The in-place-versus-allocating check was a tautology — the wrapper calls the primitive — proven by mutating the primitive and watching that line stay green while three others reddened; the seam is now gated against the oracle directly. `Variant::norm_weight_plus_one` was declared and read but never once set false, so the struct advertised coverage the file did not have; it is now flipped in the sweep. And the goldens generator pinned its oracle with a bare `assert`, which `python3 -O` strips. Verified both ways: under `-O` the old script happily dumped goldens from a corrupted source file at rc=0, the new one exits 1. The generator still reproduces the committed `.inc` byte for byte. `.agents/issue-index.md` is append-only, so the under-scoped fold sentence in the #1988 row stays where it is; the header and the spec are the surfaces a loader author reads and both are corrected there. Still NOT REACHED at this commit, unchanged from a5bd41302: the wiring is owed by W5, assembly, under #1978, and is recorded under the spec's `## Owed`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/qwen4-exp-flash-next.md | 84 +++- scripts/gen-qwen4-exp-hc-goldens.py | 6 +- src/vllm/model_executor/models/qwen4_exp_hc.h | 45 +- tests/support/max_abs_diff.h | 27 +- tests/vllm/models/test_qwen4_exp_hc.cpp | 394 ++++++++++++++++-- 5 files changed, 520 insertions(+), 36 deletions(-) diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index 910bee2cd..1d204336a 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -421,6 +421,38 @@ that does not exist. Stated because that tail is the natural thing to copy. `w_vllm = 1.0 + w_hf`. Miss it and every `hc_norm` gets a near-zero scale, which reads as a checkpoint bug rather than a port bug. +**The GGUF converter already folds it, and it folds far more than `hc_norm`.** Read at +source rather than relayed, because W5 writes the loader and the narrow version of this +sentence causes the defect it warns about. llama.cpp mainline has no `qwen4exp` at all +(`git grep -il qwen4exp` at `237ad9b96`: nothing tree-wide, so mainline can neither +convert nor load this architecture). The converter is ggml-org/llama.cpp +[#27742](https://github.com/ggml-org/llama.cpp/pull/27742), head +`035e22731a7fd70b9854b3a2d64ec68e9b1a45d3`, **still OPEN**. Its `conversion/qwen4exp.py` +declares `class Qwen4ExpTextModel(_Qwen35MRopeMixin, _LinearAttentionVReorderBase)`; +`_LinearAttentionVReorderBase` is `conversion/qwen.py:353`, a subclass of +`Qwen3NextModel` (`:270`); and the PR's `modify_tensors` has **no `hc_norm` branch**, so +`hc_norm.weight` falls through to `super()`. The `+1` is the inherited Qwen3-Next rule at +`conversion/qwen.py:302-303`: + +```python +elif name.endswith("norm.weight") and not name.endswith("linear_attn.norm.weight"): + data_torch = data_torch + 1 +``` + +So the rule a loader implements is **every `*norm.weight` carries the fold, with +`linear_attn.norm.weight` (the GDN `ssm_norm`) the one exception** -- `hc_norm`, +`attn_q_norm` and `attn_k_norm` all match it, and the PLE and indexer gammas are folded +by the PR's own early-returning branch. A loader that skips the fold for `hc_norm` alone +double-folds everything else, which is the same silent ~2x defect one tensor to the left. +Two consequences for W5. The property belongs to one in-flight converter, not to "GGUF": +#27742 can change before it merges and another publisher's tool need not match it, so the +loader treats the fold as a provenance question and checks it -- cheaply, since an +unfolded `hc_norm` is a zero-init gamma and a folded one is centred on 1.0. And it was +corroborated on published artifacts during fresh review of #1988 +(`unsloth/Qwen3.8-Flash-Next-GGUF` `UD-IQ1_S` and `UD-Q4_K_XL`, `vumpt/...-Q4_K_M`, read +by HTTP range request against the bf16 HF tensors): every `*hc_norm.weight` is HF + 1.0 +exactly, elementwise, while `ssm_norm` is unfolded and sits in [0.875, 1.023]. + **Correction to the port map above.** vLLM's grouped RMSNorm is on **`RMSNormGated`**, not the plain `RMSNorm`, whose only related knob is `var_hidden_size` -- a prefix reduction that cannot express per-group norms. Verified directly: `RMSNorm` opens at @@ -764,11 +796,53 @@ change that makes any arm reachable, not later. `.agents/model-matrix.md` is a single shared file, so three parallel waves editing one cell is the write-lock AGENTS.md "Records" names. Whichever of W1/W2/W3 lands last owes the correction. -- The **device arm of the gated residual**, and with it two checks this host - wave cannot make: that `RMSNormGated.forward_cuda`'s flash-linear-attention - Triton kernel is numerically correct in its GROUPED mode (unverified upstream, - see `## Design`), and that an fp32-accumulate device reduction still lands - inside this gate's tolerance against the double-accumulated host reference. +- The **device arm of the gated residual**, and with it one check this host wave + cannot make: that `RMSNormGated.forward_cuda`'s flash-linear-attention Triton + kernel is numerically correct in its GROUPED mode (unverified upstream, see + `## Design`). +- **W3's `kTol = 1e-5` is an absolute bound that does not survive a rescale, and + the host reference is the first thing it fails.** Recorded because an earlier + draft of the bullet above framed the tolerance question as the DEVICE arm's + problem, and it is not. Measured against the pinned oracle itself, at the + model's own shape (hidden_size 2560, hc_count 4, hc_lowrank 320, eps 1e-6, two + tokens), max|diff| on `mixed_input`: + + | | t=0 | t=1 | + |---|---|---| + | ours (fp32) vs oracle | 2.325e-05 | 2.137e-05 | + | exact double vs oracle | 1.360e-05 | 5.431e-06 | + | ours (fp32) vs exact double | 3.684e-05 | 1.606e-05 | + + At the suite's own widths (flat = 24 and 15) the implementation is bit-identical + to the oracle -- max|diff| over every golden array of cases A, B and C is + 2.384e-07 -- so kTol carries a 42x margin there and constrains nothing. At model + width our fp32 interior is 2.1x to 2.3x over it, driven by `LinearNoBias`'s + sequential fp32 accumulation over 10240 terms. **The second row is the one that + settles it: the ORACLE is itself 1.36x over kTol against an exact evaluation of + its own algorithm, because torch runs this in fp32 too.** No fp32 + implementation of this function meets a 1e-5 ABSOLUTE bound at hidden_size + 2560, and widening our accumulator cannot rescue one. W5 therefore does not + reuse kTol at model width; the file carries a real-width case with a relative + bound (`kRealWidthMixedRel`, 4e-5, derived as 6.6x the sqrt(K)*u random-walk + bound for K = 10240) that all three measurements sit inside. **What is still + owed** is agreement with the ORACLE at model width, which needs a real + checkpoint and cannot be closed in-suite: the in-suite case compares against + the double reference, because dumping one token of oracle IO at this width is + 26 MB of `.inc`. +- **The double accumulator is now gated, and the device arm inherits the + consequence.** `GroupedRmsNorm` accumulates the per-group sum of squares in + `double`, and at the suite's group sizes of 5 and 6 that convention had zero + discriminating power -- replacing it with `float` left the suite 280/280 green. + It is gated at the model's real group size of 2560, on magnitude-separated + data, where the two accumulators differ by 742x (3.168e-06 against 2.352e-03, + bound 1e-4). The convention is kept rather than dropped because it makes the + host reference more accurate than the oracle rather than less, which is what a + reference is for. What follows for the device arm, stated here so it is not + discovered: **a straight fp32-accumulate device reduction will not meet + `kRealWidthNormTol` on that data.** That is the correct signal, not a defect in + the gate -- it says the device kernel must accumulate wider than fp32 or be + gated against the oracle directly rather than against this reference. Deciding + which is the device wave's, and it is owed. - The **fused rank-1 write-back**. `GatedResidualWriteBackInPlace` is the seam and is already the primitive, but no device kernel replaces it yet. Both llama.cpp implementations of this architecture materialise the update as a diff --git a/scripts/gen-qwen4-exp-hc-goldens.py b/scripts/gen-qwen4-exp-hc-goldens.py index fae861e31..168fc2c4f 100755 --- a/scripts/gen-qwen4-exp-hc-goldens.py +++ b/scripts/gen-qwen4-exp-hc-goldens.py @@ -32,7 +32,11 @@ raw = open(SRC, "rb").read() got = hashlib.sha256(raw).hexdigest() -assert got == EXPECT_SHA, f"oracle source sha256 {got} != {EXPECT_SHA}" +# NOT an `assert`: `python3 -O` strips those, and this one guard is the entire +# mechanism tying the goldens to the pin. Under -O the stripped version would +# happily dump goldens from any file handed to it. +if got != EXPECT_SHA: + raise SystemExit(f"oracle source sha256 {got} != {EXPECT_SHA}") lines = raw.decode().splitlines(keepends=True) # 1-based, inclusive, as reported by grep -n on the pinned file. diff --git a/src/vllm/model_executor/models/qwen4_exp_hc.h b/src/vllm/model_executor/models/qwen4_exp_hc.h index 18683258e..43597b6fc 100644 --- a/src/vllm/model_executor/models/qwen4_exp_hc.h +++ b/src/vllm/model_executor/models/qwen4_exp_hc.h @@ -53,6 +53,45 @@ // * Apply it twice and the scale is ~2x. The published GGUF has the fold done // at CONVERT time, so a GGUF reader must NOT call this function. // +// ─── THE GGUF FOLD IS A BLANKET RULE, NOT AN `hc_norm` RULE ─────────────────── +// Read this before writing a loader. The sentence above is true and, stated on +// its own, it teaches the wrong lesson: a reader who skips the fold for +// `hc_norm` alone will DOUBLE-FOLD every other norm in the same file, which is +// the same silent ~2x defect moved one tensor to the left. +// +// The fold does not live in a qwen4exp code path at all. llama.cpp mainline has +// no `qwen4exp` whatsoever (read at `237ad9b96`: `git grep -il qwen4exp` returns +// nothing tree-wide, so mainline can neither convert nor load this +// architecture). The converter is ggml-org/llama.cpp PR #27742, head +// `035e22731a7fd70b9854b3a2d64ec68e9b1a45d3`, still OPEN. Its +// `conversion/qwen4exp.py` declares `class Qwen4ExpTextModel(_Qwen35MRopeMixin, +// _LinearAttentionVReorderBase)`; `_LinearAttentionVReorderBase` is +// `conversion/qwen.py:353`, a subclass of `Qwen3NextModel` (`:270`), and there +// is NO `hc_norm` branch in the PR's `modify_tensors` — `hc_norm.weight` falls +// through to `super()`. The `+1` is the INHERITED Qwen3-Next rule, present in +// mainline at `conversion/qwen.py:302-303`: +// +// elif name.endswith("norm.weight") and not name.endswith("linear_attn.norm.weight"): +// data_torch = data_torch + 1 +// +// So the rule a loader has to implement is: **every tensor whose name ends in +// `norm.weight` carries the fold, with `linear_attn.norm.weight` (the GDN +// `ssm_norm`) as the one exception.** `hc_norm`, `attn_q_norm` and `attn_k_norm` +// all match it. The PLE and indexer gammas — `.ple.norm_{key,query,conv}` and +// `.indexer.{q,k}_layernorm` — are folded too, by the PR's own branch, which +// returns early so they are never folded twice. +// +// This is a property of ONE IN-FLIGHT CONVERTER and not of "GGUF". #27742 is +// unmerged and can change before it lands, and a publisher using a different +// tool is under no obligation to match it. A loader must therefore treat the +// fold as a checkpoint-provenance question, not a format constant; the check is +// cheap, because an unfolded `hc_norm` is a ZERO-init gamma and a folded one is +// centred on 1.0. Corroborated on published artifacts during fresh review of +// #1988 (`unsloth/Qwen3.8-Flash-Next-GGUF` `UD-IQ1_S` and `UD-Q4_K_XL`, +// `vumpt/...-Q4_K_M`, read by HTTP range request against the bf16 HF tensors): +// every `*hc_norm.weight` is HF + 1.0 exactly, elementwise, while `ssm_norm` is +// not folded and sits in [0.875, 1.023]. +// // ─── PRECISION ──────────────────────────────────────────────────────────────── // fp32 interior throughout, with the per-group sum of squares accumulated in // double — the `deepseek_v4_mhc.cpp` house convention for a host reference. @@ -76,7 +115,11 @@ namespace vllm::qwen4_exp { // `w_vllm = 1.0 + w_hf`, elementwise. See the parameterization note above. Call // this exactly once, at load, on a HuggingFace `hc_norm.weight`. Never on a -// weight read out of the published GGUF, which has the fold applied already. +// weight read out of a GGUF written by ggml-org/llama.cpp#27742, which has the +// fold applied already — and note that the SAME is true of `attn_q_norm`, +// `attn_k_norm` and the PLE and indexer gammas in that file, while `ssm_norm` is +// the one exception. That rule is spelled out above; skipping the fold for +// `hc_norm` alone double-folds everything else. std::vector HcNormWeightFromHf(const std::vector& w_hf); // Grouped RMSNorm over one token's `x`, in vLLM's `RMSNormGated(group_size=)` diff --git a/tests/support/max_abs_diff.h b/tests/support/max_abs_diff.h index d81d995a3..b6a413435 100644 --- a/tests/support/max_abs_diff.h +++ b/tests/support/max_abs_diff.h @@ -28,6 +28,18 @@ // EXACT comparison, not through here. A future golden that genuinely needs a // non-finite value takes an explicit, documented carve-out at its own call site; // it never relaxes this helper. +// +// WHY THESE ARE TEMPLATES. A gate that compares an fp32 result against a +// DOUBLE-precision in-test reference — the shape every "second independent +// reference" case takes — cannot call a float-only helper, and the observed +// consequence is that it writes form B again locally. That happened on +// `test_qwen4_exp_hc.cpp` (#1988): two local `std::max(worst, std::abs(...))` +// helpers, in a file that already included this header, and a mutation that +// poisoned one path to all-NaN left its `< tol` assertion GREEN. The reduction +// is therefore templated on both operand types rather than duplicated, so the +// double-sided comparisons route through the SAME hardened scan. Mixed +// float/double operands are compared in double; the finiteness rule and its +// polarity are unchanged. #pragma once #include @@ -48,13 +60,16 @@ struct MaxAbsDiffScanResult { double worst = 0.0; // Index of the FIRST non-finite operand, or kNone when both sides are finite. size_t bad_index = kNone; - float bad_got = 0.0f; - float bad_want = 0.0f; + // Held as double so a double-sided comparison reports the value it actually + // saw; a float operand widens exactly. + double bad_got = 0.0; + double bad_want = 0.0; bool ok() const { return bad_index == kNone; } }; -inline MaxAbsDiffScanResult MaxAbsDiffScan(const float* got, const float* want, size_t count) { +template +inline MaxAbsDiffScanResult MaxAbsDiffScan(const G* got, const W* want, size_t count) { MaxAbsDiffScanResult r; double worst = 0.0; for (size_t i = 0; i < count; ++i) { @@ -77,7 +92,8 @@ inline MaxAbsDiffScanResult MaxAbsDiffScan(const float* got, const float* want, // The doctest-facing helper the model gates call. Fails loudly on a non-finite // operand and still returns +infinity, so a bound check either way reports red. -inline double MaxAbsDiff(const std::vector& got, const float* want, size_t count) { +template +inline double MaxAbsDiff(const std::vector& got, const W* want, size_t count) { REQUIRE(got.size() == count); const MaxAbsDiffScanResult r = MaxAbsDiffScan(got.data(), want, count); if (!r.ok()) { @@ -88,7 +104,8 @@ inline double MaxAbsDiff(const std::vector& got, const float* want, size_ return r.worst; } -inline double MaxAbsDiff(const std::vector& a, const std::vector& b) { +template +inline double MaxAbsDiff(const std::vector& a, const std::vector& b) { REQUIRE(a.size() == b.size()); return MaxAbsDiff(a, b.data(), b.size()); } diff --git a/tests/vllm/models/test_qwen4_exp_hc.cpp b/tests/vllm/models/test_qwen4_exp_hc.cpp index c9b40f149..0b07e1774 100644 --- a/tests/vllm/models/test_qwen4_exp_hc.cpp +++ b/tests/vllm/models/test_qwen4_exp_hc.cpp @@ -32,6 +32,8 @@ #include #include +#include +#include #include #include @@ -47,6 +49,16 @@ namespace { // The goldens are fp32 out of torch; this reference is fp32 with double-precision // reductions (the `deepseek_v4_mhc.cpp` house convention for a host reference). +// +// `kTol` IS A PROPERTY OF THESE SHAPES AND NOTHING ELSE. Say it here because the +// natural next reader is W5, wiring the module at hidden_size 2560, and reusing +// this number there is a red that is arithmetic noise. At the widths below — +// flat = 24 and 15 — the fp32 implementation is bit-identical or within one ulp +// of the oracle, measured against the pinned oracle itself: max|diff| over every +// golden array of case A, B and C is 2.384e-07, so kTol carries a 42x margin and +// is unconstrained. It does NOT survive a rescale, and the last two cases in this +// file are the measurement that says so rather than a warning that asks to be +// believed. See `kRealWidthNormTol` and `kRealWidthMixedTol`. constexpr double kTol = 1e-5; // The floor a Variant flip has to clear. It is not a round number picked to be @@ -158,24 +170,64 @@ RefOut ForwardRefD(const std::vector& hyper, const std::vector& w_ return o; } -// Compare an fp32 result against the double reference directly, for the cases -// where no oracle golden exists (a deliberately huge eps). -double MaxAbsDV(const std::vector& got, const std::vector& want) { - REQUIRE(got.size() == want.size()); - double worst = 0.0; - for (size_t i = 0; i < got.size(); ++i) { - worst = std::max(worst, std::abs(static_cast(got[i]) - want[i])); +// EVERY max|diff| in this file goes through `vllm_test::MaxAbsDiff`, including +// the double-sided ones. This file used to carry two local helpers spelled +// `worst = std::max(worst, std::abs(...))` for the fp32-vs-double and +// double-vs-golden comparisons, which is verbatim the NaN-blind "form B" that +// `support/max_abs_diff.h` exists to prevent (issue #449): `std::max(a, NaN)` +// returns `a`, so an all-NaN result reduces to 0.0 and PASSES every `< tol` +// bound. It was not hypothetical — poisoning `GroupedRmsNorm`'s large-eps path +// to all-NaN left the `< kTol` assertion below GREEN, and the suite only went +// red because a SIBLING assertion happened to use the hardened helper. The +// shared scan is now templated on both operand types so there is no reason to +// write the reduction again here, and no place left to write it wrong. + +// "Bit for bit" means the BIT PATTERN. `==` is VALUE equality, and the one +// divergence the MhcPost caveat below names — +0.0f against -0.0f — is exactly +// the case `==` cannot see. Returns the index of the first differing pattern, +// or `a.size()` when every pattern matches. +size_t FirstBitDiff(const std::vector& a, const std::vector& b) { + REQUIRE(a.size() == b.size()); + for (size_t i = 0; i < a.size(); ++i) { + uint32_t x = 0, y = 0; + std::memcpy(&x, &a[i], sizeof(x)); + std::memcpy(&y, &b[i], sizeof(y)); + if (x != y) return i; } - return worst; + return a.size(); } -double MaxAbsD(const std::vector& got, const float* want) { - double worst = 0.0; - for (size_t i = 0; i < got.size(); ++i) { - worst = std::max(worst, std::abs(got[i] - static_cast(want[i]))); +// ─── Real-width scaffolding ─────────────────────────────────────────────────── +// `Qwen/Qwen3.8-Flash-Next`'s own values (spec `## Architecture`): hc_count = 4, +// hidden_size = 2560, hc_lowrank = 320, so the residual is 10240 wide and the +// grouped norm reduces over 2560 elements. No oracle golden exists at this width +// and none should: dumping the IO of one token is 26 MB of `.inc`. What the two +// cases at the end of this file gate is the arithmetic's behaviour at SCALE, +// against the same independent double reference the sweep uses. +constexpr int64_t kModelHc = 4; +constexpr int64_t kModelHidden = 2560; +constexpr int64_t kModelRank = 320; + +// A deterministic generator written out in full rather than `std::mt19937` plus +// a distribution: `std::uniform_real_distribution` is NOT specified to produce +// the same sequence across standard libraries, and a gate whose INPUT depends on +// libstdc++ versus libc++ is not a reproducible measurement. +struct Lcg { + uint64_t s; + explicit Lcg(uint64_t seed) : s(seed) {} + // Uniform in [-1, 1), from the top 24 bits, which are the good ones. + float Next() { + s = s * 6364136223846793005ULL + 1442695040888963407ULL; + return static_cast(static_cast(s >> 40)) / 8388608.0f - 1.0f; } - return worst; -} + // Approximately unit-variance normal, four uniforms deep (Irwin-Hall). It + // matters that this is not uniform: the goldens are generated from + // `torch.randn * k`, the errors below are compared against a measurement made + // on THAT data, and a bounded uniform has visibly lighter tails, which shows + // up as a smaller dot-product error over 10240 terms. Var(U[-1,1)) = 1/3, so + // four of them scaled by sqrt(3)/2 is unit variance. + float Normal() { return 0.8660254f * (Next() + Next() + Next() + Next()); } +}; } // namespace @@ -200,7 +252,17 @@ TEST_CASE("qwen4_exp grouped RMSNorm mirrors RMSNormGated(group_size) at the lan // makes. It must be visibly different, or the case above proves nothing. Variant global; global.norm_is_grouped = false; - CHECK(MaxAbsD(NormRefD(hyper, w_hf, hc, hidden, 1e-6, global), kA_normed) > 1e-2); + CHECK(MaxAbsDiff(NormRefD(hyper, w_hf, hc, hidden, 1e-6, global), kA_normed, + hyper.size()) > 1e-2); + + // Same for the missing `1 + w` fold, through the SWEEP's reference rather than + // only through the direct call in the next case. `Variant` advertises a flag + // per known defect and this one was declared, read, and never once set false, + // so the struct claimed coverage the file did not have. + Variant raw_gamma; + raw_gamma.norm_weight_plus_one = false; + CHECK(MaxAbsDiff(NormRefD(hyper, w_hf, hc, hidden, 1e-6, raw_gamma), kA_normed, + hyper.size()) > 1e-1); // eps lives INSIDE the rsqrt, added to the mean square. At the model's real // 1e-6 the two placements differ by ~1e-6, which is UNDER this suite's fp32 @@ -211,7 +273,7 @@ TEST_CASE("qwen4_exp grouped RMSNorm mirrors RMSNormGated(group_size) at the lan // merely observing that eps does something. const std::vector big_eps = GroupedRmsNorm(hyper, w, hidden, 4.0f); CHECK(MaxAbsDiff(big_eps, got) > 1e-2); - CHECK(MaxAbsDV(big_eps, NormRefD(hyper, w_hf, hc, hidden, 4.0, Variant{})) < kTol); + CHECK(MaxAbsDiff(big_eps, NormRefD(hyper, w_hf, hc, hidden, 4.0, Variant{})) < kTol); } TEST_CASE("qwen4_exp hc_norm weight is 1 + w_hf, applied exactly once") { @@ -332,6 +394,97 @@ TEST_CASE("qwen4_exp GatedResidual refuses a stream that is not hc_count * hidde std::invalid_argument); } +// Every OTHER refusal this module makes. Split out from the two cases above +// because those two were, until this change, the ONLY refusals under any +// assertion: a mutation sweep deleted eight of the eleven guards one at a time +// and the suite stayed 280/280 green each time, including all three `RequireSize` +// calls in `GatedResidualWriteBack`. +// +// Three of those eight are not hygiene. `GatedResidualWriteBackInPlace` takes RAW +// POINTERS and the header declares it the fusion seam that a device kernel +// replaces, so the wrapper's three size checks are the only bounds check between +// a caller and an out-of-bounds read of `block_out[h]` or `injection_weights[j]`. +// What that class of guard prevents is not a tidy error: deleting the +// divisibility guard in the same sweep produced `Fatal glibc error: +// malloc.c:2599` and `Aborted (core dumped)`, with no doctest summary printed at +// all. +// +// The MESSAGE is pinned, never the type. Every guard in this translation unit +// throws `std::invalid_argument`, so `CHECK_THROWS_AS` cannot tell a deleted +// guard from a different one firing further down — which is the defect the +// hyper_input case above already documents finding. +TEST_CASE("qwen4_exp refuses every malformed shape BY NAME, not merely by type") { + const std::vector ones10(10, 1.0f); + + SUBCASE("grouped norm: a non-positive group size") { + // Deleting this guard is not a wrong answer, it is `x.size() % 0` and the + // process dies on SIGFPE. Re-run and confirmed: with the guard removed this + // case takes the whole binary down mid-run. + CHECK_THROWS_WITH_AS(GroupedRmsNorm(ones10, ones10, 0, 1e-6f), + "qwen4_exp: group_size must be positive, got 0.", std::invalid_argument); + CHECK_THROWS_WITH_AS(GroupedRmsNorm(ones10, ones10, -3, 1e-6f), + "qwen4_exp: group_size must be positive, got -3.", std::invalid_argument); + } + + SUBCASE("gated residual: hc_count <= 1 and hidden_size <= 0") { + // `configuration_qwen4_exp.py:196-197` raises `Qwen4-Exp requires hc_count > + // 1`; this is that refusal, and until now nothing held it. + GatedResidualWeights w; + w.hc_norm_weight.assign(24, 1.0f); + w.mix_down.assign(5 * 24, 0.0f); + w.mix_up.assign(24 * 5, 0.0f); + CHECK_THROWS_WITH_AS(GatedResidualForward(std::vector(6, 1.0f), w, 1, 6, 1e-6f), + "qwen4_exp: hc_count must be > 1 and hidden_size > 0, got 1 and 6.", + std::invalid_argument); + CHECK_THROWS_WITH_AS(GatedResidualForward(std::vector(24, 1.0f), w, 4, 0, 1e-6f), + "qwen4_exp: hc_count must be > 1 and hidden_size > 0, got 4 and 0.", + std::invalid_argument); + } + + SUBCASE("gated residual: the low-rank weights") { + // `rank` is INFERRED from `mix_down.size() / flat`, so a ragged `mix_down` is + // not a bad input, it is a bad rank silently propagated into two more GEMMs. + const std::vector hyper(24, 1.0f); + GatedResidualWeights w; + w.hc_norm_weight.assign(24, 1.0f); + w.mix_up.assign(24 * 5, 0.0f); + + w.mix_down.clear(); + CHECK_THROWS_WITH_AS(GatedResidualForward(hyper, w, 4, 6, 1e-6f), + "qwen4_exp: input_mix_weight_down is not a multiple of 24 (got 0).", + std::invalid_argument); + w.mix_down.assign(23, 0.0f); + CHECK_THROWS_WITH_AS(GatedResidualForward(hyper, w, 4, 6, 1e-6f), + "qwen4_exp: input_mix_weight_down is not a multiple of 24 (got 23).", + std::invalid_argument); + + w.mix_down.assign(5 * 24, 0.0f); // rank 5, so mix_up must be 120 + w.mix_up.assign(119, 0.0f); + CHECK_THROWS_WITH_AS(GatedResidualForward(hyper, w, 4, 6, 1e-6f), + "qwen4_exp: input_mix_weight_up has 119 elements, expected 120.", + std::invalid_argument); + + w.mix_up.assign(120, 0.0f); + w.block_inject.assign(95, 0.0f); // must be hc * flat = 96 + CHECK_THROWS_WITH_AS(GatedResidualForward(hyper, w, 4, 6, 1e-6f), + "qwen4_exp: block_inject_weight has 95 elements, expected 96.", + std::invalid_argument); + } + + SUBCASE("write-back: the fusion seam's only bounds check") { + const std::vector hyper24(24, 1.0f), block6(6, 1.0f), inj4(4, 1.0f); + CHECK_THROWS_WITH_AS( + GatedResidualWriteBack(std::vector(23, 1.0f), block6, inj4, 4, 6), + "qwen4_exp: hyper_input has 23 elements, expected 24.", std::invalid_argument); + CHECK_THROWS_WITH_AS( + GatedResidualWriteBack(hyper24, std::vector(5, 1.0f), inj4, 4, 6), + "qwen4_exp: block output has 5 elements, expected 6.", std::invalid_argument); + CHECK_THROWS_WITH_AS( + GatedResidualWriteBack(hyper24, block6, std::vector(3, 1.0f), 4, 6), + "qwen4_exp: injection_weights has 3 elements, expected 4.", std::invalid_argument); + } +} + // ── 3. The rank-1 write-back ────────────────────────────────────────────────── TEST_CASE("qwen4_exp write-back is hyper_input RAW plus outer(block_out, injection)") { const int64_t hc = 4, hidden = 6, flat = hc * hidden; @@ -348,14 +501,33 @@ TEST_CASE("qwen4_exp write-back is hyper_input RAW plus outer(block_out, injecti CHECK(MaxAbsDiff(GatedResidualWriteBack(normed, block, inj, hc, hidden), kA_written + t * flat, static_cast(flat)) > 1e-2); - // The in-place form is the fusion seam: it must agree bit-for-bit with the - // allocating one, so a device arm can replace one with the other. + // The in-place form is THE FUSION SEAM (`qwen4_exp_hc.h`), so it is gated + // against the ORACLE directly and not against its own wrapper. Checking it + // against `got` alone is a tautology: `GatedResidualWriteBack` copies and + // then calls this very function, so the comparison holds under any mutation + // of it. Proven — mutating the in-place body (`+=` to `=`, and + // `injection_weights[j]` to `[0]`) reddened three other assertions and never + // this one. The golden compare below is the assertion that actually holds + // the primitive; the bit check that follows holds the WRAPPER's claim to be + // free of its own arithmetic. std::vector inplace = hyper; GatedResidualWriteBackInPlace(inplace.data(), block.data(), inj.data(), hc, hidden); - for (size_t i = 0; i < inplace.size(); ++i) CHECK(inplace[i] == got[i]); + CHECK(MaxAbsDiff(inplace, kA_written + t * flat, static_cast(flat)) < kTol); + CHECK(FirstBitDiff(inplace, got) == inplace.size()); } } +// The bit helper's own polarity, pinned. `==` reports these two equal and the +// pattern comparison does not, which is the entire reason the two "bit for bit" +// claims in this file stopped using `==`. +TEST_CASE("qwen4_exp bit-for-bit means the pattern: +0.0f and -0.0f are == but not equal") { + const std::vector pos{0.0f, 1.5f}; + const std::vector neg{-0.0f, 1.5f}; + CHECK(pos[0] == neg[0]); + CHECK(FirstBitDiff(pos, neg) == 0); + CHECK(FirstBitDiff(pos, pos) == pos.size()); +} + // The spec asserts our DeepSeek-V4 `MhcPost` is a bit-exact bring-up bridge for // this write-back once its comb matrix is the identity. Verified rather than // trusted, because it is the reviewer's free mutation target. It holds on finite @@ -375,7 +547,10 @@ TEST_CASE("qwen4_exp write-back equals MhcPost with an identity comb, bit for bi const std::vector bridge = vllm::deepseek_v4::MhcPost(block, hyper, inj, comb, hc, hidden); REQUIRE(bridge.size() == ours.size()); - for (size_t i = 0; i < ours.size(); ++i) CHECK(bridge[i] == ours[i]); + // The PATTERN, not `==`. The negative-zero caveat in the comment above is + // precisely the divergence value equality cannot see, so asserting the claim + // with `==` asserted something weaker than the claim. + CHECK(FirstBitDiff(bridge, ours) == ours.size()); } } @@ -390,8 +565,8 @@ TEST_CASE("qwen4_exp goldens separate every known gated-residual trap") { Variant{}); // The independent double reference reproduces the oracle. Two references // agreeing is what makes a single-flag flip attributable to the flag. - CHECK(MaxAbsD(ok.mixed, kA_mixed + t * hidden) < kTol); - CHECK(MaxAbsD(ok.injection, kA_inj_w + t * hc) < kTol); + CHECK(MaxAbsDiff(ok.mixed, kA_mixed + t * hidden, static_cast(hidden)) < kTol); + CHECK(MaxAbsDiff(ok.injection, kA_inj_w + t * hc, static_cast(hc)) < kTol); struct Case { Variant v; @@ -414,8 +589,179 @@ TEST_CASE("qwen4_exp goldens separate every known gated-residual trap") { for (const Case& k : cases) { const RefOut bad = ForwardRefD(hyper, w_hf, kA_down, kA_up, kA_inject, hc, hidden, rank, 1e-6, k.v); - if (k.hits_mixed) CHECK(MaxAbsD(bad.mixed, kA_mixed + t * hidden) > kFlipFloor); - if (k.hits_injection) CHECK(MaxAbsD(bad.injection, kA_inj_w + t * hc) > kFlipFloor); + if (k.hits_mixed) { + CHECK(MaxAbsDiff(bad.mixed, kA_mixed + t * hidden, static_cast(hidden)) > + kFlipFloor); + } + if (k.hits_injection) { + CHECK(MaxAbsDiff(bad.injection, kA_inj_w + t * hc, static_cast(hc)) > kFlipFloor); + } + } + } +} + +// ── 5. Real width: the two things the toy shapes cannot measure ─────────────── +// +// Everything above runs at hidden_size 6 or 5 and hc_count 4 or 3. Two of this +// module's documented properties are invisible at those widths, and one of them +// is a tolerance the next wave will otherwise reuse. + +// THE DOUBLE ACCUMULATOR. `qwen4_exp_hc.h` "PRECISION" states the per-group sum +// of squares is accumulated in `double`. At group sizes 5 and 6 that claim has +// no discriminating power whatsoever: replacing `double ss` with `float ss` +// moves nothing any tolerance in this file can see, so it was a documented +// convention with zero coverage. At the model's real group size it is worth +// three orders of magnitude. +// +// The input is magnitude-separated on purpose — one element at 4096, the rest at +// 1.0 — and that is not cherry-picking, it is the only shape that CAN separate +// the two accumulators. A sum of 2560 POSITIVE squares of similar size loses +// about sqrt(N)*u either way and the two agree to within 3x. The loss appears +// when a partial sum grows past the point where the next term falls under its +// own ulp: 4096^2 is exactly 2^24, the float integer ceiling, so every one of the +// 2559 following `+1.0` terms rounds away and `float ss` misses the tail +// ENTIRELY, by 2559 out of 16779775. +// +// Measured on exactly the data below, and byte-identical at -O0 and -O2, so it +// does not rest on vectorisation. It cannot rest on FMA contraction either: this +// tree compiles with `-ffp-contract=off` everywhere. +// +// max|reference| 5.20e+01 +// ours, double accumulator 3.168430e-06 (6.1e-08 relative: one ulp) +// the same code with `float ss` 2.352230e-03 (742x worse) +// +// The bound sits 31.6x above the first and 23.5x below the second, which is a +// band rather than a fitted number. Both ends are re-run mutations, not +// estimates: `double ss` -> `float ss` in `GroupedRmsNorm` reddens this case and +// nothing else in the file. +constexpr double kRealWidthNormTol = 1e-4; + +TEST_CASE("qwen4_exp GroupedRmsNorm needs its double accumulator at group_size 2560") { + const int64_t flat = kModelHc * kModelHidden; + std::vector x(static_cast(flat)), w(static_cast(flat)); + const float dominant[4] = {4096.0f, 2048.0f, 8192.0f, 1024.0f}; + Lcg rng(12345); + for (int64_t j = 0; j < kModelHc; ++j) { + for (int64_t d = 0; d < kModelHidden; ++d) { + x[j * kModelHidden + d] = (d == 0) ? dominant[j] : 1.0f; + w[j * kModelHidden + d] = 1.0f + 0.5f * rng.Next(); + } + } + + const std::vector got = GroupedRmsNorm(x, w, kModelHidden, 1e-6f); + + // The reference, in full double, written out here rather than reusing + // `NormRefD`: that one takes the HF gamma and applies the `1 + w` fold, and + // this case is about the ACCUMULATOR, so the weight goes in already folded. + std::vector want(static_cast(flat)); + for (int64_t j = 0; j < kModelHc; ++j) { + double ss = 0.0; + for (int64_t d = 0; d < kModelHidden; ++d) { + const double v = x[j * kModelHidden + d]; + ss += v * v; + } + const double r = 1.0 / std::sqrt(ss / static_cast(kModelHidden) + 1e-6); + for (int64_t d = 0; d < kModelHidden; ++d) { + want[j * kModelHidden + d] = static_cast(x[j * kModelHidden + d]) * r * + static_cast(w[j * kModelHidden + d]); + } + } + CHECK(MaxAbsDiff(got, want) < kRealWidthNormTol); + + // And the separation is real rather than asserted: the same reduction with a + // float accumulator, over the same data, is on the far side of the bound. If + // this stops holding, the case above has stopped measuring anything. + float ss_f = 0.0f; + double ss_d = 0.0; + for (int64_t d = 0; d < kModelHidden; ++d) { + const double v = x[d]; + ss_f += static_cast(v * v); + ss_d += v * v; + } + const double r_f = 1.0 / std::sqrt(static_cast(ss_f) / kModelHidden + 1e-6); + const double r_d = 1.0 / std::sqrt(ss_d / kModelHidden + 1e-6); + CHECK(std::abs(r_f - r_d) * dominant[0] * w[0] > 10.0 * kRealWidthNormTol); +} + +// WHAT THE fp32 INTERIOR COSTS, AND WHY `kTol` DOES NOT TRAVEL. At flat = 24 the +// implementation is bit-identical to the oracle, so kTol measures nothing there. +// At flat = 10240 it is not, and the reason is not ours alone. Measured against +// the PINNED ORACLE itself (transformers v5.16.0, the same lift-and-exec path +// `scripts/gen-qwen4-exp-hc-goldens.py` uses, at hidden=2560 hc=4 lowrank=320 +// eps=1e-6, two tokens), max|diff| on `mixed_input`: +// +// t=0 t=1 +// ours (fp32) vs oracle 2.325e-05 2.137e-05 +// double ref vs oracle 1.360e-05 5.431e-06 +// ours (fp32) vs double 3.684e-05 1.606e-05 +// +// Two conclusions, and the second is the one that matters. Our fp32 interior is +// 2.1x to 2.3x over kTol at model width, driven by `LinearNoBias`'s sequential +// fp32 accumulation over 10240 terms. And the ORACLE is 1.36x over kTol against +// an exact evaluation of its own algorithm, because torch runs this in fp32 too +// — so widening OUR accumulator cannot rescue a 1e-5 absolute bound here. No +// fp32 implementation of this function meets kTol at hidden_size 2560, and the +// tolerance is the thing that is wrong at that width, not the arithmetic. +// +// The bound below is therefore RELATIVE to the reference's own magnitude, and +// the case compares against the double reference because committing an oracle +// golden at this width is 26 MB of `.inc`. It is derived rather than fitted: a +// sequential fp32 dot of length K accumulates a relative error that grows as a +// random walk, about sqrt(K)*u, with u = 2^-24 = 5.96e-08 the fp32 unit +// roundoff. At K = 10240 that is 6.03e-06, and 4e-05 is 6.6x it. The three +// measurements that exist all sit inside that: 6.79e-06 relative on this case's +// own data, and 7.32e-06 / 1.68e-05 on the two oracle tokens tabulated above. +// +// What this gates is that the fp32 interior stays within a small multiple of the +// random-walk bound for its accumulation length. What it does NOT gate is +// agreement with the ORACLE at this width — that needs a real checkpoint, and it +// is recorded under the spec's `## Owed`. +constexpr double kRealWidthMixedRel = 4e-5; + +TEST_CASE("qwen4_exp gated residual at the model's real width, hidden_size 2560") { + const int64_t flat = kModelHc * kModelHidden; + Lcg rng(20260826); + std::vector w_hf(static_cast(flat)), hyper(static_cast(flat)); + std::vector down(static_cast(kModelRank * flat)); + std::vector up(static_cast(flat * kModelRank)); + std::vector inject(static_cast(kModelHc * flat)); + // The same standard deviations `scripts/gen-qwen4-exp-hc-goldens.py` uses, so + // the numbers this case measures are comparable with the oracle table above. + for (float& v : w_hf) v = 0.5f * rng.Normal(); + for (float& v : hyper) v = 1.7f * rng.Normal(); + for (float& v : down) v = 0.3f * rng.Normal(); + for (float& v : up) v = 0.3f * rng.Normal(); + for (float& v : inject) v = 0.3f * rng.Normal(); + + GatedResidualWeights w; + w.hc_norm_weight = HcNormWeightFromHf(w_hf); + w.mix_down = down; + w.mix_up = up; + w.block_inject = inject; + + const GatedResidualResult got = GatedResidualForward(hyper, w, kModelHc, kModelHidden, 1e-6f); + const RefOut ref = ForwardRefD(hyper, w_hf, down.data(), up.data(), inject.data(), kModelHc, + kModelHidden, kModelRank, 1e-6, Variant{}); + + double scale = 0.0; + for (double v : ref.mixed) scale = std::max(scale, std::abs(v)); + REQUIRE(scale > 0.1); // the data is not degenerate, so the ratio means something + + CHECK(MaxAbsDiff(got.hyper_input_normed, ref.normed) < kRealWidthMixedRel * scale); + CHECK(MaxAbsDiff(got.mixed_input, ref.mixed) < kRealWidthMixedRel * scale); + CHECK(MaxAbsDiff(got.injection_weights, ref.injection) < kRealWidthMixedRel * 2.0); + + // The write-back at real width too: it is the seam, and 10240 is the width the + // fused kernel will see. + std::vector block(static_cast(kModelHidden)); + for (float& v : block) v = 0.9f * rng.Normal(); + const std::vector written = + GatedResidualWriteBack(hyper, block, got.injection_weights, kModelHc, kModelHidden); + REQUIRE(written.size() == static_cast(flat)); + for (int64_t j = 0; j < kModelHc; ++j) { + for (int64_t h = 0; h < kModelHidden; h += 512) { + const size_t i = static_cast(j * kModelHidden + h); + CHECK(written[i] == doctest::Approx(hyper[i] + block[h] * got.injection_weights[j])); } } } From 23483dbcc4ed246b569839abec958653367ee6f1 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 22:27:23 +0000 Subject: [PATCH 3/3] =?UTF-8?q?fix(MODEL-MM-QWEN4-EXP):=20W3=20=E2=80=94?= =?UTF-8?q?=20the=20GGUF=20fold=20guidance=20now=20cites=20the=20pinned=20?= =?UTF-8?q?llama.cpp,=20not=20a=20repudiated=20local=20commit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CORRECTION TO 09ce43f76. That commit, its header passage and the spec all attributed the Qwen3-Next `+1` fold rule to llama.cpp "mainline at `conversion/qwen.py:302-303`", read at `237ad9b96`. `237ad9b96` is not mainline. It is reachable only from the local branch `localai-paged` in the developer's llama.cpp checkout, it has no merge base with `ggml-org/llama.cpp` `origin/master`, and `.agents/oracles/llama-cpp.md` already records it as the retired, contaminated pin: 65 of this project's own performance commits past upstream tag `b9827`, built from a working tree carrying 27 uncommitted entries. The live pin is stock upstream tag `b10451`, `10bf611e533d81f739128304991c5e133c6aebd8`. The three line anchors were wrong at every ggml-org revision as well, so a W5 loader author following that passage would have landed on unrelated code in a tree they cannot fetch. Everything the passage asserts remains TRUE, and it is re-anchored rather than rewritten. Read from `git archive b10451` (the working checkout is dirty, so the tag object is the only honest source): `git grep -il qwen4exp` still returns nothing tree-wide, so a released llama.cpp can neither convert nor load this architecture. The `+1` rule and its `linear_attn.norm.weight` exception are at `conversion/qwen.py:387-388`; `_LinearAttentionVReorderBase` is `:438`; `Qwen3NextModel` is `:365`, and its signature is now `class Qwen3NextModel(_QwenMtpMixin, Qwen2MoeModel)`, which the old quotation had stale. PR ggml-org/llama.cpp#27742 is still OPEN and unmerged at head `035e22731a7fd70b9854b3a2d64ec68e9b1a45d3` (`refs/pull/27742/head` and a live `/merge` ref; the head is not an ancestor of `origin/master`), and its `modify_tensors` still carries no `hc_norm` branch, so `hc_norm.weight` falls through to `super()` and the fold covers `attn_q_norm`, `attn_k_norm` and the PLE/indexer gammas while sparing `ssm_norm`. That is the whole point of the passage: skip the fold for `hc_norm` alone and every other norm is double-folded. The word "mainline" is gone, and `237ad9b96` is cited nowhere in it. Three smaller repairs ride along. The claim that "the ORACLE is itself 1.36x over kTol" was one draw of random inputs stated as a general fact; an independent draw gives 0.91x and 0.82x. Both the test comment and the spec now say the oracle sits of the same ORDER as kTol, name both draws, and label the measurement table as one draw. The conclusion is unchanged and holds on either draw: no fp32 implementation of this function meets a 1e-5 ABSOLUTE bound at hidden_size 2560, because torch runs the reduction in fp32 too, so widening our own accumulator cannot rescue it. `scripts/gen-qwen4-exp-hc-goldens.py` still guarded "upstream must return hyper_input RAW" with a bare `assert`, one line away from the sha256 pin guard fixed for exactly this reason. `python3 -O` strips it. Proven rather than argued: with the oracle corrupted to return `hyper_input * 2` and the sha guard retargeted in a scratch copy so it could not fire first, the old script under `python3 -O` exited 0 and wrote a golden file that differs from the committed `.inc`. It is a `raise SystemExit` now, and the same experiment exits 1 with no output file under both `python3` and `python3 -O`. `tests/support/test_max_abs_diff.cpp` exercised only `float`/`float`, so the double-sided instantiation of the helper was covered only by its one caller. Two cases now compare a `std::vector` reference against an fp32 golden through both the scan and both wrapper overloads, using a difference of 2^-30 that exists in double and vanishes in float. Narrowing the operands to float inside the scan reddens those cases and nothing else; the pre-widening header refuses to compile them at all. `qwen4_exp_hc.cpp` is untouched, the goldens reproduce byte-for-byte from the pinned oracle under both `python3` and `python3 -O`, and `test_qwen4_exp_hc` is unmoved at 14 cases / 230 assertions. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/qwen4-exp-flash-next.md | 26 ++++++---- scripts/gen-qwen4-exp-hc-goldens.py | 6 ++- src/vllm/model_executor/models/qwen4_exp_hc.h | 17 ++++--- tests/support/test_max_abs_diff.cpp | 48 +++++++++++++++++++ tests/vllm/models/test_qwen4_exp_hc.cpp | 12 +++-- 5 files changed, 87 insertions(+), 22 deletions(-) diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index 1d204336a..0a34d0196 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -423,16 +423,19 @@ as a checkpoint bug rather than a port bug. **The GGUF converter already folds it, and it folds far more than `hc_norm`.** Read at source rather than relayed, because W5 writes the loader and the narrow version of this -sentence causes the defect it warns about. llama.cpp mainline has no `qwen4exp` at all -(`git grep -il qwen4exp` at `237ad9b96`: nothing tree-wide, so mainline can neither -convert nor load this architecture). The converter is ggml-org/llama.cpp +sentence causes the defect it warns about. Every anchor below is read at our recorded +llama.cpp pin, stock upstream tag `b10451` (`10bf611e533d81f739128304991c5e133c6aebd8`, +[`../oracles/llama-cpp.md`](../oracles/llama-cpp.md)). Stock upstream has no `qwen4exp` +at all there (`git grep -il qwen4exp`: nothing tree-wide, so a released llama.cpp can +neither convert nor load this architecture). The converter is ggml-org/llama.cpp [#27742](https://github.com/ggml-org/llama.cpp/pull/27742), head `035e22731a7fd70b9854b3a2d64ec68e9b1a45d3`, **still OPEN**. Its `conversion/qwen4exp.py` declares `class Qwen4ExpTextModel(_Qwen35MRopeMixin, _LinearAttentionVReorderBase)`; -`_LinearAttentionVReorderBase` is `conversion/qwen.py:353`, a subclass of -`Qwen3NextModel` (`:270`); and the PR's `modify_tensors` has **no `hc_norm` branch**, so -`hc_norm.weight` falls through to `super()`. The `+1` is the inherited Qwen3-Next rule at -`conversion/qwen.py:302-303`: +`_LinearAttentionVReorderBase` is `conversion/qwen.py:438`, a subclass of +`Qwen3NextModel` (`:365`, whose own signature is +`class Qwen3NextModel(_QwenMtpMixin, Qwen2MoeModel)`); and the PR's `modify_tensors` has +**no `hc_norm` branch**, so `hc_norm.weight` falls through to `super()`. The `+1` is the +inherited Qwen3-Next rule at `conversion/qwen.py:387-388`: ```python elif name.endswith("norm.weight") and not name.endswith("linear_attn.norm.weight"): @@ -805,7 +808,8 @@ change that makes any arm reachable, not later. draft of the bullet above framed the tolerance question as the DEVICE arm's problem, and it is not. Measured against the pinned oracle itself, at the model's own shape (hidden_size 2560, hc_count 4, hc_lowrank 320, eps 1e-6, two - tokens), max|diff| on `mixed_input`: + tokens), max|diff| on `mixed_input`. This is ONE draw of random inputs, and the + ratios below move from draw to draw; the ordering and the conclusion do not. | | t=0 | t=1 | |---|---|---| @@ -818,8 +822,10 @@ change that makes any arm reachable, not later. 2.384e-07 -- so kTol carries a 42x margin there and constrains nothing. At model width our fp32 interior is 2.1x to 2.3x over it, driven by `LinearNoBias`'s sequential fp32 accumulation over 10240 terms. **The second row is the one that - settles it: the ORACLE is itself 1.36x over kTol against an exact evaluation of - its own algorithm, because torch runs this in fp32 too.** No fp32 + settles it: the ORACLE is itself of the same ORDER as kTol against an exact + evaluation of its own algorithm -- 1.36x on the draw above, 0.91x and 0.82x on + an independent draw taken during fresh review -- because torch runs this in + fp32 too.** No fp32 implementation of this function meets a 1e-5 ABSOLUTE bound at hidden_size 2560, and widening our accumulator cannot rescue one. W5 therefore does not reuse kTol at model width; the file carries a real-width case with a relative diff --git a/scripts/gen-qwen4-exp-hc-goldens.py b/scripts/gen-qwen4-exp-hc-goldens.py index 168fc2c4f..8238488ac 100755 --- a/scripts/gen-qwen4-exp-hc-goldens.py +++ b/scripts/gen-qwen4-exp-hc-goldens.py @@ -94,7 +94,11 @@ def case(tag, hidden, hc, lowrank, eps, tokens, use_combine, seed): res = mod(hyper) if use_combine: mixed, hyper_ret, inj = res - assert hyper_ret is hyper, "upstream must return hyper_input RAW" + # NOT an `assert`, for the same reason as the sha guard above: this + # is the only check that upstream still hands back the RAW input for + # the write-back, and `python3 -O` strips an `assert`. + if hyper_ret is not hyper: + raise SystemExit("upstream must return hyper_input RAW") injection = block_out.unsqueeze(-2) * inj.unsqueeze(-1) written = hyper + injection.flatten(-2) else: diff --git a/src/vllm/model_executor/models/qwen4_exp_hc.h b/src/vllm/model_executor/models/qwen4_exp_hc.h index 43597b6fc..e70417407 100644 --- a/src/vllm/model_executor/models/qwen4_exp_hc.h +++ b/src/vllm/model_executor/models/qwen4_exp_hc.h @@ -59,17 +59,20 @@ // `hc_norm` alone will DOUBLE-FOLD every other norm in the same file, which is // the same silent ~2x defect moved one tensor to the left. // -// The fold does not live in a qwen4exp code path at all. llama.cpp mainline has -// no `qwen4exp` whatsoever (read at `237ad9b96`: `git grep -il qwen4exp` returns -// nothing tree-wide, so mainline can neither convert nor load this -// architecture). The converter is ggml-org/llama.cpp PR #27742, head +// The fold does not live in a qwen4exp code path at all. Every anchor below is +// read at our recorded llama.cpp pin, stock upstream tag `b10451` +// (`10bf611e533d81f739128304991c5e133c6aebd8`, `.agents/oracles/llama-cpp.md`). +// Stock upstream has no `qwen4exp` whatsoever there: `git grep -il qwen4exp` +// returns nothing tree-wide, so a released llama.cpp can neither convert nor +// load this architecture. The converter is ggml-org/llama.cpp PR #27742, head // `035e22731a7fd70b9854b3a2d64ec68e9b1a45d3`, still OPEN. Its // `conversion/qwen4exp.py` declares `class Qwen4ExpTextModel(_Qwen35MRopeMixin, // _LinearAttentionVReorderBase)`; `_LinearAttentionVReorderBase` is -// `conversion/qwen.py:353`, a subclass of `Qwen3NextModel` (`:270`), and there +// `conversion/qwen.py:438`, a subclass of `Qwen3NextModel` (`:365`, whose own +// signature is `class Qwen3NextModel(_QwenMtpMixin, Qwen2MoeModel)`), and there // is NO `hc_norm` branch in the PR's `modify_tensors` — `hc_norm.weight` falls -// through to `super()`. The `+1` is the INHERITED Qwen3-Next rule, present in -// mainline at `conversion/qwen.py:302-303`: +// through to `super()`. The `+1` is the INHERITED Qwen3-Next rule, at +// `conversion/qwen.py:387-388`: // // elif name.endswith("norm.weight") and not name.endswith("linear_attn.norm.weight"): // data_torch = data_torch + 1 diff --git a/tests/support/test_max_abs_diff.cpp b/tests/support/test_max_abs_diff.cpp index cbc815f14..6cccd3e2d 100644 --- a/tests/support/test_max_abs_diff.cpp +++ b/tests/support/test_max_abs_diff.cpp @@ -151,6 +151,54 @@ TEST_CASE("max|diff| scan: -FLT_MAX is FINITE and must still compare") { CHECK(moved.worst > 1e38); } +// The DOUBLE-SIDED instantiation, which is the shape the helper was templated +// for (#1988): an fp32 golden against a `std::vector` in-test reference. +// Before this case the widened templates were exercised only by +// `test_qwen4_exp_hc`, so the helper's own suite could not see a regression in +// them. Two properties are gated here: the comparison runs in DOUBLE rather than +// narrowing the reference to float, and the finiteness rule holds on the double +// side exactly as it does on the float side. +TEST_CASE("max|diff| scan: a vector reference against an fp32 golden") { + const std::vector exact(kWant, kWant + kN); + const vllm_test::MaxAbsDiffScanResult same = vllm_test::MaxAbsDiffScan(exact.data(), kWant, kN); + CHECK(same.ok()); + CHECK(same.worst == 0.0); + + // A difference that EXISTS in double and vanishes in float. 1.0 + 2^-30 is not + // representable in fp32 (whose spacing at 1.0 is 2^-23), so an implementation + // that narrowed the double operand would report 0.0 here and pass every bound. + const float one_golden[] = {1.0f, 1.0f, 1.0f, 1.0f}; + std::vector tiny(kN, 1.0); + tiny[2] = 1.0 + std::ldexp(1.0, -30); + const vllm_test::MaxAbsDiffScanResult moved = + vllm_test::MaxAbsDiffScan(tiny.data(), one_golden, kN); + CHECK(moved.ok()); + CHECK(moved.worst == std::ldexp(1.0, -30)); + + // The finiteness rule, on the DOUBLE side. A double NaN is not a zero either, + // and the reported operand keeps its double value rather than a float image. + std::vector poisoned(kN, 1.0); + poisoned[1] = std::numeric_limits::quiet_NaN(); + const vllm_test::MaxAbsDiffScanResult bad = + vllm_test::MaxAbsDiffScan(poisoned.data(), one_golden, kN); + CHECK_FALSE(bad.ok()); + CHECK(bad.bad_index == 1); + CHECK(std::isnan(bad.bad_got)); + CHECK(std::isinf(bad.worst)); + CHECK_FALSE(bad.worst < 1e-5); +} + +TEST_CASE("max|diff| wrapper: vector against an fp32 golden") { + const float one_golden[] = {1.0f, 1.0f, 1.0f, 1.0f}; + std::vector tiny(kN, 1.0); + tiny[0] = 1.0 + std::ldexp(1.0, -30); + // The `const W*` overload, the shape test_qwen4_exp_hc.cpp:569 calls. + CHECK(vllm_test::MaxAbsDiff(tiny, one_golden, kN) == std::ldexp(1.0, -30)); + // ...and the two-vector overload with mixed element types. + const std::vector want(one_golden, one_golden + kN); + CHECK(vllm_test::MaxAbsDiff(tiny, want) == std::ldexp(1.0, -30)); +} + // Named so that the `*RAISES*` CTest filter below matches exactly one case — // doctest's name matching is case-INSENSITIVE. TEST_CASE("max|diff| wrapper: a finite comparison reports nothing") { diff --git a/tests/vllm/models/test_qwen4_exp_hc.cpp b/tests/vllm/models/test_qwen4_exp_hc.cpp index 0b07e1774..55265be76 100644 --- a/tests/vllm/models/test_qwen4_exp_hc.cpp +++ b/tests/vllm/models/test_qwen4_exp_hc.cpp @@ -688,7 +688,9 @@ TEST_CASE("qwen4_exp GroupedRmsNorm needs its double accumulator at group_size 2 // At flat = 10240 it is not, and the reason is not ours alone. Measured against // the PINNED ORACLE itself (transformers v5.16.0, the same lift-and-exec path // `scripts/gen-qwen4-exp-hc-goldens.py` uses, at hidden=2560 hc=4 lowrank=320 -// eps=1e-6, two tokens), max|diff| on `mixed_input`: +// eps=1e-6, two tokens), max|diff| on `mixed_input`. These are ONE draw of +// random inputs, and the ratios below move from draw to draw; the ordering and +// the conclusion do not. // // t=0 t=1 // ours (fp32) vs oracle 2.325e-05 2.137e-05 @@ -697,9 +699,11 @@ TEST_CASE("qwen4_exp GroupedRmsNorm needs its double accumulator at group_size 2 // // Two conclusions, and the second is the one that matters. Our fp32 interior is // 2.1x to 2.3x over kTol at model width, driven by `LinearNoBias`'s sequential -// fp32 accumulation over 10240 terms. And the ORACLE is 1.36x over kTol against -// an exact evaluation of its own algorithm, because torch runs this in fp32 too -// — so widening OUR accumulator cannot rescue a 1e-5 absolute bound here. No +// fp32 accumulation over 10240 terms. And the ORACLE is itself OF THE SAME +// ORDER AS kTol against an exact evaluation of its own algorithm — 1.36x on the +// draw tabulated above, 0.91x and 0.82x on an independent draw taken during +// fresh review — because torch runs this in fp32 too, so widening OUR +// accumulator cannot rescue a 1e-5 absolute bound here. No // fp32 implementation of this function meets kTol at hidden_size 2560, and the // tolerance is the thing that is wrong at that width, not the arithmetic. //