diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 8d3d8f6d0..e26686fa4 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -748,6 +748,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#526](https://github.com/mudler/vllm.cpp/issues/526) | `SERVE-TOOL-HISTORY-ARGS` | OpenAI multi-turn tool history reaches chat templates with string-valued arguments | bug | | [#1934](https://github.com/mudler/vllm.cpp/issues/1934) | `BACKEND-ROCM` | `RocmPlatform::needs_weight_staging()` is stale-false (a W0-era placeholder never revisited despite #523/#509/#506/ROCM_ATTN/hipGraph landing since), so `CheckDeviceWeightFit` — the #1123/#1870 load-time refusal, including the `policy_forces_full_expand` fix — never runs on ROCm: measured directly, `VT_DEVICE_WEIGHT_BUDGET_BYTES=1` produced no refusal on a real load. The actual device allocation the refusal guards is not gated on this flag, so #1870's crash stays reachable until this closes; owed, not fixed in flow, because flipping the flag also moves `DirectDeviceLoadEligible` and several GDN kernel-dispatch defaults that each need their own correctness check | bug | | [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-QWEN4-EXP` | **`Qwen/Qwen3.8-Flash-Next` declares `Qwen4ExpForConditionalGeneration` / `qwen4_exp`, a new architecture vLLM does not implement, so the port runs on a split oracle: transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH.** Released 2026-08-24, 180B total / 6B activated, image-text-to-text. The `Qwen3.8` in the name is marketing continuity: `.agents/specs/qwen38-27b-bf16-gate.md`'s "one config key differs" precedent does NOT extend here. Read live 2026-08-26 at vLLM `origin/main` = `6a5e8f5979`: no `qwen4*` path, no registry entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. That is absence from vLLM `main` rather than staleness in our pin `555967922`, so a pin advance does not reach it. What exists is transformers [#48337](https://github.com/huggingface/transformers/pull/48337) "Add Qwen4Exp model", MERGED 2026-08-26, 5211 lines, and SGLang [#36497](https://github.com/sgl-project/sglang/pull/36497), still OPEN and therefore inadmissible. **Developer direction 2026-08-26, recorded verbatim: "use transformers as oracle for algorithmic side. but use ops from vllm so we account for optimized path."** Justified rather than convenient: `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so porting it as written yields a correct model at an indefensible speed, while AGENTS.md's mirror-vLLM polarity still binds every primitive vLLM implements. `Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`), all of which this tree has; GDN is an exact match for our AOT gate (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). The delta is four things, and **exactly two have no vLLM op at all**: the PLE dilated depthwise conv (kernel 4, dilation 3; `git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the n-gram hashed embedding. **The survey's load-bearing finding, and it REVERSES this row's first reading: QSA's structural twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The original call was that QSA, being plain GQA rather than MLA, had to map onto vLLM's non-MLA block-sparse case; that reasoning rested on treating `MLAAttentionSpec` as an MLA claim, and **it is not one** — M3's own indexer cache uses it while M3 is a plain-GQA model, with the comment "Key-only: MLAAttentionSpec budgets one vector/token (not 2x for K+V)". It is a budget shape. Remove that prop and the GQA-vs-MLA argument collapses. Verified line by line at `6a5e8f5979`: **nine independent structural matches with DSv4**, `compress_ratio == 4` literally the same number — MQA index with 1 key head at dim 128; `relu(q.k)` summed over index heads vs `(score.relu() * weights).sum(dim=0)`; `1/sqrt(head_dim)`; one score set per query token with no head axis vs `topk_indices_buffer[num_tokens, topk]`; pooling boundary `(position+1) % COMPRESS_RATIO == 0`; RMSNorm on the pooled key; **RoPE at the block-start position** vs `compressed_pos = (position // CR) * CR`; candidate count `visible // compress_ratio`; and one stored state per 4 tokens via `MLAAttentionSpec(tokens_per_state=compress_ratio)`, a first-class KV field documented as "Ints > 1 compress multiple tokens into one state (DSv4 sparse MLA)" that has no M3 equivalent. **M3 is a DIFFERENT ALGORITHM**, not a worse fit: its score is `tl.max(qk, axis=1)` over 128 RAW token dots with no pooling, no relu and no head reduction, it asserts `num_idx_heads == num_kv_heads` ("no topk index reduce") so it emits one block set PER KV HEAD, and its `SPARSE_BLOCK_SIZE = 128` is welded to the KV page size ("One sparse block == one KV page") on both the score and the attend side — moving it to 4 forces a page size of 4 and breaks `tl.dot`, whose tile needs >= 16. M3 contributes exactly ONE thing and it is a wiring precedent, not an algorithm: that a plain-GQA model can own a key-only side cache through `MLAAttentionSpec` and a private indexer backend. **The genuinely new work is the CONSUMER and nothing upstream supplies it** — every DSv4 sparse consumer attends to COMPRESSED MLA KV (one state per 4 tokens) and M3's attend to raw tokens only at page granularity, while QSA attends to RAW tokens selected at ratio-4 granularity. Two silent-failure traps follow: wiring QSA's top-k into a DSv4 sparse-MLA consumer attends a POOLED key/value and still emits plausible tokens, and **a short-prompt token gate cannot catch it because at context <= `indexer_budget` 2048 every candidate is selected** — so any QSA gate must run past 2048 tokens of context, which is now a stated `## Gates` requirement; and `SparseAttnCompressNormRopeStoreC4Kernel` does NOT mean-pool despite its name — it is a learned softmax pool over an OVERLAPPING window of 8 using a score channel this checkpoint does not have, and the CuteDSL variant refuses `overlap=False` at compile, so the **Triton** `head_dim=128` variant is the correct starting point. Two structural consequences beyond the module list: the residual stream is `hc_count * hidden_size` = **4 x 2560 = 10240 wide through the whole stack** with a low-rank read gate and per-branch scalar write gate around both attention and MLP, which is a change to the per-layer loop and every residual buffer rather than a drop-in module; and `number_of_conv_states = 3` on a PLE layer (GDN conv, PLE conv, n-gram token history) plus the indexer side cache, adjacent to [#1963](https://github.com/mudler/vllm.cpp/issues/1963) and [#1966](https://github.com/mudler/vllm.cpp/issues/1966). **NOTHING PUBLISHED FITS**, read live from the HF API against ~119 GB usable on GB10: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB (NVFP4 backbone with the n-gram table left at FP8, 51.2 GB) and `unsloth/...-GGUF` is a README with ZERO weight files. No GGUF exists and no tool can make one, because llama.cpp has no `qwen4_exp` either, so the standing k-quant requirement means authoring the arch on our side AND states that the quantized arms have NO llama.cpp oracle. **The chosen arm does NOT load today, and the blocker is neither the offload nor the budget: this tree cannot keep a gather table quantized at all.** `KeepQuantKDim` returns `-1` for `GgufTensorRole::kEmbeddingTable` (`src/vllm/model_executor/model_loader/gguf_keep_quant.cpp`), and `qwen3_5_gguf_weights.cpp` asserts it by name — "the embedding table cannot keep quant blocks" — so a Q4_K or Q8_0 n-gram table EXPANDS to bf16 and 51.2B params become **102.4 GB of anonymous memory**; the arm dies before the first forward. The reason was already sitting in a header comment ("a gather, not a GEMM ... A quantized-gather op is a follow-up row") and **no such row exists**. The only non-expanding gather residency is `kKeepF16`, requiring ggml type 1 exactly (102.4 GB on disk) and CPU-ONLY, because `EmbeddingKernelCuda` refuses anything but f32/bf16. **Second blocker:** `moe_intermediate_size = 640` makes `ffn_down_exps` Q4_K-illegal on its reduction dim (640 % 256 = 128), as does `hc_lowrank = 320`; llama.cpp's substitution is believed to be Q5_0 (**UNVERIFIED, owed against the pinned llama.cpp oracle**) and the dependent fact IS verified in-tree — our reader knows ggml ids `0,1,2,8,10..14,16,18,19,22..28,30,39,40,41,66` and has **no entry for 3, 6, 7 or 20**, so a stock `llama-quantize -Q4_K_M` file fails at header parse. We author the converter, so the fix is Q4_0 (block 32, same 4.5 bpw). **`ENG-WEIGHT-OFFLOAD` will not help** — it moves zero bytes today (`ConsiderWeight` has no production callers, pinned by a test) and is documented inert on GB10; the tier that DOES work already ships and is proven by the 2.4T model serving 369.97 GiB from a 119.631 GiB box at ~62 GiB resident: mmap `MAP_PRIVATE`, borrow in place, alias the host pointer, `prefault: false`. Corrected sizing: backbone ~67.7 GiB, whole process ~73.5 GiB of 119.631 at 32K single-stream, ~46 GiB of headroom for the page cache, so the ~76 GB estimate was right within 10%. The design works because per-token demand is **<= 64 KiB of reads** (16 lookups x 160 dims over at most 16 pages) against the 2.4T expert lane's 6.95 GB/token. The architecture supplies its own lever: the per-token n-gram cost is `(ngram_size-1)*heads_per_ngram` = 16 lookups of 160 dims, so **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per token** and making it non-resident is the intended design point (RadixArk reached the same split independently). Sizing arithmetic, NOT measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB (yes, ~10 GB left for KV), Q4_K_M backbone with the table non-resident ~76 GB. GB10 is UNIFIED memory so "offload to host" is not a move there; non-resident means disk-backed, and its cost is unmeasured. **Two decisions were put to the developer as explicit accept-or-reject and BOTH are settled 2026-08-26, recorded in place rather than left open.** (1) `.agents/oracles/transformers.md` pins transformers to 5.14.1, deliberately tied to what the pinned vLLM environment resolves so the environment cannot hold two `transformers` at once, and **5.14.1 does not contain `Qwen4Exp`**; the lane-scoped second pin is **ACCEPTED**, on the argument that the invariant guards a vLLM environment against drifting from its transformers and here there is no vLLM implementation to drift from, and it expires the moment vLLM registers `qwen4_exp`. **The lane pin is a real release, not a branch SHA**, which was not the expected outcome: `Qwen4Exp` merged to `main` at 12:03:40Z on 2026-08-26 and `v5.16.0` published at 12:35:15Z, and this was BOUNDED rather than assumed by fetching `models/qwen4_exp/modeling_qwen4_exp.py` at each tag — `v5.16.0` HTTP **200**, `v5.15.0` HTTP **404** — making 5.16.0 the FIRST release carrying the architecture and therefore the tightest available pin. The version string is UNMEASURED (it is the release proven to contain the model, not a `transformers.__version__` read off a running oracle) and `gateable` stays `no`. (2) The first runnable arm is the **Q4_K_M backbone with the n-gram table NON-RESIDENT** (~76 GB). Q8_0 was raised and does not fit at ~191 GB, and no partial-Q8 split reaches 119 GB with the backbone at 8 bits; Q4_K_M-throughout fits on paper at ~109 GB but leaves ~10 GB for KV and activations on a 262144-native-context model, which is not a margin. This promotes the non-resident table from a note to a first-class W6 deliverable, and it is NOT free: GB10 is UNIFIED memory, so the existing host-pinned offload seam (`ENG-WEIGHT-OFFLOAD`, mirroring vLLM's `cpu_offload_gb`) does not by itself solve it there and the mechanism must be disk-backed or genuinely unloaded — established before it is designed around. Spec: [`specs/qwen4-exp-flash-next.md`](specs/qwen4-exp-flash-next.md). No product code lands under the spec pull request | feature | +| [#1988](https://github.com/mudler/vllm.cpp/issues/1988) | `MODEL-MM-QWEN4-EXP` | **W3 of the Qwen4-Exp port: the 4-branch GATED-RESIDUAL hyper-connection stream and the grouped RMSNorm it stands on.** The residual stream is `hc_count * hidden_size` = 4 x 2560 = 10240 wide through the whole 48-layer stack, read and written twice per layer, and collapsed at the end by the same class with its injection branch switched off — a change to every residual buffer, not a drop-in module. Landed here as a HOST reference (`src/vllm/model_executor/models/qwen4_exp_hc.{h,cpp}`) gated against goldens dumped by EXECUTING the lane-pinned oracle source: transformers `v5.16.0` `models/qwen4_exp/modeling_qwen4_exp.py` (sha256 `77fec77d…`), `Qwen4ExpTextRMSNorm` (:158-181) and `Qwen4ExpTextGatedResidual` (:941-969) lifted verbatim by line range, plus an independent double-precision reference. The grouped norm mirrors vLLM's op form — `RMSNormGated` (`layers/layernorm.py:172`, `group_size` at `:187`, grouped branch `:258-264`) with the gate disabled, NOT the plain `RMSNorm` (`:37`), whose only related knob is `var_hidden_size`, a prefix reduction that cannot express per-group norms. **Three findings the gate now pins.** (1) The `1 + w` parameterization: transformers applies `out * (1.0 + weight)` on a ZERO-init weight while vLLM applies `out * weight` on a ONES-init one, they coincide only under a load-time `w = 1.0 + w_hf`, and the published GGUF has that fold applied at CONVERT time — so it lives in exactly one named function, `HcNormWeightFromHf`, and skipping it scales every `hc_norm` by ~0 (reads as a checkpoint bug) while applying it twice scales by ~2x. (2) The two divisions by `hc_count` are different: one is INSIDE the SiLU on the `[320]` low-rank intermediate BEFORE the activation (`silu(down(x)/4)`, not `silu(down(x))/4`; SiLU is not homogeneous), the other is inside the injection sigmoid with the whole sigmoid scaled by 2 (`2*sigmoid(inject(x)/4)`, range (0,2), exactly 1.0 at a zero logit), and there is NO division on the up-projection sigmoid. (3) The elementwise multiply uses the NORMED stream, the reduce over hc is a MEAN and not a sum, and `hyper_input` is written back RAW. **The spec's `MhcPost`-with-identity-comb reuse claim is VERIFIED rather than trusted**, by a bit-equality case against our DeepSeek-V4 kernel; it holds on finite inputs and is not an identity for a negative-zero or non-finite residual, neither of which is reachable here. **Not reached at its merge commit** — W1 config registration (#1986) is still in review, so nothing loads a `qwen4_exp`; the wiring is owed by W5 (assembly) under #1978 and is listed in the spec's `## Owed`. No token claim and no speed claim: no arm of this model runs on any fleet device. | feature | | [#1991](https://github.com/mudler/vllm.cpp/issues/1991) | `MODEL-MM-QWEN4-EXP` | **W4, Qwen Sparse Attention: the indexer, its side cache, and a GATHER consumer rather than a mask.** New TU `src/vllm/model_executor/models/qwen4_exp_qsa.{h,cpp}` (`indexer_n_heads` 4, `indexer_kv_heads` 1, `indexer_head_dim` 128, `indexer_budget` 2048 tokens, `indexer_compress_ratio` 4, so `block_topk` 512 and an index buffer 2051 wide). Ported onto DeepSeek-V4's C4 indexer lane per the spec's reversal, with the two things it must NOT inherit reconciled in code: DSv4's `weights_proj` + `head_scale` fold (QSA's weight is the constant `1/sqrt(128)`, applied after the sum over heads) and DSv4's GPT-J RoPE over a TRAILING span (QSA is NeoX `rotate_half` over the LEADING 64 dims with the NoPE dims trailing — the halves are swapped end for end AND the pairing convention differs). `SparseAttnCompressNormRopeStoreC4Kernel` supplies scaffolding only: its `tl.softmax(score, dim=0)` pool over an overlapping window is replaced by an unweighted mean over a non-overlapping window of 4. **The consumer is a GATHER, and that is the wave's point.** llama.cpp [#27739](https://github.com/ggml-org/llama.cpp/pull/27739) records that a sparse mask over a dense cache costs the same as dense attention under CUDA flash attention, because `flash_attn_mask_to_KV_max` only scans back to the first tile that is not all `-inf`; a mask-only QSA is CORRECT and forfeits the lever silently. Proved rather than asserted: mutating `QsaGatherAttention` to delegate to the mask reference leaves **11 of 12 cases green** and fails only `qsa-consumer: the GATHER touches only the selected rows`, so the `keys_visited` counter is the only instrument in the suite that can tell the two apart. Goldens come from RUNNING the unmodified `Qwen4ExpTextQSAIndexer.forward` at the transformers 5.16.0 lane pin (`tests/vllm/models/fixtures/gen_qwen4_exp_qsa_goldens.py`), which resolves the spec's UNMEASURED version string: `transformers.__version__` reads **5.16.0** on a live import, and the oracle runs on CPU with random weights, no checkpoint and no GPU. Both free oracles are used: sub-budget bit-identity with dense attention at kv_len = `budget + compress_ratio - 1`, and index-set equality including the ragged tail above the budget. Side cache is `MLAAttentionSpec(num_kv_heads=1, head_size=128, tokens_per_state=4)` = **64 B/token/layer at bf16**, asserted rather than defaulted, with `StatesForTokens` flooring because a ragged block writes no state. LANDS UNREACHED: `Qwen4ExpTextModel` does not exist (W2 #1987, W3 #1988, W6a #1989 are sibling waves; the registry and runner wiring is W5), row `MODEL-MM-QWEN4-EXP` owns it, [#1978](https://github.com/mudler/vllm.cpp/issues/1978) tracks it, and the spec lists it under `## Owed`. No CUDA arm and no speed claim — the speed axis opens at G4, after W6a | feature | | [#1987](https://github.com/mudler/vllm.cpp/issues/1987) | `MODEL-MM-QWEN4-EXP` | **W2 of [#1978](https://github.com/mudler/vllm.cpp/issues/1978): the hashed n-gram embedding and the PLE dilated depthwise conv, the ONLY two components of `Qwen4ExpForConditionalGeneration` with no vLLM op at all.** The negative is confirmed rather than unfound — at vLLM `origin/main` = `6a5e8f5979`, `git grep -in dilat` returns ZERO lines in `layers/mamba/`, zero in `csrc/` and zero in `tests/`, and `layers/conv.py` defines only `Conv2dLayer`/`Conv3dLayer` — so transformers **v5.16.0**, this row's accepted lane pin, is the sole oracle. Three silent divergence sites, all verified: `_splitmix64` must be `uint64_t` throughout (its `>> 30/27/31` are LOGICAL shifts on a non-negative Python int; on `int64_t` they go arithmetic and the multiplicand is wrong, with the top bit set about half the time), `_splitmix64(value) % half_bound` must be an UNSIGNED modulo (the dividend routinely exceeds 2^63), and shard reassembly is NUMERIC not lexicographic (`shard_0, shard_1, shard_10, ...` silently permutes a 95 GiB table). The forward is int64-exact and needs no bignum because `multiplier_max * vocab_size <= 2^63 - 1` — but **that bound holds only while every token id is below `vocab_size`**, so an out-of-range id overflows and diverges in silence, and this wave refuses one by name. Goldens confirmed a FOURTH way beyond #1987's three: `vocab_size = 248320`, read from the released `config.json`, is the UNIQUE preimage below 2e6 of the published `layer_multipliers = [23703573157769, 20109073645365, 8052911324071]` at the absent-`seed` default 1234. PLE specifics: the conv state is **9 columns** — `(kernel-1) * dilation`, NOT `kernel-1` — and holds the **normed** conv input while the skip term is the **un-normed** copy; the signed-sqrt gate clamps BEFORE the sqrt so the output magnitude floor is `sqrt(1e-6) = 1e-3` and tiny scores are AMPLIFIED, with exactly zero mapping to zero, making the function genuinely discontinuous at an origin that a fully masked row reaches; and the n-gram history must be padded with EOS because upstream's `update_conv_state` pads with 0, a valid token id. **Lands UNREACHED** per AGENTS.md "Nothing lands dead": no production entry point calls it until W5 assembles the model, the wiring is owned by `MODEL-MM-QWEN4-EXP` and tracked by #1978, and the spec lists it under `## Owed`. Host reference only; the batched device arm is owed, and the per-sequence signatures exist so it drops in — the gather is 16 uncoalesced random rows per token PER SEQUENCE, so at batch B it is 16*B independent gathers with no coalescing. No token claim, no speed claim: `## Gates` admits none from this row until an arm runs, and nothing published fits any fleet device | feature | | [#1963](https://github.com/mudler/vllm.cpp/issues/1963) | `ROAD-V1-MEM` | At ctx=32768 `--max-num-seqs 32` our engine consumes ~108 GB during load and never serves; vLLM and SGLang both serve there. Root cause found and fixed by `FIX-KV-GROUP-LAYER-COUNT` ([spec](specs/kv-group-layer-count.md)): thirty-three of thirty-four registries publish ONE placeholder name per KV group, `KVBytesPerBlock` reads `layer_names.size()` as the layer count, and `ResolveNumBlocks` arm 2 therefore divides an absolute `--kv-cache-memory` budget by ONE layer's page while the runner allocates one buffer per layer — measured 8.5 GiB allocated for a 1 GiB budget on the 27B | bug | diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index d6117d853..fd5784960 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -421,6 +421,41 @@ that does not exist. Stated because that tail is the natural thing to copy. `w_vllm = 1.0 + w_hf`. Miss it and every `hc_norm` gets a near-zero scale, which reads as a checkpoint bug rather than a port bug. +**The GGUF converter already folds it, and it folds far more than `hc_norm`.** Read at +source rather than relayed, because W5 writes the loader and the narrow version of this +sentence causes the defect it warns about. Every anchor below is read at our recorded +llama.cpp pin, stock upstream tag `b10451` (`10bf611e533d81f739128304991c5e133c6aebd8`, +[`../oracles/llama-cpp.md`](../oracles/llama-cpp.md)). Stock upstream has no `qwen4exp` +at all there (`git grep -il qwen4exp`: nothing tree-wide, so a released llama.cpp can +neither convert nor load this architecture). The converter is ggml-org/llama.cpp +[#27742](https://github.com/ggml-org/llama.cpp/pull/27742), head +`035e22731a7fd70b9854b3a2d64ec68e9b1a45d3`, **still OPEN**. Its `conversion/qwen4exp.py` +declares `class Qwen4ExpTextModel(_Qwen35MRopeMixin, _LinearAttentionVReorderBase)`; +`_LinearAttentionVReorderBase` is `conversion/qwen.py:438`, a subclass of +`Qwen3NextModel` (`:365`, whose own signature is +`class Qwen3NextModel(_QwenMtpMixin, Qwen2MoeModel)`); and the PR's `modify_tensors` has +**no `hc_norm` branch**, so `hc_norm.weight` falls through to `super()`. The `+1` is the +inherited Qwen3-Next rule at `conversion/qwen.py:387-388`: + +```python +elif name.endswith("norm.weight") and not name.endswith("linear_attn.norm.weight"): + data_torch = data_torch + 1 +``` + +So the rule a loader implements is **every `*norm.weight` carries the fold, with +`linear_attn.norm.weight` (the GDN `ssm_norm`) the one exception** -- `hc_norm`, +`attn_q_norm` and `attn_k_norm` all match it, and the PLE and indexer gammas are folded +by the PR's own early-returning branch. A loader that skips the fold for `hc_norm` alone +double-folds everything else, which is the same silent ~2x defect one tensor to the left. +Two consequences for W5. The property belongs to one in-flight converter, not to "GGUF": +#27742 can change before it merges and another publisher's tool need not match it, so the +loader treats the fold as a provenance question and checks it -- cheaply, since an +unfolded `hc_norm` is a zero-init gamma and a folded one is centred on 1.0. And it was +corroborated on published artifacts during fresh review of #1988 +(`unsloth/Qwen3.8-Flash-Next-GGUF` `UD-IQ1_S` and `UD-Q4_K_XL`, `vumpt/...-Q4_K_M`, read +by HTTP range request against the bf16 HF tensors): every `*hc_norm.weight` is HF + 1.0 +exactly, elementwise, while `ssm_norm` is unfolded and sits in [0.875, 1.023]. + **Correction to the port map above.** vLLM's grouped RMSNorm is on **`RMSNormGated`**, not the plain `RMSNorm`, whose only related knob is `var_hidden_size` -- a prefix reduction that cannot express per-group norms. Verified directly: `RMSNorm` opens at @@ -822,6 +857,82 @@ change that makes any arm reachable, not later. currently owns building them, and a multimodal caller with genuinely different t/h/w streams would be running an untested section layout. The wave that builds the cos/sin tables owes a case with three DISTINCT position streams. +- **W3's host reference lands UNREACHED, and this is the record of it** per + AGENTS.md "Nothing lands dead". + `src/vllm/model_executor/models/qwen4_exp_hc.{h,cpp}` + ([#1988](https://github.com/mudler/vllm.cpp/issues/1988)) is reached only by + `tests/vllm/models/test_qwen4_exp_hc.cpp`. No production entry point calls it + at its merge commit: W1 config registration + ([#1986](https://github.com/mudler/vllm.cpp/issues/1986)) was still in review, + so no `qwen4_exp` resolves through the loader and there is nothing for the + gated-residual stream to hang off. The wiring is owed by **W5, assembly**, + under [#1978](https://github.com/mudler/vllm.cpp/issues/1978), which is the + wave that widens the residual buffers to `hc_count * hidden_size` and calls + the module twice per layer. +- The **model-matrix lifecycle cell** for + `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation`, which still reads + `SPEC ONLY`. Left to W1 deliberately rather than by omission: W1 is the wave + whose scope IS registration, its pull request is already open, and + `.agents/model-matrix.md` is a single shared file, so three parallel waves + editing one cell is the write-lock AGENTS.md "Records" names. Whichever of + W1/W2/W3 lands last owes the correction. +- The **device arm of the gated residual**, and with it one check this host wave + cannot make: that `RMSNormGated.forward_cuda`'s flash-linear-attention Triton + kernel is numerically correct in its GROUPED mode (unverified upstream, see + `## Design`). +- **W3's `kTol = 1e-5` is an absolute bound that does not survive a rescale, and + the host reference is the first thing it fails.** Recorded because an earlier + draft of the bullet above framed the tolerance question as the DEVICE arm's + problem, and it is not. Measured against the pinned oracle itself, at the + model's own shape (hidden_size 2560, hc_count 4, hc_lowrank 320, eps 1e-6, two + tokens), max|diff| on `mixed_input`. This is ONE draw of random inputs, and the + ratios below move from draw to draw; the ordering and the conclusion do not. + + | | t=0 | t=1 | + |---|---|---| + | ours (fp32) vs oracle | 2.325e-05 | 2.137e-05 | + | exact double vs oracle | 1.360e-05 | 5.431e-06 | + | ours (fp32) vs exact double | 3.684e-05 | 1.606e-05 | + + At the suite's own widths (flat = 24 and 15) the implementation is bit-identical + to the oracle -- max|diff| over every golden array of cases A, B and C is + 2.384e-07 -- so kTol carries a 42x margin there and constrains nothing. At model + width our fp32 interior is 2.1x to 2.3x over it, driven by `LinearNoBias`'s + sequential fp32 accumulation over 10240 terms. **The second row is the one that + settles it: the ORACLE is itself of the same ORDER as kTol against an exact + evaluation of its own algorithm -- 1.36x on the draw above, 0.91x and 0.82x on + an independent draw taken during fresh review -- because torch runs this in + fp32 too.** No fp32 + implementation of this function meets a 1e-5 ABSOLUTE bound at hidden_size + 2560, and widening our accumulator cannot rescue one. W5 therefore does not + reuse kTol at model width; the file carries a real-width case with a relative + bound (`kRealWidthMixedRel`, 4e-5, derived as 6.6x the sqrt(K)*u random-walk + bound for K = 10240) that all three measurements sit inside. **What is still + owed** is agreement with the ORACLE at model width, which needs a real + checkpoint and cannot be closed in-suite: the in-suite case compares against + the double reference, because dumping one token of oracle IO at this width is + 26 MB of `.inc`. +- **The double accumulator is now gated, and the device arm inherits the + consequence.** `GroupedRmsNorm` accumulates the per-group sum of squares in + `double`, and at the suite's group sizes of 5 and 6 that convention had zero + discriminating power -- replacing it with `float` left the suite 280/280 green. + It is gated at the model's real group size of 2560, on magnitude-separated + data, where the two accumulators differ by 742x (3.168e-06 against 2.352e-03, + bound 1e-4). The convention is kept rather than dropped because it makes the + host reference more accurate than the oracle rather than less, which is what a + reference is for. What follows for the device arm, stated here so it is not + discovered: **a straight fp32-accumulate device reduction will not meet + `kRealWidthNormTol` on that data.** That is the correct signal, not a defect in + the gate -- it says the device kernel must accumulate wider than fp32 or be + gated against the oracle directly rather than against this reference. Deciding + which is the device wave's, and it is owed. +- The **fused rank-1 write-back**. `GatedResidualWriteBackInPlace` is the seam + and is already the primitive, but no device kernel replaces it yet. Both + llama.cpp implementations of this architecture materialise the update as a + `repeat_4d` + `mul`, i.e. 96 dense `[2560, 4, T]` broadcasts built and thrown + away per forward pass at 48 layers x 2 sites, which is where a + beat-llama.cpp-at-concurrency claim would come from. Not claimed here: no arm + runs. ## Now diff --git a/CMakeLists.txt b/CMakeLists.txt index 8eecaa3a6..e1de6ecde 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -789,6 +789,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/qwen3_moe_registry.cpp src/vllm/model_executor/models/qwen3_moe_weights.cpp src/vllm/model_executor/models/qwen3_moe.cpp + src/vllm/model_executor/models/qwen4_exp_hc.cpp src/vllm/model_executor/models/deepseek_v2_registry.cpp src/vllm/model_executor/models/deepseek_v2_weights.cpp src/vllm/model_executor/models/deepseek_v2.cpp diff --git a/scripts/gen-qwen4-exp-hc-goldens.py b/scripts/gen-qwen4-exp-hc-goldens.py new file mode 100755 index 000000000..8238488ac --- /dev/null +++ b/scripts/gen-qwen4-exp-hc-goldens.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 +"""Dump Qwen4-Exp gated-residual goldens by EXECUTING the pinned oracle source. + + usage: gen-qwen4-exp-hc-goldens.py [path/to/modeling_qwen4_exp.py] + +Fetch the oracle source first (it is not vendored -- the sha256 below is the pin): + + curl -sSLO https://raw.githubusercontent.com/huggingface/transformers/\ +v5.16.0/src/transformers/models/qwen4_exp/modeling_qwen4_exp.py + + +Oracle: huggingface/transformers v5.16.0, + src/transformers/models/qwen4_exp/modeling_qwen4_exp.py + sha256 77fec77d87f2a0eb23b95fa04276fb5779698a7c7f523cf5061e49c118bcc459 + +The two classes under test are lifted VERBATIM by line range out of that file and +exec'd; nothing is retyped. The only harness adaptation is a 4-field stand-in for +`Qwen4ExpTextConfig` (the real dataclass drags in the whole transformers package, +and the installed transformers here is 5.3.0, which predates qwen4_exp entirely). +The write-back is likewise the verbatim two lines of +`Qwen4ExpTextDecoderLayer.forward`. +""" +import hashlib +import sys + +import torch +import torch.nn as nn +import torch.nn.functional as F + +SRC = sys.argv[2] if len(sys.argv) > 2 else "modeling_qwen4_exp.py" +EXPECT_SHA = "77fec77d87f2a0eb23b95fa04276fb5779698a7c7f523cf5061e49c118bcc459" + +raw = open(SRC, "rb").read() +got = hashlib.sha256(raw).hexdigest() +# NOT an `assert`: `python3 -O` strips those, and this one guard is the entire +# mechanism tying the goldens to the pin. Under -O the stripped version would +# happily dump goldens from any file handed to it. +if got != EXPECT_SHA: + raise SystemExit(f"oracle source sha256 {got} != {EXPECT_SHA}") +lines = raw.decode().splitlines(keepends=True) + +# 1-based, inclusive, as reported by grep -n on the pinned file. +RMSNORM = (158, 181) # class Qwen4ExpTextRMSNorm +GATEDRES = (941, 969) # class Qwen4ExpTextGatedResidual + + +def lift(span): + return "".join(lines[span[0] - 1: span[1]]) + + +class Qwen4ExpTextConfig: # harness stand-in; only the 4 fields the classes read + def __init__(self, hidden_size, hc_count, hc_lowrank, rms_norm_eps): + self.hidden_size = hidden_size + self.hc_count = hc_count + self.hc_lowrank = hc_lowrank + self.rms_norm_eps = rms_norm_eps + + +ns = {"torch": torch, "nn": nn, "F": F, "Qwen4ExpTextConfig": Qwen4ExpTextConfig} +exec(lift(RMSNORM), ns) +exec(lift(GATEDRES), ns) +Qwen4ExpTextRMSNorm = ns["Qwen4ExpTextRMSNorm"] +Qwen4ExpTextGatedResidual = ns["Qwen4ExpTextGatedResidual"] + +torch.set_default_dtype(torch.float32) +out = [] + + +def emit(name, t): + flat = t.detach().reshape(-1).tolist() + out.append("const float %s[] = {" % name) + for i in range(0, len(flat), 4): + out.append(" " + ", ".join("%.9gf" % v for v in flat[i:i + 4]) + ",") + out.append("};") + + +def case(tag, hidden, hc, lowrank, eps, tokens, use_combine, seed): + g = torch.Generator().manual_seed(seed) + cfg = Qwen4ExpTextConfig(hidden, hc, lowrank, eps) + mod = Qwen4ExpTextGatedResidual(cfg, use_combine=use_combine) + hc_h = hc * hidden + with torch.no_grad(): + # hc_norm.weight is ZERO-init upstream; a zero weight makes (1 + w) == 1 + # and would hide the parameterization entirely, so it is randomized. + mod.hc_norm.weight.copy_(torch.randn(hc_h, generator=g) * 0.5) + mod.input_mix_weight_down.weight.copy_(torch.randn(lowrank, hc_h, generator=g) * 0.3) + mod.input_mix_weight_up.weight.copy_(torch.randn(hc_h, lowrank, generator=g) * 0.3) + if use_combine: + mod.block_inject_weight.weight.copy_(torch.randn(hc, hc_h, generator=g) * 0.3) + hyper = torch.randn(tokens, hc_h, generator=g) * 1.7 + block_out = torch.randn(tokens, hidden, generator=g) * 0.9 + + normed = mod.hc_norm(hyper) + res = mod(hyper) + if use_combine: + mixed, hyper_ret, inj = res + # NOT an `assert`, for the same reason as the sha guard above: this + # is the only check that upstream still hands back the RAW input for + # the write-back, and `python3 -O` strips an `assert`. + if hyper_ret is not hyper: + raise SystemExit("upstream must return hyper_input RAW") + injection = block_out.unsqueeze(-2) * inj.unsqueeze(-1) + written = hyper + injection.flatten(-2) + else: + mixed = res + + out.append("") + out.append("// ---- %s: hidden=%d hc=%d lowrank=%d eps=%g T=%d use_combine=%s seed=%d" + % (tag, hidden, hc, lowrank, eps, tokens, use_combine, seed)) + emit("k%s_norm_w_hf" % tag, mod.hc_norm.weight) + emit("k%s_down" % tag, mod.input_mix_weight_down.weight) + emit("k%s_up" % tag, mod.input_mix_weight_up.weight) + if use_combine: + emit("k%s_inject" % tag, mod.block_inject_weight.weight) + emit("k%s_hyper" % tag, hyper) + emit("k%s_normed" % tag, normed) + emit("k%s_mixed" % tag, mixed) + if use_combine: + emit("k%s_block_out" % tag, block_out) + emit("k%s_inj_w" % tag, inj) + emit("k%s_written" % tag, written) + + +out.append("// GENERATED by scripts/gen-qwen4-exp-hc-goldens.py -- do not hand-edit.") +out.append("// Oracle: transformers v5.16.0 modeling_qwen4_exp.py") +out.append("// sha256 %s" % EXPECT_SHA) +out.append("// Qwen4ExpTextRMSNorm :158-181 (grouped RMSNorm, (1.0 + weight))") +out.append("// Qwen4ExpTextGatedResidual :941-969") +out.append("// write-back: Qwen4ExpTextDecoderLayer.forward, the two lines") +out.append("// injection = hidden_states.unsqueeze(-2) * injection_weights.unsqueeze(-1)") +out.append("// hidden_states = hyper_input + injection.flatten(-2)") +out.append("// torch %s" % torch.__version__) + +case("A", hidden=6, hc=4, lowrank=5, eps=1e-6, tokens=3, use_combine=True, seed=1234) +case("B", hidden=5, hc=3, lowrank=7, eps=1e-5, tokens=2, use_combine=True, seed=99) +case("C", hidden=6, hc=4, lowrank=5, eps=1e-6, tokens=2, use_combine=False, seed=7) + +open(sys.argv[1], "w").write("\n".join(out) + "\n") +print("wrote", sys.argv[1]) diff --git a/src/vllm/model_executor/models/qwen4_exp_hc.cpp b/src/vllm/model_executor/models/qwen4_exp_hc.cpp new file mode 100644 index 000000000..56a8bff90 --- /dev/null +++ b/src/vllm/model_executor/models/qwen4_exp_hc.cpp @@ -0,0 +1,183 @@ +// Qwen4-Exp (`Qwen3.8-Flash-Next`) W3 — the 4-branch GATED-RESIDUAL +// hyper-connection stream and its grouped RMSNorm. HOST reference. +// +// Algorithm: transformers v5.16.0 (the lane pin) +// `models/qwen4_exp/modeling_qwen4_exp.py` +// ::Qwen4ExpTextRMSNorm (:158-181), ::Qwen4ExpTextGatedResidual (:941-969), +// ::Qwen4ExpTextDecoderLayer.forward write-back (:825-826, :831-832 in the +// modular file `modular_qwen4_exp.py`). +// Op form: vLLM @ origin/main 6a5e8f5979 +// `model_executor/layers/layernorm.py`::RMSNormGated (:172), `group_size` +// (:187), grouped branch in `forward_static` (:243-244, :258-264). +// See qwen4_exp_hc.h for the oracle table, the `1 + w` resolution, and the +// fusion-seam note. +#include "vllm/model_executor/models/qwen4_exp_hc.h" + +#include +#include +#include + +namespace vllm::qwen4_exp { + +namespace { + +float Sigmoid(float x) { return 1.0f / (1.0f + std::exp(-x)); } + +// y[o] = Σ_i w[o*K + i] · x[i]. PyTorch `nn.Linear(bias=False)` weight layout, +// `(out_features, in_features)` row-major. +void LinearNoBias(const float* w, const float* x, int64_t out_dim, int64_t in_dim, float* y) { + for (int64_t o = 0; o < out_dim; ++o) { + const float* row = w + o * in_dim; + float acc = 0.0f; + for (int64_t i = 0; i < in_dim; ++i) acc += row[i] * x[i]; + y[o] = acc; + } +} + +void RequireSize(size_t got, size_t want, const char* what) { + if (got != want) { + throw std::invalid_argument(std::string("qwen4_exp: ") + what + " has " + std::to_string(got) + + " elements, expected " + std::to_string(want) + "."); + } +} + +} // namespace + +std::vector HcNormWeightFromHf(const std::vector& w_hf) { + std::vector w(w_hf.size()); + for (size_t i = 0; i < w_hf.size(); ++i) w[i] = 1.0f + w_hf[i]; + return w; +} + +std::vector GroupedRmsNorm(const std::vector& x, const std::vector& weight, + int64_t group_size, float eps) { + if (group_size <= 0) { + throw std::invalid_argument("qwen4_exp: group_size must be positive, got " + + std::to_string(group_size) + "."); + } + const size_t g = static_cast(group_size); + if (x.size() % g != 0) { + // `Qwen4ExpTextRMSNorm.__init__` (:164-165) raises the same condition. + throw std::invalid_argument("qwen4_exp: hidden_size (" + std::to_string(x.size()) + + ") must be divisible by group_size (" + std::to_string(g) + ")."); + } + RequireSize(weight.size(), x.size(), "hc_norm weight"); + + std::vector out(x.size()); + for (size_t base = 0; base < x.size(); base += g) { + // The reduction is INDEPENDENT per group: this is what `RMSNormGated`'s + // `rearrange(x, "... (g d) -> ... g d")` buys and what the plain `RMSNorm` + // cannot express. Accumulated in double, per the host-reference convention. + double ss = 0.0; + for (size_t d = 0; d < g; ++d) { + const double v = x[base + d]; + ss += v * v; + } + // eps is INSIDE the rsqrt, added to the mean square, never to the norm. + const float r = 1.0f / std::sqrt(static_cast(ss / static_cast(g)) + eps); + for (size_t d = 0; d < g; ++d) { + out[base + d] = x[base + d] * r * weight[base + d]; + } + } + return out; +} + +GatedResidualResult GatedResidualForward(const std::vector& hyper_input, + const GatedResidualWeights& weights, int64_t hc, + int64_t hidden, float eps) { + if (hc <= 1 || hidden <= 0) { + // `Qwen4ExpTextConfig.__post_init__` rejects hc_count <= 1 outright. + throw std::invalid_argument("qwen4_exp: hc_count must be > 1 and hidden_size > 0, got " + + std::to_string(hc) + " and " + std::to_string(hidden) + "."); + } + const int64_t flat = hc * hidden; + if (hyper_input.size() != static_cast(flat)) { + // Mirrors `Qwen4ExpTextGatedResidual.forward` (:955-958). + throw std::invalid_argument("qwen4_exp: expected " + std::to_string(flat) + + " hyper-connection features, got " + + std::to_string(hyper_input.size()) + "."); + } + if (weights.mix_down.empty() || weights.mix_down.size() % static_cast(flat) != 0) { + throw std::invalid_argument("qwen4_exp: input_mix_weight_down is not a multiple of " + + std::to_string(flat) + " (got " + + std::to_string(weights.mix_down.size()) + ")."); + } + const int64_t rank = static_cast(weights.mix_down.size() / static_cast(flat)); + RequireSize(weights.mix_up.size(), static_cast(flat * rank), "input_mix_weight_up"); + if (!weights.block_inject.empty()) { + RequireSize(weights.block_inject.size(), static_cast(hc * flat), "block_inject_weight"); + } + + GatedResidualResult out; + out.hyper_input_normed = + GroupedRmsNorm(hyper_input, weights.hc_norm_weight, hidden, eps); + const float* normed = out.hyper_input_normed.data(); + + // DIVISION 1 — inside the SiLU, on the [rank] low-rank intermediate, BEFORE + // the activation: `F.silu(down(x) / hc_count)`. SiLU is not homogeneous, so + // `silu(a)/hc` is a different function; the placement is load-bearing. + std::vector low(static_cast(rank)); + LinearNoBias(weights.mix_down.data(), normed, rank, flat, low.data()); + // A true division, not a reciprocal multiply: upstream spells `/ self.hc_count` + // and 1/hc is inexact for any hc that is not a power of two, so the shortcut + // would put a 1-ulp wedge between this reference and the oracle at hc_count=3. + const float hc_f = static_cast(hc); + for (int64_t r = 0; r < rank; ++r) { + const float a = low[r] / hc_f; + low[r] = a * Sigmoid(a); + } + + // NO division on the up projection: `torch.sigmoid(up(...))`, full stop. + std::vector gate(static_cast(flat)); + LinearNoBias(weights.mix_up.data(), low.data(), flat, rank, gate.data()); + for (int64_t p = 0; p < flat; ++p) gate[p] = Sigmoid(gate[p]); + + // `.unflatten(-1, (hc, H))`, multiply against the NORMED stream — not the raw + // one — then `.mean(dim=-2)`. A MEAN over hc, never a sum. + out.mixed_input.assign(static_cast(hidden), 0.0f); + for (int64_t j = 0; j < hc; ++j) { + const float* g_row = gate.data() + j * hidden; + const float* s_row = normed + j * hidden; + for (int64_t h = 0; h < hidden; ++h) out.mixed_input[h] += g_row[h] * s_row[h]; + } + for (int64_t h = 0; h < hidden; ++h) out.mixed_input[h] /= hc_f; + + // `block_inject_weight is None` returns here, and that early return IS the + // model's final mixer (`use_combine=False`). + if (weights.block_inject.empty()) return out; + + // DIVISION 2 — inside the injection sigmoid, whole sigmoid scaled by 2: + // `2 * sigmoid(inject(x) / hc_count)`. Range (0, 2), exactly 1.0 at a zero + // logit, so an untrained branch is the identity rather than a half-scale. + out.injection_weights.assign(static_cast(hc), 0.0f); + LinearNoBias(weights.block_inject.data(), normed, hc, flat, out.injection_weights.data()); + for (int64_t j = 0; j < hc; ++j) { + out.injection_weights[j] = 2.0f * Sigmoid(out.injection_weights[j] / hc_f); + } + return out; +} + +void GatedResidualWriteBackInPlace(float* hyper, const float* block_out, + const float* injection_weights, int64_t hc, int64_t hidden) { + // The rank-1 update, never the materialized broadcast. See the header: this + // function is the seam a fused device kernel replaces. + for (int64_t j = 0; j < hc; ++j) { + const float w = injection_weights[j]; + float* row = hyper + j * hidden; + for (int64_t h = 0; h < hidden; ++h) row[h] += block_out[h] * w; + } +} + +std::vector GatedResidualWriteBack(const std::vector& hyper_input, + const std::vector& block_out, + const std::vector& injection_weights, int64_t hc, + int64_t hidden) { + RequireSize(hyper_input.size(), static_cast(hc * hidden), "hyper_input"); + RequireSize(block_out.size(), static_cast(hidden), "block output"); + RequireSize(injection_weights.size(), static_cast(hc), "injection_weights"); + std::vector out = hyper_input; + GatedResidualWriteBackInPlace(out.data(), block_out.data(), injection_weights.data(), hc, hidden); + return out; +} + +} // namespace vllm::qwen4_exp diff --git a/src/vllm/model_executor/models/qwen4_exp_hc.h b/src/vllm/model_executor/models/qwen4_exp_hc.h new file mode 100644 index 000000000..e70417407 --- /dev/null +++ b/src/vllm/model_executor/models/qwen4_exp_hc.h @@ -0,0 +1,197 @@ +// Qwen4-Exp (`Qwen3.8-Flash-Next`) W3 — the 4-branch GATED-RESIDUAL +// hyper-connection stream and the grouped RMSNorm it stands on. HOST reference. +// Issue #1988, spec `.agents/specs/qwen4-exp-flash-next.md` ("Gated Residual: +// what our MHC actually gives us"). +// +// ─── WHAT THIS IS A PORT OF ─────────────────────────────────────────────────── +// This row splits its oracles by design (spec `## Oracles`, developer direction +// 2026-08-26): transformers supplies the ALGORITHM, vLLM supplies the OP FORM. +// vLLM has never registered `qwen4_exp` — surveyed at `origin/main` 6a5e8f5979, +// zero hits tree-wide — so there is no vLLM model to mirror, but the grouped +// RMSNorm is a vLLM op and is mirrored as one. +// +// OURS <- ALGORITHM (transformers v5.16.0, the lane pin, +// `models/qwen4_exp/modeling_qwen4_exp.py`) +// GroupedRmsNorm <- `Qwen4ExpTextRMSNorm._norm` (:167-171) +// HcNormWeightFromHf <- `Qwen4ExpTextRMSNorm.forward` (:173-178), +// `output * (1.0 + self.weight.float())` +// GatedResidualForward <- `Qwen4ExpTextGatedResidual.forward` (:952-969) +// GatedResidualWriteBack <- `Qwen4ExpTextDecoderLayer.forward`, the two +// lines `injection = hidden_states.unsqueeze(-2) +// * injection_weights.unsqueeze(-1)` / +// `hidden_states = hyper_input + +// injection.flatten(-2)` +// +// OURS <- OP FORM (vLLM `origin/main` 6a5e8f5979, +// `model_executor/layers/layernorm.py`) +// GroupedRmsNorm <- `RMSNormGated` (:172), `group_size` (:187), +// grouped branch `forward_static` (:258-264): +// x.float() (:243), weight.float() (:244), +// rearrange "... (g d) -> ... g d", +// variance = mean(x_group^2), rsqrt(var + eps), +// out = flatten * weight. +// NOT the plain `RMSNorm` (:37): its only +// related knob is `var_hidden_size`, a PREFIX +// reduction that cannot express per-group norms. +// The gate (`z`) is disabled here; Qwen's +// hc_norm has none. +// +// ─── THE `1 + w` PARAMETERIZATION, RESOLVED EXPLICITLY ──────────────────────── +// The two oracles disagree on where the identity lives and this is the single +// most expensive detail in the module (issue #1988, trap 2): +// +// transformers: weight ZERO-init, out = normed * (1.0 + w_hf) +// vLLM: weight ONES-init, out = normed * w +// +// They coincide under `w = 1.0 + w_hf`, applied ONCE, at LOAD time. We mirror +// vLLM's op form — the kernel multiplies by `w` and knows nothing about the +// offset — and give the transform one and only one home, `HcNormWeightFromHf`, +// so that "did anyone already add the 1?" is a question about a single call site +// rather than about arithmetic buried in a kernel. It matters twice over: +// * Skip it and every `hc_norm` scales by ~0. That looks like a broken +// checkpoint, not like a broken port, and it costs a day. +// * Apply it twice and the scale is ~2x. The published GGUF has the fold done +// at CONVERT time, so a GGUF reader must NOT call this function. +// +// ─── THE GGUF FOLD IS A BLANKET RULE, NOT AN `hc_norm` RULE ─────────────────── +// Read this before writing a loader. The sentence above is true and, stated on +// its own, it teaches the wrong lesson: a reader who skips the fold for +// `hc_norm` alone will DOUBLE-FOLD every other norm in the same file, which is +// the same silent ~2x defect moved one tensor to the left. +// +// The fold does not live in a qwen4exp code path at all. Every anchor below is +// read at our recorded llama.cpp pin, stock upstream tag `b10451` +// (`10bf611e533d81f739128304991c5e133c6aebd8`, `.agents/oracles/llama-cpp.md`). +// Stock upstream has no `qwen4exp` whatsoever there: `git grep -il qwen4exp` +// returns nothing tree-wide, so a released llama.cpp can neither convert nor +// load this architecture. The converter is ggml-org/llama.cpp PR #27742, head +// `035e22731a7fd70b9854b3a2d64ec68e9b1a45d3`, still OPEN. Its +// `conversion/qwen4exp.py` declares `class Qwen4ExpTextModel(_Qwen35MRopeMixin, +// _LinearAttentionVReorderBase)`; `_LinearAttentionVReorderBase` is +// `conversion/qwen.py:438`, a subclass of `Qwen3NextModel` (`:365`, whose own +// signature is `class Qwen3NextModel(_QwenMtpMixin, Qwen2MoeModel)`), and there +// is NO `hc_norm` branch in the PR's `modify_tensors` — `hc_norm.weight` falls +// through to `super()`. The `+1` is the INHERITED Qwen3-Next rule, at +// `conversion/qwen.py:387-388`: +// +// elif name.endswith("norm.weight") and not name.endswith("linear_attn.norm.weight"): +// data_torch = data_torch + 1 +// +// So the rule a loader has to implement is: **every tensor whose name ends in +// `norm.weight` carries the fold, with `linear_attn.norm.weight` (the GDN +// `ssm_norm`) as the one exception.** `hc_norm`, `attn_q_norm` and `attn_k_norm` +// all match it. The PLE and indexer gammas — `.ple.norm_{key,query,conv}` and +// `.indexer.{q,k}_layernorm` — are folded too, by the PR's own branch, which +// returns early so they are never folded twice. +// +// This is a property of ONE IN-FLIGHT CONVERTER and not of "GGUF". #27742 is +// unmerged and can change before it lands, and a publisher using a different +// tool is under no obligation to match it. A loader must therefore treat the +// fold as a checkpoint-provenance question, not a format constant; the check is +// cheap, because an unfolded `hc_norm` is a ZERO-init gamma and a folded one is +// centred on 1.0. Corroborated on published artifacts during fresh review of +// #1988 (`unsloth/Qwen3.8-Flash-Next-GGUF` `UD-IQ1_S` and `UD-Q4_K_XL`, +// `vumpt/...-Q4_K_M`, read by HTTP range request against the bf16 HF tensors): +// every `*hc_norm.weight` is HF + 1.0 exactly, elementwise, while `ssm_norm` is +// not folded and sits in [0.875, 1.023]. +// +// ─── PRECISION ──────────────────────────────────────────────────────────────── +// fp32 interior throughout, with the per-group sum of squares accumulated in +// double — the `deepseek_v4_mhc.cpp` house convention for a host reference. +// Upstream runs the norm in fp32 (`self._norm(x.float())`) and vLLM likewise +// (`x = x.float()`), so nothing here is a widening of the model path: this is a +// CPU reference and the device arm is the thing that must be fp32-accumulate and +// gated against these numbers. No `f32` model-path buffer is introduced. +// +// ─── SCOPE ──────────────────────────────────────────────────────────────────── +// Host reference only, and NOT YET REACHED from a production entry point: the +// W1 config/registration wave (#1986) is still in review, so there is no +// `qwen4_exp` loader row for this to hang off. Named per AGENTS.md "Nothing +// lands dead": the wiring is owed by row `MODEL-MM-QWEN4-EXP` W5 (assembly) +// under issue #1978, and it is listed in the spec's `## Owed`. +#pragma once + +#include +#include + +namespace vllm::qwen4_exp { + +// `w_vllm = 1.0 + w_hf`, elementwise. See the parameterization note above. Call +// this exactly once, at load, on a HuggingFace `hc_norm.weight`. Never on a +// weight read out of a GGUF written by ggml-org/llama.cpp#27742, which has the +// fold applied already — and note that the SAME is true of `attn_q_norm`, +// `attn_k_norm` and the PLE and indexer gammas in that file, while `ssm_norm` is +// the one exception. That rule is spelled out above; skipping the fold for +// `hc_norm` alone double-folds everything else. +std::vector HcNormWeightFromHf(const std::vector& w_hf); + +// Grouped RMSNorm over one token's `x`, in vLLM's `RMSNormGated(group_size=)` +// form with the gate disabled: `x.size() / group_size` INDEPENDENT reductions, +// eps inside the rsqrt, weight applied per element after the normalize. +// +// out[g*G + d] = x[g*G + d] * rsqrt(mean_d(x[g*G + .]^2) + eps) * weight[g*G + d] +// +// Refuses an indivisible width the way `Qwen4ExpTextRMSNorm.__init__` (:164-165) +// does, and refuses a weight that is not the full flattened width. +std::vector GroupedRmsNorm(const std::vector& x, const std::vector& weight, + int64_t group_size, float eps); + +// The four `Qwen4ExpTextGatedResidual` parameters, all bias-free `nn.Linear` +// weights in PyTorch's `(out_features, in_features)` row-major layout. +struct GatedResidualWeights { + std::vector hc_norm_weight; // [hc*H] — vLLM form, i.e. 1 + w_hf + std::vector mix_down; // [R, hc*H] + std::vector mix_up; // [hc*H, R] + // EMPTY is `use_combine=False`, i.e. `block_inject_weight = None`. That arm is + // the model's FINAL mixer: `Qwen4ExpTextModel` holds one of these and there is + // NO trailing RMSNorm after it — `hc_norm` is the last normalization before + // `lm_head` (issue #1988, trap 1). + std::vector block_inject; // [hc, hc*H] or empty +}; + +struct GatedResidualResult { + std::vector mixed_input; // [H] — the block's input + std::vector hyper_input_normed; // [hc*H] — what the mean multiplies + std::vector injection_weights; // [hc] — empty on the use_combine=false arm +}; + +// One token of `Qwen4ExpTextGatedResidual.forward`. `hyper_input` is [hc*H] and +// is NOT modified: upstream returns it RAW, un-normed, and it is the raw stream +// that the write-back adds to. +// +// The two divisions by `hc` are different and both load-bearing: +// * inside the SiLU, on the [R] low-rank intermediate, BEFORE the activation — +// `silu(down(x) / hc)`, not `silu(down(x)) / hc`. SiLU is not homogeneous. +// * inside the injection sigmoid, with the whole sigmoid scaled by 2 — +// `2 * sigmoid(inject(x) / hc)`, range (0, 2), exactly 1.0 at a zero logit. +// There is NO division on the up-projection sigmoid, and the reduce over `hc` is +// a MEAN, not a sum. +GatedResidualResult GatedResidualForward(const std::vector& hyper_input, + const GatedResidualWeights& weights, int64_t hc, + int64_t hidden, float eps); + +// The rank-1 write-back, in place: +// +// hyper[j*H + h] += block_out[h] * injection_weights[j] +// +// THIS IS THE FUSION SEAM, and it is deliberately the primitive rather than the +// convenience. Upstream spells it `hidden_states.unsqueeze(-2) * +// injection_weights.unsqueeze(-1)` then `.flatten(-2)`, and both llama.cpp +// implementations of this architecture materialise it as a `repeat_4d` + `mul`, +// i.e. a dense [H, hc, T] broadcast built and thrown away 96 times per forward +// pass at 48 layers x 2 sites. It is a rank-1 update. A device arm reads +// `block_out` once per (j, h) tile and `injection_weights[j]` once per row; it +// never allocates the broadcast. Keeping the in-place pointer form as the +// primitive means the fused kernel replaces exactly this function and nothing +// above it has to change. +void GatedResidualWriteBackInPlace(float* hyper, const float* block_out, + const float* injection_weights, int64_t hc, int64_t hidden); + +// Allocating wrapper over `GatedResidualWriteBackInPlace`, for tests and for the +// host path. Bit-identical to the in-place form by construction. +std::vector GatedResidualWriteBack(const std::vector& hyper_input, + const std::vector& block_out, + const std::vector& injection_weights, int64_t hc, + int64_t hidden); + +} // namespace vllm::qwen4_exp diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index fc02e045d..27537f2fa 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -3114,6 +3114,16 @@ vllm_cpp_add_test(test_qwen4_exp_qsa vllm/models/test_qwen4_exp_qsa.cpp) # (#515). Same arrangement test_dots3_note_attn uses. target_include_directories(test_qwen4_exp_qsa PRIVATE ${CMAKE_SOURCE_DIR}/src) +# MODEL-MM-QWEN4-EXP W3 (#1988) — the 4-branch GATED-RESIDUAL hyper-connection +# stream and its grouped RMSNorm, against goldens dumped by EXECUTING the +# lane-pinned oracle source (transformers v5.16.0 `modeling_qwen4_exp.py`, the +# two classes lifted verbatim by line range) plus an independent +# double-precision reference. Host reference; no arm of this model runs yet, so +# no token and no speed claim. `qwen4_exp_hc.h` is a MODEL-PRIVATE header under +# src/ (the `dots3_note.h` precedent), hence the explicit src/ grant. +vllm_cpp_add_test(test_qwen4_exp_hc vllm/models/test_qwen4_exp_hc.cpp) +target_include_directories(test_qwen4_exp_hc PRIVATE ${CMAKE_SOURCE_DIR}/src) + # The five suites that reach src/vllm/support/platform_compat.h, granted per # target rather than globally: the file already carries 123 explicit # ${CMAKE_SOURCE_DIR}/src grants, and a blanket one in vllm_cpp_add_test would diff --git a/tests/support/max_abs_diff.h b/tests/support/max_abs_diff.h index d81d995a3..b6a413435 100644 --- a/tests/support/max_abs_diff.h +++ b/tests/support/max_abs_diff.h @@ -28,6 +28,18 @@ // EXACT comparison, not through here. A future golden that genuinely needs a // non-finite value takes an explicit, documented carve-out at its own call site; // it never relaxes this helper. +// +// WHY THESE ARE TEMPLATES. A gate that compares an fp32 result against a +// DOUBLE-precision in-test reference — the shape every "second independent +// reference" case takes — cannot call a float-only helper, and the observed +// consequence is that it writes form B again locally. That happened on +// `test_qwen4_exp_hc.cpp` (#1988): two local `std::max(worst, std::abs(...))` +// helpers, in a file that already included this header, and a mutation that +// poisoned one path to all-NaN left its `< tol` assertion GREEN. The reduction +// is therefore templated on both operand types rather than duplicated, so the +// double-sided comparisons route through the SAME hardened scan. Mixed +// float/double operands are compared in double; the finiteness rule and its +// polarity are unchanged. #pragma once #include @@ -48,13 +60,16 @@ struct MaxAbsDiffScanResult { double worst = 0.0; // Index of the FIRST non-finite operand, or kNone when both sides are finite. size_t bad_index = kNone; - float bad_got = 0.0f; - float bad_want = 0.0f; + // Held as double so a double-sided comparison reports the value it actually + // saw; a float operand widens exactly. + double bad_got = 0.0; + double bad_want = 0.0; bool ok() const { return bad_index == kNone; } }; -inline MaxAbsDiffScanResult MaxAbsDiffScan(const float* got, const float* want, size_t count) { +template +inline MaxAbsDiffScanResult MaxAbsDiffScan(const G* got, const W* want, size_t count) { MaxAbsDiffScanResult r; double worst = 0.0; for (size_t i = 0; i < count; ++i) { @@ -77,7 +92,8 @@ inline MaxAbsDiffScanResult MaxAbsDiffScan(const float* got, const float* want, // The doctest-facing helper the model gates call. Fails loudly on a non-finite // operand and still returns +infinity, so a bound check either way reports red. -inline double MaxAbsDiff(const std::vector& got, const float* want, size_t count) { +template +inline double MaxAbsDiff(const std::vector& got, const W* want, size_t count) { REQUIRE(got.size() == count); const MaxAbsDiffScanResult r = MaxAbsDiffScan(got.data(), want, count); if (!r.ok()) { @@ -88,7 +104,8 @@ inline double MaxAbsDiff(const std::vector& got, const float* want, size_ return r.worst; } -inline double MaxAbsDiff(const std::vector& a, const std::vector& b) { +template +inline double MaxAbsDiff(const std::vector& a, const std::vector& b) { REQUIRE(a.size() == b.size()); return MaxAbsDiff(a, b.data(), b.size()); } diff --git a/tests/support/test_max_abs_diff.cpp b/tests/support/test_max_abs_diff.cpp index cbc815f14..6cccd3e2d 100644 --- a/tests/support/test_max_abs_diff.cpp +++ b/tests/support/test_max_abs_diff.cpp @@ -151,6 +151,54 @@ TEST_CASE("max|diff| scan: -FLT_MAX is FINITE and must still compare") { CHECK(moved.worst > 1e38); } +// The DOUBLE-SIDED instantiation, which is the shape the helper was templated +// for (#1988): an fp32 golden against a `std::vector` in-test reference. +// Before this case the widened templates were exercised only by +// `test_qwen4_exp_hc`, so the helper's own suite could not see a regression in +// them. Two properties are gated here: the comparison runs in DOUBLE rather than +// narrowing the reference to float, and the finiteness rule holds on the double +// side exactly as it does on the float side. +TEST_CASE("max|diff| scan: a vector reference against an fp32 golden") { + const std::vector exact(kWant, kWant + kN); + const vllm_test::MaxAbsDiffScanResult same = vllm_test::MaxAbsDiffScan(exact.data(), kWant, kN); + CHECK(same.ok()); + CHECK(same.worst == 0.0); + + // A difference that EXISTS in double and vanishes in float. 1.0 + 2^-30 is not + // representable in fp32 (whose spacing at 1.0 is 2^-23), so an implementation + // that narrowed the double operand would report 0.0 here and pass every bound. + const float one_golden[] = {1.0f, 1.0f, 1.0f, 1.0f}; + std::vector tiny(kN, 1.0); + tiny[2] = 1.0 + std::ldexp(1.0, -30); + const vllm_test::MaxAbsDiffScanResult moved = + vllm_test::MaxAbsDiffScan(tiny.data(), one_golden, kN); + CHECK(moved.ok()); + CHECK(moved.worst == std::ldexp(1.0, -30)); + + // The finiteness rule, on the DOUBLE side. A double NaN is not a zero either, + // and the reported operand keeps its double value rather than a float image. + std::vector poisoned(kN, 1.0); + poisoned[1] = std::numeric_limits::quiet_NaN(); + const vllm_test::MaxAbsDiffScanResult bad = + vllm_test::MaxAbsDiffScan(poisoned.data(), one_golden, kN); + CHECK_FALSE(bad.ok()); + CHECK(bad.bad_index == 1); + CHECK(std::isnan(bad.bad_got)); + CHECK(std::isinf(bad.worst)); + CHECK_FALSE(bad.worst < 1e-5); +} + +TEST_CASE("max|diff| wrapper: vector against an fp32 golden") { + const float one_golden[] = {1.0f, 1.0f, 1.0f, 1.0f}; + std::vector tiny(kN, 1.0); + tiny[0] = 1.0 + std::ldexp(1.0, -30); + // The `const W*` overload, the shape test_qwen4_exp_hc.cpp:569 calls. + CHECK(vllm_test::MaxAbsDiff(tiny, one_golden, kN) == std::ldexp(1.0, -30)); + // ...and the two-vector overload with mixed element types. + const std::vector want(one_golden, one_golden + kN); + CHECK(vllm_test::MaxAbsDiff(tiny, want) == std::ldexp(1.0, -30)); +} + // Named so that the `*RAISES*` CTest filter below matches exactly one case — // doctest's name matching is case-INSENSITIVE. TEST_CASE("max|diff| wrapper: a finite comparison reports nothing") { diff --git a/tests/vllm/models/qwen4_exp_hc_goldens.inc b/tests/vllm/models/qwen4_exp_hc_goldens.inc new file mode 100644 index 000000000..3111bb205 --- /dev/null +++ b/tests/vllm/models/qwen4_exp_hc_goldens.inc @@ -0,0 +1,419 @@ +// GENERATED by scripts/gen-qwen4-exp-hc-goldens.py -- do not hand-edit. +// Oracle: transformers v5.16.0 modeling_qwen4_exp.py +// sha256 77fec77d87f2a0eb23b95fa04276fb5779698a7c7f523cf5061e49c118bcc459 +// Qwen4ExpTextRMSNorm :158-181 (grouped RMSNorm, (1.0 + weight)) +// Qwen4ExpTextGatedResidual :941-969 +// write-back: Qwen4ExpTextDecoderLayer.forward, the two lines +// injection = hidden_states.unsqueeze(-2) * injection_weights.unsqueeze(-1) +// hidden_states = hyper_input + injection.flatten(-2) +// torch 2.11.0+cu130 + +// ---- A: hidden=6 hc=4 lowrank=5 eps=1e-06 T=3 use_combine=True seed=1234 +const float kA_norm_w_hf[] = { + -0.0558592826f, -0.248295054f, 0.0815368518f, -0.44084388f, + 0.0269500967f, 0.334186852f, -0.0298288036f, -0.233748972f, + 0.318456322f, -0.357040048f, -0.5415622f, -0.277362108f, + 0.485842496f, -0.257504582f, 0.712763309f, 0.399342716f, + -1.26366758f, 0.738892138f, -0.0848118141f, -0.495928675f, + -0.728454232f, 0.128145874f, -0.201523483f, 0.209763691f, +}; +const float kA_down[] = { + 0.230009928f, 0.0057106209f, 0.00659922976f, 0.345959991f, + -0.101799741f, 0.0467645898f, 0.268991739f, -0.0890373439f, + -0.205710724f, -0.0148911141f, -0.374550611f, -0.255257607f, + -0.230697557f, -0.468171865f, -0.159271434f, 0.0653252229f, + -0.0849997327f, -0.169785336f, 0.106985554f, -0.136046484f, + -0.0891406462f, -0.461391181f, -0.307454944f, -0.113418475f, + 0.117305949f, 0.154736325f, -0.301274389f, 0.29578951f, + 0.340006769f, 0.255127072f, 0.316016972f, 0.110753559f, + -0.0165582672f, -0.183763459f, 0.225003466f, -0.220376238f, + 0.138669804f, 0.352784842f, 0.0643412024f, 0.160859257f, + 0.0409469977f, -0.699949861f, 0.459235817f, 0.0803977698f, + 0.135144517f, -0.0817496553f, -0.521964252f, 0.0389798582f, + -0.168896139f, -0.0848820359f, 0.0219309255f, -0.416413009f, + -0.0803379416f, -0.0376101807f, -0.451138169f, -0.0986020789f, + 0.130802929f, -0.443111718f, -0.366648406f, -0.0823910087f, + -0.103497915f, -0.214860126f, 0.173432574f, 0.114157476f, + -0.436142772f, -0.802202046f, 0.479510695f, 0.24064137f, + -0.105343677f, -0.0201012976f, -0.016012691f, -0.249461859f, + 0.440670997f, 0.379409522f, -0.0825821906f, -0.0397612788f, + 0.0318639055f, 0.0277184229f, -0.35462603f, 0.0638813004f, + 0.0464846902f, -0.195784643f, 0.215386584f, -0.134854406f, + -0.215582252f, 0.205178604f, -0.103728168f, 0.0557277352f, + -0.0475083292f, 0.396788061f, -0.803893566f, -0.0389022827f, + -0.207587719f, 0.355906367f, 0.0823413432f, -0.101632081f, + -0.0367469974f, 0.242349729f, -0.342644632f, 0.61518997f, + -0.0456919149f, -0.221436858f, -0.215664506f, 0.238124669f, + -0.053115081f, -0.269426376f, -0.295032978f, 0.226913646f, + 0.104232438f, 0.0856718495f, -0.0148635367f, 0.186736494f, + -0.576501429f, -0.395285994f, 0.368593544f, 0.210738167f, + -0.0050911135f, -0.0848935172f, -0.193378344f, -0.00596537627f, +}; +const float kA_up[] = { + 0.0336867087f, -0.198038742f, -0.187293917f, -0.222491011f, + 0.102670431f, -0.0861795396f, -0.201774433f, 0.265300244f, + 0.216334134f, 0.066431731f, -0.0821184516f, 0.258368105f, + -0.0182846691f, 0.632193625f, -0.297900349f, 0.422407985f, + -0.238362685f, 0.18431586f, -0.0819076896f, 0.496816367f, + 0.418209016f, 0.181785226f, 0.0662629679f, -0.247354791f, + 0.218663722f, -0.220087618f, 0.468721628f, -0.306239724f, + -0.00526024215f, -0.275821298f, -0.281670183f, -0.582625389f, + 0.0698702261f, -0.330426067f, -0.374202728f, -0.224535272f, + -0.293771446f, 0.248553723f, -0.0750395954f, 0.0480682962f, + 0.218860701f, -0.133215174f, 0.246407941f, -0.180457219f, + 0.272075921f, 0.470730275f, -0.0332507603f, -0.0771899447f, + 0.16894193f, 0.0188689008f, -0.0191753581f, 0.908752322f, + 0.0557283759f, -0.0511085875f, 0.308112562f, 0.0211206935f, + 0.107589111f, -0.707808971f, -0.418630153f, 0.260241687f, + -0.0675459132f, 0.0357870571f, -0.323149085f, 0.0754813105f, + -0.0658725873f, 0.181845143f, 0.0141932517f, 0.00372157129f, + 0.113248877f, 0.219974011f, -0.147438213f, -0.331458807f, + -0.0117366239f, 0.012483703f, -0.314042419f, 0.0100491336f, + -0.316349119f, -0.0591964051f, 0.255663157f, -0.121967025f, + -0.423375368f, 0.00599367917f, -0.123458393f, -0.101948835f, + -0.166899547f, -0.19523038f, -0.0653577447f, 0.230989307f, + -0.299673647f, 0.333176434f, 0.00934083387f, 0.470875531f, + 0.461682051f, 0.0895232856f, 0.166822284f, -0.206049591f, + -0.331997365f, 0.680676579f, -0.675302148f, -0.303457767f, + 0.114846051f, 0.370880038f, -0.31603983f, -0.282106221f, + -0.30173409f, 0.113918141f, -0.0895575508f, -0.408483177f, + -0.0999869928f, -0.375807047f, -0.292377025f, -0.464005977f, + 0.103854716f, -0.379565507f, -0.636228383f, 0.459145993f, + -0.323322773f, -0.0955618247f, 0.683090091f, 0.116701543f, +}; +const float kA_inject[] = { + -0.470525771f, 0.403277755f, 0.0111542391f, -0.107938446f, + 0.0309650674f, -0.488039285f, 0.17185615f, -0.209858194f, + -0.642993629f, -0.162584186f, -0.12766321f, 0.543742359f, + -0.409083843f, 0.0920040607f, -0.30982241f, 0.4147183f, + 0.278410882f, 0.760603249f, 0.222951218f, -0.277991802f, + 0.324819773f, -0.303619057f, -0.0792483464f, 0.0768021792f, + -0.0304893199f, -0.109005488f, 0.331583977f, -0.273273081f, + -0.196809694f, -0.211460099f, -0.355280548f, -0.0793590546f, + -0.385707438f, -0.064223744f, 0.0311175603f, -0.121160097f, + -0.147312239f, -0.295749545f, -0.161240309f, -0.511619389f, + -0.274669468f, 0.301221073f, 0.122041345f, -0.201354802f, + 0.137796208f, -0.202428162f, -0.0490248278f, -0.210767046f, + -0.201934978f, -0.172478542f, 0.277232081f, 0.218289748f, + 0.187215894f, 0.280916899f, 0.231201351f, -0.0376391932f, + -0.246289238f, -0.25061211f, -0.0999569371f, -0.405950665f, + -0.012512004f, 0.22387664f, 0.0996676609f, -0.0557701588f, + 0.392104f, 0.436399937f, -0.0631143376f, -0.102441408f, + -0.109501801f, 0.147821859f, -0.175981492f, 0.238440841f, + -0.303731024f, -0.197148591f, -0.172388151f, -0.623645246f, + -0.0135014346f, 0.398366183f, -0.188274488f, -0.112167157f, + -0.314300239f, 0.0936004147f, -0.248159006f, -0.257808775f, + -0.119168296f, 0.126265466f, 0.0301423147f, -0.197634354f, + -0.35174799f, 0.0275507923f, -0.169687241f, -0.357006609f, + -0.283018708f, -0.245096996f, 0.102671057f, -0.324268609f, +}; +const float kA_hyper[] = { + 0.533237576f, -1.20873201f, 0.107799865f, -5.22493458f, + -1.51860964f, -0.425679505f, 0.346284419f, -1.10036111f, + 2.06948781f, 1.03521371f, -1.91126668f, 1.29653561f, + 1.37460244f, -1.56329715f, -1.99296105f, 0.720400214f, + -1.09137011f, 2.1534636f, -0.135816127f, -2.71129417f, + 1.59412038f, -0.772991061f, -0.512841046f, -0.183839917f, + 1.23042381f, -0.663249493f, 1.21584845f, 2.0912149f, + -0.206895769f, 1.31602156f, -4.23796225f, -0.811664701f, + 2.19035053f, -0.668166578f, -0.414703071f, -0.655608356f, + -0.163112f, 1.60746503f, -2.16032863f, 0.502987266f, + 0.669123352f, -0.106363147f, 0.973783314f, -0.429280818f, + -0.124404483f, -0.0254335124f, 0.246692955f, 0.343353271f, + -0.996848762f, -0.0774152055f, -1.28514361f, -1.98718035f, + -1.5961374f, 1.50561988f, 2.33991623f, -1.33068836f, + -0.0533250868f, 0.758986533f, 2.51541495f, -1.77328277f, + 1.35424542f, 0.787682176f, 0.898809254f, 0.158229485f, + 2.1792016f, -1.06922519f, 1.23619056f, -2.43089294f, + -0.455594748f, 1.3866334f, 0.350418687f, -1.19509685f, +}; +const float kA_normed[] = { + 0.219569743f, -0.396271139f, 0.050848078f, -1.27417457f, + -0.680159092f, -0.247693375f, 0.237367317f, -0.59572506f, + 1.92782748f, 0.470276743f, -0.619072139f, 0.661979795f, + 1.30696869f, -0.742763638f, -2.18429613f, 0.645079553f, + 0.184138298f, 2.39621353f, -0.0926238596f, -1.01842678f, + 0.32257098f, -0.649831653f, -0.305144876f, -0.165730134f, + 0.91935569f, -0.394563079f, 1.04066956f, 0.925387979f, + -0.168148518f, 1.3895421f, -2.03614354f, -0.307999939f, + 1.4301523f, -0.212750986f, -0.0941500589f, -0.234621674f, + -0.210025251f, 1.0343039f, -3.20649099f, 0.609949589f, + -0.152888834f, -0.160279006f, 1.89472055f, -0.460050941f, + -0.0718210042f, -0.0610019937f, 0.418785185f, 0.883108616f, + -0.682352483f, -0.042190671f, -1.00771046f, -0.805588245f, + -1.18839931f, 1.45638013f, 1.33703506f, -0.600539446f, + -0.0414086804f, 0.287416756f, 0.679178834f, -0.754732132f, + 1.62680888f, 0.472836733f, 1.24460423f, 0.179010272f, + -0.464537263f, -1.50317037f, 0.831080019f, -0.900129199f, + -0.090880096f, 1.14914262f, 0.205540046f, -1.06206393f, +}; +const float kA_mixed[] = { + 0.218683407f, -0.342433631f, -0.0115971975f, -0.096745424f, + -0.165759042f, 0.317727774f, 0.0970179737f, -0.0145416558f, + -0.0814772174f, 0.160644844f, -0.00374818593f, 0.230426177f, + 0.389982581f, -0.14256005f, 0.00486969389f, 0.0945343599f, + -0.0895517245f, -0.208840132f, +}; +const float kA_block_out[] = { + 0.519128263f, -1.07747173f, -0.256617546f, -0.335337877f, + -1.49071062f, -1.27359354f, -0.170700669f, 1.6074369f, + -0.0265056677f, -0.220163777f, -0.416405171f, -1.34811532f, + -0.362151444f, -0.151884809f, -0.656513393f, -2.1005435f, + 0.512117267f, -0.332021594f, +}; +const float kA_inj_w[] = { + 1.25132978f, 1.10185194f, 0.939499736f, 1.02238166f, + 0.899588108f, 1.01564634f, 0.986539185f, 0.906849504f, + 0.666291654f, 0.847988546f, 0.959581435f, 1.20329559f, +}; +const float kA_written[] = { + 1.1828382f, -2.55700445f, -0.213313311f, -5.64455271f, + -3.38398027f, -2.01936507f, 0.91828692f, -2.28757548f, + 1.78673327f, 0.665721059f, -3.55380917f, -0.10677588f, + 1.86232328f, -2.57558155f, -2.23405313f, 0.405350357f, + -2.49189234f, 0.95692277f, 0.394931108f, -3.81288147f, + 1.33175933f, -1.11583436f, -2.03691626f, -1.48593855f, + 1.07686353f, 0.782781601f, 1.1920042f, 1.8931582f, + -0.581488907f, 0.103273034f, -4.41133356f, 0.820922732f, + 2.16343021f, -0.891775131f, -0.837623477f, -2.02481675f, + -0.331514895f, 3.19326448f, -2.18647742f, 0.285787076f, + 0.258323342f, -1.43633175f, 0.818983495f, 1.02842259f, + -0.148441136f, -0.225088924f, -0.130923867f, -0.879184484f, + -1.23814726f, -0.178614795f, -1.72257304f, -3.38675499f, + -1.25491798f, 1.28439665f, 2.03281593f, -1.45948493f, + -0.610040903f, -1.02225018f, 2.94968462f, -2.05483317f, + 1.00673163f, 0.641936302f, 0.268831193f, -1.85741317f, + 2.67061973f, -1.38782692f, 0.800415337f, -2.61365533f, + -1.24557447f, -1.14094138f, 0.966647148f, -1.59461701f, +}; + +// ---- B: hidden=5 hc=3 lowrank=7 eps=1e-05 T=2 use_combine=True seed=99 +const float kB_norm_w_hf[] = { + 0.30634293f, -0.587676823f, -0.382324636f, -0.333282828f, + 0.372182995f, -0.322658688f, -0.694513917f, -0.136483803f, + 0.470299363f, -1.30883467f, -0.475572467f, -0.109824248f, + 0.0838670358f, -0.00266977097f, -1.04641449f, +}; +const float kB_down[] = { + 0.0964841321f, 0.255086482f, 0.194116309f, 0.388963997f, + 0.33462137f, -0.302849412f, 0.186166987f, 0.136263818f, + 0.252059072f, -0.605638325f, -0.118919484f, -0.15502046f, + -0.700868011f, 0.210165933f, -0.0841370299f, 0.436674327f, + -0.627836585f, -0.21424897f, 0.141523972f, 0.584608614f, + 0.0808142722f, -0.402946323f, -0.125655219f, -0.118344508f, + 0.548009813f, 0.237366602f, -0.459965914f, -0.260818362f, + -0.0895831659f, -0.229434088f, 0.257640928f, -0.0247417837f, + 0.0624738671f, -0.0466020517f, -0.419715255f, -0.109153494f, + 0.143075585f, -0.058244545f, 0.119965412f, -0.240623817f, + 0.0215703472f, -0.403630406f, 0.435768038f, 0.236045897f, + -0.115460597f, -0.196635619f, -0.435137004f, -0.36175251f, + 0.507629991f, 0.52901876f, -0.03106484f, -0.522266626f, + -0.132427171f, -0.23122558f, 0.442002177f, -0.180983216f, + -0.101839982f, 0.354686797f, -0.458208293f, -0.381291807f, + -0.255328417f, 0.278687924f, -0.425551474f, -0.176636606f, + 0.15925926f, 0.316434145f, -0.258282274f, -0.187034011f, + -0.387811422f, -0.189466402f, -0.324946284f, 0.0134268887f, + 0.23262842f, -0.450703979f, 0.318678319f, -0.307124197f, + 0.184222028f, 0.033916451f, 0.241243362f, -0.175679237f, + 0.506585538f, -0.382096678f, 0.0134525402f, -0.308507323f, + 0.708951771f, -0.421255499f, -0.207631126f, -0.0347730294f, + -0.166764975f, -0.258377284f, -0.700190663f, -0.453181237f, + 0.208763883f, -0.171336681f, -0.0689195171f, 0.195338294f, + -0.0539493635f, -0.107879616f, -0.0958281383f, 0.467185408f, + -0.0806179717f, -0.202732861f, -0.0514242128f, -0.0472091548f, + -0.0386718884f, +}; +const float kB_up[] = { + 0.343490094f, 0.0721287504f, 0.185548693f, -0.17879653f, + -0.167254314f, -0.244209081f, 0.406540245f, 0.0789460614f, + -0.0533151589f, 0.114840999f, 0.21510464f, 0.0884370133f, + -0.0665651187f, -0.108309262f, -0.0655949786f, 0.573655665f, + -0.342038631f, 0.0259813108f, -0.0974591896f, 0.142308608f, + 0.19310008f, 0.139679015f, -0.0349669829f, -0.462970614f, + 0.0544645377f, 0.0513722971f, 0.230093256f, -0.0984883606f, + -0.347496063f, -0.255700141f, 0.191816881f, 0.036785312f, + 0.126049876f, 0.22464405f, 0.245288566f, -0.358890563f, + 0.545955479f, 0.36748749f, 0.0152934687f, 0.198187605f, + 0.367234081f, -0.281228781f, 0.0930237472f, -0.380721033f, + 0.0879324526f, -0.176725745f, -0.525668383f, 0.210080102f, + -0.748570025f, -0.0751311556f, 0.208635211f, 0.0851098001f, + 0.278763115f, -0.357726544f, -0.338402867f, 0.20990704f, + 0.495543897f, 0.255115837f, 0.0740242749f, 0.536688864f, + -0.3042005f, -0.204800203f, 0.160633355f, -0.508065045f, + -0.569782019f, -0.230778813f, -0.511960268f, -0.304472923f, + -0.166226864f, -0.0940455198f, -0.0832107216f, -0.635292888f, + 0.440075874f, -0.307712018f, -0.00636816584f, 0.216251418f, + 0.425538749f, 0.13552317f, 0.168148905f, 0.206026599f, + 0.481180012f, 0.0192144271f, 0.301048934f, 0.0327274986f, + -0.595184863f, 0.0550124943f, -0.187376693f, 0.0152243599f, + 0.325533718f, 0.233476982f, 0.236022383f, -0.0777259022f, + 0.0910165906f, 0.0486684665f, 0.197047994f, -0.228946328f, + -0.0614259019f, -0.0119054811f, -0.27952826f, -0.175241813f, + 0.392222583f, 0.537081897f, -0.0274822358f, 0.158765569f, + 0.315507412f, +}; +const float kB_inject[] = { + -0.099082835f, -0.0356387049f, -0.239228144f, -0.353755474f, + 0.0235928036f, 0.213960662f, -0.329626769f, -0.060693007f, + 0.240905926f, -0.544646084f, 0.142132103f, -0.264545947f, + 0.38622573f, 0.241344586f, -0.165297106f, 0.157298028f, + -0.00542231742f, 0.389258742f, -0.26785627f, 0.407752007f, + 0.214632183f, 0.334532082f, -0.154605836f, -0.10579019f, + -0.268874347f, 0.163597748f, -0.101839714f, 0.151244223f, + 0.265165776f, -0.169093236f, 0.243119746f, 0.181456268f, + -0.0887573957f, 0.315174669f, 0.308018744f, -0.107801668f, + -0.0438321307f, 0.198384956f, -0.422011763f, -0.301602066f, + 0.271917701f, -0.528458297f, 0.231092364f, 0.353875756f, + -0.0754795745f, +}; +const float kB_hyper[] = { + -0.0926936939f, -0.165322334f, -0.328608423f, -0.52173382f, + -2.85028839f, 0.375549883f, -2.49926162f, 1.3132093f, + -2.36222649f, 0.339655042f, 0.381446481f, -0.609766603f, + -1.00423896f, 0.0982440189f, -2.66434526f, -1.24888241f, + -0.502287209f, -1.04099572f, 0.30450502f, 0.308207721f, + 1.56670952f, 0.343868107f, 0.715105891f, -0.84744972f, + -3.48773026f, -0.86844635f, 1.05246472f, 3.19339752f, + 1.46291411f, -0.323694646f, +}; +const float kB_normed[] = { + -0.0926521868f, -0.0521575958f, -0.155305654f, -0.266157627f, + -2.99260306f, 0.15307501f, -0.459443659f, 0.682391405f, + -2.09004879f, -0.0631238073f, 0.152227014f, -0.413058937f, + -0.828296304f, 0.0745620355f, 0.0941058844f, -2.0775249f, + -0.263728648f, -0.818797827f, 0.25852561f, 0.538546264f, + 0.593845844f, 0.0587842502f, 0.345556051f, -0.697264194f, + 0.602762997f, -0.269262224f, 0.553898573f, 2.04633141f, + 0.862590671f, 0.00888251793f, +}; +const float kB_mixed[] = { + 0.0409843884f, -0.154900193f, -0.0681974888f, -0.337243527f, + -0.542721152f, -0.266406178f, 0.0658957139f, 0.320129067f, + 0.0727220923f, 0.19605504f, +}; +const float kB_block_out[] = { + 0.162516147f, 1.08877027f, 0.900735974f, -0.52850914f, + -0.586997807f, -1.7903043f, 0.215952665f, -1.42978597f, + -0.603830397f, -0.470523119f, +}; +const float kB_inj_w[] = { + 0.926605344f, 0.792011797f, 1.01729059f, 1.1219033f, + 0.991320074f, 1.04880357f, +}; +const float kB_written[] = { + 0.0578946397f, 0.843538046f, 0.5060184f, -1.01145315f, + -3.39420366f, 0.504264593f, -1.63694274f, 2.02660275f, + -2.78081203f, -0.125254154f, 0.546772599f, 0.497829199f, + -0.0879287124f, -0.439403355f, -3.26149273f, -3.25743055f, + -0.260009199f, -2.64507723f, -0.372934312f, -0.219673723f, + -0.208055019f, 0.557946324f, -0.702269673f, -1.44603896f, + -3.95416927f, -2.74612379f, 1.27895665f, 1.69383287f, + 0.829614639f, -0.817180991f, +}; + +// ---- C: hidden=6 hc=4 lowrank=5 eps=1e-06 T=2 use_combine=False seed=7 +const float kC_norm_w_hf[] = { + -0.41006726f, 0.197815582f, 0.449454248f, -0.694201946f, + -0.0834980085f, 0.142574981f, -0.320545763f, -0.44682771f, + -0.424454302f, -0.608464718f, -0.0405107923f, 0.680742562f, + -0.651608348f, 0.0437391885f, -0.692480683f, 0.25178194f, + -0.433350801f, -0.282003343f, -1.02823126f, -0.178361267f, + 0.547790229f, 0.241177082f, 0.699459195f, 0.702926457f, +}; +const float kC_down[] = { + -0.0406310707f, 0.224135473f, 0.183544725f, 0.5603351f, + 0.753480911f, -0.376435578f, 0.244936958f, -0.319619685f, + -0.491106421f, 0.0473135747f, 0.118722282f, -0.410321921f, + -0.0302202962f, 0.0711043775f, 0.189804628f, -0.0275031514f, + -0.114634782f, -0.350068241f, 0.10907276f, -0.0946923345f, + -0.159623668f, 0.0897425562f, -0.426133364f, -0.0490364656f, + 0.20703733f, -0.0432785526f, 0.483614028f, 0.470137149f, + -0.281651765f, -0.371090591f, -0.0187702198f, 0.134852827f, + 0.464885741f, 0.179670006f, -0.191300347f, -0.685745656f, + -0.110299252f, -0.264654309f, 0.163803622f, 0.0445536375f, + -0.226696074f, 0.117523648f, 0.224094257f, 0.413931698f, + 0.386315972f, 0.260531425f, -0.41465956f, -0.288968593f, + 0.123151451f, 0.530249238f, -0.0249568112f, 0.152611747f, + 0.33534354f, 0.338571399f, 0.0424795523f, -0.163746893f, + 0.346262723f, -0.46097061f, -0.167305231f, -0.131500423f, + 0.347156048f, 0.0266828649f, 0.0797621235f, -0.0572044179f, + -0.22258766f, -0.191264898f, -0.473584116f, 0.193800792f, + 0.0226887427f, -0.045759052f, 0.0587491132f, -0.470626235f, + -0.206827581f, -0.0148539627f, -0.367177248f, 0.192081794f, + 0.205183297f, 0.483970195f, 0.279949456f, -0.293148041f, + -0.115149453f, -0.0802590922f, 0.203814164f, -0.118406326f, + 0.398153782f, 0.0331896394f, 0.101221211f, -0.096043095f, + -0.389389396f, -0.236175284f, 0.522656798f, 0.176406369f, + 0.0826550126f, -0.280092716f, -0.0458217785f, -0.158988968f, + 0.190192476f, 0.301090837f, -0.656415582f, -0.297816485f, + 0.266549796f, -0.446002662f, -0.266949981f, 0.270158857f, + 0.20464319f, -0.308050931f, -0.0237137731f, 0.0431467891f, + 0.420761615f, -0.125009f, 0.0496361405f, -0.214707822f, + -0.202037096f, -0.213739291f, -0.207309023f, 0.0693440288f, + -0.226673052f, -0.0359355621f, 0.286342263f, 0.0447572842f, +}; +const float kC_up[] = { + -0.141083673f, -0.254465073f, 0.245550364f, 0.0618241392f, + 0.210331157f, -0.207286909f, 0.0876560062f, 0.438380867f, + 0.0921838433f, 0.227796912f, 0.459998161f, -0.232636064f, + -0.645309031f, -0.383894295f, -0.296743751f, -0.00131497451f, + -0.10726504f, 0.0559139587f, -0.349616796f, 0.265466481f, + 0.38530001f, -0.251042336f, -0.428089052f, -0.0804627314f, + -0.790298283f, -0.00693282904f, 0.221626207f, 0.198762387f, + 0.225792453f, 0.172736347f, 0.0203587636f, -0.0294060465f, + 0.183083028f, -0.27174899f, -0.264219195f, -0.258202434f, + -0.202312738f, -0.906178772f, 0.120427661f, -0.31809479f, + -0.339675665f, -0.550266445f, -0.157941684f, 0.344351232f, + -0.0843541771f, -0.207026422f, 0.444297135f, -0.0143096251f, + 0.141363353f, 0.194367304f, 0.399482399f, -0.0885263309f, + -0.00430287607f, -0.172600865f, 0.119755954f, -0.149922773f, + -0.386173874f, -0.0570468642f, -0.0244993381f, -0.25560981f, + -0.125611126f, 0.102468662f, -0.0725202113f, 0.386872202f, + -0.215037942f, 0.258619815f, -0.130791947f, -0.145139575f, + -0.188357875f, 0.602519155f, 0.226324424f, 0.0191508532f, + 0.187765375f, -0.078549698f, -0.123748176f, 0.0292070229f, + 0.2364171f, -0.242859244f, 0.0279372595f, -0.0901294649f, + 0.00190478773f, -0.451937228f, -0.187343627f, 0.135894462f, + 0.243344992f, 0.832985222f, 0.229062229f, -0.0537902117f, + -0.426362664f, -0.313000292f, -0.0858951062f, 0.235061765f, + 0.0489826053f, -0.253047436f, 0.22458221f, 0.0365615226f, + -0.0446157195f, 0.441168159f, 0.0601178966f, -0.20807749f, + 0.180020928f, -0.226440027f, 0.487573326f, 0.539725423f, + -0.150293678f, -0.183902428f, -0.108619519f, 0.464962065f, + -0.670442641f, 0.316658169f, 0.125152096f, 0.601781487f, + 0.195903376f, 0.800311983f, 0.210899025f, 0.30417484f, + 0.217664585f, 0.210095271f, -0.236560598f, 0.0700418428f, +}; +const float kC_hyper[] = { + -0.230136275f, -1.45792675f, -1.91336703f, 4.40088654f, + 0.175186977f, 1.52687156f, -2.71076894f, -1.03763235f, + -0.86155194f, 1.77705753f, -2.43255424f, 1.65857899f, + 0.738127708f, 3.41089416f, 1.9053688f, 0.119107477f, + 1.44350028f, -1.10254669f, -2.77361345f, 2.10040307f, + 0.59310782f, 0.300421387f, 1.25117958f, -0.693412304f, + 1.34150624f, 0.328173578f, -0.499845415f, 1.92204094f, + -0.120057032f, -0.353418171f, -1.14711487f, 2.3258779f, + -0.600307107f, -0.16371116f, 2.94823527f, 1.17002559f, + -0.981115401f, 2.0027864f, -0.965839982f, 0.683383763f, + 0.727447212f, -2.1764214f, 0.629573762f, 2.55347562f, + 0.392030865f, -2.98006725f, 0.911427855f, -1.61496496f, +}; +const float kC_normed[] = { + -0.0633359328f, -0.814682186f, -1.29379487f, 0.627823353f, + 0.074902758f, 0.813860238f, -0.984662235f, -0.306858748f, + -0.265091538f, 0.371969193f, -1.2477777f, 1.49029386f, + 0.144044727f, 1.99415255f, 0.328208327f, 0.0835152566f, + 0.458172768f, -0.443422288f, 0.0502088852f, 1.10659289f, + 0.588640451f, 0.239094153f, 1.36343586f, -0.757167161f, + 0.792009294f, 0.393394828f, -0.725062251f, 0.588210046f, + -0.110117443f, -0.404118478f, -0.460690677f, 0.760483265f, + -0.204218745f, -0.037887115f, 1.672032f, 1.1623553f, + -0.245411262f, 1.50083613f, -0.213247627f, 0.614186049f, + 0.295952767f, -1.12194622f, -0.00988700334f, 1.16707921f, + 0.337536007f, -2.05753636f, 0.861629903f, -1.5298425f, +}; +const float kC_mixed[] = { + -0.10553544f, 0.242798343f, -0.0706134364f, 0.164503202f, + 0.0861888379f, 0.155533314f, 0.0161547624f, 0.482847929f, + -0.0923302919f, -0.118207321f, 0.344788611f, -0.220340937f, +}; diff --git a/tests/vllm/models/test_qwen4_exp_hc.cpp b/tests/vllm/models/test_qwen4_exp_hc.cpp new file mode 100644 index 000000000..55265be76 --- /dev/null +++ b/tests/vllm/models/test_qwen4_exp_hc.cpp @@ -0,0 +1,771 @@ +// Qwen4-Exp (Qwen3.8-Flash-Next) W3 UNIT GATE — the 4-branch GATED-RESIDUAL +// hyper-connection stream and the grouped RMSNorm it stands on. +// Issue #1988, spec `.agents/specs/qwen4-exp-flash-next.md`. +// +// WHAT THE GOLDENS ARE. `qwen4_exp_hc_goldens.inc` is dumped by +// `scripts/gen-qwen4-exp-hc-goldens.py`, which lifts `Qwen4ExpTextRMSNorm` +// (:158-181) and `Qwen4ExpTextGatedResidual` (:941-969) VERBATIM by line range +// out of the lane-pinned oracle — transformers **v5.16.0** +// `src/transformers/models/qwen4_exp/modeling_qwen4_exp.py`, sha256 +// 77fec77d87f2a0eb23b95fa04276fb5779698a7c7f523cf5061e49c118bcc459 — and +// EXECUTES them under torch. Nothing was retyped, and the script re-checks that +// sha256 before it runs. The one documented harness adaptation is a four-field +// stand-in for `Qwen4ExpTextConfig`: the real dataclass drags in the whole +// transformers package, and the classes read only `hidden_size`, `hc_count`, +// `hc_lowrank` and `rms_norm_eps`. The write-back goldens replay the two +// verbatim lines of `Qwen4ExpTextDecoderLayer.forward`. +// +// WHY A DISCRIMINATION SWEEP AND NOT ONLY A GOLDEN COMPARE. Five of this +// module's guarantees are single-character defects that a plausible port makes +// silently, and a golden that cannot see them is decoration. `Variant` below is +// an INDEPENDENT double-precision reference with one flag per defect; the test +// asserts the correct spelling matches the oracle AND that each single-flag flip +// does NOT, which measures the goldens' discriminating power rather than +// assuming it (AGENTS.md "Gates", `.agents/verification.md`). +// +// SCOPE, HONESTLY. Host reference only. No token claim, no speed claim: no +// `qwen4_exp` arm runs anywhere yet (the smallest published checkpoint is +// ~128 GB against a ~119 GB budget, spec `## Hardware`), and W1 config +// registration is still in review. What this gate proves is the arithmetic. +#include "vllm/model_executor/models/qwen4_exp_hc.h" + +#include + +#include +#include +#include +#include +#include + +#include "support/max_abs_diff.h" +#include "vllm/model_executor/models/deepseek_v4_mhc.h" + +using namespace vllm::qwen4_exp; +using vllm_test::MaxAbsDiff; + +namespace { + +#include "qwen4_exp_hc_goldens.inc" // NOLINT — golden literals + +// The goldens are fp32 out of torch; this reference is fp32 with double-precision +// reductions (the `deepseek_v4_mhc.cpp` house convention for a host reference). +// +// `kTol` IS A PROPERTY OF THESE SHAPES AND NOTHING ELSE. Say it here because the +// natural next reader is W5, wiring the module at hidden_size 2560, and reusing +// this number there is a red that is arithmetic noise. At the widths below — +// flat = 24 and 15 — the fp32 implementation is bit-identical or within one ulp +// of the oracle, measured against the pinned oracle itself: max|diff| over every +// golden array of case A, B and C is 2.384e-07, so kTol carries a 42x margin and +// is unconstrained. It does NOT survive a rescale, and the last two cases in this +// file are the measurement that says so rather than a warning that asks to be +// believed. See `kRealWidthNormTol` and `kRealWidthMixedTol`. +constexpr double kTol = 1e-5; + +// The floor a Variant flip has to clear. It is not a round number picked to be +// safe: it is 6.6x under the SMALLEST separation actually measured, and 100x +// over kTol, so the two bands cannot touch. Measured on case A, max|diff| of the +// flipped double reference against the oracle goldens, per token: +// +// flip t=0 t=1 t=2 affects +// silu-div-after 1.79e-2 6.63e-3 9.54e-3 mixed_input +// div-on-up 1.97e-2 1.95e-2 1.81e-2 mixed_input +// sum-not-mean 1.03e+0 6.91e-1 1.17e+0 mixed_input +// raw-not-normed 4.77e-1 3.39e-1 5.66e-1 mixed_input +// inject-no-div 5.22e-1 2.82e-1 5.49e-1 injection_weights +// +// The SiLU-division placement is the narrowest of the five, which is exactly the +// reason it is worth a gate: it is a one-token-deep edit that moves the answer +// by well under a percent at some tokens and would survive a loose eyeball. +constexpr double kFlipFloor = 1e-3; + +std::vector Load(const float* p, size_t n) { return std::vector(p, p + n); } + +// ─── The independent double-precision reference, with one flag per known trap ── +struct Variant { + bool div_before_silu = true; // silu(down(x)/hc), NOT silu(down(x))/hc + bool div_on_up = false; // there is NO division on the up-projection + bool reduce_is_mean = true; // .mean(dim=-2), NOT .sum + bool multiply_normed = true; // against hyper_input_normed, NOT hyper_input + bool inject_div = true; // 2*sigmoid(inject(x)/hc) + bool norm_is_grouped = true; // hc independent reductions, NOT one global one + bool norm_weight_plus_one = true; // (1.0 + w_hf) +}; + +double SigmoidD(double x) { return 1.0 / (1.0 + std::exp(-x)); } + +std::vector NormRefD(const std::vector& x, const std::vector& w_hf, + int64_t hc, int64_t hidden, double eps, const Variant& v) { + std::vector out(x.size()); + const int64_t groups = v.norm_is_grouped ? hc : 1; + const int64_t gsize = v.norm_is_grouped ? hidden : hc * hidden; + for (int64_t g = 0; g < groups; ++g) { + double ss = 0.0; + for (int64_t d = 0; d < gsize; ++d) { + const double t = x[g * gsize + d]; + ss += t * t; + } + const double r = 1.0 / std::sqrt(ss / static_cast(gsize) + eps); + for (int64_t d = 0; d < gsize; ++d) { + const double scale = v.norm_weight_plus_one ? 1.0 + w_hf[g * gsize + d] : w_hf[g * gsize + d]; + out[g * gsize + d] = x[g * gsize + d] * r * scale; + } + } + return out; +} + +std::vector LinearD(const float* w, const std::vector& x, int64_t out_dim, + int64_t in_dim) { + std::vector y(static_cast(out_dim), 0.0); + for (int64_t o = 0; o < out_dim; ++o) { + double acc = 0.0; + for (int64_t i = 0; i < in_dim; ++i) acc += static_cast(w[o * in_dim + i]) * x[i]; + y[o] = acc; + } + return y; +} + +struct RefOut { + std::vector normed; + std::vector mixed; + std::vector injection; +}; + +RefOut ForwardRefD(const std::vector& hyper, const std::vector& w_hf, + const float* down, const float* up, const float* inject, int64_t hc, + int64_t hidden, int64_t rank, double eps, const Variant& v) { + const int64_t flat = hc * hidden; + RefOut o; + o.normed = NormRefD(hyper, w_hf, hc, hidden, eps, v); + + std::vector lo = LinearD(down, o.normed, rank, flat); + for (int64_t r = 0; r < rank; ++r) { + const double a = v.div_before_silu ? lo[r] / static_cast(hc) : lo[r]; + const double s = a * SigmoidD(a); + lo[r] = v.div_before_silu ? s : s / static_cast(hc); + } + std::vector gate = LinearD(up, lo, flat, rank); + for (int64_t p = 0; p < flat; ++p) { + gate[p] = SigmoidD(v.div_on_up ? gate[p] / static_cast(hc) : gate[p]); + } + + o.mixed.assign(static_cast(hidden), 0.0); + for (int64_t j = 0; j < hc; ++j) { + for (int64_t h = 0; h < hidden; ++h) { + const double stream = v.multiply_normed ? o.normed[j * hidden + h] : hyper[j * hidden + h]; + o.mixed[h] += gate[j * hidden + h] * stream; + } + } + if (v.reduce_is_mean) { + for (int64_t h = 0; h < hidden; ++h) o.mixed[h] /= static_cast(hc); + } + + if (inject != nullptr) { + std::vector logits = LinearD(inject, o.normed, hc, flat); + o.injection.resize(static_cast(hc)); + for (int64_t j = 0; j < hc; ++j) { + o.injection[j] = + 2.0 * SigmoidD(v.inject_div ? logits[j] / static_cast(hc) : logits[j]); + } + } + return o; +} + +// EVERY max|diff| in this file goes through `vllm_test::MaxAbsDiff`, including +// the double-sided ones. This file used to carry two local helpers spelled +// `worst = std::max(worst, std::abs(...))` for the fp32-vs-double and +// double-vs-golden comparisons, which is verbatim the NaN-blind "form B" that +// `support/max_abs_diff.h` exists to prevent (issue #449): `std::max(a, NaN)` +// returns `a`, so an all-NaN result reduces to 0.0 and PASSES every `< tol` +// bound. It was not hypothetical — poisoning `GroupedRmsNorm`'s large-eps path +// to all-NaN left the `< kTol` assertion below GREEN, and the suite only went +// red because a SIBLING assertion happened to use the hardened helper. The +// shared scan is now templated on both operand types so there is no reason to +// write the reduction again here, and no place left to write it wrong. + +// "Bit for bit" means the BIT PATTERN. `==` is VALUE equality, and the one +// divergence the MhcPost caveat below names — +0.0f against -0.0f — is exactly +// the case `==` cannot see. Returns the index of the first differing pattern, +// or `a.size()` when every pattern matches. +size_t FirstBitDiff(const std::vector& a, const std::vector& b) { + REQUIRE(a.size() == b.size()); + for (size_t i = 0; i < a.size(); ++i) { + uint32_t x = 0, y = 0; + std::memcpy(&x, &a[i], sizeof(x)); + std::memcpy(&y, &b[i], sizeof(y)); + if (x != y) return i; + } + return a.size(); +} + +// ─── Real-width scaffolding ─────────────────────────────────────────────────── +// `Qwen/Qwen3.8-Flash-Next`'s own values (spec `## Architecture`): hc_count = 4, +// hidden_size = 2560, hc_lowrank = 320, so the residual is 10240 wide and the +// grouped norm reduces over 2560 elements. No oracle golden exists at this width +// and none should: dumping the IO of one token is 26 MB of `.inc`. What the two +// cases at the end of this file gate is the arithmetic's behaviour at SCALE, +// against the same independent double reference the sweep uses. +constexpr int64_t kModelHc = 4; +constexpr int64_t kModelHidden = 2560; +constexpr int64_t kModelRank = 320; + +// A deterministic generator written out in full rather than `std::mt19937` plus +// a distribution: `std::uniform_real_distribution` is NOT specified to produce +// the same sequence across standard libraries, and a gate whose INPUT depends on +// libstdc++ versus libc++ is not a reproducible measurement. +struct Lcg { + uint64_t s; + explicit Lcg(uint64_t seed) : s(seed) {} + // Uniform in [-1, 1), from the top 24 bits, which are the good ones. + float Next() { + s = s * 6364136223846793005ULL + 1442695040888963407ULL; + return static_cast(static_cast(s >> 40)) / 8388608.0f - 1.0f; + } + // Approximately unit-variance normal, four uniforms deep (Irwin-Hall). It + // matters that this is not uniform: the goldens are generated from + // `torch.randn * k`, the errors below are compared against a measurement made + // on THAT data, and a bounded uniform has visibly lighter tails, which shows + // up as a smaller dot-product error over 10240 terms. Var(U[-1,1)) = 1/3, so + // four of them scaled by sqrt(3)/2 is unit variance. + float Normal() { return 0.8660254f * (Next() + Next() + Next() + Next()); } +}; + +} // namespace + +// ── 1. The grouped RMSNorm, against vLLM's `RMSNormGated(group_size=)` form ──── +// vLLM `model_executor/layers/layernorm.py` @ origin/main 6a5e8f5979: `RMSNorm` +// opens at :37 and has no group knob; `RMSNormGated` opens at :172 with +// `group_size` at :187, and its grouped branch is :258-264 — +// `rearrange(x, "... (g d) -> ... g d")`, `variance = x_group.pow(2).mean(-1)`, +// `x_normed = x_group * rsqrt(variance + eps)`, `out = flatten * weight`. That is +// the op we mirror; the SEMANTICS (which weight, which eps) come from +// transformers, and the two differ by exactly the `1 + w` transform below. +TEST_CASE("qwen4_exp grouped RMSNorm mirrors RMSNormGated(group_size) at the lane pin") { + const int64_t hc = 4, hidden = 6; + const std::vector hyper = Load(kA_hyper, static_cast(hc * hidden)); + const std::vector w_hf = Load(kA_norm_w_hf, static_cast(hc * hidden)); + + const std::vector w = HcNormWeightFromHf(w_hf); + const std::vector got = GroupedRmsNorm(hyper, w, hidden, 1e-6f); + CHECK(MaxAbsDiff(got, kA_normed, hyper.size()) < kTol); + + // A GLOBAL reduction over all 24 elements is the defect a plain RMSNorm port + // makes. It must be visibly different, or the case above proves nothing. + Variant global; + global.norm_is_grouped = false; + CHECK(MaxAbsDiff(NormRefD(hyper, w_hf, hc, hidden, 1e-6, global), kA_normed, + hyper.size()) > 1e-2); + + // Same for the missing `1 + w` fold, through the SWEEP's reference rather than + // only through the direct call in the next case. `Variant` advertises a flag + // per known defect and this one was declared, read, and never once set false, + // so the struct claimed coverage the file did not have. + Variant raw_gamma; + raw_gamma.norm_weight_plus_one = false; + CHECK(MaxAbsDiff(NormRefD(hyper, w_hf, hc, hidden, 1e-6, raw_gamma), kA_normed, + hyper.size()) > 1e-1); + + // eps lives INSIDE the rsqrt, added to the mean square. At the model's real + // 1e-6 the two placements differ by ~1e-6, which is UNDER this suite's fp32 + // tolerance — so an eps-placement defect is invisible to every golden case + // above, and a case at an eps large enough to separate them is the only thing + // that gates it. `rsqrt(m + 4)` and `1/(sqrt(m) + 4)` are far apart; the + // double reference puts eps inside, so this pins the placement rather than + // merely observing that eps does something. + const std::vector big_eps = GroupedRmsNorm(hyper, w, hidden, 4.0f); + CHECK(MaxAbsDiff(big_eps, got) > 1e-2); + CHECK(MaxAbsDiff(big_eps, NormRefD(hyper, w_hf, hc, hidden, 4.0, Variant{})) < kTol); +} + +TEST_CASE("qwen4_exp hc_norm weight is 1 + w_hf, applied exactly once") { + const std::vector w_hf = Load(kA_norm_w_hf, 24); + const std::vector w = HcNormWeightFromHf(w_hf); + REQUIRE(w.size() == w_hf.size()); + for (size_t i = 0; i < w.size(); ++i) CHECK(w[i] == doctest::Approx(1.0f + w_hf[i])); + + const std::vector hyper = Load(kA_hyper, 24); + // The HF weight used raw is the "forgot the +1" defect: near-zero scale, which + // reads as a checkpoint bug rather than a port bug (spec, trap 2). + CHECK(MaxAbsDiff(GroupedRmsNorm(hyper, w_hf, 6, 1e-6f), kA_normed, 24) > 1e-1); + // Adding 1 a SECOND time is the defect a reader of the published GGUF makes, + // because the convert step has already folded it (issue #1988, trap 2). + CHECK(MaxAbsDiff(GroupedRmsNorm(hyper, HcNormWeightFromHf(w), 6, 1e-6f), kA_normed, 24) > 1e-1); +} + +TEST_CASE("qwen4_exp grouped RMSNorm refuses an indivisible width, as upstream does") { + // The MESSAGE is asserted, not merely the type. Every guard in this TU throws + // `std::invalid_argument`, so a type-only assertion passes when the guard that + // fires is a DIFFERENT one further down — which is exactly what a mutation + // sweep found: deleting a refusal left the suite green because a later bounds + // check threw the same type. Pinning the text makes each guard its own gate. + const std::vector x(10, 1.0f), w(10, 1.0f); + CHECK_THROWS_WITH_AS(GroupedRmsNorm(x, w, 4, 1e-6f), + "qwen4_exp: hidden_size (10) must be divisible by group_size (4).", + std::invalid_argument); + CHECK_THROWS_WITH_AS(GroupedRmsNorm(x, std::vector(9, 1.0f), 5, 1e-6f), + "qwen4_exp: hc_norm weight has 9 elements, expected 10.", + std::invalid_argument); +} + +// ── 2. The gated-residual READ, both arms ───────────────────────────────────── +TEST_CASE("qwen4_exp GatedResidual read matches transformers v5.16.0, use_combine=true") { + const int64_t hc = 4, hidden = 6, rank = 5, flat = hc * hidden; + GatedResidualWeights w; + w.hc_norm_weight = HcNormWeightFromHf(Load(kA_norm_w_hf, static_cast(flat))); + w.mix_down = Load(kA_down, static_cast(rank * flat)); + w.mix_up = Load(kA_up, static_cast(flat * rank)); + w.block_inject = Load(kA_inject, static_cast(hc * flat)); + + for (int64_t t = 0; t < 3; ++t) { + const std::vector hyper = Load(kA_hyper + t * flat, static_cast(flat)); + const GatedResidualResult r = GatedResidualForward(hyper, w, hc, hidden, 1e-6f); + CHECK(MaxAbsDiff(r.hyper_input_normed, kA_normed + t * flat, static_cast(flat)) < + kTol); + CHECK(MaxAbsDiff(r.mixed_input, kA_mixed + t * hidden, static_cast(hidden)) < kTol); + CHECK(MaxAbsDiff(r.injection_weights, kA_inj_w + t * hc, static_cast(hc)) < kTol); + // Range of 2*sigmoid is (0, 2) with 1.0 at a zero logit. + for (float v : r.injection_weights) CHECK((v > 0.0f && v < 2.0f)); + } +} + +TEST_CASE("qwen4_exp GatedResidual read matches transformers v5.16.0 at hc_count=3") { + // hc_count is a config value, not the constant 4. A port that folds 4 into the + // two divisions passes case A and fails here. + const int64_t hc = 3, hidden = 5, rank = 7, flat = hc * hidden; + GatedResidualWeights w; + w.hc_norm_weight = HcNormWeightFromHf(Load(kB_norm_w_hf, static_cast(flat))); + w.mix_down = Load(kB_down, static_cast(rank * flat)); + w.mix_up = Load(kB_up, static_cast(flat * rank)); + w.block_inject = Load(kB_inject, static_cast(hc * flat)); + + for (int64_t t = 0; t < 2; ++t) { + const std::vector hyper = Load(kB_hyper + t * flat, static_cast(flat)); + const GatedResidualResult r = GatedResidualForward(hyper, w, hc, hidden, 1e-5f); + CHECK(MaxAbsDiff(r.hyper_input_normed, kB_normed + t * flat, static_cast(flat)) < + kTol); + CHECK(MaxAbsDiff(r.mixed_input, kB_mixed + t * hidden, static_cast(hidden)) < kTol); + CHECK(MaxAbsDiff(r.injection_weights, kB_inj_w + t * hc, static_cast(hc)) < kTol); + + const std::vector block = Load(kB_block_out + t * hidden, static_cast(hidden)); + CHECK(MaxAbsDiff(GatedResidualWriteBack(hyper, block, r.injection_weights, hc, hidden), + kB_written + t * flat, static_cast(flat)) < kTol); + } +} + +TEST_CASE("qwen4_exp GatedResidual with use_combine=false is the final mixer") { + const int64_t hc = 4, hidden = 6, rank = 5, flat = hc * hidden; + GatedResidualWeights w; + w.hc_norm_weight = HcNormWeightFromHf(Load(kC_norm_w_hf, static_cast(flat))); + w.mix_down = Load(kC_down, static_cast(rank * flat)); + w.mix_up = Load(kC_up, static_cast(flat * rank)); + w.block_inject.clear(); // == `block_inject_weight = None` + + for (int64_t t = 0; t < 2; ++t) { + const std::vector hyper = Load(kC_hyper + t * flat, static_cast(flat)); + const GatedResidualResult r = GatedResidualForward(hyper, w, hc, hidden, 1e-6f); + CHECK(MaxAbsDiff(r.hyper_input_normed, kC_normed + t * flat, static_cast(flat)) < + kTol); + CHECK(MaxAbsDiff(r.mixed_input, kC_mixed + t * hidden, static_cast(hidden)) < kTol); + // `Qwen4ExpTextGatedResidual.forward` returns EARLY here: no injection branch + // is computed at all, and there is no trailing model norm after it either + // (`Qwen4ExpTextModel.forward` goes straight to `lm_head` — spec, trap 1). + CHECK(r.injection_weights.empty()); + CHECK(r.mixed_input.size() == static_cast(hidden)); + } +} + +TEST_CASE("qwen4_exp GatedResidual refuses a stream that is not hc_count * hidden_size") { + GatedResidualWeights w; + w.hc_norm_weight.assign(24, 1.0f); + w.mix_down.assign(5 * 24, 0.0f); + w.mix_up.assign(24 * 5, 0.0f); + // Message-pinned for the reason given on the grouped-norm refusal above: with + // a type-only assertion the 23- and 25-wide cases were caught downstream by + // GroupedRmsNorm's divisibility guard, and deleting THIS refusal changed + // nothing the suite could see. A 30-wide stream is divisible by hidden_size, + // so it walks past that guard too and reaches the weight-size check instead. + CHECK_THROWS_WITH_AS(GatedResidualForward(std::vector(23, 1.0f), w, 4, 6, 1e-6f), + "qwen4_exp: expected 24 hyper-connection features, got 23.", + std::invalid_argument); + CHECK_THROWS_WITH_AS(GatedResidualForward(std::vector(25, 1.0f), w, 4, 6, 1e-6f), + "qwen4_exp: expected 24 hyper-connection features, got 25.", + std::invalid_argument); + CHECK_THROWS_WITH_AS(GatedResidualForward(std::vector(30, 1.0f), w, 4, 6, 1e-6f), + "qwen4_exp: expected 24 hyper-connection features, got 30.", + std::invalid_argument); +} + +// Every OTHER refusal this module makes. Split out from the two cases above +// because those two were, until this change, the ONLY refusals under any +// assertion: a mutation sweep deleted eight of the eleven guards one at a time +// and the suite stayed 280/280 green each time, including all three `RequireSize` +// calls in `GatedResidualWriteBack`. +// +// Three of those eight are not hygiene. `GatedResidualWriteBackInPlace` takes RAW +// POINTERS and the header declares it the fusion seam that a device kernel +// replaces, so the wrapper's three size checks are the only bounds check between +// a caller and an out-of-bounds read of `block_out[h]` or `injection_weights[j]`. +// What that class of guard prevents is not a tidy error: deleting the +// divisibility guard in the same sweep produced `Fatal glibc error: +// malloc.c:2599` and `Aborted (core dumped)`, with no doctest summary printed at +// all. +// +// The MESSAGE is pinned, never the type. Every guard in this translation unit +// throws `std::invalid_argument`, so `CHECK_THROWS_AS` cannot tell a deleted +// guard from a different one firing further down — which is the defect the +// hyper_input case above already documents finding. +TEST_CASE("qwen4_exp refuses every malformed shape BY NAME, not merely by type") { + const std::vector ones10(10, 1.0f); + + SUBCASE("grouped norm: a non-positive group size") { + // Deleting this guard is not a wrong answer, it is `x.size() % 0` and the + // process dies on SIGFPE. Re-run and confirmed: with the guard removed this + // case takes the whole binary down mid-run. + CHECK_THROWS_WITH_AS(GroupedRmsNorm(ones10, ones10, 0, 1e-6f), + "qwen4_exp: group_size must be positive, got 0.", std::invalid_argument); + CHECK_THROWS_WITH_AS(GroupedRmsNorm(ones10, ones10, -3, 1e-6f), + "qwen4_exp: group_size must be positive, got -3.", std::invalid_argument); + } + + SUBCASE("gated residual: hc_count <= 1 and hidden_size <= 0") { + // `configuration_qwen4_exp.py:196-197` raises `Qwen4-Exp requires hc_count > + // 1`; this is that refusal, and until now nothing held it. + GatedResidualWeights w; + w.hc_norm_weight.assign(24, 1.0f); + w.mix_down.assign(5 * 24, 0.0f); + w.mix_up.assign(24 * 5, 0.0f); + CHECK_THROWS_WITH_AS(GatedResidualForward(std::vector(6, 1.0f), w, 1, 6, 1e-6f), + "qwen4_exp: hc_count must be > 1 and hidden_size > 0, got 1 and 6.", + std::invalid_argument); + CHECK_THROWS_WITH_AS(GatedResidualForward(std::vector(24, 1.0f), w, 4, 0, 1e-6f), + "qwen4_exp: hc_count must be > 1 and hidden_size > 0, got 4 and 0.", + std::invalid_argument); + } + + SUBCASE("gated residual: the low-rank weights") { + // `rank` is INFERRED from `mix_down.size() / flat`, so a ragged `mix_down` is + // not a bad input, it is a bad rank silently propagated into two more GEMMs. + const std::vector hyper(24, 1.0f); + GatedResidualWeights w; + w.hc_norm_weight.assign(24, 1.0f); + w.mix_up.assign(24 * 5, 0.0f); + + w.mix_down.clear(); + CHECK_THROWS_WITH_AS(GatedResidualForward(hyper, w, 4, 6, 1e-6f), + "qwen4_exp: input_mix_weight_down is not a multiple of 24 (got 0).", + std::invalid_argument); + w.mix_down.assign(23, 0.0f); + CHECK_THROWS_WITH_AS(GatedResidualForward(hyper, w, 4, 6, 1e-6f), + "qwen4_exp: input_mix_weight_down is not a multiple of 24 (got 23).", + std::invalid_argument); + + w.mix_down.assign(5 * 24, 0.0f); // rank 5, so mix_up must be 120 + w.mix_up.assign(119, 0.0f); + CHECK_THROWS_WITH_AS(GatedResidualForward(hyper, w, 4, 6, 1e-6f), + "qwen4_exp: input_mix_weight_up has 119 elements, expected 120.", + std::invalid_argument); + + w.mix_up.assign(120, 0.0f); + w.block_inject.assign(95, 0.0f); // must be hc * flat = 96 + CHECK_THROWS_WITH_AS(GatedResidualForward(hyper, w, 4, 6, 1e-6f), + "qwen4_exp: block_inject_weight has 95 elements, expected 96.", + std::invalid_argument); + } + + SUBCASE("write-back: the fusion seam's only bounds check") { + const std::vector hyper24(24, 1.0f), block6(6, 1.0f), inj4(4, 1.0f); + CHECK_THROWS_WITH_AS( + GatedResidualWriteBack(std::vector(23, 1.0f), block6, inj4, 4, 6), + "qwen4_exp: hyper_input has 23 elements, expected 24.", std::invalid_argument); + CHECK_THROWS_WITH_AS( + GatedResidualWriteBack(hyper24, std::vector(5, 1.0f), inj4, 4, 6), + "qwen4_exp: block output has 5 elements, expected 6.", std::invalid_argument); + CHECK_THROWS_WITH_AS( + GatedResidualWriteBack(hyper24, block6, std::vector(3, 1.0f), 4, 6), + "qwen4_exp: injection_weights has 3 elements, expected 4.", std::invalid_argument); + } +} + +// ── 3. The rank-1 write-back ────────────────────────────────────────────────── +TEST_CASE("qwen4_exp write-back is hyper_input RAW plus outer(block_out, injection)") { + const int64_t hc = 4, hidden = 6, flat = hc * hidden; + for (int64_t t = 0; t < 3; ++t) { + const std::vector hyper = Load(kA_hyper + t * flat, static_cast(flat)); + const std::vector block = Load(kA_block_out + t * hidden, static_cast(hidden)); + const std::vector inj = Load(kA_inj_w + t * hc, static_cast(hc)); + const std::vector got = GatedResidualWriteBack(hyper, block, inj, hc, hidden); + CHECK(MaxAbsDiff(got, kA_written + t * flat, static_cast(flat)) < kTol); + + // The un-normed stream is what gets written. Feeding the NORMED stream here + // is the natural mistake and must be visible. + const std::vector normed = Load(kA_normed + t * flat, static_cast(flat)); + CHECK(MaxAbsDiff(GatedResidualWriteBack(normed, block, inj, hc, hidden), + kA_written + t * flat, static_cast(flat)) > 1e-2); + + // The in-place form is THE FUSION SEAM (`qwen4_exp_hc.h`), so it is gated + // against the ORACLE directly and not against its own wrapper. Checking it + // against `got` alone is a tautology: `GatedResidualWriteBack` copies and + // then calls this very function, so the comparison holds under any mutation + // of it. Proven — mutating the in-place body (`+=` to `=`, and + // `injection_weights[j]` to `[0]`) reddened three other assertions and never + // this one. The golden compare below is the assertion that actually holds + // the primitive; the bit check that follows holds the WRAPPER's claim to be + // free of its own arithmetic. + std::vector inplace = hyper; + GatedResidualWriteBackInPlace(inplace.data(), block.data(), inj.data(), hc, hidden); + CHECK(MaxAbsDiff(inplace, kA_written + t * flat, static_cast(flat)) < kTol); + CHECK(FirstBitDiff(inplace, got) == inplace.size()); + } +} + +// The bit helper's own polarity, pinned. `==` reports these two equal and the +// pattern comparison does not, which is the entire reason the two "bit for bit" +// claims in this file stopped using `==`. +TEST_CASE("qwen4_exp bit-for-bit means the pattern: +0.0f and -0.0f are == but not equal") { + const std::vector pos{0.0f, 1.5f}; + const std::vector neg{-0.0f, 1.5f}; + CHECK(pos[0] == neg[0]); + CHECK(FirstBitDiff(pos, neg) == 0); + CHECK(FirstBitDiff(pos, pos) == pos.size()); +} + +// The spec asserts our DeepSeek-V4 `MhcPost` is a bit-exact bring-up bridge for +// this write-back once its comb matrix is the identity. Verified rather than +// trusted, because it is the reviewer's free mutation target. It holds on finite +// inputs; it is NOT an identity for a negative-zero residual (0.0f + (-0.0f) +// is +0.0f) or a non-finite one (0.0f * inf is NaN), and neither case is +// reachable on this path. +TEST_CASE("qwen4_exp write-back equals MhcPost with an identity comb, bit for bit") { + const int64_t hc = 4, hidden = 6, flat = hc * hidden; + std::vector comb(static_cast(hc * hc), 0.0f); + for (int64_t j = 0; j < hc; ++j) comb[j * hc + j] = 1.0f; + + for (int64_t t = 0; t < 3; ++t) { + const std::vector hyper = Load(kA_hyper + t * flat, static_cast(flat)); + const std::vector block = Load(kA_block_out + t * hidden, static_cast(hidden)); + const std::vector inj = Load(kA_inj_w + t * hc, static_cast(hc)); + const std::vector ours = GatedResidualWriteBack(hyper, block, inj, hc, hidden); + const std::vector bridge = + vllm::deepseek_v4::MhcPost(block, hyper, inj, comb, hc, hidden); + REQUIRE(bridge.size() == ours.size()); + // The PATTERN, not `==`. The negative-zero caveat in the comment above is + // precisely the divergence value equality cannot see, so asserting the claim + // with `==` asserted something weaker than the claim. + CHECK(FirstBitDiff(bridge, ours) == ours.size()); + } +} + +// ── 4. Discriminating power: each trap opens a gap the goldens can see ──────── +TEST_CASE("qwen4_exp goldens separate every known gated-residual trap") { + const int64_t hc = 4, hidden = 6, rank = 5, flat = hc * hidden; + const std::vector w_hf = Load(kA_norm_w_hf, static_cast(flat)); + + for (int64_t t = 0; t < 3; ++t) { + const std::vector hyper = Load(kA_hyper + t * flat, static_cast(flat)); + const RefOut ok = ForwardRefD(hyper, w_hf, kA_down, kA_up, kA_inject, hc, hidden, rank, 1e-6, + Variant{}); + // The independent double reference reproduces the oracle. Two references + // agreeing is what makes a single-flag flip attributable to the flag. + CHECK(MaxAbsDiff(ok.mixed, kA_mixed + t * hidden, static_cast(hidden)) < kTol); + CHECK(MaxAbsDiff(ok.injection, kA_inj_w + t * hc, static_cast(hc)) < kTol); + + struct Case { + Variant v; + bool hits_mixed; + bool hits_injection; + }; + Variant a; + a.div_before_silu = false; // silu(x)/hc instead of silu(x/hc) + Variant b; + b.div_on_up = true; // a division upstream does NOT have + Variant c; + c.reduce_is_mean = false; // sum instead of mean + Variant d; + d.multiply_normed = false; // the raw stream instead of the normed one + Variant e; + e.inject_div = false; // 2*sigmoid(logits) instead of 2*sigmoid(logits/hc) + const Case cases[] = {{a, true, false}, {b, true, false}, {c, true, false}, + {d, true, false}, {e, false, true}}; + + for (const Case& k : cases) { + const RefOut bad = + ForwardRefD(hyper, w_hf, kA_down, kA_up, kA_inject, hc, hidden, rank, 1e-6, k.v); + if (k.hits_mixed) { + CHECK(MaxAbsDiff(bad.mixed, kA_mixed + t * hidden, static_cast(hidden)) > + kFlipFloor); + } + if (k.hits_injection) { + CHECK(MaxAbsDiff(bad.injection, kA_inj_w + t * hc, static_cast(hc)) > kFlipFloor); + } + } + } +} + +// ── 5. Real width: the two things the toy shapes cannot measure ─────────────── +// +// Everything above runs at hidden_size 6 or 5 and hc_count 4 or 3. Two of this +// module's documented properties are invisible at those widths, and one of them +// is a tolerance the next wave will otherwise reuse. + +// THE DOUBLE ACCUMULATOR. `qwen4_exp_hc.h` "PRECISION" states the per-group sum +// of squares is accumulated in `double`. At group sizes 5 and 6 that claim has +// no discriminating power whatsoever: replacing `double ss` with `float ss` +// moves nothing any tolerance in this file can see, so it was a documented +// convention with zero coverage. At the model's real group size it is worth +// three orders of magnitude. +// +// The input is magnitude-separated on purpose — one element at 4096, the rest at +// 1.0 — and that is not cherry-picking, it is the only shape that CAN separate +// the two accumulators. A sum of 2560 POSITIVE squares of similar size loses +// about sqrt(N)*u either way and the two agree to within 3x. The loss appears +// when a partial sum grows past the point where the next term falls under its +// own ulp: 4096^2 is exactly 2^24, the float integer ceiling, so every one of the +// 2559 following `+1.0` terms rounds away and `float ss` misses the tail +// ENTIRELY, by 2559 out of 16779775. +// +// Measured on exactly the data below, and byte-identical at -O0 and -O2, so it +// does not rest on vectorisation. It cannot rest on FMA contraction either: this +// tree compiles with `-ffp-contract=off` everywhere. +// +// max|reference| 5.20e+01 +// ours, double accumulator 3.168430e-06 (6.1e-08 relative: one ulp) +// the same code with `float ss` 2.352230e-03 (742x worse) +// +// The bound sits 31.6x above the first and 23.5x below the second, which is a +// band rather than a fitted number. Both ends are re-run mutations, not +// estimates: `double ss` -> `float ss` in `GroupedRmsNorm` reddens this case and +// nothing else in the file. +constexpr double kRealWidthNormTol = 1e-4; + +TEST_CASE("qwen4_exp GroupedRmsNorm needs its double accumulator at group_size 2560") { + const int64_t flat = kModelHc * kModelHidden; + std::vector x(static_cast(flat)), w(static_cast(flat)); + const float dominant[4] = {4096.0f, 2048.0f, 8192.0f, 1024.0f}; + Lcg rng(12345); + for (int64_t j = 0; j < kModelHc; ++j) { + for (int64_t d = 0; d < kModelHidden; ++d) { + x[j * kModelHidden + d] = (d == 0) ? dominant[j] : 1.0f; + w[j * kModelHidden + d] = 1.0f + 0.5f * rng.Next(); + } + } + + const std::vector got = GroupedRmsNorm(x, w, kModelHidden, 1e-6f); + + // The reference, in full double, written out here rather than reusing + // `NormRefD`: that one takes the HF gamma and applies the `1 + w` fold, and + // this case is about the ACCUMULATOR, so the weight goes in already folded. + std::vector want(static_cast(flat)); + for (int64_t j = 0; j < kModelHc; ++j) { + double ss = 0.0; + for (int64_t d = 0; d < kModelHidden; ++d) { + const double v = x[j * kModelHidden + d]; + ss += v * v; + } + const double r = 1.0 / std::sqrt(ss / static_cast(kModelHidden) + 1e-6); + for (int64_t d = 0; d < kModelHidden; ++d) { + want[j * kModelHidden + d] = static_cast(x[j * kModelHidden + d]) * r * + static_cast(w[j * kModelHidden + d]); + } + } + CHECK(MaxAbsDiff(got, want) < kRealWidthNormTol); + + // And the separation is real rather than asserted: the same reduction with a + // float accumulator, over the same data, is on the far side of the bound. If + // this stops holding, the case above has stopped measuring anything. + float ss_f = 0.0f; + double ss_d = 0.0; + for (int64_t d = 0; d < kModelHidden; ++d) { + const double v = x[d]; + ss_f += static_cast(v * v); + ss_d += v * v; + } + const double r_f = 1.0 / std::sqrt(static_cast(ss_f) / kModelHidden + 1e-6); + const double r_d = 1.0 / std::sqrt(ss_d / kModelHidden + 1e-6); + CHECK(std::abs(r_f - r_d) * dominant[0] * w[0] > 10.0 * kRealWidthNormTol); +} + +// WHAT THE fp32 INTERIOR COSTS, AND WHY `kTol` DOES NOT TRAVEL. At flat = 24 the +// implementation is bit-identical to the oracle, so kTol measures nothing there. +// At flat = 10240 it is not, and the reason is not ours alone. Measured against +// the PINNED ORACLE itself (transformers v5.16.0, the same lift-and-exec path +// `scripts/gen-qwen4-exp-hc-goldens.py` uses, at hidden=2560 hc=4 lowrank=320 +// eps=1e-6, two tokens), max|diff| on `mixed_input`. These are ONE draw of +// random inputs, and the ratios below move from draw to draw; the ordering and +// the conclusion do not. +// +// t=0 t=1 +// ours (fp32) vs oracle 2.325e-05 2.137e-05 +// double ref vs oracle 1.360e-05 5.431e-06 +// ours (fp32) vs double 3.684e-05 1.606e-05 +// +// Two conclusions, and the second is the one that matters. Our fp32 interior is +// 2.1x to 2.3x over kTol at model width, driven by `LinearNoBias`'s sequential +// fp32 accumulation over 10240 terms. And the ORACLE is itself OF THE SAME +// ORDER AS kTol against an exact evaluation of its own algorithm — 1.36x on the +// draw tabulated above, 0.91x and 0.82x on an independent draw taken during +// fresh review — because torch runs this in fp32 too, so widening OUR +// accumulator cannot rescue a 1e-5 absolute bound here. No +// fp32 implementation of this function meets kTol at hidden_size 2560, and the +// tolerance is the thing that is wrong at that width, not the arithmetic. +// +// The bound below is therefore RELATIVE to the reference's own magnitude, and +// the case compares against the double reference because committing an oracle +// golden at this width is 26 MB of `.inc`. It is derived rather than fitted: a +// sequential fp32 dot of length K accumulates a relative error that grows as a +// random walk, about sqrt(K)*u, with u = 2^-24 = 5.96e-08 the fp32 unit +// roundoff. At K = 10240 that is 6.03e-06, and 4e-05 is 6.6x it. The three +// measurements that exist all sit inside that: 6.79e-06 relative on this case's +// own data, and 7.32e-06 / 1.68e-05 on the two oracle tokens tabulated above. +// +// What this gates is that the fp32 interior stays within a small multiple of the +// random-walk bound for its accumulation length. What it does NOT gate is +// agreement with the ORACLE at this width — that needs a real checkpoint, and it +// is recorded under the spec's `## Owed`. +constexpr double kRealWidthMixedRel = 4e-5; + +TEST_CASE("qwen4_exp gated residual at the model's real width, hidden_size 2560") { + const int64_t flat = kModelHc * kModelHidden; + Lcg rng(20260826); + std::vector w_hf(static_cast(flat)), hyper(static_cast(flat)); + std::vector down(static_cast(kModelRank * flat)); + std::vector up(static_cast(flat * kModelRank)); + std::vector inject(static_cast(kModelHc * flat)); + // The same standard deviations `scripts/gen-qwen4-exp-hc-goldens.py` uses, so + // the numbers this case measures are comparable with the oracle table above. + for (float& v : w_hf) v = 0.5f * rng.Normal(); + for (float& v : hyper) v = 1.7f * rng.Normal(); + for (float& v : down) v = 0.3f * rng.Normal(); + for (float& v : up) v = 0.3f * rng.Normal(); + for (float& v : inject) v = 0.3f * rng.Normal(); + + GatedResidualWeights w; + w.hc_norm_weight = HcNormWeightFromHf(w_hf); + w.mix_down = down; + w.mix_up = up; + w.block_inject = inject; + + const GatedResidualResult got = GatedResidualForward(hyper, w, kModelHc, kModelHidden, 1e-6f); + const RefOut ref = ForwardRefD(hyper, w_hf, down.data(), up.data(), inject.data(), kModelHc, + kModelHidden, kModelRank, 1e-6, Variant{}); + + double scale = 0.0; + for (double v : ref.mixed) scale = std::max(scale, std::abs(v)); + REQUIRE(scale > 0.1); // the data is not degenerate, so the ratio means something + + CHECK(MaxAbsDiff(got.hyper_input_normed, ref.normed) < kRealWidthMixedRel * scale); + CHECK(MaxAbsDiff(got.mixed_input, ref.mixed) < kRealWidthMixedRel * scale); + CHECK(MaxAbsDiff(got.injection_weights, ref.injection) < kRealWidthMixedRel * 2.0); + + // The write-back at real width too: it is the seam, and 10240 is the width the + // fused kernel will see. + std::vector block(static_cast(kModelHidden)); + for (float& v : block) v = 0.9f * rng.Normal(); + const std::vector written = + GatedResidualWriteBack(hyper, block, got.injection_weights, kModelHc, kModelHidden); + REQUIRE(written.size() == static_cast(flat)); + for (int64_t j = 0; j < kModelHc; ++j) { + for (int64_t h = 0; h < kModelHidden; h += 512) { + const size_t i = static_cast(j * kModelHidden + h); + CHECK(written[i] == doctest::Approx(hyper[i] + block[h] * got.injection_weights[j])); + } + } +}