feat(MODEL-MM-QWEN4-EXP): W3 — the 4-branch gated-residual hyper-connection stream and its grouped RMSNorm

.agents/specs/qwen4-exp-flash-next.md

@@ -421,6 +421,41 @@ that does not exist. Stated because that tail is the natural thing to copy.
 `w_vllm = 1.0 + w_hf`. Miss it and every `hc_norm` gets a near-zero scale, which reads
 as a checkpoint bug rather than a port bug.
 
+**The GGUF converter already folds it, and it folds far more than `hc_norm`.** Read at
+source rather than relayed, because W5 writes the loader and the narrow version of this
+sentence causes the defect it warns about. Every anchor below is read at our recorded
+llama.cpp pin, stock upstream tag `b10451` (`10bf611e533d81f739128304991c5e133c6aebd8`,
+[`../oracles/llama-cpp.md`](../oracles/llama-cpp.md)). Stock upstream has no `qwen4exp`
+at all there (`git grep -il qwen4exp`: nothing tree-wide, so a released llama.cpp can
+neither convert nor load this architecture). The converter is ggml-org/llama.cpp
+[#27742](https://github.com/ggml-org/llama.cpp/pull/27742), head
+`035e22731a7fd70b9854b3a2d64ec68e9b1a45d3`, **still OPEN**. Its `conversion/qwen4exp.py`
+declares `class Qwen4ExpTextModel(_Qwen35MRopeMixin, _LinearAttentionVReorderBase)`;
+`_LinearAttentionVReorderBase` is `conversion/qwen.py:438`, a subclass of
+`Qwen3NextModel` (`:365`, whose own signature is
+`class Qwen3NextModel(_QwenMtpMixin, Qwen2MoeModel)`); and the PR's `modify_tensors` has
+**no `hc_norm` branch**, so `hc_norm.weight` falls through to `super()`. The `+1` is the
+inherited Qwen3-Next rule at `conversion/qwen.py:387-388`:
+
+```python
+elif name.endswith("norm.weight") and not name.endswith("linear_attn.norm.weight"):
+    data_torch = data_torch + 1
+```
+
+So the rule a loader implements is **every `*norm.weight` carries the fold, with
+`linear_attn.norm.weight` (the GDN `ssm_norm`) the one exception** -- `hc_norm`,
+`attn_q_norm` and `attn_k_norm` all match it, and the PLE and indexer gammas are folded
+by the PR's own early-returning branch. A loader that skips the fold for `hc_norm` alone
+double-folds everything else, which is the same silent ~2x defect one tensor to the left.
+Two consequences for W5. The property belongs to one in-flight converter, not to "GGUF":
+#27742 can change before it merges and another publisher's tool need not match it, so the
+loader treats the fold as a provenance question and checks it -- cheaply, since an
+unfolded `hc_norm` is a zero-init gamma and a folded one is centred on 1.0. And it was
+corroborated on published artifacts during fresh review of #1988
+(`unsloth/Qwen3.8-Flash-Next-GGUF` `UD-IQ1_S` and `UD-Q4_K_XL`, `vumpt/...-Q4_K_M`, read
+by HTTP range request against the bf16 HF tensors): every `*hc_norm.weight` is HF + 1.0
+exactly, elementwise, while `ssm_norm` is unfolded and sits in [0.875, 1.023].
+
 **Correction to the port map above.** vLLM's grouped RMSNorm is on **`RMSNormGated`**,
 not the plain `RMSNorm`, whose only related knob is `var_hidden_size` -- a prefix
 reduction that cannot express per-group norms. Verified directly: `RMSNorm` opens at
@@ -822,6 +857,82 @@ change that makes any arm reachable, not later.
   currently owns building them, and a multimodal caller with genuinely different
   t/h/w streams would be running an untested section layout. The wave that builds
   the cos/sin tables owes a case with three DISTINCT position streams.
+- **W3's host reference lands UNREACHED, and this is the record of it** per
+  AGENTS.md "Nothing lands dead".
+  `src/vllm/model_executor/models/qwen4_exp_hc.{h,cpp}`
+  ([#1988](https://github.com/mudler/vllm.cpp/issues/1988)) is reached only by
+  `tests/vllm/models/test_qwen4_exp_hc.cpp`. No production entry point calls it
+  at its merge commit: W1 config registration
+  ([#1986](https://github.com/mudler/vllm.cpp/issues/1986)) was still in review,
+  so no `qwen4_exp` resolves through the loader and there is nothing for the
+  gated-residual stream to hang off. The wiring is owed by **W5, assembly**,
+  under [#1978](https://github.com/mudler/vllm.cpp/issues/1978), which is the
+  wave that widens the residual buffers to `hc_count * hidden_size` and calls
+  the module twice per layer.
+- The **model-matrix lifecycle cell** for
+  `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation`, which still reads
+  `SPEC ONLY`. Left to W1 deliberately rather than by omission: W1 is the wave
+  whose scope IS registration, its pull request is already open, and
+  `.agents/model-matrix.md` is a single shared file, so three parallel waves
+  editing one cell is the write-lock AGENTS.md "Records" names. Whichever of
+  W1/W2/W3 lands last owes the correction.
+- The **device arm of the gated residual**, and with it one check this host wave
+  cannot make: that `RMSNormGated.forward_cuda`'s flash-linear-attention Triton
+  kernel is numerically correct in its GROUPED mode (unverified upstream, see
+  `## Design`).
+- **W3's `kTol = 1e-5` is an absolute bound that does not survive a rescale, and
+  the host reference is the first thing it fails.** Recorded because an earlier
+  draft of the bullet above framed the tolerance question as the DEVICE arm's
+  problem, and it is not. Measured against the pinned oracle itself, at the
+  model's own shape (hidden_size 2560, hc_count 4, hc_lowrank 320, eps 1e-6, two
+  tokens), max|diff| on `mixed_input`. This is ONE draw of random inputs, and the
+  ratios below move from draw to draw; the ordering and the conclusion do not.
+
+  | | t=0 | t=1 |
+  |---|---|---|
+  | ours (fp32) vs oracle | 2.325e-05 | 2.137e-05 |
+  | exact double vs oracle | 1.360e-05 | 5.431e-06 |
+  | ours (fp32) vs exact double | 3.684e-05 | 1.606e-05 |
+
+  At the suite's own widths (flat = 24 and 15) the implementation is bit-identical
+  to the oracle -- max|diff| over every golden array of cases A, B and C is
+  2.384e-07 -- so kTol carries a 42x margin there and constrains nothing. At model
+  width our fp32 interior is 2.1x to 2.3x over it, driven by `LinearNoBias`'s
+  sequential fp32 accumulation over 10240 terms. **The second row is the one that
+  settles it: the ORACLE is itself of the same ORDER as kTol against an exact
+  evaluation of its own algorithm -- 1.36x on the draw above, 0.91x and 0.82x on
+  an independent draw taken during fresh review -- because torch runs this in
+  fp32 too.** No fp32
+  implementation of this function meets a 1e-5 ABSOLUTE bound at hidden_size
+  2560, and widening our accumulator cannot rescue one. W5 therefore does not
+  reuse kTol at model width; the file carries a real-width case with a relative
+  bound (`kRealWidthMixedRel`, 4e-5, derived as 6.6x the sqrt(K)*u random-walk
+  bound for K = 10240) that all three measurements sit inside. **What is still
+  owed** is agreement with the ORACLE at model width, which needs a real
+  checkpoint and cannot be closed in-suite: the in-suite case compares against
+  the double reference, because dumping one token of oracle IO at this width is
+  26 MB of `.inc`.
+- **The double accumulator is now gated, and the device arm inherits the
+  consequence.** `GroupedRmsNorm` accumulates the per-group sum of squares in
+  `double`, and at the suite's group sizes of 5 and 6 that convention had zero
+  discriminating power -- replacing it with `float` left the suite 280/280 green.
+  It is gated at the model's real group size of 2560, on magnitude-separated
+  data, where the two accumulators differ by 742x (3.168e-06 against 2.352e-03,
+  bound 1e-4). The convention is kept rather than dropped because it makes the
+  host reference more accurate than the oracle rather than less, which is what a
+  reference is for. What follows for the device arm, stated here so it is not
+  discovered: **a straight fp32-accumulate device reduction will not meet
+  `kRealWidthNormTol` on that data.** That is the correct signal, not a defect in
+  the gate -- it says the device kernel must accumulate wider than fp32 or be
+  gated against the oracle directly rather than against this reference. Deciding
+  which is the device wave's, and it is owed.
+- The **fused rank-1 write-back**. `GatedResidualWriteBackInPlace` is the seam
+  and is already the primitive, but no device kernel replaces it yet. Both
+  llama.cpp implementations of this architecture materialise the update as a
+  `repeat_4d` + `mul`, i.e. 96 dense `[2560, 4, T]` broadcasts built and thrown
+  away per forward pass at 48 layers x 2 sites, which is where a
+  beat-llama.cpp-at-concurrency claim would come from. Not claimed here: no arm
+  runs.
 
 ## Now
 

CMakeLists.txt

@@ -789,6 +789,7 @@ add_library(vllm STATIC
   src/vllm/model_executor/models/qwen3_moe_registry.cpp
   src/vllm/model_executor/models/qwen3_moe_weights.cpp
   src/vllm/model_executor/models/qwen3_moe.cpp
+  src/vllm/model_executor/models/qwen4_exp_hc.cpp
   src/vllm/model_executor/models/deepseek_v2_registry.cpp
   src/vllm/model_executor/models/deepseek_v2_weights.cpp
   src/vllm/model_executor/models/deepseek_v2.cpp

scripts/gen-qwen4-exp-hc-goldens.py

@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+"""Dump Qwen4-Exp gated-residual goldens by EXECUTING the pinned oracle source.
+
+    usage: gen-qwen4-exp-hc-goldens.py <out.inc> [path/to/modeling_qwen4_exp.py]
+
+Fetch the oracle source first (it is not vendored -- the sha256 below is the pin):
+
+    curl -sSLO https://raw.githubusercontent.com/huggingface/transformers/\
+v5.16.0/src/transformers/models/qwen4_exp/modeling_qwen4_exp.py
+
+
+Oracle: huggingface/transformers v5.16.0,
+  src/transformers/models/qwen4_exp/modeling_qwen4_exp.py
+  sha256 77fec77d87f2a0eb23b95fa04276fb5779698a7c7f523cf5061e49c118bcc459
+
+The two classes under test are lifted VERBATIM by line range out of that file and
+exec'd; nothing is retyped.  The only harness adaptation is a 4-field stand-in for
+`Qwen4ExpTextConfig` (the real dataclass drags in the whole transformers package,
+and the installed transformers here is 5.3.0, which predates qwen4_exp entirely).
+The write-back is likewise the verbatim two lines of
+`Qwen4ExpTextDecoderLayer.forward`.
+"""
+import hashlib
+import sys
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+SRC = sys.argv[2] if len(sys.argv) > 2 else "modeling_qwen4_exp.py"
+EXPECT_SHA = "77fec77d87f2a0eb23b95fa04276fb5779698a7c7f523cf5061e49c118bcc459"
+
+raw = open(SRC, "rb").read()
+got = hashlib.sha256(raw).hexdigest()
+# NOT an `assert`: `python3 -O` strips those, and this one guard is the entire
+# mechanism tying the goldens to the pin.  Under -O the stripped version would
+# happily dump goldens from any file handed to it.
+if got != EXPECT_SHA:
+    raise SystemExit(f"oracle source sha256 {got} != {EXPECT_SHA}")
+lines = raw.decode().splitlines(keepends=True)
+
+# 1-based, inclusive, as reported by grep -n on the pinned file.
+RMSNORM = (158, 181)      # class Qwen4ExpTextRMSNorm
+GATEDRES = (941, 969)     # class Qwen4ExpTextGatedResidual
+
+
+def lift(span):
+    return "".join(lines[span[0] - 1: span[1]])
+
+
+class Qwen4ExpTextConfig:  # harness stand-in; only the 4 fields the classes read
+    def __init__(self, hidden_size, hc_count, hc_lowrank, rms_norm_eps):
+        self.hidden_size = hidden_size
+        self.hc_count = hc_count
+        self.hc_lowrank = hc_lowrank
+        self.rms_norm_eps = rms_norm_eps
+
+
+ns = {"torch": torch, "nn": nn, "F": F, "Qwen4ExpTextConfig": Qwen4ExpTextConfig}
+exec(lift(RMSNORM), ns)
+exec(lift(GATEDRES), ns)
+Qwen4ExpTextRMSNorm = ns["Qwen4ExpTextRMSNorm"]
+Qwen4ExpTextGatedResidual = ns["Qwen4ExpTextGatedResidual"]
+
+torch.set_default_dtype(torch.float32)
+out = []
+
+
+def emit(name, t):
+    flat = t.detach().reshape(-1).tolist()
+    out.append("const float %s[] = {" % name)
+    for i in range(0, len(flat), 4):
+        out.append("    " + ", ".join("%.9gf" % v for v in flat[i:i + 4]) + ",")
+    out.append("};")
+
+
+def case(tag, hidden, hc, lowrank, eps, tokens, use_combine, seed):
+    g = torch.Generator().manual_seed(seed)
+    cfg = Qwen4ExpTextConfig(hidden, hc, lowrank, eps)
+    mod = Qwen4ExpTextGatedResidual(cfg, use_combine=use_combine)
+    hc_h = hc * hidden
+    with torch.no_grad():
+        # hc_norm.weight is ZERO-init upstream; a zero weight makes (1 + w) == 1
+        # and would hide the parameterization entirely, so it is randomized.
+        mod.hc_norm.weight.copy_(torch.randn(hc_h, generator=g) * 0.5)
+        mod.input_mix_weight_down.weight.copy_(torch.randn(lowrank, hc_h, generator=g) * 0.3)
+        mod.input_mix_weight_up.weight.copy_(torch.randn(hc_h, lowrank, generator=g) * 0.3)
+        if use_combine:
+            mod.block_inject_weight.weight.copy_(torch.randn(hc, hc_h, generator=g) * 0.3)
+        hyper = torch.randn(tokens, hc_h, generator=g) * 1.7
+        block_out = torch.randn(tokens, hidden, generator=g) * 0.9
+
+        normed = mod.hc_norm(hyper)
+        res = mod(hyper)
+        if use_combine:
+            mixed, hyper_ret, inj = res
+            # NOT an `assert`, for the same reason as the sha guard above: this
+            # is the only check that upstream still hands back the RAW input for
+            # the write-back, and `python3 -O` strips an `assert`.
+            if hyper_ret is not hyper:
+                raise SystemExit("upstream must return hyper_input RAW")
+            injection = block_out.unsqueeze(-2) * inj.unsqueeze(-1)
+            written = hyper + injection.flatten(-2)
+        else:
+            mixed = res
+
+    out.append("")
+    out.append("// ---- %s: hidden=%d hc=%d lowrank=%d eps=%g T=%d use_combine=%s seed=%d"
+               % (tag, hidden, hc, lowrank, eps, tokens, use_combine, seed))
+    emit("k%s_norm_w_hf" % tag, mod.hc_norm.weight)
+    emit("k%s_down" % tag, mod.input_mix_weight_down.weight)
+    emit("k%s_up" % tag, mod.input_mix_weight_up.weight)
+    if use_combine:
+        emit("k%s_inject" % tag, mod.block_inject_weight.weight)
+    emit("k%s_hyper" % tag, hyper)
+    emit("k%s_normed" % tag, normed)
+    emit("k%s_mixed" % tag, mixed)
+    if use_combine:
+        emit("k%s_block_out" % tag, block_out)
+        emit("k%s_inj_w" % tag, inj)
+        emit("k%s_written" % tag, written)
+
+
+out.append("// GENERATED by scripts/gen-qwen4-exp-hc-goldens.py -- do not hand-edit.")
+out.append("// Oracle: transformers v5.16.0 modeling_qwen4_exp.py")
+out.append("//   sha256 %s" % EXPECT_SHA)
+out.append("//   Qwen4ExpTextRMSNorm  :158-181   (grouped RMSNorm, (1.0 + weight))")
+out.append("//   Qwen4ExpTextGatedResidual :941-969")
+out.append("//   write-back: Qwen4ExpTextDecoderLayer.forward, the two lines")
+out.append("//     injection = hidden_states.unsqueeze(-2) * injection_weights.unsqueeze(-1)")
+out.append("//     hidden_states = hyper_input + injection.flatten(-2)")
+out.append("// torch %s" % torch.__version__)
+
+case("A", hidden=6, hc=4, lowrank=5, eps=1e-6, tokens=3, use_combine=True, seed=1234)
+case("B", hidden=5, hc=3, lowrank=7, eps=1e-5, tokens=2, use_combine=True, seed=99)
+case("C", hidden=6, hc=4, lowrank=5, eps=1e-6, tokens=2, use_combine=False, seed=7)
+
+open(sys.argv[1], "w").write("\n".join(out) + "\n")
+print("wrote", sys.argv[1])