feat(MODEL-MM-QWEN4-EXP): W3 — the 4-branch gated-residual hyper-connection stream and its grouped RMSNorm - #2045
Merged
Merged
Conversation
…PED norm and writes a rank-1 update (#1988) Qwen4-Exp carries its residual as `hc_count * hidden_size` = 4 x 2560 = 10240 floats through all 48 layers, reads and rewrites it twice per layer, and collapses it at the end with the same class that gates it. This wave lands the host reference for that class and for the grouped RMSNorm underneath it. It is written against two oracles because the row has two, by developer direction: transformers v5.16.0 supplies the algorithm (vLLM has never registered `qwen4_exp` at any revision), vLLM supplies the op form. The grouped norm therefore mirrors `RMSNormGated(group_size=)` — not the plain `RMSNorm`, whose only related knob is `var_hidden_size` and which has no group concept at all — while its semantics come from `Qwen4ExpTextRMSNorm`. The reason this needs a gate rather than a careful reading is that four of its guarantees are one-character defects with no visible symptom: * The `1 + w` parameterization. transformers scales by `(1.0 + weight)` on a ZERO-init weight; vLLM scales by `weight` on a ONES-init one. They agree only under a load-time `w = 1.0 + w_hf`. Skip it and every `hc_norm` scales by ~zero, which reads as a broken checkpoint rather than a broken port; apply it twice and it scales by ~2x. It now lives in exactly one named function, `HcNormWeightFromHf`, so the question "has the 1 already been added?" is about one call site instead of about arithmetic inside a kernel. * The division by `hc_count` inside the SiLU sits BEFORE the activation. `silu(x/4)` is not `silu(x)/4`, and on this data the two answers differ by as little as 6.6e-3 — under a percent, and invisible to a loose comparison. * There is no such division on the up-projection sigmoid, and the reduce over the four streams is a MEAN and not a sum. * The elementwise multiply uses the NORMED stream while the write-back adds to the RAW one, so the two ends of the same function disagree about which copy they hold on purpose. The goldens are not derived, hand-computed, or transcribed. `Qwen4ExpTextRMSNorm` and `Qwen4ExpTextGatedResidual` are lifted verbatim by line range out of the lane-pinned oracle file, its sha256 re-checked at dump time, and EXECUTED; the generator is committed and reproduces the .inc byte for byte. A second, independent double-precision reference then reproduces the same numbers, and a flag-per-defect sweep proves each of the five traps opens a gap the goldens can actually see — measured, tabulated in the test, and floored at 6.6x under the smallest of them. The write-back is deliberately an in-place rank-1 update rather than the broadcast-and-multiply upstream spells. That shape is the fusion seam: both llama.cpp implementations of this architecture materialise it as `repeat_4d` + `mul`, which is 96 dense [2560, 4, T] tensors built and discarded per forward pass at 48 layers x 2 sites. Keeping the pointer form as the primitive means a device kernel replaces this one function and nothing above it moves. No speed is claimed here; no arm of this model runs on any device we own. The spec's claim that DeepSeek-V4's `MhcPost` is a bit-exact bring-up bridge for this write-back under an identity comb matrix is verified rather than repeated, by a bit-equality case against that kernel. It holds; it costs hc-squared work where hc suffices, and it is not an identity for a negative-zero or non-finite residual, neither of which this path reaches. NOT REACHED at this commit, per AGENTS.md "Nothing lands dead": W1 config registration (#1986) is still in review, so no `qwen4_exp` resolves through the loader and nothing production-side calls this yet. The wiring is owed by W5, assembly, under #1978, and the debt is recorded under the spec's `## Owed` together with the model-matrix cell that W1 owns. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code]
…eted refusals, or the model's own width (#1988) Fresh review of a5bd413 returned FAIL with eight findings, and its own summary is the shape of this change: "the port itself is the strongest part of this change and I could not break it ... Every finding is about the GATE, not the arithmetic." `qwen4_exp_hc.cpp` is byte-for-byte unchanged here and the header change is comments only. What changes is what the suite can detect. Issue #449, re-opened locally. `test_qwen4_exp_hc.cpp` defined two local max|diff| helpers spelled `worst = std::max(worst, std::abs(...))` — verbatim the NaN-blind "form B" that `tests/support/max_abs_diff.h` exists to prevent, in a file that already included that header. Reproduced before repairing: poisoning `GroupedRmsNorm`'s large-eps path to all-NaN left `CHECK(MaxAbsDV(big_eps, ...) < kTol)` GREEN, and the suite only went red because the sibling assertion one line up happened to use the hardened helper. The fix is not a third local copy. `MaxAbsDiffScan` and `MaxAbsDiff` are now templated on both operand types, so an fp32-result-against-double-reference comparison — the shape every "second independent reference" case takes — routes through the same hardened scan instead of being written again by hand. The same mutation now reddens the assertion itself. Eight of eleven refusals had no assertion at all. Each was deleted in turn and the suite stayed 280/280 green each time, including all three `RequireSize` calls in `GatedResidualWriteBack` — which, since `GatedResidualWriteBackInPlace` takes raw pointers and the header declares it the seam a device kernel replaces, are the only bounds check between a caller and an out-of-bounds read. That class of guard does not fail tidily: deleting the divisibility guard produces `Fatal glibc error: malloc.c:2599` and `Aborted (core dumped)` with no doctest summary at all, and deleting the `group_size` guard is `x.size() % 0` and SIGFPE. All eight are now message-pinned with `CHECK_THROWS_WITH_AS`, because every guard in the translation unit throws `std::invalid_argument` and a type-only assertion cannot tell a deleted guard from a different one firing further down. `kTol = 1e-5` was a property of the toy shapes and nothing else, and this is the part the review ranked MEDIUM that deserved the most work. At flat = 24 and 15 the implementation is bit-identical to the oracle — 2.384e-07 over every golden array — so the tolerance was unconstrained there. Measured against the pinned oracle at the model's own shape (hidden 2560, hc 4, lowrank 320), max|diff| on `mixed_input` is 2.325e-05 and 2.137e-05 over two tokens, 2.1x to 2.3x over. The reflex is to widen kTol, which AGENTS.md forbids and which would be wrong anyway. The measurement that settles it is the second row of the table now in the file: an EXACT double evaluation of the same algorithm is itself 1.360e-05 from the oracle, because torch runs this in fp32 too. No fp32 implementation meets a 1e-5 absolute bound at hidden_size 2560, and widening our own accumulator cannot rescue one — so the arithmetic is not the thing that is wrong at that width, the absolute tolerance is. kTol is unchanged and is now documented as scale-bound; a real-width case carries a RELATIVE bound derived as 6.6x the sqrt(K)*u random-walk bound for a sequential fp32 dot of K = 10240, and all three measurements sit inside it. Agreement with the oracle at model width needs a real checkpoint and is recorded as owed rather than claimed. The `double` accumulator was documented and ungated: `double ss` to `float ss` left 280/280 green, so the gate could not distinguish the convention from its opposite. It is kept, because it makes the host reference more accurate than the oracle rather than less, and it is now gated at the model's real group size of 2560 on magnitude-separated data, where the two differ by 742x (3.168430e-06 against 2.352230e-03, bound 1e-4, identical at -O0 and -O2). The consequence for the device wave is stated in the spec rather than left to be discovered: a straight fp32-accumulate device reduction will not meet that bound on that data, and that is the signal, not a defect in the gate. The GGUF fold claim was true and under-scoped in the way that causes the defect it warns about. Read at source rather than relayed: llama.cpp mainline has no `qwen4exp` at all, the converter is the still-open ggml-org/llama.cpp#27742 at 035e22731a7fd70b9854b3a2d64ec68e9b1a45d3, and it adds no `hc_norm` branch — the `+1` comes from the inherited `Qwen3NextModel` rule at `conversion/qwen.py` :302-303, `endswith("norm.weight") and not endswith("linear_attn.norm.weight")`. So the fold covers `hc_norm`, `attn_q_norm`, `attn_k_norm` and the PLE and indexer gammas, and excludes `ssm_norm`. A W5 loader author reading the old sentence would have skipped the fold for `hc_norm` and double-folded everything else. The header and the spec now state the rule, its exception, the PR and its head SHA, and that it is unmerged and may change. Three smaller ones. "Bit for bit" was asserted with `==`, which cannot see the negative-zero divergence the comment beside it names; both sites now compare bit patterns, and a case pins the helper's polarity. The in-place-versus-allocating check was a tautology — the wrapper calls the primitive — proven by mutating the primitive and watching that line stay green while three others reddened; the seam is now gated against the oracle directly. `Variant::norm_weight_plus_one` was declared and read but never once set false, so the struct advertised coverage the file did not have; it is now flipped in the sweep. And the goldens generator pinned its oracle with a bare `assert`, which `python3 -O` strips. Verified both ways: under `-O` the old script happily dumped goldens from a corrupted source file at rc=0, the new one exits 1. The generator still reproduces the committed `.inc` byte for byte. `.agents/issue-index.md` is append-only, so the under-scoped fold sentence in the #1988 row stays where it is; the header and the spec are the surfaces a loader author reads and both are corrected there. Still NOT REACHED at this commit, unchanged from a5bd413: the wiring is owed by W5, assembly, under #1978, and is recorded under the spec's `## Owed`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code]
…nned llama.cpp, not a repudiated local commit CORRECTION TO 09ce43f. That commit, its header passage and the spec all attributed the Qwen3-Next `+1` fold rule to llama.cpp "mainline at `conversion/qwen.py:302-303`", read at `237ad9b96`. `237ad9b96` is not mainline. It is reachable only from the local branch `localai-paged` in the developer's llama.cpp checkout, it has no merge base with `ggml-org/llama.cpp` `origin/master`, and `.agents/oracles/llama-cpp.md` already records it as the retired, contaminated pin: 65 of this project's own performance commits past upstream tag `b9827`, built from a working tree carrying 27 uncommitted entries. The live pin is stock upstream tag `b10451`, `10bf611e533d81f739128304991c5e133c6aebd8`. The three line anchors were wrong at every ggml-org revision as well, so a W5 loader author following that passage would have landed on unrelated code in a tree they cannot fetch. Everything the passage asserts remains TRUE, and it is re-anchored rather than rewritten. Read from `git archive b10451` (the working checkout is dirty, so the tag object is the only honest source): `git grep -il qwen4exp` still returns nothing tree-wide, so a released llama.cpp can neither convert nor load this architecture. The `+1` rule and its `linear_attn.norm.weight` exception are at `conversion/qwen.py:387-388`; `_LinearAttentionVReorderBase` is `:438`; `Qwen3NextModel` is `:365`, and its signature is now `class Qwen3NextModel(_QwenMtpMixin, Qwen2MoeModel)`, which the old quotation had stale. PR ggml-org/llama.cpp#27742 is still OPEN and unmerged at head `035e22731a7fd70b9854b3a2d64ec68e9b1a45d3` (`refs/pull/27742/head` and a live `/merge` ref; the head is not an ancestor of `origin/master`), and its `modify_tensors` still carries no `hc_norm` branch, so `hc_norm.weight` falls through to `super()` and the fold covers `attn_q_norm`, `attn_k_norm` and the PLE/indexer gammas while sparing `ssm_norm`. That is the whole point of the passage: skip the fold for `hc_norm` alone and every other norm is double-folded. The word "mainline" is gone, and `237ad9b96` is cited nowhere in it. Three smaller repairs ride along. The claim that "the ORACLE is itself 1.36x over kTol" was one draw of random inputs stated as a general fact; an independent draw gives 0.91x and 0.82x. Both the test comment and the spec now say the oracle sits of the same ORDER as kTol, name both draws, and label the measurement table as one draw. The conclusion is unchanged and holds on either draw: no fp32 implementation of this function meets a 1e-5 ABSOLUTE bound at hidden_size 2560, because torch runs the reduction in fp32 too, so widening our own accumulator cannot rescue it. `scripts/gen-qwen4-exp-hc-goldens.py` still guarded "upstream must return hyper_input RAW" with a bare `assert`, one line away from the sha256 pin guard fixed for exactly this reason. `python3 -O` strips it. Proven rather than argued: with the oracle corrupted to return `hyper_input * 2` and the sha guard retargeted in a scratch copy so it could not fire first, the old script under `python3 -O` exited 0 and wrote a golden file that differs from the committed `.inc`. It is a `raise SystemExit` now, and the same experiment exits 1 with no output file under both `python3` and `python3 -O`. `tests/support/test_max_abs_diff.cpp` exercised only `float`/`float`, so the double-sided instantiation of the helper was covered only by its one caller. Two cases now compare a `std::vector<double>` reference against an fp32 golden through both the scan and both wrapper overloads, using a difference of 2^-30 that exists in double and vanishes in float. Narrowing the operands to float inside the scan reddens those cases and nothing else; the pre-widening header refuses to compile them at all. `qwen4_exp_hc.cpp` is untouched, the goldens reproduce byte-for-byte from the pinned oracle under both `python3` and `python3 -O`, and `test_qwen4_exp_hc` is unmoved at 14 cases / 230 assertions. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code]
The branch was ten commits behind `origin/main`, and `scripts/agent-preflight.sh` answers that state by SKIPPING `commit-trailers` and `commit-style` rather than running them, so a preflight read by exit code alone would have called this branch green while neither gate had said anything about it. Merged so both gates actually run before the push. The merge is clean and touches no file this row changed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code]
`origin/main` advanced again between the first merge and the rerun of `scripts/agent-preflight.sh`, which fetches for itself, so the trailer gates SKIPPED a second time on the same "branch is behind" reason. Merged again so `commit-trailers` and `commit-style` run against a HEAD that has `origin/main` as an ancestor. Clean merge, no file this row changed is touched. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code]
The fresh scoped re-review measured this branch 3 commits behind origin/main again. That matters beyond tidiness: check-commit-style refuses a range whose base is not an ancestor of its head, and diff-scoped gates SKIP silently under a moved base while still exiting zero, so the branch's own preflight evidence was taken against a base that no longer exists. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
The remote head had gone stale enough that GitHub refused the squash with "the merge commit cannot be cleanly created", while the same three-way merge resolves cleanly in git. The overlap with main is three additive registration files — .agents/issue-index.md, CMakeLists.txt and tests/CMakeLists.txt — which is exactly the shape that automerges into something that does not build, so the merged tree was configured and built rather than assumed: cmake configure rc=0, `test_max_abs_diff` 47/47 and `test_qwen4_exp_hc` 230/230, both SUCCESS. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…oth) W4 landed as 3104a52 and W2 as 583b9a9, so this branch conflicted with its own siblings in two places. Both resolutions are take-both, and both are the shape that automerges into something that does not build, so the result was configured rather than assumed. `tests/CMakeLists.txt`: W3 and W4 each appended a test registration at the same point. Both blocks are kept, W4's first because it is already on main. The two targets are independent — `test_qwen4_exp_qsa` and `test_qwen4_exp_hc`, each with its own model-private src/ grant. `.agents/specs/qwen4-exp-flash-next.md`: both sides append `## Owed` bullets, W4's four and W3's four. All eight are kept. They document different unreached slices and neither supersedes the other. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(MODEL-MM-QWEN4-EXP): W3 — the 4-branch gated-residual hyper-connection stream and its grouped RMSNorm
Qwen4Expdoes not carry a residual ofhidden_size. It carrieshc_count * hidden_size= 4 x 2560 = 10240 through the entire stack, with a low-rank readgate and a per-branch scalar write gate around both attention and the MLP. That
is a change to the per-layer loop and to every residual buffer, not a drop-in
module, which is why it is its own wave.
This lands the stream and its grouped RMSNorm against a running transformers
5.16.0 oracle, with goldens that regenerate byte-identical from the pinned
oracle under both
python3andpython3 -O.Nothing is reachable yet. No
ModelRegistryentry resolvesQwen4ExpForConditionalGeneration, so this slice is unreached by design; thewiring is owned by W5 (#2031)
and is listed under
## Owedin the row spec.The GGUF fold guidance is anchored at the pinned oracle
The header passage that tells a W5 loader author how the
+1norm fold works isanchored to stock upstream llama.cpp at tag
b10451(
10bf611e533d81f739128304991c5e133c6aebd8), read fromgit archive b10451because the working checkout is dirty and the tag object is the only honest
source.
git grep -il qwen4expreturns nothing tree-wide at that tag, so areleased llama.cpp can neither convert nor load this architecture.
The
+1rule and itslinear_attn.norm.weightexception are atconversion/qwen.py:387-388;_LinearAttentionVReorderBaseis:438;Qwen3NextModelis:365. PR ggml-org/llama.cpp#27742 is still OPEN andunmerged at head
035e22731a7fd70b9854b3a2d64ec68e9b1a45d3, and itsmodify_tensorscarries nohc_normbranch, sohc_norm.weightfalls throughto
super()and the fold coversattn_q_norm,attn_k_normand thePLE/indexer gammas while sparing
ssm_norm. Skip the fold forhc_normaloneand every other norm is double-folded.
An earlier revision of this branch attributed that rule to llama.cpp "mainline"
at
conversion/qwen.py:302-303, read at237ad9b96. That commit is notmainline: it is reachable only from a local branch in the developer's checkout,
has no merge base with
ggml-org/llama.cpp, and.agents/oracles/llama-cpp.mdrecords it as the retired contaminated pin. The anchors were wrong at every
ggml-org revision too, so an author following the passage would have landed on
unrelated code in a tree they cannot fetch. Fresh review caught it; the
assertions all survived re-verification at
b10451and were re-anchored ratherthan rewritten.
Two defects the repair found on its own
tests/support/max_abs_diff.hnarrowed both operands to float, so astd::vector<double>compared against an fp32 golden lost any differencesmaller than fp32 resolution. Proven red-first with a 2^-30 difference that
exists in double and vanishes in float:
CHECK( 0 == 9.31323e-10 ). 7 cases /33 assertions -> 9 cases / 47 assertions.
scripts/gen-qwen4-exp-hc-goldens.py:97guarded the oracle's aliasing contractwith a bare
assert, whichpython3 -Odeletes. With the oracle corrupted andthe sha guard retargeted so it could not fire first, the old form exits rc=0
and writes a corrupted golden under
-O; the newraise SystemExitexits 1and writes nothing under both interpreters.
Evidence
test_qwen4_exp_hc: 14 cases / 230 assertions, SUCCESS77fec77d...c459)scripts/agent-preflight.sh: rc=0, all gates green, zero FAIL and zero SKIP;commit-trailersandcommit-styleboth RAN againstorigin/mainCloses #1988
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]