diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 06ade3ceb..ef88b5f90 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -748,6 +748,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#526](https://github.com/mudler/vllm.cpp/issues/526) | `SERVE-TOOL-HISTORY-ARGS` | OpenAI multi-turn tool history reaches chat templates with string-valued arguments | bug | | [#1934](https://github.com/mudler/vllm.cpp/issues/1934) | `BACKEND-ROCM` | `RocmPlatform::needs_weight_staging()` is stale-false (a W0-era placeholder never revisited despite #523/#509/#506/ROCM_ATTN/hipGraph landing since), so `CheckDeviceWeightFit` — the #1123/#1870 load-time refusal, including the `policy_forces_full_expand` fix — never runs on ROCm: measured directly, `VT_DEVICE_WEIGHT_BUDGET_BYTES=1` produced no refusal on a real load. The actual device allocation the refusal guards is not gated on this flag, so #1870's crash stays reachable until this closes; owed, not fixed in flow, because flipping the flag also moves `DirectDeviceLoadEligible` and several GDN kernel-dispatch defaults that each need their own correctness check | bug | | [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-QWEN4-EXP` | **`Qwen/Qwen3.8-Flash-Next` declares `Qwen4ExpForConditionalGeneration` / `qwen4_exp`, a new architecture vLLM does not implement, so the port runs on a split oracle: transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH.** Released 2026-08-24, 180B total / 6B activated, image-text-to-text. The `Qwen3.8` in the name is marketing continuity: `.agents/specs/qwen38-27b-bf16-gate.md`'s "one config key differs" precedent does NOT extend here. Read live 2026-08-26 at vLLM `origin/main` = `6a5e8f5979`: no `qwen4*` path, no registry entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. That is absence from vLLM `main` rather than staleness in our pin `555967922`, so a pin advance does not reach it. What exists is transformers [#48337](https://github.com/huggingface/transformers/pull/48337) "Add Qwen4Exp model", MERGED 2026-08-26, 5211 lines, and SGLang [#36497](https://github.com/sgl-project/sglang/pull/36497), still OPEN and therefore inadmissible. **Developer direction 2026-08-26, recorded verbatim: "use transformers as oracle for algorithmic side. but use ops from vllm so we account for optimized path."** Justified rather than convenient: `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so porting it as written yields a correct model at an indefensible speed, while AGENTS.md's mirror-vLLM polarity still binds every primitive vLLM implements. `Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`), all of which this tree has; GDN is an exact match for our AOT gate (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). The delta is four things, and **exactly two have no vLLM op at all**: the PLE dilated depthwise conv (kernel 4, dilation 3; `git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the n-gram hashed embedding. **The survey's load-bearing finding, and it REVERSES this row's first reading: QSA's structural twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The original call was that QSA, being plain GQA rather than MLA, had to map onto vLLM's non-MLA block-sparse case; that reasoning rested on treating `MLAAttentionSpec` as an MLA claim, and **it is not one** — M3's own indexer cache uses it while M3 is a plain-GQA model, with the comment "Key-only: MLAAttentionSpec budgets one vector/token (not 2x for K+V)". It is a budget shape. Remove that prop and the GQA-vs-MLA argument collapses. Verified line by line at `6a5e8f5979`: **nine independent structural matches with DSv4**, `compress_ratio == 4` literally the same number — MQA index with 1 key head at dim 128; `relu(q.k)` summed over index heads vs `(score.relu() * weights).sum(dim=0)`; `1/sqrt(head_dim)`; one score set per query token with no head axis vs `topk_indices_buffer[num_tokens, topk]`; pooling boundary `(position+1) % COMPRESS_RATIO == 0`; RMSNorm on the pooled key; **RoPE at the block-start position** vs `compressed_pos = (position // CR) * CR`; candidate count `visible // compress_ratio`; and one stored state per 4 tokens via `MLAAttentionSpec(tokens_per_state=compress_ratio)`, a first-class KV field documented as "Ints > 1 compress multiple tokens into one state (DSv4 sparse MLA)" that has no M3 equivalent. **M3 is a DIFFERENT ALGORITHM**, not a worse fit: its score is `tl.max(qk, axis=1)` over 128 RAW token dots with no pooling, no relu and no head reduction, it asserts `num_idx_heads == num_kv_heads` ("no topk index reduce") so it emits one block set PER KV HEAD, and its `SPARSE_BLOCK_SIZE = 128` is welded to the KV page size ("One sparse block == one KV page") on both the score and the attend side — moving it to 4 forces a page size of 4 and breaks `tl.dot`, whose tile needs >= 16. M3 contributes exactly ONE thing and it is a wiring precedent, not an algorithm: that a plain-GQA model can own a key-only side cache through `MLAAttentionSpec` and a private indexer backend. **The genuinely new work is the CONSUMER and nothing upstream supplies it** — every DSv4 sparse consumer attends to COMPRESSED MLA KV (one state per 4 tokens) and M3's attend to raw tokens only at page granularity, while QSA attends to RAW tokens selected at ratio-4 granularity. Two silent-failure traps follow: wiring QSA's top-k into a DSv4 sparse-MLA consumer attends a POOLED key/value and still emits plausible tokens, and **a short-prompt token gate cannot catch it because at context <= `indexer_budget` 2048 every candidate is selected** — so any QSA gate must run past 2048 tokens of context, which is now a stated `## Gates` requirement; and `SparseAttnCompressNormRopeStoreC4Kernel` does NOT mean-pool despite its name — it is a learned softmax pool over an OVERLAPPING window of 8 using a score channel this checkpoint does not have, and the CuteDSL variant refuses `overlap=False` at compile, so the **Triton** `head_dim=128` variant is the correct starting point. Two structural consequences beyond the module list: the residual stream is `hc_count * hidden_size` = **4 x 2560 = 10240 wide through the whole stack** with a low-rank read gate and per-branch scalar write gate around both attention and MLP, which is a change to the per-layer loop and every residual buffer rather than a drop-in module; and `number_of_conv_states = 3` on a PLE layer (GDN conv, PLE conv, n-gram token history) plus the indexer side cache, adjacent to [#1963](https://github.com/mudler/vllm.cpp/issues/1963) and [#1966](https://github.com/mudler/vllm.cpp/issues/1966). **NOTHING PUBLISHED FITS**, read live from the HF API against ~119 GB usable on GB10: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB (NVFP4 backbone with the n-gram table left at FP8, 51.2 GB) and `unsloth/...-GGUF` is a README with ZERO weight files. No GGUF exists and no tool can make one, because llama.cpp has no `qwen4_exp` either, so the standing k-quant requirement means authoring the arch on our side AND states that the quantized arms have NO llama.cpp oracle. **The chosen arm does NOT load today, and the blocker is neither the offload nor the budget: this tree cannot keep a gather table quantized at all.** `KeepQuantKDim` returns `-1` for `GgufTensorRole::kEmbeddingTable` (`src/vllm/model_executor/model_loader/gguf_keep_quant.cpp`), and `qwen3_5_gguf_weights.cpp` asserts it by name — "the embedding table cannot keep quant blocks" — so a Q4_K or Q8_0 n-gram table EXPANDS to bf16 and 51.2B params become **102.4 GB of anonymous memory**; the arm dies before the first forward. The reason was already sitting in a header comment ("a gather, not a GEMM ... A quantized-gather op is a follow-up row") and **no such row exists**. The only non-expanding gather residency is `kKeepF16`, requiring ggml type 1 exactly (102.4 GB on disk) and CPU-ONLY, because `EmbeddingKernelCuda` refuses anything but f32/bf16. **Second blocker:** `moe_intermediate_size = 640` makes `ffn_down_exps` Q4_K-illegal on its reduction dim (640 % 256 = 128), as does `hc_lowrank = 320`; llama.cpp's substitution is believed to be Q5_0 (**UNVERIFIED, owed against the pinned llama.cpp oracle**) and the dependent fact IS verified in-tree — our reader knows ggml ids `0,1,2,8,10..14,16,18,19,22..28,30,39,40,41,66` and has **no entry for 3, 6, 7 or 20**, so a stock `llama-quantize -Q4_K_M` file fails at header parse. We author the converter, so the fix is Q4_0 (block 32, same 4.5 bpw). **`ENG-WEIGHT-OFFLOAD` will not help** — it moves zero bytes today (`ConsiderWeight` has no production callers, pinned by a test) and is documented inert on GB10; the tier that DOES work already ships and is proven by the 2.4T model serving 369.97 GiB from a 119.631 GiB box at ~62 GiB resident: mmap `MAP_PRIVATE`, borrow in place, alias the host pointer, `prefault: false`. Corrected sizing: backbone ~67.7 GiB, whole process ~73.5 GiB of 119.631 at 32K single-stream, ~46 GiB of headroom for the page cache, so the ~76 GB estimate was right within 10%. The design works because per-token demand is **<= 64 KiB of reads** (16 lookups x 160 dims over at most 16 pages) against the 2.4T expert lane's 6.95 GB/token. The architecture supplies its own lever: the per-token n-gram cost is `(ngram_size-1)*heads_per_ngram` = 16 lookups of 160 dims, so **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per token** and making it non-resident is the intended design point (RadixArk reached the same split independently). Sizing arithmetic, NOT measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB (yes, ~10 GB left for KV), Q4_K_M backbone with the table non-resident ~76 GB. GB10 is UNIFIED memory so "offload to host" is not a move there; non-resident means disk-backed, and its cost is unmeasured. **Two decisions were put to the developer as explicit accept-or-reject and BOTH are settled 2026-08-26, recorded in place rather than left open.** (1) `.agents/oracles/transformers.md` pins transformers to 5.14.1, deliberately tied to what the pinned vLLM environment resolves so the environment cannot hold two `transformers` at once, and **5.14.1 does not contain `Qwen4Exp`**; the lane-scoped second pin is **ACCEPTED**, on the argument that the invariant guards a vLLM environment against drifting from its transformers and here there is no vLLM implementation to drift from, and it expires the moment vLLM registers `qwen4_exp`. **The lane pin is a real release, not a branch SHA**, which was not the expected outcome: `Qwen4Exp` merged to `main` at 12:03:40Z on 2026-08-26 and `v5.16.0` published at 12:35:15Z, and this was BOUNDED rather than assumed by fetching `models/qwen4_exp/modeling_qwen4_exp.py` at each tag — `v5.16.0` HTTP **200**, `v5.15.0` HTTP **404** — making 5.16.0 the FIRST release carrying the architecture and therefore the tightest available pin. The version string is UNMEASURED (it is the release proven to contain the model, not a `transformers.__version__` read off a running oracle) and `gateable` stays `no`. (2) The first runnable arm is the **Q4_K_M backbone with the n-gram table NON-RESIDENT** (~76 GB). Q8_0 was raised and does not fit at ~191 GB, and no partial-Q8 split reaches 119 GB with the backbone at 8 bits; Q4_K_M-throughout fits on paper at ~109 GB but leaves ~10 GB for KV and activations on a 262144-native-context model, which is not a margin. This promotes the non-resident table from a note to a first-class W6 deliverable, and it is NOT free: GB10 is UNIFIED memory, so the existing host-pinned offload seam (`ENG-WEIGHT-OFFLOAD`, mirroring vLLM's `cpu_offload_gb`) does not by itself solve it there and the mechanism must be disk-backed or genuinely unloaded — established before it is designed around. Spec: [`specs/qwen4-exp-flash-next.md`](specs/qwen4-exp-flash-next.md). No product code lands under the spec pull request | feature | +| [#1987](https://github.com/mudler/vllm.cpp/issues/1987) | `MODEL-MM-QWEN4-EXP` | **W2 of [#1978](https://github.com/mudler/vllm.cpp/issues/1978): the hashed n-gram embedding and the PLE dilated depthwise conv, the ONLY two components of `Qwen4ExpForConditionalGeneration` with no vLLM op at all.** The negative is confirmed rather than unfound — at vLLM `origin/main` = `6a5e8f5979`, `git grep -in dilat` returns ZERO lines in `layers/mamba/`, zero in `csrc/` and zero in `tests/`, and `layers/conv.py` defines only `Conv2dLayer`/`Conv3dLayer` — so transformers **v5.16.0**, this row's accepted lane pin, is the sole oracle. Three silent divergence sites, all verified: `_splitmix64` must be `uint64_t` throughout (its `>> 30/27/31` are LOGICAL shifts on a non-negative Python int; on `int64_t` they go arithmetic and the multiplicand is wrong, with the top bit set about half the time), `_splitmix64(value) % half_bound` must be an UNSIGNED modulo (the dividend routinely exceeds 2^63), and shard reassembly is NUMERIC not lexicographic (`shard_0, shard_1, shard_10, ...` silently permutes a 95 GiB table). The forward is int64-exact and needs no bignum because `multiplier_max * vocab_size <= 2^63 - 1` — but **that bound holds only while every token id is below `vocab_size`**, so an out-of-range id overflows and diverges in silence, and this wave refuses one by name. Goldens confirmed a FOURTH way beyond #1987's three: `vocab_size = 248320`, read from the released `config.json`, is the UNIQUE preimage below 2e6 of the published `layer_multipliers = [23703573157769, 20109073645365, 8052911324071]` at the absent-`seed` default 1234. PLE specifics: the conv state is **9 columns** — `(kernel-1) * dilation`, NOT `kernel-1` — and holds the **normed** conv input while the skip term is the **un-normed** copy; the signed-sqrt gate clamps BEFORE the sqrt so the output magnitude floor is `sqrt(1e-6) = 1e-3` and tiny scores are AMPLIFIED, with exactly zero mapping to zero, making the function genuinely discontinuous at an origin that a fully masked row reaches; and the n-gram history must be padded with EOS because upstream's `update_conv_state` pads with 0, a valid token id. **Lands UNREACHED** per AGENTS.md "Nothing lands dead": no production entry point calls it until W5 assembles the model, the wiring is owned by `MODEL-MM-QWEN4-EXP` and tracked by #1978, and the spec lists it under `## Owed`. Host reference only; the batched device arm is owed, and the per-sequence signatures exist so it drops in — the gather is 16 uncoalesced random rows per token PER SEQUENCE, so at batch B it is 16*B independent gathers with no coalescing. No token claim, no speed claim: `## Gates` admits none from this row until an arm runs, and nothing published fits any fleet device | feature | | [#1963](https://github.com/mudler/vllm.cpp/issues/1963) | `ROAD-V1-MEM` | At ctx=32768 `--max-num-seqs 32` our engine consumes ~108 GB during load and never serves; vLLM and SGLang both serve there. Root cause found and fixed by `FIX-KV-GROUP-LAYER-COUNT` ([spec](specs/kv-group-layer-count.md)): thirty-three of thirty-four registries publish ONE placeholder name per KV group, `KVBytesPerBlock` reads `layer_names.size()` as the layer count, and `ResolveNumBlocks` arm 2 therefore divides an absolute `--kv-cache-memory` budget by ONE layer's page while the runner allocates one buffer per layer — measured 8.5 GiB allocated for a 1 GiB budget on the 27B | bug | | [#1966](https://github.com/mudler/vllm.cpp/issues/1966) | `ROAD-V1-MEM` | The #371 recurrent-state OOM guard is 48x under and passes a config that allocates 43.4 GiB, because `recurrent_state_bytes` counts placeholder layer names. Same root defect as [#1963](https://github.com/mudler/vllm.cpp/issues/1963), different code path; both fixed by `FIX-KV-GROUP-LAYER-COUNT` ([spec](specs/kv-group-layer-count.md)) | bug | | [#1982](https://github.com/mudler/vllm.cpp/issues/1982) | `SERVE-STREAM-USAGE` | **`ChatSseStream::next` writes the `/v1/chat/completions` role frame before it reads anything from the engine, so `vllm bench serve --backend openai-chat` stamps TTFT on an empty frame and our TTFT through that harness is an HTTP round trip, not a time to first token.** Upstream builds the role chunk under `if first_iteration:` inside `async for res in result_generator:` (`vllm/entrypoints/openai/chat_completion/serving.py:477,487`) and says why at `:484-486`: an exception in the generator "needs to be sent as the FIRST response". `vllm/benchmarks/lib/endpoint_request_func.py:404-408` guards on the presence of `choices`, not on non-empty `delta.content`, and our role frame carries `delta.content = ""` with no `usage`. vLLM and SGLang order the frame after the first result, so their rows on the same harness are honest and only ours is not; this blocks the #1574 three-engine TTFT row. `.agents/specs/stream-options.md` scoped the buffering to continuous usage on purpose and both its passages are corrected here. Fixed by removing the `usage_.include_continuous_usage` guard around the first-result buffering loop, so the default path buffers too. Spec: [`specs/chat-role-frame-ordering.md`](specs/chat-role-frame-ordering.md) | bug | diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index 02aadc606..b58a12d41 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -737,6 +737,40 @@ change that makes any arm reachable, not later. - GGUF k-quant arms, including authoring the `qwen4_exp` architecture on our side, and the statement that no llama.cpp oracle exists for them. - MTP depth > 1. +- **W2 (#1987) lands UNREACHED, by AGENTS.md "Nothing lands dead".** + `src/vllm/model_executor/models/qwen4_exp_ple.{h,cpp}` is a host reference + for the n-gram hashed embedding and the PLE dilated depthwise conv. No + production entry point calls it: `qwen4_exp` has no registry entry, no + loader and no `ModelRegistry::Forward` arm until W5 assembles the model. + The wiring is owned by row `MODEL-MM-QWEN4-EXP` (W5) and tracked by + campaign issue [#1978](https://github.com/mudler/vllm.cpp/issues/1978). + Also owed from that wave: the batched device arm (the host signatures are + per-sequence precisely so it drops in), the 128-shard NUMERIC table + reassembly, and the prefix-caching decision for a conv state written by a + chunked prefill shorter than 9 columns, which `## Design` records as + AMBIGUOUS and not resolvable from upstream. +- **W2's float path has never been compared at MODEL WIDTH, and that is the one + gap its own gate cannot close.** `tests/vllm/models/test_qwen4_exp_ple.cpp` + runs at `hidden_size = 8`, `hc_count = 2`, `heads_per_ngram = 2`, + `ngram_vocab_size_base = 20`. Only the multipliers, the prime head sizes and + the offsets are pinned at the released config, and those are INTEGERS, where + width cannot change an answer. Everything float — the grouped RMSNorm, the + gate reduction that is 2560 wide in the real model, the 10240-channel dilated + conv — is gated at width 16 with 8-wide groups. Every structural mutation in + the W2 table dies there by orders of magnitude, so the instrument is sound for + structure; a REDUCTION-ORDER difference at width 2560 is what it cannot see, + and it is exactly the class of difference that a device arm introduces. + Owed: a first real-width numeric comparison against the lane pin. It must + derive a **relative** bound, not reuse W2's absolute `1e-5`. W3's repair on + the sibling branch measured the reason: an exact-double evaluation of the + oracle's own algorithm for the gated residual already exceeds a 1e-5 absolute + bound at model width, because torch runs the reduction in fp32, so an absolute + bound at that width tests the accumulator and not the port. +- The `conv_mask` contract beyond the host arm. W2 gates the masking itself + (both tensors, and through the 9-column state), but the PAIRED obligation it + documents — a masked position must already carry EOS in `input_ids`, because + the hash reads ids and not activations — is a CALLER obligation with no caller + yet. W5 owns asserting it where the mask is built. - The 1M-token RoPE extension above the native 262144. - The non-resident n-gram table on CUDA: the dequantizing gather op and the `kEmbeddingTable` keep-quant policy change (Route B), and a measurement of the diff --git a/CMakeLists.txt b/CMakeLists.txt index e5ea455ae..af13d8ed8 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -801,6 +801,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/dots3_note_attn.cpp src/vllm/model_executor/models/dots3_note_device.cpp src/vllm/model_executor/models/dots3_note_registry.cpp + src/vllm/model_executor/models/qwen4_exp_ple.cpp src/vllm/model_executor/models/laguna_registry.cpp src/vllm/model_executor/models/laguna_weights.cpp src/vllm/model_executor/models/interfaces.cpp diff --git a/scripts/gen-qwen4-exp-ple-goldens.py b/scripts/gen-qwen4-exp-ple-goldens.py new file mode 100755 index 000000000..832cdd4c6 --- /dev/null +++ b/scripts/gen-qwen4-exp-ple-goldens.py @@ -0,0 +1,417 @@ +#!/usr/bin/env python3 +"""Regenerate tests/vllm/models/qwen4_exp_ple_goldens.inc from the ORACLE. + +Issue #1987, spec `.agents/specs/qwen4-exp-flash-next.md`. + +Oracle: huggingface/transformers **v5.16.0**, this row's ACCEPTED lane pin. It +is the FIRST release containing `qwen4_exp` (`v5.15.0` returns HTTP 404 for the +same path), and vLLM implements NEITHER of the two components this file covers, +so there is no primary oracle to mirror instead. + +NOTHING HERE RE-IMPLEMENTS UPSTREAM. The script downloads the two upstream files +at the `v5.16.0` tag and `exec`s the named line ranges verbatim; only the +scaffolding around them (a config holder, a cache container) is local, and it is +the smallest thing that makes those ranges run. A golden below is therefore an +oracle observation and not a transcription. Where a value upstream computes is +not on a return path -- the n-gram ids -- it is recovered by making the gather +that consumes it invertible, never by re-running the loop that produced it; see +`ngram_ids_for`. + +Needs `torch` and network access. Usage: + + python3 scripts/gen-qwen4-exp-ple-goldens.py +""" + +import io +import math +import pathlib +import sys +import types +import urllib.request + +import torch +import torch.nn as nn +import torch.nn.functional as F + +TAG = "v5.16.0" +BASE = f"https://raw.githubusercontent.com/huggingface/transformers/{TAG}/src/transformers" +MODELING = f"{BASE}/models/qwen4_exp/modeling_qwen4_exp.py" +CACHE_UTILS = f"{BASE}/cache_utils.py" + +OUT_PATH = (pathlib.Path(__file__).resolve().parent.parent + / "tests" / "vllm" / "models" / "qwen4_exp_ple_goldens.inc") + + +def fetch(url): + with urllib.request.urlopen(url) as response: + if response.status != 200: + raise SystemExit(f"{url} returned HTTP {response.status}") + return response.read().decode("utf-8").splitlines(keepends=True) + + +def rng(lines, first, last): + """The upstream lines [first, last], 1-indexed inclusive, verbatim.""" + return "".join(lines[first - 1:last]) + + +SRC = fetch(MODELING) +CACHE = fetch(CACHE_UTILS) + +NS = {"torch": torch, "nn": nn, "F": F, "math": math, + "Cache": object, "Qwen4ExpTextConfig": object} + +# modeling_qwen4_exp.py:158-181 Qwen4ExpTextRMSNorm (the group_size arm) +# modeling_qwen4_exp.py:204-213 apply_mask_to_padding_states +# modeling_qwen4_exp.py:971-1015 _MASK64 .. _find_nth_prime_after +# modeling_qwen4_exp.py:1018-1114 Qwen4ExpTextNGramEmbedding +# modeling_qwen4_exp.py:1117-1189 Qwen4ExpTextPLELayer +for first, last in ((158, 181), (204, 213), (971, 1015), (1018, 1114), (1117, 1189)): + exec(compile(rng(SRC, first, last), f"modeling_qwen4_exp.py@{first}", "exec"), NS) + +_splitmix64 = NS["_splitmix64"] +_build_layer_multipliers = NS["_build_layer_multipliers"] +_find_nth_prime_after = NS["_find_nth_prime_after"] +NGram = NS["Qwen4ExpTextNGramEmbedding"] +PLE = NS["Qwen4ExpTextPLELayer"] + + +class Cfg(types.SimpleNamespace): + """Only the keys the two modules read.""" + + +class _MixinBase: + """The attribute surface of LinearAttentionCacheLayerMixin, nothing more.""" + + def __init__(self, n=3): + self.number_of_states = n + self.conv_states = [None] * n + self.recurrent_states = [None] * n + self.conv_kernel_size = [None] * n + self.is_conv_states_initialized = [False] * n + self.is_recurrent_states_initialized = [False] * n + self.has_previous_state = [False] * n + self.record_past = False + self.device = None + self.dtype = None + + +# cache_utils.py:1003-1075 verbatim: LinearAttentionLayer.lazy_initialization and +# .update_conv_state, which the PLE conv (state 1) and the n-gram token history +# (state 2) both ride on. The zero pad it applies on a short first chunk is +# exactly why the n-gram side EOS-pads explicitly before calling it. +_LAYER_NS = {"torch": torch, "is_torchdynamo_compiling": lambda: True, + "LinearAttentionCacheLayerMixin": _MixinBase} +exec(compile(rng(CACHE, 1003, 1075), "cache_utils.py@1003", "exec"), _LAYER_NS) +CacheLayer = _LAYER_NS["LinearAttentionLayer"] + + +class Cache: + def __init__(self, num_layers=1, n_states=3): + self.layers = [CacheLayer(n_states) for _ in range(num_layers)] + + def has_previous_state(self, layer_idx, state_idx=0): + return self.layers[layer_idx].has_previous_state[state_idx] + + def update_conv_state(self, conv_states, layer_idx, state_idx=0, **kwargs): + return self.layers[layer_idx].update_conv_state(conv_states, state_idx, **kwargs) + + +torch.manual_seed(20260826) +OUT = io.StringIO() +W = OUT.write + +# ----------------------------------------------------------- A. the real config +# `Qwen/Qwen3.8-Flash-Next` text_config.vocab_size, and the dataclass default +# `seed` (config.seed is ABSENT from the published config.json). Together they +# are the UNIQUE preimage below 2e6 of the three multipliers published in #1987, +# which is a fourth confirmation on top of that issue's three. +REAL_VOCAB = 248320 +REAL_SEED = 1234 +real_mults = _build_layer_multipliers(REAL_VOCAB, 3, 0, REAL_SEED).tolist() +sizes = [_find_nth_prime_after(20_000_000 - 1, i + 1) for i in range(16)] +offs, tot = [], 0 +for size in sizes: + offs.append(tot) + tot += size +padded = math.ceil(tot / 128) * 128 + +# The raw chain, so a signed-shift port fails on the value itself and not only +# on a derived multiplier. Half of these have their top bit set. +SPLIT_PROBES = [0, 1, 1234, REAL_SEED + 0x9E3779B97F4A7C15, + 0x9E3779B97F4A7C15, (1 << 63), (1 << 64) - 1, 0xDEADBEEFCAFEF00D] + +W("// GENERATED by scripts/gen-qwen4-exp-ple-goldens.py -- do not edit.\n") +W(f"// Oracle: huggingface/transformers {TAG},\n") +W("// src/transformers/models/qwen4_exp/modeling_qwen4_exp.py\n") +W("// src/transformers/cache_utils.py\n") +W("// Produced by exec'ing the upstream line ranges VERBATIM, never by\n") +W("// transcribing them; see the generator. transformers 5.16.0 is this row's\n") +W("// accepted lane pin (spec `## Oracles`) and vLLM implements neither of the\n") +W("// two components below, so there is no primary oracle to mirror instead.\n\n") + +W("// modeling_qwen4_exp.py:979-983 _splitmix64\n") +W("static const struct { uint64_t in; uint64_t out; } kSplitMix64[] = {\n") +for probe in SPLIT_PROBES: + W(f" {{{probe}ULL, {_splitmix64(probe)}ULL}},\n") +W("};\n\n") + +W("// modeling_qwen4_exp.py:986-995 _build_layer_multipliers, at the REAL config:\n") +W(f"// vocab_size={REAL_VOCAB}, ngram_size=3, ple_layer_index=0, seed={REAL_SEED}.\n") +W("// Matches the three values published in issue #1987 and range-read from the\n") +W("// released safetensors; vocab_size=248320 is the UNIQUE preimage below 2e6.\n") +W(f"static const int64_t kRealVocabSize = {REAL_VOCAB};\n") +W(f"static const int64_t kRealSeed = {REAL_SEED};\n") +W("static const int64_t kRealLayerMultipliers[3] = {" + + ", ".join(f"{m}LL" for m in real_mults) + "};\n\n") + +W("// modeling_qwen4_exp.py:1009-1015 _find_nth_prime_after, ngram_vocab_size_base\n") +W("// 20000000, ngram_heads 16 (ngram_size 3 x heads_per_ngram 8).\n") +W("static const int64_t kRealHeadVocabSizes[16] = {" + + ", ".join(f"{s}LL" for s in sizes) + "};\n") +W("static const int64_t kRealHeadOffsets[16] = {" + + ", ".join(f"{o}LL" for o in offs) + "};\n") +W(f"static const int64_t kRealTotalVocabSize = {tot}LL;\n") +W(f"static const int64_t kRealPaddedVocabSize = {padded}LL;\n\n") + +# ------------------------------------------------- B. a tiny runnable config +TINY = Cfg( + hidden_size=8, hc_count=2, ple_embed_dim=8, ple_conv_kernel_size=4, + ngram_size=3, heads_per_ngram=2, ngram_vocab_size_base=20, + make_ngram_vocab_size_divisible_by=8, vocab_size=64, eos_token_id=5, + seed=1234, rms_norm_eps=1e-6, +) +H, HC, E = TINY.hidden_size, TINY.hc_count, TINY.ple_embed_dim +NH = (TINY.ngram_size - 1) * TINY.heads_per_ngram +SCS = (TINY.ple_conv_kernel_size - 1) * TINY.ngram_size +EOS = TINY.eos_token_id + +ng = NGram(TINY, E, layer_idx=0, ple_layer_index=0) +W("// ---- tiny config, exercised end to end -------------------------------------\n") +W(f"static const int64_t kTinyHiddenSize = {H};\n") +W(f"static const int64_t kTinyHcCount = {HC};\n") +W(f"static const int64_t kTinyPleEmbedDim = {E};\n") +W(f"static const int64_t kTinyNgramSize = {TINY.ngram_size};\n") +W(f"static const int64_t kTinyHeadsPerNgram = {TINY.heads_per_ngram};\n") +W(f"static const int64_t kTinyNgramVocabBase = {TINY.ngram_vocab_size_base};\n") +W(f"static const int64_t kTinyVocabDivisor = {TINY.make_ngram_vocab_size_divisible_by};\n") +W(f"static const int64_t kTinyVocabSize = {TINY.vocab_size};\n") +W(f"static const int64_t kTinyEosTokenId = {EOS};\n") +W(f"static const int64_t kTinySeed = {TINY.seed};\n") +W(f"static const int64_t kTinyConvKernel = {TINY.ple_conv_kernel_size};\n") +W(f"static const int64_t kTinyShortConvStateLen = {SCS};\n") +W("static const int64_t kTinyHeadVocabSizes[%d] = {%s};\n" + % (NH, ", ".join(f"{s}LL" for s in ng.head_vocab_sizes))) +W("static const int64_t kTinyHeadOffsets[%d] = {%s};\n" + % (NH, ", ".join(f"{o}LL" for o in ng.head_offsets))) +W("static const int64_t kTinyLayerMultipliers[%d] = {%s};\n" + % (TINY.ngram_size, ", ".join(f"{m}LL" for m in ng.layer_multipliers.tolist()))) +W(f"static const int64_t kTinyTotalVocabSize = {ng.total_vocab_size}LL;\n") +W(f"static const int64_t kTinyPaddedVocabSize = {ng.ngram_embedding.num_embeddings}LL;\n\n") + +# --------------------------------------------- C. _shift_right_ignore_eos +# Deliberately EOS-dense: an EOS at the head, two in the interior, an adjacent +# pair, and a run shorter than the largest shift. +SHIFT_ROW = [EOS, 11, 12, 13, EOS, 21, EOS, EOS, 31, 32, 33, 34] +tok = torch.tensor([SHIFT_ROW], dtype=torch.long) +W("// modeling_qwen4_exp.py:1053-1067 _shift_right_ignore_eos\n") +W(f"static const int64_t kShiftSeqLen = {len(SHIFT_ROW)};\n") +W("static const int64_t kShiftInput[%d] = {%s};\n" + % (len(SHIFT_ROW), ", ".join(f"{t}LL" for t in SHIFT_ROW))) +W("static const int64_t kShiftExpected[3][%d] = {\n" % len(SHIFT_ROW)) +for shift in range(3): + row = ng._shift_right_ignore_eos(tok, shift)[0].tolist() + W(" {" + ", ".join(f"{v}LL" for v in row) + "}, // shift=%d\n" % shift) +W("};\n\n") + +# ----------------------------------------------- D. n-gram id construction +PREFILL = [7, 8, EOS, 9, 10, 11, EOS, 12, 13, 14] +DECODE = [15, 16] + + +def ngram_ids_for(chunks): + """The ids READ OUT of upstream's own forward, never rebuilt from it. + + `forward` returns embeddings rather than ids (:1114), so the obvious way to + pin the ids is to re-run its block-assembly loop -- and that is exactly the + transcription this file exists to avoid, because the generator and the port + would then share one reading of :1097-1112 and a shared misreading would + pass. Instead the gather is made INVERTIBLE: row i of `ngram_embedding` is + filled with the scalar i, so `forward` returns the ids themselves, repeated + `head_dim_per_ngram` times each, and they are read straight off the result. + The assertion below is what makes the inversion checkable rather than + assumed. Every line that computes an id is upstream's, executed. + """ + cache = Cache(num_layers=1, n_states=3) + module = NGram(TINY, E, layer_idx=0, ple_layer_index=0) + head_dim = E // module.ngram_heads + with torch.no_grad(): + for row in range(module.ngram_embedding.num_embeddings): + module.ngram_embedding.weight[row].fill_(float(row)) + # float32 holds every integer below 2**24 exactly; assert it rather than + # trust it, because a bigger tiny config would silently round. + assert module.ngram_embedding.num_embeddings < (1 << 24) + out = [] + for chunk in chunks: + ids = torch.tensor([chunk], dtype=torch.long) + with torch.no_grad(): + embedded = module.forward(ids, cache)[0] + recovered = embedded[:, ::head_dim].to(torch.long) + assert torch.equal( + embedded, recovered.repeat_interleave(head_dim, dim=-1).to(embedded.dtype)), \ + "the embedding is not invertible; the recovered ids would be a guess" + out.append(recovered.tolist()) + return out + + +ALL = PREFILL + DECODE +single = ngram_ids_for([ALL])[0] +chunked = ngram_ids_for([PREFILL, [DECODE[0]], [DECODE[1]]]) +assert single == chunked[0] + chunked[1] + chunked[2], \ + "cached decode must equal single-shot prefill" + +W("// modeling_qwen4_exp.py:1069-1114 Qwen4ExpTextNGramEmbedding.forward, id half.\n") +W("// The same 12 tokens, once as one prefill and once as prefill(10)+decode+decode;\n") +W("// upstream produces IDENTICAL ids, which is what pins the conv-state-2 history.\n") +W(f"static const int64_t kNgramPrefillLen = {len(PREFILL)};\n") +W(f"static const int64_t kNgramTotalLen = {len(ALL)};\n") +W("static const int64_t kNgramTokens[%d] = {%s};\n" + % (len(ALL), ", ".join(f"{t}LL" for t in ALL))) +W("static const int64_t kNgramExpectedIds[%d][%d] = {\n" % (len(ALL), NH)) +for row in single: + W(" {" + ", ".join(f"{v}LL" for v in row) + "},\n") +W("};\n\n") + +# ------------------------------------------------------------- E. the gate +GATE_IN = [0.0, 1e-12, -1e-12, 1e-6, -1e-6, 1e-3, -1e-3, 0.25, -0.25, 4.0, -4.0] +gate = torch.tensor(GATE_IN, dtype=torch.float32) +gate_out = gate.abs().clamp_min(1e-6).sqrt() * gate.sign() +W("// modeling_qwen4_exp.py:1181 gate.abs().clamp_min(1e-6).sqrt() * gate.sign()\n") +W("// Clamp BEFORE the sqrt: the magnitude floor is 1e-3, not 1e-6, and exactly\n") +W("// zero maps to zero because sign(0)=0. Discontinuous at the origin on purpose.\n") +W(f"static const int64_t kGateCount = {len(GATE_IN)};\n") +W("static const float kGateInput[%d] = {%s};\n" + % (len(GATE_IN), ", ".join(f"{v!r}f" for v in GATE_IN))) +W("static const float kGateExpected[%d] = {%s};\n\n" + % (len(GATE_IN), ", ".join(f"{v!r}f" for v in gate_out.tolist()))) + +# --------------------------------------------------- F. the dilated conv taps +ple = PLE(TINY, layer_idx=0, ple_layer_index=0) +with torch.no_grad(): + ple.conv1d.weight.zero_() + # Channel c carries its whole weight at kernel index c % 4, so a wrong lag, + # a unit stride or a reversed tap order all move the impulse response. + for channel in range(H * HC): + ple.conv1d.weight[channel, 0, channel % TINY.ple_conv_kernel_size] = 1.0 +TAPLEN = 14 +delta = torch.zeros(1, TAPLEN, H * HC) +delta[0, 0, :] = 3.0 +taps = ple._short_conv(delta, None) +W("// modeling_qwen4_exp.py:1150-1167 _short_conv. kernel 4, dilation ngram_size=3,\n") +W("// so output t reads input t-9, t-6, t-3, t with weights w0..w3 in that order.\n") +W("// One-hot tap per channel over an impulse at t=0: the response lands at\n") +W("// t = 9, 6, 3, 0 for w0..w3. silu is applied to the conv output.\n") +W(f"static const int64_t kTapSeqLen = {TAPLEN};\n") +W("static const float kTapImpulse = 3.0f;\n") +W("static const float kTapExpected[%d][%d] = {\n" % (TAPLEN, H * HC)) +for t in range(TAPLEN): + W(" {" + ", ".join(f"{v!r}f" for v in taps[0, t].tolist()) + "},\n") +W("};\n\n") + +# ----------------------------------------------------------- G. PLE end to end +ple2 = PLE(TINY, layer_idx=0, ple_layer_index=0) +with torch.no_grad(): + for parameter in ple2.parameters(): + parameter.uniform_(-0.6, 0.6) + ple2.ple_embedding.ngram_embedding.weight.uniform_(-0.5, 0.5) + + +def dump_tensor(name, tensor): + flat = tensor.reshape(-1).tolist() + W("static const float %s[%d] = {\n" % (name, len(flat))) + for i in range(0, len(flat), 6): + W(" " + ", ".join(f"{v!r}f" for v in flat[i:i + 6]) + ",\n") + W("};\n") + + +hidden = torch.empty(1, len(ALL), H * HC).uniform_(-1.0, 1.0) +with torch.no_grad(): + single_out = ple2(hidden, torch.tensor([ALL], dtype=torch.long), + Cache(num_layers=1, n_states=3), conv_mask=None) + cache = Cache(num_layers=1, n_states=3) + parts, lo = [], 0 + for count in (len(PREFILL), 1, 1): + parts.append(ple2(hidden[:, lo:lo + count], + torch.tensor([ALL[lo:lo + count]], dtype=torch.long), + cache, conv_mask=None)) + lo += count + incremental = torch.cat(parts, dim=1) +assert torch.allclose(single_out, incremental, atol=1e-5), \ + "incremental PLE forward must equal the single-shot one" + +W("// The full Qwen4ExpTextPLELayer forward (modeling_qwen4_exp.py:1169-1189),\n") +W("// tiny config, weights drawn once and frozen here. Prefill(10)+decode+decode\n") +W("// equals the single-shot 12-token prefill upstream, so both are one golden.\n") +dump_tensor("kPleNgramEmbeddingWeight", ple2.ple_embedding.ngram_embedding.weight) +dump_tensor("kPleKeyProjWeight", ple2.key_proj.weight) +dump_tensor("kPleValueProjWeight", ple2.value_proj.weight) +dump_tensor("kPleNormKeyWeight", ple2.norm_key.weight) +dump_tensor("kPleNormQueryWeight", ple2.norm_query.weight) +dump_tensor("kPleNormConvWeight", ple2.norm_conv.weight) +dump_tensor("kPleConv1dWeight", ple2.conv1d.weight) +dump_tensor("kPleHiddenStates", hidden) +dump_tensor("kPleExpectedOutput", single_out) + +# ----------------------------------------------------- H. the conv_mask arm +# `conv_mask` is prefill-only (`None` in steady-state decode) and upstream masks +# BOTH tensors at :1185-1187 -- `gated_value`, which is the skip term, AND +# `gated_value_normed`, which is what enters the conv AND what the 9-column +# state keeps. Masking one of the two is a real and easy port defect, so the +# mask has to be gated rather than documented. +# +# The mask is a PAIRED obligation with the caller (see the header): a masked +# position must already carry EOS in `input_ids`, because the hash reads token +# ids and not activations. These tokens honour that, so the golden pins the +# contract rather than an inconsistent state nobody would produce. +# +# Zeros at 3 and 4 are INTERIOR, not trailing: the conv is dilated by 3, so +# output t reads t-9, t-6, t-3 and t, and an interior zero therefore has to move +# t = 3, 6, 9 and 12 as well as its own row. A trailing-pad-only mask would +# leave the conv path almost untouched. +MASK_TOKENS = [7, 8, EOS, EOS, EOS, 11, EOS, 12, 13, 14, 15, EOS] +CONV_MASK = [1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0] +assert len(MASK_TOKENS) == len(CONV_MASK) == len(ALL) +assert all(MASK_TOKENS[i] == EOS for i, m in enumerate(CONV_MASK) if m == 0), \ + "a masked position must carry EOS: the hash reads ids, not activations" + +mask_t = torch.tensor([CONV_MASK], dtype=torch.bool) +with torch.no_grad(): + masked_single = ple2(hidden, torch.tensor([MASK_TOKENS], dtype=torch.long), + Cache(num_layers=1, n_states=3), conv_mask=mask_t) + cache = Cache(num_layers=1, n_states=3) + parts, lo = [], 0 + for count in (len(PREFILL), 1, 1): + parts.append(ple2(hidden[:, lo:lo + count], + torch.tensor([MASK_TOKENS[lo:lo + count]], dtype=torch.long), + cache, conv_mask=mask_t[:, lo:lo + count])) + lo += count + masked_incremental = torch.cat(parts, dim=1) +assert torch.allclose(masked_single, masked_incremental, atol=1e-5), \ + "the mask must reach the 9-column state, so both arms must agree" +assert not torch.allclose(masked_single, single_out, atol=1e-3), \ + "the mask must actually change the output, or this golden gates nothing" + +W("// modeling_qwen4_exp.py:1185-1187 + :204-213 apply_mask_to_padding_states.\n") +W("// BOTH `gated_value` (the skip term) and `gated_value_normed` (the conv\n") +W("// input, and what the 9-column state keeps) are masked. Zeros at 3, 4 and 11;\n") +W("// 3 and 4 are interior, so the dilation carries them to t = 6, 9 and 12 too.\n") +W("// Masked positions carry EOS in the tokens, which is the paired obligation.\n") +W("static const int64_t kPleMaskTokens[%d] = {%s};\n" + % (len(MASK_TOKENS), ", ".join(f"{t}LL" for t in MASK_TOKENS))) +W("static const unsigned char kPleConvMask[%d] = {%s};\n" + % (len(CONV_MASK), ", ".join(str(m) for m in CONV_MASK))) +dump_tensor("kPleMaskedExpectedOutput", masked_single) + +OUT_PATH.write_text(OUT.getvalue()) +print(f"wrote {OUT_PATH} ({len(OUT.getvalue().splitlines())} lines) from transformers {TAG}") +print(f"layer_multipliers at the real config: {real_mults}") +sys.exit(0) diff --git a/src/vllm/model_executor/models/qwen4_exp_ple.cpp b/src/vllm/model_executor/models/qwen4_exp_ple.cpp new file mode 100644 index 000000000..c3ba1d383 --- /dev/null +++ b/src/vllm/model_executor/models/qwen4_exp_ple.cpp @@ -0,0 +1,447 @@ +// Qwen4-Exp W2 — host reference for the two components with no vLLM op. +// Issue #1987, spec `.agents/specs/qwen4-exp-flash-next.md`. The header carries +// the scope statement, the upstream anchors, the batching seam and the reason +// this is host code; read it first. +// +// Oracle: huggingface/transformers v5.16.0 (this row's accepted lane pin). Line +// citations below are `modeling_qwen4_exp.py` at that tag unless stated. + +#include "vllm/model_executor/models/qwen4_exp_ple.h" + +#include +#include +#include +#include + +namespace vllm::qwen4_exp { +namespace { + +constexpr uint64_t kSplitmixGamma = 0x9E3779B97F4A7C15ULL; // :973 +constexpr uint64_t kSplitmixM1 = 0xBF58476D1CE4E5B9ULL; // :974 +constexpr uint64_t kSplitmixM2 = 0x94D049BB133111EBULL; // :975 +constexpr uint64_t kPrime1 = 10007ULL; // :976 + +[[noreturn]] void Refuse(const std::string& what) { + throw std::invalid_argument("qwen4_exp PLE: " + what); +} + +// The SECOND half of the int64 bound, and the half that has no `input_ids` loop +// to catch it. `eos_token_id` is the one id in the n-gram mix that does not +// come from the caller's tokens: `Reset` seeds the history with it and +// `ShiftRightIgnoreEos` emits it at every segment start, so it lands on the +// FIRST TOKEN OF EVERY SEQUENCE. Upstream needs no check because +// `config.eos_token_id` cannot be out of range (:1032 even unwraps a LIST); +// `PleGeometry` is a plain struct whose eos DEFAULTS TO -1, and at -1 the +// int64 product goes negative, where `torch.remainder` normalises it and our +// `static_cast` reinterprets the same bits near 2^64. Measured +// against transformers v5.16.0: upstream row 0 `[2, 35, 67, 96]` against ours +// `[8, 30, 67, 96]`, with no exception and no shape change. +void RefuseBadEos(const PleGeometry& geom) { + if (geom.eos_token_id < 0 || geom.eos_token_id >= geom.vocab_size) { + Refuse("eos_token_id " + std::to_string(geom.eos_token_id) + + " is outside [0, vocab_size=" + std::to_string(geom.vocab_size) + + "); it enters the n-gram mix at every segment start, so it would " + "overflow int64 and diverge in silence exactly like an " + "out-of-range input id"); + } +} + +// `Qwen4ExpTextRMSNorm._norm` + `.forward`, :167-178, group_size arm. +// Two traps in six lines: the reduction is over the GROUP, not the row, and the +// scale is `(1.0 + weight)` rather than `weight`, so a zeroed buffer is +// identity. Upstream promotes to float32 first; we accumulate in double, which +// is a reference choice and not a divergence — the tolerance in the gate covers +// the reduction-order difference against torch. +void GroupedRmsNorm(const float* x, int64_t rows, int64_t width, + int64_t group_size, double eps, const float* weight, + float* out) { + const int64_t groups = width / group_size; + for (int64_t r = 0; r < rows; ++r) { + const float* xr = x + r * width; + float* outr = out + r * width; + for (int64_t g = 0; g < groups; ++g) { + const int64_t base = g * group_size; + double sumsq = 0.0; + for (int64_t i = 0; i < group_size; ++i) { + const double v = static_cast(xr[base + i]); + sumsq += v * v; + } + const double scale = 1.0 / std::sqrt(sumsq / static_cast(group_size) + eps); + for (int64_t i = 0; i < group_size; ++i) { + outr[base + i] = static_cast(static_cast(xr[base + i]) * scale * + (1.0 + static_cast(weight[base + i]))); + } + } + } +} + +// `nn.Linear(bias=False)`: y[r, o] = sum_i x[r, i] * w[o, i]. +void Linear(const float* x, const float* w, int64_t rows, int64_t in_features, + int64_t out_features, float* y) { + for (int64_t r = 0; r < rows; ++r) { + const float* xr = x + r * in_features; + float* yr = y + r * out_features; + for (int64_t o = 0; o < out_features; ++o) { + const float* wo = w + o * in_features; + double acc = 0.0; + for (int64_t i = 0; i < in_features; ++i) { + acc += static_cast(xr[i]) * static_cast(wo[i]); + } + yr[o] = static_cast(acc); + } + } +} + +double Sigmoid(double v) { return 1.0 / (1.0 + std::exp(-v)); } + +double Silu(double v) { return v * Sigmoid(v); } + +} // namespace + +// :979-983. Unsigned throughout — divergence site #1. +uint64_t SplitMix64(uint64_t value) { + value = value + kSplitmixGamma; + value = (value ^ (value >> 30)) * kSplitmixM1; + value = (value ^ (value >> 27)) * kSplitmixM2; + return value ^ (value >> 31); +} + +// :998-1006. +bool IsPrime(int64_t value) { + if (value < 2) return false; + if (value % 2 == 0) return value == 2; + for (int64_t divisor = 3; divisor * divisor <= value; divisor += 2) { + if (value % divisor == 0) return false; + } + return true; +} + +// :1009-1015. +int64_t FindNthPrimeAfter(int64_t start, int64_t count) { + int64_t prime = start; + for (int64_t i = 0; i < count; ++i) { + ++prime; + while (!IsPrime(prime)) ++prime; + } + return prime; +} + +// :986-995. The modulo is UNSIGNED — divergence site #2. `SplitMix64` routinely +// returns a value above 2^63 and a signed `%` there yields a negative residue, +// which then becomes an even multiplier and a negative row index. +std::vector BuildLayerMultipliers(int64_t unigram_vocab_size, + int64_t ngram_size, + int64_t ple_layer_index, + int64_t seed) { + if (ngram_size <= 0) Refuse("ngram_size must be positive"); + const uint64_t max_long = static_cast(INT64_MAX); + const uint64_t divisor = + unigram_vocab_size > 1 ? static_cast(unigram_vocab_size) : 1ULL; + const uint64_t multiplier_max = max_long / divisor; + const uint64_t half_bound = multiplier_max / 2 > 0 ? multiplier_max / 2 : 1ULL; + const uint64_t base_seed = + static_cast(seed) + kPrime1 * static_cast(ple_layer_index); + + std::vector multipliers; + multipliers.reserve(static_cast(ngram_size)); + for (int64_t index = 0; index < ngram_size; ++index) { + const uint64_t value = + base_seed + kSplitmixGamma * static_cast(index + 1); + multipliers.push_back( + static_cast(2ULL * (SplitMix64(value) % half_bound) + 1ULL)); + } + return multipliers; +} + +// :1019-1051. The offsets are an exclusive prefix sum over the head vocab +// sizes, and `padded_vocab_size` rounds UP: the released config leaves 90 +// unaddressable rows (320001446 -> 320001536). +NGramTableLayout BuildNGramTableLayout(const PleGeometry& geom, + int64_t ple_layer_index) { + const int64_t heads = geom.ngram_heads(); + if (heads <= 0) Refuse("ngram_heads must be positive"); + if (geom.ple_embed_dim <= 0 || geom.ple_embed_dim % heads != 0) { + Refuse("ple_embed_dim must be positive and divisible by ngram_heads"); + } + if (geom.make_ngram_vocab_size_divisible_by <= 0) { + Refuse("make_ngram_vocab_size_divisible_by must be positive"); + } + RefuseBadEos(geom); + + NGramTableLayout layout; + layout.head_vocab_sizes.reserve(static_cast(heads)); + layout.head_offsets.reserve(static_cast(heads)); + for (int64_t head_idx = 0; head_idx < heads; ++head_idx) { + const int64_t global_head_idx = ple_layer_index * heads + head_idx; + const int64_t size = + FindNthPrimeAfter(geom.ngram_vocab_size_base - 1, global_head_idx + 1); + layout.head_vocab_sizes.push_back(size); + layout.head_offsets.push_back(layout.total_vocab_size); + layout.total_vocab_size += size; + } + layout.layer_multipliers = BuildLayerMultipliers( + geom.vocab_size, geom.ngram_size, ple_layer_index, geom.seed); + + const int64_t divisor = geom.make_ngram_vocab_size_divisible_by; + layout.padded_vocab_size = + ((layout.total_vocab_size + divisor - 1) / divisor) * divisor; + return layout; +} + +void PleSequenceState::Reset(const PleGeometry& geom) { + conv.assign(static_cast(geom.stream_width() * geom.short_conv_state_len()), + 0.0F); + tokens.assign(static_cast(geom.context_len()), geom.eos_token_id); +} + +// :1053-1067. +void ShiftRightIgnoreEos(const int64_t* token_ids, int64_t seq_len, int64_t shift, + int64_t eos_token_id, int64_t* out) { + if (shift == 0) { + for (int64_t i = 0; i < seq_len; ++i) out[i] = token_ids[i]; + return; + } + // `previous_eos` is the running max of EOS positions STRICTLY BEFORE i, which + // is why an EOS token belongs to the segment it terminates rather than to the + // one it opens. + int64_t previous_eos = -1; + for (int64_t i = 0; i < seq_len; ++i) { + const int64_t segment_start = previous_eos + 1; + const int64_t position_in_segment = i - segment_start; + const int64_t source = i - shift; + out[i] = (position_in_segment >= shift && source >= 0) ? token_ids[source] + : eos_token_id; + if (token_ids[i] == eos_token_id) previous_eos = i; + } +} + +// :1069-1112 for the ids, plus cache_utils.py:1037-1075 for state 2. +// +// Upstream's state-2 dance reduces to one line — `tokens := last context_len of +// (tokens ++ input_ids)` — and the reduction is worth stating because the two +// branches it collapses look different: on the first call it EOS-left-pads a +// short chunk before caching (:1080-1088) precisely to dodge +// `update_conv_state`'s zero pad, and on later calls it concatenates. With +// `tokens` EOS-seeded by `Reset` both branches produce the same bytes. +void BuildNGramIds(const PleGeometry& geom, const NGramTableLayout& layout, + const int64_t* input_ids, int64_t num_tokens, + PleSequenceState* state, int64_t* out_ids) { + if (state == nullptr) Refuse("BuildNGramIds needs a sequence state"); + const int64_t context_len = geom.context_len(); + const int64_t heads = geom.ngram_heads(); + if (static_cast(state->tokens.size()) != context_len) { + Refuse("sequence state was not Reset for this geometry"); + } + if (num_tokens <= 0) return; + + // The bound the whole int64 argument rests on. Upstream cannot be handed an + // out-of-range id; we can, and the failure is a silent overflow. Both halves + // are checked here and not only at layout time, because the geometry is a + // plain struct the caller still owns. + RefuseBadEos(geom); + for (int64_t t = 0; t < num_tokens; ++t) { + if (input_ids[t] < 0 || input_ids[t] >= geom.vocab_size) { + Refuse("token id " + std::to_string(input_ids[t]) + + " is outside [0, vocab_size=" + std::to_string(geom.vocab_size) + + "); the n-gram mix would overflow int64 and diverge in silence"); + } + } + + const int64_t history_len = context_len + num_tokens; + std::vector history(static_cast(history_len)); + for (int64_t i = 0; i < context_len; ++i) history[i] = state->tokens[i]; + for (int64_t t = 0; t < num_tokens; ++t) history[context_len + t] = input_ids[t]; + + std::vector> shifted(static_cast(geom.ngram_size)); + for (int64_t s = 0; s < geom.ngram_size; ++s) { + shifted[static_cast(s)].resize(static_cast(history_len)); + ShiftRightIgnoreEos(history.data(), history_len, s, geom.eos_token_id, + shifted[static_cast(s)].data()); + } + + // The XOR runs in uint64: every operand is non-negative by the bound checked + // above, so this is bit-identical to upstream's int64 tensor op and avoids + // signed-overflow UB on the way. + for (int64_t ngram = 2; ngram <= geom.ngram_size; ++ngram) { + const int64_t start_idx = (ngram - 2) * geom.heads_per_ngram; + for (int64_t i = 0; i < history_len; ++i) { + const int64_t row = i - context_len; + if (row < 0) continue; // upstream builds all rows, then keeps the last T + uint64_t mixed = static_cast(shifted[0][static_cast(i)]) * + static_cast(layout.layer_multipliers[0]); + for (int64_t position = 1; position < ngram; ++position) { + mixed ^= static_cast( + shifted[static_cast(position)][static_cast(i)]) * + static_cast(layout.layer_multipliers[static_cast(position)]); + } + for (int64_t h = 0; h < geom.heads_per_ngram; ++h) { + const int64_t head = start_idx + h; + const uint64_t size = + static_cast(layout.head_vocab_sizes[static_cast(head)]); + out_ids[row * heads + head] = + static_cast(mixed % size) + + layout.head_offsets[static_cast(head)]; + } + } + } + + // Advance state 2. + for (int64_t i = 0; i < context_len; ++i) { + state->tokens[static_cast(i)] = + history[static_cast(history_len - context_len + i)]; + } +} + +// :1181. +float SignedSqrtGate(float gate) { + if (gate == 0.0F) return 0.0F; // sign(0) == 0, so the origin maps to zero + const double magnitude = std::abs(static_cast(gate)); + const double clamped = magnitude < 1e-6 ? 1e-6 : magnitude; + const double out = std::sqrt(clamped); + return static_cast(gate > 0.0F ? out : -out); +} + +// :1150-1167. The pad-and-slice upstream performs unconditionally collapses to +// "prepend the 9-column state", and the state after the call is the last 9 +// columns of that same buffer. Both upstream branches — the zero left-pad on a +// short first chunk and the concatenation on every later chunk — land on those +// bytes, which is why there is no `has_previous_state` test here. +void PleShortConv(const PleGeometry& geom, const float* conv1d_weight, + const float* normed, int64_t num_tokens, + PleSequenceState* state, float* out) { + if (state == nullptr) Refuse("PleShortConv needs a sequence state"); + const int64_t width = geom.stream_width(); + const int64_t state_len = geom.short_conv_state_len(); + const int64_t kernel = geom.ple_conv_kernel_size; + const int64_t dilation = geom.ngram_size; + if (static_cast(state->conv.size()) != width * state_len) { + Refuse("sequence state was not Reset for this geometry"); + } + if (num_tokens <= 0) return; + + // [width, state_len + num_tokens], channel-major: the conv is depthwise, so + // this is the layout a device arm wants too. + const int64_t span = state_len + num_tokens; + std::vector buffer(static_cast(width * span)); + for (int64_t c = 0; c < width; ++c) { + float* row = buffer.data() + c * span; + for (int64_t s = 0; s < state_len; ++s) row[s] = state->conv[c * state_len + s]; + for (int64_t t = 0; t < num_tokens; ++t) row[state_len + t] = normed[t * width + c]; + } + + for (int64_t c = 0; c < width; ++c) { + const float* row = buffer.data() + c * span; + const float* w = conv1d_weight + c * kernel; + for (int64_t t = 0; t < num_tokens; ++t) { + double acc = 0.0; + // k = 0..3 reads lags {9, 6, 3, 0}: `t + k * dilation` against a buffer + // whose current token sits at `t + state_len`, and `(kernel-1)*dilation + // == state_len` makes the last tap the current token. Causal by that tap. + for (int64_t k = 0; k < kernel; ++k) { + acc += static_cast(w[k]) * static_cast(row[t + k * dilation]); + } + out[t * width + c] = static_cast(Silu(acc)); + } + } + + for (int64_t c = 0; c < width; ++c) { + const float* row = buffer.data() + c * span; + for (int64_t s = 0; s < state_len; ++s) { + state->conv[c * state_len + s] = row[span - state_len + s]; + } + } +} + +// :1169-1189. +void PleForward(const PleGeometry& geom, const NGramTableLayout& layout, + const PleWeights& weights, const float* hidden_states, + const int64_t* input_ids, int64_t num_tokens, + const unsigned char* conv_mask, PleSequenceState* state, + float* out) { + if (state == nullptr) Refuse("PleForward needs a sequence state"); + const int64_t hidden = geom.hidden_size; + const int64_t hc = geom.hc_count; + const int64_t width = geom.stream_width(); + const int64_t heads = geom.ngram_heads(); + const int64_t head_dim = geom.head_dim_per_ngram(); + const int64_t embed_dim = geom.ple_embed_dim; + if (num_tokens <= 0) return; + + std::vector ids(static_cast(num_tokens * heads)); + BuildNGramIds(geom, layout, input_ids, num_tokens, state, ids.data()); + + // The gather: `heads` uncoalesced random rows per token. This loop is the + // batching seam named in the header. + std::vector embeddings(static_cast(num_tokens * embed_dim)); + for (int64_t t = 0; t < num_tokens; ++t) { + for (int64_t h = 0; h < heads; ++h) { + const int64_t row = ids[t * heads + h]; + if (row < 0 || row >= layout.padded_vocab_size) { + Refuse("n-gram row " + std::to_string(row) + " is outside the padded table"); + } + const float* src = weights.ngram_embedding + row * head_dim; + float* dst = embeddings.data() + t * embed_dim + h * head_dim; + for (int64_t d = 0; d < head_dim; ++d) dst[d] = src[d]; + } + } + + std::vector key(static_cast(num_tokens * width)); + Linear(embeddings.data(), weights.key_proj, num_tokens, embed_dim, width, key.data()); + std::vector key_normed(static_cast(num_tokens * width)); + GroupedRmsNorm(key.data(), num_tokens, width, hidden, geom.rms_norm_eps, + weights.norm_key, key_normed.data()); + + std::vector value(static_cast(num_tokens * hidden)); + Linear(embeddings.data(), weights.value_proj, num_tokens, embed_dim, hidden, + value.data()); + + std::vector query_normed(static_cast(num_tokens * width)); + GroupedRmsNorm(hidden_states, num_tokens, width, hidden, geom.rms_norm_eps, + weights.norm_query, query_normed.data()); + + const double inv_scale = 1.0 / std::sqrt(static_cast(hidden)); + std::vector gated_value(static_cast(num_tokens * width)); + for (int64_t t = 0; t < num_tokens; ++t) { + for (int64_t s = 0; s < hc; ++s) { + const float* k = key_normed.data() + t * width + s * hidden; + const float* q = query_normed.data() + t * width + s * hidden; + double dot = 0.0; + for (int64_t d = 0; d < hidden; ++d) { + dot += static_cast(k[d]) * static_cast(q[d]); + } + const float gate = SignedSqrtGate(static_cast(dot * inv_scale)); + const double weight = Sigmoid(static_cast(gate)); + float* dst = gated_value.data() + t * width + s * hidden; + const float* v = value.data() + t * hidden; + for (int64_t d = 0; d < hidden; ++d) { + dst[d] = static_cast(weight * static_cast(v[d])); + } + } + } + + // THE FORK. `gated_value_normed` is what the conv sees; the skip term added + // back at the end is the UN-NORMED copy, and the 9-column state holds the + // NORMED one. + std::vector gated_value_normed(static_cast(num_tokens * width)); + GroupedRmsNorm(gated_value.data(), num_tokens, width, hidden, geom.rms_norm_eps, + weights.norm_conv, gated_value_normed.data()); + + if (conv_mask != nullptr) { + for (int64_t t = 0; t < num_tokens; ++t) { + const float m = conv_mask[t] != 0 ? 1.0F : 0.0F; + for (int64_t c = 0; c < width; ++c) { + gated_value[t * width + c] *= m; + gated_value_normed[t * width + c] *= m; + } + } + } + + std::vector conv_out(static_cast(num_tokens * width)); + PleShortConv(geom, weights.conv1d, gated_value_normed.data(), num_tokens, state, + conv_out.data()); + for (int64_t i = 0; i < num_tokens * width; ++i) { + out[i] = gated_value[static_cast(i)] + conv_out[static_cast(i)]; + } +} + +} // namespace vllm::qwen4_exp diff --git a/src/vllm/model_executor/models/qwen4_exp_ple.h b/src/vllm/model_executor/models/qwen4_exp_ple.h new file mode 100644 index 000000000..5c2b29c6d --- /dev/null +++ b/src/vllm/model_executor/models/qwen4_exp_ple.h @@ -0,0 +1,270 @@ +// Qwen4-Exp (`Qwen4ExpForConditionalGeneration`) W2 — the hashed n-gram +// embedding and the PLE dilated depthwise conv, as a portable host reference. +// +// Issue #1987, campaign issue #1978, spec +// `.agents/specs/qwen4-exp-flash-next.md` (`## Port map`, "The n-gram embedding +// is integer-exact or it is silently wrong", and "PLE: a strided-history conv +// with no vLLM op, confirmed"). +// +// ─── WHY THIS FILE HAS NO vLLM ANCHOR ──────────────────────────────────────── +// These are the ONLY two components of this model with no vLLM op at all, so +// AGENTS.md's mirror-vLLM polarity has nothing to bind here and the secondary +// oracle is the sole source. The negative is CONFIRMED rather than unfound: at +// vLLM `origin/main` = `6a5e8f5979`, `git grep -in dilat` returns zero lines in +// `vllm/model_executor/layers/mamba/`, zero in `csrc/` and zero in `tests/`; +// `layers/conv.py` defines only `Conv2dLayer` and `Conv3dLayer`. Upstream +// reached the same conclusion from the other side and hand-rolled it, saying so +// in a comment: "We cannot use the usual functions/kernels here for the short +// conv as the conv1d has dilation". +// +// ORACLE: huggingface/transformers **v5.16.0**, this row's ACCEPTED lane pin +// (spec `## Oracles`; `v5.16.0` is the FIRST release containing `qwen4_exp` — +// `v5.15.0` returns HTTP 404 for the same path). Every `file:line` below is at +// that tag. `modular_qwen4_exp.py` is the authored delta; +// `modeling_qwen4_exp.py` is its generated expansion and is what the goldens +// were exec'd out of, so both are cited. +// +// OURS <- UPSTREAM (transformers v5.16.0) +// SplitMix64 <- modeling_qwen4_exp.py::_splitmix64 (:979-983) +// = modular_qwen4_exp.py (:568-572) +// BuildLayerMultipliers <- ::_build_layer_multipliers (:986-995) +// IsPrime/FindNthPrimeAfter<- ::_is_prime (:998-1006), ::_find_nth_prime_after +// (:1009-1015) +// BuildNGramTableLayout <- ::Qwen4ExpTextNGramEmbedding.__init__ (:1019-1051) +// ShiftRightIgnoreEos <- ::Qwen4ExpTextNGramEmbedding._shift_right_ignore_eos +// (:1053-1067) +// BuildNGramIds <- ::Qwen4ExpTextNGramEmbedding.forward (:1069-1114), +// plus cache_utils.py::LinearAttentionLayer +// .update_conv_state (:1037-1075) for state 2 +// SignedSqrtGate <- ::Qwen4ExpTextPLELayer.forward (:1181) +// PleShortConv <- ::Qwen4ExpTextPLELayer._short_conv (:1150-1167) +// PleForward <- ::Qwen4ExpTextPLELayer.forward (:1169-1189) +// GroupedRmsNorm (private) <- ::Qwen4ExpTextRMSNorm (:158-181), group_size arm +// +// ─── W2 SCOPE, STATED SO NOBODY HAS TO INFER IT ────────────────────────────── +// THIS IS A HOST (CPU) REFERENCE AND IT IS NOT YET REACHED FROM ANY PRODUCTION +// ENTRY POINT. `qwen4_exp` has no registry entry, no loader and no +// `ModelRegistry::Forward` arm until W5 assembles the model, so per AGENTS.md +// "Nothing lands dead" this is a STAGED SLICE landing unreached: the wiring is +// owned by row `MODEL-MM-QWEN4-EXP` and tracked by campaign issue #1978, and +// the spec lists it under `## Owed`. Do not read the absence of a call site as +// an oversight; read it as the recorded debt it is. +// +// What is deliberately NOT here: the batched/device arm (see the batching seam +// below), the weight loader and its 128-shard NUMERIC reassembly, grouped +// RMSNorm as a shared layer, the Gated Residual stream (W3), QSA (W4), and any +// speed claim (`## Gates` admits none from this row until G2 passes). +// +// ─── THE BATCHING SEAM ─────────────────────────────────────────────────────── +// Every entry point below is PER SEQUENCE and takes its own `PleSequenceState`. +// Nothing loops over the batch, nothing owns a global, and nothing needs a +// host round trip inside a decode step. The n-gram gather is +// `(ngram_size-1) * heads_per_ngram` = 16 uncoalesced random rows per token per +// sequence, so at batch B it is 16*B independent gathers; a device arm replaces +// the body of `BuildNGramIds` + the embedding gather in `PleForward` with one +// kernel over [B, T, 16] and keeps this signature as the single-sequence +// fallback. `PleSequenceState` is deliberately two flat, contiguous buffers so +// a paged KV arm can point at cache pages instead of owning them. +#pragma once + +#include +#include + +namespace vllm::qwen4_exp { + +// The MINIMUM config surface W2 needs. W1 (#1981, PR #1986) owns the real +// `Qwen4ExpParams` / `Qwen4ExpPleParams` in +// `src/vllm/model_executor/models/qwen4_exp.h`; that branch is not on `main` at +// this commit, so this wave declares its own rather than depending on an +// unmerged branch. The three helper names below are W1's names on purpose, so +// the swap in W3 is mechanical. Defaults are the dataclass defaults at +// `modular_qwen4_exp.py:155-163`, and `seed` is 1234 because `config.seed` is +// ABSENT from the published `config.json` (spec, `### The n-gram embedding is +// integer-exact or it is silently wrong`). +struct PleGeometry { + int64_t hidden_size = 0; + int64_t hc_count = 0; + int64_t ple_embed_dim = 0; + int64_t ple_conv_kernel_size = 4; + int64_t ngram_size = 3; + int64_t heads_per_ngram = 8; + int64_t ngram_vocab_size_base = 20000000; + int64_t make_ngram_vocab_size_divisible_by = 128; + // `config.vocab_size`, the UNIGRAM vocabulary. It bounds the multiplier so + // `token_id * multiplier` cannot overflow int64 — but only while every id + // that enters the mix really is below it, which is why `BuildNGramIds` + // refuses one that is not. TWO ids enter the mix, not one; see below. + int64_t vocab_size = 0; + // -1 is an UNSET SENTINEL and is refused, not a working default: eos is the + // second id in the mix and there is no safe value to pick for it. A loader + // must set it, and `config.eos_token_id` is permitted to be a LIST upstream + // (modeling_qwen4_exp.py:1032 takes element [0]), which is the realistic way + // to get it wrong. + int64_t eos_token_id = -1; + int64_t seed = 1234; + double rms_norm_eps = 1e-6; + + // (ngram_size - 1) * heads_per_ngram; 16 for the released config. + int64_t ngram_heads() const { return (ngram_size - 1) * heads_per_ngram; } + int64_t head_dim_per_ngram() const { return ple_embed_dim / ngram_heads(); } + // NINE, not `kernel - 1`: `(4 - 1) * 3`, because the conv is dilated by + // `ngram_size`. modeling_qwen4_exp.py:1135 + // (`self.short_conv_state_len = (conv_kernel_size - 1) * conv_dilation`, + // with `conv_dilation = config.ngram_size` at :1134). + int64_t short_conv_state_len() const { + return (ple_conv_kernel_size - 1) * ngram_size; + } + // The n-gram token history kept in conv state 2. modeling_qwen4_exp.py:1023. + int64_t context_len() const { return ngram_size - 1; } + // The width of the hyper-connection residual stream: 4 * 2560 = 10240. + int64_t stream_width() const { return hc_count * hidden_size; } +}; + +// ─── the hash chain ────────────────────────────────────────────────────────── + +// `_splitmix64`, modeling_qwen4_exp.py:979-983. UNSIGNED THROUGHOUT, and that +// is divergence site #1 in the spec: upstream's `>> 30 / 27 / 31` are LOGICAL +// shifts on a non-negative Python int. On `int64_t` they become arithmetic +// shifts, the multiplicand is wrong, and the top bit is set about half the +// time, so it fires immediately and silently. +uint64_t SplitMix64(uint64_t value); + +// `_is_prime` / `_find_nth_prime_after`, modeling_qwen4_exp.py:998-1006 and +// :1009-1015. +bool IsPrime(int64_t value); +int64_t FindNthPrimeAfter(int64_t start, int64_t count); + +// `_build_layer_multipliers`, modeling_qwen4_exp.py:986-995. Divergence site +// #2 lives here: `_splitmix64(value) % half_bound` is an UNSIGNED modulo. The +// dividend routinely exceeds 2^63 and a signed modulo yields a negative +// residue. Returns `ngram_size` values. +std::vector BuildLayerMultipliers(int64_t unigram_vocab_size, + int64_t ngram_size, + int64_t ple_layer_index, + int64_t seed); + +// Everything `Qwen4ExpTextNGramEmbedding.__init__` derives before it sees a +// token. modeling_qwen4_exp.py:1019-1051. +struct NGramTableLayout { + std::vector head_vocab_sizes; // [ngram_heads] + std::vector head_offsets; // [ngram_heads], exclusive prefix sum + std::vector layer_multipliers; // [ngram_size] + int64_t total_vocab_size = 0; + int64_t padded_vocab_size = 0; // rounded UP to make_ngram_vocab_size_divisible_by +}; +NGramTableLayout BuildNGramTableLayout(const PleGeometry& geom, + int64_t ple_layer_index); + +// ─── the per-sequence state a PLE layer owns ───────────────────────────────── + +// Conv state 1 (the PLE conv) and conv state 2 (the n-gram token history). +// Upstream keeps both in the linear-attention cache alongside the GDN conv, +// which is why `number_of_conv_states == 3` on a PLE layer +// (modular_qwen4_exp.py:178-180). +// +// `Reset` seeds `tokens` with EOS and `conv` with zeros, which is EXACTLY what +// upstream's first-call path computes, and the equivalence is worth stating +// because the two states pad DIFFERENTLY: +// * tokens: upstream's `update_conv_state` pads with 0 — a VALID token id — +// so the layer works around it with an explicit EOS left-pad +// (modeling_qwen4_exp.py:1080-1088). Pad with EOS, never with zero. +// * conv: upstream pads with zeros on both the first-call and the steady +// path, so a zero-initialised buffer is bit-identical to its first call. +struct PleSequenceState { + std::vector conv; // [stream_width, short_conv_state_len] + std::vector tokens; // [context_len] + void Reset(const PleGeometry& geom); +}; + +// `_shift_right_ignore_eos`, modeling_qwen4_exp.py:1053-1067, one row. +// Position i takes token[i - shift] only when at least `shift` tokens have +// passed since the start of i's EOS-delimited segment; otherwise EOS. An EOS +// token belongs to the segment it TERMINATES, because the "previous EOS" scan +// is strictly-before. `out` holds `seq_len` ids and may not alias `token_ids`. +void ShiftRightIgnoreEos(const int64_t* token_ids, int64_t seq_len, + int64_t shift, int64_t eos_token_id, int64_t* out); + +// The id half of `Qwen4ExpTextNGramEmbedding.forward`, +// modeling_qwen4_exp.py:1069-1112. Reads `state->tokens` as the left context, +// writes `num_tokens * ngram_heads` ids to `out_ids` (row-major), and advances +// `state->tokens`. Integer-exact: there is no tolerance and no downstream gate +// that localises an error here. +// +// REFUSES BY NAME on any id outside `[0, vocab_size)` — BOTH the `input_ids` +// the caller passes AND `geom.eos_token_id`, which is the one id in the mix +// that does not come from `input_ids`. `Reset` seeds the history with eos and +// `ShiftRightIgnoreEos` emits it at every segment start, so it is on the FIRST +// TOKEN OF EVERY SEQUENCE; `BuildNGramTableLayout` refuses it at construction +// time as well. Upstream has no such check and does not need one, because its +// loader cannot admit one; ours can, and the failure is `token_id * multiplier` +// overflowing int64 and diverging in silence (spec, "That bound holds only +// while every token id is below `vocab_size`"). Measured against transformers +// v5.16.0 at the struct's own `eos = -1` default: upstream row 0 +// `[2, 35, 67, 96]`, ours `[8, 30, 67, 96]`, no exception either side. +void BuildNGramIds(const PleGeometry& geom, const NGramTableLayout& layout, + const int64_t* input_ids, int64_t num_tokens, + PleSequenceState* state, int64_t* out_ids); + +// ─── the PLE layer ─────────────────────────────────────────────────────────── + +// Row-major, bias-free, exactly as `nn.Linear.weight` stores them +// ([out_features, in_features]). `norm_*` are RMSNorm DELTAS: upstream applies +// `(1.0 + weight)` (modeling_qwen4_exp.py:177), so a zeroed buffer is identity. +struct PleWeights { + const float* ngram_embedding = nullptr; // [padded_vocab_size, head_dim_per_ngram] + const float* key_proj = nullptr; // [stream_width, ple_embed_dim] + const float* value_proj = nullptr; // [hidden_size, ple_embed_dim] + const float* norm_key = nullptr; // [stream_width] + const float* norm_query = nullptr; // [stream_width] + const float* norm_conv = nullptr; // [stream_width] + const float* conv1d = nullptr; // [stream_width, ple_conv_kernel_size] +}; + +// `gate.abs().clamp_min(1e-6).sqrt() * gate.sign()`, +// modeling_qwen4_exp.py:1181. THE CLAMP IS BEFORE THE SQRT, so the floor on the +// output MAGNITUDE is sqrt(1e-6) = 1e-3 and tiny scores are AMPLIFIED, not +// squashed. Exactly zero maps to zero because `sign(0) == 0`, so the function +// is genuinely discontinuous at the origin — and that origin is reachable on a +// fully masked row. Mirror it; do not tidy it. Clamping after the sqrt is wrong +// by three orders of magnitude in that band. +float SignedSqrtGate(float gate); + +// `_short_conv`, modeling_qwen4_exp.py:1150-1167. Depthwise, kernel 4, dilated +// by `ngram_size` = 3, so output t reads lags {9, 6, 3, 0} with weights +// w0..w3 in that order and the lag-0 tap makes it causal. `normed` is +// [num_tokens, stream_width] row-major (the NORMED conv input, never the raw +// hidden state); `out` is the same shape and carries silu already applied. +// Advances `state->conv`. +// +// The state is a genuine 9-deep ring read at stride 3 and cannot be compressed +// to 3 columns even though any single step touches only three of them: 9 +// columns x 10240 channels is ~180 KiB per sequence at bf16 for this one layer, +// which is a KV-budget line item rather than a rounding error. +void PleShortConv(const PleGeometry& geom, const float* conv1d_weight, + const float* normed, int64_t num_tokens, + PleSequenceState* state, float* out); + +// `Qwen4ExpTextPLELayer.forward`, modeling_qwen4_exp.py:1169-1189. +// hidden_states [num_tokens, stream_width] +// input_ids [num_tokens] +// conv_mask [num_tokens] of 0/1, or nullptr. `None` in steady-state +// decode, so the masking is prefill-only. BOTH the skip term +// and the conv input are masked (:1185-1187), and the masked +// conv input is what the 9-column state keeps — gated by +// `kPleMaskedExpectedOutput`, single-shot and incremental. +// It is also a PAIRED obligation with the caller: the +// activations are masked here AND `input_ids` must already +// carry EOS at padded positions, because the hash reads token +// ids rather than activations. Masking only the activations +// leaks padding into the hash. That half has no caller yet and +// is owed to W5; the spec's `## Owed` names it. +// out [num_tokens, stream_width] +// The fork the spec warns about is at the end: the skip term is the UN-NORMED +// `gated_value` and only the NORMED copy enters the conv. +void PleForward(const PleGeometry& geom, const NGramTableLayout& layout, + const PleWeights& weights, const float* hidden_states, + const int64_t* input_ids, int64_t num_tokens, + const unsigned char* conv_mask, PleSequenceState* state, + float* out); + +} // namespace vllm::qwen4_exp diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index d4d6cff71..fed847391 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -3113,3 +3113,17 @@ foreach(_pc_target target_include_directories(${_pc_target} PRIVATE ${CMAKE_SOURCE_DIR}/src) endif() endforeach() + +# Qwen4-Exp (`Qwen4ExpForConditionalGeneration`) W2 -- the hashed n-gram +# embedding and the PLE dilated depthwise conv (#1987, campaign #1978, +# .agents/specs/qwen4-exp-flash-next.md). These are the ONLY two components of +# the model with NO vLLM op, confirmed by a negative search rather than an +# unfound one, so transformers v5.16.0 -- this row's accepted lane pin -- is the +# sole oracle and the goldens beside this file were produced by EXECUTING +# upstream's own bytes. The spec admits no token gate for this row until an arm +# runs (nothing published fits any fleet device), so G0 component goldens are +# the whole instrument. CPU-only, no checkpoint, no GPU, no speed claim. +vllm_cpp_add_test(test_qwen4_exp_ple vllm/models/test_qwen4_exp_ple.cpp) +# qwen4_exp_ple.h is a MODEL-PRIVATE header under src/, same arrangement as the +# dots3-note gates above: W2 ships nothing on the public ABI. +target_include_directories(test_qwen4_exp_ple PRIVATE ${CMAKE_SOURCE_DIR}/src) diff --git a/tests/vllm/models/qwen4_exp_ple_goldens.inc b/tests/vllm/models/qwen4_exp_ple_goldens.inc new file mode 100644 index 000000000..2057f7cbe --- /dev/null +++ b/tests/vllm/models/qwen4_exp_ple_goldens.inc @@ -0,0 +1,334 @@ +// GENERATED by scripts/gen-qwen4-exp-ple-goldens.py -- do not edit. +// Oracle: huggingface/transformers v5.16.0, +// src/transformers/models/qwen4_exp/modeling_qwen4_exp.py +// src/transformers/cache_utils.py +// Produced by exec'ing the upstream line ranges VERBATIM, never by +// transcribing them; see the generator. transformers 5.16.0 is this row's +// accepted lane pin (spec `## Oracles`) and vLLM implements neither of the +// two components below, so there is no primary oracle to mirror instead. + +// modeling_qwen4_exp.py:979-983 _splitmix64 +static const struct { uint64_t in; uint64_t out; } kSplitMix64[] = { + {0ULL, 16294208416658607535ULL}, + {1ULL, 10451216379200822465ULL}, + {1234ULL, 13478418381427711195ULL}, + {11400714819323199719ULL, 10936887474700444964ULL}, + {11400714819323198485ULL, 7960286522194355700ULL}, + {9223372036854775808ULL, 5196802822362493915ULL}, + {18446744073709551615ULL, 16490336266968443936ULL}, + {16045690984503111693ULL, 10384543611796878027ULL}, +}; + +// modeling_qwen4_exp.py:986-995 _build_layer_multipliers, at the REAL config: +// vocab_size=248320, ngram_size=3, ple_layer_index=0, seed=1234. +// Matches the three values published in issue #1987 and range-read from the +// released safetensors; vocab_size=248320 is the UNIQUE preimage below 2e6. +static const int64_t kRealVocabSize = 248320; +static const int64_t kRealSeed = 1234; +static const int64_t kRealLayerMultipliers[3] = {23703573157769LL, 20109073645365LL, 8052911324071LL}; + +// modeling_qwen4_exp.py:1009-1015 _find_nth_prime_after, ngram_vocab_size_base +// 20000000, ngram_heads 16 (ngram_size 3 x heads_per_ngram 8). +static const int64_t kRealHeadVocabSizes[16] = {20000003LL, 20000023LL, 20000033LL, 20000047LL, 20000059LL, 20000063LL, 20000069LL, 20000077LL, 20000081LL, 20000093LL, 20000107LL, 20000147LL, 20000153LL, 20000159LL, 20000161LL, 20000171LL}; +static const int64_t kRealHeadOffsets[16] = {0LL, 20000003LL, 40000026LL, 60000059LL, 80000106LL, 100000165LL, 120000228LL, 140000297LL, 160000374LL, 180000455LL, 200000548LL, 220000655LL, 240000802LL, 260000955LL, 280001114LL, 300001275LL}; +static const int64_t kRealTotalVocabSize = 320001446LL; +static const int64_t kRealPaddedVocabSize = 320001536LL; + +// ---- tiny config, exercised end to end ------------------------------------- +static const int64_t kTinyHiddenSize = 8; +static const int64_t kTinyHcCount = 2; +static const int64_t kTinyPleEmbedDim = 8; +static const int64_t kTinyNgramSize = 3; +static const int64_t kTinyHeadsPerNgram = 2; +static const int64_t kTinyNgramVocabBase = 20; +static const int64_t kTinyVocabDivisor = 8; +static const int64_t kTinyVocabSize = 64; +static const int64_t kTinyEosTokenId = 5; +static const int64_t kTinySeed = 1234; +static const int64_t kTinyConvKernel = 4; +static const int64_t kTinyShortConvStateLen = 9; +static const int64_t kTinyHeadVocabSizes[4] = {23LL, 29LL, 31LL, 37LL}; +static const int64_t kTinyHeadOffsets[4] = {0LL, 23LL, 52LL, 83LL}; +static const int64_t kTinyLayerMultipliers[3] = {112381549946653559LL, 107512210695146523LL, 55314113489879299LL}; +static const int64_t kTinyTotalVocabSize = 120LL; +static const int64_t kTinyPaddedVocabSize = 120LL; + +// modeling_qwen4_exp.py:1053-1067 _shift_right_ignore_eos +static const int64_t kShiftSeqLen = 12; +static const int64_t kShiftInput[12] = {5LL, 11LL, 12LL, 13LL, 5LL, 21LL, 5LL, 5LL, 31LL, 32LL, 33LL, 34LL}; +static const int64_t kShiftExpected[3][12] = { + {5LL, 11LL, 12LL, 13LL, 5LL, 21LL, 5LL, 5LL, 31LL, 32LL, 33LL, 34LL}, // shift=0 + {5LL, 5LL, 11LL, 12LL, 13LL, 5LL, 21LL, 5LL, 5LL, 31LL, 32LL, 33LL}, // shift=1 + {5LL, 5LL, 5LL, 11LL, 12LL, 5LL, 5LL, 5LL, 5LL, 5LL, 31LL, 32LL}, // shift=2 +}; + +// modeling_qwen4_exp.py:1069-1114 Qwen4ExpTextNGramEmbedding.forward, id half. +// The same 12 tokens, once as one prefill and once as prefill(10)+decode+decode; +// upstream produces IDENTICAL ids, which is what pins the conv-state-2 history. +static const int64_t kNgramPrefillLen = 10; +static const int64_t kNgramTotalLen = 12; +static const int64_t kNgramTokens[12] = {7LL, 8LL, 5LL, 9LL, 10LL, 11LL, 5LL, 12LL, 13LL, 14LL, 15LL, 16LL}; +static const int64_t kNgramExpectedIds[12][4] = { + {13LL, 31LL, 54LL, 119LL}, + {18LL, 26LL, 72LL, 101LL}, + {18LL, 35LL, 53LL, 100LL}, + {18LL, 50LL, 59LL, 98LL}, + {6LL, 31LL, 75LL, 96LL}, + {6LL, 41LL, 57LL, 86LL}, + {18LL, 42LL, 63LL, 104LL}, + {13LL, 37LL, 67LL, 105LL}, + {20LL, 50LL, 52LL, 109LL}, + {0LL, 41LL, 54LL, 109LL}, + {6LL, 27LL, 58LL, 89LL}, + {6LL, 41LL, 70LL, 107LL}, +}; + +// modeling_qwen4_exp.py:1181 gate.abs().clamp_min(1e-6).sqrt() * gate.sign() +// Clamp BEFORE the sqrt: the magnitude floor is 1e-3, not 1e-6, and exactly +// zero maps to zero because sign(0)=0. Discontinuous at the origin on purpose. +static const int64_t kGateCount = 11; +static const float kGateInput[11] = {0.0f, 1e-12f, -1e-12f, 1e-06f, -1e-06f, 0.001f, -0.001f, 0.25f, -0.25f, 4.0f, -4.0f}; +static const float kGateExpected[11] = {0.0f, 0.0010000000474974513f, -0.0010000000474974513f, 0.0010000000474974513f, -0.0010000000474974513f, 0.03162277862429619f, -0.03162277862429619f, 0.5f, -0.5f, 2.0f, -2.0f}; + +// modeling_qwen4_exp.py:1150-1167 _short_conv. kernel 4, dilation ngram_size=3, +// so output t reads input t-9, t-6, t-3, t with weights w0..w3 in that order. +// One-hot tap per channel over an impulse at t=0: the response lands at +// t = 9, 6, 3, 0 for w0..w3. silu is applied to the conv output. +static const int64_t kTapSeqLen = 14; +static const float kTapImpulse = 3.0f; +static const float kTapExpected[14][16] = { + {0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f}, + {0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f}, + {0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f}, + {0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f}, + {0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f}, + {0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f}, + {0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f}, + {0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f}, + {0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f}, + {2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f, 2.857722520828247f, 0.0f, 0.0f, 0.0f}, + {0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f}, + {0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f}, + {0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f}, + {0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f}, +}; + +// The full Qwen4ExpTextPLELayer forward (modeling_qwen4_exp.py:1169-1189), +// tiny config, weights drawn once and frozen here. Prefill(10)+decode+decode +// equals the single-shot 12-token prefill upstream, so both are one golden. +static const float kPleNgramEmbeddingWeight[240] = { + 0.29607248306274414f, 0.4515473246574402f, 0.3767547011375427f, -0.4875754714012146f, 0.14617228507995605f, -0.20346695184707642f, + 0.26145678758621216f, 0.22933048009872437f, -0.3022264242172241f, -0.04054975509643555f, -0.21299636363983154f, 0.42029404640197754f, + 0.31615281105041504f, -0.03174692392349243f, -0.01523822546005249f, -0.3017454147338867f, -0.13954192399978638f, 0.3712848424911499f, + -0.25462543964385986f, -0.03200578689575195f, -0.026498854160308838f, 0.2152077555656433f, 0.21153640747070312f, -0.1928795576095581f, + 0.1839163899421692f, 0.4389665722846985f, -0.3569025993347168f, -0.0961194634437561f, 0.2354411482810974f, 0.3558870553970337f, + 0.10853040218353271f, -0.05076867341995239f, -0.1677362322807312f, -0.4748031497001648f, 0.4594419002532959f, -0.34059959650039673f, + 0.3074343800544739f, -0.36951106786727905f, -0.4325920343399048f, -0.4114323854446411f, -0.26788389682769775f, 0.06253695487976074f, + 0.08628761768341064f, 0.38541531562805176f, 0.1792391538619995f, 0.3165469765663147f, -0.1388283371925354f, -0.15278029441833496f, + 0.49893540143966675f, -0.05428886413574219f, -0.4209507703781128f, -0.1046488881111145f, -0.12962466478347778f, -0.4091911315917969f, + 0.09327024221420288f, -0.3359971046447754f, -0.44438445568084717f, 0.24401819705963135f, 0.14020299911499023f, 0.3838363289833069f, + 0.09823518991470337f, 0.3225310444831848f, -0.43965452909469604f, -0.03838318586349487f, -0.18560892343521118f, -0.44965261220932007f, + -0.47458744049072266f, -0.20948761701583862f, -0.2754215598106384f, -0.1357608437538147f, 0.261415958404541f, -0.26241064071655273f, + 0.033326566219329834f, 0.17034274339675903f, 0.45190149545669556f, 0.25700879096984863f, 0.14285022020339966f, 0.05718052387237549f, + -0.009894371032714844f, -0.36505240201950073f, -0.3459804654121399f, -0.1737021803855896f, 0.35892194509506226f, 0.24941492080688477f, + -0.05600243806838989f, 0.3724104166030884f, 0.15996110439300537f, -0.3501341938972473f, 0.4328814148902893f, -0.17489510774612427f, + -0.47544705867767334f, -0.24364084005355835f, 0.21671158075332642f, -0.058866798877716064f, 0.2491776943206787f, -0.4639160633087158f, + -0.3897813558578491f, 0.3342909812927246f, -0.41046464443206787f, 0.43530821800231934f, 0.15462863445281982f, 0.37929701805114746f, + 0.18661803007125854f, -0.18792980909347534f, -0.13958382606506348f, 0.10917395353317261f, 0.2515864372253418f, 0.43623054027557373f, + -0.2127615213394165f, 0.029618680477142334f, -0.11851763725280762f, -0.2905816435813904f, -0.40270912647247314f, 0.162300705909729f, + -0.4984031915664673f, -0.41838759183883667f, -0.2211417555809021f, 0.23078423738479614f, -0.3468095660209656f, 0.2105681300163269f, + 0.4900661110877991f, 0.40847599506378174f, -0.28509896993637085f, -0.26243579387664795f, -0.28390395641326904f, -0.3020915985107422f, + -0.46217578649520874f, 0.10898643732070923f, 0.27248239517211914f, 0.09091806411743164f, 0.256955623626709f, 0.46440714597702026f, + -0.04062366485595703f, -0.3300088047981262f, 0.042962729930877686f, -0.059518516063690186f, 0.28047317266464233f, 0.3311840891838074f, + -0.2948589324951172f, 0.2879871129989624f, 0.3975728750228882f, -0.39637815952301025f, 0.04120367765426636f, 0.26779574155807495f, + 0.06296747922897339f, 0.032749950885772705f, -0.49101221561431885f, 0.3784373998641968f, -0.13032817840576172f, 0.47066670656204224f, + -0.1861809492111206f, -0.3625745177268982f, -0.3580358624458313f, -0.47317689657211304f, -0.3885725736618042f, -0.2035350203514099f, + -0.11556100845336914f, 0.33243805170059204f, -0.15663939714431763f, 0.3042781949043274f, -0.048934221267700195f, -0.042385995388031006f, + -0.15107768774032593f, 0.18044829368591309f, 0.4293113350868225f, -0.15245497226715088f, 0.456515371799469f, 0.07665467262268066f, + 0.4340183138847351f, -0.27056586742401123f, -0.1076398491859436f, -0.021817505359649658f, 0.11655020713806152f, 0.38678085803985596f, + 0.1968526840209961f, 0.4483141303062439f, 0.38949018716812134f, 0.086728036403656f, 0.25227880477905273f, 0.343669056892395f, + -0.35503363609313965f, -0.13403159379959106f, 0.28352290391921997f, 0.48622655868530273f, 0.17991304397583008f, -0.40323150157928467f, + 0.0007527470588684082f, -0.12420052289962769f, -0.0036281347274780273f, -0.23127871751785278f, 0.3529236316680908f, 0.2724175453186035f, + -0.17378675937652588f, -0.18820583820343018f, -0.3883490562438965f, 0.37534844875335693f, -0.17587810754776f, -0.3242599368095398f, + -0.13182884454727173f, -0.10293388366699219f, -0.07862132787704468f, 0.3632447123527527f, 0.46584552526474f, -0.16530072689056396f, + -0.18133199214935303f, 0.1346135139465332f, 0.20124536752700806f, -0.21512848138809204f, 0.20450401306152344f, 0.14175796508789062f, + -0.09935057163238525f, 0.13034993410110474f, 0.244953453540802f, 0.2632257342338562f, -0.12532812356948853f, -0.30538785457611084f, + 0.25927555561065674f, 0.002094089984893799f, -0.1062999963760376f, -0.0956231951713562f, 0.11332488059997559f, -0.41781002283096313f, + -0.4548985958099365f, -0.28736019134521484f, -0.4222310185432434f, 0.054963111877441406f, 0.2685357928276062f, -0.48073798418045044f, + -0.37768715620040894f, -0.24495941400527954f, -0.3113873600959778f, 0.43728774785995483f, 0.15364110469818115f, -0.3158215880393982f, + -0.09596854448318481f, -0.3130941390991211f, 0.4521239995956421f, -0.11864012479782104f, -0.15588349103927612f, 0.22496312856674194f, +}; +static const float kPleKeyProjWeight[128] = { + 0.4591003656387329f, -0.3838275671005249f, 0.47984579205513f, -0.5321495532989502f, 0.0640178918838501f, -0.47101372480392456f, + -0.49932926893234253f, 0.4292030930519104f, 0.35863396525382996f, -0.16122543811798096f, -0.3132404685020447f, 0.31758007407188416f, + -0.38730254769325256f, 0.0371936559677124f, 0.5423984527587891f, 0.21764646470546722f, -0.1643335223197937f, 0.0005162000888958573f, + -0.03969419002532959f, -0.2649257183074951f, 0.04893400892615318f, 0.06728511303663254f, -0.03810139000415802f, -0.5639750361442566f, + 0.4424150884151459f, 0.3113430440425873f, -0.31804120540618896f, 0.02285327948629856f, 0.37160754203796387f, -0.22578206658363342f, + 0.2618010640144348f, -0.11273296177387238f, 0.499276340007782f, -0.4403342604637146f, -0.5626818537712097f, -0.2811174690723419f, + -0.5117231011390686f, -0.3405351936817169f, -0.021149754524230957f, -0.06537559628486633f, 0.2809096872806549f, 0.5274608731269836f, + -0.5268707275390625f, -0.47666195034980774f, -0.10992158204317093f, -0.579092800617218f, 0.20749054849147797f, 0.5563246607780457f, + -0.26022377610206604f, -0.1799774467945099f, -0.16511844098567963f, -0.18101535737514496f, -0.5631725788116455f, -0.10599253326654434f, + -0.33210068941116333f, 0.1483088582754135f, 0.3605223000049591f, 0.1284775286912918f, 0.05024013668298721f, -0.08526471257209778f, + -0.42174118757247925f, -0.5156475305557251f, 0.0027877092361450195f, 0.5321527719497681f, -0.35091784596443176f, 0.5576460361480713f, + 0.09888296574354172f, 0.3053998351097107f, 0.5865418910980225f, -0.06388256698846817f, 0.4430769979953766f, -0.46394842863082886f, + -0.2698173522949219f, 0.12287042289972305f, 0.28638553619384766f, -0.2945103645324707f, 0.37990060448646545f, -0.3027613162994385f, + -0.07831185311079025f, -0.4756072461605072f, -0.04520981386303902f, 0.2342931181192398f, -0.25496843457221985f, -0.18489110469818115f, + 0.1793263554573059f, 0.14921708405017853f, -0.055783797055482864f, -0.1577809900045395f, 0.45896676182746887f, -0.5039750933647156f, + 0.09598753601312637f, 0.5173678398132324f, 0.14493191242218018f, -0.3914191424846649f, -0.5611379742622375f, 0.4420633316040039f, + 0.11705268174409866f, 0.2799544036388397f, 0.1931530386209488f, -0.17639023065567017f, 0.28806132078170776f, -0.2956131398677826f, + -0.49118179082870483f, -0.2040523886680603f, -0.5351635813713074f, 0.5363466143608093f, -0.26806890964508057f, 0.45721158385276794f, + -0.04339563846588135f, -0.10528114438056946f, -0.021828889846801758f, -0.049129847437143326f, 0.1425381749868393f, 0.27998265624046326f, + -0.01968863047659397f, 0.40603503584861755f, -0.5708363056182861f, -0.5718393325805664f, -0.26303166151046753f, 0.26125824451446533f, + -0.02909395843744278f, 0.1772620528936386f, -0.3250634968280792f, 0.10733971744775772f, -0.34794366359710693f, -0.3203228712081909f, + -0.0026553154457360506f, -0.24786072969436646f, +}; +static const float kPleValueProjWeight[64] = { + -0.2215331643819809f, -0.20038047432899475f, 0.34290406107902527f, 0.14115722477436066f, -0.5414190292358398f, 0.06441815197467804f, + -0.004825973883271217f, 0.5811906456947327f, -0.25694549083709717f, -0.395281583070755f, -0.43106117844581604f, -0.13067029416561127f, + -0.22148488461971283f, 0.5139840245246887f, -0.36344054341316223f, -0.5971400141716003f, 0.2075289636850357f, -0.5494977235794067f, + 0.4708784818649292f, -0.48788660764694214f, -0.3912132978439331f, -0.2311331182718277f, 0.2826908230781555f, 0.4754262864589691f, + 0.43663087487220764f, -0.2233935445547104f, 0.327562540769577f, -0.2622322142124176f, 0.5103480815887451f, -0.4005710482597351f, + -0.5766379237174988f, -0.5377451181411743f, -0.2705293893814087f, -0.4322570264339447f, 0.23162220418453217f, 0.07843337208032608f, + -0.41977688670158386f, 0.2279852032661438f, 0.04813406616449356f, -0.13870325684547424f, -0.4321030378341675f, -0.4691586494445801f, + 0.5678404569625854f, 0.50094074010849f, -0.2641710638999939f, 0.03413815796375275f, 0.5855499505996704f, -0.3433586061000824f, + -0.5433874726295471f, 0.5665374398231506f, 0.3557003438472748f, -0.05505738407373428f, 0.08937893062829971f, -0.40763261914253235f, + 0.5616267919540405f, 0.03515353426337242f, 0.08695357292890549f, -0.04259448125958443f, -0.4148881733417511f, 0.42412829399108887f, + -0.027278782799839973f, -0.5527739524841309f, 0.34241390228271484f, 0.3293036222457886f, +}; +static const float kPleNormKeyWeight[16] = { + 0.2631749212741852f, -0.054456714540719986f, -0.006164646241813898f, 0.5969558954238892f, -0.4318654239177704f, 0.2769228518009186f, + -0.2967161238193512f, 0.23676568269729614f, -0.4128349721431732f, -0.5557121634483337f, -0.3780214786529541f, -0.5922777056694031f, + 0.25040125846862793f, 0.10524473339319229f, 0.00958099402487278f, -0.5378537178039551f, +}; +static const float kPleNormQueryWeight[16] = { + -0.17194662988185883f, -0.2854304015636444f, -0.41737306118011475f, -0.11213222146034241f, 0.286573588848114f, 0.3223590552806854f, + 0.46405836939811707f, 0.3436029255390167f, -0.48538193106651306f, -0.17343635857105255f, 0.04350342974066734f, 0.18877531588077545f, + 0.1901557594537735f, 0.49197065830230713f, -0.42087316513061523f, -0.1795913577079773f, +}; +static const float kPleNormConvWeight[16] = { + 0.5488592386245728f, 0.5904802083969116f, 0.5463061928749084f, 0.023768234997987747f, -0.20656658709049225f, -0.00266017927788198f, + -0.22798791527748108f, -0.32610225677490234f, 0.29204559326171875f, 0.5127481818199158f, -0.31922751665115356f, -0.06896960735321045f, + -0.4004618227481842f, -0.14266562461853027f, 0.15500608086585999f, 0.5103983879089355f, +}; +static const float kPleConv1dWeight[64] = { + 0.37107741832733154f, 0.012878037057816982f, 0.29791462421417236f, 0.04550214111804962f, -0.2697252333164215f, -0.46802738308906555f, + 0.43851250410079956f, 0.12096326798200607f, -0.5056559443473816f, 0.20142346620559692f, -0.31201738119125366f, 0.4481829106807709f, + -0.3061846196651459f, -0.5643513798713684f, 0.42496606707572937f, 0.1797054409980774f, -0.0757109671831131f, 0.18858470022678375f, + 0.3715486228466034f, -0.10761037468910217f, -0.18812908232212067f, -0.01766674593091011f, 0.183454230427742f, 0.28105223178863525f, + -0.5942596197128296f, 0.11089332401752472f, 0.10793953388929367f, 0.2716384828090668f, -0.5218613147735596f, -0.2954745292663574f, + -0.25218966603279114f, -0.2965840995311737f, -0.5234655141830444f, -0.21591375768184662f, -0.08328352123498917f, -0.5726585388183594f, + 0.5767138600349426f, -0.23417766392230988f, 0.02776172198355198f, 0.5580352544784546f, 0.11608979851007462f, 0.3435582220554352f, + 0.44253361225128174f, 0.2667587995529175f, -0.340137243270874f, 0.07867176830768585f, -0.2633977234363556f, -0.40067899227142334f, + -0.03542983531951904f, -0.48073649406433105f, -0.15345147252082825f, 0.43864360451698303f, 0.2773081064224243f, -0.5704857110977173f, + 0.16752512753009796f, -0.29237931966781616f, 0.5423178672790527f, 0.18445688486099243f, 0.1677265465259552f, 0.006297183223068714f, + 0.1701330989599228f, -0.1537785530090332f, 0.027573825791478157f, -0.09768340736627579f, +}; +static const float kPleHiddenStates[192] = { + 0.42994368076324463f, 0.12009787559509277f, -0.6813819408416748f, -0.5209351778030396f, -0.36587417125701904f, -0.8946983814239502f, + -0.8557031154632568f, -0.4928673505783081f, 0.015934109687805176f, 0.7319419384002686f, 0.9398539066314697f, 0.2542591094970703f, + -0.9013315439224243f, -0.9814925193786621f, -0.6426703929901123f, 0.728571891784668f, 0.8614739179611206f, -0.007515430450439453f, + 0.9560174942016602f, -0.48034894466400146f, -0.45703768730163574f, -0.33108770847320557f, -0.0048302412033081055f, -0.7052638530731201f, + -0.8533051013946533f, 0.2284843921661377f, 0.7791218757629395f, 0.7223368883132935f, -0.22634267807006836f, 0.383373498916626f, + -0.6495779752731323f, 0.8873920440673828f, -0.5650629997253418f, -0.890723705291748f, -0.4897725582122803f, -0.15713238716125488f, + 0.03622102737426758f, 0.7094311714172363f, -0.9478894472122192f, -0.5226631164550781f, -0.130842924118042f, -0.5946345329284668f, + -0.3645871877670288f, -0.5258170366287231f, 0.5916182994842529f, 0.39770281314849854f, -0.8541768789291382f, 0.5310550928115845f, + -0.09971380233764648f, -0.8284300565719604f, -0.06356990337371826f, 0.09872889518737793f, 0.592972993850708f, 0.6784470081329346f, + -0.15618562698364258f, -0.27540087699890137f, -0.6508510112762451f, -0.7305476665496826f, -0.5460349321365356f, 0.5673319101333618f, + -0.6061708927154541f, -0.05077850818634033f, -0.24155330657958984f, 0.5537686347961426f, -0.7746926546096802f, -0.7317371368408203f, + 0.6445295810699463f, 0.8697048425674438f, -0.7867157459259033f, -0.05190706253051758f, -0.7304003238677979f, 0.7137643098831177f, + 0.010923981666564941f, -0.18050825595855713f, 0.65085768699646f, 0.2017526626586914f, 0.5951529741287231f, -0.7780284881591797f, + 0.32583868503570557f, 0.5915882587432861f, -0.3828732967376709f, 0.19578516483306885f, 0.3399592638015747f, 0.8385756015777588f, + -0.03268551826477051f, -0.26940059661865234f, 0.2553156614303589f, 0.5924955606460571f, 0.38599610328674316f, 0.9490455389022827f, + 0.825103759765625f, -0.37143373489379883f, -0.3511406183242798f, 0.14702725410461426f, -0.5329631567001343f, 0.5110262632369995f, + 0.8423542976379395f, 0.14798283576965332f, -0.35041773319244385f, -0.029313325881958008f, 0.5864294767379761f, -0.9132624864578247f, + -0.3846536874771118f, 0.43202292919158936f, -0.8588521480560303f, 0.45785701274871826f, 0.5951042175292969f, 0.47855043411254883f, + 0.600839376449585f, 0.51995849609375f, -0.12280678749084473f, 0.13126718997955322f, 0.0936119556427002f, -0.4713503122329712f, + -0.6044362783432007f, 0.7798174619674683f, -0.8693674802780151f, 0.4778156280517578f, 0.07839822769165039f, 0.4906245470046997f, + -0.43552136421203613f, 0.373479962348938f, -0.003823518753051758f, -0.7637084722518921f, 0.770892858505249f, -0.8814637660980225f, + -0.054804444313049316f, -0.6741824150085449f, -0.03066849708557129f, 0.6769529581069946f, 0.9880853891372681f, 0.04106295108795166f, + 0.887357234954834f, 0.8562901020050049f, 0.8683667182922363f, 0.29182708263397217f, 0.4707142114639282f, 0.5562174320220947f, + -0.014180421829223633f, 0.7794408798217773f, -0.549174427986145f, -0.6584701538085938f, 0.550702691078186f, 0.6509667634963989f, + 0.24143505096435547f, 0.025332212448120117f, 0.7478917837142944f, -0.9601776599884033f, 0.574150562286377f, 0.9295830726623535f, + 0.4401681423187256f, 0.5850152969360352f, 0.9502300024032593f, 0.9177200794219971f, -0.13980627059936523f, 0.7096506357192993f, + -0.8738625049591064f, 0.043054819107055664f, 0.42393815517425537f, -0.5795249938964844f, -0.2191382646560669f, -0.5994499921798706f, + 0.8732306957244873f, 0.12746977806091309f, 0.6903105974197388f, -0.6672602891921997f, 0.9127757549285889f, 0.50844407081604f, + 0.20082366466522217f, 0.6839362382888794f, 0.4696390628814697f, -0.4303417205810547f, 0.768852949142456f, 0.04934358596801758f, + -0.22454547882080078f, 0.43524205684661865f, 0.4797931909561157f, -0.8059563636779785f, 0.8963567018508911f, 0.6971317529678345f, + -0.2866344451904297f, -0.618877649307251f, 0.513076663017273f, -0.41625237464904785f, 0.8399109840393066f, -0.3447309732437134f, + 0.2896728515625f, -0.008451461791992188f, -0.3984866142272949f, -0.15852713584899902f, -0.7220062017440796f, -0.39515388011932373f, +}; +static const float kPleExpectedOutput[192] = { + 0.06963851302862167f, 0.21422696113586426f, -0.11206153780221939f, -0.22873759269714355f, 0.0010560564696788788f, -0.137812077999115f, + -0.08115558326244354f, -0.032832808792591095f, -0.15659968554973602f, 1.0040206909179688f, -0.051519326865673065f, 0.3333406448364258f, + 0.08705511689186096f, 0.05229533463716507f, -0.04408198222517967f, 0.002707485109567642f, -0.1200815886259079f, 0.09625422209501266f, + 1.430494785308838f, 0.10445044934749603f, 0.0053314464166760445f, 0.08579660207033157f, -0.13021141290664673f, -0.0013515392784029245f, + 0.5393670797348022f, 0.33759620785713196f, 0.3999481201171875f, -0.06024087592959404f, 0.04390620440244675f, -0.019709967076778412f, + -0.05843441188335419f, -0.0001627565361559391f, 0.06671445816755295f, -0.019437722861766815f, 1.0179795026779175f, 0.11037059128284454f, + 0.007436065934598446f, -0.12247934937477112f, -0.27106982469558716f, 0.03849131613969803f, -0.11767642199993134f, -0.054666668176651f, + 0.32369378209114075f, -0.04769214615225792f, 0.04488477483391762f, 0.02709689736366272f, -0.15008333325386047f, -0.009908579289913177f, + 0.2800644338130951f, 0.9980143308639526f, 0.13671976327896118f, -0.2024199664592743f, 0.08122120052576065f, 0.1965084969997406f, + -0.2690304219722748f, -0.04112722724676132f, -0.05882590264081955f, 1.0883033275604248f, -0.007805900648236275f, 0.2821679711341858f, + 0.4248272478580475f, 0.1453343629837036f, -0.4830322861671448f, -0.05453818663954735f, -0.3176023066043854f, 0.385914146900177f, + -0.3134472668170929f, 0.39185404777526855f, -0.003477673977613449f, -0.30416393280029297f, -0.26470404863357544f, 0.012368187308311462f, + 0.4136078357696533f, 0.4574280381202698f, 0.3324526846408844f, -0.003704383969306946f, -0.2064172625541687f, -0.021773457527160645f, + -0.310366153717041f, 0.13362863659858704f, 0.6006263494491577f, -0.5882097482681274f, 0.5800319910049438f, -0.003911174833774567f, + 0.08685152232646942f, 0.21343375742435455f, 0.05491143837571144f, 0.28066036105155945f, -0.12178359925746918f, -0.4484730362892151f, + 0.7078319191932678f, -0.0474725179374218f, 0.047475192695856094f, -0.03599635511636734f, -0.09762144088745117f, 0.02086193859577179f, + 0.4411122500896454f, -0.05061407387256622f, 0.6103701591491699f, 0.5720752477645874f, 0.4108288884162903f, 0.6699548959732056f, + -0.4596680998802185f, 0.11456148326396942f, -0.1760299950838089f, -0.1293385773897171f, 0.16831707954406738f, -0.020343616604804993f, + 0.0886760726571083f, 0.2682039737701416f, -0.28017544746398926f, -0.05189185589551926f, -0.05770047754049301f, -0.1159166470170021f, + 0.8843564987182617f, 0.027845818549394608f, -0.0414850078523159f, 0.26933252811431885f, 0.10103844851255417f, -0.08441299200057983f, + -0.11158525943756104f, -0.29708218574523926f, 0.3321762979030609f, -0.08550890535116196f, 0.17899411916732788f, -0.12792372703552246f, + -0.08784984797239304f, 0.020405316725373268f, 0.4983510673046112f, -0.2341698557138443f, -0.3229801058769226f, -0.2685205936431885f, + 0.04790535196661949f, 0.4944710433483124f, 0.011328248307108879f, 0.05705556645989418f, -0.21590931713581085f, 0.29338935017585754f, + 0.34298810362815857f, 0.24224287271499634f, 0.15510395169258118f, 0.06366173923015594f, -0.09562748670578003f, 0.193404421210289f, + 0.8094600439071655f, -0.33599114418029785f, -0.3033950626850128f, -0.033202461898326874f, 0.2605142593383789f, 0.2324167937040329f, + 0.1412377655506134f, -0.09495069086551666f, -0.25441962480545044f, -0.3699108362197876f, 0.01524687185883522f, 0.5038323998451233f, + -0.22638384997844696f, -0.22415225207805634f, -0.2235727608203888f, 0.22215968370437622f, 0.08017485588788986f, -0.3407858908176422f, + -0.016165688633918762f, -0.3586617410182953f, 0.07874535024166107f, 0.03082359954714775f, 0.11969424039125443f, -0.08476392924785614f, + 0.17316974699497223f, -0.2065928429365158f, 0.5375539660453796f, 0.09750574827194214f, 0.08156198263168335f, 0.9381179213523865f, + -0.2288454920053482f, -0.11323010921478271f, 0.21552065014839172f, 0.8414918184280396f, -0.27386200428009033f, 0.6621967554092407f, + 0.06562049686908722f, 0.4316128194332123f, 0.6156079769134521f, 0.17598508298397064f, -0.17269927263259888f, -0.24345244467258453f, + -0.004823658615350723f, -0.005428016185760498f, -0.19190451502799988f, -0.07814845442771912f, -0.24328623712062836f, -0.18724854290485382f, +}; +// modeling_qwen4_exp.py:1185-1187 + :204-213 apply_mask_to_padding_states. +// BOTH `gated_value` (the skip term) and `gated_value_normed` (the conv +// input, and what the 9-column state keeps) are masked. Zeros at 3, 4 and 11; +// 3 and 4 are interior, so the dilation carries them to t = 6, 9 and 12 too. +// Masked positions carry EOS in the tokens, which is the paired obligation. +static const int64_t kPleMaskTokens[12] = {7LL, 8LL, 5LL, 5LL, 5LL, 11LL, 5LL, 12LL, 13LL, 14LL, 15LL, 5LL}; +static const unsigned char kPleConvMask[12] = {1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0}; +static const float kPleMaskedExpectedOutput[192] = { + 0.06963851302862167f, 0.21422696113586426f, -0.11206153780221939f, -0.22873759269714355f, 0.0010560564696788788f, -0.137812077999115f, + -0.08115558326244354f, -0.032832808792591095f, -0.15659968554973602f, 1.0040206909179688f, -0.051519326865673065f, 0.3333406448364258f, + 0.08705511689186096f, 0.05229533463716507f, -0.04408198222517967f, 0.002707485109567642f, -0.1200815886259079f, 0.09625422209501266f, + 1.430494785308838f, 0.10445044934749603f, 0.0053314464166760445f, 0.08579660207033157f, -0.13021141290664673f, -0.0013515392784029245f, + 0.5393670797348022f, 0.33759620785713196f, 0.3999481201171875f, -0.06024087592959404f, 0.04390620440244675f, -0.019709967076778412f, + -0.05843441188335419f, -0.0001627565361559391f, 0.06671445816755295f, -0.019437722861766815f, 1.0179795026779175f, 0.11037059128284454f, + 0.007436065934598446f, -0.12247934937477112f, -0.27106982469558716f, 0.03849131613969803f, -0.11767642199993134f, -0.054666668176651f, + 0.32369378209114075f, -0.04769214615225792f, 0.04488477483391762f, 0.02709689736366272f, -0.15008333325386047f, -0.009908579289913177f, + 0.23775668442249298f, 0.7061492800712585f, 0.08241425454616547f, -0.25137707591056824f, 0.0639595165848732f, -0.0691579282283783f, + -0.023062696680426598f, -0.0540279783308506f, -0.044211700558662415f, 0.03015831857919693f, -0.04553074389696121f, 0.28566351532936096f, + -0.01847454346716404f, -0.05523449555039406f, -0.051907993853092194f, 0.014243440702557564f, -0.2009752094745636f, 0.23813265562057495f, + -0.2737903892993927f, 0.15705174207687378f, 0.03259096294641495f, 0.037811268121004105f, -0.03212312236428261f, -0.0035384097136557102f, + 0.06810037791728973f, 0.012128463014960289f, 0.4468013346195221f, -0.07179546356201172f, -0.009762369096279144f, 0.029456978663802147f, + -0.07129035145044327f, 0.000870966468937695f, -0.07106472551822662f, -0.07339988648891449f, -0.39384520053863525f, 0.24265798926353455f, + -0.0645628497004509f, -0.46886110305786133f, 0.0020889602601528168f, 0.11040981113910675f, 0.4985309839248657f, -0.08674318343400955f, + 0.09200461953878403f, -0.08311727643013f, -0.27136898040771484f, 0.024317920207977295f, -0.09939116984605789f, -0.029169335961341858f, + 0.31887221336364746f, -0.15543347597122192f, 0.5094432830810547f, 0.15910649299621582f, 0.27248257398605347f, 0.6276422739028931f, + -0.3983144164085388f, 0.03545404225587845f, -0.17935967445373535f, -0.04118163883686066f, 0.1088288277387619f, 0.11402899026870728f, + 0.1671668440103531f, 0.15087586641311646f, -0.14943216741085052f, -0.05795801803469658f, 0.13441729545593262f, -0.2563335597515106f, + 0.7291801571846008f, -0.16095547378063202f, 0.04023926705121994f, 0.4613253176212311f, 0.1580274999141693f, 0.003852732479572296f, + -0.13347779214382172f, -0.30062347650527954f, 0.4102042615413666f, 0.034553322941064835f, 0.1440555602312088f, -0.09546968340873718f, + -0.0010418295860290527f, -0.00350874662399292f, -0.15938086807727814f, 0.05856498330831528f, 0.018491171300411224f, -0.18785607814788818f, + -0.12117926776409149f, 0.1693689078092575f, 0.010805566795170307f, 0.1680866926908493f, -0.17796170711517334f, 0.33420369029045105f, + -0.12037542462348938f, 0.1158112958073616f, 0.2365470826625824f, -0.07164666801691055f, -0.09651516377925873f, 0.1637597680091858f, + 0.7265353202819824f, -0.32033342123031616f, -0.29595038294792175f, 0.05117378383874893f, 0.16227085888385773f, 0.27207571268081665f, + 0.24354447424411774f, -0.10079684853553772f, -0.24918398261070251f, -0.3071284294128418f, -0.035049960017204285f, 0.5389229655265808f, + -0.12851496040821075f, -0.008436496369540691f, -0.1590823382139206f, 0.14573122560977936f, 0.09002771228551865f, -0.32282212376594543f, + 0.01244974136352539f, -0.25766414403915405f, 0.13496313989162445f, 0.015602927654981613f, 0.18957042694091797f, 0.021681567654013634f, + 0.02780049294233322f, -0.12827081978321075f, 0.6017333269119263f, 0.05039282888174057f, -0.014741630293428898f, 0.27826401591300964f, + -0.15154367685317993f, 0.002976030111312866f, 0.48020434379577637f, 0.3494221866130829f, -0.2699730098247528f, -0.24667038023471832f, + 0.05360059812664986f, 0.2971128821372986f, 0.561543345451355f, 0.3018703758716583f, -0.057305362075567245f, -0.007343125995248556f, + -0.17572568356990814f, 0.023971429094672203f, 0.13929112255573273f, 0.796837329864502f, -0.25649985671043396f, -0.12321992963552475f, +}; diff --git a/tests/vllm/models/test_qwen4_exp_ple.cpp b/tests/vllm/models/test_qwen4_exp_ple.cpp new file mode 100644 index 000000000..50ce7f15c --- /dev/null +++ b/tests/vllm/models/test_qwen4_exp_ple.cpp @@ -0,0 +1,540 @@ +// Qwen4-Exp W2 gate — issue #1987, campaign issue #1978, spec +// `.agents/specs/qwen4-exp-flash-next.md`. +// +// WHY THIS FILE IS THE WHOLE GATE FOR THESE TWO COMPONENTS. They are the only +// parts of `Qwen4ExpForConditionalGeneration` with no vLLM op, so there is no +// mirrored implementation to diff against, and the spec's `## Gates` admits no +// token gate for this row at all until an arm runs — nothing published fits any +// fleet device. G0, component goldens, is the only gate reachable today. Below +// it there is nothing: an n-gram id computed wrong does not crash and does not +// change a shape, it reads a different row of a 320-million-row table and the +// model emits plausible text. +// +// ORACLE: huggingface/transformers **v5.16.0**, this row's accepted lane pin +// (spec `## Oracles`). Every expected value in +// `qwen4_exp_ple_goldens.inc` was produced by EXECUTING upstream's own bytes: +// `scripts/gen-qwen4-exp-ple-goldens.py` fetches +// `models/qwen4_exp/modeling_qwen4_exp.py` and `cache_utils.py` from +// raw.githubusercontent.com at the `v5.16.0` tag and `exec`s the named line +// ranges verbatim, never transcribing them, so a golden here is an oracle +// observation and not a prediction. The n-gram ids are the one value upstream +// does not put on a return path (`forward` returns embeddings, :1114); they are +// RECOVERED from its own output by filling row i of the embedding with the +// scalar i, never rebuilt by re-running the loop at :1097-1112, because the +// generator and `BuildNGramIds` would then share one reading of those lines. +// The real-config block is confirmed a fourth way beyond the three in #1987: +// `vocab_size = 248320`, read from the released `config.json`, is the UNIQUE +// preimage below 2e6 of the published `layer_multipliers`. +// +// What it proves, on CPU, with no GPU and without the ~360 GB checkpoint: +// (1) the splitmix64 chain is UNSIGNED — divergence site #1. Four of the +// eight probes return a value above 2^63, so an `int64_t` port whose +// `>>` went arithmetic fails on the raw chain, before any derived value; +// (2) `% half_bound` is UNSIGNED — divergence site #2 — pinned through the +// three published multipliers at the REAL config; +// (3) the head vocab sizes are the successive primes after 19999999, their +// offsets are the exclusive prefix sum, and the padded table has exactly +// 90 unaddressable rows; +// (4) `_shift_right_ignore_eos` gets the EOS-SEGMENT semantics right, +// including the case that separates it from a plain shift: an EOS token +// belongs to the segment it TERMINATES, not the one it opens; +// (5) the n-gram ids from prefill(10)+decode+decode equal the single-shot +// 12-token prefill — the conv-state-2 history, EOS-padded and never +// zero-padded; +// (6) the signed-sqrt gate CLAMPS BEFORE THE SQRT: the magnitude floor is +// 1e-3, tiny scores are amplified rather than squashed, and exactly zero +// maps to zero; +// (7) the dilated depthwise conv reads lags {9, 6, 3, 0}, proved by a +// one-hot tap per channel over an impulse, so a unit-stride or reversed +// tap order moves the response; +// (8) the whole PLE forward matches upstream, and its incremental arm equals +// its single-shot arm through the 9-column state; +// (9) `conv_mask` masks BOTH the skip term and the conv input (:1185-1187), +// and the mask carries through the 9-column state across a chunk break; +// (10) `eos_token_id` is range-checked like any other id. It is the one id in +// the mix that does not come from `input_ids`, it is on the FIRST TOKEN +// OF EVERY SEQUENCE, and at the struct's own `-1` default ours and +// upstream produce DIFFERENT rows of the table with no exception on +// either side. +// +// `qwen4_exp_ple.h` is a MODEL-PRIVATE header under `src/`, like +// `dots3_note.h`: W2 ships no public ABI, because nothing is reachable from a +// production entry point until W5 assembles the model. See the header's scope +// block and the spec's `## Owed`. + +#include "vllm/model_executor/models/qwen4_exp_ple.h" + +#include + +#include "support/max_abs_diff.h" + +#include +#include +#include +#include + +namespace { + +#include "vllm/models/qwen4_exp_ple_goldens.inc" + +using vllm::qwen4_exp::BuildLayerMultipliers; +using vllm::qwen4_exp::BuildNGramIds; +using vllm::qwen4_exp::BuildNGramTableLayout; +using vllm::qwen4_exp::FindNthPrimeAfter; +using vllm::qwen4_exp::IsPrime; +using vllm::qwen4_exp::NGramTableLayout; +using vllm::qwen4_exp::PleForward; +using vllm::qwen4_exp::PleGeometry; +using vllm::qwen4_exp::PleSequenceState; +using vllm::qwen4_exp::PleShortConv; +using vllm::qwen4_exp::PleWeights; +using vllm::qwen4_exp::SignedSqrtGate; +using vllm::qwen4_exp::SplitMix64; + +// The tiny config the runnable goldens were generated at. Kept in one place so +// a test cannot silently disagree with the generator about a shape. +PleGeometry TinyGeometry() { + PleGeometry geom; + geom.hidden_size = kTinyHiddenSize; + geom.hc_count = kTinyHcCount; + geom.ple_embed_dim = kTinyPleEmbedDim; + geom.ple_conv_kernel_size = kTinyConvKernel; + geom.ngram_size = kTinyNgramSize; + geom.heads_per_ngram = kTinyHeadsPerNgram; + geom.ngram_vocab_size_base = kTinyNgramVocabBase; + geom.make_ngram_vocab_size_divisible_by = kTinyVocabDivisor; + geom.vocab_size = kTinyVocabSize; + geom.eos_token_id = kTinyEosTokenId; + geom.seed = kTinySeed; + geom.rms_norm_eps = 1e-6; + return geom; +} + +PleWeights TinyWeights() { + PleWeights w; + w.ngram_embedding = kPleNgramEmbeddingWeight; + w.key_proj = kPleKeyProjWeight; + w.value_proj = kPleValueProjWeight; + w.norm_key = kPleNormKeyWeight; + w.norm_query = kPleNormQueryWeight; + w.norm_conv = kPleNormConvWeight; + w.conv1d = kPleConv1dWeight; + return w; +} + +} // namespace + +TEST_CASE("qwen4_exp splitmix64 is unsigned end to end") { + // Divergence site #1. Assert the property that makes it load-bearing rather + // than only the values: a signed `>>` cannot be told apart on inputs whose + // top bit is clear, so the probe set has to contain some whose top bit is set. + int top_bit_set = 0; + for (const auto& probe : kSplitMix64) { + CHECK(SplitMix64(probe.in) == probe.out); + if ((probe.out >> 63) != 0U) ++top_bit_set; + } + CHECK(top_bit_set >= 3); +} + +TEST_CASE("qwen4_exp layer multipliers match the released checkpoint") { + // Divergence site #2, at the REAL config. These three values were published + // in #1987 from three independent readings (the chain, a range read of + // `model-00005-of-00131.safetensors`, and the GGUF key), and `seed` is 1234 + // because `config.seed` is ABSENT from the published config.json. + const std::vector got = + BuildLayerMultipliers(kRealVocabSize, 3, /*ple_layer_index=*/0, kRealSeed); + REQUIRE(got.size() == 3U); + for (int i = 0; i < 3; ++i) CHECK(got[static_cast(i)] == kRealLayerMultipliers[i]); + + // Every multiplier is ODD by construction (`2 * x + 1`) and POSITIVE. A + // signed modulo makes the residue negative, which makes the multiplier even + // AND negative; both halves are checked so neither alone can carry the case. + for (int64_t m : got) { + CHECK(m > 0); + CHECK(m % 2 == 1); + } + // The bound the int64 forward rests on: multiplier_max * vocab_size < 2^63. + for (int64_t m : got) CHECK(m < INT64_MAX / kRealVocabSize); +} + +TEST_CASE("qwen4_exp n-gram head vocab sizes are the successive primes") { + // `_is_prime` (modeling_qwen4_exp.py:998-1006) branch by branch. Both call + // sites start above 19999998, so nothing else in this file ever reaches the + // even arm, the `< 2` arm or 2 itself; without the block below, + // `if (value % 2 == 0) return value == 2;` can be replaced by `return false` + // and the whole gate stays green (mutation C3 in the fresh review). + CHECK(IsPrime(2)); + CHECK_FALSE(IsPrime(4)); + CHECK_FALSE(IsPrime(1)); + CHECK_FALSE(IsPrime(0)); + CHECK_FALSE(IsPrime(-7)); + CHECK(IsPrime(3)); + CHECK_FALSE(IsPrime(9)); + CHECK_FALSE(IsPrime(20000000)); // even, at the scale the model uses + CHECK_FALSE(IsPrime(20000001)); // odd composite: the trial-division arm + + int64_t running = 0; + for (int i = 0; i < 16; ++i) { + CHECK(FindNthPrimeAfter(20000000 - 1, i + 1) == kRealHeadVocabSizes[i]); + CHECK(kRealHeadOffsets[i] == running); + running += kRealHeadVocabSizes[i]; + } + CHECK(running == kRealTotalVocabSize); + + PleGeometry real; + real.hidden_size = 2560; + real.hc_count = 4; + real.ple_embed_dim = 2560; + real.ngram_size = 3; + real.heads_per_ngram = 8; + real.vocab_size = kRealVocabSize; + real.eos_token_id = 248044; + real.seed = kRealSeed; + const NGramTableLayout layout = BuildNGramTableLayout(real, /*ple_layer_index=*/0); + REQUIRE(layout.head_vocab_sizes.size() == 16U); + for (int i = 0; i < 16; ++i) { + CHECK(layout.head_vocab_sizes[static_cast(i)] == kRealHeadVocabSizes[i]); + CHECK(layout.head_offsets[static_cast(i)] == kRealHeadOffsets[i]); + } + CHECK(layout.total_vocab_size == kRealTotalVocabSize); + CHECK(layout.padded_vocab_size == kRealPaddedVocabSize); + // 90 rows the hash can never address, which is what the padding costs. + CHECK(layout.padded_vocab_size - layout.total_vocab_size == 90); + CHECK(real.head_dim_per_ngram() == 160); + CHECK(real.short_conv_state_len() == 9); // (4-1)*3, NOT kernel-1 + CHECK(real.stream_width() == 10240); +} + +TEST_CASE("qwen4_exp _shift_right_ignore_eos honours EOS segments") { + std::vector got(static_cast(kShiftSeqLen)); + for (int64_t shift = 0; shift < 3; ++shift) { + vllm::qwen4_exp::ShiftRightIgnoreEos(kShiftInput, kShiftSeqLen, shift, + kTinyEosTokenId, got.data()); + for (int64_t i = 0; i < kShiftSeqLen; ++i) { + CHECK(got[static_cast(i)] == kShiftExpected[shift][i]); + } + } + // The one case that separates this from a plain shift-with-EOS-fill: index 4 + // IS an EOS and still reads token 3, because the "previous EOS" scan is + // strictly-before, so an EOS belongs to the segment it terminates. Index 5 + // opens a new segment and therefore reads EOS. + CHECK(kShiftInput[4] == kTinyEosTokenId); + CHECK(kShiftExpected[1][4] == kShiftInput[3]); + CHECK(kShiftExpected[1][5] == kTinyEosTokenId); +} + +TEST_CASE("qwen4_exp n-gram ids: incremental decode equals single-shot prefill") { + const PleGeometry geom = TinyGeometry(); + const NGramTableLayout layout = BuildNGramTableLayout(geom, /*ple_layer_index=*/0); + REQUIRE(layout.head_vocab_sizes.size() == static_cast(geom.ngram_heads())); + for (int64_t h = 0; h < geom.ngram_heads(); ++h) { + CHECK(layout.head_vocab_sizes[static_cast(h)] == kTinyHeadVocabSizes[h]); + CHECK(layout.head_offsets[static_cast(h)] == kTinyHeadOffsets[h]); + } + for (int64_t i = 0; i < geom.ngram_size; ++i) { + CHECK(layout.layer_multipliers[static_cast(i)] == kTinyLayerMultipliers[i]); + } + CHECK(layout.total_vocab_size == kTinyTotalVocabSize); + CHECK(layout.padded_vocab_size == kTinyPaddedVocabSize); + + const int64_t heads = geom.ngram_heads(); + SUBCASE("single shot") { + PleSequenceState state; + state.Reset(geom); + std::vector ids(static_cast(kNgramTotalLen * heads)); + BuildNGramIds(geom, layout, kNgramTokens, kNgramTotalLen, &state, ids.data()); + for (int64_t t = 0; t < kNgramTotalLen; ++t) { + for (int64_t h = 0; h < heads; ++h) { + CHECK(ids[static_cast(t * heads + h)] == kNgramExpectedIds[t][h]); + } + } + // The history the next step will read is the LAST context_len tokens. + REQUIRE(state.tokens.size() == 2U); + CHECK(state.tokens[0] == kNgramTokens[kNgramTotalLen - 2]); + CHECK(state.tokens[1] == kNgramTokens[kNgramTotalLen - 1]); + } + + SUBCASE("prefill then two decode steps") { + PleSequenceState state; + state.Reset(geom); + // Seeded with EOS, never with zero: `update_conv_state` pads with 0, which + // is a VALID token id, so upstream works around it explicitly. + CHECK(state.tokens[0] == kTinyEosTokenId); + CHECK(state.tokens[1] == kTinyEosTokenId); + + std::vector ids(static_cast(kNgramTotalLen * heads)); + BuildNGramIds(geom, layout, kNgramTokens, kNgramPrefillLen, &state, ids.data()); + for (int64_t t = kNgramPrefillLen; t < kNgramTotalLen; ++t) { + BuildNGramIds(geom, layout, kNgramTokens + t, 1, &state, + ids.data() + t * heads); + } + for (int64_t t = 0; t < kNgramTotalLen; ++t) { + for (int64_t h = 0; h < heads; ++h) { + CHECK(ids[static_cast(t * heads + h)] == kNgramExpectedIds[t][h]); + } + } + } + + SUBCASE("an out-of-range token id is refused by name, not silently overflowed") { + PleSequenceState state; + state.Reset(geom); + const int64_t bad[1] = {kTinyVocabSize}; + std::vector ids(static_cast(heads)); + CHECK_THROWS_AS(BuildNGramIds(geom, layout, bad, 1, &state, ids.data()), + std::invalid_argument); + } + + SUBCASE("an out-of-range eos_token_id is refused too") { + // `eos_token_id` is the one id in the mix that does NOT come from + // `input_ids`, so the loop above cannot see it: `Reset` seeds the history + // with it and `_shift_right_ignore_eos` emits it at every segment start, + // which puts it on the FIRST TOKEN OF EVERY SEQUENCE. Measured against + // transformers v5.16.0 on the 12 tokens below, reading the ids out of + // upstream's own `forward` with an invertible embedding: + // eos = -1 upstream row 0 [2, 35, 67, 96] ours [8, 30, 67, 96] + // eos = 1000000 upstream row 0 [9, 38, 65, 108] ours [9, 38, 81, 83] + // No exception, no shape change, a different row of the table. Upstream is + // safe without a check because `config.eos_token_id` cannot be out of + // range; our geometry is a plain struct whose eos DEFAULTS TO -1, and + // upstream's config even admits a LIST (modeling_qwen4_exp.py:1032 takes + // element [0]), so a loader has a real way to mis-set it. + for (const int64_t bad_eos : {int64_t{-1}, kTinyVocabSize, int64_t{1000000}}) { + PleGeometry bad = geom; + bad.eos_token_id = bad_eos; + CHECK_THROWS_AS(BuildNGramTableLayout(bad, /*ple_layer_index=*/0), + std::invalid_argument); + PleSequenceState state; + state.Reset(bad); + std::vector ids(static_cast(heads)); + CHECK_THROWS_AS(BuildNGramIds(bad, layout, kNgramTokens, 1, &state, ids.data()), + std::invalid_argument); + } + // The valid boundary values are NOT refused, so the guard cannot be a + // blanket refusal wearing a range check. + for (const int64_t ok_eos : {int64_t{0}, kTinyVocabSize - 1}) { + PleGeometry ok = geom; + ok.eos_token_id = ok_eos; + CHECK_NOTHROW(BuildNGramTableLayout(ok, /*ple_layer_index=*/0)); + PleSequenceState state; + state.Reset(ok); + std::vector ids(static_cast(heads)); + CHECK_NOTHROW(BuildNGramIds(ok, layout, kNgramTokens, 1, &state, ids.data())); + } + } +} + +TEST_CASE("qwen4_exp signed-sqrt gate clamps before the sqrt") { + std::vector got; + got.reserve(static_cast(kGateCount)); + for (int64_t i = 0; i < kGateCount; ++i) got.push_back(SignedSqrtGate(kGateInput[i])); + // Bounds, and why they are what they are. Observed max|diff| against the + // oracle is 2.38419e-07 = 2^-22 for BOTH the conv and the full forward, and + // below 1e-07 for the gate, whose sqrt is effectively exact. That single + // number means two different things on the two sites, so state them apart: + // * conv, bound 1e-6: argmax element silu(3.0) = 2.8577, one ULP there is + // 2.384e-07, so the observation is ONE ULP and the bound is ~4x above it; + // * forward, bound 1e-5: argmax element 1.08830333, one ULP there is + // 1.192e-07, so the observation is TWO ULP and the bound is ~42x above it. + // Both bounds sit above the observation rather than on it, to survive a + // different libm, and ~4 orders BELOW the O(0.1) error any real defect here + // produces: every mutation in the table went red. R5 in the fresh review — + // dropping `rms_norm_eps` entirely, a 1e-6 perturbation — still goes red at + // 1e-5, so the looser of the two bounds discriminates. + const double worst = + vllm_test::MaxAbsDiff(got, kGateExpected, static_cast(kGateCount)); + MESSAGE("signed-sqrt gate max|diff| vs transformers v5.16.0 = " << worst); + CHECK(worst < 1e-7); + // The three properties the values encode, asserted so the intent survives a + // regenerated golden: the origin maps to zero, the magnitude floor is 1e-3 + // rather than 1e-6, and a tiny score is AMPLIFIED. + CHECK(SignedSqrtGate(0.0F) == 0.0F); + CHECK(std::abs(SignedSqrtGate(1e-12F)) > 9.9e-4F); + CHECK(std::abs(SignedSqrtGate(1e-12F)) > std::abs(1e-12F) * 1e6F); + CHECK(SignedSqrtGate(-1e-12F) < 0.0F); +} + +TEST_CASE("qwen4_exp PLE conv reads lags 9, 6, 3 and 0") { + PleGeometry geom = TinyGeometry(); + const int64_t width = geom.stream_width(); + const int64_t kernel = geom.ple_conv_kernel_size; + REQUIRE(geom.short_conv_state_len() == kTinyShortConvStateLen); + + // One-hot tap per channel: channel c carries its whole weight at kernel index + // c % 4. A unit-stride conv, a reversed tap order or a `kernel-1` state all + // move where the impulse lands. + std::vector conv_weight(static_cast(width * kernel), 0.0F); + for (int64_t c = 0; c < width; ++c) { + conv_weight[static_cast(c * kernel + (c % kernel))] = 1.0F; + } + std::vector normed(static_cast(kTapSeqLen * width), 0.0F); + for (int64_t c = 0; c < width; ++c) normed[static_cast(c)] = kTapImpulse; + + PleSequenceState state; + state.Reset(geom); + std::vector got(static_cast(kTapSeqLen * width)); + PleShortConv(geom, conv_weight.data(), normed.data(), kTapSeqLen, &state, + got.data()); + const double worst = vllm_test::MaxAbsDiff(got, &kTapExpected[0][0], + static_cast(kTapSeqLen * width)); + MESSAGE("dilated conv max|diff| vs transformers v5.16.0 = " << worst); + CHECK(worst < 1e-6); + // Read the lag structure straight off the response so a regenerated golden + // cannot quietly change it: channel c responds at t = (3 - c % 4) * 3. + for (int64_t c = 0; c < width; ++c) { + const int64_t lag = (kernel - 1 - (c % kernel)) * geom.ngram_size; + CHECK(got[static_cast(lag * width + c)] > 1.0F); + } +} + +TEST_CASE("qwen4_exp PLE forward matches transformers v5.16.0") { + const PleGeometry geom = TinyGeometry(); + const NGramTableLayout layout = BuildNGramTableLayout(geom, /*ple_layer_index=*/0); + const PleWeights weights = TinyWeights(); + const int64_t width = geom.stream_width(); + + SUBCASE("single shot") { + PleSequenceState state; + state.Reset(geom); + std::vector got(static_cast(kNgramTotalLen * width)); + PleForward(geom, layout, weights, kPleHiddenStates, kNgramTokens, kNgramTotalLen, + /*conv_mask=*/nullptr, &state, got.data()); + const double worst = vllm_test::MaxAbsDiff(got, kPleExpectedOutput, + static_cast(kNgramTotalLen * width)); + MESSAGE("PLE forward max|diff| vs transformers v5.16.0 = " << worst); + CHECK(worst < 1e-5); + } + + SUBCASE("prefill then two decode steps, through the 9-column state") { + PleSequenceState state; + state.Reset(geom); + std::vector got(static_cast(kNgramTotalLen * width)); + PleForward(geom, layout, weights, kPleHiddenStates, kNgramTokens, kNgramPrefillLen, + nullptr, &state, got.data()); + for (int64_t t = kNgramPrefillLen; t < kNgramTotalLen; ++t) { + PleForward(geom, layout, weights, kPleHiddenStates + t * width, kNgramTokens + t, + 1, nullptr, &state, got.data() + t * width); + } + const double worst = vllm_test::MaxAbsDiff(got, kPleExpectedOutput, + static_cast(kNgramTotalLen * width)); + MESSAGE("PLE forward max|diff| vs transformers v5.16.0 = " << worst); + CHECK(worst < 1e-5); + } + + SUBCASE("the skip term is the UN-NORMED copy") { + // Zero the conv weights: the conv contributes silu(0) = 0 and the output + // collapses to the skip term alone. If the skip took the NORMED copy the + // result would be RMS-normalised, so its per-group RMS would be pinned near + // 1; it is not, and that is the fork the spec warns about. + std::vector zero_conv(static_cast(width * geom.ple_conv_kernel_size), + 0.0F); + PleWeights w = weights; + w.conv1d = zero_conv.data(); + PleSequenceState state; + state.Reset(geom); + std::vector got(static_cast(kNgramTotalLen * width)); + PleForward(geom, layout, w, kPleHiddenStates, kNgramTokens, kNgramTotalLen, + nullptr, &state, got.data()); + + bool any_group_far_from_unit_rms = false; + for (int64_t t = 0; t < kNgramTotalLen; ++t) { + for (int64_t s = 0; s < geom.hc_count; ++s) { + double sumsq = 0.0; + for (int64_t d = 0; d < geom.hidden_size; ++d) { + const double v = got[static_cast(t * width + s * geom.hidden_size + d)]; + sumsq += v * v; + } + const double rms = std::sqrt(sumsq / static_cast(geom.hidden_size)); + if (std::abs(rms - 1.0) > 0.2) any_group_far_from_unit_rms = true; + } + } + CHECK(any_group_far_from_unit_rms); + } +} + +TEST_CASE("qwen4_exp PLE conv_mask masks BOTH the skip term and the conv input") { + // modeling_qwen4_exp.py:1185-1187 masks `gated_value` AND + // `gated_value_normed`; :204-213 is the multiply it does it with. Masking one + // of the two is a real port defect and it is invisible to every other case in + // this file, because they all pass `conv_mask = nullptr`. + // + // The mask is a PAIRED obligation with the caller (see the header): a masked + // position must already carry EOS in `input_ids`, because the hash reads + // token ids and not activations. `kPleMaskTokens` honours that, so this pins + // the contract rather than a state nobody would produce. Zeros sit at 3, 4 + // and 11; 3 and 4 are INTERIOR, so the dilation carries them into t = 6, 9 + // and 12 as well, and the 9-column state carries them across a chunk break. + const PleGeometry geom = TinyGeometry(); + const NGramTableLayout layout = BuildNGramTableLayout(geom, /*ple_layer_index=*/0); + const PleWeights weights = TinyWeights(); + const int64_t width = geom.stream_width(); + + SUBCASE("single shot") { + PleSequenceState state; + state.Reset(geom); + std::vector got(static_cast(kNgramTotalLen * width)); + PleForward(geom, layout, weights, kPleHiddenStates, kPleMaskTokens, kNgramTotalLen, + kPleConvMask, &state, got.data()); + const double worst = vllm_test::MaxAbsDiff(got, kPleMaskedExpectedOutput, + static_cast(kNgramTotalLen * width)); + MESSAGE("masked PLE forward max|diff| vs transformers v5.16.0 = " << worst); + CHECK(worst < 1e-5); + } + + SUBCASE("prefill then two decode steps: the mask reaches the 9-column state") { + PleSequenceState state; + state.Reset(geom); + std::vector got(static_cast(kNgramTotalLen * width)); + PleForward(geom, layout, weights, kPleHiddenStates, kPleMaskTokens, kNgramPrefillLen, + kPleConvMask, &state, got.data()); + for (int64_t t = kNgramPrefillLen; t < kNgramTotalLen; ++t) { + PleForward(geom, layout, weights, kPleHiddenStates + t * width, kPleMaskTokens + t, + 1, kPleConvMask + t, &state, got.data() + t * width); + } + const double worst = vllm_test::MaxAbsDiff(got, kPleMaskedExpectedOutput, + static_cast(kNgramTotalLen * width)); + MESSAGE("masked PLE forward max|diff| vs transformers v5.16.0 = " << worst); + CHECK(worst < 1e-5); + } + + SUBCASE("the mask is load-bearing: a nullptr mask gives a different answer") { + // Without this, a golden that happened to equal the unmasked one would gate + // nothing at all. The masked and unmasked answers must be far apart. + PleSequenceState masked_state; + masked_state.Reset(geom); + std::vector masked(static_cast(kNgramTotalLen * width)); + PleForward(geom, layout, weights, kPleHiddenStates, kPleMaskTokens, kNgramTotalLen, + kPleConvMask, &masked_state, masked.data()); + + PleSequenceState plain_state; + plain_state.Reset(geom); + std::vector plain(static_cast(kNgramTotalLen * width)); + PleForward(geom, layout, weights, kPleHiddenStates, kPleMaskTokens, kNgramTotalLen, + /*conv_mask=*/nullptr, &plain_state, plain.data()); + const double apart = vllm_test::MaxAbsDiff(masked, plain.data(), + static_cast(kNgramTotalLen * width)); + MESSAGE("masked vs unmasked max|diff| = " << apart); + CHECK(apart > 1e-2); + + // A masked row keeps NO skip term, so what survives there is the conv + // output alone. Zero the conv weights and the masked rows must be exactly + // zero, which separates "the skip term was masked" from "something was". + std::vector zero_conv( + static_cast(width * geom.ple_conv_kernel_size), 0.0F); + PleWeights w = weights; + w.conv1d = zero_conv.data(); + PleSequenceState skip_state; + skip_state.Reset(geom); + std::vector skip_only(static_cast(kNgramTotalLen * width)); + PleForward(geom, layout, w, kPleHiddenStates, kPleMaskTokens, kNgramTotalLen, + kPleConvMask, &skip_state, skip_only.data()); + int64_t masked_rows = 0; + for (int64_t t = 0; t < kNgramTotalLen; ++t) { + if (kPleConvMask[t] != 0) continue; + ++masked_rows; + for (int64_t c = 0; c < width; ++c) { + CHECK(skip_only[static_cast(t * width + c)] == 0.0F); + } + } + CHECK(masked_rows == 3); + } +}