From a4c144fb6154ecc263a6509cb7c2ca25ee5a17dc Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 15:03:11 +0000 Subject: [PATCH 1/4] =?UTF-8?q?feat(MODEL-MM-QWEN4-EXP):=20W1=20=E2=80=94?= =?UTF-8?q?=20resolve=20and=20validate=20the=20qwen4=5Fexp=20config,=20reg?= =?UTF-8?q?ister=20the=20arch,=20and=20refuse=20by=20name?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First product code on the row whose spec landed in #1980. The architecture now RESOLVES, and its config parses and validates; it does not load and it does not forward, and both refuse by name rather than dying a layer down. The resolve IS the validation. Every refusal mirrors one in upstream `Qwen4ExpTextConfig.validate_architecture` / `__post_init__` at the accepted lane pin, transformers 5.16.0: unsupported layer types, `hc_count <= 1`, the MoE bounds, the all-or-nothing QSA field group, `indexer_kv_heads != 1`, a budget that does not divide by the compress ratio, a rotary dim wider than the indexer head, and a PLE id outside the one-indexed range or landing on a sparse-attention layer. Two things the config layer gets right that a reader taking the checkpoint at face value would not. The published `layer_types` says `full_attention` for the twelve layers that actually run the QSA indexer, and upstream rewrites them in `__post_init__`. `Qwen4ExpLayerKind` therefore has no `kFullAttention` enumerator at all, so the wrong state is unrepresentable rather than merely unused. The test asserts something stronger than the rewrite: the rewritten published list and the list synthesized from `full_attention_interval` are equal, so if either path is wrong the other says so. On the real checkpoint both give sparse at 3, 7, ..., 47. `partial_rotary_factor` is read here with upstream's inherited default of 0.25 rather than taken from `config.rotary_dim`, because `IsQwen35Family` in the shared reader does not list `qwen4_exp`. An absent key defaults there to 1.0 where upstream, subclassing `Qwen3_5MoeTextConfig`, inherits 0.25 — and since upstream's own guard is `rotary_dim > indexer_head_dim`, the shared reader's value would make us REFUSE a config upstream ACCEPTS. On the published checkpoint the key is present and both agree at 64; the divergence is latent, and the regression guard is a named case. `ple_layer_ids` is stored already converted to 0-based. It is one-indexed in the checkpoint and upstream says so in terms, its validator resolves `layer_types[id - 1]`, and every PLE tensor in the released weights sits under `...layers.1.ple.`. Carrying a one-indexed value through the port is how that gets rediscovered as a bug. Two gates caught real defects in the first draft, and both fixes are recorded at the site. `[[noreturn]]` on a non-void return type is MSVC C4646, promoted to C2220 under /W4 /WX. And the forward refusal has to be `VT_CHECK(false, ...)` in the hook body: `check-runner-routing-consistency.py` recognises a refuse-by-name stub by that token and classifies the hook body itself, so a bare throw put this model in the silently exempt NONE bucket — the exact hole that checker exists to close. dots3-note's delegate hop does not help here, because it resolves `Class::ForwardDevice` across translation units or through a file-local helper, and a class in this TU's own anonymous namespace is neither. No `Qwen4ExpModel::ForwardDevice` is invented to refuse from, because there is no device forward yet and asserting a routing shape this row has not earned would be a claim, not a stub. The spec also gains two corrections the llama.cpp studies settled. Its ragged-K question is RESOLVED rather than owed: `tensor_type_fallback` maps `Q4_K -> Q5_0` and `IQ4_XS -> IQ4_NL`, so the answer depends on the recipe, and our GGUF reader supports neither type. And "no GGUF exists and no tool can produce one" is superseded — `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S is 67.56 GiB of weights that FIT GB10 with ~52 GiB of headroom, and its metadata confirms this spec's n-gram derivation to the digit, including `ple.layer_multipliers = [23703573157769, 20109073645365, 8052911324071]`. Reachability: every case drives `LoadHfConfig -> ModelRegistry::Resolve -> factory->parse_config` and the refusals through the factory's own hooks. A case that built `Qwen4ExpParams` by hand would prove the struct parses and not that anything reaches it. Tracked by #1981, under #1978. Gates: `test_qwen4_exp_scaffold` 7 cases / 151 assertions green, and load-bearing under mutation — deleting the `full_attention` rewrite reds 6 of 7 cases, and defaulting `partial_rotary_factor` to the shared reader's 1.0 reds the guard case (via a throw, so the assertion line still read 148/148, which is the documented doctest shape). Tree restored byte-for-byte after each. `test_model_registry` 24/958, `test_model_loader_gguf` 6/18, `check-supported-models` ok at 42 architectures, `check-runner-routing-consistency` classifies qwen4_exp REFUSE, `check-windows-portability` ok, `agent-preflight.sh --staged` ok. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/qwen4-exp-flash-next.md | 30 +- CMakeLists.txt | 2 + docs/FEATURES.md | 1 + src/vllm/model_executor/models/qwen4_exp.cpp | 302 +++++++++++++++ src/vllm/model_executor/models/qwen4_exp.h | 137 +++++++ .../models/qwen4_exp_registry.cpp | 175 +++++++++ tests/CMakeLists.txt | 10 + .../models/fixtures/qwen4_exp/config.json | 154 ++++++++ tests/vllm/models/test_model_registry.cpp | 13 +- tests/vllm/models/test_qwen4_exp_scaffold.cpp | 348 ++++++++++++++++++ tests/vllm/test_model_loader_gguf.cpp | 3 +- 11 files changed, 1165 insertions(+), 10 deletions(-) create mode 100644 src/vllm/model_executor/models/qwen4_exp.cpp create mode 100644 src/vllm/model_executor/models/qwen4_exp.h create mode 100644 src/vllm/model_executor/models/qwen4_exp_registry.cpp create mode 100644 tests/vllm/models/fixtures/qwen4_exp/config.json create mode 100644 tests/vllm/models/test_qwen4_exp_scaffold.cpp diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index 02aadc606..636425c66 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -563,8 +563,14 @@ it is **CPU-only**, because `EmbeddingKernelCuda` refuses anything but f32/bf16. **Second blocker, cheap to avoid because we author the converter.** `moe_intermediate_size = 640` makes `ffn_down_exps` Q4_K-illegal on its reduction dim (`640 % 256 = 128`), and `hc_lowrank = 320` is the same class. llama.cpp's substitution -for a ragged-K Q4_K tensor is believed to be Q5_0 -- **flagged as UNVERIFIED, and owed -a check against the pinned llama.cpp oracle before it becomes an assertion.** The +for a ragged-K Q4_K tensor is **Q5_0, now VERIFIED** and no longer owed: the +`tensor_type_fallback` table in `src/llama-quant.cpp` maps `Q4_K -> Q5_0`, +`Q5_K -> Q5_1`, `Q6_K -> Q8_0`, `Q2_K/Q3_K/TQ* -> Q4_0` and every `IQ*` including +`IQ4_XS -> IQ4_NL`, then falls to `F16` if the result still does not divide. So the +answer depends on the RECIPE, which is why the shipped `unsloth` UD-IQ1_S file shows +`IQ4_NL` on `ffn_down_exps` rather than Q5_0 -- it asked for an IQ type, not Q4_K. A +`-Q4_K_M` build of this model would land on Q5_0, and on Q8_0 wherever `use_more_bits` +promotes `ffn_down` to Q6_K. The dependent fact IS verified in-tree and is the one that bites: this repository's GGUF reader knows ggml type ids `0,1,2,8,10..14,16,18,19,22..28,30,39,40,41,66` and **has no entry for 3 (Q4_1), 6 (Q5_0), 7 (Q5_1) or 20 (IQ4_NL)**, so such a file fails at header @@ -741,8 +747,24 @@ change that makes any arm reachable, not later. - The non-resident n-gram table on CUDA: the dequantizing gather op and the `kEmbeddingTable` keep-quant policy change (Route B), and a measurement of the page-cache cost that the <= 64 KiB/token arithmetic only bounds. -- **UNVERIFIED and owed a check against the pinned llama.cpp oracle:** llama.cpp's exact - substitution for a ragged-K Q4_K tensor, asserted here as Q5_0. +- ~~llama.cpp's ragged-K substitution~~ **RESOLVED**: `Q4_K -> Q5_0`, `IQ4_XS -> IQ4_NL`, + read from `tensor_type_fallback` in `src/llama-quant.cpp`. Both are reachable for this + model depending on the recipe, and our reader supports NEITHER (no `case 6`, no + `case 20`), so W6 owes both. +- **A published GGUF now EXISTS**, which supersedes this spec's "no GGUF exists and no + tool can produce one": `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S, 67.56 GiB of + weights in 3 shards, `general.architecture = qwen4exp`, 1224 tensors. It FITS GB10 + with ~52 GiB of headroom, and two things in OUR tree stop us loading it: the missing + IQ4_NL reader arm, and the gather-table expansion. Its metadata independently + confirms this spec's n-gram derivation to the digit -- + `ple.layer_multipliers = [23703573157769, 20109073645365, 8052911324071]` and + `ple.head_vocab_sizes = [20000003, 20000023, ...]`. +- **Mirror the `qwen4exp` GGUF key and tensor names rather than inventing ours.** Two + competing llama.cpp PRs (#27742 open, #27739 closed-by-courtesy) already disagree on + `ple.*` key spellings and on whether the n-gram table is model-level + (`per_layer_token_embd`) or per-layer (`blk.N.ple_ngram_embd`), and a maintainer has + asked for a rename, so the names are NOT settled. Re-check before W6a commits to a + layout; a wrong guess makes every published GGUF unreadable by us. - A K-divisibility assertion in whatever writes our GGUF files. - A speed denominator, once one exists. diff --git a/CMakeLists.txt b/CMakeLists.txt index b70ed04b8..fdf5eaba9 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -799,6 +799,8 @@ add_library(vllm STATIC src/vllm/model_executor/models/dots3_note_attn.cpp src/vllm/model_executor/models/dots3_note_device.cpp src/vllm/model_executor/models/dots3_note_registry.cpp + src/vllm/model_executor/models/qwen4_exp.cpp + src/vllm/model_executor/models/qwen4_exp_registry.cpp src/vllm/model_executor/models/laguna_registry.cpp src/vllm/model_executor/models/laguna_weights.cpp src/vllm/model_executor/models/interfaces.cpp diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 4e0e26aa4..01372fd32 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -123,6 +123,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. |---|---|---|---| | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B NVFP4 (`unsloth` @`890bdef7`, `nvidia` @`0893e160`); Qwen3.5-4B BF16; **Qwen3.8-27B BF16** @`1d4bf0f2` | 27B strict 235/235 text + 32/32 image/video; 4B cached 3/3; Qwen3.8-27B 4/7 strict, 3 exact fp32 ties in band (#915) | `unsloth` 27B at/above vLLM, ModelOpt 0.85x; 4B 1.021x; 3.8-27B c4 **0.963x**, c1/c8 absolutes (#915). Loads BF16/per-tensor FP8/NVFP4 (CT+ModelOpt); `modelopt_mixed` FP8 tower NATIVE (#164), GDN qkvz merged. CUDA/CPU | | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (NVFP4 text; published BF16 text + vision tower) | NVFP4 strict 315/315 vs vLLM 0.25.0; published BF16 6/7 prompts strict 16/16 vs the pin, 7th an exact tie (#910). Image/video IMPLEMENTED, NOT GATED (#891): the tower loads and runs, mm gate OWED | gate model: 0.93x to 1.03x grid; NO BF16 or mm speed claim | +| `Qwen4ExpForConditionalGeneration` | none — **REGISTERED, NOT LOADABLE** (W1, [#1981](https://github.com/mudler/vllm.cpp/issues/1981)) | **NO GATE, and none is reachable yet.** The config resolves and validates; the loader, the forward and the KV-cache spec all REFUSE BY NAME, each naming the wave that owes it. vLLM implements `qwen4_exp` at NO revision, so the algorithm oracle is transformers **5.16.0** under an accepted lane exception; `gateable = no` because nothing published fits a fleet device — `Qwen/Qwen3.8-Flash-Next` is ~360 GB bf16, ~180 GB FP8, ~128 GB NVFP4 against ~119.6 GiB usable on GB10 | none, and no speed claim is admissible from this row until a token gate exists | | `Qwen3_5ForCausalLM`, `Qwen3_5MoeForCausalLM` | none: no text-only Qwen3.5 checkpoint fits this hardware | **NO RUN GATE, OWED.** Gated on `test_qwen3_8_text_only.cpp`; NO token claim. Loader reads stacked BF16 experts (#740) plus BF16 towers, shared expert and `lm_head` (#864), so both published indices satisfy the load plan | not measured | | `Qwen3ForCausalLM` | Qwen3 dense 0.6B/1.7B/4B/32B, NVFP4A16 | near-tie strict 16/16 vs vLLM 0.25.0 | c1 every-axis parity, c8 decode residual | | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B | strict 6/6 vs vLLM 0.25.0 | 11/16 grid cells at or above graphed vLLM | diff --git a/src/vllm/model_executor/models/qwen4_exp.cpp b/src/vllm/model_executor/models/qwen4_exp.cpp new file mode 100644 index 000000000..c87d159d9 --- /dev/null +++ b/src/vllm/model_executor/models/qwen4_exp.cpp @@ -0,0 +1,302 @@ +// Qwen4-Exp config resolution + validation (W1 of MODEL-MM-QWEN4-EXP, #1981). +// +// The resolve IS the validation: every refusal below mirrors one in upstream +// `Qwen4ExpTextConfig.validate_architecture` / `__post_init__` at the accepted +// lane pin, transformers **5.16.0**. vLLM implements `qwen4_exp` at NO +// revision, so there is nothing to mirror on this surface and transformers is +// the only source; see `.agents/specs/qwen4-exp-flash-next.md` `## Oracles`. +#include "vllm/model_executor/models/qwen4_exp.h" + +#include +#include +#include +#include + +namespace vllm { +namespace { + +constexpr const char* kSlug = "qwen4_exp"; + +[[noreturn]] void Refuse(const std::string& what) { + throw std::runtime_error(std::string(kSlug) + ": " + what + + " See .agents/specs/qwen4-exp-flash-next.md and " + "issue #1981."); +} + +// The model's own sub-object. Upstream nests everything except the wrapper's +// `architectures`/`model_type`/vision block under `text_config`; a flat config +// (what a GGUF-derived or hand-written config looks like) is accepted by +// falling back to the top level, exactly as the shared HfConfig reader does. +const nlohmann::json& TextOf(const nlohmann::json& raw) { + auto it = raw.find("text_config"); + if (it != raw.end() && it->is_object()) return *it; + return raw; +} + +int64_t OptInt(const nlohmann::json& j, const char* key, int64_t fallback) { + auto it = j.find(key); + if (it == j.end() || it->is_null()) return fallback; + if (!it->is_number_integer() && !it->is_number_unsigned()) { + Refuse(std::string("`") + key + "` must be an integer."); + } + return it->get(); +} + +double OptDouble(const nlohmann::json& j, const char* key, double fallback) { + auto it = j.find(key); + if (it == j.end() || it->is_null()) return fallback; + if (!it->is_number()) Refuse(std::string("`") + key + "` must be a number."); + return it->get(); +} + +// Upstream treats an ABSENT QSA field and a null one alike, and the group is +// all-or-nothing, so presence has to be observable separately from value. +bool HasKey(const nlohmann::json& j, const char* key) { + auto it = j.find(key); + return it != j.end() && !it->is_null(); +} + +std::vector OptIntArray(const nlohmann::json& j, const char* key) { + std::vector out; + auto it = j.find(key); + if (it == j.end() || it->is_null()) return out; + if (!it->is_array()) Refuse(std::string("`") + key + "` must be an array."); + for (const auto& e : *it) { + if (!e.is_number_integer() && !e.is_number_unsigned()) { + Refuse(std::string("`") + key + "` must contain only integers."); + } + out.push_back(e.get()); + } + return out; +} + +// `full_attention` -> `qwen_sparse_attention`. Upstream rewrites the published +// list in `__post_init__` with the comment that the checkpoint "contains +// full_attention entries for layers that are actually using an indexer". +Qwen4ExpLayerKind KindFromString(const std::string& s) { + if (s == "linear_attention") return Qwen4ExpLayerKind::kLinearAttention; + if (s == "qwen_sparse_attention" || s == "full_attention") { + return Qwen4ExpLayerKind::kQwenSparseAttention; + } + Refuse("unsupported layer type '" + s + + "'; expected `linear_attention` or `qwen_sparse_attention` (the " + "checkpoint's `full_attention` is rewritten to the latter)."); +} + +} // namespace + +Qwen4ExpParams ParseQwen4ExpParams(const HfConfig& config) { + const nlohmann::json& text = TextOf(config.raw); + Qwen4ExpParams p; + + p.hidden_size = config.hidden_size; + p.num_hidden_layers = config.num_hidden_layers; + p.vocab_size = config.vocab_size; + p.rms_norm_eps = config.rms_norm_eps; + p.num_attention_heads = config.num_attention_heads; + p.num_key_value_heads = config.num_key_value_heads; + p.head_dim = config.head_dim; + + if (p.num_hidden_layers <= 0) { + Refuse("`num_hidden_layers` must be > 0, got " + + std::to_string(p.num_hidden_layers) + "."); + } + + // --- layer_types: rewrite a published list, or synthesize from the interval. + // Both paths must agree on the real checkpoint, and the test asserts that + // they do: the published 48-entry list rewritten is byte-identical to the + // list synthesized from `full_attention_interval` = 4. + if (!config.layer_types.empty()) { + if (static_cast(config.layer_types.size()) != p.num_hidden_layers) { + Refuse("`layer_types` has " + std::to_string(config.layer_types.size()) + + " entries but `num_hidden_layers` is " + + std::to_string(p.num_hidden_layers) + "."); + } + p.layer_types.reserve(config.layer_types.size()); + for (const std::string& s : config.layer_types) { + p.layer_types.push_back(KindFromString(s)); + } + } else { + const int64_t interval = OptInt(text, "full_attention_interval", 4); + if (interval <= 0) { + Refuse("`full_attention_interval` must be > 0, got " + + std::to_string(interval) + "."); + } + p.layer_types.reserve(static_cast(p.num_hidden_layers)); + for (int64_t i = 0; i < p.num_hidden_layers; ++i) { + p.layer_types.push_back((i + 1) % interval != 0 + ? Qwen4ExpLayerKind::kLinearAttention + : Qwen4ExpLayerKind::kQwenSparseAttention); + } + } + + // --- gated residual --- + p.hc_count = OptInt(text, "hc_count", 4); + p.hc_lowrank = OptInt(text, "hc_lowrank", 320); + if (p.hc_count <= 1) { + Refuse("requires `hc_count` > 1, got " + std::to_string(p.hc_count) + "."); + } + if (p.hc_lowrank <= 0) { + Refuse("`hc_lowrank` must be > 0, got " + std::to_string(p.hc_lowrank) + + "."); + } + + // --- MoE --- + p.num_experts = config.num_experts; + p.num_experts_per_tok = config.num_experts_per_tok; + p.moe_intermediate_size = config.moe_intermediate_size; + p.shared_expert_intermediate_size = config.shared_expert_intermediate_size; + if (p.num_experts <= 0) { + Refuse("`num_experts` must be > 0, got " + std::to_string(p.num_experts) + + "."); + } + if (p.num_experts_per_tok <= 0 || p.num_experts_per_tok > p.num_experts) { + Refuse("`num_experts_per_tok` must be in [1, num_experts], got " + + std::to_string(p.num_experts_per_tok) + " and " + + std::to_string(p.num_experts) + "."); + } + if (p.moe_intermediate_size <= 0 || p.shared_expert_intermediate_size <= 0) { + Refuse("`moe_intermediate_size` and `shared_expert_intermediate_size` must " + "be > 0, got " + std::to_string(p.moe_intermediate_size) + " and " + + std::to_string(p.shared_expert_intermediate_size) + "."); + } + + // --- output gate. The shared reader already canonicalizes `swish` to `silu` + // and refuses anything outside {silu, swish, sigmoid}; upstream Qwen4-Exp + // accepts {sigmoid, silu}, so the two sets agree AFTER canonicalization. + if (config.output_gate_type != "silu" && config.output_gate_type != "sigmoid") { + Refuse("unsupported output gate activation '" + config.output_gate_type + + "'; expected `sigmoid` or `silu`."); + } + + // --- rotary. `partial_rotary_factor` is read HERE with upstream's inherited + // default of 0.25 rather than taken from `config.rotary_dim`, and that is + // deliberate. `IsQwen35Family` in the shared reader does not list + // `qwen4_exp`, so an ABSENT key defaults there to 1.0 (full rotary) where + // upstream `Qwen4ExpTextConfig`, subclassing `Qwen3_5MoeTextConfig`, inherits + // 0.25. On the published checkpoint the key is present and both agree at 64; + // on a config that omits it they would disagree 256 vs 64, and because + // upstream's own guard is `rotary_dim > indexer_head_dim`, the shared + // reader's value would make us REFUSE a config upstream ACCEPTS. Mirroring + // the inheritance is what keeps the refusal set identical. + p.partial_rotary_factor = OptDouble(text, "partial_rotary_factor", 0.25); + if (!(p.partial_rotary_factor > 0.0)) { + Refuse("`partial_rotary_factor` must be > 0, got " + + std::to_string(p.partial_rotary_factor) + "."); + } + p.rotary_dim = + static_cast(static_cast(p.head_dim) * + p.partial_rotary_factor); + + // --- QSA: all-or-nothing, then per-field. + static constexpr const char* kQsaFields[] = { + "indexer_n_heads", "indexer_kv_heads", "indexer_head_dim", + "indexer_budget", "indexer_compress_ratio"}; + int present = 0; + std::string missing; + for (const char* f : kQsaFields) { + if (HasKey(text, f)) { + ++present; + } else { + if (!missing.empty()) missing += ", "; + missing += f; + } + } + if (present > 0) { + if (present != 5) { + Refuse("QSA config is missing required fields: " + missing + "."); + } + p.qsa.n_heads = OptInt(text, "indexer_n_heads", 0); + p.qsa.kv_heads = OptInt(text, "indexer_kv_heads", 0); + p.qsa.head_dim = OptInt(text, "indexer_head_dim", 0); + p.qsa.budget = OptInt(text, "indexer_budget", 0); + p.qsa.compress_ratio = OptInt(text, "indexer_compress_ratio", 0); + if (p.qsa.n_heads <= 0 || p.qsa.kv_heads <= 0 || p.qsa.head_dim <= 0 || + p.qsa.budget <= 0 || p.qsa.compress_ratio <= 0) { + Refuse("QSA config values must be positive."); + } + if (p.qsa.kv_heads != 1) { + Refuse("QSA requires `indexer_kv_heads` = 1, got " + + std::to_string(p.qsa.kv_heads) + "."); + } + if (p.qsa.budget % p.qsa.compress_ratio != 0) { + Refuse("`indexer_budget` (" + std::to_string(p.qsa.budget) + + ") must be divisible by `indexer_compress_ratio` (" + + std::to_string(p.qsa.compress_ratio) + ")."); + } + if (p.rotary_dim > p.qsa.head_dim) { + Refuse("rotary dim " + std::to_string(p.rotary_dim) + + " exceeds `indexer_head_dim` " + std::to_string(p.qsa.head_dim) + + "."); + } + } + + // --- PLE. One-indexed on the way in, 0-based on the way out. + const std::vector raw_ple = OptIntArray(text, "ple_layer_ids"); + std::set sorted_unique(raw_ple.begin(), raw_ple.end()); + for (int64_t one_based : sorted_unique) { + if (one_based < 1 || one_based > p.num_hidden_layers) { + Refuse("`ple_layer_ids` must contain one-indexed ids in [1, " + + std::to_string(p.num_hidden_layers) + "], got " + + std::to_string(one_based) + "."); + } + const int64_t zero_based = one_based - 1; + if (p.layer_types[static_cast(zero_based)] != + Qwen4ExpLayerKind::kLinearAttention) { + Refuse("PLE is only supported on `linear_attention` layers; " + "`ple_layer_ids` names one-indexed layer " + + std::to_string(one_based) + " (0-based " + + std::to_string(zero_based) + "), which is a sparse-attention " + "layer."); + } + p.ple.layer_ids_zero_based.push_back(zero_based); + } + if (!p.ple.layer_ids_zero_based.empty()) { + p.ple.embed_dim = OptInt(text, "ple_embed_dim", p.hidden_size); + p.ple.conv_kernel_size = OptInt(text, "ple_conv_kernel_size", 4); + p.ple.ngram_size = OptInt(text, "ngram_size", 0); + p.ple.heads_per_ngram = OptInt(text, "heads_per_ngram", 0); + p.ple.ngram_vocab_size_base = OptInt(text, "ngram_vocab_size_base", 0); + p.ple.make_ngram_vocab_size_divisible_by = + OptInt(text, "make_ngram_vocab_size_divisible_by", 0); + p.ple.split_ngram_parts = OptInt(text, "split_ngram_parts", 512); + p.ple.seed = OptInt(text, "seed", 1234); + if (p.ple.ngram_size < 2) { + Refuse("`ngram_size` must be >= 2 when PLE is enabled, got " + + std::to_string(p.ple.ngram_size) + "."); + } + if (p.ple.heads_per_ngram <= 0) { + Refuse("`heads_per_ngram` must be > 0 when PLE is enabled, got " + + std::to_string(p.ple.heads_per_ngram) + "."); + } + if (p.ple.conv_kernel_size <= 0) { + Refuse("`ple_conv_kernel_size` must be > 0, got " + + std::to_string(p.ple.conv_kernel_size) + "."); + } + // 2560 / 16 = 160. A ragged split would silently mis-slice every gathered + // row, so it is refused rather than truncated. + const int64_t heads = p.ple.ngram_heads(); + if (heads <= 0 || p.ple.embed_dim % heads != 0) { + Refuse("`ple_embed_dim` (" + std::to_string(p.ple.embed_dim) + + ") must be divisible by the n-gram head count (" + + std::to_string(heads) + ")."); + } + } + + // --- MTP. `mtp_num_hidden_layers` is a sibling of the `mtp` sub-object; the + // head is a block inside THIS text config, not a separately registered + // architecture, which is why this row adds ONE model-matrix row and not two. + p.mtp_num_hidden_layers = OptInt(text, "mtp_num_hidden_layers", 0); + if (p.mtp_num_hidden_layers < 0) { + Refuse("`mtp_num_hidden_layers` must be >= 0, got " + + std::to_string(p.mtp_num_hidden_layers) + "."); + } + + return p; +} + +void ParseQwen4ExpConfig(const HfConfig& config) { + (void)ParseQwen4ExpParams(config); +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/qwen4_exp.h b/src/vllm/model_executor/models/qwen4_exp.h new file mode 100644 index 000000000..8c4f57947 --- /dev/null +++ b/src/vllm/model_executor/models/qwen4_exp.h @@ -0,0 +1,137 @@ +// Qwen4-Exp (`Qwen/Qwen3.8-Flash-Next`) — W1 config surface. +// +// Model-private header, deliberately not under include/: nothing outside this +// model needs these types yet, and `include/vllm.h` is the ABI seam a shipped +// capability is exposed through. W1 ships no capability. +// +// ORACLE. vLLM does NOT implement `qwen4_exp` at ANY revision (read live +// 2026-08-26 at `origin/main` = `6a5e8f5979`: no `qwen4*` path, no registry +// entry). Under AGENTS.md "When vLLM has no implementation" this row runs a +// SPLIT oracle, recorded in `.agents/specs/qwen4-exp-flash-next.md`: +// transformers **5.16.0** (the accepted lane pin) defines the ALGORITHM, and +// vLLM supplies the OPS for every primitive it does implement. This file is +// entirely the first half. Every anchor below is +// `transformers/models/qwen4_exp/{configuration,modular}_qwen4_exp.py` at +// v5.16.0. +#ifndef VLLM_MODEL_EXECUTOR_MODELS_QWEN4_EXP_H_ +#define VLLM_MODEL_EXECUTOR_MODELS_QWEN4_EXP_H_ + +#include +#include +#include + +#include "vllm/transformers_utils/hf_config.h" + +namespace vllm { + +// Per-layer kind AFTER upstream's rewrite. `__post_init__` replaces every +// `full_attention` entry with `qwen_sparse_attention`, because the published +// checkpoint says `full_attention` for layers that actually run the QSA +// indexer. A reader that takes the checkpoint at face value wires dense +// attention on 12 of 48 layers and is wrong WITHOUT SAYING SO, which is why +// this enum has no `kFullAttention` enumerator at all: the state is +// unrepresentable rather than merely unused. +enum class Qwen4ExpLayerKind { + kLinearAttention, // Gated DeltaNet + kQwenSparseAttention // QSA +}; + +// Qwen Sparse Attention. Upstream validates these five as a GROUP: either all +// are present or none is, and a partial set raises naming the missing fields. +struct Qwen4ExpQsaParams { + int64_t n_heads = 0; // indexer_n_heads = 4 + int64_t kv_heads = 0; // indexer_kv_heads — upstream REQUIRES exactly 1 + int64_t head_dim = 0; // indexer_head_dim = 128 + int64_t budget = 0; // indexer_budget = 2048 tokens + int64_t compress_ratio = 0; // indexer_compress_ratio = 4 + + // budget / compress_ratio = 512. Derived, never read from the config. + int64_t block_topk() const { return budget / compress_ratio; } +}; + +// Per-Layer Embedding: the hashed n-gram table plus its dilated depthwise conv. +struct Qwen4ExpPleParams { + // ONE-INDEXED in the checkpoint, and upstream says so in terms + // ("One-indexed decoder layer ids"). The lookup is + // `ple_layer_ids.index(layer_idx + 1)`, so `[2]` selects 0-based layer 1. + // Stored here ALREADY CONVERTED to 0-based, because carrying a one-indexed + // value through the port is how the off-by-one gets rediscovered. Confirmed + // three ways: the upstream docstring, the config validator's own + // `layer_types[layer_id - 1]`, and the published checkpoint, whose PLE + // tensors all sit under `model.language_model.layers.1.ple.`. + std::vector layer_ids_zero_based; + + int64_t embed_dim = 0; // ple_embed_dim = 2560, defaults to hidden_size + int64_t conv_kernel_size = 0; // ple_conv_kernel_size = 4 + int64_t ngram_size = 0; // 3 — ALSO the conv DILATION + int64_t heads_per_ngram = 0; // 8 + int64_t ngram_vocab_size_base = 0; // 20,000,000 + int64_t make_ngram_vocab_size_divisible_by = 0; // 128 + int64_t split_ngram_parts = 0; // 128 — checkpoint SHARDING only, unused in the forward + int64_t seed = 1234; // absent from the published config; the dataclass default + + // (ngram_size - 1) * heads_per_ngram = 16 hash heads per token. + int64_t ngram_heads() const { return (ngram_size - 1) * heads_per_ngram; } + // embed_dim / ngram_heads = 160. + int64_t head_dim_per_ngram() const { return embed_dim / ngram_heads(); } + // (conv_kernel_size - 1) * ngram_size = 9. NOT `kernel - 1`: the conv is + // DILATED, so its state is three times deeper than an undilated one and the + // taps sit at lags {9, 6, 3, 0}. + int64_t short_conv_state_len() const { + return (conv_kernel_size - 1) * ngram_size; + } +}; + +struct Qwen4ExpParams { + // --- geometry --- + int64_t hidden_size = 0; // 2560 + int64_t num_hidden_layers = 0; // 48 + int64_t vocab_size = 0; // 248320 + double rms_norm_eps = 1e-6; + + std::vector layer_types; // 48 entries after the rewrite + + // --- gated residual (hyper-connections) --- + // The residual stream is hc_count * hidden_size = 10240 wide through the + // WHOLE stack. Upstream requires hc_count > 1. + int64_t hc_count = 0; // 4 + int64_t hc_lowrank = 0; // 320 + + // --- MoE --- + int64_t num_experts = 0; // 512 + int64_t num_experts_per_tok = 0; // 10 routed, plus 1 shared + int64_t moe_intermediate_size = 0; // 640 + int64_t shared_expert_intermediate_size = 0; // 640 + + // --- attention --- + int64_t num_attention_heads = 0; // 24 + int64_t num_key_value_heads = 0; // 2 + int64_t head_dim = 0; // 256 + double partial_rotary_factor = 0.25; + int64_t rotary_dim = 0; // int(head_dim * partial_rotary_factor) = 64 + + Qwen4ExpQsaParams qsa; + Qwen4ExpPleParams ple; + + // --- MTP --- + int64_t mtp_num_hidden_layers = 0; // 1 + + // 3 when any PLE layer exists (GDN conv, PLE conv, n-gram token history), + // else 1. Mirrors upstream's `number_of_conv_states`. + int64_t number_of_conv_states() const { + return ple.layer_ids_zero_based.empty() ? 1 : 3; + } +}; + +// Resolves and VALIDATES. The resolve IS the validation: it throws by name on +// everything unrepresentable, mirroring upstream `validate_architecture`. +Qwen4ExpParams ParseQwen4ExpParams(const HfConfig& config); + +// ModelFactory::parse_config hook. Delegates to ParseQwen4ExpParams and +// discards the result, so a malformed config is refused at load rather than at +// first forward. +void ParseQwen4ExpConfig(const HfConfig& config); + +} // namespace vllm + +#endif // VLLM_MODEL_EXECUTOR_MODELS_QWEN4_EXP_H_ diff --git a/src/vllm/model_executor/models/qwen4_exp_registry.cpp b/src/vllm/model_executor/models/qwen4_exp_registry.cpp new file mode 100644 index 000000000..55b9adb0b --- /dev/null +++ b/src/vllm/model_executor/models/qwen4_exp_registry.cpp @@ -0,0 +1,175 @@ +// Qwen4-Exp registry TU — the ADDITIVE self-registration seam (W1 of +// MODEL-MM-QWEN4-EXP, #1981). Follows the dots3_note_registry.cpp / +// gemma4_registry.cpp seam exactly: a NEW translation unit with ONE +// REGISTER_VLLM_MODEL line and ZERO edit to any shared array. +// +// UPSTREAM. `Qwen4ExpForConditionalGeneration` is registered by NO vLLM +// revision. Read live 2026-08-26 at vLLM `origin/main` = `6a5e8f5979`: no +// `qwen4*` path, no `registry.py` entry, and a repository-wide search for +// `qwen4` returns zero results; `vllm-omni` likewise. That is absence from +// vLLM `main` rather than staleness in our parity pin `555967922`, so this TU +// deliberately carries no pinned upstream module/class anchor, the convention +// `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` follows for a beyond-pin arm. +// The ALGORITHM source is transformers **5.16.0**, the accepted lane pin; see +// `.agents/oracles/transformers.md` and `.agents/specs/qwen4-exp-flash-next.md`. +// +// The MTP head is deliberately NOT registered as a second architecture, and +// unlike dots3-note that is not a scheduling choice: upstream carries it as an +// `mtp` block INSIDE the same text config rather than as a separate registry +// entry, so there is no second architecture string to register. That is why +// this row moves the MODEL row ratchet by ONE and not by two. +// +// SCOPE HONESTY: registering this arch makes it RESOLVE and parse and validate +// its config. It does NOT make it load and it does NOT make it forward — both +// refuse BY NAME, naming the wave that owes the work. That polarity matters +// more here than usual, because no oracle for this model runs on any hardware +// this project owns yet (`gateable = no`, blocked on memory rather than +// software), so there is no downstream token gate that would catch a forward +// returning plausible garbage. Refusing is the only safe default. +#include "vllm/model_executor/models/model_registry.h" + +#include "vt/dtype.h" // VT_CHECK + +#include +#include + +#include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits complete type +#include "vllm/model_executor/models/qwen4_exp.h" +#include "vllm/v1/kv_cache_interface.h" + +namespace vllm { +namespace { + +// Text generation, multimodal (image AND video: the published config carries +// `image_token_id`, `video_token_id` and a `vision_config`), and HYBRID — +// 36 of 48 layers are Gated DeltaNet carrying recurrent state, so this belongs +// with the hybrids and not with the pure-attention arms. +inline constexpr ModelInfo kQwen4ExpInfo{ + .is_text_generation_model = true, + .is_pooling_model = false, + .is_hybrid = true, + // FALSE by the house convention the blanket assertion in + // test_model_registry.cpp enforces: our ModelInfo is a consumed subset + // whose only reader short-circuits on is_hybrid, so the GDN-hybrid + // wrappers (kQwen3_5Info, kKimiLinearInfo) all leave this false even + // though upstream's class carries HasInnerState. + .has_inner_state = false, + .supports_multimodal = true, + .score_type = "bi-encoder", +}; + +class Qwen4ExpLoadedModel final : public LoadedModel { + public: + explicit Qwen4ExpLoadedModel(const ModelRegistration& registration) + : LoadedModel(registration) {} +}; + +std::unique_ptr LoadQwen4ExpForConditionalGeneration( + const ModelRegistration& registration, const HfConfig& config, + const ModelSource& source) { + (void)registration; + (void)config; + if (source.kind == ModelSource::Kind::kGguf) { + // The GGUF k-quant arm is OWED, not optional (AGENTS.md, + // porting-a-model.md §2), and for this row it is the arm most likely to + // fit a host we own: `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S is 67.56 + // GiB of weights against ~119.6 GiB usable on GB10, where every + // safetensors artifact (bf16 ~360 GB, FP8 ~180 GB, NVFP4 ~128 GB) does + // not. Two things block it and both are ours: our GGUF reader has no + // `case 20`, so IQ4_NL — which that file uses for `ffn_down_exps` and for + // the n-gram table — fails at header parse; and the n-gram table is a + // gather, which `KeepQuantKDim` refuses to keep quantized, expanding 51.2B + // params to 102.4 GB of bf16. W6 owes both. + throw std::runtime_error( + "Qwen4ExpForConditionalGeneration: the GGUF arm is not ported yet (W6 " + "owes the `qwen4exp` architecture reader, IQ4_NL support, and a " + "quantized-gather path for the n-gram table). See " + ".agents/specs/qwen4-exp-flash-next.md and issue #1978."); + } + throw std::runtime_error( + "Qwen4ExpForConditionalGeneration: the weight loader is not ported yet " + "(W5 owes it; the config resolves and validates, which is all W1 " + "claims). See .agents/specs/qwen4-exp-flash-next.md and issue #1978."); +} + +void PrepareQwen4ExpForConditionalGeneration(LoadedModel& model, + const HfConfig& config, + vt::Queue& queue) { + (void)model; + (void)config; + (void)queue; +} + +ForwardLogits ForwardQwen4ExpForConditionalGeneration( + LoadedModel& model, const ModelForwardInput& input) { + // `ModelAs`, never a bare `static_cast`: opening a type-erased handle by + // promise is undefined behaviour on any object that is not really this type, + // and it matters MORE on a refusing forward than on a working one, because + // the type confusion happens on the way to a throw that would have happened + // anyway and is therefore invisible without a sanitizer (#775, #730). + (void)ModelAs(model, + "Qwen4ExpForConditionalGeneration"); + (void)input; + // `VT_CHECK(false, ...)` IN THE HOOK BODY, and not a bare throw behind a + // `Class::ForwardDevice` delegate. Three constraints meet here and only this + // shape satisfies all of them. + // + // `scripts/check-runner-routing-consistency.py` recognises a refuse-by-name + // stub by exactly this token (`_REFUSE`), and it classifies the hook body + // itself. A model it cannot classify lands in the silently-exempt NONE + // bucket, which is the hole that checker exists to close — so tripping it + // would be the defect, not the gate. The delegate hop dots3-note uses does + // not help a model like this one: it resolves `Class::ForwardDevice` across + // translation units or through a file-local `ForwardLogits` helper, and a + // class defined inside this TU's own anonymous namespace is neither. + // + // And `[[noreturn]]` on a non-void return type is MSVC C4646, promoted to + // C2220 under /W4 /WX; `check-windows-portability.py` caught that on the + // first draft of this function. + // + // There is no `Qwen4ExpModel::ForwardDevice` yet because there is no device + // forward yet. Inventing one to refuse from would assert a routing shape this + // row has not earned; W5 introduces it when there is something to route. + VT_CHECK(false, + "Qwen4ExpForConditionalGeneration: the forward is not ported yet. W2 " + "owes the hashed n-gram embedding and the PLE dilated depthwise conv, " + "W3 the gated-residual hyper-connection stream, W4 Qwen Sparse " + "Attention and its indexer side cache, and W5 the assembled forward, " + "vision path and MTP. See .agents/specs/qwen4-exp-flash-next.md and " + "issue #1978."); + return ForwardLogits{}; // unreachable; VT_CHECK always throws here +} + +v1::KVCacheConfig MakeQwen4ExpKVCache(const HfConfig& config, int block_size, + int num_blocks) { + (void)config; + (void)block_size; + (void)num_blocks; + // Unreachable while the loader refuses, and refusing by name anyway rather + // than returning an empty config: this model needs THREE conv states per + // linear layer (GDN conv, PLE conv, and an int64 n-gram token history) plus + // a QSA indexer side cache holding one key vector per block of four tokens, + // and a spec that silently omits them would allocate a wrong-sized cache + // that nothing downstream checks. + throw std::runtime_error( + "Qwen4ExpForConditionalGeneration: the KV-cache spec is not ported yet " + "(W4 owes the QSA indexer side cache and W2 the third conv state for the " + "n-gram token history). See .agents/specs/qwen4-exp-flash-next.md and " + "issue #1978."); +} + +const ModelFactory kQwen4ExpFactory{ + .parse_config = &ParseQwen4ExpConfig, + .load_weights = &LoadQwen4ExpForConditionalGeneration, + .prepare = &PrepareQwen4ExpForConditionalGeneration, + .forward = &ForwardQwen4ExpForConditionalGeneration, + .make_kv_cache = &MakeQwen4ExpKVCache, + .is_dense_model = false, +}; + +} // namespace + +REGISTER_VLLM_MODEL(qwen4_exp, "Qwen4ExpForConditionalGeneration", + kQwen4ExpFactory, kQwen4ExpInfo) + +} // namespace vllm diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 85e7624ce..97e84a79e 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -962,6 +962,16 @@ target_compile_definitions(test_dots3_note_scaffold PRIVATE # (#515). Same arrangement test_nemotron_h_scaffold uses. target_include_directories(test_dots3_note_scaffold PRIVATE ${CMAKE_SOURCE_DIR}/src) +# Qwen4-Exp W1 -- config resolution, validation, registration and refuse-by-name +# (#1981, .agents/specs/qwen4-exp-flash-next.md). Fixture is the published +# `Qwen/Qwen3.8-Flash-Next` config.json verbatim. +vllm_cpp_add_test(test_qwen4_exp_scaffold vllm/models/test_qwen4_exp_scaffold.cpp) +target_compile_definitions(test_qwen4_exp_scaffold PRIVATE + QWEN4_EXP_CKPT_FIXTURE_DIR="${CMAKE_CURRENT_SOURCE_DIR}/vllm/models/fixtures/qwen4_exp") +# qwen4_exp.h is a MODEL-PRIVATE header under src/, not include/vllm/: W1 ships +# nothing on the public ABI. Same arrangement test_dots3_note_scaffold uses. +target_include_directories(test_qwen4_exp_scaffold PRIVATE ${CMAKE_SOURCE_DIR}/src) + # dots3-note W3 -- the FULL-attention layer (#699, #1846, # .agents/specs/dots3-note.md §7 W3). Checks `_forward_note_mla`'s four deltas # over plain DeepSeek MLA -- the two lora rescales, `k_rope_only_layernorm`, the diff --git a/tests/vllm/models/fixtures/qwen4_exp/config.json b/tests/vllm/models/fixtures/qwen4_exp/config.json new file mode 100644 index 000000000..491017e99 --- /dev/null +++ b/tests/vllm/models/fixtures/qwen4_exp/config.json @@ -0,0 +1,154 @@ +{ + "architectures": [ + "Qwen4ExpForConditionalGeneration" + ], + "image_token_id": 248056, + "language_model_only": false, + "model_type": "qwen4_exp", + "text_config": { + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": 248044, + "dtype": "bfloat16", + "eos_token_id": 248044, + "full_attention_interval": 4, + "hc_count": 4, + "hc_lowrank": 320, + "head_dim": 256, + "heads_per_ngram": 8, + "hidden_act": "silu", + "hidden_size": 2560, + "indexer_budget": 2048, + "indexer_compress_ratio": 4, + "indexer_head_dim": 128, + "indexer_kv_heads": 1, + "indexer_n_heads": 4, + "initializer_range": 0.02, + "layer_types": [ + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention", + "linear_attention", + "linear_attention", + "linear_attention", + "full_attention" + ], + "linear_conv_kernel_dim": 4, + "linear_key_head_dim": 128, + "linear_num_key_heads": 16, + "linear_num_value_heads": 48, + "linear_value_head_dim": 128, + "make_ngram_vocab_size_divisible_by": 128, + "mamba_ssm_dtype": "float32", + "max_position_embeddings": 262144, + "model_type": "qwen4_exp_text", + "moe_intermediate_size": 640, + "mtp": { + "hybrid": true, + "layer_types": [ + "full_attention" + ], + "mtp_use_hidden_state_from_layer": null, + "num_hidden_layers": 1, + "rope_theta": 10000000 + }, + "mtp_num_hidden_layers": 1, + "mtp_use_dedicated_embeddings": false, + "ngram_size": 3, + "ngram_vocab_size_base": 20000000, + "num_attention_heads": 24, + "num_experts": 512, + "num_experts_per_tok": 10, + "num_hidden_layers": 48, + "num_key_value_heads": 2, + "output_gate_type": "sigmoid", + "output_router_logits": false, + "pad_token_id": null, + "partial_rotary_factor": 0.25, + "ple_conv_kernel_size": 4, + "ple_embed_dim": 2560, + "ple_layer_ids": [ + 2 + ], + "rms_norm_eps": 1e-06, + "rope_parameters": { + "mrope_interleaved": true, + "mrope_section": [ + 11, + 11, + 10 + ], + "partial_rotary_factor": 0.25, + "rope_theta": 10000000, + "rope_type": "default" + }, + "router_aux_loss_coef": 0.001, + "shared_expert_intermediate_size": 640, + "split_ngram_parts": 128, + "tie_word_embeddings": false, + "use_cache": true, + "vocab_size": 248320 + }, + "tie_word_embeddings": false, + "transformers_version": "5.8.0.dev0", + "video_token_id": 248057, + "vision_config": { + "deepstack_visual_indexes": [], + "depth": 27, + "hidden_act": "gelu_pytorch_tanh", + "hidden_size": 1152, + "in_channels": 3, + "initializer_range": 0.02, + "intermediate_size": 4304, + "model_type": "qwen4_exp", + "num_heads": 16, + "num_position_embeddings": 2304, + "out_hidden_size": 2560, + "patch_size": 16, + "spatial_merge_size": 2, + "temporal_patch_size": 2 + }, + "vision_end_token_id": 248054, + "vision_start_token_id": 248053 +} \ No newline at end of file diff --git a/tests/vllm/models/test_model_registry.cpp b/tests/vllm/models/test_model_registry.cpp index 1d672a1c0..10c897f7b 100644 --- a/tests/vllm/models/test_model_registry.cpp +++ b/tests/vllm/models/test_model_registry.cpp @@ -56,7 +56,7 @@ TEST_CASE("registry_imports: every registered architecture has a complete factor // @ `c205726108df54bb6fbf15b19e725a4a3add2b18`, BEYOND our parity pin). Its // speculative head `Dots3NoteMTPModel` (registry.py:670) is INVENTORIED and // deliberately NOT registered, so it adds one entry and not two. - REQUIRE(registrations.size() == 41); + REQUIRE(registrations.size() == 42); for (const ModelRegistration& registration : registrations) { CAPTURE(registration.architecture); @@ -152,7 +152,7 @@ TEST_CASE("self_registration: every arch self-registers from its own TU") { // with the kExampleConfigArchitectures ledger; adding a model appends its two // entries here. const std::vector supported = ModelRegistry::SupportedArchs(); - REQUIRE(supported.size() == 41); + REQUIRE(supported.size() == 42); CHECK(std::is_sorted(supported.begin(), supported.end())); // The full byte-order sequence. Note "MiniCPM3" < "MiniCPMF" and "Phi3" < // "PhiF" ('3' 0x33 < 'F' 0x46); "OPT" < "Olmo" ('P' 0x50 < 'l' 0x6C); and among @@ -201,6 +201,7 @@ TEST_CASE("self_registration: every arch self-registers from its own TU") { "Qwen3_5ForConditionalGeneration", "Qwen3_5MoeForCausalLM", "Qwen3_5MoeForConditionalGeneration", + "Qwen4ExpForConditionalGeneration", "StableLmForCausalLM", }; REQUIRE(supported.size() == kSortedArchs.size()); @@ -253,6 +254,7 @@ TEST_CASE("registry_model_property: Qwen registrations match pinned _ModelInfo") CHECK(registration.info.score_type == "bi-encoder"); if (registration.architecture == "Qwen3_5ForConditionalGeneration" || registration.architecture == "Qwen3_5MoeForConditionalGeneration" || + registration.architecture == "Qwen4ExpForConditionalGeneration" || registration.architecture == "KimiK3ForConditionalGeneration") { // The outer Qwen3.5 + Kimi-K3 multimodal wrappers inherit IsHybrid but not // HasInnerState; their inner language-model classes carry HasInnerState. @@ -623,7 +625,7 @@ TEST_CASE("Qwen3.5 SSM cache dtype accepts upstream torch aliases exactly") { TEST_CASE("hf_registry_coverage: every registration has an example config fixture") { // C++ fixture registry for the currently implemented subset. Keep this list // alias-for-alias with the central ordered table, mirroring HF_EXAMPLE_MODELS. - constexpr std::array kExampleConfigArchitectures{ + constexpr std::array kExampleConfigArchitectures{ "CohereForCausalLM", "DeepseekV2ForCausalLM", "DeepseekV4ForCausalLM", @@ -664,6 +666,7 @@ TEST_CASE("hf_registry_coverage: every registration has an example config fixtur "Qwen3_5ForConditionalGeneration", "Qwen3_5MoeForCausalLM", "Qwen3_5MoeForConditionalGeneration", + "Qwen4ExpForConditionalGeneration", "StableLmForCausalLM", }; const std::vector supported = ModelRegistry::SupportedArchs(); @@ -751,7 +754,7 @@ TEST_CASE("raise_for_unsupported: subset default message and order match oracle" "'Qwen3MoeForCausalLM', 'Qwen3VLForConditionalGeneration', " "'Qwen3_5ForCausalLM', 'Qwen3_5ForConditionalGeneration', " "'Qwen3_5MoeForCausalLM', " - "'Qwen3_5MoeForConditionalGeneration', 'StableLmForCausalLM'])", + "'Qwen3_5MoeForConditionalGeneration', 'Qwen4ExpForConditionalGeneration', 'StableLmForCausalLM'])", std::runtime_error); const HfConfig multiple = Config({"UnknownA", "UnknownB"}); @@ -775,7 +778,7 @@ TEST_CASE("raise_for_unsupported: subset default message and order match oracle" "'Qwen3MoeForCausalLM', 'Qwen3VLForConditionalGeneration', " "'Qwen3_5ForCausalLM', 'Qwen3_5ForConditionalGeneration', " "'Qwen3_5MoeForCausalLM', " - "'Qwen3_5MoeForConditionalGeneration', 'StableLmForCausalLM'])", + "'Qwen3_5MoeForConditionalGeneration', 'Qwen4ExpForConditionalGeneration', 'StableLmForCausalLM'])", std::runtime_error); } diff --git a/tests/vllm/models/test_qwen4_exp_scaffold.cpp b/tests/vllm/models/test_qwen4_exp_scaffold.cpp new file mode 100644 index 000000000..dc65ece2f --- /dev/null +++ b/tests/vllm/models/test_qwen4_exp_scaffold.cpp @@ -0,0 +1,348 @@ +// Qwen4-Exp W1 scaffold (MODEL-MM-QWEN4-EXP, #1981). +// +// Everything here drives the PRODUCTION entry point: +// `LoadHfConfig` -> `ModelRegistry::Resolve` -> `factory->parse_config`, and +// the refusals through `factory->load_weights` / `->forward` / `->make_kv_cache`. +// A case that built `Qwen4ExpParams` by hand would prove the struct parses and +// NOT that anything reaches it, which AGENTS.md "Nothing lands dead" refuses to +// accept as evidence. +// +// ORACLE: transformers **5.16.0**, the lane pin accepted for this row. vLLM +// implements `qwen4_exp` at no revision, so there is nothing to mirror on this +// surface. Values come from the committed fixture, which is the published +// `Qwen/Qwen3.8-Flash-Next` `config.json` verbatim. +#include +#include +#include +#include +#include +#include + +#include "doctest/doctest.h" +#include "nlohmann/json.hpp" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/model_executor/models/qwen4_exp.h" +#include "vllm/transformers_utils/hf_config.h" + +using vllm::HfConfig; +using vllm::LoadHfConfig; +using vllm::ModelRegistry; +using vllm::ParseQwen4ExpParams; +using vllm::Qwen4ExpLayerKind; +using vllm::Qwen4ExpParams; + +namespace { + +const char* FixtureDir() { +#ifdef QWEN4_EXP_CKPT_FIXTURE_DIR + return QWEN4_EXP_CKPT_FIXTURE_DIR; +#else + return "tests/vllm/models/fixtures/qwen4_exp"; +#endif +} + +// Unique to THIS PROCESS, not merely to this object. A bare `static int +// counter` makes two concurrent runs of this binary share a path and delete +// each other's directory (#1860); the failure reads as NO RESULT rather than +// as a failure, so it is worth the six lines. No `getpid()`, which MSVC spells +// differently. +std::filesystem::path UniqueTempDir(const std::string& stem) { + static const std::string kToken = [] { + std::random_device rd; + std::ostringstream os; + os << std::hex << rd() << "_" + << std::chrono::steady_clock::now().time_since_epoch().count(); + return os.str(); + }(); + static int counter = 0; + return std::filesystem::temp_directory_path() / + (stem + kToken + "_" + std::to_string(counter++)); +} + +class TempConfig { + public: + explicit TempConfig(const nlohmann::json& doc) { + dir_ = UniqueTempDir("qwen4_exp_cfg_"); + std::filesystem::create_directories(dir_); + std::ofstream(dir_ / "config.json") << doc.dump(); + } + ~TempConfig() { + std::error_code ec; + std::filesystem::remove_all(dir_, ec); + } + std::string path() const { return (dir_ / "config.json").string(); } + + private: + std::filesystem::path dir_; +}; + +nlohmann::json FixtureDoc() { + std::ifstream in(std::string(FixtureDir()) + "/config.json"); + REQUIRE_MESSAGE(in.good(), "fixture config.json missing under " << FixtureDir()); + nlohmann::json doc; + in >> doc; + return doc; +} + +// Resolve through the registry and run the model's own config hook, which is +// exactly what `ModelRegistry::Load` does before it touches a weight. +Qwen4ExpParams ParseThroughRegistry(const nlohmann::json& doc) { + TempConfig cfg(doc); + const HfConfig config = LoadHfConfig(cfg.path()); + const vllm::ModelRegistration& reg = ModelRegistry::Resolve(config); + REQUIRE(reg.factory != nullptr); + reg.factory->parse_config(config); // the production hook + return ParseQwen4ExpParams(config); +} + +std::string ThrowText(const nlohmann::json& doc) { + try { + ParseThroughRegistry(doc); + } catch (const std::exception& e) { + return e.what(); + } + return ""; +} + +} // namespace + +TEST_CASE("qwen4_exp: the published config resolves through the registry") { + const nlohmann::json doc = FixtureDoc(); + REQUIRE(doc["architectures"][0] == "Qwen4ExpForConditionalGeneration"); + REQUIRE(doc["model_type"] == "qwen4_exp"); + + const Qwen4ExpParams p = ParseThroughRegistry(doc); + + CHECK(p.hidden_size == 2560); + CHECK(p.num_hidden_layers == 48); + CHECK(p.vocab_size == 248320); + CHECK(p.hc_count == 4); + CHECK(p.hc_lowrank == 320); + CHECK(p.num_experts == 512); + CHECK(p.num_experts_per_tok == 10); + CHECK(p.moe_intermediate_size == 640); + CHECK(p.shared_expert_intermediate_size == 640); + CHECK(p.num_attention_heads == 24); + CHECK(p.num_key_value_heads == 2); + CHECK(p.head_dim == 256); + + // QSA. block_topk is DERIVED, never read: 2048 / 4 = 512. + CHECK(p.qsa.n_heads == 4); + CHECK(p.qsa.kv_heads == 1); + CHECK(p.qsa.head_dim == 128); + CHECK(p.qsa.budget == 2048); + CHECK(p.qsa.compress_ratio == 4); + CHECK(p.qsa.block_topk() == 512); + + // PLE geometry, and the derived values a port gets wrong silently. + CHECK(p.ple.ngram_size == 3); + CHECK(p.ple.heads_per_ngram == 8); + CHECK(p.ple.ngram_heads() == 16); + CHECK(p.ple.embed_dim == 2560); + CHECK(p.ple.head_dim_per_ngram() == 160); + // (4 - 1) * 3 = 9, NOT kernel-1. The conv is dilated, so its state is three + // times deeper than an undilated one. + CHECK(p.ple.short_conv_state_len() == 9); + CHECK(p.ple.split_ngram_parts == 128); + // Absent from the published config; the dataclass default. This value is + // load-bearing: it seeds the splitmix64 chain that produces the n-gram hash + // multipliers, and 1234 is what reproduces the `layer_multipliers` buffer + // stored in the released checkpoint. + CHECK(p.ple.seed == 1234); + + CHECK(p.mtp_num_hidden_layers == 1); + // Three conv states: GDN conv, PLE conv, and the n-gram token history. + CHECK(p.number_of_conv_states() == 3); +} + +TEST_CASE("qwen4_exp: full_attention is rewritten, and the rewrite equals the interval synthesis") { + nlohmann::json doc = FixtureDoc(); + // The published checkpoint says `full_attention` for layers that actually run + // the QSA indexer. A reader that takes it at face value wires DENSE attention + // on 12 of 48 layers and is wrong without saying so. + const auto& published = doc["text_config"]["layer_types"]; + REQUIRE(published.size() == 48); + bool saw_full = false; + for (const auto& e : published) { + if (e == "full_attention") saw_full = true; + CHECK_MESSAGE((e == "full_attention" || e == "linear_attention"), + "unexpected published layer type " << e); + } + REQUIRE_MESSAGE(saw_full, + "the fixture must still contain `full_attention`, or this " + "case is asserting nothing"); + + const Qwen4ExpParams from_list = ParseThroughRegistry(doc); + + // Same config with `layer_types` DELETED, so the interval path runs instead. + nlohmann::json synth = doc; + synth["text_config"].erase("layer_types"); + REQUIRE(synth["text_config"].contains("full_attention_interval")); + const Qwen4ExpParams from_interval = ParseThroughRegistry(synth); + + REQUIRE(from_list.layer_types.size() == 48); + REQUIRE(from_interval.layer_types.size() == 48); + CHECK_MESSAGE(from_list.layer_types == from_interval.layer_types, + "the rewritten published list and the interval synthesis must " + "agree; if they diverge one of the two paths is wrong and the " + "checkpoint will not say which"); + + std::vector sparse; + for (size_t i = 0; i < from_list.layer_types.size(); ++i) { + if (from_list.layer_types[i] == Qwen4ExpLayerKind::kQwenSparseAttention) { + sparse.push_back(static_cast(i)); + } + } + const std::vector expected{3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47}; + CHECK(sparse == expected); + CHECK(sparse.size() == 12); +} + +TEST_CASE("qwen4_exp: ple_layer_ids is ONE-indexed and lands on layer 1") { + const nlohmann::json doc = FixtureDoc(); + REQUIRE(doc["text_config"]["ple_layer_ids"] == nlohmann::json::array({2})); + + const Qwen4ExpParams p = ParseThroughRegistry(doc); + // `[2]` one-indexed selects 0-based layer 1. Upstream documents the field as + // one-indexed, its validator resolves `layer_types[layer_id - 1]`, and every + // PLE tensor in the released checkpoint sits under `...layers.1.ple.`. + REQUIRE(p.ple.layer_ids_zero_based.size() == 1); + CHECK(p.ple.layer_ids_zero_based[0] == 1); + // And that layer must be a linear-attention one, which is what upstream's + // own PLE validation requires. + CHECK(p.layer_types[1] == Qwen4ExpLayerKind::kLinearAttention); +} + +TEST_CASE("qwen4_exp: an omitted partial_rotary_factor keeps upstream's inherited 0.25") { + // REGRESSION GUARD, not a nicety. `IsQwen35Family` in the shared HfConfig + // reader does not list `qwen4_exp`, so an absent `partial_rotary_factor` + // defaults THERE to 1.0 (full rotary), while upstream `Qwen4ExpTextConfig` + // subclasses `Qwen3_5MoeTextConfig` and inherits 0.25. Taking the shared + // reader's value would give rotary_dim 256, and because upstream's own guard + // is `rotary_dim > indexer_head_dim` (128), we would REFUSE a config upstream + // ACCEPTS. Mirroring the inheritance is what keeps the refusal sets identical. + nlohmann::json doc = FixtureDoc(); + doc["text_config"].erase("partial_rotary_factor"); + if (doc["text_config"].contains("rope_parameters")) { + doc["text_config"]["rope_parameters"].erase("partial_rotary_factor"); + } + + const Qwen4ExpParams p = ParseThroughRegistry(doc); + CHECK(p.partial_rotary_factor == doctest::Approx(0.25)); + CHECK(p.rotary_dim == 64); + CHECK(p.rotary_dim <= p.qsa.head_dim); +} + +TEST_CASE("qwen4_exp: the config refuses every unrepresentable combination BY NAME") { + SUBCASE("an unsupported layer type") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["layer_types"][0] = "sliding_attention"; + CHECK(ThrowText(doc).find("sliding_attention") != std::string::npos); + } + SUBCASE("hc_count must exceed 1") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["hc_count"] = 1; + CHECK(ThrowText(doc).find("hc_count") != std::string::npos); + } + SUBCASE("num_experts_per_tok above num_experts") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["num_experts_per_tok"] = 513; + CHECK(ThrowText(doc).find("num_experts_per_tok") != std::string::npos); + } + SUBCASE("a partial QSA group names what is missing") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"].erase("indexer_budget"); + const std::string msg = ThrowText(doc); + CHECK(msg.find("QSA") != std::string::npos); + CHECK(msg.find("indexer_budget") != std::string::npos); + } + SUBCASE("QSA requires exactly one indexer kv head") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["indexer_kv_heads"] = 2; + CHECK(ThrowText(doc).find("indexer_kv_heads") != std::string::npos); + } + SUBCASE("the indexer budget must divide by the compress ratio") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["indexer_budget"] = 2049; + CHECK(ThrowText(doc).find("indexer_budget") != std::string::npos); + } + SUBCASE("a rotary dim wider than the indexer head") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["partial_rotary_factor"] = 1.0; + if (doc["text_config"].contains("rope_parameters")) { + doc["text_config"]["rope_parameters"]["partial_rotary_factor"] = 1.0; + } + CHECK(ThrowText(doc).find("indexer_head_dim") != std::string::npos); + } + SUBCASE("a PLE id outside the one-indexed range") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["ple_layer_ids"] = nlohmann::json::array({0}); + CHECK(ThrowText(doc).find("one-indexed") != std::string::npos); + } + SUBCASE("a PLE id on a sparse-attention layer") { + nlohmann::json doc = FixtureDoc(); + // One-indexed 4 is 0-based 3, which the rewrite makes sparse. + doc["text_config"]["ple_layer_ids"] = nlohmann::json::array({4}); + CHECK(ThrowText(doc).find("linear_attention") != std::string::npos); + } + SUBCASE("a layer_types list whose length disagrees with num_hidden_layers") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["layer_types"].erase(0); + CHECK(ThrowText(doc).find("layer_types") != std::string::npos); + } +} + +TEST_CASE("qwen4_exp: load, forward and the KV spec refuse BY NAME, naming the owing wave") { + const nlohmann::json doc = FixtureDoc(); + TempConfig cfg(doc); + const HfConfig config = LoadHfConfig(cfg.path()); + const vllm::ModelRegistration& reg = ModelRegistry::Resolve(config); + REQUIRE(reg.factory != nullptr); + + // The config hook must NOT throw: W1 claims exactly this much works. + CHECK_NOTHROW(reg.factory->parse_config(config)); + + SUBCASE("the safetensors loader") { + const vllm::ModelSource source{}; + std::string msg; + try { + (void)reg.factory->load_weights(reg, config, source); + } catch (const std::exception& e) { + msg = e.what(); + } + // Name the architecture, name what is missing, point at the record. A bare + // "not implemented" sends the reader to the wrong layer. + CHECK(msg.find("Qwen4ExpForConditionalGeneration") != std::string::npos); + CHECK(msg.find("weight loader") != std::string::npos); + CHECK(msg.find("#1978") != std::string::npos); + // And it must NOT degrade into a lower-layer shape or dtype complaint. + CHECK(msg.find("tensor not found") == std::string::npos); + } + + SUBCASE("the KV-cache spec") { + std::string msg; + try { + (void)reg.factory->make_kv_cache(config, 16, 4); + } catch (const std::exception& e) { + msg = e.what(); + } + CHECK(msg.find("Qwen4ExpForConditionalGeneration") != std::string::npos); + CHECK(msg.find("KV-cache spec") != std::string::npos); + } +} + +TEST_CASE("qwen4_exp: the registry reports it as multimodal and hybrid") { + const nlohmann::json doc = FixtureDoc(); + TempConfig cfg(doc); + const HfConfig config = LoadHfConfig(cfg.path()); + const vllm::ModelRegistration& reg = ModelRegistry::Resolve(config); + CHECK(reg.info.is_text_generation_model); + CHECK(reg.info.supports_multimodal); + // 36 of 48 layers are Gated DeltaNet carrying recurrent state. + CHECK(reg.info.is_hybrid); + // FALSE by the house convention: the ModelInfo subset's only reader + // short-circuits on is_hybrid, so every GDN-hybrid wrapper leaves this + // false even though upstream's class carries HasInnerState. + CHECK_FALSE(reg.info.has_inner_state); + CHECK_FALSE(reg.info.is_pooling_model); +} diff --git a/tests/vllm/test_model_loader_gguf.cpp b/tests/vllm/test_model_loader_gguf.cpp index e99aa8f91..9b7dbdf6f 100644 --- a/tests/vllm/test_model_loader_gguf.cpp +++ b/tests/vllm/test_model_loader_gguf.cpp @@ -165,6 +165,7 @@ TEST_CASE("FromModelDir rejects an unknown dense architecture before loading") { "'Qwen3MoeForCausalLM', 'Qwen3VLForConditionalGeneration', " "'Qwen3_5ForCausalLM', 'Qwen3_5ForConditionalGeneration', " "'Qwen3_5MoeForCausalLM', " - "'Qwen3_5MoeForConditionalGeneration', 'StableLmForCausalLM'])", + "'Qwen3_5MoeForConditionalGeneration', " + "'Qwen4ExpForConditionalGeneration', 'StableLmForCausalLM'])", std::runtime_error); } From 34dd34a1ec60ba31f58b0d34f43e040eb365878d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 17:04:51 +0000 Subject: [PATCH 2/4] =?UTF-8?q?fix(MODEL-MM-QWEN4-EXP):=20W1=20=E2=80=94?= =?UTF-8?q?=20the=20inherited=200.25=20does=20not=20exist,=20and=20four=20?= =?UTF-8?q?other=20config=20defaults=20were=20wrong?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fresh review failed this row, and the deciding finding retracts a claim a4c144fb6 made three times: in a code comment, in its commit body, and in the pull request body. This commit corrects all three and repairs the code they described. THE RETRACTION. a4c144fb6 read `partial_rotary_factor` out of the text config with a hardcoded default of 0.25, on the stated ground that `Qwen4ExpTextConfig` subclasses `Qwen3_5MoeTextConfig` and inherits that value, and that taking `config.rotary_dim` from the shared reader would therefore make us refuse a config upstream accepts. That is false at the lane pin, and it came from reading the MODULAR file and assuming its inheritance survives into the flattened class that actually executes. It does not. At transformers v5.16.0 the generated class is `class Qwen4ExpTextConfig(PreTrainedConfig)`; `partial_rotary_factor` is not among its declared fields and the string `0.25` does not occur anywhere in `configuration_qwen4_exp.py`. Its only two mentions of the name are the validator's own read, `(self.rope_parameters or {}).get("partial_rotary_factor", 1.0)`, and the `rotary_dim` it feeds. Even in the modular source the bypass is deliberate: `__post_init__` calls `PreTrainedConfig.__post_init__` DIRECTLY, skipping the `kwargs.setdefault("partial_rotary_factor", 0.25) # assign default for BC` that is the sole source of the value. So the argument was backwards in both directions. We ACCEPTED a config with no factor at all, where upstream computes rotary_dim 256, finds it wider than indexer_head_dim 128, and raises — handing W4 a 64-of-256 slice on a checkpoint that wants 256. And we REFUSED a config with top-level 1.0 and `rope_parameters.partial_rotary_factor` 0.25, which upstream accepts: the exact false refusal the comment claimed to prevent. The shared reader was right all along. `ParseRopeParameters` takes the top level first and lets the rope dict override, which is precisely upstream's `setdefault` precedence, and `IsQwen35Family` correctly does not list `qwen4_exp`. The local read is gone, `config.rotary_dim` is used, and the test that pinned 0.25 under a "REGRESSION GUARD" banner is inverted rather than deleted, so the wrong answer cannot come back quietly. THE OTHER DEFAULTS, all of the same shape and none visible to a token gate. The four PLE n-gram fields defaulted to 0 rather than upstream's 3, 8, 20_000_000 and 128, so a config that omits them — legal upstream, every one is a declared field with a default — was refused, and the two vocab fields, which have no guard at all, carried a zero-sized n-gram table into W2. They now resolve unconditionally, as dataclass fields do, which also removes a division by zero. `output_gate_type` did not fall back to `hidden_act` as `self.output_gate_type or self.hidden_act` requires, and the local check for it was a constant false: the shared reader had already refused everything outside the accepted set before that line could run. `eos_token_id` was not required when PLE is enabled, though it is a segment boundary in the hashed n-gram construction via `_shift_right_ignore_eos` and the published GGUF stores it as `qwen4exp.ple.eos_token_id`. And `ple_embed_dim <= 0` was dropped from upstream's condition, so -2560 passed, because `-2560 % 16 == 0` in C++. THE BOUNDARY IS NOW MEASURED, NOT DESCRIBED. This row has no reachable token gate, so nothing downstream will ever catch one of these by running the model, and the config layer is the last place they are checkable. The config layer is gateable even though the model is not: transformers 5.16.0 installs and imports without torch — it says so itself — and `Qwen4ExpConfig.from_dict` runs `validate_architecture` in full. W1 is therefore gated by a 39-case two-direction sweep, each config put through the oracle on one side and through `LoadHfConfig -> ModelRegistry::Resolve -> factory->parse_config` on the other. 35 agree. 4 differ, and all 4 are ours refusing what upstream accepts, never the reverse. All 15 upstream rejections are tabulated against their upstream line in the spec's new `## The refusal boundary`, with the local tighter guards listed beside them and one of those attributed to the shared reader rather than to this model. THREE THINGS NOTHING TESTED, each deletable without a red. The registered `parse_config` hook was called and then ignored: every assertion observed the free function instead, so gutting the hook to `(void)config;` left all 151 assertions green. Refusals now enter through the hook alone. The forward refusal was unreachable rather than merely untested — `ModelAs` ran first, and nothing can produce a loaded Qwen4-Exp while the loader refuses, so every reach became a type-mismatch report. The refusal moved above the downcast, which is also strictly safer on the #775 axis because no cast happens at all; W5 restores `ModelAs` when there is a real model to open. And the GGUF arm's refusal had no assertion, so deleting the whole branch would have sent a GGUF load to the safetensors message and the reader to the wrong wave. `block_topk()` and `head_dim_per_ngram()` divided by zero on a legally-parsed config — QSA is optional as a group and an absent PLE left the head count at zero — which is SIGFPE, a crash rather than a refusal, on the two helpers the header advertises to W2 and W4. Both refuse by name now. The model's local `TextOf` resolved only `text_config` where the shared `ResolveTextConfig` also handles `llm_config` and `thinker_config.text_config`, so one parse answered "what is the text config" two different ways and produced a silently half-parsed result; it mirrors the shared resolution now. Records. The model-matrix row `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` said SPEC ONLY, NO PRODUCT CODE and `READY` / unassigned while `docs/FEATURES.md` already said REGISTERED, NOT LOADABLE; the two would have contradicted each other on main. It moves to `ACTIVE`, owner `MODEL-MM-QWEN4-EXP-W1`, and the lifecycle rollup moves with it (ACTIVE 10 to 11, READY 4 to 3) because `check-model-checklist.py` requires the state row and the rollup to agree in the same commit. #1981 was in the pull request body and in every runtime refusal message this code emits, and in neither the issue index nor the spec, so a reader following the pointer a running binary gives them found nothing at the other end. Two promoted claims got the pin their promotion requires: `tensor_type_fallback` is now read at the llama-cpp oracle's recorded revision 10bf611e (b10451) rather than at master, and the published GGUF is pinned to revision 8bdc6666 with per-shard sizes and sha256s, which mattered — the repo moved after this pull request was opened. Test count 7 cases / 151 assertions to 12 / 294. `test_model_registry` 24/958, `test_model_loader_gguf` 6/18, `test_registry_downcast_refusal` 6/33, `check-supported-models`, `check-runner-routing-consistency` and `check-windows-portability` all OK. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/claims/CLAIM-MODEL-MM-QWEN4-EXP-W1.md | 5 + .agents/engine-matrix.md | 2 +- .agents/issue-index.md | 1 + .agents/model-matrix.md | 8 +- .agents/specs/qwen4-exp-flash-next.md | 162 +++++- docs/FEATURES.md | 2 +- src/vllm/model_executor/models/qwen4_exp.cpp | 230 ++++++--- src/vllm/model_executor/models/qwen4_exp.h | 41 +- .../models/qwen4_exp_registry.cpp | 42 +- tests/scripts/test_agent_record.py | 32 +- tests/vllm/models/test_qwen4_exp_scaffold.cpp | 484 +++++++++++++++++- 11 files changed, 880 insertions(+), 129 deletions(-) create mode 100644 .agents/claims/CLAIM-MODEL-MM-QWEN4-EXP-W1.md diff --git a/.agents/claims/CLAIM-MODEL-MM-QWEN4-EXP-W1.md b/.agents/claims/CLAIM-MODEL-MM-QWEN4-EXP-W1.md new file mode 100644 index 000000000..8a5a5d0f2 --- /dev/null +++ b/.agents/claims/CLAIM-MODEL-MM-QWEN4-EXP-W1.md @@ -0,0 +1,5 @@ +# CLAIM-MODEL-MM-QWEN4-EXP-W1 + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-MODEL-MM-QWEN4-EXP-W1` | `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` (`ACTIVE`) | Claude Code (opus-5), fresh implementer repairing a review — the review was written by a different agent and the code by a third | isolated worktree `/home/mudler/_git/vllm.cpp-q4w1`; CPU-only Debug build, NO GPU, NO checkpoint, NO benchmark. One download: `transformers` 5.16.0 into a scratch venv, which is what makes the config layer of this row oracle-gated at all | `row/MODEL-MM-QWEN4-EXP-W1`, PR [#1986](https://github.com/mudler/vllm.cpp/pull/1986), issue [#1981](https://github.com/mudler/vllm.cpp/issues/1981) under campaign [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | Owns ONLY the W1 config surface: `src/vllm/model_executor/models/qwen4_exp.{h,cpp}`, `src/vllm/model_executor/models/qwen4_exp_registry.cpp`, `tests/vllm/models/test_qwen4_exp_scaffold.cpp` and its fixture, the `qwen4_exp` rows in `docs/FEATURES.md` and `.agents/model-matrix.md`, and `.agents/specs/qwen4-exp-flash-next.md`. EXCLUDES the shared `HfConfig` reader, whose `partial_rotary_factor` semantics this row now DEPENDS on rather than duplicates; EXCLUDES every other registered architecture; EXCLUDES the loader, forward, KV-cache spec and GGUF arm, which W2-W6 owe and which this row only refuses by name | `ACTIVE` | 2026-08-26 — W1 landed at `a4c144fb6`, failed a fresh review, and this claim carries the repair. The deciding finding retracted an "inherited 0.25" `partial_rotary_factor` default that does not exist at the pin; four more defaults were wrong in the same direction, and three production refusals (the `parse_config` hook, the forward, the GGUF arm) could each be deleted without a red. The row's product is a refusal BOUNDARY and it is now measured rather than described: a 39-case two-direction sweep against a running `transformers` 5.16.0 (it imports without torch, so `validate_architecture` executes) agrees on 35 and differs on 4, every difference being a local guard STRICTER than upstream. All 15 upstream rejections are tabulated against their upstream line in the spec's `## The refusal boundary`. `test_qwen4_exp_scaffold` 7 cases/151 assertions to 12/294 | diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index c9a62f2b9..d274e226d 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -227,7 +227,7 @@ claims it. | `ENG-RELEASE-WINDOWS` | Native Windows x86_64 pre-alpha release extension: one adaptive MSVC/UCRT CPU bundle with AVX2 executed in CI and one Vulkan preview bundle, both deterministic ZIPs and authenticated by the existing release handoff | T0 | vLLM has no Windows release path; runtime behavior remains pinned to vLLM `555967922`. Platform substrate reference: llama.cpp `src/llama-mmap.cpp:520-590` @ `237ad9b961f009ae19ac29dbce4cd0c1251f94b3`; Win32 API is the OS authority | W14 Win32 portability/MSVC CPU, W15 deterministic ZIP/PE packaging + Vulkan, and W16 ten-tuple prerelease workflow/version/docs implemented for one PR | Linux portability/release mutation gates are local evidence only. Native `windows-2022` MSVC `/W4 /WX`, extracted runtime/ISA smokes, merged-SHA ten-tuple dry run, `v0.0.3-pre.1` publication, attestations, and exact 32-asset audit remain pending; no Windows ZIP exists yet | [windows-binary-release.md](specs/windows-binary-release.md); [#117](https://github.com/mudler/vllm.cpp/issues/117) | `ACTIVE` | `CLAIM-ENG-RELEASE-WINDOWS` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | `docker/Dockerfile` (cpu/vulkan/cuda targets calling the release scripts); `docker/healthcheck.sh`; `release/container-matrix.json`; `scripts/check-container-matrix.py`; `scripts/check-container-workflow.py`; `scripts/validate-container-image.py`; `scripts/container_tags.py`; `.github/workflows/containers.yml`; SIGTERM handler `src/vllm/entrypoints/openai/server_main.cpp` (`SignalShutdown`, all three `listen()` sites); the pre-existing `docker/Dockerfile.arm64` is an unrelated CPU bench cross-check | issues `#170`, `#312`, `#394`; `tests/scripts/test_check_container_matrix.py` 31/31; `test_check_container_workflow.py` 29/29; `test_check_cuda_fat_gencode.py` 7+4 subtests. **GB10 2026-08-11 (`promaxgb10-4ad8`, `sm_121a`, CUDA 13.3): arm64 cuda image 1.71 GB, 673/673 objects, ten-SM gencode audit PASS, and a REAL GPU boot -- `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, `--gpus all`, host driver 580.159.03 injected.** cpu amd64 783 MB gated locally; cpu+vulkan amd64 green on hosted CI **arm64 cuda lane RUNTIME-VERIFIED on GB10 2026-08-11** -- the first accelerator-hardware evidence for any lane. Four defects were removed to get there, each found by building rather than reading: the CUDA 12.9 base could not compile `sm_110`, the BuildKit cache mount outlived its toolchain (both #366), Marlin gencode had drifted from the feature table and failed the audit on 14 correctly-compiled TUs (#394, blocking BOTH cuda tuples project-wide), and the validator could only ever produce build evidence because its boot smoke never passed `--gpus`. **NOT established: nothing is published to GHCR; amd64 cuda is unbuilt; the published arm64 image is SBSA (`targets/sbsa-linux`), so Tegra -- Thor `sm_110`, Orin `sm_87` -- is untested and NOT covered** **ORIN (Tegra) 2026-08-11: the SBSA image RUNS on Jetson AGX Orin `sm_87` (L4T R36.4.3, Docker 27.5.1) -- Qwen3-0.6B (rev `c1899de2`) loads and GENERATES via `/v1/completions`, tegrastats GR3D 95-97% during decode vs 14-15% idle.** Tegra needs `--runtime nvidia --gpus all`: `--gpus` alone is refused by the hook and `--runtime` alone mounts no driver | [container-images.md](specs/container-images.md); issues [#170](https://github.com/mudler/vllm.cpp/issues/170), [#312](https://github.com/mudler/vllm.cpp/issues/312), [#394](https://github.com/mudler/vllm.cpp/issues/394) | `ACTIVE` | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | | `ENG-DOCS-SITE` | Publish the 11 `docs/*.md` as a browsable GitHub Pages site at `https://mudler.github.io/vllm.cpp/` WITHOUT a second copy of the prose. A Hugo site at `website/` mounts `../docs` READ-ONLY and derives everything else from what is already in the files: each page title from the file's first `# H1`, the sidebar order from `website/data/nav.yaml`, and links through a Goldmark render hook (internal `.md` → site URL; the 139 `../.agents/**` and `../AGENTS.md` escapes → GitHub blob URLs, since the protocol tree is deliberately NOT published). **No file under `docs/` is modified, moved, renamed, or given front matter**, so `check-doc-checkpoint.py` and every protocol path reference keep working and there is no second surface that can drift — the whole point of the row. Custom lean layouts, NO theme and NO submodule: off-the-shelf docs themes read titles, weights and menus out of front matter this design deliberately does not have, so each would need its title partial, menu and link hook overridden anyway, and hugo-book additionally floors at Hugo 0.158 against the 0.146.3 pin CI and the local toolchain share. Hard prerequisite inside the repo: `classify_path` in `scripts/check-pr-size.py` FAILS CLOSED on `website/**` (verified: raises `ValueError: unclassified repository path`), so the classifier must learn the path or the PR cannot pass the project's own size gate. Hard prerequisite outside it: GitHub Pages must be enabled with the source set to GitHub Actions — the workflow is inert otherwise. A marketing landing page is explicitly OUT of scope (`README.md` stays the front door), as is any restructuring of `docs/`; the custom domain is parked behind the pending vLLM trademark question | T1 | NO vLLM analogue — upstream's docs are a separate mkdocs site and nothing in this row mirrors upstream *behavior*, so it carries no parity obligation. The STRUCTURAL reference is LocalAI's `.github/workflows/gh-pages.yml` (two Hugo sites merged into one Pages artifact), reduced to the docs half | read-only mount `website/hugo.toml:29`; title-from-H1 `website/layouts/partials/title.html:10`; link rewriting `website/layouts/_default/_markup/render-link.html:27`; guard `scripts/check-site.py:70`; deploy `.github/workflows/gh-pages.yml` | `tests/scripts/test_check_site.py:51,56,66,80,89,97` (6 mutation cases: clean tree, H1 stripped, doc absent from nav, nav entry with no file, duplicated entry, missing nav file); build evidence 14 pages with `docs/bench-evidence` + `docs/superpowers` absent from `public/` and no `href` ending in `.md`; 48 protocol links rewritten in `docs/status/`. NO published page is claimed: GitHub Pages is not yet enabled on the repository, which is the recorded stop condition holding this row at `GATING` | [gh-pages-docs-site.md](specs/gh-pages-docs-site.md); issue [#224](https://github.com/mudler/vllm.cpp/issues/224) | `READY` | `CLAIM-ENG-DOCS-SITE` | -| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1494` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1562`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | +| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `scripts/check-agent-record.py::BARE_CITATION_RE` (the bare form), `scripts/check-agent-record.py::cell_citations` (both forms, with the adjacent-symbol rule), `scripts/check-agent-record.py::classify_citation` (OK / STALE / BROKEN), `scripts/check-agent-record.py::RECORD_ANCHOR_STATES` (gap 3: `ACTIVE` and `READY` join the count), `scripts/check-agent-record.py::check_record_anchors` (the two-way gate). SYMBOL-anchored rather than line-anchored as of `SPEC-DFLASH2` W2, which added a justification paragraph to this file's `KERNEL` count and shifted all five ranges by 14 lines at once -- the rot this row exists to measure, produced by an edit to the very file the row cites; budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1510` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1578`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | | `ENG-RECORD-CONFLICT-SURFACES` | Retire the shared record surfaces that make concurrent PRs conflict by construction. MEASURED at `origin/main` `d928e2c3` with `git merge-tree --write-tree` over every open PR: **16 of 29 conflict (55%), and 13 of the 16 conflict in bookkeeping files ONLY**, with no product code involved — `.agents/coordination.md` in 8, `.agents/NOW.md` in 5, `.agents/roadmap_v1.md` in 4, `scripts/check-public-doc-tables.py` in 4, `docs/STATUS.md` in 4, and any `src/`/`tests/` path in just 3. Three defects, each of which GUARANTEES rather than risks a collision. (1) `.agents/NOW.md` is a fixed-size shared buffer at EXACTLY 6000/6000 chars (`check-now-current.py:31`), so adding a row requires evicting another and every PR is a read-modify-write of one global — and the conflict is the LUCKY outcome, since a clean three-way merge would apply both evictions and both additions, silently dropping live rows and blowing the very budget the checker defends. (2) `STATUS_RATCHET = {"chars": 243245}` (`check-public-doc-tables.py:557`) is a hardcoded byte count of a DIFFERENT file that may only fall, so a PR owing `docs/STATUS.md` one lifecycle line must delete unrelated prose from another row to pay for it and edit the checker too; the checker's own comment at `:331` already records the failure (*"a ratchet pinned to the byte turns every concurrently merged row's one-line status edit into a spurious failure"*) and answered it with slack instead of removing the coupling. (3) `.agents/coordination.md`'s active-claims table is insert-at-one-anchor: the six ROCm GDN PRs (#334 #336 #341 #343 #345 #348) are ONE author's sequential stack that conflicts on nothing else, each appending a ~1,500-char row — the PR description, transcribed into a file every other claim also writes. It also contradicts the protocol it serves: `AGENTS.md` holds that *"History is git"* and *"There is no state log"*, yet both claims tables ARE state logs duplicating `gh pr list`, `row/` branch names and issue state; the argument that refuses a waiver registry applies unchanged to a claims registry. Precedent twice over — `policy.csv` retired in `0f3e44ee`, per-class line budgets retired 2026-08-10 because the gate fired on ordinary work. The exonerated surfaces share ONE property, one writer per file: `.agents/specs/.md` (one file per row, **zero conflicts** in the sample), the `*-matrix.md` inventories, and the append-only `.agents/benchmark-record.md`. SCOPE: remove `STATUS_RATCHET` and the doc-gating global counters while KEEPING the per-cell/per-paragraph caps (local, so they couple nothing); remove the active-claims table and derive claims from open PRs and branch names; drop `NOW.md`'s byte budget; order the roadmap's keyed tables by ID so distinct keys stop colliding at one anchor; and record the invariant — **no surface that every PR must write** — in `AGENTS.md`. No product source, kernel or gate semantic moves | T0 | NO vLLM analogue — this is local protocol machinery, so the mirror rule does not apply and no upstream `file:line` exists to port from. Governed instead by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | - | - (spec-before-code: the red-before suites are named in the spec's Tests section — `tests/scripts/test_check_public_doc_tables.py`, `tests/scripts/test_check_now_current.py`, a mutation case per removed rule proving the obligation survives in the retained caps and `check-doc-checkpoint.py`, and a `git merge-tree` merge-shape regression that must be RED before the `NOW.md`/roadmap work and GREEN after) | [retire-shared-record-surfaces.md](specs/retire-shared-record-surfaces.md); issue [#364](https://github.com/mudler/vllm.cpp/issues/364) | `READY` | `CLAIM-ENG-RECORD-CONFLICT-SURFACES` | | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate rejects CORRECT commits because of paragraph placement, and that is why `main` is red on `agent-record`. `check-commit-trailers.py` reads trailers via `git interpret-trailers --parse`, which treats ONLY the final paragraph as the block; GitHub appends `Co-authored-by:` as a SEPARATE trailing paragraph on a squash merge, so a complete correct block becomes invisible and the gate reports it missing. MEASURED: piping `git show -s --format=%B dbd0d51c` into `git interpret-trailers --parse` prints nothing but the co-author line, and 13 of the last 30 commits on `main` fail the check -- unnoticed only because those runs were cancelled (#274), which HID the defect rather than causing it. FIX: fuse consecutive trailing TRAILER-SHAPED paragraphs before parsing. Nothing is relaxed -- the block must still exist, the marker must still sit above it, each declaration must still appear exactly once, and an AI co-author is still forbidden; the block is merely FOUND where the merge tool left it. A prose paragraph still terminates it. REJECTED IN FLIGHT and recorded because it is the more instructive half: a first attempt also collapsed identical duplicate trailers to fix the multi-commit-squash shape, which relaxes the uniqueness rule an existing test already pins. Rewriting that assertion to suit the change is what AGENTS.md forbids, and the distinction is real -- a doubled block is genuinely malformed and fixable at source, whereas the co-author case is a correct commit defeated by the parser. Reverted in full. SCOPE LIMIT, stated rather than implied: this fixes ONE of five observed shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had been hiding and now correctly fails; `87308dea` (GitHub's `---------` separator), `b8293c88` (squash doubled the block) and `b580452d` (merge button, no trailers) stay red by design. Closing those is a merge-method change, not a checker change | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:60` (`join_trailing_trailer_paragraphs`, `_is_trailer_paragraph`, and the fused `parsed_trailers`) | `tests/scripts/test_check_commit_trailers.py:1` 21 cases -- the RED-BEFORE appended-co-author case plus four GUARDS that keep the fusion bounded (doubled block still fails, contradictory declarations still fail, a no-trailer merge message still fails, prose after the block still fails), all four green before and after; closure [parity-ledger.md#L941](parity-ledger.md#L941) | [trailer-merge-artifacts.md](specs/trailer-merge-artifacts.md); issue [#406](https://github.com/mudler/vllm.cpp/issues/406) | `DONE` | `157080c8` | | `ENG-FORGE-COAUTHOR` | The forbidden-AI-trailer rule was catching ATTRIBUTION rather than an authorship claim, which is why bot-opened PRs red `main` on merge. GitHub composes the squash message itself and appends the account that opened the PR — `Co-authored-by: localai-org-maint-bot <...@users.noreply.github.com>` — and most PRs here are opened by a bot, so nearly every squash trips the AI-identity check. Real instance `f64f2b71`, invisible until #406 repaired the parse, which is why it reads as a new failure and is not one. The rule exists so an AI cannot claim it WROTE the code, and that stays; GitHub is recording who pressed the button, and the AI-involvement claim is already carried separately by `AI-Assisted` and `Assisted-by` in the same block. FIX: accept a `Co-authored-by` at a GitHub account noreply address even when the name matches an AI identity token, keyed on the FORGE'S OWN DOMAIN rather than the name so the exemption cannot be borrowed. A hand-written `Co-authored-by: Claude ` still fails; `Signed-off-by` is excluded from the exemption entirely, because a sign-off is a legal assertion about provenance rather than attribution. `AGENTS.md` records the same distinction in the same change so prose and checker cannot drift | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:38` (`FORGE_ACCOUNT_EMAIL` and the forbidden-trailer skip) | `tests/scripts/test_check_commit_trailers.py:1` 25 cases -- the RED-BEFORE forge-bot case plus THREE guards that matter more than the relaxation because this LOOSENS a rule: a hand-written AI co-author still fails, `Signed-off-by` at the same noreply address still fails, and a human co-author still passes; all three green before and after. Real commit `f64f2b71` re-verified per commit | [forge-coauthor-attribution.md](specs/forge-coauthor-attribution.md); issue [#418](https://github.com/mudler/vllm.cpp/issues/418) | `ACTIVE` | `CLAIM-ENG-FORGE-COAUTHOR` | diff --git a/.agents/issue-index.md b/.agents/issue-index.md index ff108e7fd..e2ab2b8a6 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -745,3 +745,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#526](https://github.com/mudler/vllm.cpp/issues/526) | `SERVE-TOOL-HISTORY-ARGS` | OpenAI multi-turn tool history reaches chat templates with string-valued arguments | bug | | [#1934](https://github.com/mudler/vllm.cpp/issues/1934) | `BACKEND-ROCM` | `RocmPlatform::needs_weight_staging()` is stale-false (a W0-era placeholder never revisited despite #523/#509/#506/ROCM_ATTN/hipGraph landing since), so `CheckDeviceWeightFit` — the #1123/#1870 load-time refusal, including the `policy_forces_full_expand` fix — never runs on ROCm: measured directly, `VT_DEVICE_WEIGHT_BUDGET_BYTES=1` produced no refusal on a real load. The actual device allocation the refusal guards is not gated on this flag, so #1870's crash stays reachable until this closes; owed, not fixed in flow, because flipping the flag also moves `DirectDeviceLoadEligible` and several GDN kernel-dispatch defaults that each need their own correctness check | bug | | [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-QWEN4-EXP` | **`Qwen/Qwen3.8-Flash-Next` declares `Qwen4ExpForConditionalGeneration` / `qwen4_exp`, a new architecture vLLM does not implement, so the port runs on a split oracle: transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH.** Released 2026-08-24, 180B total / 6B activated, image-text-to-text. The `Qwen3.8` in the name is marketing continuity: `.agents/specs/qwen38-27b-bf16-gate.md`'s "one config key differs" precedent does NOT extend here. Read live 2026-08-26 at vLLM `origin/main` = `6a5e8f5979`: no `qwen4*` path, no registry entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. That is absence from vLLM `main` rather than staleness in our pin `555967922`, so a pin advance does not reach it. What exists is transformers [#48337](https://github.com/huggingface/transformers/pull/48337) "Add Qwen4Exp model", MERGED 2026-08-26, 5211 lines, and SGLang [#36497](https://github.com/sgl-project/sglang/pull/36497), still OPEN and therefore inadmissible. **Developer direction 2026-08-26, recorded verbatim: "use transformers as oracle for algorithmic side. but use ops from vllm so we account for optimized path."** Justified rather than convenient: `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so porting it as written yields a correct model at an indefensible speed, while AGENTS.md's mirror-vLLM polarity still binds every primitive vLLM implements. `Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`), all of which this tree has; GDN is an exact match for our AOT gate (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). The delta is four things, and **exactly two have no vLLM op at all**: the PLE dilated depthwise conv (kernel 4, dilation 3; `git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the n-gram hashed embedding. **The survey's load-bearing finding, and it REVERSES this row's first reading: QSA's structural twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The original call was that QSA, being plain GQA rather than MLA, had to map onto vLLM's non-MLA block-sparse case; that reasoning rested on treating `MLAAttentionSpec` as an MLA claim, and **it is not one** — M3's own indexer cache uses it while M3 is a plain-GQA model, with the comment "Key-only: MLAAttentionSpec budgets one vector/token (not 2x for K+V)". It is a budget shape. Remove that prop and the GQA-vs-MLA argument collapses. Verified line by line at `6a5e8f5979`: **nine independent structural matches with DSv4**, `compress_ratio == 4` literally the same number — MQA index with 1 key head at dim 128; `relu(q.k)` summed over index heads vs `(score.relu() * weights).sum(dim=0)`; `1/sqrt(head_dim)`; one score set per query token with no head axis vs `topk_indices_buffer[num_tokens, topk]`; pooling boundary `(position+1) % COMPRESS_RATIO == 0`; RMSNorm on the pooled key; **RoPE at the block-start position** vs `compressed_pos = (position // CR) * CR`; candidate count `visible // compress_ratio`; and one stored state per 4 tokens via `MLAAttentionSpec(tokens_per_state=compress_ratio)`, a first-class KV field documented as "Ints > 1 compress multiple tokens into one state (DSv4 sparse MLA)" that has no M3 equivalent. **M3 is a DIFFERENT ALGORITHM**, not a worse fit: its score is `tl.max(qk, axis=1)` over 128 RAW token dots with no pooling, no relu and no head reduction, it asserts `num_idx_heads == num_kv_heads` ("no topk index reduce") so it emits one block set PER KV HEAD, and its `SPARSE_BLOCK_SIZE = 128` is welded to the KV page size ("One sparse block == one KV page") on both the score and the attend side — moving it to 4 forces a page size of 4 and breaks `tl.dot`, whose tile needs >= 16. M3 contributes exactly ONE thing and it is a wiring precedent, not an algorithm: that a plain-GQA model can own a key-only side cache through `MLAAttentionSpec` and a private indexer backend. **The genuinely new work is the CONSUMER and nothing upstream supplies it** — every DSv4 sparse consumer attends to COMPRESSED MLA KV (one state per 4 tokens) and M3's attend to raw tokens only at page granularity, while QSA attends to RAW tokens selected at ratio-4 granularity. Two silent-failure traps follow: wiring QSA's top-k into a DSv4 sparse-MLA consumer attends a POOLED key/value and still emits plausible tokens, and **a short-prompt token gate cannot catch it because at context <= `indexer_budget` 2048 every candidate is selected** — so any QSA gate must run past 2048 tokens of context, which is now a stated `## Gates` requirement; and `SparseAttnCompressNormRopeStoreC4Kernel` does NOT mean-pool despite its name — it is a learned softmax pool over an OVERLAPPING window of 8 using a score channel this checkpoint does not have, and the CuteDSL variant refuses `overlap=False` at compile, so the **Triton** `head_dim=128` variant is the correct starting point. Two structural consequences beyond the module list: the residual stream is `hc_count * hidden_size` = **4 x 2560 = 10240 wide through the whole stack** with a low-rank read gate and per-branch scalar write gate around both attention and MLP, which is a change to the per-layer loop and every residual buffer rather than a drop-in module; and `number_of_conv_states = 3` on a PLE layer (GDN conv, PLE conv, n-gram token history) plus the indexer side cache, adjacent to [#1963](https://github.com/mudler/vllm.cpp/issues/1963) and [#1966](https://github.com/mudler/vllm.cpp/issues/1966). **NOTHING PUBLISHED FITS**, read live from the HF API against ~119 GB usable on GB10: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB (NVFP4 backbone with the n-gram table left at FP8, 51.2 GB) and `unsloth/...-GGUF` is a README with ZERO weight files. No GGUF exists and no tool can make one, because llama.cpp has no `qwen4_exp` either, so the standing k-quant requirement means authoring the arch on our side AND states that the quantized arms have NO llama.cpp oracle. **The chosen arm does NOT load today, and the blocker is neither the offload nor the budget: this tree cannot keep a gather table quantized at all.** `KeepQuantKDim` returns `-1` for `GgufTensorRole::kEmbeddingTable` (`src/vllm/model_executor/model_loader/gguf_keep_quant.cpp`), and `qwen3_5_gguf_weights.cpp` asserts it by name — "the embedding table cannot keep quant blocks" — so a Q4_K or Q8_0 n-gram table EXPANDS to bf16 and 51.2B params become **102.4 GB of anonymous memory**; the arm dies before the first forward. The reason was already sitting in a header comment ("a gather, not a GEMM ... A quantized-gather op is a follow-up row") and **no such row exists**. The only non-expanding gather residency is `kKeepF16`, requiring ggml type 1 exactly (102.4 GB on disk) and CPU-ONLY, because `EmbeddingKernelCuda` refuses anything but f32/bf16. **Second blocker:** `moe_intermediate_size = 640` makes `ffn_down_exps` Q4_K-illegal on its reduction dim (640 % 256 = 128), as does `hc_lowrank = 320`; llama.cpp's substitution is believed to be Q5_0 (**UNVERIFIED, owed against the pinned llama.cpp oracle**) and the dependent fact IS verified in-tree — our reader knows ggml ids `0,1,2,8,10..14,16,18,19,22..28,30,39,40,41,66` and has **no entry for 3, 6, 7 or 20**, so a stock `llama-quantize -Q4_K_M` file fails at header parse. We author the converter, so the fix is Q4_0 (block 32, same 4.5 bpw). **`ENG-WEIGHT-OFFLOAD` will not help** — it moves zero bytes today (`ConsiderWeight` has no production callers, pinned by a test) and is documented inert on GB10; the tier that DOES work already ships and is proven by the 2.4T model serving 369.97 GiB from a 119.631 GiB box at ~62 GiB resident: mmap `MAP_PRIVATE`, borrow in place, alias the host pointer, `prefault: false`. Corrected sizing: backbone ~67.7 GiB, whole process ~73.5 GiB of 119.631 at 32K single-stream, ~46 GiB of headroom for the page cache, so the ~76 GB estimate was right within 10%. The design works because per-token demand is **<= 64 KiB of reads** (16 lookups x 160 dims over at most 16 pages) against the 2.4T expert lane's 6.95 GB/token. The architecture supplies its own lever: the per-token n-gram cost is `(ngram_size-1)*heads_per_ngram` = 16 lookups of 160 dims, so **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per token** and making it non-resident is the intended design point (RadixArk reached the same split independently). Sizing arithmetic, NOT measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB (yes, ~10 GB left for KV), Q4_K_M backbone with the table non-resident ~76 GB. GB10 is UNIFIED memory so "offload to host" is not a move there; non-resident means disk-backed, and its cost is unmeasured. **Two decisions were put to the developer as explicit accept-or-reject and BOTH are settled 2026-08-26, recorded in place rather than left open.** (1) `.agents/oracles/transformers.md` pins transformers to 5.14.1, deliberately tied to what the pinned vLLM environment resolves so the environment cannot hold two `transformers` at once, and **5.14.1 does not contain `Qwen4Exp`**; the lane-scoped second pin is **ACCEPTED**, on the argument that the invariant guards a vLLM environment against drifting from its transformers and here there is no vLLM implementation to drift from, and it expires the moment vLLM registers `qwen4_exp`. **The lane pin is a real release, not a branch SHA**, which was not the expected outcome: `Qwen4Exp` merged to `main` at 12:03:40Z on 2026-08-26 and `v5.16.0` published at 12:35:15Z, and this was BOUNDED rather than assumed by fetching `models/qwen4_exp/modeling_qwen4_exp.py` at each tag — `v5.16.0` HTTP **200**, `v5.15.0` HTTP **404** — making 5.16.0 the FIRST release carrying the architecture and therefore the tightest available pin. The version string is UNMEASURED (it is the release proven to contain the model, not a `transformers.__version__` read off a running oracle) and `gateable` stays `no`. (2) The first runnable arm is the **Q4_K_M backbone with the n-gram table NON-RESIDENT** (~76 GB). Q8_0 was raised and does not fit at ~191 GB, and no partial-Q8 split reaches 119 GB with the backbone at 8 bits; Q4_K_M-throughout fits on paper at ~109 GB but leaves ~10 GB for KV and activations on a 262144-native-context model, which is not a margin. This promotes the non-resident table from a note to a first-class W6 deliverable, and it is NOT free: GB10 is UNIFIED memory, so the existing host-pinned offload seam (`ENG-WEIGHT-OFFLOAD`, mirroring vLLM's `cpu_offload_gb`) does not by itself solve it there and the mechanism must be disk-backed or genuinely unloaded — established before it is designed around. Spec: [`specs/qwen4-exp-flash-next.md`](specs/qwen4-exp-flash-next.md). No product code lands under the spec pull request | feature | +| [#1981](https://github.com/mudler/vllm.cpp/issues/1981) | `MODEL-MM-QWEN4-EXP` | **W1 of [#1978](https://github.com/mudler/vllm.cpp/issues/1978): the `qwen4_exp` config surface — resolve, validate, register, and refuse by name everywhere else.** Filed and closed in flow. It is indexed rather than left to the pull request body because every `Refuse()` message this code emits ends "See `.agents/specs/qwen4-exp-flash-next.md` and issue #1981", so a reader who follows the pointer a running binary gives them has to find the issue at the other end of it; AGENTS.md requires the index, the spec and the PR body to agree, and until this row only the PR body carried it. **The row's product is a BOUNDARY, and the boundary is measured.** `Qwen4ExpForConditionalGeneration` has no reachable token gate (`gateable = no`, nothing published fits a fleet device), so no downstream gate will ever catch a wrong config default by running the model, and the config layer is the last place one is checkable. The config layer itself IS gateable even though the model is not: `transformers` 5.16.0 installs and imports without torch and runs `validate_architecture` in full, so W1 is gated by a 39-case two-direction sweep — each config put through `Qwen4ExpConfig.from_dict` on one side and `LoadHfConfig -> ModelRegistry::Resolve -> factory->parse_config` on the other. **35 agree; 4 differ, and all 4 are ours refusing what upstream accepts**, never the reverse. All 15 upstream `validate_architecture` rejections are implemented and tabulated against their upstream line in the spec's `## The refusal boundary`, with the local tighter guards listed beside them. Four defaults were wrong in the first draft and every one of them is invisible to a token gate: `partial_rotary_factor` was read from the text config with a hardcoded 0.25 on the belief that `Qwen4ExpTextConfig` inherits it from `Qwen3_5MoeTextConfig` — the generated class is `class Qwen4ExpTextConfig(PreTrainedConfig)`, declares no such field, and `0.25` does not occur in the file, so the port both accepted configs upstream refuses (rotary_dim 64 where upstream computes 256 and raises) and refused one upstream accepts; the four PLE n-gram fields defaulted to 0 rather than 3 / 8 / 20000000 / 128, refusing a legal config and carrying a zero-sized n-gram vocabulary into W2; `output_gate_type` did not fall back to `hidden_act`, and its local check was a constant false the shared reader had already made unreachable; and `ple_embed_dim <= 0` was dropped from upstream's condition, so `-2560` passed the divisibility test because `-2560 % 16 == 0` in C++. Also landed: `eos_token_id` is now required when PLE is enabled (it is a segment boundary in the hashed n-gram construction, and the published GGUF stores it as `qwen4exp.ple.eos_token_id`); the forward refuses BEFORE the `ModelAs` downcast, because nothing can produce a loaded Qwen4-Exp while the loader refuses and a downcast placed first made the advertised refusal unreachable; `block_topk()` and `head_dim_per_ngram()` refuse instead of SIGFPE on a legally-parsed config with QSA or PLE absent; and the model's local `TextOf` now resolves `llm_config` and `thinker_config.text_config` like the shared `ResolveTextConfig`, which it did not, so one parse no longer answers "what is the text config" two different ways | bug | diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 6f955c418..dbf883a65 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -77,11 +77,11 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): |---|---| | INVENTORIED | 324 | | PARTIAL | 22 | -| ACTIVE | 10 | +| ACTIVE | 11 | | SPIKE | 9 | | BLOCKED | 5 | | DONE | 3 | -| READY | 4 | +| READY | 3 | | GATING | 1 | | **Total** | **378** | @@ -89,7 +89,7 @@ Engaged architectures (the 54 non-`INVENTORIED` rows): | Support | Architecture | Family / example | Status | Row | |---|---|---|---|---| -| 🚧 | `Qwen4ExpForConditionalGeneration` | Qwen3.8-Flash-Next (180B total / 6B activated, image-text-to-text) | **SPEC ONLY.** New architecture vLLM does NOT implement at any revision; split oracle by developer direction 2026-08-26 (transformers for the algorithm, vLLM ops for the optimized path). Nothing published fits ~119 GB on GB10, so `gateable = no` and the quantized arms are load-bearing; no GGUF exists and llama.cpp has no `qwen4_exp` either. NO token, NO speed. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | +| 🚧 | `Qwen4ExpForConditionalGeneration` | Qwen3.8-Flash-Next (180B total / 6B activated, image-text-to-text) | **REGISTERED, NOT LOADABLE** (W1, [#1981](https://github.com/mudler/vllm.cpp/issues/1981)): the config resolves and VALIDATES against a running transformers 5.16.0 oracle (39-case two-direction refusal sweep, 35 agree, 4 deliberately tighter and tabulated); the loader, the forward and the KV-cache spec all refuse by name. New architecture vLLM does NOT implement at any revision; split oracle by developer direction 2026-08-26 (transformers for the algorithm, vLLM ops for the optimized path). Nothing published fits ~119 GB on GB10, so `gateable = no` and the quantized arms are load-bearing; no GGUF exists and llama.cpp has no `qwen4_exp` either. NO token, NO speed. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | | ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling scope CLOSED (#323): `60e71a0e` fixed the eager path; `DenseDecodeGraphForward` ran first and replayed against stale HOST ids, so it now declines while the mirror is live and falls back to the proven eager path. Async gate 7/7 across Qwen3-0.6B/4B + Llama/Mistral/InternLM2 | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` | | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` | | ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving. **SECOND CHECKPOINT TOKEN-GATED 2026-08-15 (`Qwen/Qwen3.8-27B`@`1d4bf0f2`, bf16, [#915](https://github.com/mudler/vllm.cpp/issues/915), [spec](specs/qwen38-27b-bf16-gate.md)): 4/7 prompts STRICT 16/16 vs the pinned oracle `555967922`, and all THREE first-divergence positions are EXACT fp32 TIES** — oracle-minus-ours and top-2 gap both **0.000 mnats**, our token at rank **3 / 2 / 2** in the oracle top-20, so `ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. Every one is the [#910](https://github.com/mudler/vllm.cpp/issues/910) tie-break signature and nothing else: vLLM's pick carries the LOWER token id (1814/11/16309) and ours the HIGHER (22960/13/27180) at a bit-identical logprob. Only the first divergence per prompt is adjudicable, so this is three numbers; a raw position count over the grid is NOT a quality score and is not recorded as one. Adjudicated twice on the pinned oracle's fp32 logprobs — a greedy re-decode and an independent TEACHER-FORCED probe that asserts the echoed prefix — because the earlier `transformers` bf16 CPU probe could not resolve below one bf16 ULP (every runner-up gap it printed was a multiple of 0.125) and so could not have reported anything but a tie. **SPEED on the same checkpoint, vs vLLM's PRODUCTION graphed config at the pin, clocks 2184 MHz: 1 of 3 concurrency cells established.** c4 is the only cell where both arms completed every request — **0.963x** output throughput, **1.008x** median ITL. c1 and c8 throughput WERE withheld on 2026-08-15 (superseded, below): our server failed 1/6 in all three reps and 12/11/12 of 48 where vLLM failed none in nine legs ([#931](https://github.com/mudler/vllm.cpp/issues/931)), and `output_throughput` divides tokens by a duration still containing the dead request, so c1 read 0.677x while median TPOT in the SAME file read 1.014x in our favour. **SUPERSEDED 2026-08-19 by the c1/c8 RE-MEASURE ([#915](https://github.com/mudler/vllm.cpp/issues/915), [#979](https://github.com/mudler/vllm.cpp/issues/979), `.agents/benchmark-record.md` `BENCH-QWEN38-27B-BF16 c1/c8 RE-MEASURE`):** #931 landed, and with `VT_SERVER_SSE_PING_S=0` our arm completed **162 of 162** requests, `failed=0` on every leg — c1 **4.4040 tok/s** (CV 0.039%), c8 **22.6402 tok/s** (CV 0.205%). **Our half of the withholding is discharged; NEITHER cell became a ratio and the two halves are blocked differently.** At c1 vLLM also completed everything (**4.2835 tok/s**, CV 0.033%) and `gpu_clock_state compare` returned `PAIRING_VERDICT=DISCARD` on all three pairings — the cross-arm rule PASSED (same boot, both arms 2489 MHz median, 0.0% offset) and the WITHIN-RUN rule failed on both against the 5% ceiling ([#1354](https://github.com/mudler/vllm.cpp/issues/1354): clocks cannot be pinned inside an `rc` lease), so the c1 ratio is OWED, not withheld for being unflattering. At c8 the vLLM denominator is **NOT MEASURABLE on this box at the recorded configuration** — that is the answer, not a gap, and not a claim that vLLM is defective. Read the two output-throughput absolutes with [#1355](https://github.com/mudler/vllm.cpp/issues/1355): our `usage.prompt_tokens` reports 5,942 where vLLM reports 6,144 on identical prompts, which corrupts total-token throughput outright and biases output throughput up by more than its own CV. Cold start **53 s vs 780 s = 14.7x**; host memory after warmup **42.5 vs 110.1 GiB = 2.59x**, caveated because vLLM's is set by `--gpu-memory-utilization 0.85` pre-reserving KV | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | @@ -510,7 +510,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-qwen3-vl-moe-qwen3-vlmoe-for-conditional-generation` | `Qwen3VLMoeForConditionalGeneration` | `registry.py:552-555`; `vllm/model_executor/models/qwen3_vl_moe.py::Qwen3VLMoeForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder; video path | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | `Qwen3_5ForConditionalGeneration` | `registry.py:556`; `vllm/model_executor/models/qwen3_5.py::Qwen3_5ForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; GDN/linear-attention state; vision encoder; video path | 🚧 [family scoping](specs/mm-tools-scoping-2026-07-10.md); [plain-BF16 loader leaf](specs/qwen35-plain-bf16-direct-load.md); **[multimodal-track W-plan](specs/multimodal-track.md)**; full target spec required | `PARTIAL` (text-only) | text-only: `include/vllm/model_executor/models/qwen3_5_dense.h:40-105,146-171`; plain BF16/F32 + stacked/tied load `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:52-133,187-246,334-472`; plain execution `src/vllm/model_executor/models/qwen3_5.cpp:1993-2003,4579-4585,5249-5255,5533-5545`; loader route/queue reuse `src/vllm/entrypoints/model_loader.cpp:364-400`; real 4B gate `tests/vllm/models/test_qwen35_plain_weights.cpp:80-196`: CPU topology/load **1656/1656**, AOT CUDA direct OFF/ON full-engine token equivalence **1664/1664**. Existing W3-G immutable `ae9e8ff` default/fallback each pass **235/235 + 16/16** with the frozen 64 plans. Corrected root `/tmp/qwen35-transplant-4b-aot-557ab41d` proves ON=OFF 128/128 and records ON/OFF/vLLM total **6155.10/6064.06/6730.46 tok/s**, peak PSS **2.405/8.571/7.569 GiB**; current ON is 0.9316x historical AOT ON. Current-v0.25 oracle, sanitizer, vision, strict VRAM and external 27B/35B regressions remain unverified, with no new support claim. **MM-completion plan ([multimodal-track.md](specs/multimodal-track.md), `CLAIM-MULTIMODAL-TRACK`, 2026-07-25):** modalities = image + video (NO audio); reuses the landed GDN-hybrid text path — the mm half is the shared `Qwen3_VisionTransformer` (DeepStack, `qwen3_vl.py:519`) stood up on Qwen3-VL-4B first (M2) then attached to this wrapper (M3). Oracle-runnable (0.25.0 ships `qwen3_5.py`+`qwen3_vl.py`); NOT HW/oracle-blocked but **CHECKPOINT-gated** — the cached `unsloth/Qwen3.6-27B-NVFP4` quant is TEXT-ONLY (2111 tensors, ZERO `visual.*`; `vision_config` declared but weights absent), so a vision-inclusive checkpoint download is required (M0). Tower ~0.5-0.7 B params (~1-1.4 GiB bf16) fits GB10 trivially alongside the 27B. Plan owner `CLAIM-MULTIMODAL-TRACK` (row stays PARTIAL/narrative-only; the mm work re-claims it at M3). **M3-b LANDED 2026-07-25 (`CLAIM-MULTIMODAL-M3B`): IMAGE e2e WORKING** — `Qwen3_5VLGenerateGreedy` (`src/vllm/model_executor/models/qwen3_5.cpp`) forks the GDN-hybrid forward on inputs_embeds(scatter tower merger `[196,5120]` into image_token 248056 rows, no deepstack) + 3-section MRoPE `[11,11,10]` interleaved in the 16 full-attn layers (host `BuildMropeCosSinHost` → the `mrope_cos_sin` param on `DenseForwardLayers`, nullptr on text ⇒ byte-identical); vision loader `LoadQwen3VLVisionWeights` (`src/vllm/model_executor/models/qwen3_vl.cpp`, 27B config) + M2a tower + `LoadQwen3_5Dense` bf16 LLM. STRICT gate `tests/vllm/multimodal/test_qwen3_5_vl_e2e.cpp` **32/32 token-exact vs vLLM 0.25.0** (sha256 `ead4b484…`); text-inertness re-run cutlass-ON 27B/35B/Coder **235/315/138**. **M3d LANDED 2026-07-25 (`CLAIM-MULTIMODAL-M3D`): VIDEO e2e WORKING** — `Qwen3_5VLGenerateGreedyVideo` (`src/vllm/model_executor/models/qwen3_5.cpp`) reuses the M3-b image driver via a shared `VLGenerateCoreGdn` (video merge mask on video_token 248057 + `Qwen3VLGetRopeIndexVideo` per-frame temporal MRoPE; M3c processor/windowed-tower reused verbatim; no deepstack). STRICT gate `tests/vllm/multimodal/test_qwen3_5_vl_video_e2e.cpp` **32/32 token-exact vs vLLM 0.25.0** (oracle `scripts/mm/m3d_video_oracle_capture.py`, K=5 deterministic, near-tie gaps 0.0000); image e2e re-run STRICT 32/32 (refactor-safe); text SACRED byte-identical by construction (shared forward untouched). **Qwen video modalities COMPLETE (image+video e2e; audio N/A); speed still pending** (row stays PARTIAL). Speed lever #2 CLOSED 2026-07-27 (`CLAIM-MULTIMODAL-SPEED-DECODE`, multimodal-speed.md §8): on-GPU greedy argmax + decode embed round-trip removed on the shared `VLGenerateCoreGdn`; bit-exact (image+video STRICT 32/32 held, goldens md5-identical); 27B decode TPOT NEUTRAL (223 ms, ~222 ms bandwidth floor, at vLLM parity). **Speed lever #3 FIRST BRICK 2026-07-27 (`CLAIM-MULTIMODAL-SPEED-GRAPH`, multimodal-speed.md §9): the shared `VLGenerateCoreGdn` decode step now routes through the production `Qwen3_5DenseDecodeGraph` (cold→warm→replay captured decode) — mm decode is GRAPH-CAPTURABLE (was eager per-step). S==B==1 bit-identical rebuild; the decode-time 1-D device RoPE at p reproduces the degenerate MRoPE {p,p,p} → token-exact HELD (image+video STRICT 32/32, 30 graph replays confirmed); A/B graphed 232.5 vs eager 233.4 ms/tok = NEUTRAL at the 27B bandwidth floor. Structural gap closed; W-plan = Voxtral decode-graph (audio 1.52× gap-closer) + batched c2+ + serving ingestion. Row stays PARTIAL/speed-pending.** | unassigned | | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` | `Qwen3_5MoeForConditionalGeneration` | `registry.py:557-560`; `vllm/model_executor/models/qwen3_5.py::Qwen3_5MoeForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; GDN/linear-attention state; vision encoder; video path | 🚧 [family scoping](specs/mm-tools-scoping-2026-07-10.md); **[multimodal-track W-plan](specs/multimodal-track.md)**; target spec required | `PARTIAL` (text gated, vision NOT gated) | text-only: `include/vllm/model_executor/models/qwen3_5.h:1-17,97`; direct registry `src/vllm/model_executor/models/registry.cpp:10-20`; gate `tests/parity/test_qwen36_paged_engine.cpp:78,140`. W3-G immutable `ae9e8ff` correctness-only ratio-8 inertness passes **2/2 + 315/315**; no 35B performance claim; vision not implemented AT THAT DATE (it landed later, see M2/M3 below, and is still NOT gated). Disk load now DEFERS the routed-expert host copies and streams+frees them per layer during `PrepareMarlinResident` to bound load-phase peak PSS (`ENG-MOE-LOADSTREAM`, engine-matrix; CPU-gated, DGX pending) — device residents byte-identical. **MM-completion plan ([multimodal-track.md](specs/multimodal-track.md), `CLAIM-MULTIMODAL-TRACK`, 2026-07-25):** image + video (NO audio); same shared `Qwen3_VisionTransformer` as the 27B row, attached to the landed MoE GDN-hybrid text path (M3). Oracle-runnable (0.25.0); **CHECKPOINT-gated** — the cached `nvidia/Qwen3.6-35B-A3B-NVFP4` quant is TEXT-ONLY (`vision_config` declared, `visual.*` weights absent); vision-inclusive download required (M0). Tower fits GB10 alongside the 35B MoE per the landed text run. Plan owner `CLAIM-MULTIMODAL-TRACK` (row stays PARTIAL/narrative-only; the mm work re-claims it at M3). **M2/M3 LANDED (#891, `.agents/specs/moe-vision-tower.md`):** the loader no longer drops the checkpoint's 333 `model.visual.*` tensors (`LoadQwen3_5MoeVision` -> the SHARED `LoadQwen3VLVisionWeights` the dense arm is gated on; their ABSENCE is refused by name), and `Qwen3_5MoeVLGenerateGreedy[Video]` forks the forward gated on mm input over a greedy core now TEMPLATED on the weights arm rather than copied. Evidence: CPU suite 479/479 serial; the new `test_qwen3_5_moe_vision` proves the forked forward reduces EXACTLY to the text forward over the tower row (one visual token, 1x1x1 LLM grid) and that MRoPE is applied (8x8 grid must DIFFER from the 1-D run), with 4 mutations driven RED and restored byte-exact; on Thor (sm_110, FALLBACK attention) `test_qwen3_5_moe_vision_hw` loads the real 333 tensors and runs the tower on the fixture image. **OWED: the binding image and video token-exact gates vs the pinned oracle at 35B.** Not runnable on Thor -- vLLM cannot import there (`libcuda.so.1` absent on the host, `torch.cuda.is_available()` False) and the bf16 35B is ~67 GiB against this box's documented 25 GB single-model reboot ceiling; dgx.casa was off-limits mid-run for a sibling row. **TEXT ARM ORACLE-GATED ON THE PUBLISHED BF16 REPO 2026-08-15 ([#740](https://github.com/mudler/vllm.cpp/issues/740) + [#864](https://github.com/mudler/vllm.cpp/issues/864)), and this changes NOTHING about the vision claim:** greedy 7 prompts x 3 repeats x 16 tokens on `Qwen/Qwen3.6-35B-A3B` bf16 @`995ad96eacd98c81ed38be0c5b274b04031597b0` vs the pinned oracle gave **6/7 prompts STRICT 16/16**, the seventh one exact logit tie (`top2_gap_mnats = 0.0`) our on-device argmax breaks toward the higher id ([#910](https://github.com/mudler/vllm.cpp/issues/910)); only the FIRST divergence per prompt is adjudicable, so the raw 108/112 position count is NOT a quality score. SACRED inertness 3/3, goldens byte-identical (27B 235/235, 35B 315/315, Coder 138/138). NO throughput, latency or memory number exists for this checkpoint. The row therefore stays PARTIAL: **the binding image and video token-exact gates at 35B are still OWED**, the vision claim remains "the tower loads and computes" rather than "produces correct tokens", and the sm_110 run that proved it used the FALLBACK attention path, which is not coverage of the shipped GB10 path. [#908](https://github.com/mudler/vllm.cpp/issues/908)'s dense regression check is PARTIAL too: dense TEXT is 235/235 at `2f2bce926`, a true before/after (binary md5 `db889909d4…` vs `49ded1ece8…`, 500 TUs recompiled), while dense image/video stays UNVERIFIED (network-blocked) | unassigned | -| `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | `Qwen4ExpForConditionalGeneration` (`model_type: qwen4_exp`; campaign row `MODEL-MM-QWEN4-EXP`) | **NOT IN vLLM AT ANY REVISION** — deliberately written with no pinned module/class target, the convention `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` follows for a beyond-pin arm, and stronger here: this is absence from vLLM `main` rather than staleness in `555967922`. Read live 2026-08-26 at `origin/main` = `6a5e8f5979`: no `qwen4*` path, no `registry.py` entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. Algorithm source is [transformers#48337](https://github.com/huggingface/transformers/pull/48337) `models/qwen4_exp/modular_qwen4_exp.py`, MERGED 2026-08-26 | conditional generation / text + image + video | MM processor; vision encoder (UNCHANGED from `Qwen3_5MoeVisionModel`); GDN/linear-attention state; block-sparse attention + indexer side cache; FusedMoE/grouped GEMM; hyper-connection residual streams; hashed n-gram embedding; dilated depthwise conv; MTP | ✅ [Qwen3.8-Flash-Next](specs/qwen4-exp-flash-next.md) | `READY` | **SPEC ONLY, NO PRODUCT CODE, NO TOKEN, NO SPEED.** `Qwen/Qwen3.8-Flash-Next` (2026-08-24, 180B total / 6B activated). Split oracle by developer direction 2026-08-26: **transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH**, because `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so the reference is semantics and not a serving path. `Qwen4ExpTextModel` inherits `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`); GDN matches our AOT gate exactly (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). **Exactly two components have NO vLLM op**: the PLE dilated depthwise conv (`git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the hashed n-gram embedding. **CORRECTED after the component deep-dives: QSA's twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The first reading rested on `MLAAttentionSpec` being an MLA claim; it is a per-state BUDGET shape, and M3 — itself plain GQA — uses it. Nine structural matches with DSv4 including `compress_ratio == 4`, relu-summed MQA scoring, RoPE at the block-start position and `tokens_per_state=compress_ratio`. M3 scores by `max` over 128 RAW dots, emits one set per KV head, and welds `SPARSE_BLOCK_SIZE = 128` to the KV page size. The genuinely NEW work is the consumer: DSv4 attends COMPRESSED KV, M3 attends raw at page granularity, QSA attends RAW at ratio-4 granularity and nothing upstream does that. Any QSA gate must exceed 2048 tokens of context or it cannot distinguish a correct port from one attending pooled keys. Residual stream is `hc_count * hidden_size` = 4 x 2560 = **10240 wide through the whole stack**; `number_of_conv_states = 3` on a PLE layer plus the indexer side cache. **NOTHING PUBLISHED FITS ~119 GB on GB10**: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB, `unsloth/...-GGUF` is a README with ZERO weight files and llama.cpp has no `qwen4_exp` either, so the standing k-quant arms must be authored here and have NO llama.cpp oracle. Sizing ARITHMETIC and not measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB, Q4_K_M backbone with the 51 GB n-gram table non-resident ~76 GB — that table is 28% of the model and is touched 16 times per token, which is the offload the card itself argues for. `gateable = no` until an arm runs, and **the chosen arm does not load today**: `KeepQuantKDim` returns `-1` for `kEmbeddingTable`, so a quantized n-gram table expands to bf16 (102.4 GB) and dies at load; `kKeepF16` is the only non-expanding gather residency and is CPU-only. Also `moe_intermediate_size = 640` / `hc_lowrank = 320` are Q4_K-illegal on their reduction dims and our reader cannot open ggml types 3/6/7/20, so the converter must emit Q4_0. `ENG-WEIGHT-OFFLOAD` is inert and does not help; the shipping mmap-borrow tier does. Both blocking decisions SETTLED 2026-08-26: the transformers lane pin is **ACCEPTED at 5.16.0** ([`oracles/transformers.md`](oracles/transformers.md)), because the registry pin 5.14.1 does not contain `Qwen4Exp` — and 5.16.0 is a real release rather than a SHA, bounded by fetching the model file at each tag (`v5.16.0` HTTP 200, `v5.15.0` HTTP 404), with the version string UNMEASURED until an oracle stands up; and the first runnable arm is the **Q4_K_M backbone with a NON-RESIDENT n-gram table** (~76 GB), Q8_0 having been raised and rejected on ~191 GB. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | unassigned | +| `MODEL-MM-qwen4-exp-qwen4-exp-for-conditional-generation` | `Qwen4ExpForConditionalGeneration` (`model_type: qwen4_exp`; campaign row `MODEL-MM-QWEN4-EXP`) | **NOT IN vLLM AT ANY REVISION** — deliberately written with no pinned module/class target, the convention `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` follows for a beyond-pin arm, and stronger here: this is absence from vLLM `main` rather than staleness in `555967922`. Read live 2026-08-26 at `origin/main` = `6a5e8f5979`: no `qwen4*` path, no `registry.py` entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. Algorithm source is [transformers#48337](https://github.com/huggingface/transformers/pull/48337) `models/qwen4_exp/modular_qwen4_exp.py`, MERGED 2026-08-26 | conditional generation / text + image + video | MM processor; vision encoder (UNCHANGED from `Qwen3_5MoeVisionModel`); GDN/linear-attention state; block-sparse attention + indexer side cache; FusedMoE/grouped GEMM; hyper-connection residual streams; hashed n-gram embedding; dilated depthwise conv; MTP | ✅ [Qwen3.8-Flash-Next](specs/qwen4-exp-flash-next.md) | `ACTIVE` | **W1 LANDED: the config layer resolves, validates and REGISTERS; nothing loads and nothing forwards. NO TOKEN, NO SPEED.** The refusal boundary is measured rather than described: a 39-case two-direction sweep against a RUNNING `transformers` 5.16.0 (installable and importable without torch, which is what makes the config layer gateable while the model is not) agrees on 35 and differs on 4, each a deliberately tighter local guard tabulated in the spec's `## The refusal boundary`. W2 owes the hashed n-gram embedding and the PLE dilated depthwise conv, W3 the hyper-connection stream, W4 QSA and its indexer side cache, W5 the loader/forward/vision/MTP, W6 the GGUF arm. `Qwen/Qwen3.8-Flash-Next` (2026-08-24, 180B total / 6B activated). Split oracle by developer direction 2026-08-26: **transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH**, because `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so the reference is semantics and not a serving path. `Qwen4ExpTextModel` inherits `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`); GDN matches our AOT gate exactly (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). **Exactly two components have NO vLLM op**: the PLE dilated depthwise conv (`git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the hashed n-gram embedding. **CORRECTED after the component deep-dives: QSA's twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The first reading rested on `MLAAttentionSpec` being an MLA claim; it is a per-state BUDGET shape, and M3 — itself plain GQA — uses it. Nine structural matches with DSv4 including `compress_ratio == 4`, relu-summed MQA scoring, RoPE at the block-start position and `tokens_per_state=compress_ratio`. M3 scores by `max` over 128 RAW dots, emits one set per KV head, and welds `SPARSE_BLOCK_SIZE = 128` to the KV page size. The genuinely NEW work is the consumer: DSv4 attends COMPRESSED KV, M3 attends raw at page granularity, QSA attends RAW at ratio-4 granularity and nothing upstream does that. Any QSA gate must exceed 2048 tokens of context or it cannot distinguish a correct port from one attending pooled keys. Residual stream is `hc_count * hidden_size` = 4 x 2560 = **10240 wide through the whole stack**; `number_of_conv_states = 3` on a PLE layer plus the indexer side cache. **NOTHING PUBLISHED FITS ~119 GB on GB10**: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB, `unsloth/...-GGUF` is a README with ZERO weight files and llama.cpp has no `qwen4_exp` either, so the standing k-quant arms must be authored here and have NO llama.cpp oracle. Sizing ARITHMETIC and not measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB, Q4_K_M backbone with the 51 GB n-gram table non-resident ~76 GB — that table is 28% of the model and is touched 16 times per token, which is the offload the card itself argues for. `gateable = no` until an arm runs, and **the chosen arm does not load today**: `KeepQuantKDim` returns `-1` for `kEmbeddingTable`, so a quantized n-gram table expands to bf16 (102.4 GB) and dies at load; `kKeepF16` is the only non-expanding gather residency and is CPU-only. Also `moe_intermediate_size = 640` / `hc_lowrank = 320` are Q4_K-illegal on their reduction dims and our reader cannot open ggml types 3/6/7/20, so the converter must emit Q4_0. `ENG-WEIGHT-OFFLOAD` is inert and does not help; the shipping mmap-borrow tier does. Both blocking decisions SETTLED 2026-08-26: the transformers lane pin is **ACCEPTED at 5.16.0** ([`oracles/transformers.md`](oracles/transformers.md)), because the registry pin 5.14.1 does not contain `Qwen4Exp` — and 5.16.0 is a real release rather than a SHA, bounded by fetching the model file at each tag (`v5.16.0` HTTP 200, `v5.15.0` HTTP 404), with the version string UNMEASURED until an oracle stands up; and the first runnable arm is the **Q4_K_M backbone with a NON-RESIDENT n-gram table** (~76 GB), Q8_0 having been raised and rejected on ~191 GB. [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `CLAIM-MODEL-MM-QWEN4-EXP-W1` | | `MODEL-MM-rvl-rfor-conditional-generation` | `RForConditionalGeneration` | `registry.py:561`; `vllm/model_executor/models/rvl.py::RForConditionalGeneration` | conditional generation / video+image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-skyworkr1v-skywork-r1-vchat-model` | `SkyworkR1VChatModel` | `registry.py:562`; `vllm/model_executor/models/skyworkr1v.py::SkyworkR1VChatModel` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-smolvlm-smol-vlmfor-conditional-generation` | `SmolVLMForConditionalGeneration` | `registry.py:563`; `vllm/model_executor/models/smolvlm.py::SmolVLMForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index 636425c66..cd9c1bc4d 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -736,10 +736,120 @@ change that makes any arm reachable, not later. - No arm is made to fit any fleet device: the row holds with G0 passed and G1-G3 `PENDING` on hardware, recorded as visible debt, and no token claim is made. +## The refusal boundary + +W1's whole product is a boundary: which configs this port accepts and which it +refuses. This row has **no reachable token gate** (`## Gates`, `gateable = no`), so +nothing downstream will ever catch a wrong default by running the model — a +`partial_rotary_factor` read from the wrong place, an n-gram field defaulted to +zero, or a missing `eos_token_id` all produce a config that parses, resolves, and +is silently wrong for W2 and W4. The config layer is the last place any of it is +checkable, so the boundary is **measured** here rather than described. + +**The oracle runs.** `transformers` 5.16.0 installs and imports without torch — +it says so itself ("only tokenizers, configuration and file/data utilities can be +used") — and `Qwen4ExpConfig.from_dict()` runs `Qwen4ExpTextConfig.__post_init__` +and `validate_architecture` in full. That makes the CONFIG layer of this row +gateable even though the MODEL layer is not, and it is the only layer of this row +that is. `gateable = no` in `oracles/transformers.md` still stands: it is a +statement about running the model, and nothing here runs one. + +### Two-direction sweep + +39 configs, each derived from the committed fixture, put through +`Qwen4ExpConfig.from_dict` on one side and `LoadHfConfig -> ModelRegistry::Resolve +-> factory->parse_config -> ParseQwen4ExpParams` on the other. **35 agree; 4 +differ, and all 4 are ours refusing what upstream accepts** — never the reverse, +which is the direction that would let a bad checkpoint through. + +Reproduce (transformers 5.16.0 in a venv; the probe links `build/libvllm.a` with +`-Wl,--whole-archive` so the model's self-registration survives): + +| upstream verdict | ours | cases | +|---|---|---| +| ACCEPT | ACCEPT | baseline; `prf` top 1.0 / rope .25; `prf` top .25 / rope absent; `eos_token_id` null with PLE OFF; every n-gram default omitted; no `output_gate_type` with `hidden_act` silu; all five indexer keys erased; `layer_types` erased (interval synthesis) | +| REFUSE | REFUSE | `prf` absent everywhere; `prf` only in rope 1.0; `prf` top .25 / rope 1.0; `eos_token_id` null with PLE ON; `eos_token_id` `[]`; no `output_gate_type` with `hidden_act` gelu; `output_gate_type` swish; `output_gate_type` gelu; `ple_embed_dim` -2560; `ple_embed_dim` 2561; `hc_count` 1; `num_experts` 0; `num_experts_per_tok` 513; `moe_intermediate_size` 0; partial QSA group; `indexer_n_heads` 0; `indexer_kv_heads` 2; `indexer_budget` 2049; `sliding_attention`; `ple_layer_ids` [0]/[4]/[49]; `ngram_size` 1; `heads_per_ngram` 0; interval 0; short `layer_types`; `num_hidden_layers` 0 | +| ACCEPT | **REFUSE** | `hc_lowrank` 0; `ple_conv_kernel_size` 0; `mtp_num_hidden_layers` -1; `partial_rotary_factor` -0.25 | + +### Each upstream rejection, and the line that implements it + +`configuration_qwen4_exp.py` at `v5.16.0`; local lines in +`src/vllm/model_executor/models/qwen4_exp.cpp` unless stated. + +| # | upstream | our implementation | exercised by | +|---|---|---|---| +| 1 | `:190-192` unsupported `layer_types` | `KindFromString` | "an unsupported layer type" | +| 2 | `:193-195` `output_gate_type or hidden_act` not in {sigmoid, silu} | the raw-text gate resolution, NOT `config.output_gate_type` | "[UP] an absent output_gate_type falls back to hidden_act", "[UP] an explicit output_gate_type outside {sigmoid, silu}" | +| 3 | `:196-197` `hc_count <= 1` | the `hc_count` refusal | "hc_count must exceed 1" | +| 4 | `:198-199` `num_experts <= 0` | the `num_experts` refusal | "[UP] num_experts must be positive" | +| 5 | `:200-204` `num_experts_per_tok` outside [1, num_experts] | the `num_experts_per_tok` refusal | "num_experts_per_tok above num_experts" | +| 6 | `:205-206` MoE intermediate sizes | the MoE-size refusal | "[UP] the MoE intermediate sizes must be positive" | +| 7 | `:216-218` partial QSA group | the `present != 5` refusal, naming the missing fields | "a partial QSA group names what is missing" | +| 8 | `:219-220` QSA values not positive | the QSA positivity refusal | "[UP] QSA values must be positive" | +| 9 | `:221-222` `indexer_kv_heads != 1` | the `kv_heads` refusal | "QSA requires exactly one indexer kv head" | +| 10 | `:223-224` `indexer_budget % indexer_compress_ratio` | the divisibility refusal | "the indexer budget must divide by the compress ratio" | +| 11 | `:225-231` `rotary_dim > indexer_head_dim` | the refusal, over `config.rotary_dim` from the SHARED reader | "absent everywhere: 1.0, rotary_dim 256, and upstream REFUSES", "top-level 0.25 does NOT rescue a rope dict that says 1.0" | +| 12 | `:235-239` `ngram_heads <= 0 or ple_embed_dim <= 0 or ple_embed_dim % ngram_heads` | split three ways so the message names the field: `ngram_size < 2`, `heads_per_ngram <= 0`, then `heads <= 0 \|\| embed_dim <= 0 \|\| embed_dim % heads` | "[LOCAL] ngram_size below 2", "[LOCAL] heads_per_ngram must be positive", "[UP] a NEGATIVE ple_embed_dim", "[UP] a ple_embed_dim that does not divide by the head count" | +| 13 | `:240-247` `ple_layer_ids` outside [1, num_hidden_layers] | the one-indexed range refusal | "a PLE id outside the one-indexed range" | +| 14 | `:248-255` PLE on a non-`linear_attention` layer | the layer-kind refusal | "a PLE id on a sparse-attention layer" | +| 15 | `:256-257` `eos_token_id` unset with PLE enabled | the `eos_token_id` refusal | "[UP] eos_token_id must be set when PLE is enabled", "[UP] an EMPTY eos_token_id list is refused too" | + +`__post_init__` behaviors, which are not rejections but decide what the rejections +see: `full_attention -> qwen_sparse_attention` (`:180-184`), the interval synthesis +(`:174-179`), `ple_embed_dim` defaulting to `hidden_size` (`:168`), +`sorted(set(ple_layer_ids))` (`:167`), and `number_of_conv_states` (`:172`). Each +has its own case. + +**Upstream's ORDER inside the PLE block is mirrored**, and deliberately: head count +and embedding width first, then the id range, then the layer kind, then EOS. A +config violating two at once has to report the one upstream reports, or a reader +comparing the two runtimes is sent to a different field. + +### Refusals we impose that upstream does not + +Each is deliberate, each is exercised, and each is a row in the sweep above. None +of them lets a config through that upstream refuses. + +| ours | upstream | why we keep it | +|---|---|---| +| `num_hidden_layers <= 0` | none | a zero-layer stack is unrepresentable downstream; upstream refuses the same fixture for a different reason (the PLE id range collapses to [1, 0]) | +| `layer_types` length vs `num_hidden_layers` | none | upstream indexes `layer_types[layer_id - 1]` and would `IndexError`; in C++ that is an out-of-bounds read | +| `full_attention_interval <= 0` | none | `(i + 1) % 0` is UB in C++ where Python raises `ZeroDivisionError` | +| `hc_lowrank <= 0` | none | a non-positive rank cannot size the hyper-connection mixer W3 builds | +| `ple_conv_kernel_size <= 0` | none | `short_conv_state_len()` goes negative and W2 sizes a conv state from it | +| `mtp_num_hidden_layers < 0` | none (not even a declared field of `Qwen4ExpTextConfig`) | a negative depth cannot be built | +| `ngram_size < 2` / `heads_per_ngram <= 0` | folded into `ngram_heads <= 0` | same accept/reject boundary, a message that names the field | +| non-integer / non-array JSON where a number or list belongs | Python coerces or raises later | a typed reader has to refuse at the boundary | +| `partial_rotary_factor` outside (0, 1] | none | **belongs to the SHARED reader**, `hf_config.cpp`, not to this model. It fires before this parse runs, which is why there is no local guard: one would be unreachable. Recorded here because the sweep sees it as ours | + +### What the config layer still cannot see + +`Qwen4ExpParams` resolves the fields W1 through W5 consume. It does NOT yet carry +`linear_num_key_heads` (16), `linear_num_value_heads` (**48**, against upstream's +declared default of 32), `linear_key_head_dim`, `linear_value_head_dim`, +`linear_conv_kernel_dim`, `norm_topk_prob`, `max_position_embeddings` or the +resolved `output_gate_type` value. The shared reader types most of them, so +nothing is lost — but a wave titled "config resolution" owes the statement, and it +is listed under `## Owed`. + ## Owed -- [#1978](https://github.com/mudler/vllm.cpp/issues/1978): this port. No product - code lands under the spec pull request. +- [#1978](https://github.com/mudler/vllm.cpp/issues/1978): this port, the campaign + row. W0 landed the spec with no product code. +- [#1981](https://github.com/mudler/vllm.cpp/issues/1981): **W1**, the config + surface — resolution, validation, registration, refuse-by-name on everything + else. LANDED. Recorded here because every `Refuse()` message this code emits + ends "See `.agents/specs/qwen4-exp-flash-next.md` and issue #1981", and a reader + who follows that pointer has to find the issue at the other end of it. +- **`Qwen4ExpParams` resolves 60% of the config.** `linear_num_key_heads`, + `linear_num_value_heads` (48 in the checkpoint, against upstream's declared + default of 32 — a difference W2 must not inherit from the docstring), + `linear_key_head_dim`, `linear_value_head_dim`, `linear_conv_kernel_dim`, + `norm_topk_prob`, `max_position_embeddings` and the resolved `output_gate_type` + are read by the shared `HfConfig` and dropped by this struct. Nothing is lost + yet; W2/W3 owe carrying the ones they consume. +- **A model-layer oracle.** The config layer is gateable and now gated + (`## The refusal boundary`); nothing above it is. `gateable = no` stands. - GGUF k-quant arms, including authoring the `qwen4_exp` architecture on our side, and the statement that no llama.cpp oracle exists for them. - MTP depth > 1. @@ -747,13 +857,31 @@ change that makes any arm reachable, not later. - The non-resident n-gram table on CUDA: the dequantizing gather op and the `kEmbeddingTable` keep-quant policy change (Route B), and a measurement of the page-cache cost that the <= 64 KiB/token arithmetic only bounds. -- ~~llama.cpp's ragged-K substitution~~ **RESOLVED**: `Q4_K -> Q5_0`, `IQ4_XS -> IQ4_NL`, - read from `tensor_type_fallback` in `src/llama-quant.cpp`. Both are reachable for this +- ~~llama.cpp's ragged-K substitution~~ **RESOLVED, AND NOW READ AT THE PIN**: + `Q4_K -> Q5_0`, `IQ4_XS -> IQ4_NL`, from `tensor_type_fallback` in + `src/llama-quant.cpp:374-406` of the `llama-cpp` oracle at its recorded revision + `10bf611e533d81f739128304991c5e133c6aebd8` (`b10451`, + [`../oracles/llama-cpp.md`](../oracles/llama-cpp.md)) — not at `master`, which is + where the claim was first read and which is not an oracle. The complete table at + that revision: `IQ1_S`/`IQ1_M`/`IQ2_XXS`/`IQ2_XS`/`IQ2_S`/`IQ3_XXS`/`IQ3_S`/`IQ4_XS + -> IQ4_NL`; `Q2_0`/`Q2_K`/`Q3_K`/`TQ1_0`/`TQ2_0 -> Q4_0`; `Q4_K -> Q5_0`; + `Q5_K -> Q5_1`; `Q6_K -> Q8_0`; anything else throws. Both are reachable for this model depending on the recipe, and our reader supports NEITHER (no `case 6`, no `case 20`), so W6 owes both. - **A published GGUF now EXISTS**, which supersedes this spec's "no GGUF exists and no tool can produce one": `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S, 67.56 GiB of - weights in 3 shards, `general.architecture = qwen4exp`, 1224 tensors. It FITS GB10 + weights in 3 shards, `general.architecture = qwen4exp`, 1224 tensors. **PINNED**, and + it needed to be — the repo's `lastModified` moved to `2026-08-26T15:54:43Z`, after + W1's pull request was opened, which is exactly the re-quantize-in-place case AGENTS.md + "Say which weights, and from where" names. Revision + `8bdc666649440e9bdc97e16f3f75782c98478ff5`; at that revision, shard sizes + 10,946,624 + 49,990,818,368 + 22,544,696,352 = **72,546,461,344 bytes = 67.564 GiB**, + with sha256 `88a1420825a9304063e882ada29d438263617f51ac8923d438d927496693bafd`, + `3a62e35bbf9add4733bd1438ebd3a67649d5edd6cb0e72bb78e33c913992b2b6` and + `0e25ceaeb89b8a80aa973c6c0c7448943682f7408c2855b2ebd016b7643a861a`. Those digests are + the Hub API's `lfs.oid` values and are NOT locally computed; W6 owes a local sha256 + when it stages the file. The "1224 tensors" count remains UNVERIFIED: shard 1 is the + metadata shard and reports `n_tensors = 0`. It FITS GB10 with ~52 GiB of headroom, and two things in OUR tree stop us loading it: the missing IQ4_NL reader arm, and the gather-table expansion. Its metadata independently confirms this spec's n-gram derivation to the digit -- @@ -770,14 +898,28 @@ change that makes any arm reachable, not later. ## Now -`READY`. Spec committed, no implementation. +`ACTIVE`. **W1 landed** ([#1981](https://github.com/mudler/vllm.cpp/issues/1981)): +the config resolves, validates and the architecture is registered. Nothing loads, +nothing forwards, and there is no KV-cache spec — all three refuse by name and name +the wave that owes them. `docs/FEATURES.md` says REGISTERED, NOT LOADABLE and the +model-matrix row says the same; they are one statement in two projections. + +W1 is gated by the boundary in `## The refusal boundary` rather than by a fixture +round-trip, because this row has no reachable token gate and the config layer is the +last place a wrong default is checkable. G0 item 6 ("every rejection in +`validate_architecture`") is **met** for the config layer: all 15 upstream rejections +are implemented, each is tabulated against its upstream line, each has a case, and a +39-case two-direction sweep against a running `transformers` 5.16.0 agrees on 35 with +4 deliberately tighter local guards, all in the safe direction. Both decisions this spec was blocked on are **settled** (developer, 2026-08-26) and recorded in place rather than left as proposals: the transformers lane pin is ACCEPTED at 5.16.0 (`## Oracles`), and the first runnable arm is the Q4_K_M backbone with a non-resident n-gram table (`## Hardware`). -Next actions, in order: W0 lands this spec; W1 through W3 are reachable today -against the lane pin with tiny random configs and need neither a checkpoint nor a -GPU lease; W6b's mechanism is the unknown that decides whether the chosen arm is -schedulable, and it should be spiked before W6 is planned. +Next actions, in order: W2 (hashed n-gram embedding + PLE dilated depthwise conv) and +W3 (hyper-connection residual stream) are both reachable today against the lane pin +with tiny random configs and need neither a checkpoint nor a GPU lease — and both +inherit a config layer whose boundary is measured, so a golden that disagrees is a +port defect and not a config question. W6b's mechanism is the unknown that decides +whether the chosen arm is schedulable, and it should be spiked before W6 is planned. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 01372fd32..0e81eb586 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -123,7 +123,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. |---|---|---|---| | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B NVFP4 (`unsloth` @`890bdef7`, `nvidia` @`0893e160`); Qwen3.5-4B BF16; **Qwen3.8-27B BF16** @`1d4bf0f2` | 27B strict 235/235 text + 32/32 image/video; 4B cached 3/3; Qwen3.8-27B 4/7 strict, 3 exact fp32 ties in band (#915) | `unsloth` 27B at/above vLLM, ModelOpt 0.85x; 4B 1.021x; 3.8-27B c4 **0.963x**, c1/c8 absolutes (#915). Loads BF16/per-tensor FP8/NVFP4 (CT+ModelOpt); `modelopt_mixed` FP8 tower NATIVE (#164), GDN qkvz merged. CUDA/CPU | | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (NVFP4 text; published BF16 text + vision tower) | NVFP4 strict 315/315 vs vLLM 0.25.0; published BF16 6/7 prompts strict 16/16 vs the pin, 7th an exact tie (#910). Image/video IMPLEMENTED, NOT GATED (#891): the tower loads and runs, mm gate OWED | gate model: 0.93x to 1.03x grid; NO BF16 or mm speed claim | -| `Qwen4ExpForConditionalGeneration` | none — **REGISTERED, NOT LOADABLE** (W1, [#1981](https://github.com/mudler/vllm.cpp/issues/1981)) | **NO GATE, and none is reachable yet.** The config resolves and validates; the loader, the forward and the KV-cache spec all REFUSE BY NAME, each naming the wave that owes it. vLLM implements `qwen4_exp` at NO revision, so the algorithm oracle is transformers **5.16.0** under an accepted lane exception; `gateable = no` because nothing published fits a fleet device — `Qwen/Qwen3.8-Flash-Next` is ~360 GB bf16, ~180 GB FP8, ~128 GB NVFP4 against ~119.6 GiB usable on GB10 | none, and no speed claim is admissible from this row until a token gate exists | +| `Qwen4ExpForConditionalGeneration` | none — **REGISTERED, NOT LOADABLE** (W1, [#1981](https://github.com/mudler/vllm.cpp/issues/1981)) | **CONFIG LAYER GATED; nothing above it is.** The config resolves and validates against a RUNNING transformers 5.16.0 oracle (it imports without torch, so `validate_architecture` executes): a 39-case two-direction sweep agrees on 35 and differs on 4, all four being local guards stricter than upstream, never looser. All 15 upstream `validate_architecture` rejections are implemented and tabulated against their upstream line. The loader, the forward and the KV-cache spec all REFUSE BY NAME, each naming the wave that owes it. vLLM implements `qwen4_exp` at NO revision, so the algorithm oracle is transformers **5.16.0** under an accepted lane exception; `gateable = no` because nothing published fits a fleet device — `Qwen/Qwen3.8-Flash-Next` is ~360 GB bf16, ~180 GB FP8, ~128 GB NVFP4 against ~119.6 GiB usable on GB10 | none, and no speed claim is admissible from this row until a token gate exists | | `Qwen3_5ForCausalLM`, `Qwen3_5MoeForCausalLM` | none: no text-only Qwen3.5 checkpoint fits this hardware | **NO RUN GATE, OWED.** Gated on `test_qwen3_8_text_only.cpp`; NO token claim. Loader reads stacked BF16 experts (#740) plus BF16 towers, shared expert and `lm_head` (#864), so both published indices satisfy the load plan | not measured | | `Qwen3ForCausalLM` | Qwen3 dense 0.6B/1.7B/4B/32B, NVFP4A16 | near-tie strict 16/16 vs vLLM 0.25.0 | c1 every-axis parity, c8 decode residual | | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B | strict 6/6 vs vLLM 0.25.0 | 11/16 grid cells at or above graphed vLLM | diff --git a/src/vllm/model_executor/models/qwen4_exp.cpp b/src/vllm/model_executor/models/qwen4_exp.cpp index c87d159d9..a9c2ddaae 100644 --- a/src/vllm/model_executor/models/qwen4_exp.cpp +++ b/src/vllm/model_executor/models/qwen4_exp.cpp @@ -26,10 +26,26 @@ constexpr const char* kSlug = "qwen4_exp"; // The model's own sub-object. Upstream nests everything except the wrapper's // `architectures`/`model_type`/vision block under `text_config`; a flat config // (what a GGUF-derived or hand-written config looks like) is accepted by -// falling back to the top level, exactly as the shared HfConfig reader does. +// falling back to the top level. +// +// The alternatives BELOW `text_config` are not decoration. This has to resolve +// to the same object `HfConfig`'s own `ResolveTextConfig` picked +// (hf_config.cpp:113-121), or one parse answers "what is the text config?" +// twice. A config nested under `llm_config` resolved `hidden_size`, +// `num_hidden_layers` and `layer_types` through the shared reader while this +// function fell back to the WRAPPER and found no `hc_*`, QSA, PLE, MTP or +// interval key at all -- accepted, silently half-parsed, and reporting one conv +// state on a model that needs three. const nlohmann::json& TextOf(const nlohmann::json& raw) { auto it = raw.find("text_config"); if (it != raw.end() && it->is_object()) return *it; + it = raw.find("llm_config"); + if (it != raw.end() && it->is_object()) return *it; + it = raw.find("thinker_config"); + if (it != raw.end() && it->is_object()) { + auto text = it->find("text_config"); + if (text != it->end() && text->is_object()) return *text; + } return raw; } @@ -42,11 +58,14 @@ int64_t OptInt(const nlohmann::json& j, const char* key, int64_t fallback) { return it->get(); } -double OptDouble(const nlohmann::json& j, const char* key, double fallback) { +// Upstream's `x or y` treats an absent key, a null and an EMPTY STRING alike, +// so all three have to fall through to the alternative. +std::string OptNonEmptyString(const nlohmann::json& j, const char* key) { auto it = j.find(key); - if (it == j.end() || it->is_null()) return fallback; - if (!it->is_number()) Refuse(std::string("`") + key + "` must be a number."); - return it->get(); + if (it == j.end() || it->is_null()) return std::string(); + // A non-string value is dumped verbatim (`3`, `[]`) so the refusal names what + // was actually found rather than reporting it as absent. + return it->is_string() ? it->get() : it->dump(); } // Upstream treats an ABSENT QSA field and a null one alike, and the group is @@ -161,32 +180,73 @@ Qwen4ExpParams ParseQwen4ExpParams(const HfConfig& config) { std::to_string(p.shared_expert_intermediate_size) + "."); } - // --- output gate. The shared reader already canonicalizes `swish` to `silu` - // and refuses anything outside {silu, swish, sigmoid}; upstream Qwen4-Exp - // accepts {sigmoid, silu}, so the two sets agree AFTER canonicalization. - if (config.output_gate_type != "silu" && config.output_gate_type != "sigmoid") { - Refuse("unsupported output gate activation '" + config.output_gate_type + + // --- output gate. `output_gate_type = self.output_gate_type or + // self.hidden_act`, then refuse anything outside {sigmoid, silu} + // (configuration_qwen4_exp.py:193-195). + // + // Taken from the RAW text config rather than from `config.output_gate_type`, + // and the two differences are both real refusals. The shared reader defaults + // an ABSENT key to "silu" unconditionally (hf_config.cpp:458-460), with no + // `hidden_act` fallback, so a checkpoint whose `hidden_act` is `gelu` and + // whose gate key is missing was accepted here and would have run a silu gate + // on a model trained with something else. And the shared reader collapses + // `swish` to `silu` before this line runs, where upstream compares the raw + // string and raises on `swish` -- so the local check as written was a + // constant false that no config could ever trip. + std::string gate = OptNonEmptyString(text, "output_gate_type"); + if (gate.empty()) { + gate = OptNonEmptyString(text, "hidden_act"); + if (gate.empty()) gate = "silu"; // the dataclass default, :114 + } + if (gate != "silu" && gate != "sigmoid") { + Refuse("unsupported output gate activation '" + gate + "'; expected `sigmoid` or `silu`."); } - // --- rotary. `partial_rotary_factor` is read HERE with upstream's inherited - // default of 0.25 rather than taken from `config.rotary_dim`, and that is - // deliberate. `IsQwen35Family` in the shared reader does not list - // `qwen4_exp`, so an ABSENT key defaults there to 1.0 (full rotary) where - // upstream `Qwen4ExpTextConfig`, subclassing `Qwen3_5MoeTextConfig`, inherits - // 0.25. On the published checkpoint the key is present and both agree at 64; - // on a config that omits it they would disagree 256 vs 64, and because - // upstream's own guard is `rotary_dim > indexer_head_dim`, the shared - // reader's value would make us REFUSE a config upstream ACCEPTS. Mirroring - // the inheritance is what keeps the refusal set identical. - p.partial_rotary_factor = OptDouble(text, "partial_rotary_factor", 0.25); - if (!(p.partial_rotary_factor > 0.0)) { - Refuse("`partial_rotary_factor` must be > 0, got " + - std::to_string(p.partial_rotary_factor) + "."); - } - p.rotary_dim = - static_cast(static_cast(p.head_dim) * - p.partial_rotary_factor); + // --- rotary. TAKEN FROM THE SHARED READER, which already is the mirror. + // + // The previous shape read `partial_rotary_factor` out of the text config here + // with a default of 0.25, on the stated ground that `Qwen4ExpTextConfig` + // subclasses `Qwen3_5MoeTextConfig` and inherits that value. It does not, and + // three facts at the pin say so. The generated -- executed -- class is + // `class Qwen4ExpTextConfig(PreTrainedConfig)` + // (configuration_qwen4_exp.py:29). `partial_rotary_factor` is not among its + // declared fields (:109-164) and the string `0.25` does not occur anywhere in + // that file; its only two mentions of the name are the validator's own + // `partial_rotary_factor = (self.rope_parameters or {}).get( + // "partial_rotary_factor", 1.0)` (:225) and the `rotary_dim` it feeds + // (:226). And the modular source shows the bypass is deliberate: + // `__post_init__` calls `PreTrainedConfig.__post_init__(self, **kwargs)` + // DIRECTLY (modular_qwen4_exp.py:194), skipping the + // `kwargs.setdefault("partial_rotary_factor", 0.25) # assign default for BC` + // that is the sole source of 0.25 (configuration_qwen3_5_moe.py:124). + // + // So the default is 1.0 and the value lives in `rope_parameters`, which is + // exactly `ParseRopeParameters` (hf_config.cpp:143-205): top level first, the + // rope dict overriding. That ordering is upstream's too -- + // `convert_rope_params_to_dict` does + // `self.rope_parameters.setdefault("partial_rotary_factor", )` + // (modeling_rope_utils.py:755-757), and it runs BEFORE the generic `setattr` + // loop that would let `standardize_rope_params`:788 overwrite the dict + // (configuration_utils.py:314 vs :339), so `setdefault` is the whole + // precedence. `IsQwen35Family` correctly does NOT list `qwen4_exp`, so its + // default there is 1.0. + // + // The local read therefore diverged in BOTH directions: it ACCEPTED a config + // with no factor at all (upstream: 1.0 -> rotary_dim 256 > indexer_head_dim + // 128 -> raise) and handed W4 a 64-of-256 slice, and it REFUSED a config with + // top-level 1.0 and `rope_parameters.partial_rotary_factor` 0.25, which + // upstream accepts -- the very failure its comment claimed to prevent. + // + // NO LOCAL POSITIVITY GUARD, and its absence is measured rather than assumed. + // The shared reader already refuses anything outside (0, 1] + // (`hf_config: partial_rotary_factor must be in (0, 1]`), so a local + // `> 0` check here could never be reached — it would be the same constant + // false the output-gate check used to be. That bound is itself tighter than + // upstream, which validates the factor not at all; it belongs to the shared + // seam, and the row's spec records it there with the sweep row that shows it. + p.partial_rotary_factor = config.rope_parameters.partial_rotary_factor; + p.rotary_dim = config.rotary_dim; // --- QSA: all-or-nothing, then per-field. static constexpr const char* kQsaFields[] = { @@ -232,35 +292,42 @@ Qwen4ExpParams ParseQwen4ExpParams(const HfConfig& config) { } // --- PLE. One-indexed on the way in, 0-based on the way out. + // + // The n-gram fields resolve UNCONDITIONALLY, with upstream's own defaults. + // They are declared dataclass fields (configuration_qwen4_exp.py:149-157), so + // they hold those values whether or not any layer uses PLE, and a config that + // omits them is legal upstream. Defaulting them to 0 inside the PLE branch + // did two wrong things at once: it REFUSED such a config ("`ngram_size` must + // be >= 2 ... got 0"), and it left `ngram_heads()` at zero on a PLE-free + // config, which makes the `head_dim_per_ngram()` this header advertises a + // division by zero. + // + // `ple_embed_dim` defaults to `hidden_size`, set in `__post_init__` (:168). + p.ple.embed_dim = OptInt(text, "ple_embed_dim", p.hidden_size); + p.ple.conv_kernel_size = OptInt(text, "ple_conv_kernel_size", 4); + p.ple.ngram_size = OptInt(text, "ngram_size", 3); + p.ple.heads_per_ngram = OptInt(text, "heads_per_ngram", 8); + p.ple.ngram_vocab_size_base = OptInt(text, "ngram_vocab_size_base", 20000000); + p.ple.make_ngram_vocab_size_divisible_by = + OptInt(text, "make_ngram_vocab_size_divisible_by", 128); + p.ple.split_ngram_parts = OptInt(text, "split_ngram_parts", 512); + p.ple.seed = OptInt(text, "seed", 1234); + const std::vector raw_ple = OptIntArray(text, "ple_layer_ids"); - std::set sorted_unique(raw_ple.begin(), raw_ple.end()); - for (int64_t one_based : sorted_unique) { - if (one_based < 1 || one_based > p.num_hidden_layers) { - Refuse("`ple_layer_ids` must contain one-indexed ids in [1, " + - std::to_string(p.num_hidden_layers) + "], got " + - std::to_string(one_based) + "."); - } - const int64_t zero_based = one_based - 1; - if (p.layer_types[static_cast(zero_based)] != - Qwen4ExpLayerKind::kLinearAttention) { - Refuse("PLE is only supported on `linear_attention` layers; " - "`ple_layer_ids` names one-indexed layer " + - std::to_string(one_based) + " (0-based " + - std::to_string(zero_based) + "), which is a sparse-attention " - "layer."); - } - p.ple.layer_ids_zero_based.push_back(zero_based); - } - if (!p.ple.layer_ids_zero_based.empty()) { - p.ple.embed_dim = OptInt(text, "ple_embed_dim", p.hidden_size); - p.ple.conv_kernel_size = OptInt(text, "ple_conv_kernel_size", 4); - p.ple.ngram_size = OptInt(text, "ngram_size", 0); - p.ple.heads_per_ngram = OptInt(text, "heads_per_ngram", 0); - p.ple.ngram_vocab_size_base = OptInt(text, "ngram_vocab_size_base", 0); - p.ple.make_ngram_vocab_size_divisible_by = - OptInt(text, "make_ngram_vocab_size_divisible_by", 0); - p.ple.split_ngram_parts = OptInt(text, "split_ngram_parts", 512); - p.ple.seed = OptInt(text, "seed", 1234); + const std::set sorted_unique(raw_ple.begin(), raw_ple.end()); + if (!sorted_unique.empty()) { + // ORDER MIRRORS UPSTREAM (:233-257): head-count and embedding width first, + // then the layer-id range, then the layer kind, then EOS. Two violations at + // once must report the same one upstream reports, or a reader comparing the + // two runtimes is told to fix a different field. + // + // Upstream folds the first into one condition, + // `ngram_heads <= 0 or self.ple_embed_dim <= 0 or + // self.ple_embed_dim % ngram_heads != 0` (:235). The `ngram_size` and + // `heads_per_ngram` refusals below split that first term so the message + // names the field the reader has to edit; the accept/reject boundary is + // identical, because a sub-2 n-gram or a non-positive head count makes + // `ngram_heads` non-positive either way. if (p.ple.ngram_size < 2) { Refuse("`ngram_size` must be >= 2 when PLE is enabled, got " + std::to_string(p.ple.ngram_size) + "."); @@ -269,18 +336,55 @@ Qwen4ExpParams ParseQwen4ExpParams(const HfConfig& config) { Refuse("`heads_per_ngram` must be > 0 when PLE is enabled, got " + std::to_string(p.ple.heads_per_ngram) + "."); } - if (p.ple.conv_kernel_size <= 0) { - Refuse("`ple_conv_kernel_size` must be > 0, got " + - std::to_string(p.ple.conv_kernel_size) + "."); - } // 2560 / 16 = 160. A ragged split would silently mis-slice every gathered - // row, so it is refused rather than truncated. + // row, so it is refused rather than truncated. The `<= 0` term is upstream's + // and is not redundant in C++: `-2560 % 16 == 0`, so a negative width + // satisfies the divisibility test and `head_dim_per_ngram()` then returns + // -160. const int64_t heads = p.ple.ngram_heads(); - if (heads <= 0 || p.ple.embed_dim % heads != 0) { + if (heads <= 0 || p.ple.embed_dim <= 0 || p.ple.embed_dim % heads != 0) { Refuse("`ple_embed_dim` (" + std::to_string(p.ple.embed_dim) + - ") must be divisible by the n-gram head count (" + + ") must be > 0 and divisible by the n-gram head count (" + std::to_string(heads) + ")."); } + // LOCAL, tighter than upstream, which does not validate the PLE conv. + if (p.ple.conv_kernel_size <= 0) { + Refuse("`ple_conv_kernel_size` must be > 0, got " + + std::to_string(p.ple.conv_kernel_size) + "."); + } + for (int64_t one_based : sorted_unique) { + if (one_based < 1 || one_based > p.num_hidden_layers) { + Refuse("`ple_layer_ids` must contain one-indexed ids in [1, " + + std::to_string(p.num_hidden_layers) + "], got " + + std::to_string(one_based) + "."); + } + } + // Safe to index only because the range check above already ran over the + // whole set, exactly as upstream orders it. + for (int64_t one_based : sorted_unique) { + const int64_t zero_based = one_based - 1; + if (p.layer_types[static_cast(zero_based)] != + Qwen4ExpLayerKind::kLinearAttention) { + Refuse("PLE is only supported on `linear_attention` layers; " + "`ple_layer_ids` names one-indexed layer " + + std::to_string(one_based) + " (0-based " + + std::to_string(zero_based) + "), which is a sparse-attention " + "layer."); + } + p.ple.layer_ids_zero_based.push_back(zero_based); + } + // `if self.eos_token_id is None or isinstance(self.eos_token_id, list) and + // not self.eos_token_id` (:256-257). This is load-bearing rather than + // hygiene: the n-gram history is built with `_shift_right_ignore_eos` + // (modeling_qwen4_exp.py:1095), so EOS is a SEGMENT BOUNDARY in the hashed + // n-gram construction, and the published GGUF carries it as a first-class + // PLE key (`qwen4exp.ple.eos_token_id`). A config without one cannot have + // its n-gram ids constructed at all. + const auto eos = text.find("eos_token_id"); + if (eos == text.end() || eos->is_null() || + (eos->is_array() && eos->empty())) { + Refuse("`eos_token_id` must be set when PLE layers are enabled."); + } } // --- MTP. `mtp_num_hidden_layers` is a sibling of the `mtp` sub-object; the diff --git a/src/vllm/model_executor/models/qwen4_exp.h b/src/vllm/model_executor/models/qwen4_exp.h index 8c4f57947..f6b5e282e 100644 --- a/src/vllm/model_executor/models/qwen4_exp.h +++ b/src/vllm/model_executor/models/qwen4_exp.h @@ -17,6 +17,7 @@ #define VLLM_MODEL_EXECUTOR_MODELS_QWEN4_EXP_H_ #include +#include #include #include @@ -46,7 +47,22 @@ struct Qwen4ExpQsaParams { int64_t compress_ratio = 0; // indexer_compress_ratio = 4 // budget / compress_ratio = 512. Derived, never read from the config. - int64_t block_topk() const { return budget / compress_ratio; } + // + // REFUSES rather than divides when the group is absent. QSA is optional as a + // whole -- upstream treats all-five-absent as "QSA off" -- so a legally + // parsed config can leave `compress_ratio` at 0, and `budget / 0` is SIGFPE + // on x86: a crash, not a refusal, and one no downstream gate would attribute + // to this config. W2 and W4 are the callers, and neither has a reason to + // check the precondition before asking. + int64_t block_topk() const { + if (compress_ratio <= 0) { + throw std::runtime_error( + "qwen4_exp: block_topk() needs QSA, but `indexer_compress_ratio` is " + + std::to_string(compress_ratio) + + "; the QSA group is absent from this config."); + } + return budget / compress_ratio; + } }; // Per-Layer Embedding: the hashed n-gram table plus its dilated depthwise conv. @@ -67,13 +83,25 @@ struct Qwen4ExpPleParams { int64_t heads_per_ngram = 0; // 8 int64_t ngram_vocab_size_base = 0; // 20,000,000 int64_t make_ngram_vocab_size_divisible_by = 0; // 128 - int64_t split_ngram_parts = 0; // 128 — checkpoint SHARDING only, unused in the forward + int64_t split_ngram_parts = 512; // 128 in the checkpoint; SHARDING only, unused in the forward int64_t seed = 1234; // absent from the published config; the dataclass default // (ngram_size - 1) * heads_per_ngram = 16 hash heads per token. int64_t ngram_heads() const { return (ngram_size - 1) * heads_per_ngram; } - // embed_dim / ngram_heads = 160. - int64_t head_dim_per_ngram() const { return embed_dim / ngram_heads(); } + // embed_dim / ngram_heads = 160. Refuses rather than divides when the head + // count is zero, which `ngram_size == 1` produces on a config no PLE layer + // uses and therefore nothing validates. + int64_t head_dim_per_ngram() const { + const int64_t heads = ngram_heads(); + if (heads <= 0) { + throw std::runtime_error( + "qwen4_exp: head_dim_per_ngram() needs a positive n-gram head count, " + "got " + std::to_string(heads) + " from (ngram_size " + + std::to_string(ngram_size) + " - 1) * heads_per_ngram " + + std::to_string(heads_per_ngram) + "."); + } + return embed_dim / heads; + } // (conv_kernel_size - 1) * ngram_size = 9. NOT `kernel - 1`: the conv is // DILATED, so its state is three times deeper than an undilated one and the // taps sit at lags {9, 6, 3, 0}. @@ -107,7 +135,10 @@ struct Qwen4ExpParams { int64_t num_attention_heads = 0; // 24 int64_t num_key_value_heads = 0; // 2 int64_t head_dim = 0; // 256 - double partial_rotary_factor = 0.25; + // 1.0 is upstream's default: the validator reads + // `(self.rope_parameters or {}).get("partial_rotary_factor", 1.0)` + // (configuration_qwen4_exp.py:225) and the class declares no such field. + double partial_rotary_factor = 1.0; int64_t rotary_dim = 0; // int(head_dim * partial_rotary_factor) = 64 Qwen4ExpQsaParams qsa; diff --git a/src/vllm/model_executor/models/qwen4_exp_registry.cpp b/src/vllm/model_executor/models/qwen4_exp_registry.cpp index 55b9adb0b..a501a270d 100644 --- a/src/vllm/model_executor/models/qwen4_exp_registry.cpp +++ b/src/vllm/model_executor/models/qwen4_exp_registry.cpp @@ -58,12 +58,6 @@ inline constexpr ModelInfo kQwen4ExpInfo{ .score_type = "bi-encoder", }; -class Qwen4ExpLoadedModel final : public LoadedModel { - public: - explicit Qwen4ExpLoadedModel(const ModelRegistration& registration) - : LoadedModel(registration) {} -}; - std::unique_ptr LoadQwen4ExpForConditionalGeneration( const ModelRegistration& registration, const HfConfig& config, const ModelSource& source) { @@ -102,17 +96,27 @@ void PrepareQwen4ExpForConditionalGeneration(LoadedModel& model, ForwardLogits ForwardQwen4ExpForConditionalGeneration( LoadedModel& model, const ModelForwardInput& input) { - // `ModelAs`, never a bare `static_cast`: opening a type-erased handle by - // promise is undefined behaviour on any object that is not really this type, - // and it matters MORE on a refusing forward than on a working one, because - // the type confusion happens on the way to a throw that would have happened - // anyway and is therefore invisible without a sanitizer (#775, #730). - (void)ModelAs(model, - "Qwen4ExpForConditionalGeneration"); + (void)model; (void)input; + // THE REFUSAL COMES FIRST, AND THERE IS NO DOWNCAST ABOVE IT. That ordering + // is what makes it reachable at all, and the first draft had it the other way + // round. + // + // The house shape opens the type-erased handle with + // `ModelAs` before doing anything else, because a bare + // `static_cast` down the hierarchy is undefined behaviour on an object that + // is not really that type (#775, #730). But nothing can PRODUCE a loaded + // Qwen4-Exp while `load_weights` refuses unconditionally, so the only handle + // any caller can present is a foreign one, and a downcast placed first turns + // every reach into a type-mismatch report -- leaving the refusal below dead + // code that no test could enter and any later wave could delete without a + // red. Refusing before touching the handle is also the strictly safer + // direction on the #775 axis: no cast happens, so no type confusion can. + // W5 restores `ModelAs` at the moment there is a real forward with a real + // model to open. + // // `VT_CHECK(false, ...)` IN THE HOOK BODY, and not a bare throw behind a - // `Class::ForwardDevice` delegate. Three constraints meet here and only this - // shape satisfies all of them. + // `Class::ForwardDevice` delegate. Two constraints meet here. // // `scripts/check-runner-routing-consistency.py` recognises a refuse-by-name // stub by exactly this token (`_REFUSE`), and it classifies the hook body @@ -120,16 +124,12 @@ ForwardLogits ForwardQwen4ExpForConditionalGeneration( // bucket, which is the hole that checker exists to close — so tripping it // would be the defect, not the gate. The delegate hop dots3-note uses does // not help a model like this one: it resolves `Class::ForwardDevice` across - // translation units or through a file-local `ForwardLogits` helper, and a - // class defined inside this TU's own anonymous namespace is neither. + // translation units or through a file-local `ForwardLogits` helper, and this + // TU has neither. // // And `[[noreturn]]` on a non-void return type is MSVC C4646, promoted to // C2220 under /W4 /WX; `check-windows-portability.py` caught that on the // first draft of this function. - // - // There is no `Qwen4ExpModel::ForwardDevice` yet because there is no device - // forward yet. Inventing one to refuse from would assert a routing shape this - // row has not earned; W5 introduces it when there is something to route. VT_CHECK(false, "Qwen4ExpForConditionalGeneration: the forward is not ported yet. W2 " "owes the hashed n-gram embedding and the PLE dilated depthwise conv, " diff --git a/tests/scripts/test_agent_record.py b/tests/scripts/test_agent_record.py index 2b94772e3..dc6c6b04a 100644 --- a/tests/scripts/test_agent_record.py +++ b/tests/scripts/test_agent_record.py @@ -619,13 +619,25 @@ def test_qwen4_exp_row_is_inside_the_model_ratchet(self) -> None: every other check stays green. Only an assertion that names the row goes red. - `READY` is pinned deliberately and is the weaker half of the evidence, - stated rather than implied. The row is `READY` because its spec is - committed and no product code has landed; the structured-spec rules - already catch a move to `ACTIVE`, and the claim-ownership rules already - catch `INVENTORIED`. It is pinned anyway so that a future refactor of - those rules cannot silently take this pin with it -- which is exactly - the reasoning the dots3 test records for its own asymmetry. + The STATE pin is the weaker half of the evidence, stated rather than + implied, and it has now been moved ONCE, deliberately and with an + argument -- which is the movement it was written to make visible rather + than to prevent. It was `READY` while the spec was committed and no + product code had landed. W1 (#1981) landed the config surface: the + architecture resolves, its config parses and validates, and the loader, + forward and KV-cache spec refuse by name. That is a lifecycle change, + and AGENTS.md Records requires the owning matrix row to move with it, so + the row is `ACTIVE` and carries `CLAIM-MODEL-MM-QWEN4-EXP-W1`. + + The pin stays, at the new value, for the reason it was written: the + structured-spec rules already catch `ACTIVE` without a spec and the + claim-ownership rules already catch `ACTIVE` without a claim, but + neither would notice a silent slide BACK to `READY` on a row that has + shipped code, and neither names this row. Updating the value is not the + same as removing the assertion -- everything below still names the row, + still requires exactly one of it, and still requires it to live in + `model-matrix.md`, which is what makes 378 checkable rather than + plausible. The row is also beyond-pin in the strongest sense this file has carried: vLLM does not implement `qwen4_exp` at ANY revision, not merely after @@ -640,7 +652,11 @@ def test_qwen4_exp_row_is_inside_the_model_ratchet(self) -> None: found = [row for row in rows if row.item_id == item_id] self.assertEqual(len(found), 1, item_id) self.assertEqual(found[0].path.name, "model-matrix.md", item_id) - self.assertEqual(found[0].field("state").strip().strip("`"), "READY", item_id) + self.assertEqual(found[0].field("state").strip().strip("`"), "ACTIVE", item_id) + # ...and an ACTIVE row without a live claim is the shape the audit + # reports as ABANDONED, so the owner is pinned beside the state rather + # than left to the generic claim rule, which does not name this row. + self.assertIn("CLAIM-MODEL-MM-QWEN4-EXP-W1", found[0].field("owner"), item_id) # One row, not two: no speculative-head sibling exists for this arch. siblings = [row for row in rows if "qwen4-exp" in row.item_id] diff --git a/tests/vllm/models/test_qwen4_exp_scaffold.cpp b/tests/vllm/models/test_qwen4_exp_scaffold.cpp index dc65ece2f..89a594ed3 100644 --- a/tests/vllm/models/test_qwen4_exp_scaffold.cpp +++ b/tests/vllm/models/test_qwen4_exp_scaffold.cpp @@ -7,11 +7,22 @@ // NOT that anything reaches it, which AGENTS.md "Nothing lands dead" refuses to // accept as evidence. // +// EVERY refusal below is observed through `reg.factory->parse_config` and +// NOTHING ELSE. The earlier shape of this file called the hook and then +// returned `ParseQwen4ExpParams(config)`, so every assertion in it observed the +// FREE FUNCTION: gutting the registered hook to `(void)config;` left all 151 +// assertions green (review finding F2). `ThrowText` now enters through the hook +// alone, so that mutation reds. The value cases still need the returned struct +// -- the hook is `void` and cannot carry one -- so the hook's identity with the +// free function is pinned separately, by the equivalence case below. +// // ORACLE: transformers **5.16.0**, the lane pin accepted for this row. vLLM // implements `qwen4_exp` at no revision, so there is nothing to mirror on this // surface. Values come from the committed fixture, which is the published // `Qwen/Qwen3.8-Flash-Next` `config.json` verbatim. #include +#include +#include #include #include #include @@ -21,6 +32,10 @@ #include "doctest/doctest.h" #include "nlohmann/json.hpp" #include "vllm/model_executor/models/model_registry.h" +#include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits, *KvCache +#include "vllm/v1/attention/backend.h" // CommonAttentionMetadata +#include "vllm/v1/attention/backends/gdn_attn.h" // GDNAttentionMetadata +#include "vt/device.h" #include "vllm/model_executor/models/qwen4_exp.h" #include "vllm/transformers_utils/hf_config.h" @@ -30,6 +45,9 @@ using vllm::ModelRegistry; using vllm::ParseQwen4ExpParams; using vllm::Qwen4ExpLayerKind; using vllm::Qwen4ExpParams; +using vllm::LoadedModel; +using vllm::ModelForwardInput; +using vllm::ModelRegistration; namespace { @@ -85,7 +103,9 @@ nlohmann::json FixtureDoc() { } // Resolve through the registry and run the model's own config hook, which is -// exactly what `ModelRegistry::Load` does before it touches a weight. +// exactly what `ModelRegistry::Load` does before it touches a weight. The hook +// runs FIRST and is what refuses; the free function then supplies the resolved +// struct the value assertions read, which a `void` hook cannot return. Qwen4ExpParams ParseThroughRegistry(const nlohmann::json& doc) { TempConfig cfg(doc); const HfConfig config = LoadHfConfig(cfg.path()); @@ -95,15 +115,69 @@ Qwen4ExpParams ParseThroughRegistry(const nlohmann::json& doc) { return ParseQwen4ExpParams(config); } +// THE HOOK AND NOTHING ELSE. Every refusal case goes through this, so a hook +// that stopped validating reds the whole refusal suite. std::string ThrowText(const nlohmann::json& doc) { try { - ParseThroughRegistry(doc); + TempConfig cfg(doc); + const HfConfig config = LoadHfConfig(cfg.path()); + const vllm::ModelRegistration& reg = ModelRegistry::Resolve(config); + REQUIRE(reg.factory != nullptr); + reg.factory->parse_config(config); + } catch (const std::exception& e) { + return e.what(); + } + return ""; +} + +// The same doc put through the FREE FUNCTION alone, for the equivalence case. +std::string ThrowTextDirect(const nlohmann::json& doc) { + try { + TempConfig cfg(doc); + const HfConfig config = LoadHfConfig(cfg.path()); + (void)ParseQwen4ExpParams(config); } catch (const std::exception& e) { return e.what(); } return ""; } +// A FOREIGN `LoadedModel`: well-formed, and simply not this model's type. Same +// shape as test_registry_downcast_refusal.cpp, and the only shape available +// here, because nothing can produce a loaded Qwen4-Exp while the loader +// refuses. The forward must therefore refuse BEFORE it opens the handle, or the +// refusal it advertises is unreachable. +class ForeignLoadedModel final : public LoadedModel { + public: + explicit ForeignLoadedModel(const ModelRegistration& registration) + : LoadedModel(registration) {} +}; + +struct EmptyForwardInput { + std::vector token_ids{0}; + std::vector positions{0}; + std::vector logits_indices{0}; + vllm::v1::CommonAttentionMetadata attn_meta{}; + vllm::v1::GDNAttentionMetadata gdn_meta{}; + std::vector attn_kv; + std::vector gdn_state; + HfConfig config{}; + vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + + ModelForwardInput Get() { + return ModelForwardInput{.token_ids = token_ids, + .positions = positions, + .attn_meta = attn_meta, + .gdn_meta = gdn_meta, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = config, + .queue = queue, + .logits_indices = logits_indices, + .num_reqs = 1}; + } +}; + } // namespace TEST_CASE("qwen4_exp: the published config resolves through the registry") { @@ -213,27 +287,351 @@ TEST_CASE("qwen4_exp: ple_layer_ids is ONE-indexed and lands on layer 1") { CHECK(p.layer_types[1] == Qwen4ExpLayerKind::kLinearAttention); } -TEST_CASE("qwen4_exp: an omitted partial_rotary_factor keeps upstream's inherited 0.25") { - // REGRESSION GUARD, not a nicety. `IsQwen35Family` in the shared HfConfig - // reader does not list `qwen4_exp`, so an absent `partial_rotary_factor` - // defaults THERE to 1.0 (full rotary), while upstream `Qwen4ExpTextConfig` - // subclasses `Qwen3_5MoeTextConfig` and inherits 0.25. Taking the shared - // reader's value would give rotary_dim 256, and because upstream's own guard - // is `rotary_dim > indexer_head_dim` (128), we would REFUSE a config upstream - // ACCEPTS. Mirroring the inheritance is what keeps the refusal sets identical. - nlohmann::json doc = FixtureDoc(); - doc["text_config"].erase("partial_rotary_factor"); - if (doc["text_config"].contains("rope_parameters")) { +TEST_CASE("qwen4_exp: partial_rotary_factor comes from rope_parameters and defaults to 1.0") { + // THIS CASE IS THE INVERSION OF ITS PREDECESSOR, and the inversion is the + // point. The earlier case pinned an "inherited 0.25" that does not exist at + // the pin. `Qwen4ExpTextConfig` is generated as + // `class Qwen4ExpTextConfig(PreTrainedConfig)` + // (configuration_qwen4_exp.py:29) -- it does NOT subclass + // `Qwen3_5MoeTextConfig`, `partial_rotary_factor` is not among its declared + // fields (:109-164), and the string `0.25` does not appear in the file. Its + // only two occurrences of the name are the validator's own read, + // `partial_rotary_factor = (self.rope_parameters or {}).get( + // "partial_rotary_factor", 1.0)` (:225) and the `rotary_dim` it feeds + // (:226). The modular source confirms it deliberately: + // `Qwen4ExpTextConfig.__post_init__` calls + // `PreTrainedConfig.__post_init__(self, **kwargs)` DIRECTLY + // (modular_qwen4_exp.py:194), bypassing the `kwargs.setdefault( + // "partial_rotary_factor", 0.25) # assign default for BC` that is the + // sole source of 0.25 (configuration_qwen3_5_moe.py:124). + // + // So the default is 1.0, and the value lives in `rope_parameters`. The shared + // reader already implements exactly that: `ParseRopeParameters` takes the top + // level first and lets `rope_parameters` override, which is what + // `convert_rope_params_to_dict`'s + // `self.rope_parameters.setdefault("partial_rotary_factor", kwargs[...])` + // does (modeling_rope_utils.py:755-757) -- and it runs BEFORE the generic + // `setattr` loop that would otherwise let `standardize_rope_params`:788 + // overwrite the dict (configuration_utils.py:314 vs :339), so `setdefault` + // is the whole precedence. `IsQwen35Family` correctly excludes `qwen4_exp`, + // so `config.rotary_dim` IS upstream's `rotary_dim`. + SUBCASE("the published checkpoint, where both spellings say 0.25") { + const Qwen4ExpParams p = ParseThroughRegistry(FixtureDoc()); + CHECK(p.partial_rotary_factor == doctest::Approx(0.25)); + CHECK(p.rotary_dim == 64); + CHECK(p.rotary_dim <= p.qsa.head_dim); + } + + SUBCASE("absent everywhere: 1.0, rotary_dim 256, and upstream REFUSES") { + // Upstream: `(None or {}).get(..., 1.0)` -> 1.0 -> rotary_dim 256 > + // indexer_head_dim 128 -> ValueError. The old code answered 0.25/64 and + // ACCEPTED, handing W4 a 64-of-256 slice on a checkpoint that wants 256 -- + // a silent numerics error on a row with no reachable token gate. + nlohmann::json doc = FixtureDoc(); + doc["text_config"].erase("partial_rotary_factor"); doc["text_config"]["rope_parameters"].erase("partial_rotary_factor"); + const std::string msg = ThrowText(doc); + CHECK(msg.find("rotary dim 256") != std::string::npos); + CHECK(msg.find("indexer_head_dim") != std::string::npos); + } + + SUBCASE("rope_parameters WINS over the top level") { + // Upstream reads the rope dict and never the top-level key, so top-level + // 1.0 with rope 0.25 is ACCEPTED at rotary_dim 64. The old code read the + // top level, got 1.0, and refused -- the exact false refusal its comment + // claimed to prevent. + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["partial_rotary_factor"] = 1.0; + doc["text_config"]["rope_parameters"]["partial_rotary_factor"] = 0.25; + const Qwen4ExpParams p = ParseThroughRegistry(doc); + CHECK(p.partial_rotary_factor == doctest::Approx(0.25)); + CHECK(p.rotary_dim == 64); + } + + SUBCASE("the top level fills in when the rope dict omits the key") { + // `setdefault` semantics: the top-level key is folded into + // `rope_parameters` only where the dict has none. + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["partial_rotary_factor"] = 0.25; + doc["text_config"]["rope_parameters"].erase("partial_rotary_factor"); + const Qwen4ExpParams p = ParseThroughRegistry(doc); + CHECK(p.rotary_dim == 64); + } + + SUBCASE("top-level 0.25 does NOT rescue a rope dict that says 1.0") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["partial_rotary_factor"] = 0.25; + doc["text_config"]["rope_parameters"]["partial_rotary_factor"] = 1.0; + CHECK(ThrowText(doc).find("rotary dim 256") != std::string::npos); + } +} + +TEST_CASE("qwen4_exp: the REGISTERED hook is the validator, not a bystander") { + // The hook is `void`, so no value can flow through it and no value case can + // be made unreachable without it. What CAN be pinned is its identity with the + // free function: for the same document the two must give the same answer, + // refusal text included. A hook gutted to `(void)config;` reds every row. + const nlohmann::json fixture = FixtureDoc(); + + nlohmann::json bad_gate = fixture; + bad_gate["text_config"].erase("output_gate_type"); + bad_gate["text_config"]["hidden_act"] = "gelu"; + + nlohmann::json bad_eos = fixture; + bad_eos["text_config"]["eos_token_id"] = nullptr; + + nlohmann::json bad_hc = fixture; + bad_hc["text_config"]["hc_count"] = 1; + + nlohmann::json bad_rope = fixture; + bad_rope["text_config"].erase("partial_rotary_factor"); + bad_rope["text_config"]["rope_parameters"].erase("partial_rotary_factor"); + + for (const nlohmann::json* doc : {&bad_gate, &bad_eos, &bad_hc, &bad_rope}) { + const std::string through_hook = ThrowText(*doc); + const std::string direct = ThrowTextDirect(*doc); + CHECK_FALSE(through_hook.empty()); + CHECK(through_hook == direct); } + // ...and the fixture passes through BOTH. + CHECK(ThrowText(fixture).empty()); + CHECK(ThrowTextDirect(fixture).empty()); +} +TEST_CASE("qwen4_exp: the PLE defaults are upstream's, not zero") { + // Every one of these is a declared field with a default + // (configuration_qwen4_exp.py:149-157), so a config that omits them is legal + // upstream. Defaulting them to 0 made us REFUSE such a config with + // "`ngram_size` must be >= 2 ... got 0" (review probe F), and carried a zero + // n-gram vocabulary silently into W2 for the two fields that have no guard. + nlohmann::json doc = FixtureDoc(); + for (const char* key : {"ngram_size", "heads_per_ngram", + "ngram_vocab_size_base", + "make_ngram_vocab_size_divisible_by", + "split_ngram_parts", "ple_conv_kernel_size", + "ple_embed_dim", "seed"}) { + doc["text_config"].erase(key); + } const Qwen4ExpParams p = ParseThroughRegistry(doc); - CHECK(p.partial_rotary_factor == doctest::Approx(0.25)); - CHECK(p.rotary_dim == 64); - CHECK(p.rotary_dim <= p.qsa.head_dim); + CHECK(p.ple.ngram_size == 3); + CHECK(p.ple.heads_per_ngram == 8); + CHECK(p.ple.ngram_vocab_size_base == 20000000); + CHECK(p.ple.make_ngram_vocab_size_divisible_by == 128); + CHECK(p.ple.split_ngram_parts == 512); + CHECK(p.ple.conv_kernel_size == 4); + CHECK(p.ple.seed == 1234); + // `ple_embed_dim` defaults to `hidden_size` in upstream's `__post_init__`. + CHECK(p.ple.embed_dim == 2560); + CHECK(p.ple.ngram_heads() == 16); } +TEST_CASE("qwen4_exp: the n-gram fields resolve even when no layer uses PLE") { + // Upstream carries them as dataclass fields, so they hold their defaults + // whether or not `ple_layer_ids` is set. Reading them only inside the PLE + // branch left `ngram_size == 0`, which made `ngram_heads()` zero and + // `head_dim_per_ngram()` a division by zero on a legally-parsed config. + nlohmann::json doc = FixtureDoc(); + doc["text_config"].erase("ple_layer_ids"); + const Qwen4ExpParams p = ParseThroughRegistry(doc); + CHECK(p.ple.layer_ids_zero_based.empty()); + CHECK(p.number_of_conv_states() == 1); + CHECK(p.ple.ngram_size == 3); + CHECK(p.ple.ngram_heads() == 16); + CHECK(p.ple.head_dim_per_ngram() == 160); +} + +TEST_CASE("qwen4_exp: the derived helpers refuse instead of dividing by zero") { + // `block_topk()` and `head_dim_per_ngram()` are advertised by the header as + // the derived values a port should use, and W2/W4 will call them. Both + // divide, and both divisors can legally be zero: QSA is optional as a group, + // and `ngram_size == 1` makes the head count zero. `budget / 0` is SIGFPE on + // x86 -- a crash, not a refusal, and one no gate downstream would attribute. + SUBCASE("QSA absent: block_topk refuses by name") { + nlohmann::json doc = FixtureDoc(); + for (const char* f : {"indexer_n_heads", "indexer_kv_heads", + "indexer_head_dim", "indexer_budget", + "indexer_compress_ratio"}) { + doc["text_config"].erase(f); + } + const Qwen4ExpParams p = ParseThroughRegistry(doc); + REQUIRE(p.qsa.compress_ratio == 0); + CHECK_THROWS_WITH_AS(p.qsa.block_topk(), + doctest::Contains("indexer_compress_ratio"), + std::runtime_error); + } + SUBCASE("a zero n-gram head count: head_dim_per_ngram refuses by name") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"].erase("ple_layer_ids"); // no PLE => not validated + doc["text_config"]["ngram_size"] = 1; // (1 - 1) * 8 == 0 + const Qwen4ExpParams p = ParseThroughRegistry(doc); + REQUIRE(p.ple.ngram_heads() == 0); + CHECK_THROWS_WITH_AS(p.ple.head_dim_per_ngram(), + doctest::Contains("n-gram head count"), + std::runtime_error); + } +} + +TEST_CASE("qwen4_exp: a text config nested under llm_config resolves the SAME way") { + // `HfConfig`'s own `ResolveTextConfig` handles `text_config`, `llm_config` + // and `thinker_config.text_config`; the model's local `TextOf` handled only + // the first, so on an `llm_config` wrapper the shared reader found + // `hidden_size`/`layer_types` while the model found NO `hc_*`, QSA, PLE or + // MTP key and silently produced a half-parsed config. + nlohmann::json doc = FixtureDoc(); + nlohmann::json nested = doc; + nested["llm_config"] = doc["text_config"]; + nested.erase("text_config"); + + const Qwen4ExpParams a = ParseThroughRegistry(doc); + const Qwen4ExpParams b = ParseThroughRegistry(nested); + CHECK(b.hc_count == a.hc_count); + CHECK(b.qsa.budget == a.qsa.budget); + CHECK(b.ple.ngram_size == a.ple.ngram_size); + CHECK(b.ple.layer_ids_zero_based == a.ple.layer_ids_zero_based); + CHECK(b.mtp_num_hidden_layers == a.mtp_num_hidden_layers); + CHECK(b.number_of_conv_states() == a.number_of_conv_states()); +} + +// EVERY refusal in `ParseQwen4ExpParams` has a subcase here, and that is a GATE +// OBLIGATION rather than thoroughness: the row's `## Gates` G0 item 6 is "every +// rejection in `validate_architecture`", and this row has no reachable token +// gate, so the config layer is the last place the refusal boundary is checkable +// at all. A single mutation deleting 13 of the 22 refusals used to leave the +// suite green (review finding F10). +// +// The `[UP]` rows mirror one upstream raise; the `[LOCAL]` rows are refusals +// this port imposes that `validate_architecture` does not. Both directions are +// tabulated against their upstream line in +// `.agents/specs/qwen4-exp-flash-next.md` `## The refusal boundary`. TEST_CASE("qwen4_exp: the config refuses every unrepresentable combination BY NAME") { + SUBCASE("[LOCAL] num_hidden_layers must be positive") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["num_hidden_layers"] = 0; + CHECK(ThrowText(doc).find("num_hidden_layers") != std::string::npos); + } + SUBCASE("[LOCAL] full_attention_interval must be positive") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"].erase("layer_types"); + doc["text_config"]["full_attention_interval"] = 0; + CHECK(ThrowText(doc).find("full_attention_interval") != std::string::npos); + } + SUBCASE("[LOCAL] hc_lowrank must be positive") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["hc_lowrank"] = 0; + CHECK(ThrowText(doc).find("hc_lowrank") != std::string::npos); + } + SUBCASE("[UP] num_experts must be positive") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["num_experts"] = 0; + CHECK(ThrowText(doc).find("num_experts") != std::string::npos); + } + SUBCASE("[UP] the MoE intermediate sizes must be positive") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["moe_intermediate_size"] = 0; + CHECK(ThrowText(doc).find("moe_intermediate_size") != std::string::npos); + } + SUBCASE("[UP] an absent output_gate_type falls back to hidden_act") { + // `output_gate_type = self.output_gate_type or self.hidden_act` + // (configuration_qwen4_exp.py:193). The shared reader defaults an ABSENT + // key to "silu" unconditionally, which made us accept a checkpoint whose + // gate is whatever `hidden_act` says (review probe G) and left the local + // check a constant false. + nlohmann::json doc = FixtureDoc(); + doc["text_config"].erase("output_gate_type"); + doc["text_config"]["hidden_act"] = "gelu"; + const std::string msg = ThrowText(doc); + CHECK(msg.find("output gate") != std::string::npos); + CHECK(msg.find("gelu") != std::string::npos); + } + SUBCASE("[UP] an explicit output_gate_type outside {sigmoid, silu}") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["output_gate_type"] = "swish"; + // Upstream compares the RAW string against {"sigmoid", "silu"}, so `swish` + // raises there. The shared reader canonicalizes `swish` to `silu` for the + // GDN family, so the refusal has to be taken on the raw value here. + CHECK(ThrowText(doc).find("swish") != std::string::npos); + } + SUBCASE("[SHARED, tighter than upstream] a factor outside (0, 1]") { + // Named for what it IS: this refusal comes from the SHARED reader + // (`hf_config: partial_rotary_factor must be in (0, 1]`), before this + // model's parse runs, and upstream validates the factor not at all. A local + // `> 0` guard here would be unreachable, which is why there is not one -- + // the same constant-false shape the output-gate check used to have. The + // assertion names the shared message so that a later widening of that bound + // shows up here rather than passing on a substring both messages share. + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["rope_parameters"]["partial_rotary_factor"] = -0.25; + CHECK(ThrowText(doc).find("must be in (0, 1]") != std::string::npos); + } + SUBCASE("[UP] QSA values must be positive") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["indexer_n_heads"] = 0; + CHECK(ThrowText(doc).find("positive") != std::string::npos); + } + SUBCASE("[LOCAL] ngram_size below 2") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["ngram_size"] = 1; + CHECK(ThrowText(doc).find("ngram_size") != std::string::npos); + } + SUBCASE("[LOCAL] heads_per_ngram must be positive") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["heads_per_ngram"] = 0; + CHECK(ThrowText(doc).find("heads_per_ngram") != std::string::npos); + } + SUBCASE("[LOCAL] ple_conv_kernel_size must be positive") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["ple_conv_kernel_size"] = 0; + CHECK(ThrowText(doc).find("ple_conv_kernel_size") != std::string::npos); + } + SUBCASE("[UP] a NEGATIVE ple_embed_dim, which -2560 % 16 == 0 lets through") { + // Upstream's condition is `ngram_heads <= 0 or self.ple_embed_dim <= 0 or + // self.ple_embed_dim % ngram_heads != 0` (:235). Dropping the middle term + // accepted -2560 in C++, where the remainder is 0, and + // `head_dim_per_ngram()` then returned -160 (review probe H). + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["ple_embed_dim"] = -2560; + CHECK(ThrowText(doc).find("ple_embed_dim") != std::string::npos); + } + SUBCASE("[UP] a ple_embed_dim that does not divide by the head count") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["ple_embed_dim"] = 2561; + CHECK(ThrowText(doc).find("ple_embed_dim") != std::string::npos); + } + SUBCASE("[UP] eos_token_id must be set when PLE is enabled") { + // `configuration_qwen4_exp.py:256-257`. Not cosmetic: the n-gram history + // uses `_shift_right_ignore_eos`, so EOS is a SEGMENT BOUNDARY in the + // hashed n-gram construction, and the published GGUF stores it as + // `qwen4exp.ple.eos_token_id`. A null there hands W2 a config whose n-gram + // ids cannot be built, against a gate that is integer equality. + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["eos_token_id"] = nullptr; + CHECK(ThrowText(doc).find("eos_token_id") != std::string::npos); + } + SUBCASE("[UP] an EMPTY eos_token_id list is refused too") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["eos_token_id"] = nlohmann::json::array(); + CHECK(ThrowText(doc).find("eos_token_id") != std::string::npos); + } + SUBCASE("[LOCAL] mtp_num_hidden_layers must not be negative") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["mtp_num_hidden_layers"] = -1; + CHECK(ThrowText(doc).find("mtp_num_hidden_layers") != std::string::npos); + } + SUBCASE("[LOCAL] a non-integer where an integer belongs") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["hc_lowrank"] = "three hundred and twenty"; + CHECK(ThrowText(doc).find("must be an integer") != std::string::npos); + } + SUBCASE("[LOCAL] ple_layer_ids that is not an array") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["ple_layer_ids"] = 2; + CHECK(ThrowText(doc).find("must be an array") != std::string::npos); + } + SUBCASE("[LOCAL] ple_layer_ids holding a non-integer") { + nlohmann::json doc = FixtureDoc(); + doc["text_config"]["ple_layer_ids"] = nlohmann::json::array({"two"}); + CHECK(ThrowText(doc).find("only integers") != std::string::npos); + } SUBCASE("an unsupported layer type") { nlohmann::json doc = FixtureDoc(); doc["text_config"]["layer_types"][0] = "sliding_attention"; @@ -319,6 +717,60 @@ TEST_CASE("qwen4_exp: load, forward and the KV spec refuse BY NAME, naming the o CHECK(msg.find("tensor not found") == std::string::npos); } + SUBCASE("the GGUF arm, which refuses for a DIFFERENT reason than safetensors") { + // The GGUF k-quant arm is OWED, not optional, and it is the arm most likely + // to fit a host we own. Its refusal names what W6 owes; nothing asserted it, + // so deleting the whole branch left the gate green (review mutation M7) and + // a GGUF load would have fallen through to the safetensors message, sending + // the reader to the wrong wave. + vllm::ModelSource source{}; + source.kind = vllm::ModelSource::Kind::kGguf; + std::string msg; + try { + (void)reg.factory->load_weights(reg, config, source); + } catch (const std::exception& e) { + msg = e.what(); + } + CHECK(msg.find("Qwen4ExpForConditionalGeneration") != std::string::npos); + CHECK(msg.find("GGUF") != std::string::npos); + CHECK(msg.find("W6") != std::string::npos); + CHECK(msg.find("IQ4_NL") != std::string::npos); + // ...and it must NOT be the safetensors message. + CHECK(msg.find("weight loader") == std::string::npos); + } + + SUBCASE("the forward") { + // The case was TITLED "load, forward and the KV spec refuse" and had no + // forward subcase: deleting the `VT_CHECK` left the forward returning an + // empty `ForwardLogits{}` and the gate green (review mutation M6). + // + // The refusal has to come BEFORE the `ModelAs` downcast, or it is + // unreachable rather than merely untested: nothing can produce a loaded + // Qwen4-Exp while the loader refuses, so the only handle any caller can + // present is a foreign one, and a downcast placed first turns every reach + // into a type-mismatch report instead. + REQUIRE(reg.factory->forward != nullptr); + ForeignLoadedModel foreign(reg); + EmptyForwardInput in; + const ModelForwardInput input = in.Get(); + std::string msg; + try { + (void)reg.factory->forward(foreign, input); + } catch (const std::exception& e) { + msg = e.what(); + } + CHECK(msg.find("Qwen4ExpForConditionalGeneration") != std::string::npos); + CHECK(msg.find("forward is not ported") != std::string::npos); + // Each wave that owes a piece of it is named, so the reader is not sent to + // the loader for work W2/W3/W4 owe. + CHECK(msg.find("W2") != std::string::npos); + CHECK(msg.find("W4") != std::string::npos); + CHECK(msg.find("#1978") != std::string::npos); + // And it is NOT the type-mismatch report, which would mean the refusal this + // model advertises is unreachable behind a downcast. + CHECK(msg.find("was not produced by") == std::string::npos); + } + SUBCASE("the KV-cache spec") { std::string msg; try { From 4263abb9cd069e43cb6b6be6aedfbc96bb9b929b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 17:23:03 +0000 Subject: [PATCH 3/4] =?UTF-8?q?fix(MODEL-MM-QWEN4-EXP):=20W1=20=E2=80=94?= =?UTF-8?q?=20two=20of=20the=20new=20refusal=20cases=20were=20substring=20?= =?UTF-8?q?gates=20that=20a=20deleted=20guard=20still=20satisfied?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mutating each of the 23 refusals in `ParseQwen4ExpParams` one at a time, rather than as the union the review used, found two of my own cases passing with their subject removed. `num_hidden_layers = 0` asserted that the message contains `num_hidden_layers`. Delete that guard and the next refusal down fires instead — "`layer_types` has 48 entries but `num_hidden_layers` is 0" — which contains the same word, so the case stayed green. `num_experts = 0` had the identical shape against the `num_experts_per_tok must be in [1, num_experts]` message. Both now assert the distinguishing text, and both go red when their guard is deleted. The general rule this is an instance of: a substring assertion is a weak gate wherever two refusals share a word, and only a per-guard mutation finds it. A union mutation cannot: the first one attempted here took SIGFPE on `(i + 1) % 0` at its second member and never reached the other eleven, so eleven guards would have been reported as covered on the strength of one crash. All 23 refusals now red individually, restored byte-for-byte after each, and the spec records the method beside the sweep it complements. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/qwen4-exp-flash-next.md | 24 +++++++++++++++++++ tests/vllm/models/test_qwen4_exp_scaffold.cpp | 12 ++++++++-- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index cd9c1bc4d..56e61852d 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -805,6 +805,30 @@ and embedding width first, then the id range, then the layer kind, then EOS. A config violating two at once has to report the one upstream reports, or a reader comparing the two runtimes is sent to a different field. +### Every refusal is mutated ONE AT A TIME + +A sweep is an accept/reject comparison; it does not say whether OUR TESTS would +notice a refusal going missing. So each of the 23 refusals in +`ParseQwen4ExpParams` was deleted individually — `if () {` rewritten to +`if (false) {`, proved applied by a non-empty `git diff --stat`, rebuilt, run, +and restored by byte comparison. **All 23 red.** Before this change a single +mutation deleting 13 of them at once left the suite green. + +Deleting them as a UNION is not equivalent and would have hidden two defects: the +first union mutation SIGFPE'd on `(i + 1) % 0` at the second subcase and never +reached the other eleven. Run one at a time, two of the new subcases turned out +to be weak — `num_hidden_layers = 0` asserted the bare field name, which the next +refusal down ("`layer_types` has 48 entries but `num_hidden_layers` is 0") also +prints, and `num_experts = 0` the same against the `num_experts_per_tok` range +message. Both now assert the distinguishing text. That is the general shape: +**a substring assertion is a weak gate wherever two refusals share a word**, and +only a per-guard mutation finds it. + +The three production entry points were mutated too. Gutting the registered +`parse_config` hook to `(void)config;` reds 3 cases / 42 assertions; removing the +forward's `VT_CHECK` reds 5 assertions; removing the GGUF arm's throw reds 4. +Before this change all three were green. + ### Refusals we impose that upstream does not Each is deliberate, each is exercised, and each is a row in the sweep above. None diff --git a/tests/vllm/models/test_qwen4_exp_scaffold.cpp b/tests/vllm/models/test_qwen4_exp_scaffold.cpp index 89a594ed3..501d78df7 100644 --- a/tests/vllm/models/test_qwen4_exp_scaffold.cpp +++ b/tests/vllm/models/test_qwen4_exp_scaffold.cpp @@ -505,9 +505,15 @@ TEST_CASE("qwen4_exp: a text config nested under llm_config resolves the SAME wa // `.agents/specs/qwen4-exp-flash-next.md` `## The refusal boundary`. TEST_CASE("qwen4_exp: the config refuses every unrepresentable combination BY NAME") { SUBCASE("[LOCAL] num_hidden_layers must be positive") { + // The DISTINGUISHING text, not the bare field name. The next refusal down + // ("`layer_types` has 48 entries but `num_hidden_layers` is 0") also names + // the field, so deleting this guard left the case green -- caught by + // mutating this refusal alone, and it is the shape a substring assertion + // takes whenever two refusals share a word. nlohmann::json doc = FixtureDoc(); doc["text_config"]["num_hidden_layers"] = 0; - CHECK(ThrowText(doc).find("num_hidden_layers") != std::string::npos); + CHECK(ThrowText(doc).find("`num_hidden_layers` must be > 0") != + std::string::npos); } SUBCASE("[LOCAL] full_attention_interval must be positive") { nlohmann::json doc = FixtureDoc(); @@ -521,9 +527,11 @@ TEST_CASE("qwen4_exp: the config refuses every unrepresentable combination BY NA CHECK(ThrowText(doc).find("hc_lowrank") != std::string::npos); } SUBCASE("[UP] num_experts must be positive") { + // Same trap: the `num_experts_per_tok` range refusal below also prints + // `num_experts`, so the bare field name passed with this guard deleted. nlohmann::json doc = FixtureDoc(); doc["text_config"]["num_experts"] = 0; - CHECK(ThrowText(doc).find("num_experts") != std::string::npos); + CHECK(ThrowText(doc).find("`num_experts` must be > 0") != std::string::npos); } SUBCASE("[UP] the MoE intermediate sizes must be positive") { nlohmann::json doc = FixtureDoc(); From d90c6b6941332b6ca217cd24511935b21caa022d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 21:45:25 +0000 Subject: [PATCH 4/4] =?UTF-8?q?fix(MODEL-MM-QWEN4-EXP):=20W1=20=E2=80=94?= =?UTF-8?q?=20"never=20the=20reverse"=20was=20an=20absolute,=20and=20a=20f?= =?UTF-8?q?ortieth=20case=20falsifies=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The refusal-boundary sweep said "35 agree; 4 differ, and all 4 are ours refusing what upstream accepts — never the reverse". The first half is a measurement. The second half was an absolute, and a fresh re-review broke it with a case outside the 39. `rope_parameters` carrying `rope_dim = 64` alongside `partial_rotary_factor = 1.0`: upstream ignores `rope_dim` entirely and computes `int(head_dim * partial_rotary_factor)` = 256 unconditionally, so it REFUSES at 256 > indexer_head_dim 128. We prefer `rope_dim`, following vLLM's `get_rope` semantics in the shared reader, and ACCEPT at rotary_dim 64 — handing W4 a 64-of-256 slice. Same failure mode and same direction as the finding that failed this wave's first review, reached through a different key. Scoped rather than repaired, and the distinction matters. The divergence is narrow: `rope_dim` has zero occurrences in `modeling_rope_utils.py` at v5.16.0, so no transformers path writes or reads it and no published checkpoint carries it. It also does not live in this model — it is the shared reader deliberately mirroring vLLM instead of transformers on that point, which is the polarity AGENTS.md sets. Repairing it here would mean changing a shared rope resolution from inside a model row. What is not acceptable is leaving the absolute standing. In the row whose entire product is a refusal boundary, a claim about that boundary has to be either true or bounded, and this one is now bounded to the 39 cases actually measured, with the fortieth stated in full and owed to whoever reconciles the shared reader. Tracked by #1981, under #1978. Gates: `check-agent-record` ok, `check-model-checklist` ok. Records-only; no code changed, so the focused gate is unmoved at 12 cases / 294 assertions. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/qwen4-exp-flash-next.md | 28 +++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index 56e61852d..b0aca00eb 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -759,8 +759,32 @@ statement about running the model, and nothing here runs one. 39 configs, each derived from the committed fixture, put through `Qwen4ExpConfig.from_dict` on one side and `LoadHfConfig -> ModelRegistry::Resolve -> factory->parse_config -> ParseQwen4ExpParams` on the other. **35 agree; 4 -differ, and all 4 are ours refusing what upstream accepts** — never the reverse, -which is the direction that would let a bad checkpoint through. +differ, and over these 39 all 4 are ours refusing what upstream accepts** — the +safe direction, since the reverse is what lets a bad checkpoint through. + +**That is a claim about the measured set, and it is bounded on purpose.** An earlier +draft said "never the reverse" as an absolute, and a fresh re-review falsified it with +a fortieth case outside the sweep: `rope_parameters` carrying **`rope_dim = 64` +alongside `partial_rotary_factor = 1.0`**. Upstream ignores `rope_dim` entirely — +`validate_architecture` computes `int(self.head_dim * partial_rotary_factor)` = 256 +unconditionally at `configuration_qwen4_exp.py:225-226` — and refuses, because +256 > `indexer_head_dim` 128. We take `rope_dim` in preference, following vLLM's +`get_rope` semantics in the shared reader (`hf_config.cpp:545-547`), and **ACCEPT at +`rotary_dim = 64`**, handing W4 a 64-of-256 slice. That is the same failure mode and +the same direction as the finding that failed this wave's first review, reached +through a different key. + +It is narrow and it is not a defect in this model's code: `rope_dim` has **zero +occurrences** in `modeling_rope_utils.py` at v5.16.0, so no transformers path writes +or reads it and no published checkpoint carries it — the oracle tolerates the key and +ignores it. The divergence lives in the shared reader, which is deliberately mirroring +vLLM rather than transformers on that point. + +It is recorded rather than repaired because the fix belongs to whoever reconciles the +shared reader's rope resolution, not to this row, and because the honest form of a +boundary claim in the row whose whole product is that boundary is either **true or +bounded**. Owed: either a `rope_dim` case in the sweep with the divergence stated, or +a shared-reader change that makes it moot. Reproduce (transformers 5.16.0 in a venv; the probe links `build/libvllm.a` with `-Wl,--whole-archive` so the model's self-registration survives):