Qwen3.8-Flash-Next is a new architecture (`qwen4_exp`) that vLLM does not implement: port it with transformers as the algorithmic oracle and vLLM ops as the optimized path

[localai-bot]

Qwen/Qwen3.8-Flash-Next (released 2026-08-24, 180B total / 6B activated) declares
Qwen4ExpForConditionalGeneration / model_type: qwen4_exp. The name is misleading:
this is not the Qwen3.8-27B shape retrained, it is the architecture Qwen's own card
calls "the architecture that will underpin Qwen4". Nothing in this tree matches it
(grep over .agents/, docs/, src/, include/, tests/ for qwen4_exp /
Flash-Next returns zero hits).

The oracle problem, and the split this row adopts

vLLM implements nothing. Read live at origin/main = 6a5e8f5979: no
qwen4* file, no registry entry, and a GitHub search for qwen4 across
vllm-project/vllm returns 0 results. vllm-omni likewise. Advancing the
parity pin does not help, because the model is absent from vLLM main, not merely
from our pin (555967922, 2026-07-26).

What exists as of 2026-08-26:

Upstream	State
huggingface/transformers #48337 "Add Qwen4Exp model"	MERGED 2026-08-26, 5211 lines
sgl-project/sglang #36497 "Introduce Qwen 3.8 Flash Next"	OPEN (cookbook #36496 merged)
vllm-project/vllm	nothing

Developer direction, 2026-08-26: transformers is the oracle for the ALGORITHM,
and vLLM supplies the OPS. That is not a compromise, it is the correct reading of
what each upstream is. transformers #48337 is a semantics reference and states so in
its own code: the QSA indexer's forward loops in Python over (batch, query), and its
comment reads "we only allow eager and sdpa". Porting it as written would produce a
correct model at an indefensible speed. vLLM is where the optimized form of each of
these primitives already lives, and AGENTS.md's "Mirror vLLM" polarity still binds
every op that vLLM implements even when vLLM has never assembled this particular model.

So each component resolves against exactly one of them, and the spec records which.

The delta, and where each half comes from

The transformers modular file inherits from Qwen3_5Moe / Qwen3Next, which this
tree already implements, and leaves rotary embedding, MLP, experts, TopK router and
the whole vision tower (Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass)
unchanged. What is actually new is four things, and only two of them lack a vLLM op.

Component	Algorithm (transformers)	Optimized op (vLLM origin/main)	This tree
GDN linear attention	Qwen4ExpTextGatedDeltaNet	layers/mamba/gdn/qwen_gdn_linear_attn.py	HAVE. Config is K=V=128, Hg=16, Hv=48, an exact match for the AOT gate at src/vt/cuda/cuda_gdn.cu:5264 (K=V=128, Hg=16, H in {48,32})
Grouped RMSNorm	Qwen4ExpTextRMSNorm(group_size=)	layers/layernorm.py:187 group_size	new, small; mirror vLLM
QSA block scoring + top-k	Qwen4ExpTextQSAIndexer (eager, Python loops)	models/minimax_m3/common/indexer.py, common/ops/index_topk.py (_index_block_score_kernel, bitonic _topk_index_kernel), common/sparse_attention.py	new
QSA pooled-key build (mean-pool -> norm -> RoPE at block start)	indexer forward	models/deepseek_v4/nvidia/ops/sparse_attn_compress_cutedsl.py (SparseAttnCompressNormRopeStoreC4Kernel, carries compress_ratio)	partial: deepseek_v4_compressor.h
Indexer side cache	Cache.update_indexer	MiniMaxM3IndexerCache, v1/attention/backends/mla/indexer.py	new KV spec
Gated Residual (hyper-connections)	Qwen4ExpTextGatedResidual	layers/mhc.py + kernels/mhc/{torch,triton,tilelang,aiter}.py (different math, same fused shape)	partial: deepseek_v4_mhc.cpp
MoE 512 experts, top-10 + 1 shared, intermediate 640	Qwen4ExpTextSparseMoeBlock	FusedMoE / grouped GEMM	HAVE, shape change only
MTP (1 layer, hybrid: true)	config mtp	qwen3_5_mtp.py	HAVE, needs extension
PLE dilated depthwise conv (kernel 4, dilation 3)	Qwen4ExpTextPLELayer._short_conv	NONE. causal_conv1d has no dilation anywhere in vLLM (git grep dilation over layers/mamba/ = 0 hits)	new, no vLLM op
N-gram hashed embedding (20M entries, 51B params)	Qwen4ExpTextNGramEmbedding	NONE	new, no vLLM op

The single most useful finding of the survey: QSA's structural twin in vLLM is
MiniMax-M3, not DeepSeek-V4. DSA is an MLA indexer; QSA is not MLA (plain GQA,
24 Q heads / 2 KV, head_dim 256). MiniMax-M3's lightning indexer is the non-MLA
case that already scores KV blocks, takes top-k blocks, and keeps its own side
cache of one index-key vector per token, feeding a block-sparse attention consumer.
That is QSA's exact shape. Building QSA on the DSA/MLA path instead would be the
wrong port, and it is the mistake this row exists to prevent.

Two structural consequences beyond the module list:

• The residual stream is hc_count x hidden_size = 4 x 2560 = 10240 wide through
  the entire stack, with a low-rank read gate and per-branch scalar write gate
  around both attention and MLP. This is a change to the per-layer loop and to every
  residual buffer, not a drop-in module.
• number_of_conv_states = 3 on a PLE layer (GDN conv, PLE conv, n-gram token
  history), so the KV-cache spec grows a third conv stream. Adjacent to the
  divisor work in #1963 /
  #1966.

Nothing published fits the hardware, so the quantized arms are load-bearing

Read live from the HF API, 2026-08-26. Usable budget on GB10 is about 119 GB.

Artifact	On disk	Verdict
Qwen/Qwen3.8-Flash-Next BF16	~360 GB (safetensors.parameters.BF16 = 179,999,981,424)	no
Qwen/Qwen3.8-Flash-Next-FP8 (official)	~180 GB	no
RadixArk/Qwen3.8-Flash-Next-NVFP4	~128 GB (NVFP4 backbone + the n-gram table kept at FP8, 51.2 GB)	no, over budget before KV
unsloth/Qwen3.8-Flash-Next-GGUF	README only, 2 files, zero weights	does not exist

No GGUF exists and none can be produced by an existing tool, because llama.cpp has
no qwen4_exp architecture either. Per AGENTS.md the quantized arms are a standing
requirement and not a choice, so this row owes them, and producing them means
authoring the arch on our side rather than consuming someone else's file.

The architecture hands us the lever itself. Its card argues n-gram embedding is
"more amenable to offloading than MoE", and it is right: the per-token cost is
(ngram_size - 1) * heads_per_ngram = 16 lookups of 160 dims. 51 GB of the 180 GB,
28% of the model, is a table touched 16 times per token. Making it non-resident is
the intended design point, not a workaround, and RadixArk independently reached the
same split by quantizing the backbone to NVFP4 while leaving that one table at FP8.

Sizing arithmetic for the arms this row owes, at about 119 GB usable:

Arm	Backbone (125B)	N-gram (51B)	Resident	Fits
Q8_0 throughout	~133 GB	~54 GB	~191 GB	no
Q4_K_M throughout	~76 GB	~31 GB	~109 GB	yes, ~10 GB left for KV and activations
Q4_K_M backbone, n-gram table non-resident	~76 GB	0	~76 GB	yes, with room

MTP (4B) is separable and a first gate can drop it. Note Qwen3.8-Flash (the cloud
product) is a different, unreleased thing; only -Next is open-weight.

What this issue asks for

A committed spec before any code, per AGENTS.md "Spec before code", recording the
oracle split above per component, the QSA-maps-to-MiniMax-M3 finding, the KV-cache
consequences, and the quantized-arm plan with its owed items. No product code lands
under this issue.

Known open decision, not settled here

.agents/oracles/transformers.md pins transformers to 5.14.1, deliberately tied
to whatever the pinned vLLM environment resolves, with the stated reason that an
independent pin "would let the oracle environment hold two different transformers
at once". 5.14.1 does not contain Qwen4Exp. Using transformers as this row's
algorithmic oracle therefore needs a lane-scoped second pin and an argued exception.
The argument is available and narrow: the invariant guards against drift between a
vLLM env and its transformers, and here there is no vLLM implementation to drift
from. The spec proposes it; a reviewer should accept or reject it explicitly rather
than let it pass as housekeeping.