diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index 02aadc606..c9425d063 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -500,11 +500,39 @@ Usable budget on GB10 is about 119 GB. Read live from the HF API, 2026-08-26: | `Qwen/Qwen3.8-Flash-Next` BF16 | ~360 GB (`BF16 = 179,999,981,424` params) | no | | `Qwen/Qwen3.8-Flash-Next-FP8` (official) | ~180 GB | no | | `RadixArk/Qwen3.8-Flash-Next-NVFP4` | ~128 GB; NVFP4 backbone with the n-gram table kept at **FP8, 51.2 GB** | no, over budget before KV | -| `unsloth/Qwen3.8-Flash-Next-GGUF` | **README only, zero weight files** | does not exist | - -No GGUF exists and no existing tool can produce one, because llama.cpp has no -`qwen4_exp` architecture either. Per AGENTS.md the quantized arms are a standing -requirement, so this row owes them and owes authoring the arch on our side. +| `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S | **67.56 GiB**, 3 shards | **YES, and it is the ONLY published artifact that does** | + +**CORRECTED 2026-08-26.** This table previously read "README only, zero weight files +-- does not exist", and that was true when it was written and false a few hours later. +The repository was populated at 13:32Z with `UD-IQ1_S/Qwen3.8-Flash-Next-UD-IQ1_S-0000{1,2,3}-of-00003.gguf`, +72,546,461,344 bytes = **67.56 GiB**, read from the files' own headers: +`general.architecture = "qwen4exp"`, `split.tensors.count = 1224`, +`general.file_type = 24`. **It fits GB10 with roughly 52 GiB of headroom**, where every +safetensors artifact does not fit at all. The whole shape of this row's `## Work +breakdown` follows from that, which is why the correction is called out rather than +quietly applied. + +Its metadata independently confirms this spec's own n-gram derivation to the digit: +`qwen4exp.ple.layer_multipliers = [23703573157769, 20109073645365, 8052911324071]`, +`qwen4exp.ple.head_vocab_sizes` starting `[20000003, 20000023, 20000033, ...]`, and +`qwen4exp.ple.layers = [1]` (0-based) corroborating the one-indexed conversion. + +**Two things in OUR tree stop us loading it**, and both are W6a's +([#1989](https://github.com/mudler/vllm.cpp/issues/1989)): our GGUF reader has no +`case 20`, so the IQ4_NL that file uses for `ffn_down_exps` and for the n-gram table +fails at header parse; and `KeepQuantKDim` returns `-1` for `kEmbeddingTable`, so a +quantized gather table expands to bf16 and 51.2B params become 102.4 GB. + +It carries **no MTP weights** — zero `nextn`/`mtp` tensors of 1224 — while the +safetensors repo has 31. That is [#1993](https://github.com/mudler/vllm.cpp/issues/1993)'s +problem and `docs/USAGE.md` must say so beside the arm. + +**A revision and a per-shard sha256 are still OWED.** The repo's `lastModified` moved +again after this row first read it, which is exactly the re-quantized-in-place case +AGENTS.md "Say which weights, and from where" names; a repo id alone is not a pin. + +llama.cpp still has no *merged* `qwen4_exp` architecture -- two competing PRs are open +or withdrawn -- so authoring our own converter remains owed for arms nobody publishes. **The architecture hands us the lever.** Its card argues n-gram embedding is "more amenable to offloading than MoE", and the arithmetic agrees: the per-token cost is @@ -709,9 +737,56 @@ No token gate is claimable until an arm runs. In order: 4. **G3, quantized arms.** Per arm, with the lower-bound requirement this repository places on quantized gates, and with the missing-llama.cpp-oracle limitation stated in the result rather than omitted. -5. **Speed: nothing.** No throughput, latency or memory number is admissible from - this row until G2 passes. There is no vLLM denominator for this model, so when a - speed axis does open, the spec must first say what the denominator is. +5. **G4, speed against llama.cpp at its pin.** A denominator now exists and the + earlier "there is no denominator" clause is superseded: `llama-cpp` is a registered, + pinned, `gateable = yes` oracle whose scope is "GGUF k-quant speed and memory floors, + **quant-matched against the same weights**", and `unsloth/Qwen3.8-Flash-Next-GGUF` + UD-IQ1_S is one published artifact both engines can run. The axis opens when W6a makes + that file loadable, and not before: no throughput, latency or memory number is + admissible from this row until then. + + **The target is binding** (developer, 2026-08-26): faster than llama.cpp on this + model, *especially at high concurrency*. Therefore: + + - **A concurrency LADDER is the headline, not a point.** c = 1, 4, 8, 16, 32 at + minimum. A c=1 result neither confirms nor refutes this target. + - **Prefill and decode reported separately**, because input length splits them and an + aggregate hides which lever moved. + - Memory is an axis: peak RSS and peak device bytes at each concurrency. + - llama.cpp runs in its production configuration. A handicapped denominator is not a + result, and this repository already has the `--enforce-eager` precedent for how that + goes wrong. + - Identical artifact, prompts, token counts, sampling and concurrency; idle host; + reproduced with a same-binary A/B. + +### Where the speed is expected to come from, and what would forfeit it + +Four levers, from a source study of the two llama.cpp implementations (#27742 open, +#27739 closed by courtesy). **Both are UNMERGED**; each item is a reading of a pinned SHA +and not a measurement. Recorded here because three of them constrain waves that have not +started. + +1. **Continuous batching and paged KV — the concurrency lever.** This engine mirrors + vLLM's scheduler and block manager; llama.cpp's server allocates fixed parallel slots. + That gap grows with concurrency rather than shrinking, which is where the target aims. +2. **The QSA consumer — the long-context lever, and the one this row can forfeit by + accident.** #27739 records that a sparse **mask** over a dense cache costs the same as + dense attention under CUDA flash attention, because `flash_attn_mask_to_KV_max` only + scans back to the first tile that is not all `-inf`. #27742 is mask-only and so buys + correctness without decode speed. **W4 builds the gather, not the mask.** +3. **N-gram table residency.** #27742 makes the table CPU-resident by tensor class + regardless of `-ngl`, so every token's 16 gathers are host work. At IQ4_NL the table is + ~28.8 GB inside a 67.56 GiB file against ~119.6 GiB usable, so it can be + device-resident and quantized. At batch B that is 16xB uncoalesced random gathers, so + host-versus-device here is a scaling difference, not a constant. W6a's residency + decision must be **asserted**, not defaulted. +4. **The hyper-connection write-back.** Both PRs materialise the rank-1 update as a + `repeat_4d` + `mul`: 96 materialised `[2560, 4, T]` broadcasts per forward at 48 + layers x 2 sites. W3 leaves the fused seam reachable rather than building it. + +**No ceiling may be declared** if a first measurement disappoints. An apparent +same-artifact limit is an unresolved implementation difference with a next traceable +hypothesis, every time. ## Evidence required