From b33615b774ddf684633b7f8e239883ea312bf2f0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 15:44:04 +0000 Subject: [PATCH 1/2] record(MODEL-MM-QWEN4-EXP): a speed denominator exists, and the target is to beat llama.cpp at concurrency MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The row spec's `## Gates` said "Speed: nothing ... There is no vLLM denominator for this model, so when a speed axis does open, the spec must first say what the denominator is." Two of those sentences are now stale and the third is answerable, so this says it. `llama-cpp` is already a registered oracle with a pin and `gateable = yes`, and its recorded scope is "GGUF k-quant speed and memory floors, quant-matched against the same weights". `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S is one published artifact both engines can run, which is exactly that condition. The axis opens when W6a makes the file loadable and not before. The developer's target is binding rather than aspirational: faster than llama.cpp on this model, especially at high concurrency. The gate is written to match, because a c=1 number would neither confirm nor refute it — a concurrency ladder is the headline, prefill and decode are reported separately since input length splits them, memory is an axis rather than a footnote, and llama.cpp runs in its production configuration. This repository already has the `--enforce-eager` precedent for what a handicapped denominator does to a comparison. The larger part of this change is the four levers, and they are recorded now because three of them constrain waves that have not started. The QSA one is the reason for the urgency: #27739 records that a sparse mask over a dense cache costs the same as dense attention under CUDA flash attention, because `flash_attn_mask_to_KV_max` only scans back to the first tile that is not all -inf. #27742 is mask-only and therefore buys correctness without decode speed. W4 can forfeit the long-context lever by accident, by building the obvious thing, so the spec now says to build the gather. Likewise W6a's n-gram residency: #27742 pins that table to the CPU by tensor class regardless of -ngl, so every token's 16 gathers are host work, and at batch B that is 16xB uncoalesced random gathers — host versus device there is a scaling difference and not a constant. At IQ4_NL the table is ~28.8 GB inside a 67.56 GiB file against ~119.6 GiB usable, so device-resident and quantized is available to us. The spec now requires that decision to be asserted rather than defaulted. Both llama.cpp PRs are UNMERGED and every lever above is a reading of a pinned SHA, not a measurement. No number is claimed here, nothing has been benchmarked, and no arm runs yet. The no-ceiling rule is restated in place, because the moment to write it down is before a first measurement disappoints rather than after. Tracked by #1990, under #1978. Gates: `check-agent-record` ok, `check-model-checklist` ok. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/qwen4-exp-flash-next.md | 53 +++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 3 deletions(-) diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index 02aadc606..ccc509b06 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -709,9 +709,56 @@ No token gate is claimable until an arm runs. In order: 4. **G3, quantized arms.** Per arm, with the lower-bound requirement this repository places on quantized gates, and with the missing-llama.cpp-oracle limitation stated in the result rather than omitted. -5. **Speed: nothing.** No throughput, latency or memory number is admissible from - this row until G2 passes. There is no vLLM denominator for this model, so when a - speed axis does open, the spec must first say what the denominator is. +5. **G4, speed against llama.cpp at its pin.** A denominator now exists and the + earlier "there is no denominator" clause is superseded: `llama-cpp` is a registered, + pinned, `gateable = yes` oracle whose scope is "GGUF k-quant speed and memory floors, + **quant-matched against the same weights**", and `unsloth/Qwen3.8-Flash-Next-GGUF` + UD-IQ1_S is one published artifact both engines can run. The axis opens when W6a makes + that file loadable, and not before: no throughput, latency or memory number is + admissible from this row until then. + + **The target is binding** (developer, 2026-08-26): faster than llama.cpp on this + model, *especially at high concurrency*. Therefore: + + - **A concurrency LADDER is the headline, not a point.** c = 1, 4, 8, 16, 32 at + minimum. A c=1 result neither confirms nor refutes this target. + - **Prefill and decode reported separately**, because input length splits them and an + aggregate hides which lever moved. + - Memory is an axis: peak RSS and peak device bytes at each concurrency. + - llama.cpp runs in its production configuration. A handicapped denominator is not a + result, and this repository already has the `--enforce-eager` precedent for how that + goes wrong. + - Identical artifact, prompts, token counts, sampling and concurrency; idle host; + reproduced with a same-binary A/B. + +### Where the speed is expected to come from, and what would forfeit it + +Four levers, from a source study of the two llama.cpp implementations (#27742 open, +#27739 closed by courtesy). **Both are UNMERGED**; each item is a reading of a pinned SHA +and not a measurement. Recorded here because three of them constrain waves that have not +started. + +1. **Continuous batching and paged KV — the concurrency lever.** This engine mirrors + vLLM's scheduler and block manager; llama.cpp's server allocates fixed parallel slots. + That gap grows with concurrency rather than shrinking, which is where the target aims. +2. **The QSA consumer — the long-context lever, and the one this row can forfeit by + accident.** #27739 records that a sparse **mask** over a dense cache costs the same as + dense attention under CUDA flash attention, because `flash_attn_mask_to_KV_max` only + scans back to the first tile that is not all `-inf`. #27742 is mask-only and so buys + correctness without decode speed. **W4 builds the gather, not the mask.** +3. **N-gram table residency.** #27742 makes the table CPU-resident by tensor class + regardless of `-ngl`, so every token's 16 gathers are host work. At IQ4_NL the table is + ~28.8 GB inside a 67.56 GiB file against ~119.6 GiB usable, so it can be + device-resident and quantized. At batch B that is 16xB uncoalesced random gathers, so + host-versus-device here is a scaling difference, not a constant. W6a's residency + decision must be **asserted**, not defaulted. +4. **The hyper-connection write-back.** Both PRs materialise the rank-1 update as a + `repeat_4d` + `mul`: 96 materialised `[2560, 4, T]` broadcasts per forward at 48 + layers x 2 sites. W3 leaves the fused seam reachable rather than building it. + +**No ceiling may be declared** if a first measurement disappoints. An apparent +same-artifact limit is an unresolved implementation difference with a next traceable +hypothesis, every time. ## Evidence required From cd7199e73409b9fd12cbcd78411e755d4a6d2f8c Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 26 Aug 2026 17:06:39 +0000 Subject: [PATCH 2/2] record(MODEL-MM-QWEN4-EXP): the GGUF exists, and saying otherwise on main was actively misleading parallel waves The spec's `## Hardware` table said `unsloth/Qwen3.8-Flash-Next-GGUF` was "README only, zero weight files -- does not exist". True when written, false a few hours later: the repository was populated at 13:32Z with three shards totalling 72,546,461,344 bytes = 67.56 GiB, and it FITS GB10 with roughly 52 GiB of headroom where every safetensors artifact does not fit at all. The correction is being made here, on a records-only branch, because of where it was first made. It went into the W1 branch alongside product code, W1 then failed its fresh review, and the correction is now stuck behind a repair. Meanwhile W3's implementer read `main`, found the spec asserting the artifact does not exist, correctly flagged it as contradicting the issue it had been given, and had no way to know which side was stale. That is the cost of bundling a time-critical record fix into a change that can fail for unrelated reasons, and it is the reason AGENTS.md allows a record-only pull request when the record IS the work. The whole shape of this row's `## Work breakdown` follows from that file existing, so the correction is called out in place rather than quietly applied. It also carries what the file independently confirms -- `ple.layer_multipliers`, `ple.head_vocab_sizes` and a 0-based `ple.layers = [1]`, all matching this spec's own derivation -- and the two things in OUR tree that still stop us loading it, both W6a's: no `case 20` for IQ4_NL in the reader, and `KeepQuantKDim` returning -1 for a gather table. A revision and a per-shard sha256 are recorded as OWED rather than invented. The repo's `lastModified` moved again after this row first read it, which is precisely the re-quantized-in-place case AGENTS.md names when it says a repo id alone is not a pin. Tracked by #1990, under #1978. Gates: `check-agent-record` ok, `check-model-checklist` ok. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --- .agents/specs/qwen4-exp-flash-next.md | 38 +++++++++++++++++++++++---- 1 file changed, 33 insertions(+), 5 deletions(-) diff --git a/.agents/specs/qwen4-exp-flash-next.md b/.agents/specs/qwen4-exp-flash-next.md index ccc509b06..c9425d063 100644 --- a/.agents/specs/qwen4-exp-flash-next.md +++ b/.agents/specs/qwen4-exp-flash-next.md @@ -500,11 +500,39 @@ Usable budget on GB10 is about 119 GB. Read live from the HF API, 2026-08-26: | `Qwen/Qwen3.8-Flash-Next` BF16 | ~360 GB (`BF16 = 179,999,981,424` params) | no | | `Qwen/Qwen3.8-Flash-Next-FP8` (official) | ~180 GB | no | | `RadixArk/Qwen3.8-Flash-Next-NVFP4` | ~128 GB; NVFP4 backbone with the n-gram table kept at **FP8, 51.2 GB** | no, over budget before KV | -| `unsloth/Qwen3.8-Flash-Next-GGUF` | **README only, zero weight files** | does not exist | - -No GGUF exists and no existing tool can produce one, because llama.cpp has no -`qwen4_exp` architecture either. Per AGENTS.md the quantized arms are a standing -requirement, so this row owes them and owes authoring the arch on our side. +| `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S | **67.56 GiB**, 3 shards | **YES, and it is the ONLY published artifact that does** | + +**CORRECTED 2026-08-26.** This table previously read "README only, zero weight files +-- does not exist", and that was true when it was written and false a few hours later. +The repository was populated at 13:32Z with `UD-IQ1_S/Qwen3.8-Flash-Next-UD-IQ1_S-0000{1,2,3}-of-00003.gguf`, +72,546,461,344 bytes = **67.56 GiB**, read from the files' own headers: +`general.architecture = "qwen4exp"`, `split.tensors.count = 1224`, +`general.file_type = 24`. **It fits GB10 with roughly 52 GiB of headroom**, where every +safetensors artifact does not fit at all. The whole shape of this row's `## Work +breakdown` follows from that, which is why the correction is called out rather than +quietly applied. + +Its metadata independently confirms this spec's own n-gram derivation to the digit: +`qwen4exp.ple.layer_multipliers = [23703573157769, 20109073645365, 8052911324071]`, +`qwen4exp.ple.head_vocab_sizes` starting `[20000003, 20000023, 20000033, ...]`, and +`qwen4exp.ple.layers = [1]` (0-based) corroborating the one-indexed conversion. + +**Two things in OUR tree stop us loading it**, and both are W6a's +([#1989](https://github.com/mudler/vllm.cpp/issues/1989)): our GGUF reader has no +`case 20`, so the IQ4_NL that file uses for `ffn_down_exps` and for the n-gram table +fails at header parse; and `KeepQuantKDim` returns `-1` for `kEmbeddingTable`, so a +quantized gather table expands to bf16 and 51.2B params become 102.4 GB. + +It carries **no MTP weights** — zero `nextn`/`mtp` tensors of 1224 — while the +safetensors repo has 31. That is [#1993](https://github.com/mudler/vllm.cpp/issues/1993)'s +problem and `docs/USAGE.md` must say so beside the arm. + +**A revision and a per-shard sha256 are still OWED.** The repo's `lastModified` moved +again after this row first read it, which is exactly the re-quantized-in-place case +AGENTS.md "Say which weights, and from where" names; a repo id alone is not a pin. + +llama.cpp still has no *merged* `qwen4_exp` architecture -- two competing PRs are open +or withdrawn -- so authoring our own converter remains owed for arms nobody publishes. **The architecture hands us the lever.** Its card argues n-gram embedding is "more amenable to offloading than MoE", and the arithmetic agrees: the per-token cost is