record(MODEL-MM-QWEN4-EXP): a speed denominator exists, and the target is to beat llama.cpp at concurrency - #1995
Open
localai-bot wants to merge 2 commits into
Open
record(MODEL-MM-QWEN4-EXP): a speed denominator exists, and the target is to beat llama.cpp at concurrency#1995localai-bot wants to merge 2 commits into
localai-bot wants to merge 2 commits into
Conversation
…t is to beat llama.cpp at concurrency The row spec's `## Gates` said "Speed: nothing ... There is no vLLM denominator for this model, so when a speed axis does open, the spec must first say what the denominator is." Two of those sentences are now stale and the third is answerable, so this says it. `llama-cpp` is already a registered oracle with a pin and `gateable = yes`, and its recorded scope is "GGUF k-quant speed and memory floors, quant-matched against the same weights". `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S is one published artifact both engines can run, which is exactly that condition. The axis opens when W6a makes the file loadable and not before. The developer's target is binding rather than aspirational: faster than llama.cpp on this model, especially at high concurrency. The gate is written to match, because a c=1 number would neither confirm nor refute it — a concurrency ladder is the headline, prefill and decode are reported separately since input length splits them, memory is an axis rather than a footnote, and llama.cpp runs in its production configuration. This repository already has the `--enforce-eager` precedent for what a handicapped denominator does to a comparison. The larger part of this change is the four levers, and they are recorded now because three of them constrain waves that have not started. The QSA one is the reason for the urgency: #27739 records that a sparse mask over a dense cache costs the same as dense attention under CUDA flash attention, because `flash_attn_mask_to_KV_max` only scans back to the first tile that is not all -inf. #27742 is mask-only and therefore buys correctness without decode speed. W4 can forfeit the long-context lever by accident, by building the obvious thing, so the spec now says to build the gather. Likewise W6a's n-gram residency: #27742 pins that table to the CPU by tensor class regardless of -ngl, so every token's 16 gathers are host work, and at batch B that is 16xB uncoalesced random gathers — host versus device there is a scaling difference and not a constant. At IQ4_NL the table is ~28.8 GB inside a 67.56 GiB file against ~119.6 GiB usable, so device-resident and quantized is available to us. The spec now requires that decision to be asserted rather than defaulted. Both llama.cpp PRs are UNMERGED and every lever above is a reading of a pinned SHA, not a measurement. No number is claimed here, nothing has been benchmarked, and no arm runs yet. The no-ceiling rule is restated in place, because the moment to write it down is before a first measurement disappoints rather than after. Tracked by #1990, under #1978. Gates: `check-agent-record` ok, `check-model-checklist` ok. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code]
Collaborator
Author
|
Gate state, stated rather than implied.
So the gate could not find a quiet window in which to test its guarantee. It is being re-run on a quiet host before this merges; it is not being waived, and this comment exists so that the re-run is owed in writing rather than remembered. |
…main was actively misleading parallel waves The spec's `## Hardware` table said `unsloth/Qwen3.8-Flash-Next-GGUF` was "README only, zero weight files -- does not exist". True when written, false a few hours later: the repository was populated at 13:32Z with three shards totalling 72,546,461,344 bytes = 67.56 GiB, and it FITS GB10 with roughly 52 GiB of headroom where every safetensors artifact does not fit at all. The correction is being made here, on a records-only branch, because of where it was first made. It went into the W1 branch alongside product code, W1 then failed its fresh review, and the correction is now stuck behind a repair. Meanwhile W3's implementer read `main`, found the spec asserting the artifact does not exist, correctly flagged it as contradicting the issue it had been given, and had no way to know which side was stale. That is the cost of bundling a time-critical record fix into a change that can fail for unrelated reasons, and it is the reason AGENTS.md allows a record-only pull request when the record IS the work. The whole shape of this row's `## Work breakdown` follows from that file existing, so the correction is called out in place rather than quietly applied. It also carries what the file independently confirms -- `ple.layer_multipliers`, `ple.head_vocab_sizes` and a 0-based `ple.layers = [1]`, all matching this spec's own derivation -- and the two things in OUR tree that still stop us loading it, both W6a's: no `case 20` for IQ4_NL in the reader, and `KeepQuantKDim` returning -1 for a gather table. A revision and a per-shard sha256 are recorded as OWED rather than invented. The repo's `lastModified` moved again after this row first read it, which is precisely the re-quantized-in-place case AGENTS.md names when it says a repo id alone is not a pin. Tracked by #1990, under #1978. Gates: `check-agent-record` ok, `check-model-checklist` ok. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The row spec's
## Gatessaid "Speed: nothing ... There is no vLLM denominator forthis model, so when a speed axis does open, the spec must first say what the
denominator is." Two of those sentences are now stale and the third is answerable,
so this says it.
llama-cppis already a registered oracle with a pin andgateable = yes, and itsrecorded scope is "GGUF k-quant speed and memory floors, quant-matched against the
same weights".
unsloth/Qwen3.8-Flash-Next-GGUFUD-IQ1_S is one published artifactboth engines can run, which is exactly that condition. The axis opens when W6a makes
the file loadable and not before.
The developer's target is binding rather than aspirational: faster than llama.cpp on
this model, especially at high concurrency. The gate is written to match, because a
c=1 number would neither confirm nor refute it — a concurrency ladder is the headline,
prefill and decode are reported separately since input length splits them, memory is
an axis rather than a footnote, and llama.cpp runs in its production configuration.
This repository already has the
--enforce-eagerprecedent for what a handicappeddenominator does to a comparison.
The larger part of this change is the four levers, and they are recorded now because
three of them constrain waves that have not started. The QSA one is the reason for the
urgency: #27739 records that a sparse mask over a dense cache costs the same as dense
attention under CUDA flash attention, because
flash_attn_mask_to_KV_maxonly scansback to the first tile that is not all -inf. #27742 is mask-only and therefore buys
correctness without decode speed. W4 can forfeit the long-context lever by accident,
by building the obvious thing, so the spec now says to build the gather.
Likewise W6a's n-gram residency: #27742 pins that table to the CPU by tensor class
regardless of -ngl, so every token's 16 gathers are host work, and at batch B that is
16xB uncoalesced random gathers — host versus device there is a scaling difference and
not a constant. At IQ4_NL the table is ~28.8 GB inside a 67.56 GiB file against ~119.6
GiB usable, so device-resident and quantized is available to us. The spec now requires
that decision to be asserted rather than defaulted.
Both llama.cpp PRs are UNMERGED and every lever above is a reading of a pinned SHA,
not a measurement. No number is claimed here, nothing has been benchmarked, and no arm
runs yet. The no-ceiling rule is restated in place, because the moment to write it down
is before a first measurement disappoints rather than after.
Tracked by #1990, under #1978.
Gates:
check-agent-recordok,check-model-checklistok.FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]