Skip to content

record(MODEL-MM-QWEN4-EXP): a speed denominator exists, and the target is to beat llama.cpp at concurrency - #1995

Open
localai-bot wants to merge 2 commits into
mainfrom
row/MODEL-MM-QWEN4-EXP-PERF
Open

record(MODEL-MM-QWEN4-EXP): a speed denominator exists, and the target is to beat llama.cpp at concurrency#1995
localai-bot wants to merge 2 commits into
mainfrom
row/MODEL-MM-QWEN4-EXP-PERF

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

The row spec's ## Gates said "Speed: nothing ... There is no vLLM denominator for
this model, so when a speed axis does open, the spec must first say what the
denominator is." Two of those sentences are now stale and the third is answerable,
so this says it.

llama-cpp is already a registered oracle with a pin and gateable = yes, and its
recorded scope is "GGUF k-quant speed and memory floors, quant-matched against the
same weights". unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ1_S is one published artifact
both engines can run, which is exactly that condition. The axis opens when W6a makes
the file loadable and not before.

The developer's target is binding rather than aspirational: faster than llama.cpp on
this model, especially at high concurrency. The gate is written to match, because a
c=1 number would neither confirm nor refute it — a concurrency ladder is the headline,
prefill and decode are reported separately since input length splits them, memory is
an axis rather than a footnote, and llama.cpp runs in its production configuration.
This repository already has the --enforce-eager precedent for what a handicapped
denominator does to a comparison.

The larger part of this change is the four levers, and they are recorded now because
three of them constrain waves that have not started. The QSA one is the reason for the
urgency: #27739 records that a sparse mask over a dense cache costs the same as dense
attention under CUDA flash attention, because flash_attn_mask_to_KV_max only scans
back to the first tile that is not all -inf. #27742 is mask-only and therefore buys
correctness without decode speed. W4 can forfeit the long-context lever by accident,
by building the obvious thing, so the spec now says to build the gather.

Likewise W6a's n-gram residency: #27742 pins that table to the CPU by tensor class
regardless of -ngl, so every token's 16 gathers are host work, and at batch B that is
16xB uncoalesced random gathers — host versus device there is a scaling difference and
not a constant. At IQ4_NL the table is ~28.8 GB inside a 67.56 GiB file against ~119.6
GiB usable, so device-resident and quantized is available to us. The spec now requires
that decision to be asserted rather than defaulted.

Both llama.cpp PRs are UNMERGED and every lever above is a reading of a pinned SHA,
not a measurement. No number is claimed here, nothing has been benchmarked, and no arm
runs yet. The no-ceiling rule is restated in place, because the moment to write it down
is before a first measurement disappoints rather than after.

Tracked by #1990, under #1978.

Gates: check-agent-record ok, check-model-checklist ok.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]

…t is to beat llama.cpp at concurrency

The row spec's `## Gates` said "Speed: nothing ... There is no vLLM denominator for
this model, so when a speed axis does open, the spec must first say what the
denominator is." Two of those sentences are now stale and the third is answerable,
so this says it.

`llama-cpp` is already a registered oracle with a pin and `gateable = yes`, and its
recorded scope is "GGUF k-quant speed and memory floors, quant-matched against the
same weights". `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S is one published artifact
both engines can run, which is exactly that condition. The axis opens when W6a makes
the file loadable and not before.

The developer's target is binding rather than aspirational: faster than llama.cpp on
this model, especially at high concurrency. The gate is written to match, because a
c=1 number would neither confirm nor refute it — a concurrency ladder is the headline,
prefill and decode are reported separately since input length splits them, memory is
an axis rather than a footnote, and llama.cpp runs in its production configuration.
This repository already has the `--enforce-eager` precedent for what a handicapped
denominator does to a comparison.

The larger part of this change is the four levers, and they are recorded now because
three of them constrain waves that have not started. The QSA one is the reason for the
urgency: #27739 records that a sparse mask over a dense cache costs the same as dense
attention under CUDA flash attention, because `flash_attn_mask_to_KV_max` only scans
back to the first tile that is not all -inf. #27742 is mask-only and therefore buys
correctness without decode speed. W4 can forfeit the long-context lever by accident,
by building the obvious thing, so the spec now says to build the gather.

Likewise W6a's n-gram residency: #27742 pins that table to the CPU by tensor class
regardless of -ngl, so every token's 16 gathers are host work, and at batch B that is
16xB uncoalesced random gathers — host versus device there is a scaling difference and
not a constant. At IQ4_NL the table is ~28.8 GB inside a 67.56 GiB file against ~119.6
GiB usable, so device-resident and quantized is available to us. The spec now requires
that decision to be asserted rather than defaulted.

Both llama.cpp PRs are UNMERGED and every lever above is a reading of a pinned SHA,
not a measurement. No number is claimed here, nothing has been benchmarked, and no arm
runs yet. The no-ceiling rule is restated in place, because the moment to write it down
is before a first measurement disappoints rather than after.

Tracked by #1990, under #1978.

Gates: `check-agent-record` ok, `check-model-checklist` ok.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]
@localai-bot

Copy link
Copy Markdown
Collaborator Author

Gate state, stated rather than implied.

check-agent-record and check-model-checklist pass. The full agent-preflight.sh exits 1 on test_cpu_x86_llamacpp_floor, and that red is not attributable to this diff:

  • this change is a single markdown file, +50/-3 to .agents/specs/qwen4-exp-flash-next.md, and cannot reach a CPU floor harness;
  • the failing case is test_a_contended_leg_is_discarded_and_never_summarised, asserting returncode == 2 (GIVING_UP) and getting 4 (NO_QUIET_WINDOW), with the harness reporting busy=103% load=33.93;
  • that is exactly #618, whose own text says the load-dependent exit means "the guarantee goes untested and the red reads as a defect in whatever diff is in flight";
  • host loadavg was 157 at the time, from five parallel implementation agents building this row's other waves.

So the gate could not find a quiet window in which to test its guarantee. It is being re-run on a quiet host before this merges; it is not being waived, and this comment exists so that the re-run is owed in writing rather than remembered.

…main was actively misleading parallel waves

The spec's `## Hardware` table said `unsloth/Qwen3.8-Flash-Next-GGUF` was "README
only, zero weight files -- does not exist". True when written, false a few hours
later: the repository was populated at 13:32Z with three shards totalling
72,546,461,344 bytes = 67.56 GiB, and it FITS GB10 with roughly 52 GiB of headroom
where every safetensors artifact does not fit at all.

The correction is being made here, on a records-only branch, because of where it was
first made. It went into the W1 branch alongside product code, W1 then failed its
fresh review, and the correction is now stuck behind a repair. Meanwhile W3's
implementer read `main`, found the spec asserting the artifact does not exist,
correctly flagged it as contradicting the issue it had been given, and had no way to
know which side was stale. That is the cost of bundling a time-critical record fix
into a change that can fail for unrelated reasons, and it is the reason AGENTS.md
allows a record-only pull request when the record IS the work.

The whole shape of this row's `## Work breakdown` follows from that file existing, so
the correction is called out in place rather than quietly applied. It also carries
what the file independently confirms -- `ple.layer_multipliers`, `ple.head_vocab_sizes`
and a 0-based `ple.layers = [1]`, all matching this spec's own derivation -- and the
two things in OUR tree that still stop us loading it, both W6a's: no `case 20` for
IQ4_NL in the reader, and `KeepQuantKDim` returning -1 for a gather table.

A revision and a per-shard sha256 are recorded as OWED rather than invented. The
repo's `lastModified` moved again after this row first read it, which is precisely the
re-quantized-in-place case AGENTS.md names when it says a repo id alone is not a pin.

Tracked by #1990, under #1978.

Gates: `check-agent-record` ok, `check-model-checklist` ok.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants