feat(KV-FP8): W3 -- half-sized KV blocks the runner actually serves, and a default scale that knows it was declared (#1593)

[localai-bot]

W1 landed the CPU fp8-e4m3 KV store and read, W2 the CUDA arm, and both landed
with nothing reaching either from a production entry point. W3 is the wiring:
--kv-cache-dtype on the server flag, the checkpoint's own kv_cache_quant_algo
honoured when no flag is typed, KV blocks sized at one byte per element, and the
k_scale/v_scale path.

Now based on main. W2 merged as squash 53f5b74be, so this branch's
previous base row/KV-FP8-W2 is dead and GitHub was computing mergeability
against it. origin/main is merged in and the conflict set was re-derived here,
not inherited.

Issue: #1593 (the issue that carries W2
and W3). Spec: .agents/specs/fp8-kv-cache.md
## W3 — the runner integration.

What the FIRST fresh review found, and what changed

The review of 68d2e0d5a returned FAIL — not safe to merge, with one
functional bug. Every finding is repaired here, and the second review re-proved
every one of them by mutation.

The bug: --kv-cache-dtype fp8 threw at the first forward on every real checkpoint

FullAttnBlockPaged cast K and V to the cache dtype only under
if (kv.dtype == DType::kBF16), and on the fp8 route kv.dtype is kI8. So K
still arrived as f32 — attn_dt is f32 for every fp8 cache, because
kv.dtype == DType::kBF16 is a term of both FA2 eligibility tests — while V
arrived as whatever the v_proj GEMM emitted. That is bf16 on the block-wise
fp8 arm (MatmulFp8BlockScaledD), bf16 on the NVFP4 arm under the default
VT_BF16_GEMM_OUT, and bf16 on ordinary torch safetensors (MatmulBf16D).
vt::ReshapeAndCacheFp8 quantizes from ONE source dtype, so the flag threw
reshape_and_cache_fp8: k/v must share one float dtype at the first forward on
every weight arm a user has — including the #1574 campaign checkpoint this
wave exists to serve, so W3 as written did not deliver the matched comparison
it is on the critical path for.

The gate could not see it. MakeDenseWeights builds projection weights with no
nk flag, ProjectFullAttnQkv serves those through MatmulF32D, and f32 pairs
with f32. Only the per-tensor fp8 arm and that synthetic arm ever paired.

The fp8 route now normalises both to bf16, exactly as the bf16-cache route
already did, reusing the two DBufs that were already allocated. bf16 rather
than f32 because bf16 is the dtype upstream quantizes from: vLLM's model IS bf16
where reshape_and_cache_flash takes key/value
(cache_kernels.cu:314-401). G9 is the new case, and it enters through
LoadedEngine with the raw torch Linear layout ([N=out, K=in], nk) a real
safetensors checkpoint ships.

The merge conflict in model_loader.cpp was SEMANTIC, and both sides survive

main's SPEC-DRAFTER-CHAIN W1 refusal (#1522) and this row's resolution stanza
both insert as the FIRST statement of FromModelDir, and both are load-bearing:
that row's G5 requires the chain refusal before any weight I/O, and
ReadQuantConfigJson opens a file inside model_dir. Taking either side drops a
guarantee. Both survive, chain refusal first, and the order is gated.

tests/vllm/entrypoints/test_drafter_chain_reach.cpp cannot catch an inversion:
it points at a NONEXISTENT directory, and ReadQuantConfigJson answers "" for
one of those without opening anything, so the chain refusal arrives either way.
G10's ordering case points at a directory that EXISTS and declares fp8, so an
inverted order announces the declaration first — and that line is the evidence.
Mutation M19b MOVES the refusal below the stanza (rather than deleting it)
and reddens exactly that assertion.

The keyed records — .agents/engine-matrix.md, .agents/quantization-matrix.md,
.agents/issue-index.md — take the target-branch version with the scoped edit
re-applied by hand. The union merge had duplicated two issue-index rows
(#1593 and #1636 appear once on main and twice after the automatic merge);
that is the known union-driver trap and it is undone here.

Three surfaces that no gate reached

Surface	The review's mutation	Now gated by
FromModelDir's resolution stanza + ReadQuantConfigJson, whose only caller it is	deleting the whole stanza left 19/19, 89/89 SUCCESS	G10, through LoadedEngine::FromModelDir on a real directory
the --kv-cache-dtype parse arm and engine_params.kv_cache_dtype = args.kv_cache_dtype	deleting both left the same green, and no test in the tree named the flag	new tests/vllm/entrypoints/openai/test_serve_kv_cache_dtype.cpp, driving the REAL VllmServerMain
ApplyCacheDType's per_layer_attn_specs loop	deleting it left the same green	G11, on the arithmetic it claims: every non-null spec retyped, and KVBytesPerBlock exactly halved

The serve gate works on a polarity rather than on a printed value: the loader
announces the checkpoint's declaration only when nothing explicit was typed, so
CASE 1 (no flag) requires the line and CASE 2 (--kv-cache-dtype bfloat16)
requires its absence. Break either half of the flag's chain and the default
auto arrives, the checkpoint wins, and the line comes back.

The file precedence was inverted against upstream

ReadQuantConfigJson read hf_quant_config.json first and fell back to
config.json:quantization_config. vllm/transformers_utils/config.py:751-761
does the reverse, under upstream's own comments: ModelOpt writes the inline
document from 0.31.0 on, and the standalone file is what 0.29.0 and before wrote,
consulted only when the inline one is None. Mirrored, and gated at both the
resolver and the loader by G10 — a checkpoint re-quantized in place, inline
document added and stale legacy file left beside it, otherwise resolves to the
OLD declaration and quantizes a KV cache nobody asked to quantize, at half the
page, silently.

What turning it on costs, recorded rather than claimed

An fp8 KV cache takes the model off every fast attention kernel this engine
has. qwen3_5.cpp makes kv.dtype == DType::kBF16 a term of both
fa2_prefill and fa2_decode; src/vt/cuda/cuda_paged_attn.cu:2941-2960 routes
an fp8 read only through tiled prefill and block decode, because the WMMA ladder,
the vendored FA-2 launchers and the vectorized decode-opt/GQA kernels are
bf16-native by construction.

W3's rationale is that fp8 KV makes campaign #1574 a matched comparison, where we
currently lead 11.06 vs 9.71 tok/s on the bf16 path. Turning it on plausibly
LOSES that lead, and this session had no GPU and measured nothing. So the spec's
## W3, docs/USAGE.md, docs/FEATURES.md and both matrix rows now say the
exchange rate is unmeasured, and the fast-kernel routing is named under
## Owed. No direction is claimed.

What the SECOND fresh review found, and what changed

The review of 947c5b65f returned FAIL, with one blocking finding. Every
finding is repaired here, and the first round's repairs are untouched.

BLOCKING: the shared seam's fp8 routing was DEAD CODE

dense_attn::AttnBlock is named in AGENTS.md ## Shared seams as the decode
seam, and W3 listed it as routed. It was not. The routing was written — fp8_kv,
WriteKvCache, ApplyKvCacheQuant — underneath a preamble guard that still
admitted only kBF16 and kF32, and IsFp8KvCache is true only for kI8. So
fp8_kv was provably false at every call and neither fp8 arm could execute. Not
an inference: a contradiction between two lines of the same function.

The gate was blind to it. The reviewer reverted the ENTIRE routing from the
seam and got 26 cases / 120 assertions SUCCESS, because every case in the file
entered through Qwen3_5DenseModel::Forward and none entered here.

The seam is the production forward for Qwen3 dense (qwen3.cpp:185), Qwen3-MoE
(qwen3_moe.cpp:84), Voxtral (voxtral.cpp:102) and the Llama, Mistral and
InternLM2 registries that share Qwen3DenseModel. --kv-cache-dtype fp8 on any
of them sized the pool at half and then threw a message naming neither fp8, nor
the flag, nor the row.

The guard is widened exactly as qwen3_5.cpp:5313 already was, and G12 is
the new gate. It enters through Qwen3DenseModel::Forward, which is what
ForwardQwen3ForCausalLM calls under ModelRegistry::Forward
(qwen3_dense.cpp:113), and it asserts three things a "it ran" case would not:
the half-width pages carry bytes after the forward, two fresh stacks agree bit
for bit, and the fp8 logits DIFFER from the same forward over a bf16 cache
(320/320, max |delta| 3.39e-4) — which is what says the read really dequantized
rather than the cache behaving like a float one. Mutation MSEAM reverts the
seam's routing the way the reviewer did and reddens both cases.

The #1574 checkpoint does NOT take the declared-fp8 path, and five places said it did

Verified from the live artifact @ 36f717a2: config.json:quantization_config
carries quant_method: "modelopt", quant_algo: "MIXED_PRECISION" and no
kv_cache_* key at all; only hf_quant_config.json declares
kv_cache_quant_algo: "FP8". The first round's own file-precedence repair makes
the inline document win, so the declaration is never read for this checkpoint —
on this engine and on vLLM alike.

Nothing is functionally wrong; the mirror is correct on both halves. What was
wrong is what the spec, kv_cache.h, cache.h, model_loader.cpp and this body
said about it. The campaign consequence, stated plainly: for #1574 the flag has
to be typed explicitly on both sides. The competitors' own serve.sh already
passes --kv-cache-dtype fp8, so this is consistent rather than a blocker. A new
G10 case writes BOTH real documents into one directory and pins the answer at
auto.

The modelopt marker: NOT a divergence, and the reason took running upstream

The review reported src/vllm/config/cache.cpp accepting producer.name as a
divergence, on the grounds that get_kv_cache_quant_algo_string
(torch_utils.py:319) gates on a top-level quant_method and nothing else. That
transcription is of one function; the answer needs two.
ModelArchConfigConvertorBase._normalize_quantization_config
(transformers_utils/model_arch_config_convertor.py:208-247) runs FIRST, at
ModelConfig.__post_init__ (config/model.py:577), and INJECTS
quant_cfg["quant_method"] = "modelopt" when producer["name"] == "modelopt",
mutating the very dict hf_config.quantization_config names.

MEASURED by extracting both functions with ast from the files at 555967922
and running them over the live documents rather than retyping them:
nvidia/Llama-3.3-70B-Instruct-FP8's producer-only hf_quant_config.json
answers None before normalization and 'fp8_e4m3' after it. So accepting the
producer name is the MIRROR. The comment that claimed it on the wrong grounds
("modelopt_mixed_precision.h already reads both shapes") is replaced by the
chain and the measurement.

Reading the injector did surface three real differences, and the marker set is
now exactly upstream's:

• quantization.quant_method is no longer accepted. Upstream writes and reads
  the marker at the TOP level and never looks inside, so a document whose only
  marker is nested resolved fp8 KV here and nowhere else.
• A nested modelopt_quant_config key is now accepted (:218-220), which
  upstream treats as the legacy modelopt marker and we ignored.
• The producer name is compared raw against the literal (:222 is ==), not
  prefix-matched and case-folded. quant_method keeps both, because :238-246
  lower-cases it before :319 prefix-tests it. Same file, two tests, mirrored
  separately.

G1's new marker case pins all six arms; restoring the nested-marker widening
reddens two of them.

One difference remains and is recorded rather than copied. The injector
RAISES ValueError: Unknown ModelOpt quant algo: <algo> (:235) when the
producer is modelopt and the nested quant_algo is neither FP8-family nor
NVFP4 — which is what the #1574 legacy file (MIXED_PRECISION) gets, and we
answer fp8_e4m3 for it instead. That refusal is a WEIGHT-half validation
living in a config convertor this port does not have, and moving it into the KV
resolver would refuse a checkpoint whose weights modelopt_mixed_precision.h
loads. It is unreachable for that checkpoint anyway, because its inline document
wins. Spec ## Owed names it.

The refusal mechanism was wrong for 13 of the 16 unrouted architectures

"Every other architecture is refused BY NAME at vt::ReshapeAndCache" holds for
three. granite:95, minicpm:96, phi3:78, gemma3:121, opt:125,
stablelm:86, glm4:93, commandr:93, gemma:53, gemma2:135, phi:98,
muse_glimmer:144 and olmo2:94 each refuse at their own
"<arch>: KV cache must be bf16 or f32" guard first, which names neither fp8
nor the flag. Only gemma4, qwen3_vl and nemotron_h_device reach the store
guard. The safety property holds — nothing writes floats into a half-sized page
— but the described mechanism did not, and the count was 16 architectures at 17
call sites rather than 17 architectures. Round 3 traced each of the three and
found only qwen3_vl reaches the store guard; the split is 14 / 1 / 1. See
below. Corrected in the spec, in
docs/USAGE.md, in docs/FEATURES.md and in the engine-matrix row, and the
13-guard gap is now its own ## Owed bullet.

docs/FEATURES.md promised a doubled pool on a path that halves it

ResolveNumBlocks (model_loader.cpp:1446-1451) returns the fixed 256-block
fallback when neither --num-blocks nor --kv-cache-memory is given, so on the
default path fp8 KV halves the pool BYTES rather than doubling the pool. The
one-liner said "halves the block, doubles the pool" unqualified;
docs/USAGE.md's example passes --kv-cache-memory and was correct, and now
says so explicitly.

Preflight was reporting nothing about two gates

origin/main had moved to 08c81a89 and was no longer an ancestor, so the
diff-scoped commit-trailers and commit-style checks SKIPPED. origin/main is
merged up and both execute.

What the THIRD fresh review found, and what changed

The review of d0bcf80dd returned FAIL, with one blocking finding. Every
finding is repaired here, and rounds 1 and 2 are untouched. The round-2
producer.name MEDIUM was FALSIFIED by that review, which ast-extracted both
pinned upstream functions and ran them: _normalize_quantization_config mutates
hf_config.quantization_config in place during ModelConfig.__post_init__,
before resolve_kv_cache_dtype_string reads it, so accepting producer.name is
the mirror. It stays.

BLOCKING: the gate could not distinguish a working fp8 KV cache from a corrupted one

G12's counter-case asserted differing > 0 and max_abs < 1.0 on a delta its
own MESSAGE reported as 0.000339303. The bound sat about 3000x above the
signal, so it measured this toy model's insensitivity to its KV cache and not
the cache. Two mutations of kv_cache_route.h:63 walked through the whole 30/30
suite:

Mutation	Delta on the old axis	Old suite
N1_KVSWAP — pass (q, v, k, k_cache, v_cache, ...), so the fp8 arm stores V into k_cache and K into v_cache	0.0247495 (73x)	30/30 SUCCESS
N4_STORE_SCALE8 — store with kv.k_scale * 8.0F / kv.v_scale * 8.0F while the read uses the unscaled values	0.0063844 (19x)	30/30 SUCCESS

The second one is exactly the defect the k_scale/v_scale plumbing exists to
prevent. Nothing above G12 closed the hole either: G5 and G9, the two
production-entry cases, assert token counts and determinism across two fresh
stacks, and neither compares an fp8 run against a bf16-cache run.

The repair compares the CACHE BYTES, inside an envelope read off the FORMAT.
A new G12 case runs the same prompt twice — once with a bf16 page, once with an
fp8 page — and checks every layer-0 element of both K and V:

|dequant(store(ref)) - ref|  <=  2^-4 * |ref|  +  2^-10 * scale

e4m3fn carries three explicit mantissa bits and rounds to nearest even
(include/vt/fp8_kv.h), so a normal magnitude is within a half ulp — at most
2^-4 of its own value — and anything below the smallest normal 2^-6 sits on
a uniform 2^-9 grid whose half step is 2^-10, in absolute terms scaled by the
per-tensor scale. Not one constant is fitted to a measurement, which is the point:
a threshold sized to today's 3.4e-4 is the same hole one decimal place tighter.

Layer 0 is the whole population, and that is the design rather than a
limitation. Its K and V are functions of the embedding and the input layernorm
alone, so the bf16 run and the fp8 run hand the store bit-identical floats and
the float run's page IS the reference the fp8 page has to round. From layer 1 on
the fp8 run's inputs already carry the previous layer's dequantization and no
per-element envelope holds.

The scales are 0.125 for K and 0.25 for V — non-unit and unequal, so a
dropped, swapped or one-sided scale IN THE STORE leaves the envelope rather than
staying inside it. Both are below one because the store divides by the scale and this
model's layer-0 K and V land around 1e-2; a scale above one pushes most
elements under 2^-6, where only the weaker absolute arm applies. Two anti-vacuity
REQUIREs hold the population to that: every one of the ten pages (five tokens,
K and V) must carry at least one NORMAL, and no reference may reach e4m3's finite
maximum, so the bound is the pure rounding envelope and never a clamp.

MEASURED on the repaired tree, and both mutations are RED against it:

Tree	Elements outside the envelope	Worst ratio to the bound
repaired	0 / 320 (281/320 normals over 10/10 pages)	0
N1_KVSWAP	316 / 320	416.9
N4_STORE_SCALE8	319 / 320	13.9

A bf16-versus-fp8 comparison in G5/G9 through LoadedEngine was considered and
is NOT what closes this. Both STORE mutations left every other case green, G5 and
G9 included, which is the direct measurement that an engine-level token or
determinism comparison cannot see a dequant defect this model absorbs. The old
two assertions are kept beside the new case, not replaced.

Round 4: that envelope gated the STORE, and the READ was still open

The fourth review found the hole the third one's repair left. The envelope case
decodes the cache bytes with its own vt::LoadKvFp8E4M3 and never enters the
production dequant (cpu_paged_attn.cpp:167); the one value in the case that IS
downstream of that dequant carried no assertion at all. Every other case in the
file that asserts a number downstream of the read runs at
k_scale == v_scale == 1, where a k/v scale SWAP on the read is arithmetically
inert. Mutating the production read to const float v_scale = args.k_scale;
therefore left test_kv_cache_fp8_wiring at 31/31 and test_ops_fp8_kv_cache at
8/8, both SUCCESS, while every V the softmax saw was halved.

The read's SCALE is closed by INVARIANCE, and EXACTLY rather than by a
tolerance — its ROUTING is not, and that half is under ## Owed.
e4m3fn's normal grid is relative — for |y| in [2^e, 2^(e+1)) the
representable points are m * 2^(e-3) — and dividing by a power of two is exact
in binary floating point, shifting e without touching the mantissa. So for any
two power-of-two scales that both leave a value normal and unsaturated,

s * Dequant(Quantize(x / s))  ==  s' * Dequant(Quantize(x / s'))

bit for bit: the two runs' cache BYTES differ in every element's exponent field,
and the floats the attention kernel is handed do not. The case runs the seam
twice more, at (2^-7, 2^-13) and (2^-11, 2^-9), and requires memcmp-level
agreement on the logits. There is no constant to fit and none to widen later,
which is the same discipline the envelope above is written to. Both pairs move
BOTH sides on purpose: a pair that moved only one would let a read-side defect
that depends on the other scale reproduce itself identically in both runs and
cancel out.

That exactness is ASSERTED for LAYER 0, and the logit equality is an EMPIRICAL
result for this fixture rather than a theorem. The scale_exact REQUIRE
decodes bf16.buf[0] and holds 320/320 LAYER-0 elements normal and unsaturated
at all four scales — the measured magnitudes are 1.76e-4 to 1.32e-1 for K and
5.41e-5 to 4.22e-2 for V, against all-normal windows of
(2.94e-4, 1.13e-2] and (9.43e-5, 3.46e-3] — because one subnormal would round
on the ABSOLUTE 2^-9 grid, which is not scale invariant. MakeSeamConfig sets
num_hidden_layers = 2, so that is 320 of the 640 elements each run stores,
while the logits the case compares are a function of BOTH layers' caches. Layer 1
does NOT satisfy the precondition: decoding each run's own buf[1] at its own
scales, 3/320 of its elements disagree by up to 7.62939e-06, all K-side and
an order of magnitude below layer 0's 1.76e-4 minimum, so at
kInvAKScale = 2^-7 they land in e4m3's SUBNORMAL region and that layer carries
2 SATURATED elements besides. CHECK(inv_differing == 0) therefore holds by the
format property for layer 0 and by ABSORPTION for layer 1 — a 7.6e-6 cache
perturbation vanishing in f32 accumulation before it reaches a logit. Extending
scale_exact over every layer was the preferred repair and this fixture cannot
satisfy it, so the claim is narrowed here and the fragility is recorded under
## Owed.

The second anti-vacuity REQUIRE is that the two caches really do hold different
bytes, so the comparison is a statement about the read rather than about two
identical buffers.

MEASURED on the repaired tree: 0/320 logits differ, max |delta| exactly 0.
Three read-side mutations of cpu_paged_attn.cpp:167 are RED, and the store
envelope above reads 0/320 under all three:

Read mutation	Logits differing	max delta
v_scale = args.k_scale	320 / 320	0.0673
k_scale = args.v_scale	256 / 320	1.08e-4
v_scale = 1.0F	320 / 320	2.47e-3

The K-side one is why this is stated as exact equality rather than a bound. The
three signals span 621x, so a bound sized against the largest keeps nothing
for the smallest: even a tight 12.4x margin against 0.0673 puts the constant at
5.4e-3, which admits 1.08e-4 by 50x. An absolute 1e-4 would in fact
have CAUGHT the K-side one, by 8.4%, and a relative 1e-4 against the fixture's
largest logit (0.0627671) is 6.3e-6 absolute and catches it by 17x — the
argument is the span, and the refusal to fit a constant to whichever defect
happened to be measured first, never any one number.

What this comparison still cannot see is a read-side defect that is a
function of BYTES and INDICES rather than of scales, because both fp8 runs commit
it identically and it cancels. Measured on production code, both of these PASS
the whole file at 31/31, 487/487, with 0/320 logits differ: serving V out
of the K page with K's scale (cpu_paged_attn.cpp:174 v_base = k_cache.data
with :167 v_scale = args.k_scale), the read-side twin of the N1_KVSWAP
store mutation the envelope DOES catch; and dropping the in-page token offset
from the V read (:270, off & 0), which is pure indexing and scale-free.
Closing that class needs a comparison against a REFERENCE — the bf16 run's page,
or the case's own decode — and never a second fp8 run. It is under ## Owed.

The same review found scripts/agent-integration.py --base origin/main exiting 1
with commit-trailers and commit-style both SKIPPED: origin/main had moved to
c98ffd4d0 while this branch's last merge took ff8f72807, so neither gate
reported anything about this tree. main is merged up and both execute again.

Round 5: the code was correct, and two sentences about it were not

The fifth review passed the change and rejected two claims written around it. It
did not ask for a production edit and none was made; the two paragraphs above are
where the repair lands, and both numbers in them were re-measured here rather
than carried over. origin/main had moved again, to ea9b7e30e, so it is
merged up once more and commit-trailers and commit-style execute against
this tree rather than SKIP on a base that has moved.

The first was the tolerance sentence. "A tolerance sized at 1e-4 would have let
it through" is false as stated and false on either reading: 1.08408e-4 is 8.4%
ABOVE an absolute 1e-4, and a relative 1e-4 against the fixture's largest
logit 0.0627671 is 6.3e-6 and catches it by 17x. The argument that survives
is the 621x SPAN between the three read signals, not that constant.

The second was the exactness scope. The comment said the invariance gate is exact
"as a property of the format rather than a lucky measurement" and that
"the scale_exact REQUIRE below holds the population to it". scale_exact
decodes bf16.buf[0], which is 320 of the 640 elements each run stores, and the
logits it compares depend on both layers. Extending the assertion over every
layer was the preferred repair and the fixture cannot carry it: layer 1 measures
3/320 elements disagreeing by up to 7.62939e-06, subnormal at 2^-7. So the
prose is narrowed to what is asserted, the empirical part is named as empirical,
and the fragility and the unclosed routing half are both under ## Owed.

The ## Owed refusal accounting was wrong for 2 of the 3 architectures it named

Round 2 said gemma4, qwen3_vl and nemotron_h_device reach the store guard.
Traced with a kI8 page against a bf16 model dtype, only qwen3_vl does. The
true split is 14 / 1 / 1:

• gemma4.cpp:306-315 takes kv.dtype != adt, allocates
  DBuf kcast(d, kv.dtype /* kI8 */, ...) and calls vt::CastF32, which refuses
  at src/vt/ops.cpp:4087 with "cast_f32: out must be f32". It never reaches
  the store, and that message names neither fp8, nor the flag, nor gemma4 — so
  it is the WORST of the sixteen, not one of the three best.
• nemotron_h_device.cpp:1589-1593 carries an explicit
  else { VT_CHECK(false, "NemotronH paged forward: ... The fp8 KV scheme ... is not selected here"); } on the same cast, which fires first. It is the ONE
  refusal in the sixteen that names fp8 to the operator.
• qwen3_vl.cpp:198-200 carries no guard and no cast, so vt::ReshapeAndCache
  is what refuses it.

Safety is unaffected — nothing writes floats into a half-sized page on any of the
sixteen — and what moves is message quality: the store guard's better message is
reached by 1 of 16, not 3. Corrected in the spec's narrative and in its ## Owed
bullet, which now also records that G7 hand-builds its K/V tensors and calls
vt::ReshapeAndCache directly, so it gates the store guard's MESSAGE and never
the claim that any particular architecture reaches it.

The marker mirror accepted three shapes upstream answers None for

_normalize_quantization_config injects quant_method only
if quant_algo is not None (:224), and it reads that key out of
quant_cfg.get("quantization", {}) — an EMPTY-object fallback, unlike the
reader's quant_cfg.get("quantization", quant_cfg) at torch_utils.py:321.
cache.cpp accepted the producer name and the legacy key unconditionally, and
read the legacy key out of the reader's inner. Three documents therefore
resolved to fp8_e4m3 here while upstream resolved None:

Document	Upstream	Before	Now
producer.name == "modelopt", quantization with no quant_algo	None	fp8_e4m3	auto
quantization.modelopt_quant_config with no quant_algo	None	fp8_e4m3	auto
top-level modelopt_quant_config, no quantization key at all	None	fp8_e4m3	auto

Each of the three would halve the KV page on a checkpoint vLLM runs at the model
dtype. Arm (g) of G1's marker case pins all three, alongside the same
documents made acceptable by adding the quant_algo upstream requires, so what
refuses them is the guard and not the shape. No shipped fixture moved: every
real document in the suite carries a quant_algo.

The MIXED_PRECISION raise stays under ## Owed, and that is why the mirrored
test is exactly quant_algo is not None rather than the FP8/NVFP4 family set:
the only two upstream outcomes for a quant_algo that IS present are "inject"
and "raise", and taking the raise's arm collapses them into one. Importing the
raise would refuse a checkpoint whose weights this tree loads.

This PR falsified four statements elsewhere and now carries the repair

Before this branch nothing in production read hf_quant_config.json.
src/vllm/config/cache.cpp:207 now does, from LoadedEngine::FromModelDir
(model_loader.cpp:1988). Four statements written against the old world read
false at this head, and a record edit rides in the pull request whose change made
it stale:

Site	Was	Now
modelopt_mixed_precision.h:1006	"a file no production path in this tree reads at all"	reads it, but only as the LEGACY FALLBACK behind config.json's inline document, and this checkpoint ships that document
.agents/specs/qwen38-27b-quant-arms.md:1629	"the declaration is invisible to the loader rather than ignored by it"	the same conclusion, reached through the precedence rather than through absence
.agents/specs/qwen38-27b-quant-arms.md:1841	"No production path in this tree reads hf_quant_config.json"	names the reader, the fallback order, and that BOTH artifacts carry the inline document
test_qwen38_27b_modelopt_mtp_arm.cpp:482 (case NAME) and :487-488	"where NO production path reads it" / "the ONLY document any production path in this tree reads"	"in the file this artifact's config.json outranks", with the precedence spelled out

No assertion changed. They stay valid because config.json's inline
quantization_config wins on the one path that reads either.

The 14 / 1 / 1 split had to reach four more records

The recount above landed in the spec and stopped there. docs/USAGE.md,
docs/FEATURES.md, the KV-FP8 engine-matrix row and the QUANT-KV-FP8
quantization-matrix row — all four written by THIS pull request — still told a
reader that Gemma-4, Qwen3-VL and Nemotron-H are the three that name the flag.
None of them names the flag. All four now describe what an operator actually
sees, and the two matrix rows take the new gate count (31 cases, G1-G12) with one
line on what G12's third case bounds; the quantization-matrix row was still
saying 26 cases, G1-G11, and 17 unrouted architectures.

A GCC 13 -O2 build of this branch did not compile

KvAlgoFromObject's const auto num_bits = cond ? std::optional<int>(...) : std::nullopt; fails -Werror=maybe-uninitialized at -O2 on GCC 13.3, which is
every Release and RelWithDebInfo build of this tree. CI configures with no
CMAKE_BUILD_TYPE and never saw it; build-newest-gcc uses Release on a
newer compiler that does not emit it. Written as a statement instead, and the
translation unit now compiles clean under
-O2 -g -DNDEBUG -Wall -Wextra -Werror.

Gates

tests/vllm/entrypoints/test_kv_cache_fp8_wiring.cpp — 31 cases / 481
assertions GREEN, and
tests/vllm/entrypoints/openai/test_serve_kv_cache_dtype.cpp — 3 cases / 26
assertions GREEN, plus tests/vllm/models/test_qwen38_27b_modelopt_mtp_arm.cpp
— 22 cases / 1687 assertions GREEN, whose case name this round narrowed.
CPU-only, -Wall -Wextra -Werror, zero warnings over a full build. The
src/vllm/config/cache.cpp translation unit is compiled a second time at
-O2 -g -DNDEBUG for the GCC 13 -Werror=maybe-uninitialized repair above,
because CI configures with no CMAKE_BUILD_TYPE and cannot see it.

ctest 584/584 PASSED, 0 failed on the pushed head, on an idle box, over the
whole tree built from scratch with -Wall -Wextra -Werror and zero warnings.
test_runner PASSES here in 36.9 s, so the CI red
(#1602) is not reproducible on
this host; it is still counted as the pre-existing red rather than as fixed,
because nothing in this branch touched it. Four suites are Skipped for absent
checkpoints or an absent config:
test_modelopt_mixed_precision_checkpoint, test_voxtral_e2e,
test_cli_offload_config, test_qwen35_paged_engine. The count is 584 rather
than CI's 593 because this configure sets -DVLLM_CPP_BUILD_EXAMPLES=OFF.

The round-2 measurement, kept for the record:
ctest 591/593 after the second round, on a CONTENDED box (1-minute load
average 62 to 100 through the run, other sessions building in their own
worktrees). test_runner is the pre-existing red
(#1602) and fails on an
attention-backend selection message under a non-multiple-of-16 block size, which
this change does not touch; it fails the same way on a serial re-run
(20 cases, 543/544 assertions), so it is a code red that predates this branch and
not a load artefact. test_engine_core_proc failed under ctest -j 4 and
PASSES on a serial re-run (14 cases / 118 assertions, Status: SUCCESS!), so
it is counted as green rather than as a regression. Three suites are Skipped for
absent checkpoints: test_modelopt_mixed_precision_checkpoint,
test_voxtral_e2e, test_qwen35_paged_engine.

scripts/agent-preflight.sh --staged is green except
test_cpu_x86_llamacpp_floor
(#618), which fails on one run
and passes on the next depending on the box's load: its harness leg is DISCARDED
and RETRIED under contention and the case asserts returncode == 2 against the
4 the retry returns. Nothing here was repaired.

CI READ on this head (b7921ceb0, run
32627552373,
CONCLUDED): 17 pass, 6 skipping, 3 fail, and all three are pre-existing and
filed.

build-test-cpu PASSES here, 593/593. test_runner
(#1602) is green in that lane
and in both sanitize lanes on this head, and green locally in 36.9 s. On the
previous head it was the ONLY failing test in all three of build-test-cpu,
sanitize-cpu (address,undefined) and sanitize-cpu (thread) — one flake
reported as three job failures, not three defects. cuda-fat-build,
build-newest-gcc, build-test-cpu-arm64, build-test-vulkan, both verify
lanes, commit-protocol-tag, documentation-checkpoint, pr-size,
device-leakage, last-gated-commit, cuda-arch-features,
vulkan-spirv-freshness, plan and drift-check all pass.

The three reds:

Job	What it reports	Whose
windows-msvc-cpu	test_openai_api_server.exe exited with status -1073740791 (0xC0000409, STATUS_STACK_BUFFER_OVERRUN)	#584, open, reproduced unchanged on PR #1744
windows-msvc-vulkan	the same executable, the same status	#584
agent-record	FileNotFoundError: [Errno 2] No such file or directory: 'hugo' at tests/scripts/test_check_site.py:130	#1722, #1754, #1764; PR #1726 is the fix in flight

The windows-msvc-* signature MOVED, and that settles what the catch-up merge
bought. Both jobs were failing at step 3, "Prove PowerShell, static CRT, and
unsupported-tier contracts" —
#1649, whose fix 2f2a70925
was NOT reachable from this branch's previous base. On this head step 3 PASSES in
both, and what fails is step 4, on the pre-existing #584 crash. #1649 is closed
for this branch; #584 is not this branch's and is not repaired here.

agent-record's case was added by #1714 (1db7e59cf, on main) and shells out
to a binary the job never installs (grep hugo .github/workflows/ci.yml finds
nothing). It passes here 7/7, because this host has hugo. Red on main and on
every pull request; not repaired here.

commit-trailers and commit-style now RUN. origin/main had moved to
08c81a89 and was no longer an ancestor, so both were SKIPPED and reported
nothing at all about this tree. After the catch-up merge preflight prints
Commit trailers vs origin/main 08c81a89218906cac08209a63c6301f03fdc8ec7: with
both green, alongside now-current range and issue-index append-only.

Twelve mutations from the first round remain valid, eight more were run for the
second, two for the third, and three more for this round, each applied with its sha256 and applied
diff printed, built with its exit status printed, run red, and restored
byte-identically:

Mutation	Gate that went red
M13 the fp8 route stops normalising K/V to one dtype	G9 (the production arm)
M14 the --kv-cache-dtype parse arm is dropped	serve CASE 2 and CASE 3
M15 the parsed flag never reaches EngineParams	serve CASE 2 and CASE 3
M16 the loader never reads the checkpoint's declaration	G10
M17 the legacy file outranks the current one again	G10 (precedence, both halves)
M18 the per-layer specs are never retyped	G11 (9 assertions)
M19 the chain refusal is deleted	G10 (ordering)
M19b the chain refusal MOVES below the stanza	G10 (ordering) — the mutation M19 cannot make
MSEAM the shared seam's fp8 routing is reverted whole (cast_dt back to kv.dtype, WriteKvCache -> vt::ReshapeAndCache, ApplyKvCacheQuant deleted)	G12, both cases — the mutation that stayed GREEN before this round
MMARKER the nested quantization.quant_method widening is restored	G1's marker case, arms (c) and (d)
N1_KVSWAP the fp8 store is handed (q, v, k, ...), so V lands in k_cache and K in v_cache	G12's envelope case: 316/320 elements outside, worst ratio 416.9 — GREEN 30/30 before this round
N4_STORE_SCALE8 the fp8 store uses k_scale * 8 / v_scale * 8 against an unscaled read	G12's envelope case: 319/320 outside, worst ratio 13.9 — GREEN 30/30 before this round
N5_READ_VSWAP the production read dequantizes V with K's scale (cpu_paged_attn.cpp:167)	G12's envelope case, the new invariance half: 320/320 logits differ at max delta 0.0673 — GREEN 31/31 AND test_ops_fp8_kv_cache 8/8 before this round
N6_READ_KSWAP the production read dequantizes K with V's scale	same: 256/320 logits differ at max delta 1.08e-4
N7_READ_VDROP the production read drops the V scale (v_scale = 1.0F)	same: 320/320 logits differ at max delta 2.47e-3
N_LOW1 the marker predicate is restored to its pre-repair form (unguarded producer/legacy key, legacy key read out of inner)	G1's marker case, arm (g): 7 assertions red across all three shapes

The driver prints the applied diff and the build exit status for every one,
because git diff --stat is EMPTY for a file git does not track yet, and a
mutation that never applied — or that failed to build — reads as a passing test.

Unreached and owed

All under #1593 and listed in the spec's ## Owed:

• The C ABI carries no kv_cache_dtype field. A C-ABI caller reaches an fp8
  KV cache only through a checkpoint that declares one, which the loader does
  honour on that path.
• An fp8 KV cache is served by the SLOW attention kernels only (the section
  above). Owned by KV-FP8 under KV-FP8 W2 and W3: the CUDA fp8 KV store, its paged-attention read, and the runner integration — the only thing now gating every cell of the #1574 three-way #1593.
• The heterogeneous per-layer fp8 arm is SIZING-only. G11 gates the
  arithmetic; the only architecture that populates per_layer_attn_specs is
  Gemma-4, which is one of the 16 that refuse the store, so a Gemma-4 run gets a
  correctly halved pool and then a named refusal — the intended order.
• 16 architectures refuse rather than route, at 17 call sites: glm4,
  minicpm, opt, gemma, gemma2, gemma3, gemma4 (two sites),
  commandr, phi, phi3, muse_glimmer, stablelm, qwen3_vl, olmo2,
  granite and nemotron_h_device.
• 15 of those 16 refuse with a message naming neither fp8 nor the flag.
  Thirteen have their own "<arch>: KV cache must be bf16 or f32" guard;
  gemma4 dies earlier still, inside vt::CastF32, with a message that names
  no architecture at all. Only qwen3_vl reaches the store guard, and only
  nemotron_h_device names fp8 to the operator. Widening those 15 is the same
  edit that routes them, so it is owed with the bullet above.
• The resolver accepts one document upstream REFUSES, where the nested
  quant_algo IS present and is neither FP8-family nor NVFP4, and upstream's
  config convertor raises instead of injecting. The quant_algo is not None
  guard itself is now mirrored (round 3 above), so this is the one remaining
  arm. Recorded, argued and unreachable for the Qwen3.8-27B three-way at each engine's best: one checkpoint ours/vLLM/SGLang all serve, and three of the four recorded NVFP4 blockers are properties of the unsloth artifact rather than the format #1574 subject; porting the
  convertor's weight-half validation is its own row.
• No engine auto-selects fp8 KV for the Qwen3.8-27B three-way at each engine's best: one checkpoint ours/vLLM/SGLang all serve, and three of the four recorded NVFP4 blockers are properties of the unsloth artifact rather than the format #1574 subject, so the campaign types
  the flag on both sides and the checkpoint-declaration path this row builds is
  gated by G1 and G10 rather than exercised by the benchmark it was built for.
• No weight loader extracts k_scale/v_scale, so the two
  checkpoint-loaded arms of the resolver are unit-gated and unreached, and the
  scale is per-engine rather than per-layer.
• --calculate-kv-scales is refused, not implemented (upstream deprecates it
  for v0.19).
• Every CUDA case in this feature is still UNEXECUTED, for the reason W2
  recorded: no nvcc and no device in this session, and a CPU build compiles
  zero .cu translation units.
• G12's read-side exactness is ASSERTED for LAYER 0 only (scale_exact
  decodes bf16.buf[0], 320 of the 640 elements each run stores), and layer 1
  carries 3/320 elements that dequantize 7.62939e-06 apart between the two
  invariance runs, in e4m3's subnormal region where the covariance argument does
  not hold. The logit equality is empirical there, absorbed below f32, and a
  change to the fixture, the layer count, the thread count or the accumulation
  order could redden a CORRECT tree with a defect-shaped message.
• The read comparison closes the SCALE half of the read and not the ROUTING
  half. Two fp8 runs cannot see a defect that is a function of bytes and
  indices; a V-from-the-K-page read and a V read that drops the in-page offset
  both PASS the whole file. Closing that class needs a REFERENCE comparison.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]