perf(GFX1100-TG200): the 200 tok/s gfx1100 decode campaign and fp8 KV cache decode attention

.agents/feature-matrix.md

@@ -82,7 +82,7 @@ Confirmed NON-gap: vLLM has removed prompt adapters.
 | SGLang RadixAttention behavior parity (fuse-or-flag) | SGLang v0.5.15 `f63458b` `mem_cache/radix_cache.py`, `managers/schedule_policy.py`, `constrained/outlines_jump_forward.py` | `ACTIVE` T2 | **Scoped 2026-07-27 (`CLAIM-SGLANG-RADIX-SCOPE`); W1+W2 IMPLEMENTED 2026-07-27 (`CLAIM-SGLANG-IMPL`, rows now `ACTIVE`).** VERDICT: SGLang's radix TREE == our block-hash APC ⇒ RadixAttention is **already FUSED**, `--enable-radix-attention` is an ALIAS for the APC toggle (LANDED: server alias + C-ABI `enable_prefix_caching` tri-state, ABI v7). Genuinely-distinct behavior = cache-aware **LPM scheduling `--schedule-policy=lpm`** (LANDED: `SchedulerPolicy::kLPM` reorders the FCFS waiting deque by APC longest-match, ported FROM `schedule_policy.py:205,229`, output-neutral; gate `test_scheduler_lpm` 6/6). **SW2 in-batch prefix-collision de-prioritization LANDED 2026-07-27 (`CLAIM-SGLANG-SW2`)** inside the `kLPM` reorder (block-hash APC keys, no second trie; ported FROM `schedule_policy.py:253-301,311`), output-neutral; its throughput lever is NOT-APPLICABLE — our APC caches at allocation time so the 2nd same-step collider already hits (within-step dedup subsumes it). overlap scheduler == `ENG-ASYNC-SCHED` (fused). **SW3 jump-forward decoding — safe TOKEN-UNIQUE subset LANDED 2026-07-28 (`CLAIM-SGLANG-SW3`)**: forced-token detection hook `StructuredOutputGrammar::forced_token()` + opt-in driver `DrainForcedTokens` (env `VT_ENABLE_JUMP_FORWARD`, default OFF), provably byte-identical to per-token constrained decode (jumps only where the grammar leaves exactly one valid token — no re-tokenization); gate `test_jump_forward` 5/5 (RED-first). Residual: SW4 + the general re-tokenization span + production scheduler splice (named). **ABI/API/flag EXPOSURE 2026-07-28 (`CLAIM-SGLANG-ABI-DOCS`, reconciled to ABI v10):** LPM + jump-forward made first-class DOCUMENTED knobs on ALL THREE surfaces (were server-only / env-only) — LPM via the concurrent session's C-ABI **string** field `vllm_model_params.scheduling_policy="lpm"` (ABI v9; NO duplicate int knob) + C++ `EngineParams::policy=kLPM` + server `--scheduling-policy lpm`; jump-forward via new C-ABI `vllm_model_params.enable_jump_forward` (tri-state int, ABI **v10** appended after the v9 fields) + C++ `EngineParams::enable_jump_forward` + server `--[enable\|disable]-jump-forward`. `VT_ENABLE_JUMP_FORWARD` retained as env override. User docs [docs/SGLANG-COMPAT.md](../docs/SGLANG-COMPAT.md) + spec [sglang-enablement.md](specs/sglang-enablement.md); ABI e2e `tests/capi/test_capi.cpp` (2 v10 jump-forward cases; `vllm_abi_version()`==10). Default-inert (all-zero ⇒ byte-identical). Rows `KV-SGLANG-RADIX-CACHE` + `ENG-SGLANG-BEHAVIOR-FLAG`. Sibling benchmark track = `BACKEND-GATE-CUDA-SGLANG*` (unchanged) | [sglang-radixattention.md](specs/sglang-radixattention.md) |
 | **SGLang parity PROGRAM** (whole-surface inventory + oracle) | SGLang v0.5.15 `f63458b` — full runtime surface | `SPIKE` T2 | **Elevated 2026-07-27 (`CLAIM-SGLANG-PARITY-PROGRAM`).** The vLLM-parity approach replicated for SGLang: a tabular whole-surface inventory (44 rows) classifying every SGLang capability **FUSED (23) / SGLANG-DISTINCT (8) / INVENTORIED (5) / OUT-OF-SCOPE (8)**, plus SGLang stood up as a correctness + performance ORACLE (dgx GB10 via the arm64 cu130 image — no from-source build needed). SGLang is a competitor perf FLOOR + correctness cross-check, NOT the mirror source (vLLM stays behavior truth). Headline SGLANG-DISTINCT opt-ins: LPM scheduling, in-batch prefix de-prioritization, radix eviction strategies, jump-forward, custom logit processors, batch-invariant determinism, PD disaggregation, two-batch EP overlap. Full map + ranked plan in the matrix. Sibling benchmark rows `BACKEND-GATE-CUDA-SGLANG*` unchanged | [sglang-matrix.md](sglang-matrix.md); [sglang-parity-oracle.md](specs/sglang-parity-oracle.md) |
 | SlidingWindowSpec + ChunkedLocalAttentionSpec | `v1/kv_cache_interface.py` | `PARTIAL` T1 | Both execution leaves are implemented: W1 sliding-window and W3 chunked-local sizing, registry/grouping, manager prefix/recycling policy, admission and hybrid-disabled conversion pass their ported CPU/property/sanitizer gates (G1/G2). The compute-locality consumers are now GPU-gated (2026-07-27 `CLAIM-ROADMAP-C5`, dgx GB10: Gemma-2/Gemma-3 sliding-window model gates 48/48; `test_chunked_local_attention` 5/5). The KV memory-OPTIMIZATION path (optimized-manager held-block cap vs the full-allocation fallback the current model gates use) still needs a model-level hybrid-manager memory gate (G8) — kept `PARTIAL` honestly | [sliding-local-yarn-long-context.md](specs/sliding-local-yarn-long-context.md) |
-| fp8 KV cache (`cache_dtype=fp8*`) | `layers/quantization/kv_cache.py` | ◐ T1 | `KV-FP8` ACTIVE / `QUANT-KV-FP8` PARTIAL — W1 CPU fp8-e4m3 store+read+config-parse landed; CUDA + memory-halving e2e later | [fp8-kv-cache](specs/fp8-kv-cache.md) |
+| fp8 KV cache (`cache_dtype=fp8*`) | `layers/quantization/kv_cache.py` | ◐ T1 | `KV-FP8` ACTIVE / `QUANT-KV-FP8` PARTIAL — W1 CPU fp8-e4m3 store+read+config-parse landed; W2 CUDA store+read landed; W6 ROCm store+read landed; memory-halving e2e later | [fp8-kv-cache](specs/fp8-kv-cache.md) |
 | nvfp4 / per-token-head / turboquant KV | `config/cache.py` | ☐ T2 | | `planned: specs/nvfp4-kv-cache.md` |
 | KV offload (CPU tiering, LRU/ARC) | `v1/kv_offload/` | ☐ T2 | | `planned: specs/kv-offload.md` |
 | External KV-cache provider ABI + LMCache (MP service and in-process connectors) | `config/kv_transfer.py`, `distributed/kv_transfer/kv_connector/v1/{base,lmcache_connector,lmcache_mp_connector}.py` | ☐ T2 | explicit roadmap outcome `KV-EXTERNAL-CACHE`: mirror `kv_producer`/`kv_consumer`/`kv_both`, scheduler/worker metadata, async layer load/store, dynamic external connector modules, failure policy, metrics and cache-lifecycle ownership; gate the official LMCache shared-prefix quickstart plus Qwen3.6 hybrid behavior | `planned: specs/external-kv-cache-lmcache.md` |

.agents/quantization-matrix.md

@@ -157,7 +157,7 @@ Pinned vLLM source: `vllm/config/cache.py:19-36`.
 
 | ID | Item | Upstream | Our code | Tests/evidence | Spike/spec | State | Owner |
 |---|---|---|---|---|---|---|---|
-| `QUANT-KV-FP8` | fp8, fp8_e4m3, fp8_e5m2 | `vllm/config/cache.py:19-25`; `vllm/model_executor/layers/quantization/kv_cache.py:42-191`; store `cache_kernels.cu:241-252`; scale convention `quant_utils.cuh:296-308` | **W1 CPU fp8-e4m3 store+read LANDED**: [codec](../include/vt/fp8_kv.h#L39), [store kernel](../src/vt/cpu/cpu_cache.cpp#L143), [read dequant](../src/vt/cpu/cpu_paged_attn.cpp#L82), [config parse](../include/vllm/v1/kv_cache_dtype.h#L37). **W2 CUDA fp8-e4m3 store+read LANDED** ([#1593](https://github.com/mudler/vllm.cpp/issues/1593)): [store kernel](../src/vt/cuda/cuda_cache.cu), [read dequant](../src/vt/cuda/cuda_paged_attn.cu) -- gate [test_cuda_fp8_kv_cache](../tests/vt/test_cuda_fp8_kv_cache.cpp), whose DEVICE cases are UNEXECUTED; the CUDA TUs COMPILE (CI `cuda-fat-build`, ten architectures, run 32495320287 on `4d71e776e`) but that job builds with tests OFF, so none has been executed (spec `## Owed`). **W3 runner integration LANDED** ([#1593](https://github.com/mudler/vllm.cpp/issues/1593)): [`--kv-cache-dtype`](../src/vllm/entrypoints/openai/server_main.cpp) reaches `EngineParams`, the checkpoint's `kv_cache_quant_algo` is honoured when no flag is typed, [`ApplyCacheDType`](../src/vllm/v1/kv_cache_interface.cpp) retypes every attention spec (group and heterogeneous per-layer alike) so one byte budget buys exactly 2x the blocks, and [`kv_cache_route.h`](../include/vllm/model_executor/models/kv_cache_route.h) is the ONE place the store and the read decide float versus fp8. An fp8 cache DISABLES FA-2 prefill and decode, the WMMA ladder and the vectorized decode kernels, which are bf16-native; the resulting throughput cost is unmeasured and recorded as such. e5m2 compute, per-head scales, the Metal/ROCm arms, the C-ABI field, the 16 unrouted architectures (15 of which refuse with a message that names neither fp8 nor the flag) and the `k_scale`/`v_scale` weight-loader read are named later bricks (see spec) | [test_ops_fp8_kv_cache](../tests/vt/test_ops_fp8_kv_cache.cpp#L1) — 8 cases / 511 assertions, round-trip within the e4m3 band + fp8-vs-bf16 NMSE<1% + paged-attention e2e; RED-first (wrong store direction fails 3/480); W3 [test_kv_cache_fp8_wiring](../tests/vllm/entrypoints/test_kv_cache_fp8_wiring.cpp) — 31 cases, G1-G12, entering through `LoadedEngine` rather than by building a spec by hand, and bounding the fp8 pages against a bf16 run inside e4m3's own round-trip envelope, and [test_serve_kv_cache_dtype](../tests/vllm/entrypoints/openai/test_serve_kv_cache_dtype.cpp) — the flag through the REAL `VllmServerMain` | [fp8-kv-cache](specs/fp8-kv-cache.md) | `PARTIAL` | - |
+| `QUANT-KV-FP8` | fp8, fp8_e4m3, fp8_e5m2 | `vllm/config/cache.py:19-25`; `vllm/model_executor/layers/quantization/kv_cache.py:42-191`; store `cache_kernels.cu:241-252`; scale convention `quant_utils.cuh:296-308` | **W1 CPU fp8-e4m3 store+read LANDED**: [codec](../include/vt/fp8_kv.h#L39), [store kernel](../src/vt/cpu/cpu_cache.cpp#L143), [read dequant](../src/vt/cpu/cpu_paged_attn.cpp#L82), [config parse](../include/vllm/v1/kv_cache_dtype.h#L37). **W2 CUDA fp8-e4m3 store+read LANDED** ([#1593](https://github.com/mudler/vllm.cpp/issues/1593)): [store kernel](../src/vt/cuda/cuda_cache.cu), [read dequant](../src/vt/cuda/cuda_paged_attn.cu) -- gate [test_cuda_fp8_kv_cache](../tests/vt/test_cuda_fp8_kv_cache.cpp), whose DEVICE cases are UNEXECUTED; the CUDA TUs COMPILE (CI `cuda-fat-build`, ten architectures, run 32495320287 on `4d71e776e`) but that job builds with tests OFF, so none has been executed (spec `## Owed`). **W3 runner integration LANDED** ([#1593](https://github.com/mudler/vllm.cpp/issues/1593)): [`--kv-cache-dtype`](../src/vllm/entrypoints/openai/server_main.cpp) reaches `EngineParams`, the checkpoint's `kv_cache_quant_algo` is honoured when no flag is typed, [`ApplyCacheDType`](../src/vllm/v1/kv_cache_interface.cpp) retypes every attention spec (group and heterogeneous per-layer alike) so one byte budget buys exactly 2x the blocks, and [`kv_cache_route.h`](../include/vllm/model_executor/models/kv_cache_route.h) is the ONE place the store and the read decide float versus fp8. An fp8 cache DISABLES FA-2 prefill and decode, the WMMA ladder and the vectorized decode kernels, which are bf16-native; the resulting throughput cost is unmeasured and recorded as such. e5m2 compute, per-head scales, the Metal/ROCm arms, the C-ABI field, the 16 unrouted architectures (15 of which refuse with a message that names neither fp8 nor the flag) and the `k_scale`/`v_scale` weight-loader read are named later bricks (see spec) | [test_ops_fp8_kv_cache](../tests/vt/test_ops_fp8_kv_cache.cpp#L1) — 8 cases / 511 assertions, round-trip within the e4m3 band + fp8-vs-bf16 NMSE<1% + paged-attention e2e; RED-first (wrong store direction fails 3/480); W3 [test_kv_cache_fp8_wiring](../tests/vllm/entrypoints/test_kv_cache_fp8_wiring.cpp) — 31 cases, G1-G12, entering through `LoadedEngine` rather than by building a spec by hand, and bounding the fp8 pages against a bf16 run inside e4m3's own round-trip envelope, and [test_serve_kv_cache_dtype](../tests/vllm/entrypoints/openai/test_serve_kv_cache_dtype.cpp) — the flag through the REAL `VllmServerMain`. **W6 ROCm fp8-e4m3 store+read LANDED** ([#2065](https://github.com/mudler/vllm.cpp/issues/2065)): [store kernel](../src/vt/rocm/rocm_dense_basic.hip), [read dequant](../src/vt/rocm/rocm_paged_attn.hip) -- gate [test_rocm_fp8_kv_cache](../tests/vt/test_rocm_fp8_kv_cache.cpp), 7/7 cases 28/28 assertions on gfx1100 | [fp8-kv-cache](specs/fp8-kv-cache.md) | `PARTIAL` | - |
 | `QUANT-KV-FP8-VENDOR` | fp8_inc, fp8_ds_mla | `vllm/config/cache.py:24-25`; vendor KV implementations selected by attention backend | - | no quantized KV cache | `planned: specs/vendor-fp8-kv-cache.md` | `INVENTORIED` | - |
 | `QUANT-KV-TURBO` | k8v4, 4bit_nc, k3v4_nc, 3bit_nc | `vllm/config/cache.py:28-33`; TurboQuant dependency path | - | no quantized KV cache | `planned: specs/turboquant-kv-cache.md` | `INVENTORIED` | - |
 | `QUANT-KV-PER-HEAD` | int4/int8/fp8 per-token-head | `vllm/config/cache.py:34`; quantized cache kernels selected by backend | - | no quantized KV cache | `planned: specs/per-head-kv-cache.md` | `INVENTORIED` | - |