From b917fe26c27eb34b262085e86d9725f77d2ac147 Mon Sep 17 00:00:00 2001 From: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> Date: Sun, 23 Aug 2026 03:09:37 +0000 Subject: [PATCH 1/2] docs: repair public benchmark links The benchmark split left three reader-facing references on the old document shape. Link each claim to its public detail page and remove the deleted STATUS.md reference. No benchmark result, support claim, or README news headline changes. Closes #1756 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] --- .agents/issue-index.md | 1 + .agents/specs/public-doc-projections.md | 9 +++++++++ README.md | 5 +++-- docs/SPECULATIVE-DECODING.md | 15 ++++++++------- docs/benchmarks/llama-cpp-cpu.md | 2 +- 5 files changed, 22 insertions(+), 10 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 600bf392b..bc1a5b503 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -814,3 +814,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#2117](https://github.com/mudler/vllm.cpp/issues/2117) | `SPEC-DFLASH2` | **Two silent graph-loss mechanisms on the c=8 spec step, and the cost model that made the first look measurable.** A step that admits a prefill loses the CUDA graph for the WHOLE step (`GraphEligibleQueryLen` returns `nullopt`, `cudagraph_dispatch.h:161-176`) AND drops its decode rows onto the `num_splits=1` prefill ladder (`uniform_spec_query_len` becomes 0, `runner.cpp:2065-2069`, and `PagedAttnIsPrefill` then reads true, `include/vt/paged_attn_route.h:47-51`). W13 lands the INSTRUMENT and corrects the arithmetic rather than fixing either mechanism: #2117 prices mechanism 1 by multiplying the mixed step's TOTAL cost by its frequency, but ~1976 of a 2048-token mixed step is prefill work the engine owes on any lane, so the defect's MARGINAL cost is only the eager term (~24% of a decode step at c=64, an upper bound at c=8, `runner.cpp:2003-2004`) plus the attention-lane term (+9 ms on a ~113-122 ms step, `specs/dflash2-spec-as-decode.md`), which over 3% to 7% of steps is **1.0% to 2.2%, under the rung's own 5.9% spread** — mechanism 1 can be neither confirmed nor refuted by a c=8 throughput delta, so a fix landed against it now would be unfalsifiable in both directions. What ships instead is the readout #2112 owes plus a THREE-WAY split of `ragged_steps` into `ragged_mixed_steps`, `ragged_prefill_only_steps` and `ragged_spec_only_steps`, because the flat counter is consistent both with #2117's 3-7% admission prediction and with its own "far above 10% means #1943", and therefore discriminates nothing. All three candidate fixes are assessed and NONE lands: **(a) PIECEWISE** is a real primary-oracle gap (`cudagraph_dispatcher.py:307-324`, ours goes fully eager, `cudagraph_dispatch.h:204-207` already names it owed) but `include/vt/breakable_graph.h:21-26` states the seam's own admissibility rule — a speed claim needs a path "BOTH currently eager AND currently host-bound" — and a 2048-token mixed step is the prefill regime, 3.8% host-idle at >96% GPU-busy, so the lever is inert there and the construction is several waves without a compiler; **(b) keep the decode rows on the decode lane** is the half with a measured number, and its unlanded PRECONDITION is itself a mirror gap this issue found — `runner.cpp:1691` calls `reorder_batch_to_split_decodes_and_prefills` with the default `decode_threshold = 1` while vLLM passes `reorder_batch_threshold`, which `_init_reorder_batch_threshold` (`backend.py:657-687` @ pin `5559679229`) raises to `1 + 2k` for a spec-as-decode backend, so at k=8 the threshold should be 17 and is 1 and a 9-token verify row sorts as `long_extend` beside chunked-prefill continuations, leaving no boundary any split could cut at; **(c) SGLang's prefill/decode step separation** is a divergence from the mirror source, not a porting gap, and is REJECTED pending the measurement. NOT re-run, and named so nobody re-runs it: the `mnbt=2048` vs `8192` A/B at `model_loader.cpp:1100-1103`, whose stated reason is mechanism 1 in its own words. No GPU taken, no throughput claim made; O1/O2/O3 in [`specs/dflash2-mixed-step-readout.md`](specs/dflash2-mixed-step-readout.md) carry the remainder | bug | | [#2129](https://github.com/mudler/vllm.cpp/issues/2129) | `SPEC-DFLASH2` | **The spec-as-decode reorder threshold is computed and never passed to the reorder, so the runner reorders every batch at 1.** `runner.cpp:1691` calls `reorder_batch_to_split_decodes_and_prefills(input_batch_, scheduler_output)` with no `decode_threshold`, taking the declaration default of 1 (`runner.h:102-104`), while `SpecAsDecodeReorderThreshold` (`include/vllm/v1/attention/backend.h:180-184`) already mirrors upstream's `1 + (parallel_drafting ? 2 : 1) * k` and has exactly ONE caller — `backend.h:201`, inside the same header, serving the spec-as-decode *classification* that `runner.cpp:2065` consumes. Upstream feeds it: `_may_reorder_batch` passes `decode_threshold=self.reorder_batch_threshold` (`gpu_model_runner.py:1126-1130` @ pin `5559679229`), `calculate_reorder_batch_threshold` resolves it as the min over the attention groups after their builders exist (`:7122`, `:7194-7212`), and `_init_reorder_batch_threshold` raises it for a spec-as-decode backend (`backend.py:657-687`), which `gdn_attn.py:112` requests for every speculative configuration via `supports_spec_as_decode=self.use_spec_decode`. At k=8 with `parallel_drafting` — true for `dflash` and `dspark` and only those (`speculative.py:963-964`, mirrored at `include/vllm/config/speculative.h:158`, `:346`) — the threshold should be 17 and is 1, so a `1+k = 9`-token verify row classifies `long_extend` (region 2) instead of `decode` (region 0) and sorts among the chunked-prefill continuations, leaving no decode/prefill boundary for any consumer of the reordered order to cut at. **A mirror obligation, not a speed claim**: the formula was ported by W10 and left unwired, and NOTHING here is measured — the c=8 floor on the #1574 ladder is ~5.9 %. Found by the #2117 implementer while scoping the mixed-step work and named in [#2128](https://github.com/mudler/vllm.cpp/issues/2128) as lever (b)'s unlanded precondition; verified independently before dispatch. Fixed in flow, spec [reorder-threshold-wiring.md](specs/reorder-threshold-wiring.md), which also records under `## Owed` the divergence this change does NOT close: upstream skips the reorder entirely when every group reports `None` (`flash_attn.py` and `triton_attn.py` set no threshold), and we reorder unconditionally | bug | | [#2116](https://github.com/mudler/vllm.cpp/issues/2116) | `SPEC-DFLASH2` | **A speculator vetoes the async input and sampler path at `runner.cpp:470` (`:425` at the base tree the measurement below names), so every spec step drains the queue in step, while vLLM keeps async scheduling ON for dflash because `DFlashModelTypes` is inside `EagleModelTypes`.** Scoped by [`specs/dflash2-async-spec-sampler.md`](specs/dflash2-async-spec-sampler.md), which discharges `## Owed` A2 of [`specs/spec-decode-async-scheduling.md`](specs/spec-decode-async-scheduling.md). The veto was MEASURED load-bearing on the CPU tier rather than argued: deleting `!spec_config_.has_value()` at both construction sites reds `test_mtp_depth`'s W7 identity case (10 cases / 123 assertions / exit 0 becomes 9 passed / 1 failed / exit 1) through a production refusal at `runner.cpp:1833`, because `sample_tokens_async` carries no verify arm — no rejection sampler and no propose — a reason the veto's own comment did not name. The comment's stated reason holds too, and holds invisibly: under the same mutation the non-draft-aware combine overwrites the LAST DRAFT of every verify block with the previous step's committed token (`draft=[6 18]` becomes `draft=[6 5]` where the previous step emitted `5`, at every position), and the emitted tokens never move, so every identity assertion still passes. That is #1366's acceptance-only shape a second time. The row therefore stays vetoed and the fix is staged A2-1 through A2-5, with the draft-equality gate G2 owed by the first wave | bug | +| [#1756](https://github.com/mudler/vllm.cpp/issues/1756) | `ENG-PUBLIC-DOC-PROJECTIONS` | Three user-facing benchmark references retained the old public-document shape after #1714: README sent two claims to the internal benchmark record, the speculative-decoding guide described the compact index as the old record, and the llama.cpp detail page named deleted `docs/STATUS.md`. Fixed in flow by linking the public benchmark detail pages directly. No result, support claim, or README news headline changed | bug | diff --git a/.agents/specs/public-doc-projections.md b/.agents/specs/public-doc-projections.md index 0c260ab78..d2224cae8 100644 --- a/.agents/specs/public-doc-projections.md +++ b/.agents/specs/public-doc-projections.md @@ -318,3 +318,12 @@ The current-main reconciliation preserved the `docs/benchmarks/at-a-glance.md`. The operator gate passed every change-related check. The unrelated shared-host contention means this result does not claim fully green integration. + +### Follow-up link repair + +Issue [#1756](https://github.com/mudler/vllm.cpp/issues/1756) repaired three +references that retained the old public-document shape after the split. README +benchmark claims now lead to public detail pages. The speculative-decoding +guide links its benchmark detail directly. The llama.cpp detail page no longer +names the deleted `docs/STATUS.md`. No result, support claim, or news headline +changed. diff --git a/README.md b/README.md index 2e3cbbb06..9c8a20fc7 100644 --- a/README.md +++ b/README.md @@ -112,7 +112,7 @@ treat c2 through c32 as ties. The tokens are identical either way, and the insta 9.1 GiB. Cold start to first `/health`: **36.5 s vs vLLM's 221.5 s (6.1x)**, provisional -([detail](.agents/benchmark-record.md)). +([measurement details](docs/benchmarks/vllm-online-serving.md)). Peak host memory is a clean win at **24.88 GiB vs vLLM's 28.18 GiB**, with no Python stack behind it: @@ -159,7 +159,8 @@ shape-gated to prefill (95.9% on the default build). MTP is **token-identical to vLLM's MTP and about 4% faster** at c1 on Qwen3.6-27B-NVFP4, on both gate models end to end. Block-diffusion DFlash runs about 2x over spec-off but stays below vLLM's -throughput. That gap is an open bf16 acceptance floor, tracked in the benchmark record. +throughput. The [speculative-decoding measurements](docs/benchmarks/speculative-decoding.md) track +that open bf16 acceptance floor. Full per-axis grids, memory tables, the nine residual axes, and exact reproduction recipes: [docs/BENCHMARKS.md](docs/BENCHMARKS.md). The two figures above are rendered from these measured diff --git a/docs/SPECULATIVE-DECODING.md b/docs/SPECULATIVE-DECODING.md index d6395ff18..91597ae07 100644 --- a/docs/SPECULATIVE-DECODING.md +++ b/docs/SPECULATIVE-DECODING.md @@ -362,11 +362,12 @@ with a SINGLE COLD oracle invocation per paired run, so the denominator paid compile-JIT the numerator did not; with the oracle warm and generation length matched, the paired measurement is **0.834x**. Those two cannot be differenced directly, because the gate host was reimaged in between and is no longer the same -machine. The deciding experiment, a single cold oracle invocation on the CURRENT -box, is specified in the benchmark record and has not yet run. Until it does, no -speed claim in either direction is supportable. The acceptance-rate band and the -other target families remain owed. A GGUF target, and a target architecture with no aux -multi-tap, are both refused by name. +machine. The deciding experiment is a single cold oracle invocation on the +current box. The [speculative-decoding benchmark page](benchmarks/speculative-decoding.md) +records the pending run. Until it runs, no speed claim in either direction is +supportable. The acceptance-rate band and the other target families remain +owed. The loader refuses a GGUF target or a target architecture without an +auxiliary multi-tap by name. ```bash main --model /models/Qwen3-4B \ @@ -457,8 +458,8 @@ Speculation helps both engines about 1.5 to 1.6x at this operating point, and ou engine is already about 4% faster than vLLM with speculation off, so the lead is preserved with it on. The extra state speculation needs (a doubled recurrent-state slot at k=1, plus the draft cache and head) costs about 3.6 GB, well inside the -box's unified memory. The full A/B, including the higher-concurrency numbers, is -in [the benchmark record](BENCHMARKS.md). +box's unified memory. The [speculative-decoding benchmark page](benchmarks/speculative-decoding.md) +contains the full A/B, including the higher-concurrency numbers. ## Concurrency above 1 diff --git a/docs/benchmarks/llama-cpp-cpu.md b/docs/benchmarks/llama-cpp-cpu.md index 0365b4309..d2dd17b7e 100644 --- a/docs/benchmarks/llama-cpp-cpu.md +++ b/docs/benchmarks/llama-cpp-cpu.md @@ -34,7 +34,7 @@ set is swept over every tracked file, not listed: the query is in the | Flips first | Verdict | Exposure | |---|---|---| | 1 | Vulkan `BENCH-VK-LLAMA` decode 4.36 vs 4.35, `MET` | 0.23% margin inside a 0.69% 7-leg spread. Its own source calls it "a narrow pass, not a comfortable one", so any move can flip it. The README states it as "matches" | -| 2 | Muse Glimmer in128 prefill `1.023x` (also `STATUS.md`) | 2.3% margin inside our own arm's 4.5% leg spread, n=4. Its denominator is already stock `7044859`, 84 commits from `b10451`, so the noise floor is what exposes it | +| 2 | Muse Glimmer in128 prefill `1.023x` (also in [At a glance](at-a-glance.md)) | 2.3% margin inside our own arm's 4.5% leg spread, n=4. Its denominator is already stock `7044859`, 84 commits from `b10451`, so the noise floor is what exposes it | | 3 | This table's peak memory `1.01x` PARITY and decode `0.97x` tie | ties by declaration, not wins. A denominator that moves at all in llama.cpp's favour turns both into recorded gaps | | 4 | keep-f16's `1.01x` RSS and "prefill `1.18x` AHEAD", quoted in product code for `VT_GGUF_KEEP_F16` default ON | buys 1.05 GiB of peak RSS (3.885 to 2.832) for ~9% prefill (224 to 204 tok/s) and ~1.4% decode, tokens identical. **Decided 2026-08-17: the default stays ON on our own arms, so it needs no re-take.** Both ratios do | | 5 | `KERNEL-GEMM-CPU-TILED` NEON vs stock ggml sgemm, "at parity, ahead on 4 of 6 shapes" | off this page, in the kernel matrix. Bands overlap, 216-242 vs 208-215 GFLOP/s, and one shape is already behind | From 55c88e4d3ae2ade25ecb668bac3ae4a5db1b2232 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 28 Aug 2026 07:35:34 +0000 Subject: [PATCH 2/2] docs: point the two repointed claims at figures the target pages carry #1756 repaired three links that still named the internal record or the deleted `docs/STATUS.md`. Two of them now resolve to a public page that states a DIFFERENT number from the sentence citing it, which is worse than the dead link it replaced, because a reader can now check. Cold start: the README's "36.5 s vs 221.5 s (6.1x), provisional" appears nowhere in the benchmark record and nowhere on the page it now cites. `docs/benchmarks/vllm-online-serving.md` measures 53 s vs 780 s, medians of three, and says in the next row why the ratio is not like-for-like. The README takes the measured pair and that caveat. Speculative decoding: `docs/benchmarks/speculative-decoding.md` puts DFlash at 2.9x over spec-off and at vLLM's own DFlash (1.003x); the arm that is still behind is DFlash2 at 0.8017x. The README conflated the two, so it named the right floor against the wrong speculator. The guide's "contains the full A/B, including the higher-concurrency numbers" is also untrue of that page and was untrue of `BENCHMARKS.md` before it, so it now says what the page holds and records the higher-concurrency legs as owed. No measurement is added, and no claim moves in our favour: both edited ratios were replaced by the published ones. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- README.md | 11 +++++++---- docs/SPECULATIVE-DECODING.md | 3 ++- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 9c8a20fc7..14ca94d7a 100644 --- a/README.md +++ b/README.md @@ -111,7 +111,9 @@ We are ahead at all six, but only c1 at 4.5% is clearly outside our 0.5% run-to- treat c2 through c32 as ties. The tokens are identical either way, and the install is 66 MiB against 9.1 GiB. -Cold start to first `/health`: **36.5 s vs vLLM's 221.5 s (6.1x)**, provisional +Cold start to first `/health`: **53 s vs vLLM's 780 s (14.7x)**, medians of three, and +deliberately NOT like-for-like -- we answer `/health` on process liveness, so the first +request then pays 91.6 s of real first-inference cost that vLLM front-loads ([measurement details](docs/benchmarks/vllm-online-serving.md)). Peak host memory is a clean win at **24.88 GiB vs vLLM's 28.18 GiB**, with no Python stack behind it: @@ -158,9 +160,10 @@ shape-gated to prefill (95.9% on the default build). ### Speculative decoding MTP is **token-identical to vLLM's MTP and about 4% faster** at c1 on Qwen3.6-27B-NVFP4, on both gate -models end to end. Block-diffusion DFlash runs about 2x over spec-off but stays below vLLM's -throughput. The [speculative-decoding measurements](docs/benchmarks/speculative-decoding.md) track -that open bf16 acceptance floor. +models end to end. Block-diffusion DFlash runs **2.9x over spec-off** and lands at vLLM's own +DFlash (1.003x); its successor DFlash2 is the one still **behind, at 0.8017x**, on an open bf16 +acceptance floor. Every speculator's current position is in +[the speculative-decoding measurements](docs/benchmarks/speculative-decoding.md). Full per-axis grids, memory tables, the nine residual axes, and exact reproduction recipes: [docs/BENCHMARKS.md](docs/BENCHMARKS.md). The two figures above are rendered from these measured diff --git a/docs/SPECULATIVE-DECODING.md b/docs/SPECULATIVE-DECODING.md index 91597ae07..b6d668785 100644 --- a/docs/SPECULATIVE-DECODING.md +++ b/docs/SPECULATIVE-DECODING.md @@ -459,7 +459,8 @@ engine is already about 4% faster than vLLM with speculation off, so the lead is preserved with it on. The extra state speculation needs (a doubled recurrent-state slot at k=1, plus the draft cache and head) costs about 3.6 GB, well inside the box's unified memory. The [speculative-decoding benchmark page](benchmarks/speculative-decoding.md) -contains the full A/B, including the higher-concurrency numbers. +carries every speculator's current published position. The higher-concurrency legs of this +particular A/B are not among them and stay owed. ## Concurrency above 1