Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/issue-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -761,3 +761,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
| [#2040](https://github.com/mudler/vllm.cpp/issues/2040) | `ENG-HYBRID-PLACEMENT` | W3c: replace W3b's hand-written per-model round trip with ONE shared seam every architecture routes through, exploiting that every MoE block already has the same `(Dev, weights, params, [T,H] dh, T)` shape — and correct `docs/ENVIRONMENT.md` and `docs/FEATURES.md`, which W3b made false by leaving them saying the knobs place nothing | bug |
| [#2046](https://github.com/mudler/vllm.cpp/issues/2046) | `ENG-HYBRID-PLACEMENT` | `qwen3_5.cpp` kept private `Dev`/`DBuf`/`MakeTensor`/`Reshape` copies instead of the shared `dense_device_glue.h` set — the off-framework divergence its own `ResidentWeight` comment records, where a repair reached 25 model files and not this one. The private types also had INTERNAL LINKAGE, which is what forced the MoE placement seam to carry a glue-templated second spelling; migrating collapses it back to one | bug |
| [#2050](https://github.com/mudler/vllm.cpp/issues/2050) | `ENG-HYBRID-PLACEMENT` | Laguna's FFN is host-orchestrated token-at-a-time — per-token host rows, the router on the host through `MatmulNK`, and a host scalar combine loop — so a device-shaped MoE entry wrapping those loops would put it in the placement seam's wired list while moving nothing and adding a round trip: supported to read, a regression to measure. The real repair is a device-resident batched FFN, which is a model rework with a performance gate | gap |
| [#1756](https://github.com/mudler/vllm.cpp/issues/1756) | `ENG-PUBLIC-DOC-PROJECTIONS` | Three user-facing benchmark references retained the old public-document shape after #1714: README sent two claims to the internal benchmark record, the speculative-decoding guide described the compact index as the old record, and the llama.cpp detail page named deleted `docs/STATUS.md`. Fixed in flow by linking the public benchmark detail pages directly. No result, support claim, or README news headline changed | bug |
9 changes: 9 additions & 0 deletions .agents/specs/public-doc-projections.md
Original file line number Diff line number Diff line change
Expand Up @@ -318,3 +318,12 @@ The current-main reconciliation preserved the
`docs/benchmarks/at-a-glance.md`. The operator gate passed every
change-related check. The unrelated shared-host contention means this result
does not claim fully green integration.

### Follow-up link repair

Issue [#1756](https://github.com/mudler/vllm.cpp/issues/1756) repaired three
references that retained the old public-document shape after the split. README
benchmark claims now lead to public detail pages. The speculative-decoding
guide links its benchmark detail directly. The llama.cpp detail page no longer
names the deleted `docs/STATUS.md`. No result, support claim, or news headline
changed.
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ treat c2 through c32 as ties. The tokens are identical either way, and the insta
9.1 GiB.

Cold start to first `/health`: **36.5 s vs vLLM's 221.5 s (6.1x)**, provisional
([detail](.agents/benchmark-record.md)).
([measurement details](docs/benchmarks/vllm-online-serving.md)).

Peak host memory is a clean win at **24.88 GiB vs vLLM's 28.18 GiB**, with no Python stack behind it:

Expand Down Expand Up @@ -156,7 +156,8 @@ shape-gated to prefill (95.9% on the default build).

MTP is **token-identical to vLLM's MTP and about 4% faster** at c1 on Qwen3.6-27B-NVFP4, on both gate
models end to end. Block-diffusion DFlash runs about 2x over spec-off but stays below vLLM's
throughput. That gap is an open bf16 acceptance floor, tracked in the benchmark record.
throughput. The [speculative-decoding measurements](docs/benchmarks/speculative-decoding.md) track
that open bf16 acceptance floor.

Full per-axis grids, memory tables, the nine residual axes, and exact reproduction recipes:
[docs/BENCHMARKS.md](docs/BENCHMARKS.md). The two figures above are rendered from these measured
Expand Down
15 changes: 8 additions & 7 deletions docs/SPECULATIVE-DECODING.md
Original file line number Diff line number Diff line change
Expand Up @@ -362,11 +362,12 @@ with a SINGLE COLD oracle invocation per paired run, so the denominator paid
compile-JIT the numerator did not; with the oracle warm and generation length
matched, the paired measurement is **0.834x**. Those two cannot be differenced
directly, because the gate host was reimaged in between and is no longer the same
machine. The deciding experiment, a single cold oracle invocation on the CURRENT
box, is specified in the benchmark record and has not yet run. Until it does, no
speed claim in either direction is supportable. The acceptance-rate band and the
other target families remain owed. A GGUF target, and a target architecture with no aux
multi-tap, are both refused by name.
machine. The deciding experiment is a single cold oracle invocation on the
current box. The [speculative-decoding benchmark page](benchmarks/speculative-decoding.md)
records the pending run. Until it runs, no speed claim in either direction is
supportable. The acceptance-rate band and the other target families remain
owed. The loader refuses a GGUF target or a target architecture without an
auxiliary multi-tap by name.

```bash
main --model /models/Qwen3-4B \
Expand Down Expand Up @@ -457,8 +458,8 @@ Speculation helps both engines about 1.5 to 1.6x at this operating point, and ou
engine is already about 4% faster than vLLM with speculation off, so the lead is
preserved with it on. The extra state speculation needs (a doubled recurrent-state
slot at k=1, plus the draft cache and head) costs about 3.6 GB, well inside the
box's unified memory. The full A/B, including the higher-concurrency numbers, is
in [the benchmark record](BENCHMARKS.md).
box's unified memory. The [speculative-decoding benchmark page](benchmarks/speculative-decoding.md)
contains the full A/B, including the higher-concurrency numbers.

## Concurrency above 1

Expand Down
2 changes: 1 addition & 1 deletion docs/benchmarks/llama-cpp-cpu.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ set is swept over every tracked file, not listed: the query is in the
| Flips first | Verdict | Exposure |
|---|---|---|
| 1 | Vulkan `BENCH-VK-LLAMA` decode 4.36 vs 4.35, `MET` | 0.23% margin inside a 0.69% 7-leg spread. Its own source calls it "a narrow pass, not a comfortable one", so any move can flip it. The README states it as "matches" |
| 2 | Muse Glimmer in128 prefill `1.023x` (also `STATUS.md`) | 2.3% margin inside our own arm's 4.5% leg spread, n=4. Its denominator is already stock `7044859`, 84 commits from `b10451`, so the noise floor is what exposes it |
| 2 | Muse Glimmer in128 prefill `1.023x` (also in [At a glance](at-a-glance.md)) | 2.3% margin inside our own arm's 4.5% leg spread, n=4. Its denominator is already stock `7044859`, 84 commits from `b10451`, so the noise floor is what exposes it |
| 3 | This table's peak memory `1.01x` PARITY and decode `0.97x` tie | ties by declaration, not wins. A denominator that moves at all in llama.cpp's favour turns both into recorded gaps |
| 4 | keep-f16's `1.01x` RSS and "prefill `1.18x` AHEAD", quoted in product code for `VT_GGUF_KEEP_F16` default ON | buys 1.05 GiB of peak RSS (3.885 to 2.832) for ~9% prefill (224 to 204 tok/s) and ~1.4% decode, tokens identical. **Decided 2026-08-17: the default stays ON on our own arms, so it needs no re-take.** Both ratios do |
| 5 | `KERNEL-GEMM-CPU-TILED` NEON vs stock ggml sgemm, "at parity, ahead on 4 of 6 shapes" | off this page, in the kernel matrix. Bands overlap, 216-242 vs 208-215 GFLOP/s, and one shape is already behind |
Expand Down
Loading