docs: add recent server features to README - #1895
Closed
localai-org-maint-bot wants to merge 1 commit into
Closed
Conversation
The README News section predates the serving paths for FP8 KV cache storage and prompt log probabilities. Add both surfaces and link the FP8 memory trade-off to the focused usage guide. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:GPT-5 [Codex]
localai-bot
added a commit
that referenced
this pull request
Aug 25, 2026
…st an independent double reference (#699) (#1898) W3 of the dots3-note row (#699): the full-attention layer. The first dots3 brick that COMPUTES rather than refuses. WHY THE REFERENCE IS THE ARGUMENT. Under spec §6.4 option B this row has no oracle and will not get one at this stage -- the model fits on no host we own. There is no token gate, so an in-test double-precision reference IS the correctness claim. That only means anything if the reference is independent, so it is a DIFFERENT ALGORITHM at every step: rotation by complex multiply with angles recomputed per element rather than a cos/sin cache, softmax WITHOUT max subtraction in `long double`, top-k by full stable sort rather than the shared partial-selection port. Agreement 1.75e-16 to 3.21e-16; indexer selection identical in all 24 slots. A fresh reviewer tested that independence in the strong direction rather than by reading: it mutated the SHARED helper the implementation routes through and the reference does not -- dropped the ReLU in `deepseek_v4::DsaIndexerLogits` -- and the gate went RED. The reference VALIDATES the shared helper instead of agreeing with it, which is the inverse of the shared-helper failure mode this project has recorded elsewhere. A second mutation replaced the sigmoid with a hard step that preserves every structural property -- still in (0,1), still per-head, `gated/attn_out` still equal to the traced gate -- and ONLY the two reference comparisons fired. THE FIVE MECHANISMS, each caught by mutation with its compiler exit recorded: the two LoRA rescales, `k_rope_only_layernorm` (the RMSNorm DeepSeek lacks), the headwise gate, the indexer's GPT-J pairing (§4 trap 2), and #1846's LEADING rope slice. On the rescales the MISSING/MISPLACED distinction is earned rather than asserted: moving the multiply before the norm is provably a no-op (6.03e-14) and the gate detects that structurally, so only an absent rescale moves output. #1846's tail slice moves 10 of 24 selection slots and 0.793, and gives a DIFFERENT answer than the NeoX flip -- the two axes are independent defects, not one wearing two costumes. A MUTATION CAME BACK GREEN AND THE CODE CHANGED, TWICE. First, feeding the indexer an unrescaled `q_c` alters nothing: `sum_h w·ReLU(dot)` scales uniformly per row under a positive rescale, so the argmax cannot move. A source comment claimed otherwise; it was corrected and the invariance pinned with a power-of-two factor so the ratio is an exact equality. Second, review found `indexer_k_norm_eps` was the one shared scalar no case pinned -- moved three orders of magnitude, the gate stayed green. Both halves of the fix landed, and re-running the mutation CORRECTED the author's own claim about it: the pin reds, the reference comparison does not, because at this fixture the eps shift moves the selection margin without moving the chosen keys. The commit message that claimed either edit alone would catch it was wrong and says so. ANCHOR HYGIENE, WHICH THIS PR IS PARTLY ABOUT. Spec §2.2's W0-era line numbers were stale on byte-identical sources -- the headwise gate is `model.py:190-197`, not `:246-262`, which is a constructor argument list. Nine further anchors were re-derived at `06ecec7a84` and corrected. One of them, the MLA softmax scale, was miscounted by BOTH the author (`:1027`, which is `max_position_embeddings`) and the reviewer (`:1025`, which is blank); it is `:1026`, and the record says two readers miscounted the same line in opposite directions. The author's own first draft of the §2.2 fix cleared two items as "verified unchanged" without checking them, and they were stale by the more dangerous margin -- W0's `:154` lands exactly on the line the NEXT item names, so a reader checking it finds plausible code and moves on. That near-miss is recorded rather than quietly fixed. `check-symbol-anchors.py` cannot catch any of this; its own docstring says it does not verify line citations, which is why #1139 exists. NOT ON THE DECODE PATH, AND SAID SO. `Dots3NoteModel::ForwardDevice` still refuses by name and the last gate case asserts that refusal through the real factory. Three debts are recorded in `## Owed` rather than implied: the device wiring and the `mla::ForwardMlaAttentionBlock` extension (both W4), and the bf16 memory format -- a double reference cannot see that upstream computes the gate's sigmoid in fp32 and casts back. Review added a fourth: upstream quantizes the indexer's q per 128-element group to fp8, which neither arm models, and M13's argmax invariance does NOT cover it because fp8 rounding is not a positive rescale. Not extending the shared MLA seam was judged right for a stronger reason than the author gave: `mla::ForwardMlaAttentionBlock` is a device seam over a paged cache with a decode backend and no sparse mask, so a host, cache-free, whole sequence `double` reference cannot route through it at all -- four of four deltas it cannot represent. Duplication is contained to one new TU with no existing header edited, and the deviation is recorded in porting-inventory 18. Gate: `test_dots3_note_attn`, 12 cases / 199 assertions, CPU-only, no GPU lease. Nineteen mutations, every compiler exit status recorded. Operator gate rerun rather than taken on report: 0 behind main at merge, both commit checkers RUN and pass (they skip on a behind branch, and a skip is not a pass), `check-agent-record` and `check-model-checklist` OK, the target rebuilt and 12/199 green, and one spot-mutation reproduced by hand -- flipping `IndexerRopeOffset` to the tail slice builds clean and reddens 3 cases / 8 assertions, with the tree restored and md5 verified. CI on the reviewed head: `build-test-cpu`, BOTH `sanitize-cpu` arms, `cuda-fat-build`, `build-newest-gcc`, arm64 and vulkan all pass; the reviewer additionally ran an ASAN+UBSAN+LSAN build of the target with zero diagnostics. Only `windows-msvc-*` fail, #584, corroborated across #1896, #1899 and the docs-only #1895. Row stays SPIKE. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
jefby
pushed a commit
to jefby/vllm.cpp
that referenced
this pull request
Aug 25, 2026
…st an independent double reference (mudler#699) (mudler#1898) W3 of the dots3-note row (mudler#699): the full-attention layer. The first dots3 brick that COMPUTES rather than refuses. WHY THE REFERENCE IS THE ARGUMENT. Under spec §6.4 option B this row has no oracle and will not get one at this stage -- the model fits on no host we own. There is no token gate, so an in-test double-precision reference IS the correctness claim. That only means anything if the reference is independent, so it is a DIFFERENT ALGORITHM at every step: rotation by complex multiply with angles recomputed per element rather than a cos/sin cache, softmax WITHOUT max subtraction in `long double`, top-k by full stable sort rather than the shared partial-selection port. Agreement 1.75e-16 to 3.21e-16; indexer selection identical in all 24 slots. A fresh reviewer tested that independence in the strong direction rather than by reading: it mutated the SHARED helper the implementation routes through and the reference does not -- dropped the ReLU in `deepseek_v4::DsaIndexerLogits` -- and the gate went RED. The reference VALIDATES the shared helper instead of agreeing with it, which is the inverse of the shared-helper failure mode this project has recorded elsewhere. A second mutation replaced the sigmoid with a hard step that preserves every structural property -- still in (0,1), still per-head, `gated/attn_out` still equal to the traced gate -- and ONLY the two reference comparisons fired. THE FIVE MECHANISMS, each caught by mutation with its compiler exit recorded: the two LoRA rescales, `k_rope_only_layernorm` (the RMSNorm DeepSeek lacks), the headwise gate, the indexer's GPT-J pairing (§4 trap 2), and mudler#1846's LEADING rope slice. On the rescales the MISSING/MISPLACED distinction is earned rather than asserted: moving the multiply before the norm is provably a no-op (6.03e-14) and the gate detects that structurally, so only an absent rescale moves output. mudler#1846's tail slice moves 10 of 24 selection slots and 0.793, and gives a DIFFERENT answer than the NeoX flip -- the two axes are independent defects, not one wearing two costumes. A MUTATION CAME BACK GREEN AND THE CODE CHANGED, TWICE. First, feeding the indexer an unrescaled `q_c` alters nothing: `sum_h w·ReLU(dot)` scales uniformly per row under a positive rescale, so the argmax cannot move. A source comment claimed otherwise; it was corrected and the invariance pinned with a power-of-two factor so the ratio is an exact equality. Second, review found `indexer_k_norm_eps` was the one shared scalar no case pinned -- moved three orders of magnitude, the gate stayed green. Both halves of the fix landed, and re-running the mutation CORRECTED the author's own claim about it: the pin reds, the reference comparison does not, because at this fixture the eps shift moves the selection margin without moving the chosen keys. The commit message that claimed either edit alone would catch it was wrong and says so. ANCHOR HYGIENE, WHICH THIS PR IS PARTLY ABOUT. Spec §2.2's W0-era line numbers were stale on byte-identical sources -- the headwise gate is `model.py:190-197`, not `:246-262`, which is a constructor argument list. Nine further anchors were re-derived at `06ecec7a84` and corrected. One of them, the MLA softmax scale, was miscounted by BOTH the author (`:1027`, which is `max_position_embeddings`) and the reviewer (`:1025`, which is blank); it is `:1026`, and the record says two readers miscounted the same line in opposite directions. The author's own first draft of the §2.2 fix cleared two items as "verified unchanged" without checking them, and they were stale by the more dangerous margin -- W0's `:154` lands exactly on the line the NEXT item names, so a reader checking it finds plausible code and moves on. That near-miss is recorded rather than quietly fixed. `check-symbol-anchors.py` cannot catch any of this; its own docstring says it does not verify line citations, which is why mudler#1139 exists. NOT ON THE DECODE PATH, AND SAID SO. `Dots3NoteModel::ForwardDevice` still refuses by name and the last gate case asserts that refusal through the real factory. Three debts are recorded in `## Owed` rather than implied: the device wiring and the `mla::ForwardMlaAttentionBlock` extension (both W4), and the bf16 memory format -- a double reference cannot see that upstream computes the gate's sigmoid in fp32 and casts back. Review added a fourth: upstream quantizes the indexer's q per 128-element group to fp8, which neither arm models, and M13's argmax invariance does NOT cover it because fp8 rounding is not a positive rescale. Not extending the shared MLA seam was judged right for a stronger reason than the author gave: `mla::ForwardMlaAttentionBlock` is a device seam over a paged cache with a decode backend and no sparse mask, so a host, cache-free, whole sequence `double` reference cannot route through it at all -- four of four deltas it cannot represent. Duplication is contained to one new TU with no existing header edited, and the deviation is recorded in porting-inventory 18. Gate: `test_dots3_note_attn`, 12 cases / 199 assertions, CPU-only, no GPU lease. Nineteen mutations, every compiler exit status recorded. Operator gate rerun rather than taken on report: 0 behind main at merge, both commit checkers RUN and pass (they skip on a behind branch, and a skip is not a pass), `check-agent-record` and `check-model-checklist` OK, the target rebuilt and 12/199 green, and one spot-mutation reproduced by hand -- flipping `IndexerRopeOffset` to the tail slice builds clean and reddens 3 cases / 8 assertions, with the tree restored and md5 verified. CI on the reviewed head: `build-test-cpu`, BOTH `sanitize-cpu` arms, `cuda-fat-build`, `build-newest-gcc`, arm64 and vulkan all pass; the reviewer additionally ran an ASAN+UBSAN+LSAN build of the target with zero diagnostics. Only `windows-msvc-*` fail, mudler#584, corroborated across mudler#1896, mudler#1899 and the docs-only mudler#1895. Row stays SPIKE. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Collaborator
Author
|
Superseded by the current-main documentation audit in #1956, which carries this PR's source-verified additions and corrects the registry count. |
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row
Documentation audit only. This PR does not move a lifecycle row.
Before starting
scripts/ready-for-helper.pyresult when applicable: not applicable because no lifecycle state changes.src/vllm/entrypoints/openai/server_main.cpp,include/vllm/config/cache.h,tests/vllm/entrypoints/openai/test_serve_kv_cache_dtype.cpp,tests/vllm/entrypoints/test_kv_cache_fp8_wiring.cpp,docs/USAGE.md, anddocs/FEATURES.md.What changed
README News now includes the FP8 KV cache and prompt log probabilities. The feature overview also links the FP8 memory and kernel trade-offs to the usage guide.
Evidence
scripts/agent-preflight.shpasses. The documentation gates pass, but the full local preflight does not. This host lacks PyYAML, NumPy, and Ninja, uses a Pythontarfilewithout the required filter API, and reports unrelated environment-sensitive baseline failures.python3 scripts/check-readme-structure.pypython3 -m unittest tests.scripts.test_check_readme_structure tests.scripts.test_check_quickstart_recipes(51 tests)python3 scripts/check-agent-record.pypython3 scripts/check-quickstart-recipes.pypython3 scripts/check-commit-trailers.py --range HEAD^..HEADpython3 scripts/check-commit-style.py --range HEAD^..HEADgit diff upstream/main...HEAD --checkSpeed claims
Honest gaps
No GPU tests or measurements ran. This is a README-only alignment change. The full preflight limitations are listed under Evidence.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:GPT-5 [Codex]