Repository navigation
Release 0.1.4 - #40
Merged
Merged
Conversation
The README changes of #13-#17 as one change: - Speed: a short overview with the best measured number for each system and concurrency (PCIe reference and the community NVLink result) and the cold prefill table; the breakdown moves to Performance (MTP4, DFlash2 against MTP4 by prompt category, NVLink 0.1.1 with its notes). - Weight and KV formats: GPU KV capacity with MTP4 and with DFlash2, and an FP16 KV row. - Feature list: the largest measured KV pool and the best speeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Qwen3.8 Flash-Next accepts images and video, not audio, so the
reference --limit-mm-per-prompt no longer sets an audio count. The
README launch command, the V100 media notes and the multimodal input
guide now use '{"image": 16, "video": 1}' and the generic image count
'{"image": 8, "video": 1}'.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Pair KV capacity with 512K concurrency and label measured versions. Add community NVLink results, state replay costs, and graph defaults. Use the measured 0.1.2 PCIe cold-prefill values. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
A profile of the replay on 4x V100 (12 steady steps at 1, 3 and 8 requests) showed about 150 small kernel launches per step for the replay index tables, 0.3 to 0.4 ms of GPU time, next to 0.7 to 1.4 ms for the 36 verifier relaunches. A Triton kernel now builds a cache group's table in one launch, with the batch gathers folded in: one program per verifier row writes the padded row, the running block at the last accepted position, and the prefix-cache block at a boundary passed before it. build_replay_state_indices stays as the reference; a CPU test runs the kernel under the Triton interpreter against it on 480 random cases (block sizes 4 to 1664, graph padding rows, strided base ids, speculative rows among others). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Add a "Checkpoints, KV scales and MTP drafts" section with a matrix and step-by-step setup for the AWQ checkpoints, Standard NVFP4 (combined scale and FP8 MTP draft pack v1.2) and NVIDIA FP4-Local. Document strict scale loading, the unit-scale fallback risk, FP16 and INT8 alternatives, and that an uncalibrated E4M3 draft cache is rejected. Recommend DFlash2 with GDN speculative state replay off: replay adds about 7 ms per decoding step when the target samples. Update the DFlash2 KV figures (1,379,634 tokens with replay off, 1,542,558 with replay on). Also add sampling parameter guidance, a note on static YaRN scaling, and correct the Standard NVFP4 --quantization note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Keep the README as a landing page and move setup, configuration, measurements and support details into seven focused guides. Retain linked README headings and apply the three reviewed setup fixes. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
The FP4-Local scale pack v1.1-base-b9ce18a6 adds the two K/V scales of the checkpoint's built-in MTP layer. Point the checkpoint table, the download step and the launch example at v1.1 and use fp8_e4m3 for the draft KV cache, with the paired acceptance result. Keep the note that v1.0 needs an FP16 draft cache. Also say that the 0.1.3 AWQ DFlash2 capacity rows cover state replay off (recommended) and on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Mirror v1.1-base-b9ce18a6 of the FP4-Local scale pack: add the two calibrated K/V scales of the checkpoint's built-in MTP layer and their manifest, and update the materializer, which merges them into the view index next to the 24 unchanged target scales (26 entries). Without the draft file the materializer still builds the original 24-scale view. The README now describes the draft scales and the E4M3 draft KV option. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Say that the greedy replay cost at eight requests comes from development runs and that a single release-image check measured replay slightly faster, so the difference is small. Correct the replay capacity gain to about 12% (1,542,558 vs 1,379,634) and compare the 26% peak KV use with 55% without replay and 60% on 0.1.2. Write the upstream issue as vllm-project/vllm#57740. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
A torch profiler capture of five real sampled requests on our 4x V100 PCIe deployment showed the replay's GPU work unchanged (about 1.0 ms per step) but the GPU busy only 7% of the time the replay call ran: the target's sampling had drained the queue, and the host-side launches (one table kernel per cache group, one verifier relaunch per GDN layer, 36 in all) set the step time. With greedy decoding the same launches overlapped queued target work (GPU busy about 90%), which hid the cost. GDNSpecReplayGraphs captures a FULL step's replay once per shape, the second time the shape runs (the first, eager, run compiles the kernels), and then replays it after copying three per-step inputs (speculative rows, idx_mapping, num_sampled) into the graph's fixed-size buffers. The key is the shape plus the persistent tensors the replay reads, so a step backed by other tensors captures its own graph. Padding rows are marked -1 in the speculative rows and keep only their start column. Mixed steps keep the eager replay. sm70_dflash2.spec_state_replay_graph (default on) turns it off. CPU tests: the table kernel with -1-padded rows equals the reference on 200 random FULL batches (Triton interpreter); mixed steps and the disabled switch stay eager; the second FULL step captures and replays with the step's inputs; a step backed by other tensors gets a new key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
A second release-image run of the recommended DFlash2 configuration measured 335.8 tokens/s at eight greedy requests, against 302.1 in the first run, so say that the replay difference at high concurrency is within run-to-run variation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Point the current-release line, the pip and Docker commands and the version check at 0.1.3, and drop the "in testing" notes from the 0.1.3 DFlash2 figures. The INT8 disk-tier limit is unchanged, so say "yet" instead of naming 0.1.2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
[Doc] README: speed overview, NVLink results and KV capacity by method
[Tools] FP4-Local KV scales: mirror the v1.1 pack with MTP draft scales
The first page now answers what a reader needs to decide whether to try ReVolta, and leaves everything else to docs/: - what it is and who it is for: a stable daily service on V100 32 GB servers, built for four or more GPUs and validated on four, backed by the release gate rather than adjectives; built on 1Cat-vLLM, so the vllm command and API work as usual; - the hardware and memory needed, in one line; - two small tables: context and GPU KV capacity, and the best measured speed on PCIe and NVLink; - a model table with precision (bits per weight for the routed experts and the whole model), download size, host memory for the PLE table, GPU KV pool, quality against each BF16 parent and notes, followed by what stays BF16 in each format. Test status moves to docs/checkpoints.md. Also point the direct wheel URL at v0.1.3, and replace "in 0.1.2" where the 0.1.3 behaviour is the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
[Doc] README: a first page for deciding whether ReVolta fits
The README said the AWQ PLE table always needs about 48 GiB of host memory and that PLE cascade is unsupported. Read together, that hides the experimental option added in 0.1.3, which keeps the whole PLE table on disk and ran under 64 GiB and 32 GiB container limits. State the default, point to the option, and separate it from cascade, which spills part of the table, stays unsupported and off by default, and is planned for 0.2. docs/known-limits.md now lists the switches, the tested scope, the numeric difference and what was not measured, as in the 0.1.3 release notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
[Doc] PLE: document the experimental whole-table disk option
Exercise replay admission, allocator accounting, host state slots, and the traced-source cache boundary on CPU. Add selector ownership and loader shape contracts for vocabulary-sharded codebooks. Keep unresolved contract failures visible for review. Runtime code is unchanged; CUDA and NCCL qualification remains a separate GPU gate. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com> (cherry picked from commit f7a50a0)
…racts The independent review cases for #20 and #9 were written before the fixes they prompted, and five of them asserted the pre-fix behaviour: - Qualified default policy: under the qualified contract replay=None now yields to the packed verifier, and an explicit replay=True turns the automatic packed default off (#21). The case asserts that precedence. - Verifier admission (4, 8 requests): with replay on, the layer admits every verify batch (QwenGatedDeltaNetAttention._mixed_qkv_verify_admits) and reads row-strided q/k/v; the module token band is unchanged for replay off. The case checks the layer-level admission _forward_core uses. - Negative zero: summing the owner's row with +0.0 from the other ranks keeps finite nonzero values bitwise and zero values equal, but may clear the sign of -0.0, as #9 and the 0.1.3 release notes state. The case asserts exactly that. No runtime code changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
[Test] Replay and selector review cases, aligned with the 0.1.3 contracts
Reject the incompatible INT8 v1 kernel-page layout regardless of the draft cache dtype. Recommend FP8 E4M3, FP16 or auto with DFlash and MTP with INT8, and remove the misleading separate-cache workaround. Fixes #25 Co-authored-by: Codex <codex@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Adapt vllm-project/vllm#52246 by SayHelloToWorld. Upstream merge: 70aaec832bde70ada17cbad2061105bbd07541dc Use the shared exception mapping in messages and count_tokens before building the Anthropic envelope. Preserve the status-derived error types from #8, including invalid_request_error, not_found_error and api_error. Cover overlong token validation, conversion validation, missing models and internal failures without a model or GPU. Guard the existing HTTP 200/SSE error behavior after a stream starts. Fixes #26 Co-authored-by: Codex <codex@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Adapt vllm-project/vllm#63010 by AdvayMonga. Upstream merge: 6f51433838b858a9cd0478b2ffee4d2c610a1712 Reset next_stored_block_idx with the block-ID mirror when a request restarts. Prefix blocks evicted from the CPU tier can then be stored again; prepare_store filters keys that are still resident. Keep this before allocation so a CPU hit can advance the cursor afterwards. Port both cpu_hit cases of test_request_preemption for synchronous and asynchronous scheduling, retaining the existing pending-store fence. Requests that never restart keep their scheduling and numerical paths. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Adapt vllm-project/vllm#55700, by wenjinhust and Jiangyun Zhu, from upstream commit 1c879919786497217b3b5fb0258d1d72f2442cb0. Keep monitoring disabled unless dump_dir is set. Suspend it for idle input, feed around RPCs and steps, and stop the thread during cleanup. Handle SIGTERM re-entry and dump errors without changing model execution. Keep diagnostic settings out of the computation-graph configuration hash. Cover the CLI, short stalls, recovery, signals and thread lifecycle on CPU. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
[Bugfix] Anthropic 4xx for client errors, and KV offload store cursor reset on restart
[Bugfix] Reject INT8 KV with DFlash2 at configuration time; opt-in watchdog stack dumps
One info line per capture: the verify shape, the layer count, the capture time and the allocator reserve it added, and the number of replay graphs so far, so a deployment can check the first-capture stall and the memory the replay graphs take beyond the reserve sized at startup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Replay becomes an opt-in for more KV capacity: SM70_DFLASH2_SPEC_STATE_REPLAY_DEFAULT is now False, so spec_state_replay=None resolves off, matching the recommended configuration. A user who turns replay on gets the replay CUDA graph by default (spec_state_replay_graph stays True). Configuration hash: the replay factor enters it only when replay is on, so a deployment that already sets spec_state_replay=false keeps its factors. Code hash: vllm/config/sm70_dflash2.py is likely among the files Dynamo traces (the layer-norm predicates call sm70_dflash2_enabled in forward), so editing it changes the compile cache key as in 0.1.3 and deployments compile once; with replay off the arithmetic is unchanged. Tests follow the default: replay off unless requested; the release-default switch and the policy precedence cases set the default explicitly; the allocation and admission cases request replay so they still exercise it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
docs/dflash2.md gains a "State replay (optional)" section: how to turn it
on, the KV gain (1,379,634 to 1,542,558 tokens, +11.8%), the cost measured
in one session (about +0.44 ms per step with sampling, 1-4% at one to four
greedy requests, level at eight), who it suits, and the replay CUDA graph:
eight graphs with the {4: 3, 5: 4} width policy, 35-90 ms once per shape,
no measurable GPU memory, and a full KV pool at eight 96K-token requests
without preemption. docs/performance.md adds the four-way comparison.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
[GDN] Replay: one-launch replay tables and a CUDA graph for the post-sampling replay
Route non-push FP16 sum2 shapes directly to PyNccl on SM70 TP4 PCIe-switch groups. Match the native raw-environment gates, including M2's explicit FASTPATH requirement, and retain every push launch. Reuse the existing PyNccl/torch fallback without retrying ordinary custom all-reduce. Keep ordinary dispatch and its byte limit unchanged. CPU validation: 350 tests pass; all 165 new baseline failures are fixed. GPU numerical, performance and model-quality checks remain pending. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Fall back to all_reduce(a + b) for non-push SM70 PCIe sum2 sizes. This lets small captured sums use ordinary push instead of bypassing custom all-reduce for PyNccl. Keep the native sum2 push path unchanged. Exercise T=1..64 both inside and outside graph capture through the complete ordinary dispatch, including the PyNccl and torch fallbacks. Ordinary admission, native code and the byte limit remain unchanged. GPU numerical, performance and model-quality checks remain pending. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Distinguish measured KV capacity by draft dtype and release, and clarify quality references, PLE disk checks, and historical replay behavior. Refresh setup and support details against the released measurements and current source, including the Anthropic Messages routes. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
[Doc] Correct public documentation inconsistencies
Adapt vllm-project/vllm#52807 by yifjiang. Upstream merge: da8ec2826898d47bd2fb424bc607052a6dc515c1 update_state_after_alloc scanned each group's block table from index 0 for the first non-null block without a hash and took it as the start of the load region. A non-null unhashed block below the local prefix hit pulled that boundary below num_locally_computed_tokens; the boundary assertion then failed and the engine core exited. Start the scan at cdiv(num_locally_computed_tokens, gpu_block_size), keep skipping null padding above it, and assert that the local hit is block aligned. Rename the value to load_start_gpu_block_idx as upstream does, keeping this tree's gpu_block_size and block_size_factor names. Requests without a local hit, and requests whose blocks below the hit are all hashed or null, select the same blocks and keys as before. Port the regression with the 1632/8 and 1664/16 state-page geometries and pin the unchanged cold and hashed-prefix paths. Co-authored-by: Codex <codex@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Adapt vllm-project/vllm#44409 by Saddss, which supersedes vllm-project/vllm#33775. Upstream merge: 588db1836245bb40589d45b3e87048a53a13cfe5 allocate_new_computed_blocks ran each KV cache group in turn: touch the group's local prefix hits, then take blocks for external (connector) tokens. An earlier group's get_new_blocks could therefore pop a later group's untouched, evictable hit block from the free-queue head, clear its hash and use it as a load destination; the later group then touched the same block and kept it below its hit. One physical block was owned by two groups. The offloading connector found the unhashed block below the local hit and failed its load-boundary assertion, so the engine core exited; without that assertion the load would overwrite the later group's prefix KV. Touch every group's hits first, then allocate external blocks for every group. As upstream does, split the manager method into add_local_computed_blocks and allocate_external_computed_blocks and move the running-request fast path into the coordinator. The ring, KPool tail and cross-attention managers keep their no-op behavior in both phases. With the DFlash2 group order (full attention, ring, GDN/PLE states, the drafter window last), the drafter pages below the hit are the victims when a GPU hit is followed by an external hit shorter than the drafter window. The MTP4 layout holds hits only in its first group. Requests without external tokens run the same single phase as before. Port the upstream tests and add a real-manager test that reproduces the connector assertion with the DFlash2 layout and passes with MTP4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
[Bugfix][KV Cache] Two-phase allocation for cross-group prefix hits; offload load boundary scan
mypy 3.12 needs an element type for the empty set literal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
[Perf] SM70 PCIe TP4: avoid the generic sum2 pull kernel (MTP4 at 2-3 requests)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 0.1.4 from release/0.1.4 into prod. The frozen source is dfc092c.
Merge commit only. The merge tree must equal the accepted source tree. The v0.1.4 tag and the published wheel, image and PyPI file point to the accepted source commit; the deployment tag points to this merge commit.
Changes since 0.1.3: the engine no longer exits with an OffloadingConnector assertion under long-context traffic with CPU KV offload and DFlash2 (#36); KV offload stores a prefix again after a request restarts from token 0, and the Anthropic endpoint returns 4xx for client errors (#32, fixes #26); INT8 KV with DFlash2 is rejected at configuration time, and an opt-in stack-dump watchdog is added (#33, fixes #25); DFlash2 state replay becomes optional and off by default, with the replay running as a CUDA graph (#24); MTP4 at two and three concurrent requests avoids the generic sum2 pull kernel on PCIe-switch TP4 (#34); FP4-Local KV scales include the MTP draft scales (#27); documentation updates (#19, #29, #30, #35).
Release gate on the release image (0.1.4 native, 4 x V100 PCIe, reference configuration, 24 arms) passed. Single-request greedy output is identical to 0.1.3 for MTP4 and for DFlash2 with greedy drafting. Both stress replays passed, and the CPU-offload replay that triggered #36 ran clean on 0.1.4 (2,808 of 2,808 requests). Multimodal, tools, CPU and SSD offload, developer-mode endpoints, the draft RoPE check, INT8 KV with MTP4, the Anthropic error paths and 512K context passed; 30 minutes of eight concurrent requests showed flat host memory. Two checks were accepted with reasons: the C8 nucleus check in the DFlash2 arm is a false positive of its separate re-scoring prefill (decode-time probabilities show no violations), and the 0.5 ms limit of the optional state-replay session check was set for greedy drafting (0.8 ms with probabilistic drafting; replay is off by default). MTP4 at two concurrent requests is about 3% faster than 0.1.3; a repeated A/B measurement at three to five requests found DFlash2 within about 2% per step of 0.1.3, with the same acceptance length.
Go-live: the reference deployment moves from 0.1.3 DFlash2 (greedy drafting, state replay off) to 0.1.4 DFlash2 with probabilistic drafting and state replay off. This PR is merged only after the post-promotion check and the go-live smoke pass on the reference machine.
Reference deployment parameters keep TP4, DCP2, E4M3 KV, prefix caching, align mode, 48 GiB CPU offload, 512K maximum context, eight sequences, and 8192 batched tokens. The multimodal input limit is 16 images and one video. For DFlash2, the profile supplies the graph memory budget and disables tail graphs, selector codebook sharding uses its default, drafting is probabilistic ("draft_sample_method": "probabilistic"), and state replay stays off ("sm70_dflash2": {"spec_state_replay": false}, now also the default).
AI assistance: Codex and Claude assisted with the fixes, release tooling and validation. The submitting human reviews the changes, actual checks, and hardware acceptance evidence.