diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 97ad947fa..19d865b6e 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -5,14 +5,14 @@ body: - type: markdown attributes: value: | - **Support scope for 0.1.0** (see the support matrix in the README): + **Support scope** (see the support matrix in the README): - - **Validated:** 4 × V100 32 GB PCIe, TP4 + DCP2, Qwen3.8 Flash-Next + - **Validated:** 4 × V100 32 GB, TP4 + DCP2, Qwen3.8 Flash-Next (multimodal) with the AWQ g32, standard NVFP4 or NVIDIA FP4-Local checkpoints; E4M3 or INT8 KV cache; KV offload off, CPU, or CPU + SSD; - PCIe switch with P2P. PCIe without P2P was validated by disabling P2P on - P2P-capable hardware. - - **Unverified, best effort:** NVLink, TP2/TP8, V100 16 GB, other GPU counts, + NVLink, or a PCIe switch with P2P (the recommended interconnects). PCIe + without P2P was validated by disabling P2P on P2P-capable hardware. + - **Unverified, best effort:** TP2/TP8, V100 16 GB, other GPU counts, other models and other topologies. - **Out of scope:** GPUs other than SM70, ROCm/CPU/XPU/TPU, GGUF. INT8 KV with the SSD tier is not supported in 0.1.0. @@ -52,7 +52,7 @@ body: options: - PCIe switch with P2P - PCIe without P2P - - NVLink (unverified) + - NVLink - Other (describe in the reproduction) validations: required: true diff --git a/.github/SUPPORT.md b/.github/SUPPORT.md index 5b1e38d67..dd8d75509 100644 --- a/.github/SUPPORT.md +++ b/.github/SUPPORT.md @@ -5,17 +5,17 @@ In short: | Area | Scope | | --- | --- | -| GPUs | 4 × NVIDIA V100 32 GB PCIe | -| PCIe switch with P2P | Validated | +| GPUs | 4 × NVIDIA V100 32 GB (PCIe or SXM2) | +| PCIe switch with P2P | Recommended; validated | | PCIe without P2P | Validated by disabling P2P on P2P-capable hardware; about 30–55% of P2P speed | -| NVLink | Community results on 4 × V100-SXM2 32 GB ([#2](https://github.com/Leonccaa/ReVolta/issues/2)); outside our release gate | +| NVLink | Recommended; tested with community results on 4 × V100-SXM2 32 GB ([#2](https://github.com/Leonccaa/ReVolta/issues/2), [#38](https://github.com/Leonccaa/ReVolta/issues/38), [#37](https://github.com/Leonccaa/ReVolta/issues/37)) | | Host memory | 128 GiB recommended; 80 GiB measured minimum with KV offload off | | KV offload | Off, CPU, or CPU + SSD tier | | KV cache | E4M3 (default) or INT8; INT8 with the SSD tier is not supported | | Models | Qwen3.8 Flash-Next (QSA, multimodal) | | Checkpoints | AWQ g32, standard NVFP4 `RadixArk/Qwen3.8-Flash-Next-NVFP4`, `nvidia/Qwen3.8-Flash-Next-FP4-Local` | -Configurations outside our release gate (NVLink, TP2/TP8, V100 16 GB, other models or +Configurations outside our release gate (TP2/TP8, V100 16 GB, other models or topologies) are handled on a best-effort basis. Other GPU architectures, ROCm/CPU/XPU/TPU serving and GGUF are not built. diff --git a/README.md b/README.md index 69c60dbbc..32fd8fb16 100644 --- a/README.md +++ b/README.md @@ -4,24 +4,24 @@ ReVolta turns a server with V100 32 GB GPUs and plenty of system memory into a dependable daily AI service. It runs Qwen3.8 Flash-Next for text, images and video behind an OpenAI-compatible API. ReVolta is built on [1Cat-vLLM](https://github.com/1CatAI/1Cat-vLLM), a V100-focused fork of vLLM, so the `vllm` command, its options and its OpenAI-compatible API work as usual. -It is built for four or more V100 32 GB GPUs and validated on four. Every release passes the same gate on our own server before it ships: single-request greedy output identical to the previous release token for token, 4 and 8 concurrent requests, images and video, CPU cache readback, and 512K-token prompts. That server runs ReVolta every day. +It is built for four or more V100 32 GB GPUs and validated on four. We recommend, and test, the two interconnects that give the best speed: NVLink, or all GPUs under one PCIe switch with P2P. Every release passes the same gate on our own server before it ships: single-request greedy output identical to the previous release token for token, 4 and 8 concurrent requests, images and video, CPU cache readback, and 512K-token prompts. That server runs ReVolta every day. -**You need:** 4 × V100 32 GB (PCIe with P2P, or NVLink), at least 64 GB of system memory (128 GB recommended), Linux x86-64 with glibc 2.34 or newer, and an NVIDIA driver for CUDA 12.8. Tested hardware and limits: [Support and known limits](docs/known-limits.md). +**You need:** 4 × V100 32 GB (NVLink or PCIe; NVLink or one PCIe switch with P2P recommended), at least 64 GB of system memory (128 GB recommended), Linux x86-64 with glibc 2.34 or newer, and an NVIDIA driver for CUDA 12.8. Tested hardware and limits: [Support and known limits](docs/known-limits.md). -Current release: [0.1.3](https://github.com/Leonccaa/ReVolta/releases/tag/v0.1.3). +Current release: [0.1.4](https://github.com/Leonccaa/ReVolta/releases/tag/v0.1.4). ## At a glance -**Context and capacity** (Uncensored AWQ, E4M3 KV cache, 0.1.3) +**Context and capacity** (Uncensored AWQ, E4M3 KV cache, 0.1.4) | | MTP4 (default) | DFlash2 (recommended settings) | | --- | ---: | ---: | | Context per request | 512K tokens | 512K tokens | -| GPU KV pool | 1.15 million tokens | **1.38 million tokens** | -| GPU KV pool / 512K tokens (capacity ratio) | 2.20× | **2.63×** | +| GPU KV pool | 1.15 million tokens | **1.37 million tokens** | +| GPU KV pool / 512K tokens (capacity ratio) | 2.20× | **2.61×** | | Concurrent requests | 8 | 8 | -Prefixes evicted from the GPU stay in CPU memory or on disk: a cached 32K-token document is answered in **1.28 s instead of 10.06 s**. See [KV offload](docs/kv-offload.md). +Prefixes evicted from the GPU stay in CPU memory or on disk: a cached 32K-token document comes back from CPU memory in **0.6 s**, and after an engine restart from a local NVMe disk tier in **1.5 s**, instead of **10.1 s** when recomputed (0.1.4, DFlash2). See [KV offload](docs/kv-offload.md). **Speed** (total output tokens/s, 512-token responses; best of MTP4 and DFlash2) @@ -36,7 +36,7 @@ Versions, power caps and the per-method breakdown are in [Performance](docs/perf | Model | Weights (bits per weight: routed experts / whole model) | Download | PLE table in host memory | GPU KV pool (MTP4) | Quality vs its BF16 parent: KL ↓ / top-1 ↑ | Notes | | --- | --- | ---: | ---: | ---: | ---: | --- | -| [Uncensored AWQ g32](https://huggingface.co/leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32) **(recommended)** | 4-bit AWQ g32 experts, FP8 PLE, BF16 rest (**4.63 / 5.95**) | 142 GB, MTP draft included | 48 GiB | 1.15 M (1.38 M with DFlash2) | 0.0214 / 97.1% | Uncensored fine-tune; the reference setup | +| [Uncensored AWQ g32](https://huggingface.co/leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32) **(recommended)** | 4-bit AWQ g32 experts, FP8 PLE, BF16 rest (**4.63 / 5.95**) | 142 GB, MTP draft included | 48 GiB | 1.15 M (1.37 M with DFlash2) | 0.0214 / 97.1% | Uncensored fine-tune; the reference setup | | [Base AWQ g32](https://huggingface.co/leoncca/Qwen3.8-Flash-Next-AWQ-g32) | 4-bit AWQ g32 experts, FP8 PLE, BF16 rest (**4.63 / 5.95**) | 142 GB | 48 GiB | Same layout as Uncensored | 0.0207 / 97.1% | The original model | | [Standard NVFP4](https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4) | NVFP4 experts, FP8 PLE, BF16 rest (**4.50 / 5.86**) | 135 GB + 5 GB draft pack | 48 GiB | 1.25 M | 0.0303 / 96.4% | Widely used community format | | [NVIDIA FP4-Local](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local) | NVFP4 experts and PLE, MXFP8 attention and shared experts (**4.26 / 4.53**) | 102 GB, MTP layer built in | **27 GiB** | **1.47 M** | 0.0380 / 95.9% | NVIDIA's official checkpoint; least host memory | @@ -53,14 +53,14 @@ The package is `revolta`; the Python import and CLI remain `vllm`. ```bash python3.12 -m venv /data/revolta/venv . /data/revolta/venv/bin/activate -pip install revolta==0.1.3 -vllm --version # 0.1.3 +pip install revolta==0.1.4 +vllm --version # 0.1.4 ``` Or pull the Docker image: ```bash -docker pull ghcr.io/leonccaa/revolta:0.1.3 +docker pull ghcr.io/leonccaa/revolta:0.1.4 ``` Follow [Getting started](docs/getting-started.md) to download the pinned checkpoint, @@ -83,10 +83,10 @@ For another checkpoint, use its [preparation guide](docs/checkpoints.md) before ## Known limits -- V100 16 GB, TP2/TP8, and other models have not been tested; NVLink has community results and is outside the release gate. +- V100 16 GB, TP2/TP8, and other models have not been tested. - This release builds SM70 only; other GPU architectures, ROCm, CPU, XPU, TPU, and GGUF are outside the build. - Greedy outputs at C2–C8 can change between runs; fresh compiler autotuning can also change outputs. -- The AWQ PLE table needs about 48 GiB of host memory by default. An existing experimental option keeps the whole table on disk instead; functional checks in 0.1.3 ran under 64 GiB and 32 GiB container limits without CPU KV offload, but speed was not measured. See [Support and known limits](docs/known-limits.md). +- The AWQ PLE table needs about 48 GiB of host memory by default. An experimental option keeps the whole table on a local SSD instead: in 0.1.4 it served under 64 GiB and 32 GiB memory limits, also with a 32 GiB CPU KV pool under 64 GiB, and greedy output matched the default path. Single-request decoding is about 23–25% slower, and a cold first token takes 1.7–2.2 times as long. See [Support and known limits](docs/known-limits.md). - PLE cascade (spilling part of the table to disk, `ple_disk_cascade=true`) is not supported yet and is off by default in the ReVolta profile; full support is planned for 0.2. - INT8 KV cannot be combined with DFlash2; the disk tier does not support INT8 yet. @@ -98,8 +98,10 @@ ReVolta focuses on dependable daily service on V100. Releases check output corre long-context retrieval, concurrent use, multimodal input, cache recovery, and speed on the reference machine. -Next: faster INT8 KV decoding, simpler NVFP4 setup, INT8 disk caching, and community -testing on NVLink systems. Longer-term work includes reducing host-memory use and +Next (0.1.5): faster all-reduce on NVLink above 16 tokens per step, a cheaper DFlash2 +state replay, smaller DFlash2 CPU and disk cache pages, and a 512K long-context +retrieval self-test. Also planned: faster INT8 KV decoding, simpler NVFP4 setup and +INT8 disk caching. Longer-term work includes reducing host-memory use and runtime dependencies. ## Getting help @@ -113,7 +115,10 @@ Reports from NVLink V100 owners are especially welcome; share measured results w [Test results](https://github.com/Leonccaa/ReVolta/issues/new?template=test_results.md) template ([`.github/ISSUE_TEMPLATE/test_results.md`](.github/ISSUE_TEMPLATE/test_results.md); agents can fill it in and open the issue through the API). Thanks to -[@jttdev](https://github.com/jttdev) for sharing their results ([#2](https://github.com/Leonccaa/ReVolta/issues/2)). Report security issues +[@jttdev](https://github.com/jttdev), [@That80-20](https://github.com/That80-20) and +[@vincentyan2023](https://github.com/vincentyan2023) for sharing their results +([#2](https://github.com/Leonccaa/ReVolta/issues/2), [#38](https://github.com/Leonccaa/ReVolta/issues/38), +[#37](https://github.com/Leonccaa/ReVolta/issues/37)). Report security issues through GitHub private vulnerability reporting. For project status and development background, read the [pinned overview (#1)](https://github.com/Leonccaa/ReVolta/issues/1). diff --git a/docs/known-limits.md b/docs/known-limits.md index 65562fe50..7f7ae82c2 100644 --- a/docs/known-limits.md +++ b/docs/known-limits.md @@ -3,6 +3,9 @@ This reference lists the tested support scope and known limitations. Check it before using the [launch tutorial](getting-started.md) or changing hardware, formats or request limits. +ReVolta is designed for four or more V100 32 GB GPUs and validated on four. NVLink, or +all GPUs under one PCIe switch with P2P, are the recommended and tested interconnects. + The reference system uses four V100 32 GB PCIe cards behind one PCIe switch (PIX/PXB in `nvidia-smi topo -m`), with P2P enabled and a 128 GiB container memory limit. @@ -10,7 +13,7 @@ The reference system uses four V100 32 GB PCIe cards behind one PCIe switch | --- | --- | | 4 × V100 32 GB PCIe, TP4 + DCP2, P2P | Full acceptance passed on the reference configuration | | P2P disabled on the same machine | Correct output; about 30–55% of P2P throughput | -| 4 × V100-SXM2 32 GB, NVLink (NV2 mesh) | Community test by [@jttdev](https://github.com/jttdev) ([#2](https://github.com/Leonccaa/ReVolta/issues/2)): speed, retrieval to about 490K tokens, vision. Not part of our release gate | +| 4 × V100-SXM2 32 GB, NVLink | Tested by the community: [@jttdev](https://github.com/jttdev) ([#2](https://github.com/Leonccaa/ReVolta/issues/2)): speed, retrieval to about 490K tokens, vision; also [#38](https://github.com/Leonccaa/ReVolta/issues/38) and [#37](https://github.com/Leonccaa/ReVolta/issues/37) | | V100 16 GB, other GPU counts | Not yet tested | **Full acceptance on the reference setup:** default Uncensored AWQ, Standard NVFP4, @@ -18,8 +21,7 @@ and FP4-Local with their documented settings. Base AWQ has quality measurements. 80 GiB RAM has memory-stress results. Disk caching has readback results; its full release gate passed with an 8 GiB CPU pool and a 40 GiB disk budget. -V100 16 GB, TP2/TP8, and other models have not been tested. NVLink has a community -test ([#2](https://github.com/Leonccaa/ReVolta/issues/2)) but is not part of our release gate. +V100 16 GB, TP2/TP8, and other models have not been tested. This release builds SM70 only; other GPU architectures, ROCm, CPU, XPU, TPU, and GGUF are outside the build. @@ -44,14 +46,14 @@ GGUF are outside the build. and is off by default in the ReVolta profile; full support is planned for 0.2. Profiling with it enabled used about 4 GiB more GPU memory per 32 GB card, which left room for about 345K context tokens. -- **Whole PLE table on disk (experimental; first checked in 0.1.3).** Instead of pinned host memory, the +- **Whole PLE table on disk (experimental; first checked in 0.1.3, measured in 0.1.4).** Instead of pinned host memory, the whole table can stay as file-backed mappings of the checkpoint's safetensors files, read by a separate PLE process. This is a different path from cascade and is off by default. To try it, set `VLLM_PLE_CPU_OFFLOAD=1`, `VLLM_PLE_DISK_OFFLOAD=1` and `VLLM_SM70_QWEN38_HYBRID_PLE=0`, add `--safetensors-load-strategy lazy`, keep - `"ple_disk_cascade": false` in `--kernel-config` (the check also set - `"ple_disk_release_pages": true`), and run without CPU KV offload - (`--kv-transfer-config`); that combination was not tested. + `"ple_disk_cascade": false` in `--kernel-config` (the checks also set + `"ple_disk_release_pages": true`). CPU KV offload (`--kv-transfer-config`) can stay + on; 0.1.4 measured it with a 32 GiB CPU pool. - Checked on the 0.1.3 candidate with Uncensored AWQ and E4M3 KV: MTP4 served 131K-token prompts, 4 concurrent requests and images under both a 64 GiB and a 32 GiB container memory limit, and a 491K-token prompt under 64 GiB; DFlash2 @@ -62,9 +64,21 @@ GGUF are outside the build. single-request outputs diverged at a near-tie, so the two paths are not bitwise identical. If the PLE process dies, the client gets an error after about 9 s and the engine exits after 11–13 s instead of hanging. - - Not measured: decode speed and time to first token, a machine with little - physical RAM, a local SSD (the check read the checkpoint from network storage), - and the net memory saving against the default path. + - Measured on 0.1.4 with DFlash2 (recommended settings), Uncensored AWQ, E4M3 KV and + 250 W, with the table read from a local NVMe or SATA SSD. Under a 64 GiB limit + (NVMe and SATA), a 32 GiB limit (NVMe), and a 64 GiB limit with a 32 GiB CPU KV + pool (NVMe and SATA), the engine started and served without running out of memory, + and three 64-token greedy outputs matched the pinned-memory path exactly. + Single-request decoding was 101.5–104.9 tokens/s against 135.5 with the pinned + table, about 23–25% slower. A cold first token took 5.4–5.9 / 16.8–17.9 / + 65.6–69.8 s at 8K / 32K / 131K prompt tokens, against 2.61 / 9.66 / 40.40 s, so + 1.7–2.2 times as long; once the table pages for a prompt had been read, the first + token was within 2–3% of the pinned path (5–8% with the 32 GiB CPU KV pool). NVMe + and SATA, and the 32 and 64 GiB limits, differed little. Peak memory of the + serving container was about 57 GiB under the 64 GiB limit (62 GiB with the CPU KV + pool) and 30 GiB under 32 GiB. + - Not measured: a machine with little physical RAM (the limits above were set on the + container that runs the engine), and long-running stability on this path. - **FP4-Local vision.** Keep the default vision-tower TP4. Forcing data parallelism removes the tested memory arrangement. Non-empty request-level `mm_processor_kwargs` and `media_io_kwargs` are rejected with HTTP 400 by diff --git a/docs/kv-offload.md b/docs/kv-offload.md index d7607ae82..3186b6333 100644 --- a/docs/kv-offload.md +++ b/docs/kv-offload.md @@ -30,8 +30,11 @@ computed again. } ``` -`cpu_bytes_to_use` sets the pool size (total for all workers). With 48 GiB, the pool -holds about 2.4M tokens. `mamba_state_slots_reference_tokens` sets how many +`cpu_bytes_to_use` sets the pool size (total for all workers). With MTP4, a 48 GiB pool +holds about 2.4M tokens. With DFlash2, the drafter uses a different cache block size, so +every cache group takes pages of the full block size and a prefix needs about 50 KB per +token in the CPU pool and on disk (measured in 0.1.4: 1.84 GB for a 32K-token prefix, +6.36 GB for 131K), about 2.5 times as much as with MTP4. `mamba_state_slots_reference_tokens` sets how many recurrent-state slots each layer group gets. A smaller reference gives more slots, which helps when many short sessions share the pool. @@ -78,15 +81,21 @@ use a larger CPU pool and set `max_size_bytes` to your disk budget. - The SSD tier is a disposable cache, not model storage. - A prefix restored from disk is staged in the CPU pool first, so `cpu_bytes_to_use` must be larger than the largest prefix you expect to restore from disk (about 21 KB per - token; a 100K-token prefix needs more than 2 GiB). Otherwise the request is computed + token with MTP4 and about 50 KB with DFlash2; a 100K-token prefix needs more than + 2 GiB with MTP4 and about 5 GB with DFlash2). Otherwise the request is computed again. The quick-start 48 GiB pool holds about 2.4M tokens. - The SSD tier is **not supported with INT8 KV** yet. That combination is rejected at configuration time. -Validation scope: the SSD tier was tested with its root on a network file system. The -full release gate passed with an 8 GiB CPU pool and a 40 GiB disk budget, including a -99K-token prefix restored from disk (96,288 cached tokens, output identical). SSD -throughput was not measured. +Validation scope: the full release gate passed with the SSD tier on a network file +system (8 GiB CPU pool, 40 GiB disk budget), including a 99K-token prefix restored from +disk (96,288 cached tokens, output identical). In 0.1.4 the tier was also measured on a +local NVMe SSD with DFlash2 (7 GiB CPU pool, 64 GiB disk budget, one read and one write +thread). After an engine restart, 11 of 11 documents were read back from disk with the +same 64-token greedy output as a cold run, at about 2.2–2.4 GB/s per request; a +32K / 99K / 131K-token prefix came back in 1.47 / 2.93 / 3.77 s instead of +10.12 / 31.88 / 42.43 s when recomputed. A root holding 27.8 GB in 1,088 files did not +slow the restart. See [Coming back to a document](performance.md#coming-back-to-a-document). ## See also diff --git a/docs/performance.md b/docs/performance.md index 89aaf2328..225814448 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -26,7 +26,26 @@ method (MTP4 and DFlash2) and prompt type is in [Performance](#performance). ### Coming back to a document -No ReVolta version is specified for this readback measurement; its request and cache conditions follow below. +Measured on 0.1.4 with DFlash2 (recommended settings), Uncensored AWQ, E4M3 KV, 250 W, +the pinned PLE table, a 7 GiB CPU pool and a disk tier on a local NVMe SSD. Each +document was first computed cold. The CPU readback followed after the GPU prefix cache +was cleared; the disk readback followed an engine restart with the page cache dropped. +Both readbacks hit the whole cacheable prefix, and the disk readbacks gave the same +64-token greedy output as the cold run. + +| Prompt tokens | Recomputed | From CPU memory | From NVMe after a restart | +| ---: | ---: | ---: | ---: | +| 32,768 | 10.12 s | 0.61 s | 1.47 s | +| 99,000 | 31.88 s | 0.77 s | 2.93 s | +| 131,072 | 42.43 s | 0.99 s | 3.77 s | + +Times are to the first token, one request at a time. Eight more 32K-token documents +gave 0.62–0.66 s from CPU memory and 1.41–1.46 s from NVMe for the first four of them. +Four 32K-token documents read from NVMe at the same time after the restart took +1.48–3.84 s each (median 2.67 s). + +An earlier measurement, with no ReVolta version recorded, under the request and cache +conditions below: A cached 32K-token document answered a follow-up in **1.28 seconds from memory**, compared with **10.06 seconds when recomputed**. Both produced identical output. @@ -39,8 +58,8 @@ compared with **10.06 seconds when recomputed**. Both produced identical output. | CPU memory + filesystem cache | 1.60 s | The request generated 64 tokens and reused 31,008 cached prompt tokens. A small GPU -pool forced eviction. The filesystem tier was tested on a network filesystem; -local SSD performance has not yet been measured. +pool forced eviction. The filesystem tier in this earlier measurement was on a network +filesystem. ### Quantization quality