From 87f3314a05adb1b78f28e077a12390fe1c374769 Mon Sep 17 00:00:00 2001 From: Leonccaa <166551845+Leonccaa@users.noreply.github.com> Date: Tue, 27 Oct 2026 11:30:46 -0700 Subject: [PATCH 1/2] [Doc] 0.1.4: current release, PLE and KV disk-tier measurements Update the README and guides for 0.1.4 and add the measurements taken after its release: - README: 0.1.4 is the current release (install commands, image tag, capacity table with the recommended probabilistic DFlash2 pool of 1.37M tokens / 2.61x), readback times from CPU memory and from a local NVMe disk tier after a restart, the measured cost of the experimental whole-table PLE disk option, 0.1.5 plans, and thanks to the authors of #38 and #37. - Positioning: ReVolta is built for 4 x V100 32 GB; NVLink and one PCIe switch with P2P are the recommended and tested interconnects (README, known-limits, SUPPORT.md, bug report form). - known-limits: whole-table PLE disk option measured on 0.1.4 (local NVMe and SATA, 64 and 32 GiB limits, with and without a 32 GiB CPU KV pool). - kv-offload: DFlash2 needs about 50 KB per token in the CPU pool and on disk (about 2.5x MTP4); local NVMe results after a restart. - performance: 0.1.4 readback table (recomputed, CPU memory, NVMe after a restart). No code changes. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug_report.yml | 12 ++++----- .github/SUPPORT.md | 8 +++--- README.md | 37 +++++++++++++++------------ docs/known-limits.md | 34 ++++++++++++++++-------- docs/kv-offload.md | 23 ++++++++++++----- docs/performance.md | 25 +++++++++++++++--- 6 files changed, 93 insertions(+), 46 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 97ad947fa7..19d865b6e5 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -5,14 +5,14 @@ body: - type: markdown attributes: value: | - **Support scope for 0.1.0** (see the support matrix in the README): + **Support scope** (see the support matrix in the README): - - **Validated:** 4 × V100 32 GB PCIe, TP4 + DCP2, Qwen3.8 Flash-Next + - **Validated:** 4 × V100 32 GB, TP4 + DCP2, Qwen3.8 Flash-Next (multimodal) with the AWQ g32, standard NVFP4 or NVIDIA FP4-Local checkpoints; E4M3 or INT8 KV cache; KV offload off, CPU, or CPU + SSD; - PCIe switch with P2P. PCIe without P2P was validated by disabling P2P on - P2P-capable hardware. - - **Unverified, best effort:** NVLink, TP2/TP8, V100 16 GB, other GPU counts, + NVLink, or a PCIe switch with P2P (the recommended interconnects). PCIe + without P2P was validated by disabling P2P on P2P-capable hardware. + - **Unverified, best effort:** TP2/TP8, V100 16 GB, other GPU counts, other models and other topologies. - **Out of scope:** GPUs other than SM70, ROCm/CPU/XPU/TPU, GGUF. INT8 KV with the SSD tier is not supported in 0.1.0. @@ -52,7 +52,7 @@ body: options: - PCIe switch with P2P - PCIe without P2P - - NVLink (unverified) + - NVLink - Other (describe in the reproduction) validations: required: true diff --git a/.github/SUPPORT.md b/.github/SUPPORT.md index 5b1e38d67e..dd8d75509b 100644 --- a/.github/SUPPORT.md +++ b/.github/SUPPORT.md @@ -5,17 +5,17 @@ In short: | Area | Scope | | --- | --- | -| GPUs | 4 × NVIDIA V100 32 GB PCIe | -| PCIe switch with P2P | Validated | +| GPUs | 4 × NVIDIA V100 32 GB (PCIe or SXM2) | +| PCIe switch with P2P | Recommended; validated | | PCIe without P2P | Validated by disabling P2P on P2P-capable hardware; about 30–55% of P2P speed | -| NVLink | Community results on 4 × V100-SXM2 32 GB ([#2](https://github.com/Leonccaa/ReVolta/issues/2)); outside our release gate | +| NVLink | Recommended; tested with community results on 4 × V100-SXM2 32 GB ([#2](https://github.com/Leonccaa/ReVolta/issues/2), [#38](https://github.com/Leonccaa/ReVolta/issues/38), [#37](https://github.com/Leonccaa/ReVolta/issues/37)) | | Host memory | 128 GiB recommended; 80 GiB measured minimum with KV offload off | | KV offload | Off, CPU, or CPU + SSD tier | | KV cache | E4M3 (default) or INT8; INT8 with the SSD tier is not supported | | Models | Qwen3.8 Flash-Next (QSA, multimodal) | | Checkpoints | AWQ g32, standard NVFP4 `RadixArk/Qwen3.8-Flash-Next-NVFP4`, `nvidia/Qwen3.8-Flash-Next-FP4-Local` | -Configurations outside our release gate (NVLink, TP2/TP8, V100 16 GB, other models or +Configurations outside our release gate (TP2/TP8, V100 16 GB, other models or topologies) are handled on a best-effort basis. Other GPU architectures, ROCm/CPU/XPU/TPU serving and GGUF are not built. diff --git a/README.md b/README.md index 69c60dbbcf..bc79bc6772 100644 --- a/README.md +++ b/README.md @@ -4,24 +4,24 @@ ReVolta turns a server with V100 32 GB GPUs and plenty of system memory into a dependable daily AI service. It runs Qwen3.8 Flash-Next for text, images and video behind an OpenAI-compatible API. ReVolta is built on [1Cat-vLLM](https://github.com/1CatAI/1Cat-vLLM), a V100-focused fork of vLLM, so the `vllm` command, its options and its OpenAI-compatible API work as usual. -It is built for four or more V100 32 GB GPUs and validated on four. Every release passes the same gate on our own server before it ships: single-request greedy output identical to the previous release token for token, 4 and 8 concurrent requests, images and video, CPU cache readback, and 512K-token prompts. That server runs ReVolta every day. +It is built for servers with 4 × V100 32 GB. We recommend, and test, the two interconnects that give the best speed: NVLink, or all four GPUs under one PCIe switch with P2P. Every release passes the same gate on our own server before it ships: single-request greedy output identical to the previous release token for token, 4 and 8 concurrent requests, images and video, CPU cache readback, and 512K-token prompts. That server runs ReVolta every day. -**You need:** 4 × V100 32 GB (PCIe with P2P, or NVLink), at least 64 GB of system memory (128 GB recommended), Linux x86-64 with glibc 2.34 or newer, and an NVIDIA driver for CUDA 12.8. Tested hardware and limits: [Support and known limits](docs/known-limits.md). +**You need:** 4 × V100 32 GB (NVLink or PCIe; NVLink or one PCIe switch with P2P recommended), at least 64 GB of system memory (128 GB recommended), Linux x86-64 with glibc 2.34 or newer, and an NVIDIA driver for CUDA 12.8. Tested hardware and limits: [Support and known limits](docs/known-limits.md). -Current release: [0.1.3](https://github.com/Leonccaa/ReVolta/releases/tag/v0.1.3). +Current release: [0.1.4](https://github.com/Leonccaa/ReVolta/releases/tag/v0.1.4). ## At a glance -**Context and capacity** (Uncensored AWQ, E4M3 KV cache, 0.1.3) +**Context and capacity** (Uncensored AWQ, E4M3 KV cache, 0.1.4) | | MTP4 (default) | DFlash2 (recommended settings) | | --- | ---: | ---: | | Context per request | 512K tokens | 512K tokens | -| GPU KV pool | 1.15 million tokens | **1.38 million tokens** | -| GPU KV pool / 512K tokens (capacity ratio) | 2.20× | **2.63×** | +| GPU KV pool | 1.15 million tokens | **1.37 million tokens** | +| GPU KV pool / 512K tokens (capacity ratio) | 2.20× | **2.61×** | | Concurrent requests | 8 | 8 | -Prefixes evicted from the GPU stay in CPU memory or on disk: a cached 32K-token document is answered in **1.28 s instead of 10.06 s**. See [KV offload](docs/kv-offload.md). +Prefixes evicted from the GPU stay in CPU memory or on disk: a cached 32K-token document comes back from CPU memory in **0.6 s**, and after an engine restart from a local NVMe disk tier in **1.5 s**, instead of **10.1 s** when recomputed (0.1.4, DFlash2). See [KV offload](docs/kv-offload.md). **Speed** (total output tokens/s, 512-token responses; best of MTP4 and DFlash2) @@ -36,7 +36,7 @@ Versions, power caps and the per-method breakdown are in [Performance](docs/perf | Model | Weights (bits per weight: routed experts / whole model) | Download | PLE table in host memory | GPU KV pool (MTP4) | Quality vs its BF16 parent: KL ↓ / top-1 ↑ | Notes | | --- | --- | ---: | ---: | ---: | ---: | --- | -| [Uncensored AWQ g32](https://huggingface.co/leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32) **(recommended)** | 4-bit AWQ g32 experts, FP8 PLE, BF16 rest (**4.63 / 5.95**) | 142 GB, MTP draft included | 48 GiB | 1.15 M (1.38 M with DFlash2) | 0.0214 / 97.1% | Uncensored fine-tune; the reference setup | +| [Uncensored AWQ g32](https://huggingface.co/leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32) **(recommended)** | 4-bit AWQ g32 experts, FP8 PLE, BF16 rest (**4.63 / 5.95**) | 142 GB, MTP draft included | 48 GiB | 1.15 M (1.37 M with DFlash2) | 0.0214 / 97.1% | Uncensored fine-tune; the reference setup | | [Base AWQ g32](https://huggingface.co/leoncca/Qwen3.8-Flash-Next-AWQ-g32) | 4-bit AWQ g32 experts, FP8 PLE, BF16 rest (**4.63 / 5.95**) | 142 GB | 48 GiB | Same layout as Uncensored | 0.0207 / 97.1% | The original model | | [Standard NVFP4](https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4) | NVFP4 experts, FP8 PLE, BF16 rest (**4.50 / 5.86**) | 135 GB + 5 GB draft pack | 48 GiB | 1.25 M | 0.0303 / 96.4% | Widely used community format | | [NVIDIA FP4-Local](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local) | NVFP4 experts and PLE, MXFP8 attention and shared experts (**4.26 / 4.53**) | 102 GB, MTP layer built in | **27 GiB** | **1.47 M** | 0.0380 / 95.9% | NVIDIA's official checkpoint; least host memory | @@ -53,14 +53,14 @@ The package is `revolta`; the Python import and CLI remain `vllm`. ```bash python3.12 -m venv /data/revolta/venv . /data/revolta/venv/bin/activate -pip install revolta==0.1.3 -vllm --version # 0.1.3 +pip install revolta==0.1.4 +vllm --version # 0.1.4 ``` Or pull the Docker image: ```bash -docker pull ghcr.io/leonccaa/revolta:0.1.3 +docker pull ghcr.io/leonccaa/revolta:0.1.4 ``` Follow [Getting started](docs/getting-started.md) to download the pinned checkpoint, @@ -83,10 +83,10 @@ For another checkpoint, use its [preparation guide](docs/checkpoints.md) before ## Known limits -- V100 16 GB, TP2/TP8, and other models have not been tested; NVLink has community results and is outside the release gate. +- V100 16 GB, TP2/TP8, and other models have not been tested. - This release builds SM70 only; other GPU architectures, ROCm, CPU, XPU, TPU, and GGUF are outside the build. - Greedy outputs at C2–C8 can change between runs; fresh compiler autotuning can also change outputs. -- The AWQ PLE table needs about 48 GiB of host memory by default. An existing experimental option keeps the whole table on disk instead; functional checks in 0.1.3 ran under 64 GiB and 32 GiB container limits without CPU KV offload, but speed was not measured. See [Support and known limits](docs/known-limits.md). +- The AWQ PLE table needs about 48 GiB of host memory by default. An experimental option keeps the whole table on a local SSD instead: in 0.1.4 it served under 64 GiB and 32 GiB memory limits, also with a 32 GiB CPU KV pool under 64 GiB, and greedy output matched the default path. Single-request decoding is about 23–25% slower, and a cold first token takes 1.7–2.2 times as long. See [Support and known limits](docs/known-limits.md). - PLE cascade (spilling part of the table to disk, `ple_disk_cascade=true`) is not supported yet and is off by default in the ReVolta profile; full support is planned for 0.2. - INT8 KV cannot be combined with DFlash2; the disk tier does not support INT8 yet. @@ -98,8 +98,10 @@ ReVolta focuses on dependable daily service on V100. Releases check output corre long-context retrieval, concurrent use, multimodal input, cache recovery, and speed on the reference machine. -Next: faster INT8 KV decoding, simpler NVFP4 setup, INT8 disk caching, and community -testing on NVLink systems. Longer-term work includes reducing host-memory use and +Next (0.1.5): faster all-reduce on NVLink above 16 tokens per step, a cheaper DFlash2 +state replay, smaller DFlash2 CPU and disk cache pages, and a 512K long-context +retrieval self-test. Also planned: faster INT8 KV decoding, simpler NVFP4 setup and +INT8 disk caching. Longer-term work includes reducing host-memory use and runtime dependencies. ## Getting help @@ -113,7 +115,10 @@ Reports from NVLink V100 owners are especially welcome; share measured results w [Test results](https://github.com/Leonccaa/ReVolta/issues/new?template=test_results.md) template ([`.github/ISSUE_TEMPLATE/test_results.md`](.github/ISSUE_TEMPLATE/test_results.md); agents can fill it in and open the issue through the API). Thanks to -[@jttdev](https://github.com/jttdev) for sharing their results ([#2](https://github.com/Leonccaa/ReVolta/issues/2)). Report security issues +[@jttdev](https://github.com/jttdev), [@That80-20](https://github.com/That80-20) and +[@vincentyan2023](https://github.com/vincentyan2023) for sharing their results +([#2](https://github.com/Leonccaa/ReVolta/issues/2), [#38](https://github.com/Leonccaa/ReVolta/issues/38), +[#37](https://github.com/Leonccaa/ReVolta/issues/37)). Report security issues through GitHub private vulnerability reporting. For project status and development background, read the [pinned overview (#1)](https://github.com/Leonccaa/ReVolta/issues/1). diff --git a/docs/known-limits.md b/docs/known-limits.md index 65562fe501..b80a39f5d9 100644 --- a/docs/known-limits.md +++ b/docs/known-limits.md @@ -3,6 +3,9 @@ This reference lists the tested support scope and known limitations. Check it before using the [launch tutorial](getting-started.md) or changing hardware, formats or request limits. +ReVolta is designed for 4 × V100 32 GB. NVLink, or all four GPUs under one PCIe switch +with P2P, are the recommended and tested interconnects. + The reference system uses four V100 32 GB PCIe cards behind one PCIe switch (PIX/PXB in `nvidia-smi topo -m`), with P2P enabled and a 128 GiB container memory limit. @@ -10,7 +13,7 @@ The reference system uses four V100 32 GB PCIe cards behind one PCIe switch | --- | --- | | 4 × V100 32 GB PCIe, TP4 + DCP2, P2P | Full acceptance passed on the reference configuration | | P2P disabled on the same machine | Correct output; about 30–55% of P2P throughput | -| 4 × V100-SXM2 32 GB, NVLink (NV2 mesh) | Community test by [@jttdev](https://github.com/jttdev) ([#2](https://github.com/Leonccaa/ReVolta/issues/2)): speed, retrieval to about 490K tokens, vision. Not part of our release gate | +| 4 × V100-SXM2 32 GB, NVLink | Tested by the community: [@jttdev](https://github.com/jttdev) ([#2](https://github.com/Leonccaa/ReVolta/issues/2)): speed, retrieval to about 490K tokens, vision; also [#38](https://github.com/Leonccaa/ReVolta/issues/38) and [#37](https://github.com/Leonccaa/ReVolta/issues/37) | | V100 16 GB, other GPU counts | Not yet tested | **Full acceptance on the reference setup:** default Uncensored AWQ, Standard NVFP4, @@ -18,8 +21,7 @@ and FP4-Local with their documented settings. Base AWQ has quality measurements. 80 GiB RAM has memory-stress results. Disk caching has readback results; its full release gate passed with an 8 GiB CPU pool and a 40 GiB disk budget. -V100 16 GB, TP2/TP8, and other models have not been tested. NVLink has a community -test ([#2](https://github.com/Leonccaa/ReVolta/issues/2)) but is not part of our release gate. +V100 16 GB, TP2/TP8, and other models have not been tested. This release builds SM70 only; other GPU architectures, ROCm, CPU, XPU, TPU, and GGUF are outside the build. @@ -44,14 +46,14 @@ GGUF are outside the build. and is off by default in the ReVolta profile; full support is planned for 0.2. Profiling with it enabled used about 4 GiB more GPU memory per 32 GB card, which left room for about 345K context tokens. -- **Whole PLE table on disk (experimental; first checked in 0.1.3).** Instead of pinned host memory, the +- **Whole PLE table on disk (experimental; first checked in 0.1.3, measured in 0.1.4).** Instead of pinned host memory, the whole table can stay as file-backed mappings of the checkpoint's safetensors files, read by a separate PLE process. This is a different path from cascade and is off by default. To try it, set `VLLM_PLE_CPU_OFFLOAD=1`, `VLLM_PLE_DISK_OFFLOAD=1` and `VLLM_SM70_QWEN38_HYBRID_PLE=0`, add `--safetensors-load-strategy lazy`, keep - `"ple_disk_cascade": false` in `--kernel-config` (the check also set - `"ple_disk_release_pages": true`), and run without CPU KV offload - (`--kv-transfer-config`); that combination was not tested. + `"ple_disk_cascade": false` in `--kernel-config` (the checks also set + `"ple_disk_release_pages": true`). CPU KV offload (`--kv-transfer-config`) can stay + on; 0.1.4 measured it with a 32 GiB CPU pool. - Checked on the 0.1.3 candidate with Uncensored AWQ and E4M3 KV: MTP4 served 131K-token prompts, 4 concurrent requests and images under both a 64 GiB and a 32 GiB container memory limit, and a 491K-token prompt under 64 GiB; DFlash2 @@ -62,9 +64,21 @@ GGUF are outside the build. single-request outputs diverged at a near-tie, so the two paths are not bitwise identical. If the PLE process dies, the client gets an error after about 9 s and the engine exits after 11–13 s instead of hanging. - - Not measured: decode speed and time to first token, a machine with little - physical RAM, a local SSD (the check read the checkpoint from network storage), - and the net memory saving against the default path. + - Measured on 0.1.4 with DFlash2 (recommended settings), Uncensored AWQ, E4M3 KV and + 250 W, with the table read from a local NVMe or SATA SSD. Under a 64 GiB limit + (NVMe and SATA), a 32 GiB limit (NVMe), and a 64 GiB limit with a 32 GiB CPU KV + pool (NVMe and SATA), the engine started and served without running out of memory, + and three 64-token greedy outputs matched the pinned-memory path exactly. + Single-request decoding was 101.5–104.9 tokens/s against 135.5 with the pinned + table, about 23–25% slower. A cold first token took 5.4–5.9 / 16.8–17.9 / + 65.6–69.8 s at 8K / 32K / 131K prompt tokens, against 2.61 / 9.66 / 40.40 s, so + 1.7–2.2 times as long; once the table pages for a prompt had been read, the first + token was within 2–3% of the pinned path (5–8% with the 32 GiB CPU KV pool). NVMe + and SATA, and the 32 and 64 GiB limits, differed little. Peak memory of the + serving container was about 57 GiB under the 64 GiB limit (62 GiB with the CPU KV + pool) and 30 GiB under 32 GiB. + - Not measured: a machine with little physical RAM (the limits above were set on the + container that runs the engine), and long-running stability on this path. - **FP4-Local vision.** Keep the default vision-tower TP4. Forcing data parallelism removes the tested memory arrangement. Non-empty request-level `mm_processor_kwargs` and `media_io_kwargs` are rejected with HTTP 400 by diff --git a/docs/kv-offload.md b/docs/kv-offload.md index d7607ae82b..3186b6333e 100644 --- a/docs/kv-offload.md +++ b/docs/kv-offload.md @@ -30,8 +30,11 @@ computed again. } ``` -`cpu_bytes_to_use` sets the pool size (total for all workers). With 48 GiB, the pool -holds about 2.4M tokens. `mamba_state_slots_reference_tokens` sets how many +`cpu_bytes_to_use` sets the pool size (total for all workers). With MTP4, a 48 GiB pool +holds about 2.4M tokens. With DFlash2, the drafter uses a different cache block size, so +every cache group takes pages of the full block size and a prefix needs about 50 KB per +token in the CPU pool and on disk (measured in 0.1.4: 1.84 GB for a 32K-token prefix, +6.36 GB for 131K), about 2.5 times as much as with MTP4. `mamba_state_slots_reference_tokens` sets how many recurrent-state slots each layer group gets. A smaller reference gives more slots, which helps when many short sessions share the pool. @@ -78,15 +81,21 @@ use a larger CPU pool and set `max_size_bytes` to your disk budget. - The SSD tier is a disposable cache, not model storage. - A prefix restored from disk is staged in the CPU pool first, so `cpu_bytes_to_use` must be larger than the largest prefix you expect to restore from disk (about 21 KB per - token; a 100K-token prefix needs more than 2 GiB). Otherwise the request is computed + token with MTP4 and about 50 KB with DFlash2; a 100K-token prefix needs more than + 2 GiB with MTP4 and about 5 GB with DFlash2). Otherwise the request is computed again. The quick-start 48 GiB pool holds about 2.4M tokens. - The SSD tier is **not supported with INT8 KV** yet. That combination is rejected at configuration time. -Validation scope: the SSD tier was tested with its root on a network file system. The -full release gate passed with an 8 GiB CPU pool and a 40 GiB disk budget, including a -99K-token prefix restored from disk (96,288 cached tokens, output identical). SSD -throughput was not measured. +Validation scope: the full release gate passed with the SSD tier on a network file +system (8 GiB CPU pool, 40 GiB disk budget), including a 99K-token prefix restored from +disk (96,288 cached tokens, output identical). In 0.1.4 the tier was also measured on a +local NVMe SSD with DFlash2 (7 GiB CPU pool, 64 GiB disk budget, one read and one write +thread). After an engine restart, 11 of 11 documents were read back from disk with the +same 64-token greedy output as a cold run, at about 2.2–2.4 GB/s per request; a +32K / 99K / 131K-token prefix came back in 1.47 / 2.93 / 3.77 s instead of +10.12 / 31.88 / 42.43 s when recomputed. A root holding 27.8 GB in 1,088 files did not +slow the restart. See [Coming back to a document](performance.md#coming-back-to-a-document). ## See also diff --git a/docs/performance.md b/docs/performance.md index 89aaf23287..225814448e 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -26,7 +26,26 @@ method (MTP4 and DFlash2) and prompt type is in [Performance](#performance). ### Coming back to a document -No ReVolta version is specified for this readback measurement; its request and cache conditions follow below. +Measured on 0.1.4 with DFlash2 (recommended settings), Uncensored AWQ, E4M3 KV, 250 W, +the pinned PLE table, a 7 GiB CPU pool and a disk tier on a local NVMe SSD. Each +document was first computed cold. The CPU readback followed after the GPU prefix cache +was cleared; the disk readback followed an engine restart with the page cache dropped. +Both readbacks hit the whole cacheable prefix, and the disk readbacks gave the same +64-token greedy output as the cold run. + +| Prompt tokens | Recomputed | From CPU memory | From NVMe after a restart | +| ---: | ---: | ---: | ---: | +| 32,768 | 10.12 s | 0.61 s | 1.47 s | +| 99,000 | 31.88 s | 0.77 s | 2.93 s | +| 131,072 | 42.43 s | 0.99 s | 3.77 s | + +Times are to the first token, one request at a time. Eight more 32K-token documents +gave 0.62–0.66 s from CPU memory and 1.41–1.46 s from NVMe for the first four of them. +Four 32K-token documents read from NVMe at the same time after the restart took +1.48–3.84 s each (median 2.67 s). + +An earlier measurement, with no ReVolta version recorded, under the request and cache +conditions below: A cached 32K-token document answered a follow-up in **1.28 seconds from memory**, compared with **10.06 seconds when recomputed**. Both produced identical output. @@ -39,8 +58,8 @@ compared with **10.06 seconds when recomputed**. Both produced identical output. | CPU memory + filesystem cache | 1.60 s | The request generated 64 tokens and reused 31,008 cached prompt tokens. A small GPU -pool forced eviction. The filesystem tier was tested on a network filesystem; -local SSD performance has not yet been measured. +pool forced eviction. The filesystem tier in this earlier measurement was on a network +filesystem. ### Quantization quality From 687160da27dfd05d83544a9cacb6097feb417d74 Mon Sep 17 00:00:00 2001 From: Leonccaa <166551845+Leonccaa@users.noreply.github.com> Date: Tue, 27 Oct 2026 11:56:49 -0700 Subject: [PATCH 2/2] [Doc] Keep "four or more V100 32 GB GPUs, validated on four" The positioning keeps the scope of four or more GPUs, validated on four, and adds the recommended and tested interconnects (NVLink, or all GPUs under one PCIe switch with P2P). Co-Authored-By: Claude Opus 5.5 Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com> --- README.md | 2 +- docs/known-limits.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index bc79bc6772..32fd8fb164 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ ReVolta turns a server with V100 32 GB GPUs and plenty of system memory into a dependable daily AI service. It runs Qwen3.8 Flash-Next for text, images and video behind an OpenAI-compatible API. ReVolta is built on [1Cat-vLLM](https://github.com/1CatAI/1Cat-vLLM), a V100-focused fork of vLLM, so the `vllm` command, its options and its OpenAI-compatible API work as usual. -It is built for servers with 4 × V100 32 GB. We recommend, and test, the two interconnects that give the best speed: NVLink, or all four GPUs under one PCIe switch with P2P. Every release passes the same gate on our own server before it ships: single-request greedy output identical to the previous release token for token, 4 and 8 concurrent requests, images and video, CPU cache readback, and 512K-token prompts. That server runs ReVolta every day. +It is built for four or more V100 32 GB GPUs and validated on four. We recommend, and test, the two interconnects that give the best speed: NVLink, or all GPUs under one PCIe switch with P2P. Every release passes the same gate on our own server before it ships: single-request greedy output identical to the previous release token for token, 4 and 8 concurrent requests, images and video, CPU cache readback, and 512K-token prompts. That server runs ReVolta every day. **You need:** 4 × V100 32 GB (NVLink or PCIe; NVLink or one PCIe switch with P2P recommended), at least 64 GB of system memory (128 GB recommended), Linux x86-64 with glibc 2.34 or newer, and an NVIDIA driver for CUDA 12.8. Tested hardware and limits: [Support and known limits](docs/known-limits.md). diff --git a/docs/known-limits.md b/docs/known-limits.md index b80a39f5d9..7f7ae82c2d 100644 --- a/docs/known-limits.md +++ b/docs/known-limits.md @@ -3,8 +3,8 @@ This reference lists the tested support scope and known limitations. Check it before using the [launch tutorial](getting-started.md) or changing hardware, formats or request limits. -ReVolta is designed for 4 × V100 32 GB. NVLink, or all four GPUs under one PCIe switch -with P2P, are the recommended and tested interconnects. +ReVolta is designed for four or more V100 32 GB GPUs and validated on four. NVLink, or +all GPUs under one PCIe switch with P2P, are the recommended and tested interconnects. The reference system uses four V100 32 GB PCIe cards behind one PCIe switch (PIX/PXB in `nvidia-smi topo -m`), with P2P enabled and a 128 GiB container memory limit.