feat(vllm): support CPU KV offloading on vLLM 0.29 and 0.30 by RixinLiu · Pull Request #598 · ovg-project/kvcached

RixinLiu · 2026-10-25T21:05:20Z

Problem

From vLLM 0.29 the KV cache that OffloadingConnector copies is allocated through kvcached, and kvcached may unmap a page once all of its blocks are free. In three cases a block is freed, kvcached can unmap its page, and only then does the connector copy the block:

A request that finishes in the step that writes its KV. The store is in that step's build_connector_meta, but the worker submits it only at the start of the next step. The request frees its blocks in between, so the page can be unmapped before the store is submitted.
What a finished request still has to store. When the request frees its blocks, no job exists yet: the connector creates it in the next build_connector_meta (request_finished: "so _build_store_jobs can create store jobs for the last block(s) on the next schedule step"), so the page can be unmapped before that build.
Sliding-window blocks. They are freed while the request runs, and the connector may store them in any later build, as long as vLLM has not reallocated them, so the page can be unmapped at any point before that.

vLLM waits for such a copy only when it reallocates the block (jobs_to_flush), and an unmap is not a reallocation. main refuses offloading at startup (#351):

kvcached does not support vLLM CPU KV offloading (OffloadingConnector, --kv-offloading-size); remove --kv-offloading-size or disable kvcached

With that check removed, on vLLM 0.30 with --no-async-scheduling, Qwen2.5-0.5B, KVCACHED_MAX_CACHED_TOKENS=0 and KVCACHED_MAX_RESERVED_PAGES=0, 2 of 60 requests served from the CPU cache returned '5' and '0' where native vLLM returns ' River'. Both generate one token, so they finish in the step that writes their KV.

Change

Each transfer job pins its pages in the kvcached manager from the build that returns it until every worker has reported it, and a page emptied by a free waits for one more build. This is driven from the connector's public build_connector_meta, update_connector_output and has_pending_push_work; vLLM is unchanged.
Pages of sliding-window blocks (and of Mamba groups outside "align" mode) stay pinned from allocation until their request's last build. The pins only keep pages mapped; vLLM still reuses the blocks. Mamba "align" states are stored in the pass that hands them off, so the one-build wait covers full + linear attention models.
Offloading is accepted on vLLM 0.29 and 0.30 with Model Runner V2, TP=1, PP=1 and OffloadingConnector as the only connector, and refused at startup otherwise. Before 0.29 nothing changes (Error when vllm-v0.16.0 uses --kv-offloading-size #267).

#269 and #361 add offloading for vLLM 0.16's cross-layer allocation, which 0.29 replaced.

Validation

New tests/test_offload_copy_pins.py (24 tests, fail on main). The CPU suite goes from 1537 to 1563 passed.
A100-SXM4-40GB, vllm/vllm-openai:v0.30.0 and v0.29.0, kvcached 8d026c4 with this change (the branch was then rebased onto main with fix(controller): preserve latest activity after out-of-order requests #596, which touches only the controller), KVCACHED_MAX_CACHED_TOKENS=0, KVCACHED_MAX_RESERVED_PAGES=0. Each run sends the same prompts (4.5k-7.5k tokens, 1-24 output tokens) twice with a GPU prefix-cache reset in between, so the second pass loads its prefix from the CPU. Outputs are compared by greedy tokens and top-5 logprobs against native vLLM with the same flags. The block-size and KVCACHED_CONTIGUOUS_LAYOUT=false runs below used this branch (2ae013c).

Case	main	this change
`--kv-offloading-size` on 0.29 / 0.30	refused at startup	accepted
Qwen2.5-0.5B, 0.30, sync, second pass (prefix loaded from the CPU)	2 / 60 wrong (`'5'`, `'0'`)*	identical to native, 60 / 60
Qwen2.5-0.5B, 0.30 async; 0.29 sync and async	refused	identical to native, 60 / 60

* main with the startup check from #351 removed, so that offloading runs.

With this change, sync and async scheduling:

Model (KV groups)	vLLM	CPU hit per request	second pass
Qwen3.5-4B (8 full, 24 Gated DeltaNet)	0.30, 0.29	5120-7168 tokens	identical to native with `--block-size 1024` (0.30 sync, either KV layout)†; equal to the first pass in the other runs (16 prompts, 14 GB CPU cache)
DeepSeek-V2-Lite (MLA)	0.30	5008-6736 tokens	identical to native
Gemma-4-E2B (full + sliding window)	0.30	4512-5888 tokens	identical to native with `KVCACHED_CONTIGUOUS_LAYOUT=false`; with the default layout 19 of 40 prompts hit 16 tokens more than native, and 6 of them change a near-tie token‡
gpt-oss-20b (full + 128-token window)	0.30	4752-6320 tokens	differs from the first pass by at most 0.70 (sync) / 0.75 (async) in top-5 logprobs; native's two passes differ by up to 1.38

† kvcached selects 1024-token blocks for this model (#522); with native's default of 528 even the first pass differs from kvcached at 4 of 16 prompts.

‡ For the default contiguous layout kvcached requires a block-outermost KV layout (BLNHC), for which vLLM groups this model in 16-token blocks instead of native's 32 and 16, so its hits are multiples of 16 instead of 32 tokens. Native vLLM 0.30 cannot run that grouping: with VLLM_KV_CACHE_LAYOUT=BLNHC it fails at startup with a KeyError on layer 15, Gemma-4's first KV-shared layer.

With KVCACHED_CONTIGUOUS_LAYOUT=false on 0.30, Qwen2.5-0.5B (sync, 60 prompts) and Gemma-4-E2B (sync and async, 40 prompts) match native in both passes, including per-request hits and top-5 logprobs.

On 0.29 gpt-oss gets no CPU hits even natively, so only its stores ran. A counter of jobs that read freed blocks stayed at 0 for gpt-oss with and without the sliding-window pins, also with 4096-token offload chunks: kvcached reallocates freed window blocks right away, and the connector then drops their ids. The window pins follow the connector's contract rather than a reproduced failure.

From vLLM 0.29 OffloadingConnector allocates its KV through kvcached, but it copies blocks after vLLM has freed them. The store that a request's last step needs is submitted at the start of the next step, after the request finished and its blocks went back; whatever a finished request still has to store is built in the next build_connector_meta; and a sliding-window group keeps freed block ids and may store them any number of builds later. vLLM waits for such a copy only when it reallocates the block, so kvcached could unmap the page first and the copy would read whatever was mapped there next. kvcached therefore refused --kv-offloading-size at startup. With that refusal removed, on vLLM 0.30 (Model Runner V2, --no-async-scheduling), Qwen2.5-0.5B on an A100 with KVCACHED_MAX_CACHED_TOKENS=0 and KVCACHED_MAX_RESERVED_PAGES=0, two of 60 requests served from the CPU cache returned '5' and '0' where native vLLM returns ' River'. Both are requests that generate one token, so they finish in the step that writes their KV. Each transfer job now pins its pages in the kvcached manager from the build that returns it until every worker has reported it, and a page left empty by a free waits for one more build. Pages of sliding-window blocks (and of Mamba groups outside "align" mode) stay pinned from allocation until their request's last build; the pins keep pages mapped, and vLLM still reuses the blocks. Mamba "align" states are stored in the pass that hands them off, so the one-build wait covers them. All of this is driven from the connector's public build, completion and pending-work methods; vLLM is unchanged. Offloading is accepted on vLLM 0.29 and 0.30 with Model Runner V2, TP=1 and PP=1, and refused at startup otherwise.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

RixinLiu · 2026-10-26T01:44:36Z

/ci run

github-actions · 2026-10-26T01:44:47Z

GPU CI (basic) on 2ae013c, requested by @RixinLiu: run

github-actions · 2026-10-26T02:16:45Z

Basic correctness · 2026-10-26 · PASS

Run · kvcached 2ae013c (merged into main at 3170a0a)
2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each · lmsysorg/sglang:v0.5.20, vllm/vllm-openai:v0.30.0

All 22 cases passed.

Setup of every case

On request in a pull request (gpu-pr.yml) on 2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each. kvcached is installed from the pull request merged into main into each engine's official image; each case starts one server in it.

engine	image	versions
SGLang	`lmsysorg/sglang:v0.5.20` (`sha256:06e4f2ed21af…`)	sglang 0.5.20, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
vLLM	`vllm/vllm-openai:v0.30.0` (`sha256:8a69ffad015f…`)	vllm 0.30.0, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0

Every SGLang server: HF_HUB_OFFLINE=1 python3 -m sglang.launch_server --model-path MODEL --served-model-name e2e --host 127.0.0.1 --port 19000 --random-seed 42 --enable-cache-report, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.
Every vLLM server: HF_HUB_OFFLINE=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_SERVER_DEV_MODE=1 vllm serve MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --seed 42 --enable-prompt-tokens-details --enable-prefix-caching --async-scheduling --compilation-config.backend=eager, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.

Cases

Each kvcached case is compared with the native case of the same engine and model.

case	model	KV cache	TP	KV dtype	case flags
sglang-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
vllm-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --block-size 1024`
vllm-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`

Each case:

gen_seq: 9 prompts (6 short, and 300, 900 and 1800 words of filler text), greedy, 64 output tokens, one request at a time
gen_conc: the same 9 prompts, all at once
the prefix cache is reset
apc_0, apc_1, apc_2: 4 questions about one shared 1300-word document, greedy, 32 output tokens, one request at a time
apc_conc: the same 4 questions, all at once
the server is stopped with SIGINT to its process group (E2E_STOP_SIGNAL), SIGKILL after 90 s

check	passes when
`server_ready`	the server answers /health within the ready timeout
`<phase>_ok`	every request of the phase returns HTTP 200 with output tokens
`prefix_cache_reset`	the prefix-cache reset endpoint succeeds
`server_alive_after_load`	the server process group is still running after the load
`clean_exit_on_signal`	the server exits within 90 s of the stop signal
`no_error_signatures`	no traceback, CUDA error or patch failure in the log before the stop signal
`kvcached_active`	kvcached applied every required patch and sized the KV cache
`kvcached_inactive`	the native server's log has no [kvcached] line
`no_gpu_process_left`	no process is left on the GPUs of the run
`<phase>_tokens_equal`	in gen_seq and in each of apc_0..2, more than half of the prompts give the same greedy tokens as native
`apc_<n>_cached_tokens_equal`	every request reports the same cached_tokens as native

Elastic tests

case	GPU	server A	server B
elastic-sglang-qwen05b+sglang-qwen35_4b	1	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.8 --max-running-requests 128`	SGLang, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.85 --max-running-requests 128`
elastic-vllm-qwen05b+vllm-qwen35_4b	0	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.8 --max-num-seqs 128`	vLLM, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.9 --max-num-seqs 128`

both servers use the contiguous KV layout and a batch limit of 128; A starts first, then B, on the same GPU
every load request is a distinct 3000-word filler prompt with 1024 output tokens and ignore_eos
calibration: 4 such requests per server measure the KV memory one request holds
each server's load is sized to need 70% of the GPU memory free after calibration, at most 128 requests
phases: (1) A gets its load; (2) A's prefix cache is reset; (3) B gets its load and its prefix cache is reset; (4) both get their loads at the same time
3 greedy probe prompts of 32 tokens on each server before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`a_requests_ok, b_requests_ok`	every request of the server's own load succeeds
`a_filled_memory`	A's KV usage grows by more than half of the free memory
`a_released_memory`	after its prefix cache is reset, A keeps less than half of what it grew
`b_reused_a_memory`	B's KV usage grows past what was free at A's peak
`contended_requests_ok`	every request succeeds while both loads run
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of each server

Limit tests

case	server
limit-sglang-qwen05b	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
limit-vllm-qwen05b	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`

one kvcached server, contiguous KV layout
fill: 24 distinct 1500-word filler prompts with 1 output token, 8 at a time
kvctl limit lowers the server's limit to its idle usage plus a quarter of what the fill added
load under the lower limit: 100 distinct 150-word prompts with 16 output tokens, 8 at a time, then 4 distinct 1200-word prompts with 16 output tokens, one at a time
the limit is restored and the fill is repeated with new prompts
3 greedy probe prompts of 32 tokens before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`limit_set`	`kvctl limit` succeeds, when lowering and when restoring the limit
`limit_cache_filled`	every fill request succeeds and the KV usage grows
`limit_released_idle`	within 30 s of lowering the limit, with no requests, the KV usage is within it
`limit_requests_ok`	every request under the lower limit succeeds
`limit_held_under_load`	the KV usage stays within the lower limit under that load
`limit_raised_regrows`	after the limit is restored, the KV usage grows past the lower one
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of the server

Logs: artifact, kept 30 days: serve.log and cmd.txt per case, summary.json with every request's tokens, setup.json, run.log, GPU state at the end.

RixinLiu · 2026-10-26T02:48:32Z

/ci run weekly

github-actions · 2026-10-26T02:48:42Z

GPU CI (weekly) on 2ae013c, requested by @RixinLiu: run

github-actions · 2026-10-26T03:50:59Z

Weekly correctness · 2026-10-26 · PASS

Run · kvcached 2ae013c (merged into main at 3170a0a)
2x NVIDIA L4 (23034 MiB, driver 580.178.04) · vllm/vllm-openai:v0.30.0, lmsysorg/sglang:v0.5.20

All 26 cases passed.

These cases pass, though some prompts' greedy tokens differ from native: fewer than half of each phase's prompts, which is most likely numerical. Greedy decoding flips between near-tied tokens, and the engines also do this between their own runs.

case	check	prompts that differ
vllm-qwen35_9b-kv_c1-tp2	`gen_seq_tokens_equal`	2 of 9 prompts differ: prompt 6: {'index': 8, 'a': 6165, 'b': 17326}; prompt 7: {'index': 27, 'a': 9705, 'b': 4618}
vllm-qwen35_9b-kv_c1-tp2	`apc_0_tokens_equal`	1 of 4 prompts differ: prompt 2: {'index': 16, 'a': 10597, 'b': 6165}
vllm-qwen35_9b-kv_c1-tp2	`apc_1_tokens_equal`	1 of 4 prompts differ: prompt 2: {'index': 16, 'a': 10597, 'b': 6165}
vllm-qwen35_9b-kv_c1-tp2	`apc_2_tokens_equal`	1 of 4 prompts differ: prompt 2: {'index': 16, 'a': 10597, 'b': 6165}

Setup of every case

On request in a pull request (gpu-pr.yml) on 2x NVIDIA L4 (23034 MiB, driver 580.178.04). kvcached is installed from the pull request merged into main into each engine's official image; each case starts one server in it.

engine	image	versions
vLLM	`vllm/vllm-openai:v0.30.0` (`sha256:8a69ffad015f…`)	vllm 0.30.0, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
SGLang	`lmsysorg/sglang:v0.5.20` (`sha256:06e4f2ed21af…`)	sglang 0.5.20, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0

Every vLLM server: HF_HUB_OFFLINE=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_SERVER_DEV_MODE=1 vllm serve MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --seed 42 --enable-prompt-tokens-details --enable-prefix-caching --async-scheduling --compilation-config.backend=eager, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.
Every SGLang server: HF_HUB_OFFLINE=1 python3 -m sglang.launch_server --model-path MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --random-seed 42 --enable-cache-report, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.

Cases

Each kvcached case is compared with the native case of the same engine and model.

case	model	KV cache	TP	KV dtype	case flags
vllm-qwen05b-kv_c1-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-kv_c0-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-dsv2lite-kv_c1-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code`
vllm-dsv2lite-kv_c0-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code`
vllm-dsv2lite-native-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code`
vllm-gemma4_12b-kv_c1-tp2	`google/gemma-4-12B-it` @ 707f0a3	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4_12b-kv_c0-tp2	`google/gemma-4-12B-it` @ 707f0a3	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4_12b-native-tp2	`google/gemma-4-12B-it` @ 707f0a3	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_9b-kv_c1-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_9b-kv_c0-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_9b-native-tp2	`Qwen/Qwen3.5-9B` @ c202236	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --block-size 1024`
sglang-qwen05b-kv_c1-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-kv_c0-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-dsv2lite-kv_c1-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code`
sglang-dsv2lite-kv_c0-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code`
sglang-dsv2lite-native-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	native	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code`
sglang-qwen35_9b-kv_c1-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_9b-kv_c0-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_9b-native-tp2	`Qwen/Qwen3.5-9B` @ c202236	native	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
vllm-qwen05b-kv_c1-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	fp8	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	fp8	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
sglang-qwen05b-kv_c1-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	fp8_e4m3	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	fp8_e4m3	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`

Each case:

gen_seq: 9 prompts (6 short, and 300, 900 and 1800 words of filler text), greedy, 64 output tokens, one request at a time
gen_conc: the same 9 prompts, all at once
the prefix cache is reset
apc_0, apc_1, apc_2: 4 questions about one shared 1300-word document, greedy, 32 output tokens, one request at a time
apc_conc: the same 4 questions, all at once
the server is stopped with SIGINT to its process group (E2E_STOP_SIGNAL), SIGKILL after 90 s

check	passes when
`server_ready`	the server answers /health within the ready timeout
`<phase>_ok`	every request of the phase returns HTTP 200 with output tokens
`prefix_cache_reset`	the prefix-cache reset endpoint succeeds
`server_alive_after_load`	the server process group is still running after the load
`clean_exit_on_signal`	the server exits within 90 s of the stop signal
`no_error_signatures`	no traceback, CUDA error or patch failure in the log before the stop signal
`kvcached_active`	kvcached applied every required patch and sized the KV cache
`kvcached_inactive`	the native server's log has no [kvcached] line
`no_gpu_process_left`	no process is left on the GPUs of the run
`<phase>_tokens_equal`	in gen_seq and in each of apc_0..2, more than half of the prompts give the same greedy tokens as native
`apc_<n>_cached_tokens_equal`	every request reports the same cached_tokens as native

Elastic tests

case	GPU	server A	server B
elastic-vllm-qwen05b+sglang-qwen35_4b	0	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.8 --max-num-seqs 128`	SGLang, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.85 --max-running-requests 128`

both servers use the contiguous KV layout and a batch limit of 128; A starts first, then B, on the same GPU
every load request is a distinct 3000-word filler prompt with 1024 output tokens and ignore_eos
calibration: 4 such requests per server measure the KV memory one request holds
each server's load is sized to need 70% of the GPU memory free after calibration, at most 128 requests
phases: (1) A gets its load; (2) A's prefix cache is reset; (3) B gets its load and its prefix cache is reset; (4) both get their loads at the same time
3 greedy probe prompts of 32 tokens on each server before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`a_requests_ok, b_requests_ok`	every request of the server's own load succeeds
`a_filled_memory`	A's KV usage grows by more than half of the free memory
`a_released_memory`	after its prefix cache is reset, A keeps less than half of what it grew
`b_reused_a_memory`	B's KV usage grows past what was free at A's peak
`contended_requests_ok`	every request succeeds while both loads run
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of each server

Logs: artifact, kept 30 days: serve.log and cmd.txt per case, summary.json with every request's tokens, setup.json, run.log, GPU state at the end.

shipiyouniao

[P2] Drop obsolete window holds before resumed admission

Can we release a preempted request's window holds when the connector discards its old block list, before allocating the resumed blocks? Currently the holds are only dropped after successful resumption or finish.

With offloading and sliding-window groups, lowering the memory limit below the held pages and then resetting the prefix cache with reset_running_requests=True can leave those pages pinned and retired. If the waiting request needs more blocks than the remaining unretired capacity, it cannot resume while shrink is pending, so it never reaches the code that drops its holds. Resize retries cannot complete while those held pages exceed the target.

The native connector already clears the old block IDs in get_num_new_matched_tokens, before admission. Could we mirror that lifetime boundary while retaining in-flight transfer-job pins, and cover this shrink/preemption interaction?

kvcached/kvcached/integration/vllm/offload_pins.py

Lines 153 to 174 in 2ae013c

    
                           self.manager.pin_copy_job(key, block_ids) 
        
                           self.held.setdefault(req_id, []).append(key) 
        
                   # Their last stores are in this build's metadata, pinned above. 
        
                   for req_id in scheduler_output.finished_req_ids or (): 
        
                       self._drop(req_id) 
        
               def _drop(self, req_id: str) -> None: 
        
                   for key in self.held.pop(req_id, ()): 
        
                       self.manager.unpin_copy_job(key) 
        
           def install_offload_copy_pins(engine_core: Any, manager: Any) -> bool: 
        
               """Wrap the scheduler's OffloadingConnector, if any. Returns True if wrapped.""" 
        
               scheduler: Any = getattr(engine_core, "scheduler", None) 
        
               connector = getattr(scheduler, "connector", None) 
        
               if connector is None or type(connector).__name__ != OFFLOADING_CONNECTOR: 
        
                   return False 
        
               try: 
        
                   from vllm.distributed.kv_transfer.kv_connector.utils import yield_req_data 
        
                   from vllm.distributed.kv_transfer.kv_connector.v1.offloading.common import ( 
        
                       OffloadingConnectorMetadata, 
        
                       OffloadingWorkerMetadata,

kvcached/kvcached/kv_cache_manager.py

Lines 1455 to 1476 in 2ae013c

    
           def available_size(self) -> int: 
        
               avail_blocks = self.num_avail_blocks + len(self.reserved_blocks) 
        
               if self.in_shrink: 
        
                   # alloc() does not reuse pages retired for a pending shrink. 
        
                   for page_id in self._retiring_page_ids(): 
        
                       page = self.avail_pages.get(page_id) 
        
                       if page is not None: 
        
                           avail_blocks -= page.num_free_blocks() 
        
               # Also surfaces a fatal background-preallocation failure during shrink. 
        
               try: 
        
                   virtual_free_pages = self.page_allocator.get_num_free_pages() 
        
               except Exception as e: 
        
                   # This is the capacity check on the alloc path: it raises before 
        
                   # _alloc()'s own alloc_page() classification is reached, so the 
        
                   # verdict must be recorded here or the phase never leaves 
        
                   # DEGRADED (the #478 review's T4 repro). 
        
                   self._record_native_fatal("available_size", e) 
        
                   raise 
        
               if self.in_shrink: 
        
                   blocks_from_free_pages = 0 
        
               else: 
        
                   physical_free_pages = (

https://github.com/vllm-project/vllm/blob/v0.30.0/vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py#L1041-L1050

shipiyouniao · 2026-10-27T05:13:33Z

/ci run weekly

github-actions · 2026-10-27T05:13:43Z

GPU CI (weekly) on 62fffb9, requested by @shipiyouniao: run

github-actions · 2026-10-27T06:14:58Z

Weekly correctness · 2026-10-27 · PASS

Run · kvcached 62fffb9 (merged into main at 3170a0a)
2x NVIDIA L4 (23034 MiB, driver 580.178.04) · vllm/vllm-openai:v0.30.0, lmsysorg/sglang:v0.5.20

All 26 cases passed.

These cases pass, though some prompts' greedy tokens differ from native: fewer than half of each phase's prompts, which is most likely numerical. Greedy decoding flips between near-tied tokens, and the engines also do this between their own runs.

case	check	prompts that differ
vllm-qwen35_9b-kv_c1-tp2	`gen_seq_tokens_equal`	2 of 9 prompts differ: prompt 6: {'index': 8, 'a': 6165, 'b': 17326}; prompt 7: {'index': 27, 'a': 9705, 'b': 4618}
vllm-qwen35_9b-kv_c1-tp2	`apc_0_tokens_equal`	1 of 4 prompts differ: prompt 2: {'index': 16, 'a': 10597, 'b': 6165}
vllm-qwen35_9b-kv_c1-tp2	`apc_1_tokens_equal`	1 of 4 prompts differ: prompt 2: {'index': 16, 'a': 10597, 'b': 6165}
vllm-qwen35_9b-kv_c1-tp2	`apc_2_tokens_equal`	1 of 4 prompts differ: prompt 2: {'index': 16, 'a': 10597, 'b': 6165}

Setup of every case

On request in a pull request (gpu-pr.yml) on 2x NVIDIA L4 (23034 MiB, driver 580.178.04). kvcached is installed from the pull request merged into main into each engine's official image; each case starts one server in it.

engine	image	versions
vLLM	`vllm/vllm-openai:v0.30.0` (`sha256:8a69ffad015f…`)	vllm 0.30.0, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
SGLang	`lmsysorg/sglang:v0.5.20` (`sha256:06e4f2ed21af…`)	sglang 0.5.20, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0

Every vLLM server: HF_HUB_OFFLINE=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_SERVER_DEV_MODE=1 vllm serve MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --seed 42 --enable-prompt-tokens-details --enable-prefix-caching --async-scheduling --compilation-config.backend=eager, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.
Every SGLang server: HF_HUB_OFFLINE=1 python3 -m sglang.launch_server --model-path MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --random-seed 42 --enable-cache-report, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.

Cases

Each kvcached case is compared with the native case of the same engine and model.

case	model	KV cache	TP	KV dtype	case flags
vllm-qwen05b-kv_c1-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-kv_c0-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-dsv2lite-kv_c1-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code`
vllm-dsv2lite-kv_c0-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code`
vllm-dsv2lite-native-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code`
vllm-gemma4_12b-kv_c1-tp2	`google/gemma-4-12B-it` @ 707f0a3	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4_12b-kv_c0-tp2	`google/gemma-4-12B-it` @ 707f0a3	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4_12b-native-tp2	`google/gemma-4-12B-it` @ 707f0a3	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_9b-kv_c1-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_9b-kv_c0-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_9b-native-tp2	`Qwen/Qwen3.5-9B` @ c202236	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --block-size 1024`
sglang-qwen05b-kv_c1-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-kv_c0-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-dsv2lite-kv_c1-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code`
sglang-dsv2lite-kv_c0-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code`
sglang-dsv2lite-native-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	native	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code`
sglang-qwen35_9b-kv_c1-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_9b-kv_c0-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_9b-native-tp2	`Qwen/Qwen3.5-9B` @ c202236	native	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
vllm-qwen05b-kv_c1-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	fp8	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	fp8	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
sglang-qwen05b-kv_c1-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	fp8_e4m3	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	fp8_e4m3	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`

Each case:

gen_seq: 9 prompts (6 short, and 300, 900 and 1800 words of filler text), greedy, 64 output tokens, one request at a time
gen_conc: the same 9 prompts, all at once
the prefix cache is reset
apc_0, apc_1, apc_2: 4 questions about one shared 1300-word document, greedy, 32 output tokens, one request at a time
apc_conc: the same 4 questions, all at once
the server is stopped with SIGINT to its process group (E2E_STOP_SIGNAL), SIGKILL after 90 s

check	passes when
`server_ready`	the server answers /health within the ready timeout
`<phase>_ok`	every request of the phase returns HTTP 200 with output tokens
`prefix_cache_reset`	the prefix-cache reset endpoint succeeds
`server_alive_after_load`	the server process group is still running after the load
`clean_exit_on_signal`	the server exits within 90 s of the stop signal
`no_error_signatures`	no traceback, CUDA error or patch failure in the log before the stop signal
`kvcached_active`	kvcached applied every required patch and sized the KV cache
`kvcached_inactive`	the native server's log has no [kvcached] line
`no_gpu_process_left`	no process is left on the GPUs of the run
`<phase>_tokens_equal`	in gen_seq and in each of apc_0..2, more than half of the prompts give the same greedy tokens as native
`apc_<n>_cached_tokens_equal`	every request reports the same cached_tokens as native

Elastic tests

case	GPU	server A	server B
elastic-vllm-qwen05b+sglang-qwen35_4b	0	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.8 --max-num-seqs 128`	SGLang, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.85 --max-running-requests 128`

both servers use the contiguous KV layout and a batch limit of 128; A starts first, then B, on the same GPU
every load request is a distinct 3000-word filler prompt with 1024 output tokens and ignore_eos
calibration: 4 such requests per server measure the KV memory one request holds
each server's load is sized to need 70% of the GPU memory free after calibration, at most 128 requests
phases: (1) A gets its load; (2) A's prefix cache is reset; (3) B gets its load and its prefix cache is reset; (4) both get their loads at the same time
3 greedy probe prompts of 32 tokens on each server before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`a_requests_ok, b_requests_ok`	every request of the server's own load succeeds
`a_filled_memory`	A's KV usage grows by more than half of the free memory
`a_released_memory`	after its prefix cache is reset, A keeps less than half of what it grew
`b_reused_a_memory`	B's KV usage grows past what was free at A's peak
`contended_requests_ok`	every request succeeds while both loads run
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of each server

Logs: artifact, kept 30 days: serve.log and cmd.txt per case, summary.json with every request's tokens, setup.json, run.log, GPU state at the end.

shipiyouniao

The obsolete window holds are released when native matching drops the old block lists, while in-flight transfer pins remain protected. Final-head CPU/static checks, weekly CI, and native-versus-patched vLLM 0.30 offload/reload validation passed.

Copilot AI balanced review requested due to automatic review settings October 25, 2026 21:05

Copilot AI reviewed Oct 25, 2026

RixinLiu requested a review from shipiyouniao October 25, 2026 21:12

shipiyouniao mentioned this pull request Oct 26, 2026

feat(vllm): support single-node TP/PP CPU KV offloading #602

Open

shipiyouniao reviewed Oct 26, 2026

View reviewed changes

fix(vllm): release obsolete window holds before admission

62fffb9

shipiyouniao approved these changes Oct 27, 2026

View reviewed changes

shipiyouniao merged commit 775795b into main Oct 27, 2026
19 checks passed

shipiyouniao mentioned this pull request Oct 29, 2026

System Design Discussion for Scheduling Policy + CPU Offloading #585

Open

	self.manager.pin_copy_job(key, block_ids)
	self.held.setdefault(req_id, []).append(key)
	# Their last stores are in this build's metadata, pinned above.
	for req_id in scheduler_output.finished_req_ids or ():
	self._drop(req_id)

	def _drop(self, req_id: str) -> None:
	for key in self.held.pop(req_id, ()):
	self.manager.unpin_copy_job(key)


	def install_offload_copy_pins(engine_core: Any, manager: Any) -> bool:
	"""Wrap the scheduler's OffloadingConnector, if any. Returns True if wrapped."""
	scheduler: Any = getattr(engine_core, "scheduler", None)
	connector = getattr(scheduler, "connector", None)
	if connector is None or type(connector).__name__ != OFFLOADING_CONNECTOR:
	return False
	try:
	from vllm.distributed.kv_transfer.kv_connector.utils import yield_req_data
	from vllm.distributed.kv_transfer.kv_connector.v1.offloading.common import (
	OffloadingConnectorMetadata,
	OffloadingWorkerMetadata,

	def available_size(self) -> int:
	avail_blocks = self.num_avail_blocks + len(self.reserved_blocks)
	if self.in_shrink:
	# alloc() does not reuse pages retired for a pending shrink.
	for page_id in self._retiring_page_ids():
	page = self.avail_pages.get(page_id)
	if page is not None:
	avail_blocks -= page.num_free_blocks()
	# Also surfaces a fatal background-preallocation failure during shrink.
	try:
	virtual_free_pages = self.page_allocator.get_num_free_pages()
	except Exception as e:
	# This is the capacity check on the alloc path: it raises before
	# _alloc()'s own alloc_page() classification is reached, so the
	# verdict must be recorded here or the phase never leaves
	# DEGRADED (the #478 review's T4 repro).
	self._record_native_fatal("available_size", e)
	raise
	if self.in_shrink:
	blocks_from_free_pages = 0
	else:
	physical_free_pages = (

Conversation

RixinLiu commented Oct 25, 2026

Problem

Change

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

RixinLiu commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Basic correctness · 2026-10-26 · PASS

Cases

Elastic tests

Limit tests

Uh oh!

RixinLiu commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

github-actions Bot commented Oct 26, 2026

Weekly correctness · 2026-10-26 · PASS

Cases

Elastic tests

Uh oh!

shipiyouniao left a comment

Choose a reason for hiding this comment

Uh oh!

shipiyouniao commented Oct 27, 2026

Uh oh!

github-actions Bot commented Oct 27, 2026

Uh oh!

github-actions Bot commented Oct 27, 2026

Weekly correctness · 2026-10-27 · PASS

Cases

Elastic tests

Uh oh!

shipiyouniao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants