feat: add native physical KV reservation APIs by shipiyouniao · Pull Request #487 · ovg-project/kvcached

shipiyouniao · 2026-09-14T11:47:09Z

This PR reserves native physical KV growth before allocation and retains each charge until physical release and unmapping are confirmed. The implementation uses flock with totals derived from per-pool records.

Accounting and calibration

Grants and instance limits use a short flock around the per-pool records. CUDA allocation, mapping and release run outside that lock. There are no stored aggregate counters.

Calibration no longer requires global idle or an active/version check. A nonblocking kernel lease coordinates only calibrators across the driver sample and publication, preventing older samples from overwriting newer ones. Grants, refunds and owner retirement continue while sampling. Publication sums the current pool charges and sets budget = max(0, current charges + sampled free memory - safety margin), retaining negative headroom until the final clamp.

Successful releases refund in byte batches below 64 MiB per releasing thread and budget connection. A physical handle of at least 64 MiB refunds directly; outer eviction, trim and destroy scopes no longer defer all refunds until completion. Failed release does not refund, and a failed accounting flush retains its backlog for retry.

The budget remains an estimate. All concurrent unmaterialized grants, queued refunds, the largest indivisible physical handles, sample/publication delay and unrelated allocations can contribute error. Five logical pages are not a global bound. Accounting failures can add further backlog; the safety margin is not a guarantee against CUDA OOM.

A pool's kernel owner lease determines whether registration can retire it. Retirement reduces both its charge and the GPU budget, creating no spendable credit and making no claim about completed driver cleanup. A later sample can recover memory actually observed free.

Workers need a trusted local Linux filesystem supporting flock and OFD byte-range locks. Equal pathnames in isolated container mounts do not coordinate. The default /dev/shm setup assumes one Unix UID. Multiple UIDs require provisioned access through a common group, setgid directory and group-write-preserving umask (for example 0007), or equivalent default ACLs. All participants must be able to read and write the ledger, use the same protocol, and restart together after a layout change. The existing ledger pathname is retained so incompatible live layouts fail instead of creating independent budgets. Do not remove or replace live coordination files. The implementation does not chmod files or bypass permission errors. GPU partitions sharing a PCI identity are not separate accounting domains in this version.

Use a consistent KVCACHED_IPC_NAME across an instance's workers. Limits apply to the sum of its pools on the GPU. A limit reduction blocks new growth without reclaiming existing pages. Failed physical cleanup retains its charge and ownership for retry; uncertain mappings remain quarantined and may require worker restart.

Engine boundary

The existing SGLang prefill guard and Jeff's admission-retry fix remain intact: cleanup preserves each request's original is_retracted, retracted_stain and retraction_count, including requeue. Prefix-lock/Mamba cleanup and the retained-chunk path remain intact. The guard is scoped to SGLang 0.5.20 single-rank, page-size-one prefill; it is not general distributed admission. The manager.world_size == 1 check does not establish TP=1. The earlier collective concern remains a code inference, not a normal-run reproduction. No #554/#578 integration is included.

Calibration and refund timing

Before the flock simplification, global activity could postpone calibration and an outer release scope deferred all refunds:

sequenceDiagram
    participant W as Worker
    participant L as Ledger
    participant C as Calibrator
    participant D as Driver
    W->>L: Begin outer mutation
    C->>L: Check global active count
    L-->>C: Skip sample while active
    loop Whole eviction or trim
        W->>D: Unmap and release handle
        W->>W: Accumulate refund
    end
    W->>L: Refund entire outer scope
    C->>L: Exclude grants and check idle
    C->>D: Sample free memory
    C->>L: Publish if version is unchanged

The current flock implementation coordinates only calibrators across sampling and publication. Ordinary accounting continues and confirmed refunds flush in bounded batches:

sequenceDiagram
    participant W as Worker
    participant L as Ledger
    participant C as Calibrator
    participant D as Driver
    C->>C: Acquire sampler-only lease
    C->>D: Sample free memory outside ledger lock
    par Normal accounting continues
        W->>L: Grant under short lock
        W->>D: Create or release physical handles
        W->>L: Flush confirmed refunds in byte batches
    and Publish sample
        C->>L: Sum current pools and update budget
    end
    C->>C: Release sampler-only lease

Validation

Native compilation, the GPU-free C++ suite, four budget-interface tests, the compiled Python snapshot export check and clang-format passed. A private source review found no actionable findings.

Historical validation and measurements at 519b6a2

The native build and GPU-free C++ suite passed for 519b6a2. Real-VMM and admission regressions passed: 85 tests, with three two-GPU skips. The isolated Python 3.11 CPU suite passed 1,247 tests with 12 SGLang-runtime skips; the classification manifest passed. Changed-file formatting, lint, spelling and Markdown checks passed. A dedicated private Codex source review found no actionable defects; runtime checks were performed separately.

Two normal SGLang 0.5.15 servers using the cached Qwen2.5-0.5B model completed 152 HTTP requests on both old and new flock. Each instance grew from 96 to 768 MiB, returned to 96 MiB after a public limit change plus cache flush, and grew again to 288 MiB. Repeated 2,048-token prompts reported 2,047 cached tokens before and after the flush/regrowth sequence. Both runs recorded zero admission misses, retractions, OOM or state-inconsistency errors.

This is normal serving validation, not a reproduction of Jeff's admission-refusal case: no refusal occurred. It covers the available 0.5.15 runtime, not 0.5.20; the latter still has the previously disclosed kernel-version mismatch.

Full private pre-commit hooks and strict MyPy checks for Python 3.10–3.13 passed. Public CPU CI passed on Python 3.10–3.13, with 1,449 passed and 14 skipped in each version; public pre-commit CI also passed. Public GPU CI also passed all 22 normal-serving cases for 519b6a2 merged into main at 04ccd1d, using SGLang 0.5.20 and vLLM 0.30.0 on two L4 GPUs (formal report). The elastic SGLang Qwen3.5-4B run logged one admission refusal returning one request to scheduling, and the case passed. This exercises rejection/requeue through normal serving, but does not correlate that retry with prefix-cache accounting or reproduce the old cached_tokens defect. Each server used TP=1; this is not TP=2 validation.

Three paired native runs compared the new flock code with 0315b5d on the same T4, alternating order, with two processes and 60 cycles per worker. KVCACHED_GPU_UTILIZATION=0.9 set the safety margin to approximately 10% of the CUDA-reported total memory. Each sampling loop attempted explicit calibration, read the ledger and then queried driver memory, with a 5 ms sleep between loops. Calibration, waiting and read time add to that interval; this was not a fixed sampling frequency. This is a constructed allocator benchmark, not normal serving or the default one-second refresh cadence.

Physical layout	New/old cycle throughput, median (range)	Release-path p50, old → new	Applied calibration attempts, old → new
512 MiB grant, 2 MiB handles	1.017× (0.989–1.051)	33.24 → 30.02 ms	624/736 → 714/766
480 MiB grant, 96 MiB handles	1.003× (0.989–1.053)	6.17 → 6.24 ms	485/603 → 602/630

Both throughput ranges cross 1.0; these runs establish no consistent speedup. The median accounting-lock acquisitions per cycle increased from 10.0 to 16.3 and 8.8 to 11.2, respectively, including sampler and observation queries. Calibration-call p50 medians were 24.67 → 26.17 ms and 1.85 → 5.32 ms; these include allocator-lock waits and skipped attempts. Release timings include unmap, physical release and accounting, not isolated refund cost. All measured cycles completed with zero admission misses and zero OOM.

Across the three paired native runs, the largest positive differences between a ledger snapshot's available credit and a subsequent driver free-memory-minus-margin reading were 470 MiB for 2 MiB handles and 864 MiB for 96 MiB handles (old flock: 2 MiB and 480 MiB, respectively). These sequential observations combine possible estimation error with concurrent changes between reads. They do not isolate true simultaneous error, establish an upper bound, or demonstrate that the margin prevents OOM. No near-OOM or multi-GPU validation was performed in this benchmark.

Earlier historical validation

The previous flock dbacefb merged with main passed 22 normal-serving GPU CI cases, all at TP=1. The earlier CAS 22-case L4 run also predates this change.

Historical five-round A/B: flock 3310135 / CAS 6d63913

On one T4, identical optimized builds and benchmark code alternated order across five paired rounds. Median paired CAS/flock throughput ratios (minimum–maximum):

Processes	GPU growth: 8 MiB	GPU growth: 40 MiB	CPU ledger cycle
1	0.988× (0.960–1.093)	0.988× (0.983–1.026)	1.401× (1.388–1.407)
2	1.077× (0.882–1.136)	1.077× (0.999–1.126)	3.733× (3.436–3.923)
4	1.011× (0.927–1.112)	0.992× (0.947–1.021)	5.389× (4.677–5.922)

All GPU-growth ranges cross 1.0. CAS improved CPU-only accounting under contention, but also replaced owner-table scans with aggregate counters; this was a whole-implementation comparison. Runs used a two-core container quota, unfixed GPU clocks, 200 growth iterations after 20 warmups, 2,000 reserve-reuse iterations and 5,000 CPU-ledger attempts per worker. Ranges are descriptive, not confidence intervals. These are allocator measurements, not model-serving throughput or multi-GPU results, and neither head includes the current simplification.

Refs #467

shipiyouniao · 2026-10-14T06:42:00Z

/ci run

github-actions · 2026-10-14T06:42:11Z

GPU CI (basic) on 0e1ec05, requested by @shipiyouniao: run

github-actions · 2026-10-14T07:13:33Z

Basic correctness · 2026-10-14 · PASS

Run · kvcached 0e1ec05 (merged into main at a7dfbfe)
2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each · lmsysorg/sglang:v0.5.20, vllm/vllm-openai:v0.30.0

All 22 cases passed.

Setup of every case

On request in a pull request (gpu-pr.yml) on 2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each. kvcached is installed from the pull request merged into main into each engine's official image; each case starts one server in it.

engine	image	versions
SGLang	`lmsysorg/sglang:v0.5.20` (`sha256:06e4f2ed21af…`)	sglang 0.5.20, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
vLLM	`vllm/vllm-openai:v0.30.0` (`sha256:8a69ffad015f…`)	vllm 0.30.0, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0

Every SGLang server: HF_HUB_OFFLINE=1 python3 -m sglang.launch_server --model-path MODEL --served-model-name e2e --host 127.0.0.1 --port 19000 --random-seed 42 --enable-cache-report, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.
Every vLLM server: HF_HUB_OFFLINE=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_SERVER_DEV_MODE=1 vllm serve MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --seed 42 --enable-prompt-tokens-details --enable-prefix-caching --async-scheduling --compilation-config.backend=eager, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.

Cases

Each kvcached case is compared with the native case of the same engine and model.

case	model	KV cache	TP	KV dtype	case flags
sglang-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
vllm-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --block-size 1024`
vllm-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`

Each case:

gen_seq: 9 prompts (6 short, and 300, 900 and 1800 words of filler text), greedy, 64 output tokens, one request at a time
gen_conc: the same 9 prompts, all at once
the prefix cache is reset
apc_0, apc_1, apc_2: 4 questions about one shared 1300-word document, greedy, 32 output tokens, one request at a time
apc_conc: the same 4 questions, all at once
the server is stopped with SIGINT to its process group (E2E_STOP_SIGNAL), SIGKILL after 90 s

check	passes when
`server_ready`	the server answers /health within the ready timeout
`<phase>_ok`	every request of the phase returns HTTP 200 with output tokens
`prefix_cache_reset`	the prefix-cache reset endpoint succeeds
`server_alive_after_load`	the server process group is still running after the load
`clean_exit_on_signal`	the server exits within 90 s of the stop signal
`no_error_signatures`	no traceback, CUDA error or patch failure in the log before the stop signal
`kvcached_active`	kvcached applied every required patch and sized the KV cache
`kvcached_inactive`	the native server's log has no [kvcached] line
`no_gpu_process_left`	no process is left on the GPUs of the run
`<phase>_tokens_equal`	gen_seq and apc_0..2 give the same greedy tokens as native
`apc_<n>_cached_tokens_equal`	every request reports the same cached_tokens as native

Elastic tests

case	GPU	server A	server B
elastic-sglang-qwen05b+sglang-qwen35_4b	1	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.8 --max-running-requests 128`	SGLang, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.85 --max-running-requests 128`
elastic-vllm-qwen05b+vllm-qwen35_4b	0	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.8 --max-num-seqs 128`	vLLM, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.9 --max-num-seqs 128`

both servers use the contiguous KV layout and a batch limit of 128; A starts first, then B, on the same GPU
every load request is a distinct 3000-word filler prompt with 1024 output tokens and ignore_eos
calibration: 4 such requests per server measure the KV memory one request holds
each server's load is sized to need 70% of the GPU memory free after calibration, at most 128 requests
phases: (1) A gets its load; (2) A's prefix cache is reset; (3) B gets its load and its prefix cache is reset; (4) both get their loads at the same time
3 greedy probe prompts of 32 tokens on each server before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`a_requests_ok, b_requests_ok`	every request of the server's own load succeeds
`a_filled_memory`	A's KV usage grows by more than half of the free memory
`a_released_memory`	after its prefix cache is reset, A keeps less than half of what it grew
`b_reused_a_memory`	B's KV usage grows past what was free at A's peak
`contended_requests_ok`	every request succeeds while both loads run
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of each server

Limit tests

case	server
limit-sglang-qwen05b	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
limit-vllm-qwen05b	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`

one kvcached server, contiguous KV layout
fill: 24 distinct 1500-word filler prompts with 1 output token, 8 at a time
kvctl limit lowers the server's limit to its idle usage plus a quarter of what the fill added
load under the lower limit: 100 distinct 150-word prompts with 16 output tokens, 8 at a time, then 4 distinct 1200-word prompts with 16 output tokens, one at a time
the limit is restored and the fill is repeated with new prompts
3 greedy probe prompts of 32 tokens before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`limit_set`	`kvctl limit` succeeds, when lowering and when restoring the limit
`limit_cache_filled`	every fill request succeeds and the KV usage grows
`limit_released_idle`	within 30 s of lowering the limit, with no requests, the KV usage is within it
`limit_requests_ok`	every request under the lower limit succeeds
`limit_held_under_load`	the KV usage stays within the lower limit under that load
`limit_raised_regrows`	after the limit is restored, the KV usage grows past the lower one
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of the server

Logs: artifact, kept 30 days: serve.log and cmd.txt per case, summary.json with every request's tokens, setup.json, run.log, GPU state at the end.

Co-authored-by: 石皮幼鸟 <2960474346@qq.com>

shipiyouniao · 2026-10-16T09:04:14Z

@RixinLiu @zou-zhiheng could you review the two candidates and the A/B results above? The implementation choice is still open. I’d especially appreciate your thoughts on the recovery/maintenance trade-off and whether #585 is likely to make budget accounting a frequent operation.

shipiyouniao · 2026-10-16T09:45:58Z

/ci run

github-actions · 2026-10-16T09:46:08Z

GPU CI (basic) on 87f6fab, requested by @shipiyouniao: run

shipiyouniao · 2026-10-16T10:10:18Z

Synchronization at the current CAS head:

Normal cross-process grant/refund accounting uses CAS. The local GpuBudget::mutex_ still protects owner records, pending accounting and local bookkeeping. mutation_mutex_ protects mutation nesting and accumulated refunds at accounting boundaries; it is not held throughout CUDA release. See the lock wrapper and mutation/refund paths.
The per-allocator mtx_ still covers CUDA allocation, mapping, unmapping and release. Prepare holds it while reserving backing, including any budget-admission wait; commit/abort take it for their own work. It is released between API calls. CUDA therefore runs outside the shared accounting flock, while allocator-local serialization remains. See allocator operations.
Shared flock remains for segment setup/teardown, slot registration/reclamation, revisioned limits and calibration setup/publication. The local budget mutex can be held while waiting for this control lock. These paths do not hold the shared data flock across CUDA allocation or rank-communication waits. See registration and limit updates.
Calibration samples device memory outside the ledger data locks while retaining an OFD lease. An ordinary grant encountering calibration now drops the ledger data mutex, waits for the lease, then retries admission. CAS first unwinds its pending/active state. The repaired short-lock candidate follows the same wait/retry rule and also drops its shared data flock. Recursive admission from the sampler is rejected to avoid self-deadlock. An outer allocator mutex can remain held. Release activity invalidates stale samples through active/version checks; actual capacity denial still rejects. See grant/wait and calibration.

The normal path still checks file identity and the calibration lease, and may enter calibration when its sample is due. This is not a lock-free allocator. Source inspection alone does not show that a remaining lock is a serving bottleneck; the PR description separates historical microbenchmarks, current validation and the completed matched A/B.

github-actions · 2026-10-16T10:17:25Z

Basic correctness · 2026-10-16 · FAIL (1 of 22 cases)

Run · kvcached 87f6fab (merged into main at 04ccd1d)
2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each · lmsysorg/sglang:v0.5.20, vllm/vllm-openai:v0.30.0

case	setup	failed checks
elastic-sglang-qwen05b+sglang-qwen35_4b	SGLang Qwen/Qwen2.5-0.5B-Instruct + SGLang Qwen/Qwen3.5-4B on GPU 1, batch limit 128	`b_requests_ok` (11/11 failed); `contended_requests_ok` (11/131 failed); `elastic-sglang-qwen35_4b-kv_c1_probe_unchanged`; `elastic-sglang-qwen35_4b-kv_c1_alive`; `elastic-sglang-qwen35_4b-kv_c1_no_error_signatures` (Traceback (most recent call last))

Setup of every case

On request in a pull request (gpu-pr.yml) on 2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each. kvcached is installed from the pull request merged into main into each engine's official image; each case starts one server in it.

engine	image	versions
SGLang	`lmsysorg/sglang:v0.5.20` (`sha256:06e4f2ed21af…`)	sglang 0.5.20, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
vLLM	`vllm/vllm-openai:v0.30.0` (`sha256:8a69ffad015f…`)	vllm 0.30.0, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0

Every SGLang server: HF_HUB_OFFLINE=1 python3 -m sglang.launch_server --model-path MODEL --served-model-name e2e --host 127.0.0.1 --port 19000 --random-seed 42 --enable-cache-report, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.
Every vLLM server: HF_HUB_OFFLINE=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_SERVER_DEV_MODE=1 vllm serve MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --seed 42 --enable-prompt-tokens-details --enable-prefix-caching --async-scheduling --compilation-config.backend=eager, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.

Cases

Each kvcached case is compared with the native case of the same engine and model.

case	model	KV cache	TP	KV dtype	case flags
sglang-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
vllm-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --block-size 1024`
vllm-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`

Each case:

gen_seq: 9 prompts (6 short, and 300, 900 and 1800 words of filler text), greedy, 64 output tokens, one request at a time
gen_conc: the same 9 prompts, all at once
the prefix cache is reset
apc_0, apc_1, apc_2: 4 questions about one shared 1300-word document, greedy, 32 output tokens, one request at a time
apc_conc: the same 4 questions, all at once
the server is stopped with SIGINT to its process group (E2E_STOP_SIGNAL), SIGKILL after 90 s

check	passes when
`server_ready`	the server answers /health within the ready timeout
`<phase>_ok`	every request of the phase returns HTTP 200 with output tokens
`prefix_cache_reset`	the prefix-cache reset endpoint succeeds
`server_alive_after_load`	the server process group is still running after the load
`clean_exit_on_signal`	the server exits within 90 s of the stop signal
`no_error_signatures`	no traceback, CUDA error or patch failure in the log before the stop signal
`kvcached_active`	kvcached applied every required patch and sized the KV cache
`kvcached_inactive`	the native server's log has no [kvcached] line
`no_gpu_process_left`	no process is left on the GPUs of the run
`<phase>_tokens_equal`	in gen_seq and in each of apc_0..2, more than half of the prompts give the same greedy tokens as native
`apc_<n>_cached_tokens_equal`	every request reports the same cached_tokens as native

Elastic tests

case	GPU	server A	server B
elastic-sglang-qwen05b+sglang-qwen35_4b	1	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.8 --max-running-requests 128`	SGLang, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.85 --max-running-requests 128`
elastic-vllm-qwen05b+vllm-qwen35_4b	0	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.8 --max-num-seqs 128`	vLLM, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.9 --max-num-seqs 128`

both servers use the contiguous KV layout and a batch limit of 128; A starts first, then B, on the same GPU
every load request is a distinct 3000-word filler prompt with 1024 output tokens and ignore_eos
calibration: 4 such requests per server measure the KV memory one request holds
each server's load is sized to need 70% of the GPU memory free after calibration, at most 128 requests
phases: (1) A gets its load; (2) A's prefix cache is reset; (3) B gets its load and its prefix cache is reset; (4) both get their loads at the same time
3 greedy probe prompts of 32 tokens on each server before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`a_requests_ok, b_requests_ok`	every request of the server's own load succeeds
`a_filled_memory`	A's KV usage grows by more than half of the free memory
`a_released_memory`	after its prefix cache is reset, A keeps less than half of what it grew
`b_reused_a_memory`	B's KV usage grows past what was free at A's peak
`contended_requests_ok`	every request succeeds while both loads run
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of each server

Limit tests

case	server
limit-sglang-qwen05b	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
limit-vllm-qwen05b	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`

one kvcached server, contiguous KV layout
fill: 24 distinct 1500-word filler prompts with 1 output token, 8 at a time
kvctl limit lowers the server's limit to its idle usage plus a quarter of what the fill added
load under the lower limit: 100 distinct 150-word prompts with 16 output tokens, 8 at a time, then 4 distinct 1200-word prompts with 16 output tokens, one at a time
the limit is restored and the fill is repeated with new prompts
3 greedy probe prompts of 32 tokens before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`limit_set`	`kvctl limit` succeeds, when lowering and when restoring the limit
`limit_cache_filled`	every fill request succeeds and the KV usage grows
`limit_released_idle`	within 30 s of lowering the limit, with no requests, the KV usage is within it
`limit_requests_ok`	every request under the lower limit succeeds
`limit_held_under_load`	the KV usage stays within the lower limit under that load
`limit_raised_regrows`	after the limit is restored, the KV usage grows past the lower one
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of the server

Logs: artifact, kept 30 days: serve.log and cmd.txt per case, summary.json with every request's tokens, setup.json, run.log, GPU state at the end.

shipiyouniao · 2026-10-16T11:54:35Z

/ci run

github-actions · 2026-10-17T05:35:10Z

Basic correctness · 2026-10-17 · PASS

Run · kvcached dbacefb (merged into main at 04ccd1d)
2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each · lmsysorg/sglang:v0.5.20, vllm/vllm-openai:v0.30.0

All 22 cases passed.

Setup of every case

On request in a pull request (gpu-pr.yml) on 2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each. kvcached is installed from the pull request merged into main into each engine's official image; each case starts one server in it.

engine	image	versions
SGLang	`lmsysorg/sglang:v0.5.20` (`sha256:06e4f2ed21af…`)	sglang 0.5.20, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
vLLM	`vllm/vllm-openai:v0.30.0` (`sha256:8a69ffad015f…`)	vllm 0.30.0, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0

Every SGLang server: HF_HUB_OFFLINE=1 python3 -m sglang.launch_server --model-path MODEL --served-model-name e2e --host 127.0.0.1 --port 19000 --random-seed 42 --enable-cache-report, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.
Every vLLM server: HF_HUB_OFFLINE=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_SERVER_DEV_MODE=1 vllm serve MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --seed 42 --enable-prompt-tokens-details --enable-prefix-caching --async-scheduling --compilation-config.backend=eager, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.

Cases

Each kvcached case is compared with the native case of the same engine and model.

case	model	KV cache	TP	KV dtype	case flags
sglang-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
vllm-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --block-size 1024`
vllm-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`

Each case:

gen_seq: 9 prompts (6 short, and 300, 900 and 1800 words of filler text), greedy, 64 output tokens, one request at a time
gen_conc: the same 9 prompts, all at once
the prefix cache is reset
apc_0, apc_1, apc_2: 4 questions about one shared 1300-word document, greedy, 32 output tokens, one request at a time
apc_conc: the same 4 questions, all at once
the server is stopped with SIGINT to its process group (E2E_STOP_SIGNAL), SIGKILL after 90 s

check	passes when
`server_ready`	the server answers /health within the ready timeout
`<phase>_ok`	every request of the phase returns HTTP 200 with output tokens
`prefix_cache_reset`	the prefix-cache reset endpoint succeeds
`server_alive_after_load`	the server process group is still running after the load
`clean_exit_on_signal`	the server exits within 90 s of the stop signal
`no_error_signatures`	no traceback, CUDA error or patch failure in the log before the stop signal
`kvcached_active`	kvcached applied every required patch and sized the KV cache
`kvcached_inactive`	the native server's log has no [kvcached] line
`no_gpu_process_left`	no process is left on the GPUs of the run
`<phase>_tokens_equal`	in gen_seq and in each of apc_0..2, more than half of the prompts give the same greedy tokens as native
`apc_<n>_cached_tokens_equal`	every request reports the same cached_tokens as native

Elastic tests

case	GPU	server A	server B
elastic-sglang-qwen05b+sglang-qwen35_4b	1	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.8 --max-running-requests 128`	SGLang, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.85 --max-running-requests 128`
elastic-vllm-qwen05b+vllm-qwen35_4b	0	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.8 --max-num-seqs 128`	vLLM, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.9 --max-num-seqs 128`

both servers use the contiguous KV layout and a batch limit of 128; A starts first, then B, on the same GPU
every load request is a distinct 3000-word filler prompt with 1024 output tokens and ignore_eos
calibration: 4 such requests per server measure the KV memory one request holds
each server's load is sized to need 70% of the GPU memory free after calibration, at most 128 requests
phases: (1) A gets its load; (2) A's prefix cache is reset; (3) B gets its load and its prefix cache is reset; (4) both get their loads at the same time
3 greedy probe prompts of 32 tokens on each server before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`a_requests_ok, b_requests_ok`	every request of the server's own load succeeds
`a_filled_memory`	A's KV usage grows by more than half of the free memory
`a_released_memory`	after its prefix cache is reset, A keeps less than half of what it grew
`b_reused_a_memory`	B's KV usage grows past what was free at A's peak
`contended_requests_ok`	every request succeeds while both loads run
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of each server

Limit tests

case	server
limit-sglang-qwen05b	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
limit-vllm-qwen05b	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`

one kvcached server, contiguous KV layout
fill: 24 distinct 1500-word filler prompts with 1 output token, 8 at a time
kvctl limit lowers the server's limit to its idle usage plus a quarter of what the fill added
load under the lower limit: 100 distinct 150-word prompts with 16 output tokens, 8 at a time, then 4 distinct 1200-word prompts with 16 output tokens, one at a time
the limit is restored and the fill is repeated with new prompts
3 greedy probe prompts of 32 tokens before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`limit_set`	`kvctl limit` succeeds, when lowering and when restoring the limit
`limit_cache_filled`	every fill request succeeds and the KV usage grows
`limit_released_idle`	within 30 s of lowering the limit, with no requests, the KV usage is within it
`limit_requests_ok`	every request under the lower limit succeeds
`limit_held_under_load`	the KV usage stays within the lower limit under that load
`limit_raised_regrows`	after the limit is restored, the KV usage grows past the lower one
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of the server

Logs: artifact, kept 30 days: serve.log and cmd.txt per case, summary.json with every request's tokens, setup.json, run.log, GPU state at the end.

shipiyouniao · 2026-10-18T09:21:05Z

With this simplification, I prefer flock for the smaller accounting and recovery surface. It keeps per-pool records and short critical sections, allows calibration during activity, and returns confirmed releases in bounded byte batches. The T4 comparisons do not establish a consistent throughput advantage; the trade-off includes higher refund-lock activity and sampling error, documented in the PR. CAS is unchanged.

shipiyouniao · 2026-10-18T09:21:42Z

/ci run

github-actions · 2026-10-18T09:21:52Z

GPU CI (basic) on 519b6a2, requested by @shipiyouniao: run

github-actions · 2026-10-18T09:53:07Z

Basic correctness · 2026-10-18 · PASS

Run · kvcached 519b6a2 (merged into main at 04ccd1d)
2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each · lmsysorg/sglang:v0.5.20, vllm/vllm-openai:v0.30.0

All 22 cases passed.

Setup of every case

On request in a pull request (gpu-pr.yml) on 2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each. kvcached is installed from the pull request merged into main into each engine's official image; each case starts one server in it.

engine	image	versions
SGLang	`lmsysorg/sglang:v0.5.20` (`sha256:06e4f2ed21af…`)	sglang 0.5.20, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
vLLM	`vllm/vllm-openai:v0.30.0` (`sha256:8a69ffad015f…`)	vllm 0.30.0, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0

Every SGLang server: HF_HUB_OFFLINE=1 python3 -m sglang.launch_server --model-path MODEL --served-model-name e2e --host 127.0.0.1 --port 19000 --random-seed 42 --enable-cache-report, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.
Every vLLM server: HF_HUB_OFFLINE=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_SERVER_DEV_MODE=1 vllm serve MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --seed 42 --enable-prompt-tokens-details --enable-prefix-caching --async-scheduling --compilation-config.backend=eager, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.

Cases

Each kvcached case is compared with the native case of the same engine and model.

case	model	KV cache	TP	KV dtype	case flags
sglang-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
vllm-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --block-size 1024`
vllm-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`

Each case:

gen_seq: 9 prompts (6 short, and 300, 900 and 1800 words of filler text), greedy, 64 output tokens, one request at a time
gen_conc: the same 9 prompts, all at once
the prefix cache is reset
apc_0, apc_1, apc_2: 4 questions about one shared 1300-word document, greedy, 32 output tokens, one request at a time
apc_conc: the same 4 questions, all at once
the server is stopped with SIGINT to its process group (E2E_STOP_SIGNAL), SIGKILL after 90 s

check	passes when
`server_ready`	the server answers /health within the ready timeout
`<phase>_ok`	every request of the phase returns HTTP 200 with output tokens
`prefix_cache_reset`	the prefix-cache reset endpoint succeeds
`server_alive_after_load`	the server process group is still running after the load
`clean_exit_on_signal`	the server exits within 90 s of the stop signal
`no_error_signatures`	no traceback, CUDA error or patch failure in the log before the stop signal
`kvcached_active`	kvcached applied every required patch and sized the KV cache
`kvcached_inactive`	the native server's log has no [kvcached] line
`no_gpu_process_left`	no process is left on the GPUs of the run
`<phase>_tokens_equal`	in gen_seq and in each of apc_0..2, more than half of the prompts give the same greedy tokens as native
`apc_<n>_cached_tokens_equal`	every request reports the same cached_tokens as native

Elastic tests

case	GPU	server A	server B
elastic-sglang-qwen05b+sglang-qwen35_4b	1	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.8 --max-running-requests 128`	SGLang, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.85 --max-running-requests 128`
elastic-vllm-qwen05b+vllm-qwen35_4b	0	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.8 --max-num-seqs 128`	vLLM, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.9 --max-num-seqs 128`

both servers use the contiguous KV layout and a batch limit of 128; A starts first, then B, on the same GPU
every load request is a distinct 3000-word filler prompt with 1024 output tokens and ignore_eos
calibration: 4 such requests per server measure the KV memory one request holds
each server's load is sized to need 70% of the GPU memory free after calibration, at most 128 requests
phases: (1) A gets its load; (2) A's prefix cache is reset; (3) B gets its load and its prefix cache is reset; (4) both get their loads at the same time
3 greedy probe prompts of 32 tokens on each server before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`a_requests_ok, b_requests_ok`	every request of the server's own load succeeds
`a_filled_memory`	A's KV usage grows by more than half of the free memory
`a_released_memory`	after its prefix cache is reset, A keeps less than half of what it grew
`b_reused_a_memory`	B's KV usage grows past what was free at A's peak
`contended_requests_ok`	every request succeeds while both loads run
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of each server

Limit tests

case	server
limit-sglang-qwen05b	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
limit-vllm-qwen05b	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`

one kvcached server, contiguous KV layout
fill: 24 distinct 1500-word filler prompts with 1 output token, 8 at a time
kvctl limit lowers the server's limit to its idle usage plus a quarter of what the fill added
load under the lower limit: 100 distinct 150-word prompts with 16 output tokens, 8 at a time, then 4 distinct 1200-word prompts with 16 output tokens, one at a time
the limit is restored and the fill is repeated with new prompts
3 greedy probe prompts of 32 tokens before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`limit_set`	`kvctl limit` succeeds, when lowering and when restoring the limit
`limit_cache_filled`	every fill request succeeds and the KV usage grows
`limit_released_idle`	within 30 s of lowering the limit, with no requests, the KV usage is within it
`limit_requests_ok`	every request under the lower limit succeeds
`limit_held_under_load`	the KV usage stays within the lower limit under that load
`limit_raised_regrows`	after the limit is restored, the KV usage grows past the lower one
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of the server

Logs: artifact, kept 30 days: serve.log and cmd.txt per case, summary.json with every request's tokens, setup.json, run.log, GPU state at the end.

One document for sharing a GPU between kvcached instances: the per-GPU ledger this PR adds, and the planned per-instance controls, watermarks, reclaim, CPU tier and coordination API.

…dles misses ovg-project#554 returns a prefill batch that the pools cannot back to the waiting queue, so the separate admission guard is no longer needed; both wrapped Scheduler._get_new_batch_prefill_raw. Also drop the A/B benchmark script.

RixinLiu · 2026-10-18T21:09:46Z

/ci run

github-actions · 2026-10-18T21:09:55Z

GPU CI (basic) on cae31f2, requested by @RixinLiu: run

github-actions · 2026-10-18T21:41:25Z

Basic correctness · 2026-10-18 · PASS

Run · kvcached cae31f2 (merged into main at a2af415)
2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each · lmsysorg/sglang:v0.5.20, vllm/vllm-openai:v0.30.0

All 22 cases passed.

Setup of every case

On request in a pull request (gpu-pr.yml) on 2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each. kvcached is installed from the pull request merged into main into each engine's official image; each case starts one server in it.

engine	image	versions
SGLang	`lmsysorg/sglang:v0.5.20` (`sha256:06e4f2ed21af…`)	sglang 0.5.20, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
vLLM	`vllm/vllm-openai:v0.30.0` (`sha256:8a69ffad015f…`)	vllm 0.30.0, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0

Every SGLang server: HF_HUB_OFFLINE=1 python3 -m sglang.launch_server --model-path MODEL --served-model-name e2e --host 127.0.0.1 --port 19000 --random-seed 42 --enable-cache-report, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.
Every vLLM server: HF_HUB_OFFLINE=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_SERVER_DEV_MODE=1 vllm serve MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --seed 42 --enable-prompt-tokens-details --enable-prefix-caching --async-scheduling --compilation-config.backend=eager, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.

Cases

Each kvcached case is compared with the native case of the same engine and model.

case	model	KV cache	TP	KV dtype	case flags
sglang-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
vllm-qwen05b-kv_c1	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-kv_c0	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen35_4b-kv_c1	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-kv_c0	`Qwen/Qwen3.5-4B` @ 851bf6e	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_4b-native	`Qwen/Qwen3.5-4B` @ 851bf6e	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --block-size 1024`
vllm-gemma4e2b-kv_c1	`google/gemma-4-E2B-it` @ 3e22461	kvcached, contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-kv_c0	`google/gemma-4-E2B-it` @ 3e22461	kvcached, non-contiguous	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4e2b-native	`google/gemma-4-E2B-it` @ 3e22461	native	1	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`

Each case:

gen_seq: 9 prompts (6 short, and 300, 900 and 1800 words of filler text), greedy, 64 output tokens, one request at a time
gen_conc: the same 9 prompts, all at once
the prefix cache is reset
apc_0, apc_1, apc_2: 4 questions about one shared 1300-word document, greedy, 32 output tokens, one request at a time
apc_conc: the same 4 questions, all at once
the server is stopped with SIGINT to its process group (E2E_STOP_SIGNAL), SIGKILL after 90 s

check	passes when
`server_ready`	the server answers /health within the ready timeout
`<phase>_ok`	every request of the phase returns HTTP 200 with output tokens
`prefix_cache_reset`	the prefix-cache reset endpoint succeeds
`server_alive_after_load`	the server process group is still running after the load
`clean_exit_on_signal`	the server exits within 90 s of the stop signal
`no_error_signatures`	no traceback, CUDA error or patch failure in the log before the stop signal
`kvcached_active`	kvcached applied every required patch and sized the KV cache
`kvcached_inactive`	the native server's log has no [kvcached] line
`no_gpu_process_left`	no process is left on the GPUs of the run
`<phase>_tokens_equal`	in gen_seq and in each of apc_0..2, more than half of the prompts give the same greedy tokens as native
`apc_<n>_cached_tokens_equal`	every request reports the same cached_tokens as native

Elastic tests

case	GPU	server A	server B
elastic-sglang-qwen05b+sglang-qwen35_4b	1	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.8 --max-running-requests 128`	SGLang, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.85 --max-running-requests 128`
elastic-vllm-qwen05b+vllm-qwen35_4b	0	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.8 --max-num-seqs 128`	vLLM, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.9 --max-num-seqs 128`

both servers use the contiguous KV layout and a batch limit of 128; A starts first, then B, on the same GPU
every load request is a distinct 3000-word filler prompt with 1024 output tokens and ignore_eos
calibration: 4 such requests per server measure the KV memory one request holds
each server's load is sized to need 70% of the GPU memory free after calibration, at most 128 requests
phases: (1) A gets its load; (2) A's prefix cache is reset; (3) B gets its load and its prefix cache is reset; (4) both get their loads at the same time
3 greedy probe prompts of 32 tokens on each server before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`a_requests_ok, b_requests_ok`	every request of the server's own load succeeds
`a_filled_memory`	A's KV usage grows by more than half of the free memory
`a_released_memory`	after its prefix cache is reset, A keeps less than half of what it grew
`b_reused_a_memory`	B's KV usage grows past what was free at A's peak
`contended_requests_ok`	every request succeeds while both loads run
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of each server

Limit tests

case	server
limit-sglang-qwen05b	SGLang, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
limit-vllm-qwen05b	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`

one kvcached server, contiguous KV layout
fill: 24 distinct 1500-word filler prompts with 1 output token, 8 at a time
kvctl limit lowers the server's limit to its idle usage plus a quarter of what the fill added
load under the lower limit: 100 distinct 150-word prompts with 16 output tokens, 8 at a time, then 4 distinct 1200-word prompts with 16 output tokens, one at a time
the limit is restored and the fill is repeated with new prompts
3 greedy probe prompts of 32 tokens before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`limit_set`	`kvctl limit` succeeds, when lowering and when restoring the limit
`limit_cache_filled`	every fill request succeeds and the KV usage grows
`limit_released_idle`	within 30 s of lowering the limit, with no requests, the KV usage is within it
`limit_requests_ok`	every request under the lower limit succeeds
`limit_held_under_load`	the KV usage stays within the lower limit under that load
`limit_raised_regrows`	after the limit is restored, the KV usage grows past the lower one
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of the server

Logs: artifact, kept 30 days: serve.log and cmd.txt per case, summary.json with every request's tokens, setup.json, run.log, GPU state at the end.

RixinLiu · 2026-10-18T21:42:53Z

/ci run weekly

github-actions · 2026-10-18T21:43:01Z

GPU CI (weekly) on cae31f2, requested by @RixinLiu: run

github-actions · 2026-10-18T22:44:33Z

Weekly correctness · 2026-10-18 · PASS

Run · kvcached cae31f2 (merged into main at a2af415)
2x NVIDIA L4 (23034 MiB, driver 580.178.04) · vllm/vllm-openai:v0.30.0, lmsysorg/sglang:v0.5.20

All 26 cases passed.

Setup of every case

On request in a pull request (gpu-pr.yml) on 2x NVIDIA L4 (23034 MiB, driver 580.178.04). kvcached is installed from the pull request merged into main into each engine's official image; each case starts one server in it.

engine	image	versions
vLLM	`vllm/vllm-openai:v0.30.0` (`sha256:8a69ffad015f…`)	vllm 0.30.0, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
SGLang	`lmsysorg/sglang:v0.5.20` (`sha256:06e4f2ed21af…`)	sglang 0.5.20, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0

Every vLLM server: HF_HUB_OFFLINE=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_SERVER_DEV_MODE=1 vllm serve MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --seed 42 --enable-prompt-tokens-details --enable-prefix-caching --async-scheduling --compilation-config.backend=eager, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.
Every SGLang server: HF_HUB_OFFLINE=1 python3 -m sglang.launch_server --model-path MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --random-seed 42 --enable-cache-report, followed by the case's flags.
- kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
- native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.

Cases

Each kvcached case is compared with the native case of the same engine and model.

case	model	KV cache	TP	KV dtype	case flags
vllm-qwen05b-kv_c1-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-kv_c0-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-dsv2lite-kv_c1-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code`
vllm-dsv2lite-kv_c0-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code`
vllm-dsv2lite-native-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code`
vllm-gemma4_12b-kv_c1-tp2	`google/gemma-4-12B-it` @ 707f0a3	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4_12b-kv_c0-tp2	`google/gemma-4-12B-it` @ 707f0a3	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-gemma4_12b-native-tp2	`google/gemma-4-12B-it` @ 707f0a3	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_9b-kv_c1-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_9b-kv_c0-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9`
vllm-qwen35_9b-native-tp2	`Qwen/Qwen3.5-9B` @ c202236	native	2	model dtype	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --block-size 1024`
sglang-qwen05b-kv_c1-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-kv_c0-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native-tp2	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-dsv2lite-kv_c1-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code`
sglang-dsv2lite-kv_c0-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code`
sglang-dsv2lite-native-tp2	`deepseek-ai/DeepSeek-V2-Lite-Chat` @ 8586474	native	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code`
sglang-qwen35_9b-kv_c1-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_9b-kv_c0-tp2	`Qwen/Qwen3.5-9B` @ c202236	kvcached, non-contiguous	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
sglang-qwen35_9b-native-tp2	`Qwen/Qwen3.5-9B` @ c202236	native	2	model dtype	`--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85`
vllm-qwen05b-kv_c1-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	fp8	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
vllm-qwen05b-native-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	fp8	`--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8`
sglang-qwen05b-kv_c1-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	kvcached, contiguous	1	fp8_e4m3	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`
sglang-qwen05b-native-fp8kv	`Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576	native	1	fp8_e4m3	`--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8`

Each case:

gen_seq: 9 prompts (6 short, and 300, 900 and 1800 words of filler text), greedy, 64 output tokens, one request at a time
gen_conc: the same 9 prompts, all at once
the prefix cache is reset
apc_0, apc_1, apc_2: 4 questions about one shared 1300-word document, greedy, 32 output tokens, one request at a time
apc_conc: the same 4 questions, all at once
the server is stopped with SIGINT to its process group (E2E_STOP_SIGNAL), SIGKILL after 90 s

check	passes when
`server_ready`	the server answers /health within the ready timeout
`<phase>_ok`	every request of the phase returns HTTP 200 with output tokens
`prefix_cache_reset`	the prefix-cache reset endpoint succeeds
`server_alive_after_load`	the server process group is still running after the load
`clean_exit_on_signal`	the server exits within 90 s of the stop signal
`no_error_signatures`	no traceback, CUDA error or patch failure in the log before the stop signal
`kvcached_active`	kvcached applied every required patch and sized the KV cache
`kvcached_inactive`	the native server's log has no [kvcached] line
`no_gpu_process_left`	no process is left on the GPUs of the run
`<phase>_tokens_equal`	in gen_seq and in each of apc_0..2, more than half of the prompts give the same greedy tokens as native
`apc_<n>_cached_tokens_equal`	every request reports the same cached_tokens as native

Elastic tests

case	GPU	server A	server B
elastic-vllm-qwen05b+sglang-qwen35_4b	0	vLLM, `Qwen/Qwen2.5-0.5B-Instruct` @ 7ae5576, kvcached, contiguous, `--dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.8 --max-num-seqs 128`	SGLang, `Qwen/Qwen3.5-4B` @ 851bf6e, kvcached, contiguous, `--dtype bfloat16 --context-length 8192 --mem-fraction-static 0.85 --max-running-requests 128`

both servers use the contiguous KV layout and a batch limit of 128; A starts first, then B, on the same GPU
every load request is a distinct 3000-word filler prompt with 1024 output tokens and ignore_eos
calibration: 4 such requests per server measure the KV memory one request holds
each server's load is sized to need 70% of the GPU memory free after calibration, at most 128 requests
phases: (1) A gets its load; (2) A's prefix cache is reset; (3) B gets its load and its prefix cache is reset; (4) both get their loads at the same time
3 greedy probe prompts of 32 tokens on each server before and after

check	passes when
`<server>_ready`	the server answers /health within the ready timeout
`a_requests_ok, b_requests_ok`	every request of the server's own load succeeds
`a_filled_memory`	A's KV usage grows by more than half of the free memory
`a_released_memory`	after its prefix cache is reset, A keeps less than half of what it grew
`b_reused_a_memory`	B's KV usage grows past what was free at A's peak
`contended_requests_ok`	every request succeeds while both loads run
`<server>_probe_unchanged`	the probes give the same tokens before and after
`<server>_alive`	the server is still running at the end
`<server>_<shutdown check>`	the shutdown and cleanup checks of each server

Logs: artifact, kept 30 days: serve.log and cmd.txt per case, summary.json with every request's tokens, setup.json, run.log, GPU state at the end.

Under TP each SGLang rank has a kvcached pool of its own, and its available_size() is the physical headroom of that rank's GPU. The scheduler reads that count to admit (PrefillAdder.rem_total_tokens), to evict the token shortfall before every allocation and inside the decode gate (evict_from_tree_cache), and to evict the Mamba shortfall in alloc_req_slots. With a second process on one GPU only, the ranks read different counts, admit different batches or evict different amounts, and the forward desyncs, which the agreed miss alone does not prevent since both allocations can succeed locally. Once per scheduling round (get_next_batch_to_run, or the prefill and decode entries the PD and multiplexing loops call directly) every rank reads its managers' counts and one MIN all-reduce over the attention TP and CP CPU groups makes the smallest the budget of all. The count each manager reports is the forecast ovg-project#487 clamps to the per-GPU ledger, so the agreed budget is the shortest rank's ledger forecast. The elastic allocators serve the agreed count plus a tally of the round's own allocations and frees, never the live count, which re-reads the GPU's physical free pages after every allocation. The reduce runs whatever the local counts, once per outermost round, before the MAX reduces of the round's attempts, so every rank issues the same collectives in the same order. Without a queue group no round opens and every read stays local, and a pipeline does not open a round, as the miss patches keep SGLang's own behaviour there. Refs ovg-project#547 Signed-off-by: Rishabh Sinha <rsinha17@terpmail.umd.edu>

This was referenced Sep 14, 2026

perf: coalesce physical growth retries and signal released capacity #488

Draft

fix: coordinate distributed physical KV mapping outcomes #455

Open

shipiyouniao force-pushed the feature/physical-kv-reservations branch from a885b66 to fe09783 Compare September 14, 2026 16:54

This was referenced Sep 22, 2026

fix(vllm): defer physical unmap until async batches complete #464

Merged

feat: allocate physical KV pages in request-sized batches #489

Draft

shipiyouniao force-pushed the feature/physical-kv-reservations branch from fe09783 to 83fbfc7 Compare September 24, 2026 11:26

shipiyouniao mentioned this pull request Sep 24, 2026

vLLM 0.28 compatibility #490

Closed

11 of 20 tasks

feat: reserve physical KV pages under a short fair admission turn

0e1ec05

shipiyouniao force-pushed the feature/physical-kv-reservations branch from 53925d6 to 0e1ec05 Compare October 8, 2026 20:25

shipiyouniao marked this pull request as ready for review October 8, 2026 20:30

shipiyouniao mentioned this pull request Oct 14, 2026

feat(sglang): back native DeepSeek-V4 C128 request state #569

Draft

RixinLiu self-requested a review October 14, 2026 16:48

This was referenced Oct 15, 2026

test(allocator): cover local growth ledger regressions #574

Closed

Feature request: support DeepSeek-V4 KV cache pooling in vLLM and SGLang #369

Open

zou-zhiheng and others added 3 commits October 15, 2026 13:34

fix(allocator): restore device after physical capacity checks

4819660

Co-authored-by: 石皮幼鸟 <2960474346@qq.com>

test: keep device regression entry mergeable with main

17f81b6

Co-authored-by: 石皮幼鸟 <2960474346@qq.com>

feat(allocator): account physical growth with short shared budget locks

9b9be3c

shipiyouniao force-pushed the feature/physical-kv-reservations branch 2 times, most recently from 6462d60 to 87f6fab Compare October 16, 2026 09:44

shipiyouniao mentioned this pull request Oct 16, 2026

System Design Discussion for Scheduling Policy + CPU Offloading #585

Open

shipiyouniao force-pushed the feature/physical-kv-reservations branch from 87f6fab to 6d63913 Compare October 16, 2026 11:54

jeff3071 reviewed Oct 17, 2026

View reviewed changes

Comment thread kvcached/integration/sglang/admission.py Resolved

fix: preserve retract history on SGLang admission retry

0315b5d

shipiyouniao mentioned this pull request Oct 18, 2026

fix: query physical KV capacity on the pool's GPU #589

Open

refactor(allocator): simplify flock calibration and refund batches

1cc0c19

shipiyouniao force-pushed the feature/physical-kv-reservations branch from 519b6a2 to 1cc0c19 Compare October 18, 2026 16:24

RixinLiu added 3 commits October 18, 2026 16:49

docs: describe the coordinator: ledger, controls, reclaim, CPU tier

07d3841

One document for sharing a GPU between kvcached instances: the per-GPU ledger this PR adds, and the planned per-instance controls, watermarks, reclaim, CPU tier and coordination API.

docs(coordinator): allow reclaim to preempt and leave reads unlocked

cae31f2

JasperChan929 mentioned this pull request Oct 18, 2026

PageAllocator background capacity queries use the wrong GPU #588

Open

RixinLiu added 2 commits October 19, 2026 02:47

docs: remove the shared segment lifecycle testing notes

675e383

Merge main into feature/physical-kv-reservations

d40961c

shipiyouniao mentioned this pull request Oct 19, 2026

feat(allocator): add independent atomic budget observations #590

Open

RixinLiu merged commit d100fa7 into ovg-project:main Oct 19, 2026
9 checks passed

This was referenced Oct 19, 2026

Discussion: unify kvctl and API limit semantics with the per-GPU ledger #591

Closed

feat(allocator): claim exact free block IDs atomically #565

Draft

rishabhsinha17 mentioned this pull request Oct 20, 2026

ci: make the weekly elastic case starve one rank of a TP=2 server #577

Merged

RixinLiu mentioned this pull request Oct 28, 2026

feat(control): add min, priority and headroom instance controls #605

Merged

konstde00 mentioned this pull request Oct 29, 2026

fix(ipc): distinct default instance names for co-located engines #607

Open

Conversation

shipiyouniao commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Accounting and calibration

Engine boundary

Calibration and refund timing

Validation

Earlier historical validation

Uh oh!

shipiyouniao commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Basic correctness · 2026-10-14 · PASS

Cases

Elastic tests

Limit tests

Uh oh!

shipiyouniao commented Oct 16, 2026

Uh oh!

shipiyouniao commented Oct 16, 2026

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

shipiyouniao commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 16, 2026

Basic correctness · 2026-10-16 · FAIL (1 of 22 cases)

Cases

Elastic tests

Limit tests

Uh oh!

shipiyouniao commented Oct 16, 2026

Uh oh!

github-actions Bot commented Oct 17, 2026

Basic correctness · 2026-10-17 · PASS

Cases

Elastic tests

Limit tests

Uh oh!

Uh oh!

shipiyouniao commented Oct 18, 2026

Uh oh!

shipiyouniao commented Oct 18, 2026

Uh oh!

github-actions Bot commented Oct 18, 2026

Uh oh!

github-actions Bot commented Oct 18, 2026

Basic correctness · 2026-10-18 · PASS

Cases

Elastic tests

Limit tests

Uh oh!

RixinLiu commented Oct 18, 2026

Uh oh!

github-actions Bot commented Oct 18, 2026

Uh oh!

github-actions Bot commented Oct 18, 2026

Basic correctness · 2026-10-18 · PASS

Cases

Elastic tests

Limit tests

Uh oh!

RixinLiu commented Oct 18, 2026

Uh oh!

github-actions Bot commented Oct 18, 2026

Uh oh!

github-actions Bot commented Oct 18, 2026

Weekly correctness · 2026-10-18 · PASS

Cases

Elastic tests

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

shipiyouniao commented Sep 14, 2026 •

edited

Loading

shipiyouniao commented Oct 16, 2026 •

edited

Loading