Skip to content

fix: release the prefix cache when the KV memory limit is lowered - #552

Merged
RixinLiu merged 1 commit into
ovg-project:mainfrom
Jeffwan:jiaxin/release-prefix-cache
Oct 12, 2026
Merged

RixinLiu merged 1 commit into
ovg-project:mainfrom
Jeffwan:jiaxin/release-prefix-cache

Conversation

@Jeffwan

@Jeffwan Jeffwan commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Lowering a kvcached limit did not release prefix-cache pages: the shrink stayed deferred while idle and under traffic, and a kvctl limit was only applied at the next allocation.

When a resize is deferred, kvcached now asks the prefix cache to evict whole pages until the instance fits, and applies kvctl limits while the engine is idle (vLLM EngineCore wakeup, SGLang scheduler idle hook). Pages of running requests are never revoked. SGLang with TP/PP > 1 is unchanged.

A10, Qwen3-4B, MiB mapped after kvctl limit:

limit main, idle 10s fix, idle 10s main, after traffic fix, after traffic
vLLM 0.30 630 2304 576 2304 576
SGLang 0.5.20 756 2592 288 2160 720

Copilot AI balanced review requested due to automatic review settings October 8, 2026 23:53
@Jeffwan
Jeffwan force-pushed the jiaxin/release-prefix-cache branch from d61241f to 5b7aa88 Compare October 8, 2026 23:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Valid zero and preexisting resize targets can remain unapplied.

2 open findings
What changed in this PR

Enables deferred KV-limit reductions to reclaim prefix-cache pages during traffic and idle periods.

Changes:

  • Adds shrink reclaimers and retirement-aware allocation.
  • Integrates idle limit handling with vLLM and SGLang.
  • Adds CPU coverage and updates memory-control documentation.
File Description
kvcached/​kv_cache_manager.py Coordinates deferred shrink reclamation.
kvcached/​integration/​vllm/​patches.py Adds prefix eviction and idle limit watcher.
kvcached/​integration/​vllm/​page_eviction.py Limits page-aware victim selection by page count.
kvcached/​integration/​sglang/​patches.py Adds SGLang prefix reclaimers and idle processing.
kvcached/​integration/​sglang/​autopatch.py Registers the scheduler idle patch.
tests/​test_prefix_cache_shrink.py Tests shrinking, traffic, idle, and parallel behavior.
tests/​manifests/​cpu.txt Includes the new CPU test suite.
examples/​09_prefix_caching/​README.md Documents memory-limit precedence.
examples/​02_memory_control/​README.md Documents shrink behavior and exceptions.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread kvcached/integration/vllm/patches.py Resolved
Comment thread kvcached/kv_cache_manager.py Resolved
@Jeffwan
Jeffwan force-pushed the jiaxin/release-prefix-cache branch from 2a9e1a7 to e6297cc Compare October 9, 2026 00:01
@cui36

cui36 commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Thanks for contributing, @Jeffwan! We will take a closer look.

@shipiyouniao

shipiyouniao commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Could you add real-GPU vLLM TP and PP validation before merging? This changes when cached pages are reclaimed, so the single-GPU results do not establish that release remains correctly ordered across workers.

TP=2/PP=1 and TP=1/PP=2 would cover the two paths. Please lower the limit while batches are queued and prefix-cache hits are active, finish or cancel requests, then check that all ranks release only after the worker barrier and that mapped usage converges while idle. Please exercise both kvctl and set_instance_memory_limit().

A delayed or failed rank during release should not lead to a false applied result or silently inconsistent mappings. Please include the vLLM version, topology, scheduling mode, request results, and any CUDA errors or worker restarts.

@shipiyouniao shipiyouniao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found three shrink-path cases that need fixing, detailed inline. I reproduced them in CPU control-flow tests using the PR's fake page allocator and engine/cache doubles; these are not GPU results. The existing 16 tests passed, while the added cases produced four failing parameterizations and one async-only skip.

Comment thread kvcached/integration/vllm/patches.py Resolved
Comment thread kvcached/integration/sglang/patches.py Resolved
Comment thread kvcached/kv_cache_manager.py Resolved

@rishabhsinha17 rishabhsinha17 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this belongs in kvcached. the limit is a kvcached construct, kvctl limit and set_memory_limit, with no counterpart in vllm or sglang. only this layer can turn a lowered limit into prefix cache evictions, and without it kvctl limit on a prefix caching engine never converges while idle. our 0.5.20 validation saw the limit hold through live reclamation only (finding 5 in #509).

not ready at e6297cc, one blocker on top of shipiyouniao's three, on the same path as his first. a direct api call evicts the engine's prefix cache on the caller's thread. set_memory_limit enters _resize, which now runs the reclaimers (the callbacks that evict cached pages) inline, and the memory control README tells controllers to call it from a handler in the engine process, which runs off the engine thread. i reproduced it with your fixtures. fill the cache to 9 pages, call set_memory_limit for 4 pages from a thread named control-handler, and _evict_block_ids runs on control-handler while the status comes back applied. your watcher docstring already names the engine thread as the only one allowed to change the block pool, the sglang reclaimer evicts the radix cache (its prefix cache) the same way, and before this pr the call touched allocator and manager state only. one fix covers both findings. let set_memory_limit publish the shrink and hand eviction and the drain of retired pages to the engine thread, through the idle callback and wakeup the watcher already uses, and the idle hook on sglang.

everything else checked out. victims returns only pages whose whole occupancy is evictable and rechecks each page before selection, a running request keeps its blocks, and max_pages only caps #538's heap walk. _idle_state_callbacks exists on 0.17 through 0.30 and WAKEUP from 0.20, and vllm runs those callbacks on the engine thread while has_work is false, 0.17 to 0.19 skip the watcher. the C++ allocator retains the last published target, the seen check reports once per change, and kvctl limit 0 is the zero case shipiyouniao flagged.

your 16 tests pass at e6297cc on my mac, cpu manifest 1248 passed with the failure set identical to main, and the eviction and capacity guard files are green at your head. a10 numbers stay yours, happy to push the thread repro as a test if useful.

@Jeffwan

Jeffwan commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

I have not tested TP/PP yet. I will make it later.

@RixinLiu

RixinLiu commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

/ci run

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown

GPU CI (nightly) on e6297cc, requested by @RixinLiu: run

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown

Nightly correctness · 2026-10-09 · FAIL (14 of 20 cases)

Run · kvcached e6297cc (merged into main at 0222845)
2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each · lmsysorg/sglang:v0.5.20, vllm/vllm-openai:v0.30.0

case setup failed checks
sglang-qwen05b-kv_c1 SGLang 0.5.20, Qwen/Qwen2.5-0.5B-Instruct@7ae5576, kvcached, contiguous, TP 1, KV model dtype shm_removed (e2e_10090419_sglang_qwen05b_kv_c1)
sglang-qwen05b-kv_c0 SGLang 0.5.20, Qwen/Qwen2.5-0.5B-Instruct@7ae5576, kvcached, non-contiguous, TP 1, KV model dtype shm_removed (e2e_10090419_sglang_qwen05b_kv_c0)
sglang-qwen35_4b-kv_c1 SGLang 0.5.20, Qwen/Qwen3.5-4B@851bf6e, kvcached, contiguous, TP 1, KV model dtype shm_removed (e2e_10090419_sglang_qwen35_4b_kv_c1, e2e_10090419_sglang_qwen35_4b_kv_c1_g1000)
sglang-qwen35_4b-kv_c0 SGLang 0.5.20, Qwen/Qwen3.5-4B@851bf6e, kvcached, non-contiguous, TP 1, KV model dtype shm_removed (e2e_10090419_sglang_qwen35_4b_kv_c0, e2e_10090419_sglang_qwen35_4b_kv_c0_g1000)
sglang-gemma4e2b-kv_c1 SGLang 0.5.20, google/gemma-4-E2B-it@3e22461, kvcached, contiguous, TP 1, KV model dtype prefix_cache_reset; shm_removed (e2e_10090419_sglang_gemma4e2b_kv_c1, e2e_10090419_sglang_gemma4e2b_kv_c1_g1); apc_0_cached_tokens_equal (kvcached [2, 2, 1409, 1409] vs native [0, 0, 1409, 1409])
sglang-gemma4e2b-kv_c0 SGLang 0.5.20, google/gemma-4-E2B-it@3e22461, kvcached, non-contiguous, TP 1, KV model dtype prefix_cache_reset; shm_removed (e2e_10090419_sglang_gemma4e2b_kv_c0, e2e_10090419_sglang_gemma4e2b_kv_c0_g1); apc_0_cached_tokens_equal (kvcached [2, 2, 1409, 1409] vs native [0, 0, 1409, 1409])
elastic-sglang-qwen05b+sglang-qwen35_4b SGLang Qwen/Qwen2.5-0.5B-Instruct + SGLang Qwen/Qwen3.5-4B on GPU 1, batch limit 128 b_requests_ok (11/11 failed); contended_requests_ok (11/131 failed); elastic-sglang-qwen35_4b-kv_c1_probe_unchanged; elastic-sglang-qwen35_4b-kv_c1_alive; elastic-sglang-qwen05b-kv_c1_shm_removed (e2e_10090419_elastic_sglang_qwen05b_kv_c1); elastic-sglang-qwen35_4b-kv_c1_no_error_signatures (Traceback (most recent call last)); elastic-sglang-qwen35_4b-kv_c1_shm_removed (e2e_10090419_elastic_sglang_qwen35_4b_kv_c1, e2e_10090419_elastic_sglang_qwen35_4b_kv_c1_g1000)
vllm-qwen05b-kv_c1 vLLM 0.30.0, Qwen/Qwen2.5-0.5B-Instruct@7ae5576, kvcached, contiguous, TP 1, KV model dtype socket_dirs_removed (kvcached-tp-e2e_10090419_vllm_qwen05b_kv_c1-5da7b27a)
vllm-qwen05b-kv_c0 vLLM 0.30.0, Qwen/Qwen2.5-0.5B-Instruct@7ae5576, kvcached, non-contiguous, TP 1, KV model dtype socket_dirs_removed (kvcached-tp-e2e_10090419_vllm_qwen05b_kv_c0-fea90eac)
vllm-qwen35_4b-kv_c1 vLLM 0.30.0, Qwen/Qwen3.5-4B@851bf6e, kvcached, contiguous, TP 1, KV model dtype socket_dirs_removed (kvcached-tp-e2e_10090419_vllm_qwen35_4b_kv_c1-e7ecf74c)
vllm-qwen35_4b-kv_c0 vLLM 0.30.0, Qwen/Qwen3.5-4B@851bf6e, kvcached, non-contiguous, TP 1, KV model dtype socket_dirs_removed (kvcached-tp-e2e_10090419_vllm_qwen35_4b_kv_c0-763618d1)
vllm-gemma4e2b-kv_c1 vLLM 0.30.0, google/gemma-4-E2B-it@3e22461, kvcached, contiguous, TP 1, KV model dtype socket_dirs_removed (kvcached-tp-e2e_10090419_vllm_gemma4e2b_kv_c1-e310eec7)
vllm-gemma4e2b-kv_c0 vLLM 0.30.0, google/gemma-4-E2B-it@3e22461, kvcached, non-contiguous, TP 1, KV model dtype socket_dirs_removed (kvcached-tp-e2e_10090419_vllm_gemma4e2b_kv_c0-66fb684c)
elastic-vllm-qwen05b+vllm-qwen35_4b vLLM Qwen/Qwen2.5-0.5B-Instruct + vLLM Qwen/Qwen3.5-4B on GPU 0, batch limit 128 elastic-vllm-qwen05b-kv_c1_socket_dirs_removed (kvcached-tp-e2e_10090419_elastic_vllm_qwen05b_kv_c1-c9057ca9); elastic-vllm-qwen35_4b-kv_c1_socket_dirs_removed (kvcached-tp-e2e_10090419_elastic_vllm_qwen35_4b_kv_c1-8269b47b)
Setup of every case

On request in a pull request (gpu-pr.yml) on 2x NVIDIA L4 (23034 MiB, driver 580.178.04); the SGLang and vLLM cases run at the same time, one GPU each. kvcached is installed from the pull request merged into main into each engine's official image; each case starts one server in it.

engine image versions
SGLang lmsysorg/sglang:v0.5.20 (sha256:06e4f2ed21af…) sglang 0.5.20, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
vLLM vllm/vllm-openai:v0.30.0 (sha256:8a69ffad015f…) vllm 0.30.0, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
  • Every SGLang server: python3 -m sglang.launch_server --model-path MODEL --served-model-name e2e --host 127.0.0.1 --port 19000 --random-seed 42 --enable-cache-report, followed by the case's flags.
    • kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
    • native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.
  • Every vLLM server: VLLM_USE_V2_MODEL_RUNNER=1 VLLM_SERVER_DEV_MODE=1 vllm serve MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --seed 42 --enable-prompt-tokens-details --enable-prefix-caching --async-scheduling --compilation-config.backend=eager, followed by the case's flags.
    • kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
    • native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.

Cases

Each kvcached case is compared with the native case of the same engine and model.

case model KV cache TP KV dtype case flags
sglang-qwen05b-kv_c1 Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576 kvcached, contiguous 1 model dtype --dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8
sglang-qwen05b-kv_c0 Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576 kvcached, non-contiguous 1 model dtype --dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8
sglang-qwen05b-native Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576 native 1 model dtype --dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8
sglang-qwen35_4b-kv_c1 Qwen/Qwen3.5-4B @ 851bf6e kvcached, contiguous 1 model dtype --dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85
sglang-qwen35_4b-kv_c0 Qwen/Qwen3.5-4B @ 851bf6e kvcached, non-contiguous 1 model dtype --dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85
sglang-qwen35_4b-native Qwen/Qwen3.5-4B @ 851bf6e native 1 model dtype --dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85
sglang-gemma4e2b-kv_c1 google/gemma-4-E2B-it @ 3e22461 kvcached, contiguous 1 model dtype --dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85
sglang-gemma4e2b-kv_c0 google/gemma-4-E2B-it @ 3e22461 kvcached, non-contiguous 1 model dtype --dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85
sglang-gemma4e2b-native google/gemma-4-E2B-it @ 3e22461 native 1 model dtype --dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85
vllm-qwen05b-kv_c1 Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576 kvcached, contiguous 1 model dtype --dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8
vllm-qwen05b-kv_c0 Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576 kvcached, non-contiguous 1 model dtype --dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8
vllm-qwen05b-native Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576 native 1 model dtype --dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8
vllm-qwen35_4b-kv_c1 Qwen/Qwen3.5-4B @ 851bf6e kvcached, contiguous 1 model dtype --dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9
vllm-qwen35_4b-kv_c0 Qwen/Qwen3.5-4B @ 851bf6e kvcached, non-contiguous 1 model dtype --dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9
vllm-qwen35_4b-native Qwen/Qwen3.5-4B @ 851bf6e native 1 model dtype --dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --block-size 1024
vllm-gemma4e2b-kv_c1 google/gemma-4-E2B-it @ 3e22461 kvcached, contiguous 1 model dtype --dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9
vllm-gemma4e2b-kv_c0 google/gemma-4-E2B-it @ 3e22461 kvcached, non-contiguous 1 model dtype --dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9
vllm-gemma4e2b-native google/gemma-4-E2B-it @ 3e22461 native 1 model dtype --dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9

Each case:

  • gen_seq: 9 prompts (6 short, and 300, 900 and 1800 words of filler text), greedy, 64 output tokens, one request at a time
  • gen_conc: the same 9 prompts, all at once
  • the prefix cache is reset
  • apc_0, apc_1, apc_2: 4 questions about one shared 1300-word document, greedy, 32 output tokens, one request at a time
  • apc_conc: the same 4 questions, all at once
  • the server is stopped with SIGINT to its process group (E2E_STOP_SIGNAL), SIGKILL after 90 s
check passes when
server_ready the server answers /health within the ready timeout
<phase>_ok every request of the phase returns HTTP 200 with output tokens
prefix_cache_reset the prefix-cache reset endpoint succeeds
server_alive_after_load the server process group is still running after the load
clean_exit_on_signal the server exits within 90 s of the stop signal
no_error_signatures no traceback, CUDA error or patch failure in the log before the stop signal
kvcached_active kvcached applied every required patch and sized the KV cache
kvcached_inactive the native server's log has no [kvcached] line
shm_removed no /dev/shm segment of the server is left
socket_dirs_removed no /tmp/kvcached-tp-* directory of the server is left
no_gpu_process_left no process is left on the GPUs of the run
<phase>_tokens_equal gen_seq and apc_0..2 give the same greedy tokens as native
apc_<n>_cached_tokens_equal every request reports the same cached_tokens as native

Elastic tests

case GPU server A server B
elastic-sglang-qwen05b+sglang-qwen35_4b 1 SGLang, Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576, kvcached, contiguous, --dtype bfloat16 --context-length 8192 --mem-fraction-static 0.8 --max-running-requests 128 SGLang, Qwen/Qwen3.5-4B @ 851bf6e, kvcached, contiguous, --dtype bfloat16 --context-length 8192 --mem-fraction-static 0.85 --max-running-requests 128
elastic-vllm-qwen05b+vllm-qwen35_4b 0 vLLM, Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576, kvcached, contiguous, --dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.8 --max-num-seqs 128 vLLM, Qwen/Qwen3.5-4B @ 851bf6e, kvcached, contiguous, --dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.9 --max-num-seqs 128
  • both servers use the contiguous KV layout and a batch limit of 128; A starts first, then B, on the same GPU
  • every load request is a distinct 3000-word filler prompt with 1024 output tokens and ignore_eos
  • calibration: 4 such requests per server measure the KV memory one request holds
  • each server's load is sized to need 70% of the GPU memory free after calibration, at most 128 requests
  • phases: (1) A gets its load; (2) A's prefix cache is reset; (3) B gets its load and its prefix cache is reset; (4) both get their loads at the same time
  • 3 greedy probe prompts of 32 tokens on each server before and after
check passes when
<server>_ready the server answers /health within the ready timeout
a_requests_ok, b_requests_ok every request of the server's own load succeeds
a_filled_memory A's KV usage grows by more than half of the free memory
a_released_memory after its prefix cache is reset, A keeps less than half of what it grew
b_reused_a_memory B's KV usage grows past what was free at A's peak
contended_requests_ok every request succeeds while both loads run
<server>_probe_unchanged the probes give the same tokens before and after
<server>_alive the server is still running at the end
<server>_<shutdown check> the shutdown and cleanup checks of each server

Logs: artifact, kept 30 days: serve.log and cmd.txt per case, summary.json with every request's tokens, setup.json, run.log, GPU state at the end.

@RixinLiu

RixinLiu commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

GPU CI result for e6297cc merged into main at 0222845 (run), on 2x L4:

The kvcached cases match native's greedy tokens and cached_tokens, and every native case passes. All of the failures are known and not caused by this PR:

None of these cases lowers the KV memory limit, so this run shows that normal serving still works, but it does not exercise the path this PR changes.

Jeffwan added a commit to Jeffwan/kvcached that referenced this pull request Oct 9, 2026
Address review on ovg-project#552:
- resize() and set_memory_limit() only record a shrink, as a controller
  may call them from its own thread; alloc(), request completion and the
  idle wakeup evict on the engine thread
- the vLLM idle wakeup also follows set_memory_limit() revisions and
  drains the retired pages
- SGLang registers its reclaimers when the scheduler is built, before
  any idle iteration
- a published resize target of 0 is applied; -1 means none (test fakes
  now return -1 like the C++ allocator)

Signed-off-by: Jiaxin Shan <seedjeffwan@gmail.com>
A lower limit (kvctl limit / set_memory_limit) stayed deferred while
cached prefixes held its pages, idle or under traffic: allocations reuse
cached slots one LRU victim at a time and never empty a page, and a kvctl
target was only applied by the next allocation.

- resize() only records a shrink; the prefix cache evicts whole pages on
  the engine thread (alloc, request completion, idle), and pages retired
  for the shrink are not reallocated
- idle engines apply new limits: a vLLM EngineCore wakeup that follows
  kvctl targets and set_memory_limit() revisions, and an SGLang scheduler
  idle hook (SGLang TP/PP > 1 unchanged)
- a published resize target of 0 is applied; -1 means none
- tests: tests/test_prefix_cache_shrink.py (CPU) and tests/e2e/limit.py
  (nightly GPU profile)

Signed-off-by: Jiaxin Shan <seedjeffwan@gmail.com>
@Jeffwan
Jeffwan force-pushed the jiaxin/release-prefix-cache branch from 9cb8ad0 to a84b6d8 Compare October 9, 2026 20:25
@Jeffwan

Jeffwan commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor Author

@rishabhsinha17 thanks. eviction no longer runs inside resize(). the control-handler case is test_set_memory_limit_leaves_eviction_to_the_engine_thread

I finished the PP/TP tests.

  • Setup: TP/PP (vLLM 0.30.0, Qwen3-4B, 2x A40, driver 580).
  • Per run: fill the prefix cache; lower the limit while 24 requests on one shared 1500-word document are queued (--max-num-seqs 8) and hitting the prefix cache, half of them cancelled by a 4 s client timeout; then idle, 100 more requests, restore the limit and refill
topology limit via limit filled when lowered load done idle GPU 0 / GPU 1 MiB (filled → idle) peak under 100 requests prefix hits / cancelled API status outputs
TP=2 kvctl 342 1152 1296 288 288 6824 / 6557 → 5240 / 4973 288 12 / 12 — same
TP=2 API 342 1152 1296 288 288 6824 / 6557 → 5240 / 4973 288 12 / 12 deferred → applied same
PP=2 kvctl 315 1152 360 288 288 6948 / 6713 → 5726 / 5489 288 12 / 12 — same
PP=2 API 315 1152 1224 288 288 6948 / 6713 → 5726 / 5489 288 12 / 12 deferred → applied same
TP=2, rank 1 barrier +5 s API 342 1152 1296 288 288 6824 / 6557 → 5240 / 4973 288 12 / 12 deferred → applied same

every non-cancelled request succeeded, and no run had a traceback, CUDA/NCCL error or worker restart before shutdown.

@shipiyouniao shipiyouniao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from my side! We still need to land a few other PRs to sort out CI before merging this one. Thanks for your patience :)

@RixinLiu RixinLiu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. But I think there's a problem when kvctl limit and set_memory_limit() are both used. With the fakes in test_prefix_cache_shrink.py: kvctl limit to 12 pages (9 mapped), then set_memory_limit() to 4 pages, which stays deferred. When the idle callback runs, apply_resize_target() applies the old kvctl target again, so the shrink gets cancelled and memory_limit_state() says applied while 9 pages are still mapped.

But this problem is not related to this PR. No blocking from my side.

@rishabhsinha17 rishabhsinha17 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM at a84b6d8. _resize now only records the shrink. eviction runs in alloc() and reclaim_for_shrink(), called on the engine thread by the vllm idle callback and the sglang on_idle wrapper. no controller thread touches the block pool.

my repro passes. set_memory_limit for 4 pages from a thread named control-handler evicts nothing and returns deferred, the idle pass then evicts on the engine thread to applied, 4 mapped. the reclaim call put back into _resize turns test_set_memory_limit_leaves_eviction_to_the_engine_thread red. shipiyouniao's three hold, revision followed, reclaimer registered after Scheduler.init, zero reaches resize.

your file 23 passed, cpu manifest 1293 with the failure set identical to main, the #469, #511 and #538 files green. rixin's kvctl then api case is pre-existing, main's alloc() re-applies the kept kvctl target the same way, a two line follow-up applies a published target once, not every pass. a10 and a40 stay yours.

@rishabhsinha17

Copy link
Copy Markdown
Collaborator

the kvctl then api case from rixin's note is now #558. pre-existing at main, alloc() re-applied the retained kvctl quota on every pass, so two lines in that block, _apply_resize_target at this head, apply a published quota once.

@RixinLiu

Copy link
Copy Markdown
Collaborator

Merged

@RixinLiu
RixinLiu merged commit dc0d860 into ovg-project:main Oct 12, 2026
9 checks passed
@Jeffwan

Jeffwan commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

thanks for all your help! I will put more efforts other issues from my side.

@Jeffwan
Jeffwan deleted the jiaxin/release-prefix-cache branch October 12, 2026 18:32
rishabhsinha17 added a commit to rishabhsinha17/kvcached that referenced this pull request Oct 15, 2026
…nto _apply_resize_target

ovg-project#552 extracted the kvctl block of _alloc_impl() into _apply_resize_target()
and accepts a zero quota. The applied-quota check and its record move into
that helper unchanged, so alloc() and the idle pass share them.

Signed-off-by: Rishabh Sinha <rsinha17@terpmail.umd.edu>
rishabhsinha17 added a commit to rishabhsinha17/kvcached that referenced this pull request Oct 15, 2026
…arget

shipiyouniao's review of ovg-project#558 found that applying the quota once stalls
a kvctl shrink the pool refused while reserved pages were the only pages
in use. _resize() frees the reserved blocks before it enters the shrink,
so that free() cannot complete it, and the next allocation skipped the
retained quota. At main the next allocation applied it again and the
shrink completed.

The control published last is the one in force now. _apply_resize_target()
applies the retained quota on every pass while kvctl published last, as
at main, and skips it once resize() published a later target, which
set_memory_limit() calls. A changed kvctl value is a new publication and
takes the limit back. clear() resets nothing, since it publishes nothing.
The -1 and zero handling from ovg-project#552 stays.

Refs ovg-project#552

Signed-off-by: Rishabh Sinha <rsinha17@terpmail.umd.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants