[JasperChan929]

Fixes #588

Background capacity queries can use GPU 0 for an FTensor backend initialized on GPU 1. In the normal SGLang startup reported in #588, this leaves null-block initialization waiting while GPU 1 has available memory.

Resolve an implicit CUDA device at initialization, query the backend's device and restore the caller's device afterward. Standalone IPC allocators without a local backend retain their current-device behavior.

The complete launch configuration, inline pressure helper, original output and controls are in #588. On SGLang 0.5.20 / Torch cu130 with two RTX 4090s, the fixed version and all controls complete; all 15 requests' full output token arrays match native. This serving validation is TP=1. Combined-path verification remains pending #487 landing and rebasing this change onto it.

[shipiyouniao]

Could you provide a runnable repro starting vLLM or SGLang normally, with the launch command, CUDA_VISIBLE_DEVICES setting and exact steps that trigger this?

The current test constructs a native pool and starts a query thread directly. It demonstrates the low-level device behavior, but bypasses the engine's initialization and execution order. We need to establish whether a supported serving setup actually reaches the same failure. With only one GPU visible per process, the pool uses logical GPU 0, so that setup doesn't demonstrate this particular mismatch.

Please share the failing output from the original code through the normal engine entry points, without changing internal state or manually substituting the call sequence. That will help us judge the real impact before reviewing the fix.

[shipiyouniao]

@JasperChan929 This applies to both this PR and #587, and to future bug reports and fixes: provide a runnable reproduction that triggers through the normal execution flow in a real, supported environment, including the launch configuration and exact steps.

An invasive test that bypasses the engine's execution order or manually creates an internal call sequence is not acceptable as evidence that normal usage is broken. Calling internal methods in an arbitrary order can produce failures that the engine's own orchestration prevents. Regression tests do not replace a valid reproduction of the reported problem.

Please meet this requirement before submitting further bug-fix PRs. If this happens again, I will close the PR without another reminder. I do not want to keep repeating this requirement.

[shipiyouniao]

LGTM from my side. Let's wait for #487 to land, then rebase this and verify the combined path before merging.

Withdrawing my approval.

[shipiyouniao]

@JasperChan929 I asked for a real, runnable reproduction in the description of issue #588, not a new document and a one-off GPU-memory pressure script committed to this repository.
I already made this explicit in #584: #584 (comment). Documentation is maintained by the maintainers; these PRs should not include documentation changes. Adding a reproduction document is not an exception. This requirement should not need repeating.
I thought the documentation and helper linked in the PR description were on a separate branch for me to inspect as reproduction evidence. I hadn't noticed they had been pushed into this PR branch. Had I noticed that, I would not have approved this version.
The reproduction document and helper need to be removed from this PR. Put the complete steps, helper snippet, environment and observed results in #588's description. Keep this PR focused on the fix and relevant regression tests. My approval should not stand while these scope issues remain.
From the "non-human" line breaks—obvious at a single glance in your PR description—to the "eyeball-it" debugging and fixing process (which would be humanly impossible to perform accurately), everything about your work screams an uncritical, total reliance on generative AI.
We aren’t against AI, but I have a deep aversion to "vibe coders" who lean entirely on AI while showing no respect for the codebase or the repository itself.
You’re welcome to challenge my view, but can you actually explain—line by line, in your own words—what these changes of yours are actually doing? It’s quite possible that the reviewer understands your own code better than you do.
I will no longer review or approve your PRs. Please ask another maintainer to review future submissions. The explicit scope requirements have not been followed, and I cannot continue spending time repeating them and cleaning up submissions.

[JasperChan929]

@shipiyouniao First, I want to apologize and thank you for the time and care you’ve put into reviewing my PRs. You’ve been quick to respond and detailed in your feedback, and I really appreciate that. I’m sorry I made you repeat clear requirements and spend more time on avoidable rework.

I really enjoy contributing to this project. I find the direction interesting and promising. When I have an idea or find a bug, I’ve been too eager to get a PR out quickly. In doing so, I haven’t given enough attention to the quality of what I submit or the extra work it creates for you.

You had already asked me to leave documentation out of these PRs, and I still included it. I used AI assistance without reviewing the final changes carefully enough. It was my responsibility to understand and check what I was submitting.

I would still like to contribute to the project, and I want those contributions to help rather than create more work. I’ll slow down, review the existing feedback, and make sure I understand and verify my changes before submitting them. Thank you again for the time you’ve already given me.

[shipiyouniao]

/ci run weekly

[github-actions]

GPU CI (weekly) on 30e4203, requested by @shipiyouniao: run

[github-actions]

Weekly correctness · 2026-10-23 · PASS

Run · kvcached 30e4203 (merged into main at 3170a0a)
2x NVIDIA L4 (23034 MiB, driver 580.178.04) · vllm/vllm-openai:v0.30.0, lmsysorg/sglang:v0.5.20

All 26 cases passed.

Setup of every case

On request in a pull request (gpu-pr.yml) on 2x NVIDIA L4 (23034 MiB, driver 580.178.04). kvcached is installed from the pull request merged into main into each engine's official image; each case starts one server in it.

engine	image	versions
vLLM	vllm/vllm-openai:v0.30.0 (sha256:8a69ffad015f…)	vllm 0.30.0, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0
SGLang	lmsysorg/sglang:v0.5.20 (sha256:06e4f2ed21af…)	sglang 0.5.20, kvcached 0.1.6, torch 2.13.0+cu130, CUDA 13.0

• Every vLLM server: HF_HUB_OFFLINE=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_SERVER_DEV_MODE=1 vllm serve MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --seed 42 --enable-prompt-tokens-details --enable-prefix-caching --async-scheduling --compilation-config.backend=eager, followed by the case's flags.
  • kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
  • native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.
• Every SGLang server: HF_HUB_OFFLINE=1 python3 -m sglang.launch_server --model-path MODEL --served-model-name e2e --host 127.0.0.1 --port 18000 --random-seed 42 --enable-cache-report, followed by the case's flags.
  • kvcached cases add ENABLE_KVCACHED=true KVCACHED_AUTOPATCH=1 and a per-server KVCACHED_IPC_NAME; KVCACHED_CONTIGUOUS_LAYOUT follows the KV layout.
  • native cases add ENABLE_KVCACHED=false KVCACHED_AUTOPATCH=0.

Cases

Each kvcached case is compared with the native case of the same engine and model.

case	model	KV cache	TP	KV dtype	case flags
vllm-qwen05b-kv_c1-tp2	Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576	kvcached, contiguous	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8
vllm-qwen05b-kv_c0-tp2	Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576	kvcached, non-contiguous	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8
vllm-qwen05b-native-tp2	Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576	native	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8
vllm-dsv2lite-kv_c1-tp2	deepseek-ai/DeepSeek-V2-Lite-Chat @ 8586474	kvcached, contiguous	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code
vllm-dsv2lite-kv_c0-tp2	deepseek-ai/DeepSeek-V2-Lite-Chat @ 8586474	kvcached, non-contiguous	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code
vllm-dsv2lite-native-tp2	deepseek-ai/DeepSeek-V2-Lite-Chat @ 8586474	native	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --trust-remote-code
vllm-gemma4_12b-kv_c1-tp2	google/gemma-4-12B-it @ 707f0a3	kvcached, contiguous	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9
vllm-gemma4_12b-kv_c0-tp2	google/gemma-4-12B-it @ 707f0a3	kvcached, non-contiguous	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9
vllm-gemma4_12b-native-tp2	google/gemma-4-12B-it @ 707f0a3	native	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9
vllm-qwen35_9b-kv_c1-tp2	Qwen/Qwen3.5-9B @ c202236	kvcached, contiguous	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9
vllm-qwen35_9b-kv_c0-tp2	Qwen/Qwen3.5-9B @ c202236	kvcached, non-contiguous	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9
vllm-qwen35_9b-native-tp2	Qwen/Qwen3.5-9B @ c202236	native	2	model dtype	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 16 --gpu-memory-utilization 0.9 --block-size 1024
sglang-qwen05b-kv_c1-tp2	Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576	kvcached, contiguous	2	model dtype	--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8
sglang-qwen05b-kv_c0-tp2	Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576	kvcached, non-contiguous	2	model dtype	--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8
sglang-qwen05b-native-tp2	Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576	native	2	model dtype	--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8
sglang-dsv2lite-kv_c1-tp2	deepseek-ai/DeepSeek-V2-Lite-Chat @ 8586474	kvcached, contiguous	2	model dtype	--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code
sglang-dsv2lite-kv_c0-tp2	deepseek-ai/DeepSeek-V2-Lite-Chat @ 8586474	kvcached, non-contiguous	2	model dtype	--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code
sglang-dsv2lite-native-tp2	deepseek-ai/DeepSeek-V2-Lite-Chat @ 8586474	native	2	model dtype	--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85 --trust-remote-code
sglang-qwen35_9b-kv_c1-tp2	Qwen/Qwen3.5-9B @ c202236	kvcached, contiguous	2	model dtype	--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85
sglang-qwen35_9b-kv_c0-tp2	Qwen/Qwen3.5-9B @ c202236	kvcached, non-contiguous	2	model dtype	--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85
sglang-qwen35_9b-native-tp2	Qwen/Qwen3.5-9B @ c202236	native	2	model dtype	--dtype bfloat16 --context-length 8192 --max-running-requests 16 --mem-fraction-static 0.85
vllm-qwen05b-kv_c1-fp8kv	Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576	kvcached, contiguous	1	fp8	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8
vllm-qwen05b-native-fp8kv	Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576	native	1	fp8	--dtype bfloat16 --max-model-len 8192 --max-num-seqs 32 --gpu-memory-utilization 0.8
sglang-qwen05b-kv_c1-fp8kv	Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576	kvcached, contiguous	1	fp8_e4m3	--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8
sglang-qwen05b-native-fp8kv	Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576	native	1	fp8_e4m3	--dtype bfloat16 --context-length 8192 --max-running-requests 32 --mem-fraction-static 0.8

Each case:

• gen_seq: 9 prompts (6 short, and 300, 900 and 1800 words of filler text), greedy, 64 output tokens, one request at a time
• gen_conc: the same 9 prompts, all at once
• the prefix cache is reset
• apc_0, apc_1, apc_2: 4 questions about one shared 1300-word document, greedy, 32 output tokens, one request at a time
• apc_conc: the same 4 questions, all at once
• the server is stopped with SIGINT to its process group (E2E_STOP_SIGNAL), SIGKILL after 90 s

check	passes when
server_ready	the server answers /health within the ready timeout
<phase>_ok	every request of the phase returns HTTP 200 with output tokens
prefix_cache_reset	the prefix-cache reset endpoint succeeds
server_alive_after_load	the server process group is still running after the load
clean_exit_on_signal	the server exits within 90 s of the stop signal
no_error_signatures	no traceback, CUDA error or patch failure in the log before the stop signal
kvcached_active	kvcached applied every required patch and sized the KV cache
kvcached_inactive	the native server's log has no [kvcached] line
no_gpu_process_left	no process is left on the GPUs of the run
<phase>_tokens_equal	in gen_seq and in each of apc_0..2, more than half of the prompts give the same greedy tokens as native
apc_<n>_cached_tokens_equal	every request reports the same cached_tokens as native

Elastic tests

case	GPU	server A	server B
elastic-vllm-qwen05b+sglang-qwen35_4b	0	vLLM, Qwen/Qwen2.5-0.5B-Instruct @ 7ae5576, kvcached, contiguous, --dtype bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.8 --max-num-seqs 128	SGLang, Qwen/Qwen3.5-4B @ 851bf6e, kvcached, contiguous, --dtype bfloat16 --context-length 8192 --mem-fraction-static 0.85 --max-running-requests 128

• both servers use the contiguous KV layout and a batch limit of 128; A starts first, then B, on the same GPU
• every load request is a distinct 3000-word filler prompt with 1024 output tokens and ignore_eos
• calibration: 4 such requests per server measure the KV memory one request holds
• each server's load is sized to need 70% of the GPU memory free after calibration, at most 128 requests
• phases: (1) A gets its load; (2) A's prefix cache is reset; (3) B gets its load and its prefix cache is reset; (4) both get their loads at the same time
• 3 greedy probe prompts of 32 tokens on each server before and after

check	passes when
<server>_ready	the server answers /health within the ready timeout
a_requests_ok, b_requests_ok	every request of the server's own load succeeds
a_filled_memory	A's KV usage grows by more than half of the free memory
a_released_memory	after its prefix cache is reset, A keeps less than half of what it grew
b_reused_a_memory	B's KV usage grows past what was free at A's peak
contended_requests_ok	every request succeeds while both loads run
<server>_probe_unchanged	the probes give the same tokens before and after
<server>_alive	the server is still running at the end
<server>_<shutdown check>	the shutdown and cleanup checks of each server

Logs: artifact, kept 30 days: serve.log and cmd.txt per case, summary.json with every request's tokens, setup.json, run.log, GPU state at the end.

[shipiyouniao]

LGTM at 30e4203. Capacity queries now use the initialized pool's GPU, restore the caller's device, and preserve mainline physical budget limits without creating missing allocators.

Validation on this head: Python 3.10 through 3.13 each had 1536 tests pass; the single-T4 native run had 79 pass. CPU CI, pre-commit CI, and weekly GPU CI passed; weekly covered 26 cases on two L4s. Codex review with xhigh reasoning and fast mode found no actionable regressions.

The eight new dual-GPU native pressure/query/preallocation cases remain unexecuted: the T4 host has one GPU, and weekly does not run these native tests. They were among 11 two-GPU skips in the T4 run. @RixinLiu, please take a look at this remaining coverage gap.