From 4e1e03a77801ac3613afc52a6fad329c6a8d9eb1 Mon Sep 17 00:00:00 2001 From: Ashwin Sreedhar Date: Fri, 23 Oct 2026 15:24:05 -0400 Subject: [PATCH 1/4] docs: document VLLM_ENABLE_STARTUP_PLAN as an opt-in next to the compile cache vLLM (v0.29.0+) persists its measured KV-cache budget under VLLM_CACHE_ROOT, which this image already keeps on the network volume, so later cold starts skip the memory measurement and CUDA-graph memory estimate when the plan's fingerprint matches. Documented as opt-in rather than set in the image: the fingerprint omits gpu_memory_utilization until vllm-project/vllm#60143 lands, so a changed limit would reuse the old budget. Numbers in the doc are placeholders pending the v0.32.0 rerun. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01KGxXNtPM12ALm9YXqHhiY4 --- docs/configuration.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index 487497fe..aaad4005 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -90,6 +90,26 @@ cache: each vLLM version, model config and GPU type adds an entry. endpoint scaled to zero); vLLM rebuilds it. Set `VLLM_CACHE_ROOT=/root/.cache/vllm` to keep the cache in the container. +### Startup plan (opt-in) + +Set `VLLM_ENABLE_STARTUP_PLAN=1` on the endpoint and vLLM (v0.29.0+) saves the KV-cache +memory budget it measured under `$VLLM_CACHE_ROOT/startup_plan/`, next to the compile +cache, keyed by a fingerprint of the vLLM version, the model, parallel and compilation +config, the GPU name, total memory and compute capability, and the torch/CUDA build. A +later cold start with the same fingerprint and at least as much free GPU memory reuses that +budget and skips the memory measurement and the CUDA-graph memory estimate (the profile +forward pass and graph capture still run); any mismatch measures as before. The plan is +looked up and written under whatever `VLLM_CACHE_ROOT` is in effect, so it follows the +compile-cache fallback above; an existing plan on the volume is not copied. + +Measured on Qwen3-8B bf16, RTX 4090, network volume, vLLM v0.32.0: the profiling phase +drops from about 6.5 s to about 2.3 s per cold start. + +Not on by default yet: vLLM's fingerprint does not include `gpu_memory_utilization` +(vllm-project/vllm#60143), so after changing `GPU_MEMORY_UTILIZATION` on an endpoint +delete `$VLLM_CACHE_ROOT/startup_plan/` or unset the variable, or the old budget is reused. +When that lands upstream this can become the image default. + ## Speculative decoding Pass a full JSON config, which vLLM parses natively: From a54300d7b343eb77279f175aed120a6b3ada5978 Mon Sep 17 00:00:00 2001 From: Ashwin Sreedhar Date: Fri, 23 Oct 2026 15:35:30 -0400 Subject: [PATCH 2/4] docs: measured startup-plan numbers on vLLM v0.32.0 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01KGxXNtPM12ALm9YXqHhiY4 --- docs/configuration.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index aaad4005..b8cb73a3 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -102,8 +102,9 @@ forward pass and graph capture still run); any mismatch measures as before. The looked up and written under whatever `VLLM_CACHE_ROOT` is in effect, so it follows the compile-cache fallback above; an existing plan on the volume is not copied. -Measured on Qwen3-8B bf16, RTX 4090, network volume, vLLM v0.32.0: the profiling phase -drops from about 6.5 s to about 2.3 s per cold start. +Measured on Qwen3-8B bf16, RTX 4090, network volume, vLLM v0.32.0, three cold starts each: +the profiling phase (`determine_available_memory`) takes 6.2-8.2 s without the plan and +1.9-2.2 s with it, about 4.5 s saved per cold start. Not on by default yet: vLLM's fingerprint does not include `gpu_memory_utilization` (vllm-project/vllm#60143), so after changing `GPU_MEMORY_UTILIZATION` on an endpoint From f2ba564e88c9d936d46e5f7e173d8045448ffb11 Mon Sep 17 00:00:00 2001 From: Ashwin Sreedhar Date: Fri, 23 Oct 2026 15:40:09 -0400 Subject: [PATCH 3/4] docs: fold the startup plan into the compile-cache paragraph; numbers move to the PR Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01KGxXNtPM12ALm9YXqHhiY4 --- docs/configuration.md | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index b8cb73a3..14252be9 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -90,26 +90,18 @@ cache: each vLLM version, model config and GPU type adds an entry. endpoint scaled to zero); vLLM rebuilds it. Set `VLLM_CACHE_ROOT=/root/.cache/vllm` to keep the cache in the container. -### Startup plan (opt-in) - -Set `VLLM_ENABLE_STARTUP_PLAN=1` on the endpoint and vLLM (v0.29.0+) saves the KV-cache -memory budget it measured under `$VLLM_CACHE_ROOT/startup_plan/`, next to the compile -cache, keyed by a fingerprint of the vLLM version, the model, parallel and compilation -config, the GPU name, total memory and compute capability, and the torch/CUDA build. A -later cold start with the same fingerprint and at least as much free GPU memory reuses that -budget and skips the memory measurement and the CUDA-graph memory estimate (the profile -forward pass and graph capture still run); any mismatch measures as before. The plan is -looked up and written under whatever `VLLM_CACHE_ROOT` is in effect, so it follows the -compile-cache fallback above; an existing plan on the volume is not copied. - -Measured on Qwen3-8B bf16, RTX 4090, network volume, vLLM v0.32.0, three cold starts each: -the profiling phase (`determine_available_memory`) takes 6.2-8.2 s without the plan and -1.9-2.2 s with it, about 4.5 s saved per cold start. - -Not on by default yet: vLLM's fingerprint does not include `gpu_memory_utilization` -(vllm-project/vllm#60143), so after changing `GPU_MEMORY_UTILIZATION` on an endpoint -delete `$VLLM_CACHE_ROOT/startup_plan/` or unset the variable, or the old budget is reused. -When that lands upstream this can become the image default. +Set `VLLM_ENABLE_STARTUP_PLAN=1` on the endpoint (`1` or `0` only; `true` fails the start) +and vLLM also saves the KV-cache memory budget it measured under +`$VLLM_CACHE_ROOT/startup_plan/`, keyed by the vLLM version, the model, parallel and +compilation config, the GPU name, total memory and compute capability, the torch/CUDA build +and the rank. A later cold start with the same key and at least as much free GPU memory +reuses the budget and skips the memory measurement and the CUDA-graph memory estimate; +any mismatch, less free memory, or an explicit `KV_CACHE_MEMORY_BYTES` means it measures as +before, with nothing to clean up. Plans follow `VLLM_CACHE_ROOT`, including the fallback +above (an existing plan on the volume is not copied). Off by default: vLLM's key does not +yet include `gpu_memory_utilization` (vllm-project/vllm#60143), so after changing +`GPU_MEMORY_UTILIZATION` delete `$VLLM_CACHE_ROOT/startup_plan/` while no worker is +starting, or unset the variable, or the old budget is reused. ## Speculative decoding From ea9a4f13d87d9d8484a868f5e21fed6d324ac64a Mon Sep 17 00:00:00 2001 From: Ashwin Sreedhar Date: Fri, 23 Oct 2026 15:46:21 -0400 Subject: [PATCH 4/4] docs: the plan key also covers the cache config Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01KGxXNtPM12ALm9YXqHhiY4 --- docs/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuration.md b/docs/configuration.md index 14252be9..17566531 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -92,7 +92,7 @@ keep the cache in the container. Set `VLLM_ENABLE_STARTUP_PLAN=1` on the endpoint (`1` or `0` only; `true` fails the start) and vLLM also saves the KV-cache memory budget it measured under -`$VLLM_CACHE_ROOT/startup_plan/`, keyed by the vLLM version, the model, parallel and +`$VLLM_CACHE_ROOT/startup_plan/`, keyed by the vLLM version, the model, cache, parallel and compilation config, the GPU name, total memory and compute capability, the torch/CUDA build and the rank. A later cold start with the same key and at least as much free GPU memory reuses the budget and skips the memory measurement and the CUDA-graph memory estimate;