[ashwinsreedhar28]

Summary

Documents VLLM_ENABLE_STARTUP_PLAN=1 as an opt-in endpoint setting, in the compile-cache paragraph of docs/configuration.md. With it, vLLM saves its measured KV-cache memory budget under $VLLM_CACHE_ROOT/startup_plan/, which this image already keeps on the network volume, and later cold starts with a matching key skip the memory measurement and the CUDA-graph memory estimate. Docs only; no image change.

Why opt-in and not the image default

vLLM's plan key does not yet include gpu_memory_utilization (vllm-project/vllm#60143 proposes adding it). Until it does, a plan recorded at one limit is reused after the limit changes while free memory still clears the gate, so a user lowering GPU_MEMORY_UTILIZATION after an OOM would keep the old budget. The doc says so and gives the workaround. When #60143 lands this can become one ENV line in the Dockerfile; I have that branch ready.

Measured

Runpod Serverless, this image's base (vLLM v0.32.0), Qwen3-8B bf16, RTX 4090, network volume, compile cache warm. Three measured cold starts each; the with-plan endpoint first ran one discarded boot that wrote the plan (profile 7.5 s on that boot). Per-phase wall time recorded inside the vLLM worker:

determine_available_memory	without plan	with plan
runs, in order	8.2, 6.5, 6.5 s	1.90, 1.97, 2.18 s
median	6.5 s	1.97 s

4.6 s saved per cold start (median). Weight load, warm-up and graph capture are not affected (weight load varies with the host page cache in both arms). The three with-plan runs landed on three different workers, so the plan is reused across workers on the volume; on v0.30.0 earlier the same test showed 6.9 → 2.3 s, including a plan written by one endpoint applied by a second. Raw data and the runner: https://github.com/ashwinsreedhar28/vllm-coldstart/tree/main/bench/results

Also found while measuring: vLLM stages the plan file as <path>.tmp.<pid>, which can collide across containers sharing a cache root; fix proposed in vllm-project/vllm#63388. Not a blocker for opt-in use.

Notes

• Plans are looked up and written under whatever VLLM_CACHE_ROOT is in effect, so they follow src/compile_cache.py's fallback to the container; an existing plan on the volume is not copied.
• A plan hit still runs the profile forward pass and CUDA-graph capture; only the memory measurement and graph-memory estimate are skipped.
• The variable parses as int: 1/0 only; true raises inside the worker. Base images that predate it (it is present in v0.30.0; I have not checked older) log Unknown vLLM environment variable and ignore it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KGxXNtPM12ALm9YXqHhiY4