diff --git a/docs/configuration.md b/docs/configuration.md index 487497fe..17566531 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -90,6 +90,19 @@ cache: each vLLM version, model config and GPU type adds an entry. endpoint scaled to zero); vLLM rebuilds it. Set `VLLM_CACHE_ROOT=/root/.cache/vllm` to keep the cache in the container. +Set `VLLM_ENABLE_STARTUP_PLAN=1` on the endpoint (`1` or `0` only; `true` fails the start) +and vLLM also saves the KV-cache memory budget it measured under +`$VLLM_CACHE_ROOT/startup_plan/`, keyed by the vLLM version, the model, cache, parallel and +compilation config, the GPU name, total memory and compute capability, the torch/CUDA build +and the rank. A later cold start with the same key and at least as much free GPU memory +reuses the budget and skips the memory measurement and the CUDA-graph memory estimate; +any mismatch, less free memory, or an explicit `KV_CACHE_MEMORY_BYTES` means it measures as +before, with nothing to clean up. Plans follow `VLLM_CACHE_ROOT`, including the fallback +above (an existing plan on the volume is not copied). Off by default: vLLM's key does not +yet include `gpu_memory_utilization` (vllm-project/vllm#60143), so after changing +`GPU_MEMORY_UTILIZATION` delete `$VLLM_CACHE_ROOT/startup_plan/` while no worker is +starting, or unset the variable, or the old budget is reused. + ## Speculative decoding Pass a full JSON config, which vLLM parses natively: