Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,19 @@ cache: each vLLM version, model config and GPU type adds an entry.
endpoint scaled to zero); vLLM rebuilds it. Set `VLLM_CACHE_ROOT=/root/.cache/vllm` to
keep the cache in the container.

Set `VLLM_ENABLE_STARTUP_PLAN=1` on the endpoint (`1` or `0` only; `true` fails the start)
and vLLM also saves the KV-cache memory budget it measured under
`$VLLM_CACHE_ROOT/startup_plan/`, keyed by the vLLM version, the model, cache, parallel and
compilation config, the GPU name, total memory and compute capability, the torch/CUDA build
and the rank. A later cold start with the same key and at least as much free GPU memory
reuses the budget and skips the memory measurement and the CUDA-graph memory estimate;
any mismatch, less free memory, or an explicit `KV_CACHE_MEMORY_BYTES` means it measures as
before, with nothing to clean up. Plans follow `VLLM_CACHE_ROOT`, including the fallback
above (an existing plan on the volume is not copied). Off by default: vLLM's key does not
yet include `gpu_memory_utilization` (vllm-project/vllm#60143), so after changing
`GPU_MEMORY_UTILIZATION` delete `$VLLM_CACHE_ROOT/startup_plan/` while no worker is
starting, or unset the variable, or the old budget is reused.

## Speculative decoding

Pass a full JSON config, which vLLM parses natively:
Expand Down
Loading