Repository navigation
Very slow cold starts even with flashboot #111
Description
Hi, we're experiencing very slow cold starts even when activating flash boot, and this didn't happen before, with the same model architecture (Llama-3.1-8B) (different custom versions though). In fact, when it's about 1' since the last request, the worker appears initializing again, downloading weights etc. We've tried attaching a data storage to it, thinking that would lower cold start times (our hypothesis was that weights were downloaded there once and then just loaded from that disk at subsequent requests). However, that made things even worse, the delay times going up to 2' for a request.
We're about to launch an AI-based app, and have been using Runpod for development for some months now, but these delay times are not acceptable for the app to run properly. We need to scale down to 0 to keep costs variable at the beginning until we have more customers (then we will use active workers, for sure).
Can you please provide a solution for this? Is there some way to configure Runpod serverless endpoints so that delay times come back to where they were a week or two ago (1-2s)? @alpayariyak @Jorghi12 @pandyamarut @justinmerrell @carlson-svg @mikljohansson @casper-hansen @joennlae @willsamu @rachfop @vladmihaisima
Thanks in advance.
Activity
pandyamarut commentedon Sep 18, 2024
@avacaondata Sorry for this inconvenience. I can understand the frustration. We changed the flow a bit, for making it easy to update the vllm version but there's a model caching feature we are rolling out soon, this should solve it.
Model is indeed downloaded once and it's loaded from disk then, but let me check this as well.
Alternatively, you can try with model baked in a docker image.
avacaondata commentedon Sep 19, 2024
Thanks for the quick reply and for your understanding @pandyamarut :)
Could you please guide us a little bit on best practices for deploying models effectively before that model caching feature release? We would really appreciate that.
If we pre-build the docker container instead of using the web-interface vLLM template you provide, would that help? Or what other things can we do to improve cold-start times?
Thanks again for checking this out, I hope this can be fixed soon so that we can still use Runpod for our production deployment, except for this, our experience with your cloud has been very nice so far.
Yhlong00 commentedon Sep 19, 2024
If you build the model as part of the Docker image, it’ll definitely help with the cold start. Loading the model from the local disk on the host server to GPU vRAM is faster than pulling it from a network volume. You can also extend the idle timeout a bit so the worker doesn’t go to sleep right after finishing a request, which helps avoid cold starts. And of course, keeping an active worker is the best way to prevent cold starts.
ComposerKevin commentedon Mar 7, 2025
Hello is this being addressed?
juancampa commentedon Mar 24, 2025
Also interested in this. Are custom docker images the best current way to speed up cold starts? I saw a PR that enabled model caching (#157) but got reverted soon after (#161)
MiMillieuh commentedon May 20, 2025
The real issue is that it's slow to load (about 2-3 minutes for a 8b model) and the loading time COSTS ! it cost me about 50 cents each time it has to load the model... that's huge... and I'm pretty disapointed
HoMi264 commentedon Jul 29, 2025
Any updates about this I tried network volumes but this will limiting my choice of GPUs too right?
AlexisMDP commentedon Aug 3, 2025
Any update on this?
@pandyamarut why did your PR got reverted?
ashwinsreedhar28 commentedon Oct 8, 2026
Some numbers on this, since the thread has been quiet. I measured Qwen3-8B cold boots on an RTX 4090 Serverless endpoint, network volume attached, FlashBoot off, every sample a full cold boot. From the worker log, "vllm serve" launch to the "vLLM is healthy" line:
As of worker-vllm v2.29.0 (#351), vLLM's torch.compile cache lives on the network volume next to the HF cache, so it's compiled once per GPU type and then loaded on later cold starts, including on hosts that never ran the model before. Same setup after the change, five cache hits across three different hosts:
So roughly 40-60 s off each cold start for an 8B model (measured on Qwen3-8B, same size class as Llama-3.1-8B). What's still on the clock is the weight read from the volume, vLLM's own Triton kernels and FlashInfer JIT, and CUDA graph capture, none of which this release changes. Baking the model into the image still beats the volume for the weight read; the compile cache helps either way. v2.29.0 also bumps vLLM to 0.31.0, so the first cold start after upgrading recompiles once, then it's cached again. Nothing is needed on your side beyond having a volume attached, and the old behavior is available with VLLM_CACHE_ROOT=/root/.cache/vllm.
Two things that bit me on the way, in case they match what people here are seeing. In my testing the volume wasn't mounted into the worker while the endpoint's model-cache option was on, so if your volume "made it worse" or weights kept re-downloading, check that first. And Runpod's delayTime includes waiting for a GPU on Low availability; in my runs that was 70-860 s on top of the boot, which is a different problem from the boot itself.
Raw logs and per-run numbers: #351
EloWeld commentedon Oct 16, 2026
Not Serverless, but the same trade-off on plain pods at qubu.ai: we pause idle pods and wake them on the next request instead of keeping workers warm. The API answers 202 with Retry-After while the pod boots, and the client polls. If you have to scale to zero early on, that contract may be easier to live with than chasing 1–2 s cold starts.