Startup OOM: eager relaunch costs 2x on an 8B model, MAX_NUM_SEQS=32 avoids most of it, 'Cannot auto-fit' exits 1 #366

I measured the startup-OOM relaunch from #346 on v2.30.1 (71d503e6, vLLM v0.32.0) to see what it costs when it fires. Three things came out of it: the eager relaunch halves throughput on a small model with nothing in the responses, the Hub default MAX_NUM_SEQS=256 costs boot time and context without a throughput gain at the worker's concurrency, and Cannot auto-fit max_model_len isn't matched by memory_shortfall(), so that case still ends in sys.exit(1) and a restart on the same footprint, the loop #346 set out to remove.

Setup. Not a deployed endpoint: vllm/vllm-openai:v0.32.0 (the worker's base image) running the args build_vllm_args() builds from the Hub form defaults (MAX_MODEL_LEN=auto, GPU_MEMORY_UTILIZATION=0.95, MAX_NUM_SEQS=256). For each failed boot, the last 400 log lines went through the worker's own startup_errors.py with main.py's rules to get what the worker would do. Benchmark: vllm bench serve, random 1024 in / 512 out, at 1, 8 and 30 concurrent (MAX_CONCURRENCY). Two cards, neither in the Hub's GPU list, so the numbers are from these cards only:

• RTX 5090 (32 GB), deepseek-ai/DeepSeek-R1-Distill-Llama-8B (the Hub preset)
• RTX PRO 6000 Blackwell (96 GB), RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic

Output tok/s:

1. The relaunch's cost depends on the model. Eager is 2.0x slower at one request on the 8B model (1.5x at 30) and 0-3% slower on the 70B. Nothing in the responses says it happened, so an 8B endpoint that lands on this path runs at half speed with no signal outside the logs.
2. MAX_NUM_SEQS=256 costs boot time and context. Capturing graphs for 256 sequences took 158 s of the 5090 boot (75 + 83 s) and 101 s of the PRO 6000 boot; with 32 sequences it took 2-3 s. MAX_NUM_SEQS=32 is still above MAX_CONCURRENCY=30, matched the default's throughput at every concurrency tested, booted in 80 s instead of 251 s and 90 s instead of 210 s, and fit 3.3K (5090) and 2.3K (PRO 6000) more tokens of context under auto. So with it as the default, fewer deploys reach the relaunch at all, and every cold start is 2-3 min shorter.
3. Most of the relaunch band can keep graphs. At an explicit MAX_MODEL_LEN between what the default and MAX_NUM_SEQS=32 fit (108,440 on the 5090, 63,680 on the PRO 6000), the default failed with the KV shortfall memory_shortfall() matches, and both the eager relaunch and MAX_NUM_SEQS=32 with graphs served it. Above the MAX_NUM_SEQS=32 fit, only eager helps: it frees the whole graph estimate and fits about 1K (5090) and 2.2K (PRO 6000) tokens more.
4. Cannot auto-fit exits 1. When the weights leave no room for KV under MAX_MODEL_LEN=auto (forced here with GPU_MEMORY_UTILIZATION=0.45, standing in for a model that fills the card), vLLM v0.32.0 raises Cannot auto-fit max_model_len: not enough GPU memory available to serve even a single token (vllm/v1/core/kv_cache_utils.py:2453-2457). Under auto the auto-fit runs before the capacity check that prints No available memory for the cache blocks, so a negative KV budget ends in this message too. Neither memory_shortfall() nor classify() matches it, and main.py:305-311 exits 1. I didn't deploy an endpoint, so the restart on the same config is from the code, not observed.

Suggested changes:

• Default MAX_NUM_SEQS in .runpod/hub.json to 32, or tie it to MAX_CONCURRENCY.
• If 256 stays the default, the OOM relaunch could first try MAX_NUM_SEQS at MAX_CONCURRENCY with graphs on and keep eager as the last step. That adds a launch against Auto-retry startup OOM once with a reduced memory footprint #346's bound of three, so it's a tradeoff.
• Match Cannot auto-fit max_model_len in memory_shortfall(), or at least in classify(), so jobs get the memory advice instead of an exit 1.

I can send a PR for the first and third if that helps. The kits, with the exact args per variant and the results: https://github.com/adi-suresh01/sm120-kit/tree/main/issues/runpod-worker-vllm-r1d8b and https://github.com/adi-suresh01/sm120-kit/tree/main/issues/runpod-worker-vllm-l70b