Skip to content

fix: answer jobs when a model's own vLLM code rejects the setup - #363

Open
ashwinsreedhar28 wants to merge 4 commits into
runpod-workers:mainfrom
ashwinsreedhar28:fix/model-code-startup-error
Open

ashwinsreedhar28 wants to merge 4 commits into
runpod-workers:mainfrom
ashwinsreedhar28:fix/model-code-startup-error

Conversation

@ashwinsreedhar28

Copy link
Copy Markdown
Contributor

A model's own vLLM implementation can refuse a configuration outright, and then it fails the same way on every start. The worker doesn't recognise those errors, so it exits, gets restarted into the same failure, and submitted jobs receive no result. RecursiveSI/flint does this twice with vLLM's day-0 image (vllm/vllm-openai:flint, vLLM 0.32.0rc1.dev348) on Hopper:

  • With default settings vLLM picks FLASH_ATTN, and flint's model code raises AssertionError: Flint requires FlashInfer attention while building the first layer. On an H200 the worker failed that way 5 times, about every 32 s (log).
  • With ATTENTION_BACKEND=FLASHINFER, warmup fails with RuntimeError: Flint BF16 decode MoE requires the FlashInfer TRTLLM BF16 MoE backend without padding (got UnquantizedMoeBackend.TRITON); flint's own sglang/serve_h200.sh notes the TRT-LLM MoE kernels are Blackwell-only. The worker exited 5 times in 10.6 minutes (log). The same image and setting serve flint on a B200 (result).

This adds one narrow rule to startup_errors.classify, checked after every existing match. It applies when an AssertionError, RuntimeError, NotImplementedError or ValueError (not logged as a handled DEBUG/INFO/WARNING record) has model code as its innermost frame (vllm/model_executor/models/, or the per-model packages under vllm/models/, which v0.32.0 already uses for models such as DeepSeek V4, Kimi K3 and MiniMax M3) and its message states a requirement (require(s), only support(s)/only supported, not supported on/with/for/by). The job is then answered with that exact message instead of the API server's "Engine core initialization failed" wrapper. Where an error is raised isn't enough on its own, since a driver fault or a dropped download inside load_weights also ends in model code. So these still restart:

  • any other model-code error, including autograd's "requires grad" errors;
  • any failed start whose output shows a CUDA, NCCL, NVML, cuBLAS/cuDNN, ECC or Xid fault, including a CUDA cause wrapped by a model-code exception;
  • a matched exception whose chain shows a transport fault (connection reset, timeout, or HTTP status text such as Too Many Requests or Service Unavailable);
  • a match followed by a different exception raised outside model code (other than vLLM's "Engine core initialization failed" and "WorkerProc initialization failed" wrappers, or a repeat of the same cause), since then the match isn't what ended the launch;
  • an engine killed by a signal.

A requirement can depend on the GPU type: on an endpoint listing both H200 and B200, an H200 worker now answers jobs with the cause instead of crash-looping while a B200 worker serves them. The out-of-memory handling already behaves this way; the message says to remove GPU types the model can't use.

Evidence:

check result
python -m pytest tests 179 pass (144 on main + 35 new in tests/test_startup_errors.py). The flint fixtures follow the real log order (logged traceback, Process EngineCore:, raw traceback, API server tail) and assert the exact quoted cause. Cases that must still restart: CUDA driver error, NVML internal assert, connection reset and timeout inside load_weights; illegal memory access in model code; a wrapped CUDA cause; a CUDA error on a continuation line; no stated requirement; "requires grad"; a requirement raised outside model code; the next log record, or a line printed after a raw traceback, supplying the word "requires"; a traceback logged at WARNING; an unrelated exception after the match; a transport fault (connection reset, HTTP 503) chained under the match; an engine killed by SIGKILL. Harmless NVML, pynvml, HF retry and c10d connect warnings, and vLLM's "NVLink detection failed" record, elsewhere in the launch don't block a match.
classify() on real worker logs: the last 400 lines of vLLM's output before each exit both full flint H200 logs (all 5 launches each) quote AssertionError: Flint requires FlashInfer attention and the TRTLLM MoE RuntimeError exactly; the K2-Horizon logs (v2.27.2, v2.28.0) still get the TRUST_REMOTE_CODE hint, and a 48 GB out-of-memory log still gets the memory advice
image with this change (FROM vllm/vllm-openai:flint; this branch's head a17e9aa plus #360's version-guard commit, which a day-0 base tag needs; build) on Serverless, RecursiveSI/flint, H100, default settings one vLLM launch; job FAILED with startup_error 9.5 min after endpoint creation (about 8.7 min elapsed before vLLM launched): "RecursiveSI/flint failed inside vLLM's implementation of the model during startup: AssertionError: Flint requires FlashInfer attention. The model's code requires something this worker's GPU or settings don't provide, so restarting this worker will not change it. …" (result, log)

On a 48 GB Ada/Ampere card vLLM picks FlashInfer for flint by itself and the model simply runs out of memory, which the existing OOM handling already answers; the evidence run is on an H100 for that reason.

The image-build and Serverless jobs need upstream secrets to complete on a fork PR; happy for them to be mirrored like #352.

Study, build logs and results: https://github.com/ashwinsreedhar28/day0-ready

A model's vLLM implementation can refuse a configuration outright, and it
fails the same way on every start. RecursiveSI/flint on an H200 (vLLM
0.32.0rc1 day-0 image) did it twice: "AssertionError: Flint requires
FlashInfer attention" with the default backend, then "RuntimeError: Flint
BF16 decode MoE requires the FlashInfer TRTLLM BF16 MoE backend" with
ATTENTION_BACKEND=FLASHINFER (a Blackwell-only kernel). Both exited and
restarted into the same failure, so jobs got no answer.

classify() now treats an AssertionError, RuntimeError, NotImplementedError,
ValueError or TypeError whose innermost frame is in model code
(vllm/model_executor/models/ or vllm/models/) as a startup_error, quoting
the root cause rather than the API server's wrapper. CUDA/NCCL errors and
anything raised outside model code still exit for a platform restart.
Checked last, so every specific match keeps priority.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kxoD6sb3tVrig5ddmB8f1
…fault

Review found the location rule too broad: a CUDA driver fault, an NVML
internal assert or a dropped connection inside load_weights also ends in
model code, and would have been answered forever. Now a model-code
exception counts only when its message (all lines) states a requirement
("requires", "only supports", "not supported on ..."), and any CUDA, NCCL,
NVML, cuBLAS/cuDNN, ECC, Xid or connection/timeout error anywhere in the
failed launch's output sends it back to the platform restart, including a
CUDA cause wrapped by a model-code ValueError. Both flint logs, the H100
log and the K2 logs classify as before. 161 tests (17 new).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kxoD6sb3tVrig5ddmB8f1
…ult match

Second review found the multi-line joining appended the next log record
("Process EngineCore:") to the quoted cause on every real flint log, and
could borrow a requirement word from an unrelated INFO line. Continuation
lines now count only within the same log record (same prefix).

Also from the review:
- host faults are matched by fault signatures (CUDA/NCCL/NVML errors,
  cuBLAS/cuDNN status, ECC, Xid, driver too old, collective timeout), not
  any NVML/timeout/connection wording, so torch's NVML warnings or a retried
  download elsewhere in the launch don't send flint back to restarting;
  transport wording only vetoes when it is in the quoted error itself
- "requires grad" autograd errors don't count as requirements
- tracebacks logged below ERROR, and engines killed by a signal, don't count
- a model-code frame no longer attaches to a later frameless error
- vLLM 0.30's bare "(APIServer pid=N) " raw-output prefix is stripped
- the message names this worker's GPU or settings and says what to do on
  endpoints with several GPU types; README lists the case with the others

Fixtures now follow the real log order (prefixed traceback, "Process
EngineCore:", raw traceback, API server tail) and assert the exact quoted
cause. All 5 launches of both full H200 logs, the H100 log, the 48 GB OOM
log and both K2 logs classify as intended. 173 tests (29 new).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kxoD6sb3tVrig5ddmB8f1
…aunch

From the re-review:
- multi-line messages join only inside a timestamped log record; raw
  tracebacks (no record boundary) never take the next line, e.g. torch's
  destroy_process_group() warning at exit
- a different exception after the match (other than vLLM's "Engine core /
  WorkerProc initialization failed" wrappers or a repeat of the cause)
  cancels it: the match wasn't what ended the launch
- transport wording (now also 429/5xx status text) is checked over the
  matched exception's whole chain, not just its message
- vLLM's "NVLink detection failed. This is normal ..." record, logged with
  an NVMLError traceback on PCIe hosts, isn't read as a host fault
- the bare raw-output prefix is anchored to "(Name pid=N) ", so a message
  line starting with a parenthesis isn't stripped
- message: the ATTENTION_BACKEND advice no longer reads as conditional

All 5 launches of both full flint H200 logs, the H100 log, the 48 GB OOM
log and the K2 logs classify as before. 179 tests (35 new).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kxoD6sb3tVrig5ddmB8f1
@velaraptor-runpod
velaraptor-runpod self-requested a review October 22, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants