Repository navigation
fix: answer jobs when a model's own vLLM code rejects the setup - #363
Open
ashwinsreedhar28 wants to merge 4 commits into
Open
ashwinsreedhar28 wants to merge 4 commits into
ashwinsreedhar28 wants to merge 4 commits into
Conversation
A model's vLLM implementation can refuse a configuration outright, and it fails the same way on every start. RecursiveSI/flint on an H200 (vLLM 0.32.0rc1 day-0 image) did it twice: "AssertionError: Flint requires FlashInfer attention" with the default backend, then "RuntimeError: Flint BF16 decode MoE requires the FlashInfer TRTLLM BF16 MoE backend" with ATTENTION_BACKEND=FLASHINFER (a Blackwell-only kernel). Both exited and restarted into the same failure, so jobs got no answer. classify() now treats an AssertionError, RuntimeError, NotImplementedError, ValueError or TypeError whose innermost frame is in model code (vllm/model_executor/models/ or vllm/models/) as a startup_error, quoting the root cause rather than the API server's wrapper. CUDA/NCCL errors and anything raised outside model code still exit for a platform restart. Checked last, so every specific match keeps priority. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016kxoD6sb3tVrig5ddmB8f1
…fault
Review found the location rule too broad: a CUDA driver fault, an NVML
internal assert or a dropped connection inside load_weights also ends in
model code, and would have been answered forever. Now a model-code
exception counts only when its message (all lines) states a requirement
("requires", "only supports", "not supported on ..."), and any CUDA, NCCL,
NVML, cuBLAS/cuDNN, ECC, Xid or connection/timeout error anywhere in the
failed launch's output sends it back to the platform restart, including a
CUDA cause wrapped by a model-code ValueError. Both flint logs, the H100
log and the K2 logs classify as before. 161 tests (17 new).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kxoD6sb3tVrig5ddmB8f1
…ult match
Second review found the multi-line joining appended the next log record
("Process EngineCore:") to the quoted cause on every real flint log, and
could borrow a requirement word from an unrelated INFO line. Continuation
lines now count only within the same log record (same prefix).
Also from the review:
- host faults are matched by fault signatures (CUDA/NCCL/NVML errors,
cuBLAS/cuDNN status, ECC, Xid, driver too old, collective timeout), not
any NVML/timeout/connection wording, so torch's NVML warnings or a retried
download elsewhere in the launch don't send flint back to restarting;
transport wording only vetoes when it is in the quoted error itself
- "requires grad" autograd errors don't count as requirements
- tracebacks logged below ERROR, and engines killed by a signal, don't count
- a model-code frame no longer attaches to a later frameless error
- vLLM 0.30's bare "(APIServer pid=N) " raw-output prefix is stripped
- the message names this worker's GPU or settings and says what to do on
endpoints with several GPU types; README lists the case with the others
Fixtures now follow the real log order (prefixed traceback, "Process
EngineCore:", raw traceback, API server tail) and assert the exact quoted
cause. All 5 launches of both full H200 logs, the H100 log, the 48 GB OOM
log and both K2 logs classify as intended. 173 tests (29 new).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016kxoD6sb3tVrig5ddmB8f1
…aunch From the re-review: - multi-line messages join only inside a timestamped log record; raw tracebacks (no record boundary) never take the next line, e.g. torch's destroy_process_group() warning at exit - a different exception after the match (other than vLLM's "Engine core / WorkerProc initialization failed" wrappers or a repeat of the cause) cancels it: the match wasn't what ended the launch - transport wording (now also 429/5xx status text) is checked over the matched exception's whole chain, not just its message - vLLM's "NVLink detection failed. This is normal ..." record, logged with an NVMLError traceback on PCIe hosts, isn't read as a host fault - the bare raw-output prefix is anchored to "(Name pid=N) ", so a message line starting with a parenthesis isn't stripped - message: the ATTENTION_BACKEND advice no longer reads as conditional All 5 launches of both full flint H200 logs, the H100 log, the 48 GB OOM log and the K2 logs classify as before. 179 tests (35 new). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016kxoD6sb3tVrig5ddmB8f1
velaraptor-runpod
self-requested a review
October 22, 2026 06:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A model's own vLLM implementation can refuse a configuration outright, and then it fails the same way on every start. The worker doesn't recognise those errors, so it exits, gets restarted into the same failure, and submitted jobs receive no result. RecursiveSI/flint does this twice with vLLM's day-0 image (
vllm/vllm-openai:flint, vLLM 0.32.0rc1.dev348) on Hopper:FLASH_ATTN, and flint's model code raisesAssertionError: Flint requires FlashInfer attentionwhile building the first layer. On an H200 the worker failed that way 5 times, about every 32 s (log).ATTENTION_BACKEND=FLASHINFER, warmup fails withRuntimeError: Flint BF16 decode MoE requires the FlashInfer TRTLLM BF16 MoE backend without padding (got UnquantizedMoeBackend.TRITON); flint's ownsglang/serve_h200.shnotes the TRT-LLM MoE kernels are Blackwell-only. The worker exited 5 times in 10.6 minutes (log). The same image and setting serve flint on a B200 (result).This adds one narrow rule to
startup_errors.classify, checked after every existing match. It applies when anAssertionError,RuntimeError,NotImplementedErrororValueError(not logged as a handled DEBUG/INFO/WARNING record) has model code as its innermost frame (vllm/model_executor/models/, or the per-model packages undervllm/models/, which v0.32.0 already uses for models such as DeepSeek V4, Kimi K3 and MiniMax M3) and its message states a requirement (require(s),only support(s)/only supported,not supported on/with/for/by). The job is then answered with that exact message instead of the API server's "Engine core initialization failed" wrapper. Where an error is raised isn't enough on its own, since a driver fault or a dropped download insideload_weightsalso ends in model code. So these still restart:A requirement can depend on the GPU type: on an endpoint listing both H200 and B200, an H200 worker now answers jobs with the cause instead of crash-looping while a B200 worker serves them. The out-of-memory handling already behaves this way; the message says to remove GPU types the model can't use.
Evidence:
python -m pytest teststests/test_startup_errors.py). The flint fixtures follow the real log order (logged traceback,Process EngineCore:, raw traceback, API server tail) and assert the exact quoted cause. Cases that must still restart: CUDA driver error, NVML internal assert, connection reset and timeout insideload_weights; illegal memory access in model code; a wrapped CUDA cause; a CUDA error on a continuation line; no stated requirement; "requires grad"; a requirement raised outside model code; the next log record, or a line printed after a raw traceback, supplying the word "requires"; a traceback logged at WARNING; an unrelated exception after the match; a transport fault (connection reset, HTTP 503) chained under the match; an engine killed by SIGKILL. Harmless NVML, pynvml, HF retry and c10d connect warnings, and vLLM's "NVLink detection failed" record, elsewhere in the launch don't block a match.classify()on real worker logs: the last 400 lines of vLLM's output before each exitAssertionError: Flint requires FlashInfer attentionand the TRTLLM MoERuntimeErrorexactly; the K2-Horizon logs (v2.27.2, v2.28.0) still get theTRUST_REMOTE_CODEhint, and a 48 GB out-of-memory log still gets the memory adviceFROM vllm/vllm-openai:flint; this branch's head a17e9aa plus #360's version-guard commit, which a day-0 base tag needs; build) on Serverless, RecursiveSI/flint, H100, default settingsstartup_error9.5 min after endpoint creation (about 8.7 min elapsed before vLLM launched): "RecursiveSI/flint failed inside vLLM's implementation of the model during startup: AssertionError: Flint requires FlashInfer attention. The model's code requires something this worker's GPU or settings don't provide, so restarting this worker will not change it. …" (result, log)On a 48 GB Ada/Ampere card vLLM picks FlashInfer for flint by itself and the model simply runs out of memory, which the existing OOM handling already answers; the evidence run is on an H100 for that reason.
The image-build and Serverless jobs need upstream secrets to complete on a fork PR; happy for them to be mirrored like #352.
Study, build logs and results: https://github.com/ashwinsreedhar28/day0-ready