fix(serving): surface failed model init as UNHEALTHY instead of hanging (#146 Bug 2) - #148
Merged
Merged
Conversation
_initialize_model() runs as an un-awaited asyncio task with a bare
try/finally, so any load failure (bad checkpoint, config, etc.) was
stored on the task and never surfaced: /health stayed
{"status":"starting"} forever instead of flipping to "unhealthy", making
hard init failures look like indefinite hangs (issue #146 Bug 2).
Wrap the init body in `except Exception`: log the traceback and
set_unhealthy(reason) so /health returns 503 {"status":"unhealthy",
"reason": ...} and the failure is visible. CancelledError still
propagates, so shutdown cancellation is unaffected.
Also brings the stale _FakeRuntimeEngine test double in line with the
real ContinuousBatchingEngine signature: it lacked the speculative_draft
kwarg that _initialize_model has passed since the DFlash integration, so
the new except turned its latent TypeError from a collection-time error
into an assertion failure.
Adds a regression test asserting a failed init flips health to
unhealthy with a reason.
Refs: #146
…to_experts transformers 5.15 removed GlmMoeDsaMoE.route_tokens_to_experts, which glm_moe_dsa.py and test_glm_routing.py rely on. CI installs the latest 5.x (requirements pin transformers>=5.3.0,<6), so unit-tests (3.10)/(3.12) fail on dev (and every open PR) with "AttributeError: type object 'GlmMoeDsaMoE' has no attribute route_tokens_to_experts". Skip the GLM routing module when the method is absent (mirrors the existing importorskip guard), and resolve it via getattr so the block raises a clear, actionable error at routing time instead of a cryptic AttributeError at construction. Refs: #146
drunkcoding
pushed a commit
that referenced
this pull request
Aug 11, 2026
_initialize_model passes speculative_draft to ContinuousBatchingEngine (since the DFlash integration), but the watchdog test's _FakeRuntimeEngine mock never accepted it. On transformers 5.15, once the earlier GLM failure stops masking it (pytest runs fail-fast), test_watchdog_integration fails with "unexpected keyword argument 'speculative_draft'". Match the real ContinuousBatchingEngine signature (same fix already in #148). Refs: #146
drunkcoding
pushed a commit
that referenced
this pull request
Aug 11, 2026
_initialize_model passes speculative_draft to ContinuousBatchingEngine (since the DFlash integration), but the watchdog test's _FakeRuntimeEngine mock never accepted it. On transformers 5.15, once the earlier GLM failure stops masking it (pytest runs fail-fast), test_watchdog_integration fails with "unexpected keyword argument 'speculative_draft'". Match the real ContinuousBatchingEngine signature (same fix already in #148). Refs: #146
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #146 Bug 2: a failed async model init left
/healthstuck at{"status":"starting"}forever instead of flipping tounhealthy, so hardinit failures (bad checkpoint, unresolved config, etc.) looked like indefinite
hangs.
_initialize_model()runs as an un-awaitedasynciotask and wrapped itsheavy loads in a bare
try/finallywith noexcept. Any exception wastherefore stored on the task and never surfaced —
set_healthy()was skippedand the health state never left
STARTING.Change
_initialize_model: wrap the init body inexcept Exception→ log thefull traceback (
_logger.exception) and_health_state.set_unhealthy(reason)./healthnow returns503 {"status":"unhealthy","reason": "..."}on initfailure.
CancelledError(aBaseException) still propagates, so shutdowncancellation is unaffected.
test_initialize_model_failure_sets_unhealthy): drivesthe real
_initialize_modelwith a forced tokenizer-load failure and assertshealth flips to
unhealthywith a reason (andengine is None)._FakeRuntimeEnginelacked thespeculative_draftkwarg that
_initialize_modelhas passed since the DFlash integration. Ondevthis raised a latentTypeErrorinside_initialize_model; before thischange it propagated (test errored), and the new
exceptturned it into anassertion failure. The mock is brought in line with the real
ContinuousBatchingEnginesignature.Verification
tests/python/unit— 460 passed, 1 skipped (the one remaining failure,test_examples_smoke, is an environment-onlyMKL_THREADING_LAYERconflicton
import torchin a subprocess — passes withMKL_THREADING_LAYER=GNU, andis unrelated to this change).
tests/python/serving— 140 passed, 1 skipped.ruff format/ruff checkclean; LSP clean.Relationship to #147
Independent of #147 (the multi-GPU device-placement hang, Bug 1). Both target
devand reference #146; this PR is the health-surfacing robustness fix thatalso explains why Bug 1 manifested as a silent hang rather than an error.
Refs: #146