Skip to content

fix(trtllm): honor served model name in Dynamo sidecars - #622

Open
zhongdaor-nv wants to merge 1 commit into
mainfrom
codex/fix-trtllm-sidecar-model-name
Open

zhongdaor-nv wants to merge 1 commit into
mainfrom
codex/fix-trtllm-sidecar-model-name

Conversation

@zhongdaor-nv

Copy link
Copy Markdown
Collaborator

Summary

When a cluster model alias resolves to local weights, TensorRT-LLM loads /model while clients request engine.served_model_name (for example, Qwen/Qwen3-0.6B). The Dynamo sidecar previously registered /model, so requests using the configured served name failed with 404 unknown_model.

Use the existing served-name resolver for the sidecar's --model-path, retaining the local-path and pinned-HF-ID fallbacks when no served name is configured. Add regression coverage for aggregate, prefill, and decode launches and document that this name must resolve to matching tokenizer/config artifacts; arbitrary display-only aliases are unsupported by this sidecar interface.

Verified that the ai-dynamo-runtime==1.6.0.dev20261016 sidecar CLI has no separate served-name option. The upstream registration code inspected at 27541b1 uses model_path as the registration name.

Validation

  • Regression tests failed with the original implementation and passed with the fix.
  • make check: Ruff, formatting, type checking, and generated-doc checks passed; 5,181 tests passed, 3 skipped, 6 deselected, and 11 xfailed, including launch snapshot checks.
  • Ran make snapshots and reviewed the generated launch plans; no snapshot changes are needed for the existing examples.
  • uv run srtctl dry-run -f examples/trtllm/dynamo-sidecar-disagg.yaml passed.
  • No GPU or Slurm cluster execution was performed.

Signed-off-by: Zhongdao Ren <zhongdaor@nvidia.com>
@github-actions

Copy link
Copy Markdown

src/srtctl diff: +3 -3

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant