[nv-kmcgill53]

Problem

In a Dynamo sidecar job (dynamo.sidecar: true), the engine runs as its own process and a Dynamo sidecar talks to it over gRPC. Two kinds of worker metrics exist:

• The sidecar's runtime metrics: dynamo_component_*, dynamo_work_handler_* and kv_publisher_*. These are served on the sidecar's system status server (DYN_SYSTEM_PORT).
• The engine's native Prometheus metrics: vllm:* and sglang:*. The engine serves these on its own HTTP port. The sidecar does not forward them.

Before this PR, Tachometer and DSight saw only part of this, and some of what they were given was wrong:

• Tachometer scraped DYN_SYSTEM_PORT on every rank. In a cross-node replica, the sidecar runs only on the first node, so the follower targets pointed at ports with nothing listening. No engine endpoint was scraped at all.
• vLLM: the engine's HTTP server listened on 127.0.0.1, so nothing on another node could reach it. When no HTTP port was allocated, the port fell back to one derived from the gRPC port (grpc_port + 1).
• SGLang: sidecar commands omitted --enable-metrics, so SGLang never mounted /metrics. Follower nodes ran with --port 0, so their per-node scheduler metrics were unreachable.
• TRT-LLM: the custom-benchmark AIPERF_SERVER_METRICS_URLS pointed at the OpenEngine gRPC port, which serves no HTTP.
• DSight had no catalog entries for the engine families. It also did not treat the engine's data-parallel labels as ranks, so per-DP-rank engine series were not attributed to ranks.

Changes

Which processes are scraped. Two new Backend members answer this per process, with neutral defaults in backends/base.py:

• runs_dynamo_sidecar(process, endpoint_processes): whether this process starts a sidecar.
• sidecar_engine_metrics_port(process, endpoint_processes): the allocated port the native engine serves Prometheus on, or None.

Each scraped process gets up to two Tachometer targets. Both carry the same worker metadata, so DSight attributes them to one worker:

• backend_<mode><idx>_rank<r>: the sidecar runtime endpoint. Present only where a sidecar runs.
• backend_<mode><idx>_rank<r>_engine: the native engine endpoint. Present only where the engine serves one.

Engine	Sidecar target	Engine target
vLLM	Every node that runs the Rust frontend; headless followers of a cross-node replica run none.	That node's allocated HTTP port. The --host changes from 127.0.0.1 to 0.0.0.0, the grpc_port + 1 fallback is removed, and a missing HTTP port is now an error.
SGLang	The leader, plus any follower that runs a sidecar to relay KV events.	The HTTP port on every node. --enable-metrics is passed unless the role sets it. In sidecar mode, followers now get an allocated HTTP port for their health server's /metrics.
TRT-LLM	MPI rank zero, on the leader.	None: OpenEngine gRPC serves no Prometheus endpoint.

Recipe overrides. srtctl now owns the engine's host and port in sidecar mode. A recipe value for either is dropped and logged, so it cannot move the listener that Tachometer and the sidecar use. SGLang's enable-metrics/enable_metrics override is honoured per role, and the engine target exists only when the rendered command carries the flag.

Metrics URLs given to AIPerf (AIPERF_SERVER_METRICS_URLS):

• Built-in AIPerf runners get exactly the Tachometer worker targets. TRT-LLM keeps its existing publication gate.
• Custom runners keep the existing contract: one URL per logical worker, in the same order as SRT_*_ENDPOINTS, not gated.
  • For vLLM and SGLang, that URL is the leader's engine endpoint, as before.
  • For TRT-LLM, it is now the leader's sidecar system port, because the engine port is gRPC only.

DSight:

• analysis/metric_catalog.py catalogs the vllm:* and sglang:* families into Workers groups, each with a unit and title. The same concept shares a group across engines. kv_publisher_* lands under "Engine KV events".
• dsight/metrics.py reads a data-parallel rank (rank_kind: dp) from the engine's inline dp_rank (SGLang) or engine (vLLM) label. It does this only on worker series, so the router's dp_rank label on frontend series, or an extra_metadata key named engine, does not create ranks.

Docs and example:

• New Metrics section in docs/sidecars.md.
• Updates to docs/observability.md, docs/benchmarks.md, docs/dsight.md and docs/dsight-data-flow.md.
• New example examples/vllm/dynamo-sidecar-agg.yaml.
• Regenerated schema docs. The only change is the dynamo.sidecar_port comment, which no longer says "loopback".

Design-rule baseline. One entry is deleted from tests/test_design_rules.py: the derived grpc_port + 1 port.

Before and after

Tachometer worker targets from the mock orchestrator (src/srtctl/mock.py), on the commit this branch is based on and on this branch:

Recipe	Before	After
vLLM TP8 across 2 nodes	rank0 :7500, rank1 :7501 (headless follower, no sidecar)	rank0 :7500, rank0_engine :6100
TRT-LLM TP8 across 2 nodes	rank0 :7500, rank1 :7501 (no sidecar on rank 1)	rank0 :7500
SGLang TP2/DP2 attention-DP across 2 nodes	rank0 :7500, rank1 :7501	rank0, rank0_engine, rank1, rank1_engine
vLLM TP1 × DP2 on 1 node	rank0 :7500	rank0 :7500, rank0_engine :6100 (vllm:* labelled engine 0 and 1)

Launch snapshot changes

• features__sglang-sidecar-multinode-dp.txt and features__sglang-sidecar-multinode-disagg-dp.txt:
  • every sglang.launch_server command gains --enable-metrics;
  • follower commands change from --port 0 to an allocated --port, which binds on the follower's own node.
• vllm__dynamo-sidecar-agg.txt is new, for the new example. It shows --host 0.0.0.0 --port <allocated HTTP port> --grpc-port <gRPC port>.

Upstream behaviour relied on

Code comments cite the upstream main commits the implementation was read against. The links below show the same code at the released versions validated on hardware.

• vLLM v0.32.0:
  • the gRPC listener follows the HTTP host, so binding HTTP to 0.0.0.0 also binds gRPC there;
  • the Rust frontend serves /metrics on its HTTP router;
  • engine metrics are labelled model_name and engine.
• SGLang v0.5.22:
  • enable_metrics defaults to off, and the HTTP server's Prometheus route depends on it;
  • nodes after the first run a health server on --port that mounts /metrics when metrics are enabled;
  • scheduler series carry a dp_rank label.
• TRT-LLM v1.4.0rc1: --grpc --grpc-protocol openengine replaces the OpenAI server with the OpenEngine gRPC server, which opens only its gRPC port.
• Dynamo: the sidecar creates a DistributedRuntime, whose system status server and /metrics run when DYN_SYSTEM_PORT is set.

Validation

make check: 5141 passed, 3 skipped. It also reports 11 expected failures, all from main's TRT-LLM iteration-log tests in tests/test_dsight_engines.py, which this PR does not touch. New tests:

• tests/test_sidecar_metrics.py: per-engine targets and commands, plus every sidecar example recipe through the mock orchestrator.
• tests/test_sidecar_metrics_adversarial.py, which checks:
  • port uniqueness on colocated shapes;
  • that the custom-runner URL contract holds on every sidecar example;
  • that recipe host/port overrides cannot move the scraped listener;
  • that the SGLang metrics override is per role;
  • IPv6 bracketing;
  • that non-sidecar recipes are unchanged.
• tests/test_dsight_sidecar_metrics.py and tests/test_dsight_sidecar_metrics_adversarial.py: worker attribution, DP ranks only from worker-series labels, and catalog coverage of every engine family.

End to end on GB200 (4 GPUs per node, 1 or 2 nodes per job). Images: vllm/vllm-openai:v0.32.0, lmsysorg/sglang:v0.5.22 and nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc1, with ai-dynamo==1.6.0.dev20261021 and Qwen/Qwen3-0.6B.

Each job ran a custom benchmark that streamed chat requests for 90 s at concurrency 8 and fetched every advertised metrics URL. Each job then checked:

• the job completed;
• the Tachometer targets were exactly the expected set, and every target was scraped with the expected metric families;
• the sidecar request counter equalled the requests sent;
• the custom-runner URLs were one per logical worker, aligned with SRT_*_ENDPOINTS, and answered under load;
• srtctl dsight build attributed the sidecar and engine series to the right worker, with the expected DP ranks.

Case	Result
vLLM TP1 × DP2, 1 node	Pass
SGLang TP1, 1 node	Pass
TRT-LLM TP1, 1 node	Pass
vLLM TP8 across 2 nodes	Pass
SGLang TP2/DP2 attention-DP across 2 nodes	Pass
TRT-LLM TP8 across 2 nodes	Pass

The harness used for these runs is not part of this PR.

Limitations and review notes

• The vLLM engine now listens on every interface (0.0.0.0) instead of loopback. This is needed so Tachometer and AIPerf can reach it from other nodes. It also exposes vLLM's HTTP and gRPC APIs on the job's compute nodes, which SGLang's sidecar commands already did. Draft fix: capture native engine and sidecar metrics with Tachometer #465 covers the same metrics gap with a different choice: it keeps loopback and adds per-node scrapers. The two should be reconciled before either merges.
• TRT-LLM sidecar jobs collect no engine-native metrics. The OpenEngine gRPC server serves none, and the sidecar does not forward its GetLoad RPC. Only the sidecar's runtime metrics are scraped.
• Disaggregated sidecar topologies, and vLLM hybrid DP across nodes, are covered by the mock orchestrator and launch snapshots only, not by the hardware runs above.

🤖 Generated with Claude Code

[github-actions]

src/srtctl diff: +450 -87