Skip to content

feat(monitoring): add Token Router observability - #5381

Open
m199369309 wants to merge 29 commits into
xorbitsai:mainfrom
m199369309:feat/token-router-monitoring
Open

feat(monitoring): add Token Router observability#5381
m199369309 wants to merge 29 commits into
xorbitsai:mainfrom
m199369309:feat/token-router-monitoring

Conversation

@m199369309

Copy link
Copy Markdown
Collaborator

Summary

  • add Prometheus metrics for Token Router Runtime requests, routing decisions, backend latency/errors, concurrency pools, tokenization, and process resources
  • export Supervisor snapshots for Router Agents, Assignments, Runtime health/config state, Tokenizer Asset Bindings, and logical Router availability
  • add authenticated Prometheus HTTP service discovery for dynamically assigned Router Runtime ports
  • add localized Token Router Grafana dashboards, synchronized alert rules, Alertmanager inhibition examples, and Filebeat Router log inputs
  • expose Token Router monitoring metadata in Monitor Center and document the monitoring setup

Scope

This PR is intentionally limited to Token Router observability. It does not include the separate generic model-replica lifecycle, placement, recovery, or model-load dashboard changes from the local customization history.

Dependencies

Depends on #5375
Depends on #5379

Please review now if convenient, but merge only after the dependency PRs have landed.

Validation

  • pytest focused monitoring/router/API suite: 42 passed
  • npm ci
  • npm run lint (existing repository warnings only)
  • npm run build
  • pre-commit run --files <all changed files>
  • Python compileall for changed runtime, API, metrics, and test modules

A broader combined async test invocation reproduced the repository's existing asyncio teardown hang after its completed tests had passed; the focused changed-code suite above completes successfully.

@XprobeBot XprobeBot added this to the v3.x milestone Aug 18, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive Token Router feature for DeepSeek-V4 in Xinference, including orchestration, scheduling, and deployment management on the Supervisor, along with independent Router Agent and Runtime services. It also adds localized Grafana dashboards, Prometheus alert rules, Filebeat logging configurations, and extensive unit tests. Feedback on the changes highlights a high-severity issue in _proxy_token_router_chat_completion where the asynchronous HTTP client could leak its connection pool if an exception other than httpx.HTTPError (such as asyncio.CancelledError) is raised during client.send before the response stream takes ownership.

Comment thread xinference/api/restful_api.py Outdated
@m199369309
m199369309 force-pushed the feat/token-router-monitoring branch 3 times, most recently from 2a7fa83 to ccf9dff Compare August 18, 2026 18:08
…deepseek-tokenizer-asset

# Conflicts:
#	xinference/core/supervisor.py
…agent-orchestration

# Conflicts:
#	xinference/api/tests/test_token_router_api.py
#	xinference/core/supervisor.py
#	xinference/router/app.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants