Skip to content

feat(router): route Anthropic Messages through virtual models - #5391

Open
m199369309 wants to merge 9 commits into
xorbitsai:mainfrom
m199369309:feat/router-anthropic-messages
Open

feat(router): route Anthropic Messages through virtual models#5391
m199369309 wants to merge 9 commits into
xorbitsai:mainfrom
m199369309:feat/router-anthropic-messages

Conversation

@m199369309

Copy link
Copy Markdown
Collaborator

Summary

  • route Anthropic Messages requests for Token Router virtual model UIDs through the existing OpenAI-compatible Router data plane
  • reuse the canonical Anthropic request/response/SSE adapter from feat(api): add unified Anthropic Messages protocol adapter #5389
  • convert Router OpenAI errors and streams back into native Anthropic protocol responses
  • keep external Anthropic x-api-key/Bearer credentials at the Supervisor boundary; only a configured internal Router token is forwarded
  • close upstream responses and per-request HTTP clients on success, error, disconnect, and stream completion

Dependencies

This branch intentionally contains both dependency histories so it can be tested before they merge. After the dependencies merge, it will be rebased onto the latest official main so this PR's diff contains only the Anthropic Token Router integration.

Tests

  • pytest -q xinference/api/tests/test_token_router_dispatch.py -k anthropic
  • pytest -q xinference/api/tests/test_anthropic_messages.py xinference/api/tests/test_anthropic_protocol.py
  • pre-commit run --files xinference/api/restful_api.py xinference/api/tests/test_anthropic_messages.py xinference/api/tests/test_token_router_dispatch.py

@XprobeBot XprobeBot added this to the v3.x milestone Aug 19, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an independent, token-aware router process for DeepSeek-V4 on Xinference, adding new control-plane REST endpoints, schemas, and an Anthropic Messages protocol adapter to integrate token-aware routing with the existing RESTful API. Feedback on these changes highlights critical startup issues where asyncio.Condition and asyncio.Lock are instantiated outside of an active event loop, which will cause immediate crashes. Additionally, a performance concern was raised regarding the creation of a new httpx.AsyncClient instance for every request, which should be replaced with a shared client to avoid latency and socket exhaustion.

Comment thread xinference/router/admission.py
Comment thread xinference/router/runtime.py
Comment thread xinference/router/metrics.py
Comment thread xinference/router/tokenization.py
Comment thread xinference/api/restful_api.py Outdated
@m199369309

Copy link
Copy Markdown
Collaborator Author

Additional CI fix included in b6d2d4c:

  • Normalize request header names in _request_credential() so both Starlette Headers and plain mapping-based test requests preserve HTTP case-insensitive header semantics.
  • This fixes the two failing advanced-auth tests for unauthorized and invalid API keys.

Validation:

  • 45 passed across the focused router, tokenization, Anthropic Messages, dispatch, and advanced-auth suites.
  • pre-commit run --files xinference/api/restful_api.py xinference/api/tests/test_token_router_dispatch.py passed all hooks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants