feat(router): add tokenizer asset registry

[m199369309]

Summary

• add a managed Tokenizer asset registry for Token-aware Router deployments
• validate manifests, required files, checksums, path traversal and symlink safety
• support external asset roots through explicit configuration and isolated smoke tests
• add focused registry tests

Depends on #5375

Validation

• pytest -q xinference/core/tests/test_tokenizer_asset_registry.py
• pre-commit run --files xinference/constants.py xinference/core/tokenizer_asset_registry.py xinference/core/tests/test_tokenizer_asset_registry.py

Please review now if convenient, but merge only after the dependency PRs have landed.

[gemini-code-assist]

Code Review

This pull request introduces an independent, token-aware router service for DeepSeek-V4 on Xinference, adding a persistent SQLite configuration store, an in-memory runtime registry, a dedicated router process, and REST endpoints to proxy chat completions based on token budgets or typed routing rules. The review feedback highlights several important improvements: ensuring compatibility with Python 3.9 and below by catching asyncio.TimeoutError instead of the built-in TimeoutError in control_plane.py and admission.py, reusing a single httpx.AsyncClient instance in restful_api.py to avoid socket exhaustion, and optimizing performance in tokenizer.py by replacing expensive copy.deepcopy calls with shallow copies.

[qinxuye]

Three findings from the current-head review:

[qinxuye]

[P1] Do not execute asset Python with Supervisor privileges

This calls DeepSeekV4TokenEstimator, which imports and executes the asset's encoding/encoding_dsv4.py. A spawned daemon process is not a security sandbox: it still inherits the Supervisor's environment, filesystem identity, and network access. Unlike the Router tokenization worker, this child does not even remove the API/internal tokens, and the checksum is declared by the same asset manifest rather than an independent trust root. A compromised asset can therefore read control-plane secrets or modify Supervisor-owned state during create/validate. Please run this in a genuinely least-privilege sandbox (or remove executable asset code from Supervisor validation), with secrets, filesystem/network access, and the trust anchor isolated from the asset.

[m199369309]

Fixed in 462cd8e. The Supervisor no longer executes any asset Python.

• Removed _execute_smoke_test / _smoke_test_worker / _smoke_test and _SMOKE_TEST_TIMEOUT_SECONDS from xinference/core/tokenizer_asset_registry.py. DeepSeekV4TokenEstimator is never instantiated in the Supervisor or any child process it spawns.
• Validation is now purely static: manifest structure/field checks, SHA-256 checksum verification, path-safety checks (absolute paths, .. traversal, escape from the asset directory), and file hashing for the fingerprint.
• validate_path() and _inspect_entry() keep the smoke_test parameter only for API compatibility; it is unused.
• Tests updated to assert the static results (required_files, checksums, manifest, capabilities) instead of smoke-test checks, including test_registered_asset_list_resolve_and_validate and test_declared_capabilities_are_reported.

[qinxuye]

[P1] Report the asset that was actually loaded

These values are copied from the control-plane config, and validate_token_router() compares them with that same stored config. The Router never hashes the files loaded by TokenizationService, so a different asset at the same path (for example on another host/mount, or after a TOCTOU replacement) still reports the expected fingerprint and passes validation. Please compute the fingerprint/revision from the files actually loaded by the Router/tokenization worker and publish that measured value in the heartbeat; add a test where the configured fingerprint differs from the local asset contents.

[m199369309]

Fixed in 462cd8e. The Router now measures the asset it actually loaded and publishes the measured values.

• New xinference/router/tokenizer_asset.py: compute_tokenizer_asset_fingerprint() hashes the files the worker uses (tokenizer.json, encoding/encoding_dsv4.py) with a path-safety check, and read_tokenizer_asset_revision() reads asset.json revision statically (JSON only, no code execution). The same aggregation function is shared with the registry so both sides compute identical fingerprints.
• initialize_tokenization_worker() now computes the fingerprint and revision before and after constructing DeepSeekV4TokenEstimator; if they differ (TOCTOU replacement during load) initialization raises. It also scrubs all three Router credential env vars.
• ping_worker() returns the measured fingerprint/revision; TokenizationService.start() verifies every worker loaded the same asset (single non-empty fingerprint and single revision) and exposes asset_fingerprint / asset_revision.
• The heartbeat process.tokenizer_asset and RouterRuntime.summary() now report the measured values from the snapshot instead of copying the control-plane config, and Supervisor validation compares against the measured values.
• Regression coverage: test_start_measures_tokenizer_asset_fingerprint and test_validation_detects_loaded_fingerprint_mismatch (configured fingerprint differs from local asset contents).

[qinxuye]

[P2] Enforce the declared asset capabilities at runtime

tools and thinking currently only decide which smoke tests run. resolve() drops the capability map, RouterConfig has no corresponding field, and the request path still accepts tool/thinking payloads unconditionally. An asset declaring tools: false or thinking: false can therefore be deployed and used for an unsupported request, causing a tokenization failure or incorrect budget/routing. Please propagate the validated capabilities into the Router config and reject unsupported requests/configured rules, with regression coverage for both false flags.

[m199369309]

Fixed in 462cd8e. Validated capabilities now flow into the Router config and are enforced on both the request path and rule validation.

• resolve() returns tokenizer_asset_capabilities and tokenizer_asset_files; RouterConfig gains both fields and config_from_control_plane() reads them (manifest may declare capabilities as a dict or a list).
• Request path in xinference/router/app.py: tool requests are pre-checked before tokenization (400 tools_not_allowed), and thinking-mode requests are rejected after budget computation (400 thinking_not_allowed).
• Rule validation rejects rules that require tools/thinking when the asset does not support them, in both _validate_config() and Supervisor v2 validation in validate_token_router().
• Custom paths keep the phase-1 behavior (full capabilities + default required files) for backward compatibility.
• Regression coverage: test_tools_request_rejected_when_asset_lacks_tools_capability, test_thinking_request_rejected_when_asset_lacks_thinking_capability, v2 rule-rejection cases, and test_v2_validation_rejects_rules_asset_cannot_support.

[qinxuye]

Two remaining findings after reviewing the fixes in 462cd8e:

[qinxuye]

[P1] Fail closed when the measured asset differs from the configuration

This only proves that the tokenization workers agree with one another. TokenizationService is never given the expected fingerprint/revision, so if every worker on this host loads the same wrong asset, startup succeeds, the runtime ACKs the revision and reports ready. resolve_token_router_runtime() also selects ready/ACKed instances without checking the measured heartbeat metadata; the manual validation endpoint therefore does not prevent traffic from reaching the mismatched tokenizer. I reproduced a configured sha256:expected plus heartbeat sha256:swapped resolving as available=True. Please compare the measured values with RouterConfig before the snapshot becomes ready (or exclude mismatched heartbeats in Supervisor runtime selection), and add a regression proving such an instance cannot be resolved/routed.

[qinxuye]

[P2] Reject unsupported thinking before invoking the asset tokenizer

This capability check runs only after estimate(). An asset declaring thinking: false may fail while rendering the thinking payload, so control never reaches this branch and the client receives a generic invalid_request_error instead of thinking_not_allowed. The new _payload_thinking() helper is currently unused. Please perform equivalent normalized detection before tokenization (including the existing top-level / extra_body / JSON-string chat_template_kwargs precedence), and cover the case where the estimator would raise if it were invoked.

[m199369309]

CI failure: click 8.4 compatibility for cmdline tests (fixed in eb3a4d6)

The failing xinference/deploy/test/test_cmdline.py tests are not caused by this PR — the same two failures appeared on earlier commits (e.g. 140df640, run 32388623931) before the recent review fixes, and the file is untouched by the feature work.

Root cause: dependency drift. pyproject.toml pins click<8.2.0, but the CI optional deps (diffusers → newly released huggingface-hub>=1.23) resolve click to 8.4.2. Pip installs it and warns xinference ... requires click<8.2.0, but you have click 8.4.2 which is incompatible. click 8.4 removed CliRunner(mix_stderr=...) (replaced by capture=sys) and captures stdout/stderr separately by default.

Two failing tests:

• test_cmdline_model_path_error: CliRunner(mix_stderr=False) raises TypeError on click 8.4.
• test_cmdline_of_custom_model: list_model_registrations writes its table to stderr (print(..., file=sys.stderr)); click 8.1 default mix_stderr=True mixed stderr into result.stdout, click 8.4 does not.

Fix (eb3a4d6): make the assertions version-agnostic without changing CLI behavior:

• CliRunner(mix_stderr=False) → CliRunner() (the test only inspects exit_code/exc_info).
• Assert on result.output instead of result.stdout — result.output is the combined stream on both click <8.2 (mix_stderr=True default) and click >=8.4 (capture=sys).

Verified with minimal repros on both click 8.1.8 and 8.4.2; pre-commit passes and the fast unit test in the file passes. New CI run (32428725269) is in progress on eb3a4d6.

[qinxuye]

One additional finding in the shutdown follow-up:

[qinxuye]

[P1] Clean up a replacement snapshot when shutdown cancels apply()

This cancellation can land while control_plane.run() is awaiting runtime.apply(). RouterRuntime.apply() only catches Exception around replacement.tokenization.start(), but asyncio.CancelledError is a BaseException, so the replacement snapshot is abandoned without closing its HTTP client, tokenization service, or process pool; the subsequent runtime.aclose() only closes the current snapshot. I reproduced cancellation during replacement startup with all three replacement resources still open, which can make the shutdown hang on orphaned worker processes. Please make the pre-swap portion of apply() cancellation-safe (clean up on CancelledError/BaseException, then re-raise; use bounded/cancellation-safe executor cleanup as needed), and add a regression that blocks replacement startup, cancels apply(), and asserts the replacement snapshot is closed.