feat(service-pools): add priority failover and AI routing

[ctkm-aelf]

Summary

Closes #1680.

A service pool currently selects one connection before dispatch, so an upstream 429 or transport failure reaches the caller even when another usable connection exists. This adds priority pools that keep one stable route, balance within each priority tier, and try eligible backups under a bounded failover policy. For example, a platform connection can serve first and an authorized BYOK connection can serve after a retryable rejection.

• Extend the existing generic ServicePool with member priorities, independent round-robin/weighted tier balancing, bounded attempts and deadlines, replay limits, and durable operation-specific cooldown. Existing round-robin/weighted pools retain single-attempt behavior. Routing has no Oracle dependency.
• Support same_api compatibility and ai_chat translation between supported OpenAI Completions, Responses, and Anthropic Messages providers, with explicit member models and the gateway alias pool:<slug>. Preserve configured destinations and apply live ownership, scopes, grants, operation policy, approvals, and node capability checks before member execution.
• Expose configuration, candidate suitability, health/reset, and atomic revision-checked updates through REST, the AI Services dashboard, and nyxid pool. Add per-attempt audit/response attribution, exact member accounting, durable settlement/recovery, and cancellation support for node-routed attempts.

Behavior and CLI

The issue's CLI sketch is supported:

nyxid pool create --slug my-llm --name "My LLM" --strategy priority
nyxid pool add-member my-llm --service platform-chat --priority 0
nyxid pool add-member my-llm --service my-chat-key --priority 10
nyxid pool set-failover my-llm --retry-on 429,5xx,timeout,node_offline --max-attempts 3
nyxid pool set-strategy my-llm priority --tier-balance weighted
nyxid pool health my-llm --method POST --path /chat/completions

Member examples assume existing compatible connections owned by the pool's person or organization. Platform access uses an ordinary same-owner platform-bound UserService with live grants. ai_chat members additionally specify a model. Candidate inspection reports unsuitable members and required compatibility declarations or node upgrades.

Inline policy flags merge only supplied fields, including nested cooldown settings. JSON files support full configuration and atomic membership/contract changes. Policy updates read one pool ID/revision snapshot and send one PUT; conflicts do not silently retry. 5xx expands to the supported 500/502/503/504/529 triggers.

Ambiguous unsafe-method replay requires explicit --retry-ambiguous-dispatch opt-in. Configuring 5xx/timeout triggers alone does not enable it. Provider switching stops after the first nonempty client-visible output. Caller cancellation and billing lease loss cancel work without penalizing a healthy upstream. Known usage is settled for each attempted provider; unknown token usage remains unknown. Local authorization, approval, rate-limit, and billing denials are terminal.

See the service pool guide for complete configuration and operating behavior, and the routing proof for authorization and routing invariants.

Test Plan

• Backend follow-up verification: 737 distinct cases with passing latest results across affected runs and targeted reruns, using a real MongoDB 8 replica set. This is a deduplicated union, not a fresh full-repository suite. Covers retry/cooldown, streaming commitment, caller/lease cancellation, authorization, exact funding and recovery, locked-health EOF, rejected-body failures, node dispatch evidence, provider translation, custom destinations, encoding, and signed node/Codex transport.
• CLI: 23/23 pool tests passed, including Clap-to-command-to-mock-HTTP execution of the enhancement sketch, aliases, nested policy preservation, organization selection, concrete-ID/revision updates, and conflicts. cargo build -p nyxid-cli passed.
• Frontend follow-up: 11/11 affected component/hook tests passed; TypeScript and changed-schema ESLint/Prettier passed.
• cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, git diff --check, and scripts/check-rci-backend-boundary.sh passed on the final implementation snapshot.
• Earlier implementation checkpoint: full frontend 3,948/3,948, selected backend/CLI/adapter suite 1,608/1,608, KMS feature/release builds, embedded-assets guard, wizard freshness, and standalone billing smoke passed. These precede the final follow-up changes and are separate from the current affected verification above.
• Primary personally reviewed the implementation and corrections; all confirmed findings from the independent gap review were resolved and verified. All 81 committed file contents match the accepted final source manifest.

CI will validate the PR against the current base. The branch merges cleanly with main at 82dcf3bf; the local verification above was performed on the implementation branch based on e96a5078.

Rollout and rollback

Upgrade all backend replicas and management clients before creating priority pools. Node-routed priority members require an upgraded, reconnected node advertising http_cancellation; inspection explains when this is missing. Before downgrading, convert or delete every priority pool, including disabled pools. These requirements and conversion examples are documented in the guide. No Oracle migration is needed.

Checklist

• Code follows the project's architecture rules.
• No hardcoded production secrets or credentials.
• Errors and attempt summaries do not expose credentials or internal destination details.
• Documentation and service-pool skill reference updated.

[github-actions]

📊 Code coverage

Component	Lines	Threshold	Status	Δ vs base
Backend (nyxid)	87.57%	73%	✅	🔺 +0.13
CLI (nyxid-cli)	72.59%	64%	✅	🔺 +0.14
Frontend (vitest)	71.35%	15%	✅	🔺 +0.35

Gate: line coverage must stay at or above the threshold. Ratchet plan (W21): Backend → 55%, CLI → 50%, Frontend → 30% by quarter end.