diff --git a/docs-site/src/content/docs/fr/reference/cli/providers-accounts.md b/docs-site/src/content/docs/fr/reference/cli/providers-accounts.md index f4e17ef8221..4a8824336de 100644 --- a/docs-site/src/content/docs/fr/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/fr/reference/cli/providers-accounts.md @@ -214,6 +214,10 @@ faire basculer la requête vers un autre compte de pool admissible. Ces transiti { ok: true, provider, type, activeId } ``` +### `ocx account pause|resume anthropic [--json]` + +La commande CLI suspend ou reprend un compte Anthropic OAuth par id ou alias unique (correspondance exacte, puis sans distinction de casse). Utilise `PUT /api/oauth/accounts/pause` avec `{ provider: "anthropic", accountId, paused }`, également utilisé par le tableau de bord. L’état `paused` est enregistré dans le compte et exposé par `GET /api/oauth/accounts`. La suspension s’applique même si le pool proactif est désactivé : le compte est exclu de la sélection, des affinités et des successeurs 429. Si tous les comptes sont suspendus, les requêtes renvoient 403 jusqu’à une reprise. Les requêtes déjà envoyées continuent ; les identifiants et l’état de santé sont conservés. La suspension survit au redémarrage et à une nouvelle connexion, et disparaît avec la suppression du compte. Les seuils individuels ne font pas partie de cette commande. + ### `ocx account clear [--json]` Efface la sélection manuelle du compte Codex sans résoudre d'id de compte, donc fonctionne même lorsqu'un compte s'appelle littéralement `auto`. Pools Codex uniquement ; les autres types de fournisseur n'ont pas de sélection automatique à rétablir. diff --git a/docs-site/src/content/docs/fr/reference/management-api.md b/docs-site/src/content/docs/fr/reference/management-api.md index 6b0f60fc5e5..207e6a4c965 100644 --- a/docs-site/src/content/docs/fr/reference/management-api.md +++ b/docs-site/src/content/docs/fr/reference/management-api.md @@ -283,6 +283,7 @@ Tant qu’une liste initiale fiable n’est pas disponible, les requêtes PUT va | `GET, PUT, PATCH /api/oauth/accounts/pool` | Lire ou mettre à jour la stratégie du pool OAuth Anthropic | 400 fournisseur non Anthropic ou stratégie invalide | | `POST /api/oauth/accounts/clear-cooldown` | Effacer le temps de recharge d'un compte OAuth | 400 invalide provider/account | | `PUT /api/oauth/accounts/alias` | Définir ou supprimer un alias de compte OAuth | 400 invalide provider/account/alias | +| `PUT /api/oauth/accounts/pause` | Suspendre/reprendre Anthropic ou un compte OAuth générique. Body `{ provider, accountId, paused }` ; la suspension du compte actif sélectionne un autre compte utilisable s’il existe. | 400 fournisseur non pris en charge ou body invalide ; 404 compte absent ; `oauth_mutation_busy` | | `GET, POST, DELETE /api/providers/keys` | Répertorier les clés de fournisseur masquées, en ajouter ou en activer une, ou en supprimer une | 400 saisie invalide ; 404 fournisseur ou clé manquante | | `PUT /api/providers/keys/active` | Sélectionnez la clé active d'un fournisseur | 400 saisie invalide ; 404 provider/key manquant | | `PUT /api/providers/keys/alias` | Définir ou supprimer un alias de clé de fournisseur | 400 saisie invalide ; 404 provider/key manquant | @@ -291,6 +292,10 @@ Tant qu’une liste initiale fiable n’est pas disponible, les requêtes PUT va Les réponses qui répertorient les identifiants sont délibérément masquées. Les jetons d'accès OAuth et les clés API complètes des fournisseurs ne sont pas renvoyés aux clients du tableau de bord. +#### Anthropic OAuth: `pause` / `resume` + +La commande CLI suspend ou reprend un compte Anthropic OAuth par id ou alias unique (correspondance exacte, puis sans distinction de casse). Utilise `PUT /api/oauth/accounts/pause` avec `{ provider: "anthropic", accountId, paused }`, également utilisé par le tableau de bord. L’état `paused` est enregistré dans le compte et exposé par `GET /api/oauth/accounts`. La suspension s’applique même si le pool proactif est désactivé : le compte est exclu de la sélection, des affinités et des successeurs 429. Si tous les comptes sont suspendus, les requêtes renvoient 403 jusqu’à une reprise. Les requêtes déjà envoyées continuent ; les identifiants et l’état de santé sont conservés. La suspension survit au redémarrage et à une nouvelle connexion, et disparaît avec la suppression du compte. Les seuils individuels ne font pas partie de cette commande. + ### Fournisseurs | Méthode et chemin | Objectif | Erreurs notables | diff --git a/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md b/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md index 479fac0f0da..2d4b72755a6 100644 --- a/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md @@ -159,6 +159,10 @@ Codex pool selection applies to the next request after clearing existing affinit { ok: true, provider, type, activeId } ``` +### `ocx account pause|resume anthropic [--json]` + +CLI コマンドは Anthropic OAuth アカウントを id または一意の別名で一時停止・再開します。別名は完全一致を優先し、次に大文字と小文字を区別せず照合します。ダッシュボードと同じ `PUT /api/oauth/accounts/pause` に `{ provider: "anthropic", accountId, paused }` を送信します。`paused` はアカウントに保存され、`GET /api/oauth/accounts` にも表示されます。プロアクティブなプールが無効でも、停止中のアカウントは選択、セッションの紐付け、429 の切り替え候補から除外されます。全アカウントが停止中なら、再開するまでリクエストは 403 を返します。送信済みのリクエストは継続し、認証情報と健全性の状態は保持されます。再起動や再ログインでも停止は維持され、アカウント削除時に消えます。アカウント別のしきい値はこの操作に含まれません。 + ### `ocx account clear [--json]` アカウント id を解決せずに Codex アカウントの手動選択を解除するため、`auto` という id のアカウントが存在しても機能します。Codex プール専用です。他のプロバイダー種別には復元する自動選択がありません。 diff --git a/docs-site/src/content/docs/ja/reference/management-api.md b/docs-site/src/content/docs/ja/reference/management-api.md index 9821c5d358f..3fee2a3a801 100644 --- a/docs-site/src/content/docs/ja/reference/management-api.md +++ b/docs-site/src/content/docs/ja/reference/management-api.md @@ -245,6 +245,7 @@ Aside プロファイルの変更はこの場合でも一つだけ保存しま | `GET, PUT, PATCH /api/oauth/accounts/pool` | Anthropic OAuth プール ポリシーの読み取りまたは更新 | 400 非 Anthropic プロバイダーまたは無効なポリシー | | `POST /api/oauth/accounts/clear-cooldown` | 1 つの OAuth アカウントのランタイム クールダウンをクリアする | 400 無効なプロバイダー/アカウント | | `PUT /api/oauth/accounts/alias` | OAuth アカウント エイリアスを設定またはクリアする | 400 無効なプロバイダー/アカウント/エイリアス | +| `PUT /api/oauth/accounts/pause` | Anthropic または汎用 OAuth アカウントを一時停止・再開。Body `{ provider, accountId, paused }`。アクティブなアカウントを停止すると、利用可能な別のアカウントがあれば切り替えます。 | 400 未対応のプロバイダーまたは無効な body；404 アカウントなし；`oauth_mutation_busy` | | `GET, POST, DELETE /api/providers/keys` |マスクされたプロバイダー キーを一覧表示し、1 つを追加/アクティブ化するか、1 つを削除します。 400 無効な入力。 404 プロバイダー/キーがありません | | `PUT /api/providers/keys/active` |プロバイダーのアクティブなキーを選択します | 400 無効な入力。 404 プロバイダー/キーがありません | | `PUT /api/providers/keys/alias` |プロバイダー キー エイリアスを設定またはクリアする | 400 無効な入力。 404 プロバイダー/キーがありません | @@ -252,6 +253,10 @@ Aside プロファイルの変更はこの場合でも一つだけ保存しま 資格情報リストの応答は意図的にマスクされます。 OAuth アクセス トークンと完全なプロバイダー API キーはダッシュボード クライアントに返されません。 +#### Anthropic OAuth: `pause` / `resume` + +CLI コマンドは Anthropic OAuth アカウントを id または一意の別名で一時停止・再開します。別名は完全一致を優先し、次に大文字と小文字を区別せず照合します。ダッシュボードと同じ `PUT /api/oauth/accounts/pause` に `{ provider: "anthropic", accountId, paused }` を送信します。`paused` はアカウントに保存され、`GET /api/oauth/accounts` にも表示されます。プロアクティブなプールが無効でも、停止中のアカウントは選択、セッションの紐付け、429 の切り替え候補から除外されます。全アカウントが停止中なら、再開するまでリクエストは 403 を返します。送信済みのリクエストは継続し、認証情報と健全性の状態は保持されます。再起動や再ログインでも停止は維持され、アカウント削除時に消えます。アカウント別のしきい値はこの操作に含まれません。 + ### プロバイダー |メソッドとパス |目的 |注目すべきエラー | diff --git a/docs-site/src/content/docs/ko/reference/cli/providers-accounts.md b/docs-site/src/content/docs/ko/reference/cli/providers-accounts.md index 5f94fd837b1..856390db88c 100644 --- a/docs-site/src/content/docs/ko/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/ko/reference/cli/providers-accounts.md @@ -254,6 +254,10 @@ Codex pool selection applies to the next request after clearing existing affinit { ok: true, provider, type, activeId } ``` +### `ocx account pause|resume anthropic [--json]` + +CLI 명령은 Anthropic OAuth 계정을 id 또는 유일한 alias로 일시 정지하거나 재개합니다. alias는 정확히 일치하는 값을 먼저 찾고, 없으면 대소문자를 구분하지 않고 찾습니다. 대시보드와 같은 `PUT /api/oauth/accounts/pause`에 `{ provider: "anthropic", accountId, paused }`를 보냅니다. 계정에 저장되는 `paused` 상태는 `GET /api/oauth/accounts`에도 표시됩니다. 사전 계정 전환 풀이 꺼져 있어도 정지된 계정은 선택, 세션 바인딩, 429 대체 후보에서 제외됩니다. 모든 계정이 정지되면 하나를 재개할 때까지 요청은 403을 반환합니다. 이미 전송한 요청은 유지하며 자격 증명과 건강 상태를 지우지 않습니다. 재시작·재로그인 후에도 정지는 유지되고, 계정을 삭제하면 함께 제거됩니다. 계정별 전환 임계값은 이 기능에 포함되지 않습니다. + ### `ocx account clear [--json]` 계정 id를 해석하지 않고 Codex 계정의 수동 선택을 지우므로 `auto`라는 id의 계정이 있어도 동작합니다. Codex 풀 전용이며 다른 공급자 유형에는 복원할 자동 선택이 없습니다. diff --git a/docs-site/src/content/docs/ko/reference/management-api.md b/docs-site/src/content/docs/ko/reference/management-api.md index cd428d4ef35..1a884ba64d4 100644 --- a/docs-site/src/content/docs/ko/reference/management-api.md +++ b/docs-site/src/content/docs/ko/reference/management-api.md @@ -258,6 +258,7 @@ Aside 프로필 변경은 이때도 한 가지를 저장합니다. 확인을 보 | `GET, PUT, PATCH /api/oauth/accounts/pool` | Anthropic과 일반 OAuth provider의 기존 pool policy입니다. `/api/pool/settings`로 대체되었고 기존 클라이언트를 위해 유지합니다 | 400 codex 또는 API 키 provider, 잘못된 policy | | `POST /api/oauth/accounts/clear-cooldown` | OAuth 계정 하나의 런타임 cooldown을 지웁니다 | 400 잘못된 provider/account | | `PUT /api/oauth/accounts/alias` | OAuth 계정 alias를 설정하거나 지웁니다 | 400 잘못된 provider/account/alias | +| `PUT /api/oauth/accounts/pause` | Anthropic 또는 일반 OAuth 계정을 정지·재개합니다. Body `{ provider, accountId, paused }`. 활성 계정을 정지하면 사용 가능한 다른 계정이 있을 때 전환합니다. | 400 지원하지 않는 provider 또는 잘못된 body; 404 계정 없음; `oauth_mutation_busy` | | `GET, POST, DELETE /api/providers/keys` | 마스킹된 provider key를 나열, 추가/활성화, 또는 제거합니다 | 400 잘못된 입력; 404 provider/key 없음 | | `PUT /api/providers/keys/active` | provider의 활성 key를 선택합니다 | 400 잘못된 입력; 404 provider/key 없음 | | `PUT /api/providers/keys/alias` | provider-key alias를 설정하거나 지웁니다 | 400 잘못된 입력; 404 provider/key 없음 | @@ -265,6 +266,10 @@ Aside 프로필 변경은 이때도 한 가지를 저장합니다. 확인을 보 자격 증명 목록 응답은 의도적으로 마스킹됩니다. OAuth access token과 완전한 provider API key는 대시보드 클라이언트에 반환되지 않습니다. +#### Anthropic OAuth: `pause` / `resume` + +CLI 명령은 Anthropic OAuth 계정을 id 또는 유일한 alias로 일시 정지하거나 재개합니다. alias는 정확히 일치하는 값을 먼저 찾고, 없으면 대소문자를 구분하지 않고 찾습니다. 대시보드와 같은 `PUT /api/oauth/accounts/pause`에 `{ provider: "anthropic", accountId, paused }`를 보냅니다. 계정에 저장되는 `paused` 상태는 `GET /api/oauth/accounts`에도 표시됩니다. 사전 계정 전환 풀이 꺼져 있어도 정지된 계정은 선택, 세션 바인딩, 429 대체 후보에서 제외됩니다. 모든 계정이 정지되면 하나를 재개할 때까지 요청은 403을 반환합니다. 이미 전송한 요청은 유지하며 자격 증명과 건강 상태를 지우지 않습니다. 재시작·재로그인 후에도 정지는 유지되고, 계정을 삭제하면 함께 제거됩니다. 계정별 전환 임계값은 이 기능에 포함되지 않습니다. + ### 제공자 | HTTP 메서드와 경로 | 목적 | 주요 오류 | diff --git a/docs-site/src/content/docs/reference/cli/providers-accounts.md b/docs-site/src/content/docs/reference/cli/providers-accounts.md index b8ce75d6486..02edd827e13 100644 --- a/docs-site/src/content/docs/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/reference/cli/providers-accounts.md @@ -469,16 +469,21 @@ Clear the manual Codex account selection without resolving an account id, so it ### `ocx account pause|resume [--json]` -Pause or resume one account in the Codex pool or a generic OAuth provider pool, including +Pause or resume one account in the Codex, Anthropic, or generic OAuth provider pool, including `google-antigravity`. For the Codex pool, `main` identifies only the built-in Codex account; -generic OAuth accounts must be identified by id or a unique alias. A paused generic OAuth account +OAuth accounts must be identified by id or a unique alias. A paused OAuth account is excluded from request selection, 429 failover, and proactive token refresh, and cannot be selected manually. Pausing the active account switches to the next usable account when one exists. If every account is paused, requests that need that pool return 403 until an account is resumed. -For a generic OAuth provider, identify the account by id or by a unique exact or case-insensitive +For Anthropic and generic OAuth providers, identify the account by id or by a unique exact or case-insensitive alias. The JSON response reports the account id, pause state, and active account id. +Anthropic pause applies even when proactive pooling is disabled, including session affinity and +429 successors. It survives restart and reauthentication, preserves credentials and health, +and does not interrupt a turn already sent. Removing the account removes its pause state. +Per-account Anthropic auto-switch thresholds are not part of this control. + ```bash ocx account pause google-antigravity ocx account resume google-antigravity diff --git a/docs-site/src/content/docs/reference/management-api.md b/docs-site/src/content/docs/reference/management-api.md index 5e2ea2b0602..3c60b3505db 100644 --- a/docs-site/src/content/docs/reference/management-api.md +++ b/docs-site/src/content/docs/reference/management-api.md @@ -557,10 +557,10 @@ outcome fields from an older server do not establish successful recovery. | `POST /api/oauth/login/cancel` | Cancel a public in-progress OAuth flow | 400 unknown provider | | `GET /api/oauth/status` | Poll one provider's OAuth flow | 400 unknown provider | | `POST /api/oauth/logout` | Remove the selected provider credential | 400 unknown provider; `oauth_mutation_busy` | -| `GET /api/oauth/accounts` | List masked accounts; generic OAuth account rows include their `paused` state. Kiro rows include `autoSelectable` and a closed `skipReason` when excluded from automatic selection; an active singleton may still send. Quota remains opt-in. | 400 invalid provider | +| `GET /api/oauth/accounts` | List masked accounts; Anthropic and generic OAuth account rows include their `paused` state. Kiro rows include `autoSelectable` and a closed `skipReason` when excluded from automatic selection; an active singleton may still send. Quota remains opt-in. | 400 invalid provider | | `DELETE /api/oauth/accounts` | Remove one account | 400 invalid provider/id; 404 account missing; `oauth_mutation_busy` | | `PUT /api/oauth/accounts/active` | Select the active OAuth account | 400 invalid provider/account; 404 account missing; 409 account paused; `oauth_mutation_busy` | -| `PUT /api/oauth/accounts/pause` | Pause or resume one generic OAuth account. Body `{ provider, accountId, paused }`; pausing the active account selects the next usable account when available | 400 unsupported provider or invalid body; 404 account missing; `oauth_mutation_busy` | +| `PUT /api/oauth/accounts/pause` | Pause or resume one Anthropic or generic OAuth account. Body `{ provider, accountId, paused }`; pausing the active account selects the next usable account when available. Pause is durable and independent of pool enablement; resume preserves health and credentials. | 400 unsupported provider or invalid body; 404 account missing; `oauth_mutation_busy` | | `GET, PUT, PATCH /api/pool/settings` | Read or update pool policy for any kind (codex, anthropic, generic); answers with the same keys for all three and declares in `supported` which the kind honours | 400 unknown provider, a field the kind does not support, or an invalid value | | `GET, PUT, PATCH /api/oauth/accounts/pool` | Legacy per-pool policy for Anthropic and generic OAuth providers; superseded by `/api/pool/settings` and kept for existing clients | 400 codex or api-key provider, or invalid policy | | `POST /api/oauth/accounts/clear-cooldown` | Clear one OAuth account's runtime cooldown | 400 invalid provider/account | diff --git a/docs-site/src/content/docs/ru/reference/cli/providers-accounts.md b/docs-site/src/content/docs/ru/reference/cli/providers-accounts.md index 7339d3f53bb..bd3daa00013 100644 --- a/docs-site/src/content/docs/ru/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/ru/reference/cli/providers-accounts.md @@ -190,6 +190,10 @@ credential'а, это состояние тоже печатается, но к { ok: true, provider, type, activeId } ``` +### `ocx account pause|resume anthropic [--json]` + +Команда CLI приостанавливает или возобновляет аккаунт Anthropic OAuth по id или уникальному alias: сначала точное совпадение, затем без учёта регистра. CLI и дашборд используют `PUT /api/oauth/accounts/pause` с `{ provider: "anthropic", accountId, paused }`. Поле `paused` сохраняется в аккаунте и возвращается через `GET /api/oauth/accounts`. Пауза действует и при отключённом проактивном пуле: аккаунт исключается из выбора, привязок сессий и кандидатов после 429. Если приостановлены все аккаунты, запросы получают 403 до возобновления одного из них. Уже отправленные запросы продолжаются; учётные данные и состояние здоровья сохраняются. Пауза переживает перезапуск и повторный вход, но удаляется вместе с аккаунтом. Индивидуальные пороги в эту операцию не входят. + ### `ocx account clear [--json]` Снимает ручной выбор аккаунта Codex без разрешения id, поэтому работает, даже когда аккаунт буквально называется `auto`. Только для пулов Codex; у других типов провайдеров нет автоматического выбора для восстановления. diff --git a/docs-site/src/content/docs/ru/reference/management-api.md b/docs-site/src/content/docs/ru/reference/management-api.md index 30a02a70ed7..62b74ab5452 100644 --- a/docs-site/src/content/docs/ru/reference/management-api.md +++ b/docs-site/src/content/docs/ru/reference/management-api.md @@ -276,6 +276,7 @@ Endpoint'ы storage cleanup могут перемещать или навсег | `GET, PUT, PATCH /api/oauth/accounts/pool` | Прежняя policy пула для Anthropic и обычных OAuth-провайдеров; заменена на `/api/pool/settings` и сохранена для существующих клиентов | 400 codex или api-key provider, либо недопустимая policy | | `POST /api/oauth/accounts/clear-cooldown` | Очистить runtime cooldown одного OAuth-аккаунта | 400 invalid provider/account | | `PUT /api/oauth/accounts/alias` | Задать или очистить alias OAuth-аккаунта | 400 invalid provider/account/alias | +| `PUT /api/oauth/accounts/pause` | Приостановить/возобновить Anthropic или обычный OAuth-аккаунт. Body `{ provider, accountId, paused }`; при паузе активного аккаунта выбирается другой пригодный аккаунт, если он есть. | 400 неподдерживаемый provider или неверный body; 404 аккаунт не найден; `oauth_mutation_busy` | | `GET, POST, DELETE /api/providers/keys` | Показать список masked provider-key'ов, добавить/активировать один или удалить один | 400 invalid input; 404 provider/key missing | | `PUT /api/providers/keys/active` | Выбрать активный ключ провайдера | 400 invalid input; 404 provider/key missing | | `PUT /api/providers/keys/alias` | Задать или очистить alias provider-key'а | 400 invalid input; 404 provider/key missing | @@ -284,6 +285,10 @@ Endpoint'ы storage cleanup могут перемещать или навсег Ответы со списками credential'ов намеренно маскируются. OAuth access-token'ы и полные API-key'и провайдеров клиентам дашборда не возвращаются. +#### Anthropic OAuth: `pause` / `resume` + +Команда CLI приостанавливает или возобновляет аккаунт Anthropic OAuth по id или уникальному alias: сначала точное совпадение, затем без учёта регистра. CLI и дашборд используют `PUT /api/oauth/accounts/pause` с `{ provider: "anthropic", accountId, paused }`. Поле `paused` сохраняется в аккаунте и возвращается через `GET /api/oauth/accounts`. Пауза действует и при отключённом проактивном пуле: аккаунт исключается из выбора, привязок сессий и кандидатов после 429. Если приостановлены все аккаунты, запросы получают 403 до возобновления одного из них. Уже отправленные запросы продолжаются; учётные данные и состояние здоровья сохраняются. Пауза переживает перезапуск и повторный вход, но удаляется вместе с аккаунтом. Индивидуальные пороги в эту операцию не входят. + ### Провайдеры | Метод и путь | Назначение | Особые ошибки | diff --git a/docs-site/src/content/docs/tr/reference/cli/providers-accounts.md b/docs-site/src/content/docs/tr/reference/cli/providers-accounts.md index 15d2c125d1f..6c98e625a2d 100644 --- a/docs-site/src/content/docs/tr/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/tr/reference/cli/providers-accounts.md @@ -235,6 +235,10 @@ ayar yalnızca kullanıma dayalı proaktif geçişi devre dışı bırakır. { ok: true, provider, type, activeId } ``` +### `ocx account pause|resume anthropic [--json]` + +CLI komutu Anthropic OAuth hesabını id veya benzersiz takma ad ile duraklatır ya da sürdürür. Önce tam eşleşme, ardından büyük/küçük harf duyarsız eşleşme aranır. CLI ve kontrol paneli aynı `PUT /api/oauth/accounts/pause` uç noktasına `{ provider: "anthropic", accountId, paused }` gönderir. `paused` hesapta saklanır ve `GET /api/oauth/accounts` yanıtında gösterilir. Proaktif havuz kapalı olsa bile duraklatılan hesap seçimden, oturum bağlarından ve 429 sonrası adaylardan çıkarılır. Tüm hesaplar duraklatılmışsa biri sürdürülene kadar istekler 403 döndürür. Önceden gönderilmiş istekler devam eder; kimlik bilgileri ve sağlık durumu korunur. Duraklatma yeniden başlatma ve yeniden girişten sonra da sürer, hesap silinince kaldırılır. Hesaba özel eşikler bu işleme dahil değildir. + ### `ocx account clear [--json]` Bir hesap id'si çözümlemeden Codex hesabının elle seçimini temizler; `auto` adında bir hesap olsa bile çalışır. Yalnızca Codex havuzları içindir; diğer sağlayıcı türlerinde geri yüklenecek otomatik seçim yoktur. diff --git a/docs-site/src/content/docs/tr/reference/management-api.md b/docs-site/src/content/docs/tr/reference/management-api.md index 1bd688931ae..0a19213029c 100644 --- a/docs-site/src/content/docs/tr/reference/management-api.md +++ b/docs-site/src/content/docs/tr/reference/management-api.md @@ -301,6 +301,7 @@ Güvenilir ilk model listesi hazır olana kadar `/api/selected-models` ve `/api/ | `GET, PUT, PATCH /api/oauth/accounts/pool` | Anthropic OAuth havuz politikasını okuyun veya güncelleyin | 400 Anthropic olmayan sağlayıcı veya geçersiz politika | | `POST /api/oauth/accounts/clear-cooldown` | Bir OAuth hesabının çalışma zamanı soğuma süresini temizleyin | 400 geçersiz sağlayıcı/hesap | | `PUT /api/oauth/accounts/alias` | Bir OAuth hesap takma adını ayarlayın veya temizleyin | 400 geçersiz sağlayıcı/hesap/takma ad | +| `PUT /api/oauth/accounts/pause` | Anthropic veya genel OAuth hesabını duraklatın/sürdürün. Body `{ provider, accountId, paused }`; etkin hesap duraklatıldığında varsa kullanılabilir başka hesaba geçilir. | 400 desteklenmeyen sağlayıcı veya geçersiz body; 404 hesap yok; `oauth_mutation_busy` | | `GET, POST, DELETE /api/providers/keys` | Maskelenmiş sağlayıcı anahtarlarını listeleyin, bir tane ekleyin/etkinleştirin veya kaldırın | 400 geçersiz girdi; 404 sağlayıcı/anahtar eksik | | `PUT /api/providers/keys/active` | Bir sağlayıcının etkin anahtarını seçin | 400 geçersiz girdi; 404 sağlayıcı/anahtar eksik | | `PUT /api/providers/keys/alias` | Bir sağlayıcı anahtarı takma adını ayarlayın veya temizleyin | 400 geçersiz girdi; 404 sağlayıcı/anahtar eksik | @@ -310,6 +311,10 @@ Kimlik bilgisi listesi yanıtları kasıtlı olarak maskelenir. OAuth erişim belirteçleri ve eksiksiz sağlayıcı API anahtarları kontrol paneli istemcilerine döndürülmez. +#### Anthropic OAuth: `pause` / `resume` + +CLI komutu Anthropic OAuth hesabını id veya benzersiz takma ad ile duraklatır ya da sürdürür. Önce tam eşleşme, ardından büyük/küçük harf duyarsız eşleşme aranır. CLI ve kontrol paneli aynı `PUT /api/oauth/accounts/pause` uç noktasına `{ provider: "anthropic", accountId, paused }` gönderir. `paused` hesapta saklanır ve `GET /api/oauth/accounts` yanıtında gösterilir. Proaktif havuz kapalı olsa bile duraklatılan hesap seçimden, oturum bağlarından ve 429 sonrası adaylardan çıkarılır. Tüm hesaplar duraklatılmışsa biri sürdürülene kadar istekler 403 döndürür. Önceden gönderilmiş istekler devam eder; kimlik bilgileri ve sağlık durumu korunur. Duraklatma yeniden başlatma ve yeniden girişten sonra da sürer, hesap silinince kaldırılır. Hesaba özel eşikler bu işleme dahil değildir. + ### Sağlayıcılar | Yöntem ve yol | Amaç | Dikkate değer hatalar | diff --git a/docs-site/src/content/docs/zh-cn/reference/cli/providers-accounts.md b/docs-site/src/content/docs/zh-cn/reference/cli/providers-accounts.md index 2a102ecec5c..1806ae38843 100644 --- a/docs-site/src/content/docs/zh-cn/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/zh-cn/reference/cli/providers-accounts.md @@ -173,6 +173,10 @@ Codex Pool 选择会清除进程本地 affinity，并从下一次请求开始生 { ok: true, provider, type, activeId } ``` +### `ocx account pause|resume anthropic [--json]` + +CLI 命令通过 id 或唯一别名暂停或恢复 Anthropic OAuth 账户。别名先精确匹配，再进行不区分大小写的匹配。CLI 和仪表板使用同一个 `PUT /api/oauth/accounts/pause`，请求体为 `{ provider: "anthropic", accountId, paused }`。`paused` 保存在账户中，并通过 `GET /api/oauth/accounts` 返回。即使主动账户池已关闭，暂停账户也会从选择、会话绑定和 429 后继候选中排除。所有账户暂停时，请求返回 403，直到恢复一个账户。已经发送的请求继续执行，凭证和健康状态保持不变。重启或重新登录仍保留暂停，删除账户时一并清除。此操作不包含账户级自动切换阈值。 + ### `ocx account clear [--json]` 在不解析账号 id 的情况下清除 Codex 账号的手动选择，因此即使存在名为 `auto` 的账号也有效。仅适用于 Codex Pool；其他提供商类型没有可恢复的自动选择。 diff --git a/docs-site/src/content/docs/zh-cn/reference/management-api.md b/docs-site/src/content/docs/zh-cn/reference/management-api.md index 6280fb8b9d8..4abf420ad36 100644 --- a/docs-site/src/content/docs/zh-cn/reference/management-api.md +++ b/docs-site/src/content/docs/zh-cn/reference/management-api.md @@ -238,6 +238,7 @@ Aside 配置档的变更在这种情况下仍会保存一件事：确认之后 | `GET, PUT, PATCH /api/oauth/accounts/pool` | 读取或更新 Anthropic OAuth 池策略 | 400 非 Anthropic provider 或策略无效 | | `POST /api/oauth/accounts/clear-cooldown` | 清除一个 OAuth 账户的运行时冷却 | 400 provider/账户无效 | | `PUT /api/oauth/accounts/alias` | 设置或清除 OAuth 账户别名 | 400 provider/账户/别名无效 | +| `PUT /api/oauth/accounts/pause` | 暂停或恢复 Anthropic 或通用 OAuth 账户。Body `{ provider, accountId, paused }`；暂停活跃账户时，如有其他可用账户则切换过去。 | 400 不支持的 provider 或无效 body；404 账户不存在；`oauth_mutation_busy` | | `GET, POST, DELETE /api/providers/keys` | 列出已脱敏的 provider 密钥，添加/激活一个，或移除一个 | 400 输入无效；404 provider/密钥缺失 | | `PUT /api/providers/keys/active` | 选择某个 provider 的活跃密钥 | 400 输入无效；404 provider/密钥缺失 | | `PUT /api/providers/keys/alias` | 设置或清除 provider 密钥别名 | 400 输入无效；404 provider/密钥缺失 | @@ -245,6 +246,10 @@ Aside 配置档的变更在这种情况下仍会保存一件事：确认之后 凭证列表响应会刻意脱敏。OAuth 访问令牌和完整的 provider API 密钥不会返回给仪表板客户端。 +#### Anthropic OAuth: `pause` / `resume` + +CLI 命令通过 id 或唯一别名暂停或恢复 Anthropic OAuth 账户。别名先精确匹配，再进行不区分大小写的匹配。CLI 和仪表板使用同一个 `PUT /api/oauth/accounts/pause`，请求体为 `{ provider: "anthropic", accountId, paused }`。`paused` 保存在账户中，并通过 `GET /api/oauth/accounts` 返回。即使主动账户池已关闭，暂停账户也会从选择、会话绑定和 429 后继候选中排除。所有账户暂停时，请求返回 403，直到恢复一个账户。已经发送的请求继续执行，凭证和健康状态保持不变。重启或重新登录仍保留暂停，删除账户时一并清除。此操作不包含账户级自动切换阈值。 + ### Providers | 方法和路径 | 用途 | 典型错误 | diff --git a/docs-site/src/content/docs/zh-tw/reference/cli/providers-accounts.md b/docs-site/src/content/docs/zh-tw/reference/cli/providers-accounts.md index 141467062a6..6abdcc7e7bc 100644 --- a/docs-site/src/content/docs/zh-tw/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/zh-tw/reference/cli/providers-accounts.md @@ -144,15 +144,19 @@ Codex 池選擇套用於清除既有親和性後的下一個請求；進行中 ### `ocx account pause|resume [--json]` -暫停或恢復 Codex 帳號池或通用 OAuth 供應商池中的單一帳號，包括 -`google-antigravity`。在 Codex 池中，`main` 僅代表 Codex 內建帳號；通用 OAuth 帳號必須用 id 或唯一別名識別。 -已暫停的通用 OAuth 帳號不會參與請求選帳、429 輪替或主動 Token 刷新，也不能手動選取。 +暫停或恢復 Codex、Anthropic 或通用 OAuth 供應商池中的單一帳號，包括 +`google-antigravity`。在 Codex 池中，`main` 僅代表 Codex 內建帳號；OAuth 帳號必須用 id 或唯一別名識別。 +已暫停的 OAuth 帳號不會參與請求選帳、429 輪替或主動 Token 刷新，也不能手動選取。 若暫停目前使用中的帳號，系統會在有其他可用帳號時切換過去。若全部帳號都已暫停， 需要該池的請求會回覆 403，直到恢復其中一個帳號。 -通用 OAuth 供應商可用帳號 id，或唯一且完全相符／不區分大小寫的別名識別帳號。 +Anthropic 和通用 OAuth 供應商可用帳號 id，或唯一且完全相符／不區分大小寫的別名識別帳號。 JSON 回應會提供帳號 id、暫停狀態與目前 active 帳號 id。 +Anthropic 暫停不受帳號池啟用開關影響，包含工作階段綁定與 429 後繼選帳。 +重新啟動或登入仍保留暫停，憑證與健康狀態不會清除，已送出的請求不會中斷。 +刪除帳號會一併刪除暫停狀態；個別帳號的自動切換門檻不在此功能範圍內。 + ```bash ocx account pause google-antigravity ocx account resume google-antigravity diff --git a/docs-site/src/content/docs/zh-tw/reference/management-api.md b/docs-site/src/content/docs/zh-tw/reference/management-api.md index 9e3549dbb42..e987e1f8a72 100644 --- a/docs-site/src/content/docs/zh-tw/reference/management-api.md +++ b/docs-site/src/content/docs/zh-tw/reference/management-api.md @@ -233,10 +233,10 @@ Aside 設定檔的變更在這種情況下仍會儲存一件事：確認之後 | `POST /api/oauth/login/cancel` | 取消公開進行中的 OAuth 流程 | 400 未知供應商 | | `GET /api/oauth/status` | 輪詢一個供應商的 OAuth 流程 | 400 未知供應商 | | `POST /api/oauth/logout` | 移除所選的供應商憑證 | 400 未知供應商；`oauth_mutation_busy` | -| `GET /api/oauth/accounts` | 列出遮罩帳號；通用 OAuth 帳號列也會提供 `paused` 狀態。Kiro 列包含自動選取狀態 `autoSelectable`，排除時還包含封閉集合的 `skipReason`。唯一的有效帳號仍可傳送請求，配額查詢仍為選用。 | 400 無效供應商 | +| `GET /api/oauth/accounts` | 列出遮罩帳號；Anthropic 與通用 OAuth 帳號列也會提供 `paused` 狀態。Kiro 列包含自動選取狀態 `autoSelectable`，排除時還包含封閉集合的 `skipReason`。唯一的有效帳號仍可傳送請求，配額查詢仍為選用。 | 400 無效供應商 | | `DELETE /api/oauth/accounts` | 移除一個帳號 | 400 無效供應商/id；404 帳號缺失；`oauth_mutation_busy` | | `PUT /api/oauth/accounts/active` | 選擇現用 OAuth 帳號 | 400 無效供應商／帳號；404 帳號缺失；409 帳號已暫停；`oauth_mutation_busy` | -| `PUT /api/oauth/accounts/pause` | 暫停或恢復一個通用 OAuth 帳號。Body `{ provider, accountId, paused }`；若暫停現用帳號，且有可用帳號，會切換至下一個 | 400 不支援的供應商或無效 body；404 帳號缺失；`oauth_mutation_busy` | +| `PUT /api/oauth/accounts/pause` | 暫停或恢復一個 Anthropic 或通用 OAuth 帳號。Body `{ provider, accountId, paused }`；若暫停現用帳號，且有可用帳號，會切換至下一個。暫停會持久儲存且不受帳號池開關影響，恢復保留健康狀態與憑證。 | 400 不支援的供應商或無效 body；404 帳號缺失；`oauth_mutation_busy` | | `GET, PUT, PATCH /api/oauth/accounts/pool` | 讀取或更新 Anthropic OAuth 池政策 | 400 非 Anthropic 供應商或無效政策 | | `POST /api/oauth/accounts/clear-cooldown` | 清除一個 OAuth 帳號的 runtime 冷卻 | 400 無效供應商／帳號 | | `PUT /api/oauth/accounts/alias` | 設定或清除 OAuth 帳號別名 | 400 無效供應商／帳號／別名 | diff --git a/gui/tests/provider-quota-refresh-controls.test.tsx b/gui/tests/provider-quota-refresh-controls.test.tsx index 5766221a13f..2c1d868550a 100644 --- a/gui/tests/provider-quota-refresh-controls.test.tsx +++ b/gui/tests/provider-quota-refresh-controls.test.tsx @@ -172,12 +172,12 @@ test("the accounts surface omits the control when the page cannot force a read", expect(findButton("Refresh quotas")).toBeNull(); }); -test("generic OAuth accounts expose pause and resume controls", async () => { +test.each(["google-antigravity", "anthropic"])("%s OAuth accounts expose pause and resume controls", async provider => { const calls: Array<{ provider: string; accountId: string; paused: boolean }> = []; const handlers = authHandlers({ onPauseAccount: async (provider, row, paused) => { calls.push({ provider, accountId: row.id, paused }); }, }); - const item = { ...oauthItem, name: "google-antigravity" }; + const item = { ...oauthItem, name: provider }; await render( { const pause = findButton("Pause"); expect(pause).not.toBeNull(); await act(async () => { pause!.click(); }); - expect(calls).toEqual([{ provider: "google-antigravity", accountId: "ga-active", paused: true }]); + expect(calls).toEqual([{ provider, accountId: "ga-active", paused: true }]); await render( { expect(host.textContent).not.toContain(en["codexAuth.pausedHint"]); expect(en["pws.accountPausedHint"]).not.toBe(en["codexAuth.pausedHint"]); await act(async () => { resume!.click(); }); - expect(calls[1]).toEqual({ provider: "google-antigravity", accountId: "ga-active", paused: false }); + expect(calls[1]).toEqual({ provider, accountId: "ga-active", paused: false }); }); test("API-key rows use independent shared credit readings and the same awaited refresh control", async () => { diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 96c50b8f374..c0898f6e213 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -100,6 +100,8 @@ "ambiguous-resend-composition.test.ts": "lib", "ambiguous-resend-gate.test.ts": "lib", "anthropic-account-pool.test.ts": "adapters/anthropic", + "anthropic-account-pause-outbound.test.ts": "adapters/anthropic", + "anthropic-account-pause.test.ts": "adapters/anthropic", "anthropic-combo-account-cooldown.test.ts": "adapters/anthropic", "anthropic-model-routes.test.ts": "adapters/anthropic", "anthropic-agentrouter-language-framing.test.ts": "adapters/anthropic", diff --git a/skills/ocx/references/01_management_surface.md b/skills/ocx/references/01_management_surface.md index 1945073ed2b..110b56be1e9 100644 --- a/skills/ocx/references/01_management_surface.md +++ b/skills/ocx/references/01_management_surface.md @@ -803,7 +803,7 @@ JSON mode: `payload`. ### `ocx account pause` -Exclude one account in a Codex or supported generic OAuth pool from automatic selection. +Exclude one account in a Codex, Anthropic or supported generic OAuth pool from automatic selection. | Method | Route | |---|---| @@ -817,11 +817,11 @@ Exclude one account in a Codex or supported generic OAuth pool from automatic se JSON mode: `envelope`. -- Codex pause unbinds pinned threads and selects a fallback when possible; with no fallback, a paused-but-selected Codex account still receives requests. Generic OAuth pause never dispatches to that account: it is excluded from new requests, failover and refresh, and an all-paused pool answers 403. Anthropic is unsupported. +- Codex pause unbinds pinned threads and selects a fallback when possible; with no fallback, a paused-but-selected Codex account still receives requests. Anthropic and generic OAuth pause exclude the account from new requests, failover and refresh, and an all-paused pool answers 403. Credentials and health are preserved; already-sent turns are not cancelled. ### `ocx account resume` -Return a paused account to a Codex or supported generic OAuth pool. +Return a paused account to a Codex, Anthropic or supported generic OAuth pool. | Method | Route | |---|---| diff --git a/src/cli/account-extended.ts b/src/cli/account-extended.ts index 4fce819000d..5260b979a99 100644 --- a/src/cli/account-extended.ts +++ b/src/cli/account-extended.ts @@ -787,7 +787,7 @@ function resolveGenericOAuthPauseTarget(accounts: unknown[], requested: string): return { error: `Account not found: no OAuth account has the id or alias "${requested}"` }; } -/** Pause or resume a Codex account or a generic OAuth provider account. */ +/** Pause or resume a Codex or OAuth provider account, including Anthropic. */ export async function cmdPause(args: string[], deps: AccountDeps, paused: boolean): Promise { const wantsJson = flag(args, "--json"); const name = args.shift(); @@ -800,7 +800,6 @@ export async function cmdPause(args: string[], deps: AccountDeps, paused: boolea if (!baseUrl) return proxyUnreachable(); if (classified.type === "oauth") { - if (name === "anthropic") return usage(`Error: ${verb} is not supported for the Anthropic OAuth pool`); const list = await apiJson(deps, baseUrl, "GET", `/api/oauth/accounts?provider=${encodeURIComponent(name)}`); if (list.status === 0) return proxyUnreachable(list.transportError); if (list.status !== 200) return apiError(list.json, `failed to list ${name} OAuth accounts`, list.status); diff --git a/src/cli/capabilities.ts b/src/cli/capabilities.ts index 86fecb5a5f0..c64edc31d30 100644 --- a/src/cli/capabilities.ts +++ b/src/cli/capabilities.ts @@ -541,7 +541,7 @@ export const CAPABILITIES: readonly Capability[] = [ }, { command: ["account", "pause"], - summary: "Exclude one account in a Codex or supported generic OAuth pool from automatic selection.", + summary: "Exclude one account in a Codex, Anthropic or supported generic OAuth pool from automatic selection.", // Resume uses the same endpoints with `paused: false`. routes: [ { method: "PUT", path: "/api/codex-auth/accounts/pause" }, @@ -552,12 +552,12 @@ export const CAPABILITIES: readonly Capability[] = [ mutates: true, json: "envelope", details: [ - "Codex pause unbinds pinned threads and selects a fallback when possible; with no fallback, a paused-but-selected Codex account still receives requests. Generic OAuth pause never dispatches to that account: it is excluded from new requests, failover and refresh, and an all-paused pool answers 403. Anthropic is unsupported.", + "Codex pause unbinds pinned threads and selects a fallback when possible; with no fallback, a paused-but-selected Codex account still receives requests. Anthropic and generic OAuth pause exclude the account from new requests, failover and refresh, and an all-paused pool answers 403. Credentials and health are preserved; already-sent turns are not cancelled.", ], }, { command: ["account", "resume"], - summary: "Return a paused account to a Codex or supported generic OAuth pool.", + summary: "Return a paused account to a Codex, Anthropic or supported generic OAuth pool.", routes: [ { method: "PUT", path: "/api/codex-auth/accounts/pause" }, { method: "GET", path: "/api/oauth/accounts" }, diff --git a/src/codex/catalog/gather-capture.ts b/src/codex/catalog/gather-capture.ts index 7f67743cc0c..64eadad1c2e 100644 --- a/src/codex/catalog/gather-capture.ts +++ b/src/codex/catalog/gather-capture.ts @@ -120,6 +120,8 @@ export interface CatalogGatherProviderModelOutcome { export interface ModelsAuthResolution { readonly apiKey: string | undefined; readonly observed: boolean; + readonly oauthAccountId?: string; + readonly oauthGeneration?: string; readonly oauthApiBaseUrl?: string; readonly oauthProjectId?: string; } diff --git a/src/codex/catalog/provider-models.ts b/src/codex/catalog/provider-models.ts index f43b20904c0..a3db3bb898f 100644 --- a/src/codex/catalog/provider-models.ts +++ b/src/codex/catalog/provider-models.ts @@ -6,7 +6,7 @@ import { copyFileSync, existsSync, mkdirSync, readFileSync, realpathSync } from import { delimiter, dirname, join, resolve } from "node:path"; import { atomicWriteFile, expandUserPath, getConfigDir, websocketsEnabled } from "../../config"; import { resolveProviderApiKey } from "../../providers/key-store"; -import { getAccountSet } from "../../oauth/store"; +import { captureOAuthAccountSelection, credentialGeneration, getAccountCredentialWithStatus, getAccountSet } from "../../oauth/store"; import { readKiroAccountModels, kiroObservedContextWindow } from "../../providers/kiro-model-catalog"; import { CODEX_CONFIG_PATH, CODEX_MODELS_CACHE_PATH, DEFAULT_CATALOG_PATH, readRootTomlString, resolveCodexConfigPath } from "../paths"; import { @@ -74,6 +74,7 @@ import { import type { NormalizedComboConfig } from "../../combos/types"; import { ProviderOutboundPolicyError, + ProviderOutboundSendCancelledError, providerOutboundGet, providerOutboundPost, providerRedirectError, @@ -149,6 +150,8 @@ export function observedModelsAuthResolver( return { apiKey: observation.snapshot.accessToken, observed: true, + oauthAccountId: observation.snapshot.accountId, + oauthGeneration: observation.snapshot.generation, ...(observation.snapshot.apiBaseUrl ? { oauthApiBaseUrl: observation.snapshot.apiBaseUrl } : {}), ...(observation.snapshot.projectId ? { oauthProjectId: observation.snapshot.projectId } : {}), }; @@ -174,6 +177,8 @@ export async function fetchProviderModelsWithAuth( // generation, so a request started with the former account cannot later publish its result. const cacheGeneration = captureModelCacheGeneration(name); const isCurrentCacheGeneration = () => isModelCacheGenerationCurrent(name, cacheGeneration); + const anthropicSelection = name === "anthropic" && prov.authMode === "oauth" + ? captureOAuthAccountSelection(name) : null; if (prov.authMode === "forward") return observed([], "authoritative"); // ChatGPT backend has no /models const seedVertexDefault = prov.adapter === "google" && prov.googleMode === "vertex" @@ -256,6 +261,8 @@ export async function fetchProviderModelsWithAuth( .then(snapshot => ({ apiKey: snapshot.accessToken, observed: false, + oauthAccountId: snapshot.accountId, + oauthGeneration: snapshot.generation, ...(snapshot.apiBaseUrl ? { oauthApiBaseUrl: snapshot.apiBaseUrl } : {}), ...(snapshot.projectId ? { oauthProjectId: snapshot.projectId } : {}), })) @@ -263,6 +270,15 @@ export async function fetchProviderModelsWithAuth( : { apiKey: await resolveModelsAuthToken(name, prov), observed: false } : resolveAuth.resolve(name, prov)); const apiKey = auth.apiKey; + const maySendAnthropicDiscovery = () => { + if (name !== "anthropic" || prov.authMode !== "oauth") return true; + const selected = captureOAuthAccountSelection(name); + const row = auth.oauthAccountId ? getAccountCredentialWithStatus(name, auth.oauthAccountId) : null; + return !!anthropicSelection && !!selected && !!row && !row.paused && !row.needsReauth + && selected.accountId === anthropicSelection.accountId && selected.revision === anthropicSelection.revision + && row.credential.expires > Date.now() && auth.oauthAccountId === selected.accountId + && credentialGeneration(row.credential) === auth.oauthGeneration && row.credential.access === apiKey; + }; // A configured default is a real callable selector and must remain discoverable when a // compatible provider's live /models request fails (issue #308). Static providers already seed // their default selector above when no explicit model list exists. @@ -609,7 +625,8 @@ export async function fetchProviderModelsWithAuth( // proof is on the URL — not the provider name — because an OAuth/forward // name matches any baseUrl by design. Retargeted or renamed custom rows // fetch a different URL and keep the rejection. - const outboundDependencies = { isCanonicalUrl: isRegistryModelDiscoveryUrl }; + if (!maySendAnthropicDiscovery()) return observed(withConfiguredRetention(failedDiscoveryConfigured), "degraded"); + const outboundDependencies = { isCanonicalUrl: isRegistryModelDiscoveryUrl, beforeSend: maySendAnthropicDiscovery }; const res = request.method === "POST" ? await providerOutboundPost(name, prov, url, { headers, @@ -774,6 +791,9 @@ export async function fetchProviderModelsWithAuth( markProviderDiscoveryOk(name, liveModelCount); return observed(returned, "authoritative"); } catch (error) { + if (error instanceof ProviderOutboundSendCancelledError) { + return observed(withConfiguredRetention(failedDiscoveryConfigured), "degraded"); + } if (error instanceof ProviderOutboundPolicyError) { const { models, fallback, shouldLog } = failedDiscoveryFallback({ reason: "blocked" }); if (shouldLog) { diff --git a/src/lib/provider-outbound.ts b/src/lib/provider-outbound.ts index 995e9b4c8fb..87b752b71f0 100644 --- a/src/lib/provider-outbound.ts +++ b/src/lib/provider-outbound.ts @@ -28,12 +28,18 @@ export interface ProviderOutboundDependencies { * that forgets the seam fails closed, never open. */ isCanonicalUrl?: (name: string, url: string) => boolean; + /** Recheck caller-owned credential authority after DNS and immediately before transport. */ + beforeSend?: () => boolean; } export class ProviderOutboundPolicyError extends Error { override readonly name = "ProviderOutboundPolicyError"; } +export class ProviderOutboundSendCancelledError extends Error { + override readonly name = "ProviderOutboundSendCancelledError"; +} + function pickPinnedAddress(addresses: Array<{ address: string; family: number }>): { address: string; family: number } { return addresses.find(address => address.family === 4) ?? addresses[0]!; } @@ -157,6 +163,9 @@ async function providerOutboundRequest( // See PROVIDER_OUTBOUND_DEFAULT_USER_AGENT: this wrapper only carries proxy-originated // diagnostic traffic, so it identifies itself unless the caller already did. const init = withDefaultOutboundUserAgent(rawInit); + const assertSendAllowed = () => { + if (dependencies.beforeSend?.() === false) throw new ProviderOutboundSendCancelledError("provider credential changed before send"); + }; const postUrl = method === "POST" ? new URL(url) : undefined; if (postUrl?.protocol !== undefined && postUrl.protocol !== "https:") { throw new ProviderOutboundPolicyError("provider POST URL must use HTTPS"); @@ -195,6 +204,7 @@ async function providerOutboundRequest( }); if (destinationError) throw new ProviderOutboundPolicyError(destinationError); } + assertSendAllowed(); return provider.fetch(url, { ...init, method, redirect: "manual" }); } const parsed = postUrl ?? new URL(url); @@ -272,6 +282,7 @@ async function providerOutboundRequest( warnProxyDnsDegradationOnce(); // An explicit provider proxy stays pinned through the degradation too; re-inferring the // route from the environment here would quietly move the request to a different exit. + assertSendAllowed(); return configuredOutboundFetch(url, { ...init, method, redirect: "manual", ...(providerProxy ? { proxy: providerProxy } : {}), @@ -286,6 +297,7 @@ async function providerOutboundRequest( // An explicit provider proxy is always pinned, for the same reason and unconditionally: // the operator named the exit for this provider, so the environment must not re-decide it. const proxy = providerProxy ?? ((allowMihomoIpv6FakeIp && bindingProxy) ? bindingProxy : undefined); + assertSendAllowed(); return configuredOutboundFetch(url, { ...init, method, redirect: "manual", ...(proxy ? { proxy } : {}) }); } if (proxyApplies && resolved.privateNetwork) { @@ -300,6 +312,7 @@ async function providerOutboundRequest( context: "provider response", }; const pinned = pickPinnedAddress(resolved.addresses); + assertSendAllowed(); if (method === "POST") { return pinnedPost(url, pinned, (init as ProviderPostInit).body, init.signal ?? undefined, requestOptions); } diff --git a/src/oauth/anthropic-routing.ts b/src/oauth/anthropic-routing.ts index 60865a9d05a..e0a0db6970a 100644 --- a/src/oauth/anthropic-routing.ts +++ b/src/oauth/anthropic-routing.ts @@ -34,6 +34,7 @@ import type { OcxAccountPoolQuotaWindow, OcxAccountPoolRotationStrategy, OcxConf import { sweepExpiredOnWrite } from "../lib/state-store-sweeper"; import { retainedUtf8Bytes } from "../lib/admission"; import { routeCandidates, type AnthropicRouteDecision } from "./anthropic-model-routes"; +import { subscribeOAuthAccountPauseChanges } from "../lib/account-selection-events"; /** * The read side of a `Headers` object, so a caller can pass the live upstream response's @@ -57,6 +58,16 @@ export const ANTHROPIC_POOL_MAX_FAILOVERS_PER_REQUEST = 3; export type AnthropicAccountPoolConfig = NonNullable; +/** Local admission refusal across async dispatch; never treat it as provider reachability evidence. */ +export class AnthropicAccountCooldownError extends Error { + constructor(readonly retryAfterSeconds: number | null, readonly routePosition?: number) { + super(routePosition === undefined + ? "All Anthropic OAuth accounts are temporarily rate-limited" + : "Anthropic OAuth accounts for this model route are temporarily rate-limited"); + this.name = "AnthropicAccountCooldownError"; + } +} + /** * Where a cooldown's length came from. Same vocabulary as `CodexCooldownSource`, because it * answers the same question for the same reason: `retry-after` is upstream answering THIS @@ -266,7 +277,7 @@ export function getEligibleAnthropicAccounts(now = Date.now()): string[] { if (!set) return []; return set.accounts .filter(account => - account.needsReauth !== true + account.paused !== true && account.needsReauth !== true && !isCooled(account.id, now) && isPoolCredentialUsable(account.id, now)) .map(account => account.id); @@ -301,6 +312,8 @@ export function getEligibleAnthropicAccounts(now = Date.now()): string[] { const QUORUM_CACHE_TTL_MS = 2_000; let quorumCache: { value: boolean; readAt: number } | null = null; +// Pause changes eligibility, not health. Do not reset cooldowns or cancel sent turns. +subscribeOAuthAccountPauseChanges(provider => { if (provider === PROVIDER) quorumCache = null; }); /** * Whether a 429 has somewhere to go: two or more accounts that could serve traffic if asked. @@ -331,7 +344,7 @@ export function hasAnthropicFailoverQuorum(now = Date.now()): boolean { if (set) { let usable = 0; for (const account of set.accounts) { - if (account.needsReauth === true) continue; + if (account.paused === true || account.needsReauth === true) continue; if (!isPoolCredentialUsable(account.id, now)) continue; if (++usable >= 2) { value = true; break; } } @@ -355,8 +368,8 @@ export function getAnthropicPoolRetryAfterSeconds(now = Date.now(), decision: An && !getEligibleAnthropicAccounts(now).some(id => decision.accounts.includes(id)); let earliest: number | null = null; for (const account of set.accounts) { + if (account.paused === true || account.needsReauth === true || !isPoolCredentialUsable(account.id, now)) continue; if (decision && !fallbackExpanded && !decision.accounts.includes(account.id)) continue; - if (fallbackExpanded && (account.needsReauth === true || !isPoolCredentialUsable(account.id, now))) continue; const snap = getAnthropicAccountHealthSnapshot(account.id, now); if (!snap?.cooldownUntil) continue; if (earliest === null || snap.cooldownUntil < earliest) earliest = snap.cooldownUntil; @@ -486,6 +499,7 @@ export type AnthropicAccountSelectionReason = | "manual" | "fill-first" | "none" + | "paused" | "all-cooled"; export interface AnthropicAccountSelection { @@ -568,7 +582,7 @@ function pickUnboundStrategyAccount( /** * Resolve which Anthropic OAuth account should serve this session. - * When the pool is disabled, always returns the store's active account. + * When the pool is disabled, retain the active account unless the operator paused it. */ export function resolveAnthropicAccountForSession( sessionKey: string | null | undefined, @@ -579,6 +593,11 @@ export function resolveAnthropicAccountForSession( pruneExpiredAffinity(now); const set = getAccountSet(PROVIDER); if (!set || set.accounts.length === 0) return { accountId: null, reason: "none", ...(decision ? { routePosition: decision.position } : {}) }; + const scoped = decision && !decision.fallback + ? set.accounts.filter(account => decision.accounts.includes(account.id)) : set.accounts; + if (scoped.length > 0 && scoped.every(account => account.paused === true)) { + return { accountId: null, reason: "paused", routePosition: decision?.position }; + } if (manualPreference === undefined) { manualPreference = set.selectionRevision !== undefined @@ -586,23 +605,24 @@ export function resolveAnthropicAccountForSession( : null; } - if (!isAnthropicAccountPoolEnabled(config)) { - return { accountId: set.activeAccountId, reason: "pool-disabled" }; + const eligible = routeCandidates(getEligibleAnthropicAccounts(now), decision); + if (eligible.length === 0) { + // Pause, removal and reauthentication are not cooldown evidence. Classify only + // usable members of this strict route (or the ordinary pool after fallback widens). + const recoverable = set.accounts.filter(account => + (!decision || decision.fallback || decision.accounts.includes(account.id)) + && account.paused !== true && account.needsReauth !== true && isPoolCredentialUsable(account.id, now)); + const cooled = recoverable.length > 0 && recoverable.every(account => isCooled(account.id, now)); + if (cooled || decision) return { accountId: null, reason: cooled ? "all-cooled" : "none", routePosition: decision?.position }; } - const eligible = routeCandidates(getEligibleAnthropicAccounts(now), decision); - if (decision && eligible.length === 0) { - let cooled: boolean; - if (decision.fallback) { - // A removed route member is not a candidate, but fallback can still use the - // ordinary pool once those accounts recover. Only usable stored accounts count. - const ordinary = set.accounts.filter(account => - account.needsReauth !== true && isPoolCredentialUsable(account.id, now)); - cooled = ordinary.length > 0 && ordinary.every(account => isCooled(account.id, now)); - } else { - cooled = decision.accounts.every(id => set.accounts.some(account => account.id === id && isCooled(id, now))); + if (!isAnthropicAccountPoolEnabled(config)) { + const active = set.accounts.find(account => account.id === set.activeAccountId); + // Disabled proactive rotation does not authorize a paused slot or an entirely cooled pool. + if (active?.paused || isCooled(set.activeAccountId, now)) { + return { accountId: eligible[0] ?? null, reason: eligible.length > 0 ? "only-eligible" : "none" }; } - return { accountId: null, reason: cooled ? "all-cooled" : "none", routePosition: decision.position }; + return { accountId: set.activeAccountId, reason: "pool-disabled" }; } // A manual choice is a one-dispatch preference, not a lower-priority quota hint. @@ -626,7 +646,7 @@ export function resolveAnthropicAccountForSession( if (key) { const affined = sessionAffinity.get(key); if (affined && now - affined.lastUsedAt <= AFFINITY_IDLE_TTL_MS) { - const stillThere = set.accounts.some(a => a.id === affined.accountId && a.needsReauth !== true); + const stillThere = set.accounts.some(a => a.id === affined.accountId && a.paused !== true && a.needsReauth !== true); const stillUsable = stillThere && !isCooled(affined.accountId, now) && isPoolCredentialUsable(affined.accountId, now); if (stillUsable && eligible.includes(affined.accountId)) { @@ -694,13 +714,36 @@ export function resolveAnthropicAccountForSession( } if (!accountId) { - const anyCooled = set.accounts.some(a => (!decision || decision.accounts.includes(a.id)) && isCooled(a.id, now)); + const anyCooled = set.accounts.some(a => !a.paused && (!decision || decision.accounts.includes(a.id)) && isCooled(a.id, now)); return { accountId: null, reason: anyCooled ? "all-cooled" : "none", routePosition: decision?.position }; } return { accountId, reason, routePosition: decision?.position }; } +/** Shared local refusal policy for Responses and native Messages after asynchronous waits. */ +export async function resolveAnthropicDispatchAccountId( + config: OcxConfig, + sessionKey: string | null = null, + decision: AnthropicRouteDecision | null = null, +): Promise { + const now = Date.now(); + const selection = resolveAnthropicAccountForSession(sessionKey, config, now, decision); + if (selection.reason === "all-cooled") { + throw new AnthropicAccountCooldownError(getAnthropicPoolRetryAfterSeconds(now, decision), decision?.position); + } + if (selection.reason === "paused" || !selection.accountId || !getEligibleAnthropicAccounts(now).includes(selection.accountId)) { + const { OAuthAccountPausedError, OAuthLoginRequiredError } = await import("./index"); + const active = getAccountSet(PROVIDER)?.activeAccountId; + // Pool-off fresh admission resolves the active credential first. Preserve that + // paused-active refusal when no usable survivor could take over from it. + if (selection.reason === "paused" || (!isAnthropicAccountPoolEnabled(config) && active + && getAccountCredentialWithStatus(PROVIDER, active)?.paused)) throw new OAuthAccountPausedError(); + throw new OAuthLoginRequiredError(PROVIDER); + } + return selection.accountId; +} + export function bindAnthropicSessionAffinity( sessionKey: string | null | undefined, accountId: string, @@ -770,7 +813,10 @@ export function recordAnthropicAccount429( // just rate-limited is a different thing: it only ever runs after a refusal, and stranding a // 429 while a second logged-in account sits idle is a defect, not a configuration choice. // Presence is the activation rule, the same one an apiKeyPool of two keys already uses. - if (!isAnthropicAccountPoolEnabled(config) && !hasAnthropicFailoverQuorum(now)) return false; + // A sent turn may finish after its account is paused; its one remaining successor is + // still a valid reactive recovery even though the current unpaused quorum is now one. + if (!isAnthropicAccountPoolEnabled(config) && !hasAnthropicFailoverQuorum(now) + && !getAccountCredentialWithStatus(PROVIDER, failedAccountId)?.paused) return false; // Retry-After first: it is the header written FOR this decision. The rejected window's // reset is the fallback, because a 429 that omits Retry-After still carries it -- and @@ -825,7 +871,7 @@ export function commitAnthropicSelectionRouting( committed: OAuthAccountSelection, options: AnthropicSelectionRoutingOptions, ): boolean { - if (committed.accountId !== accountId || (options.routeDecision + if (getAccountCredentialWithStatus(PROVIDER, accountId)?.paused || committed.accountId !== accountId || (options.routeDecision && !routeCandidates(getEligibleAnthropicAccounts(), options.routeDecision).includes(accountId))) return false; const current = captureOAuthAccountSelection(PROVIDER); if (current?.accountId !== committed.accountId || current.revision !== committed.revision) return false; @@ -869,7 +915,12 @@ export function resetAnthropicRoutingForManualSelection(accountId: string): void * as quota probes). */ export async function getAnthropicPoolAccessToken(accountId: string): Promise { - const stored = getAccountCredential(PROVIDER, accountId); + const row = getAccountCredentialWithStatus(PROVIDER, accountId); + if (row?.paused) { + const { OAuthAccountPausedError } = await import("./index"); + throw new OAuthAccountPausedError(); + } + const stored = row?.credential; if (!stored) { const { OAuthLoginRequiredError } = await import("./index"); throw new OAuthLoginRequiredError(PROVIDER); @@ -886,6 +937,10 @@ export async function getAnthropicPoolAccessToken(accountId: string): Promise { const accessToken = await getAnthropicPoolAccessToken(accountId); const row = getAccountCredentialWithStatus(PROVIDER, accountId); + if (row?.paused) { + const { OAuthAccountPausedError } = await import("./index"); + throw new OAuthAccountPausedError(); + } if (!row || row.needsReauth || row.credential.access !== accessToken || row.credential.expires <= Date.now()) { throw new Error("Anthropic pool credential changed during account selection"); @@ -899,6 +954,7 @@ export async function getAnthropicPoolAccessSnapshot(accountId: string): Promise */ export function canRefreshAnthropicPoolAccount(accountId: string): boolean { const set = getAccountSet(PROVIDER); + if (set?.accounts.find(account => account.id === accountId)?.paused) return false; const cred = getAccountCredential(PROVIDER, accountId); if (!cred) return false; if (cred.source !== "local-cli") return true; diff --git a/src/oauth/index.ts b/src/oauth/index.ts index db7e5b7c5ad..3d6a0efcf68 100644 --- a/src/oauth/index.ts +++ b/src/oauth/index.ts @@ -925,8 +925,10 @@ export async function refreshAnthropicAccountWithLock( const now = deps.now ?? Date.now; const guard = await (deps.intentLock ?? createOAuthRefreshIntentLock(provider, accountId)).acquire(); try { - const stored = getAccountCredential(provider, accountId); - if (!stored) throw new OAuthLoginRequiredError(provider); + const row = getAccountCredentialWithStatus(provider, accountId); + if (!row) throw new OAuthLoginRequiredError(provider); + if (row.paused) throw new OAuthAccountPausedError(); + const stored = row.credential; const account = getAccountSet(provider)?.accounts.find(candidate => candidate.id === accountId); const generation = credentialGeneration(stored); let pendingIntent = readOAuthRefreshIntent(provider, accountId); diff --git a/src/oauth/store.ts b/src/oauth/store.ts index 5c9fff453fe..7d085c96034 100644 --- a/src/oauth/store.ts +++ b/src/oauth/store.ts @@ -1380,4 +1380,6 @@ export async function markAccountNeedsReauth( } export async function mergeAccountCredential(provider:string,accountId:string,credential:OAuthCredentials,opts:{expectedGeneration?:string;afterPrePersistRead?:()=>void|Promise;assertOwnership?: (store: AuthStore) => void}={}):Promise<{superseded:false}|{superseded:true;stored:OAuthCredentials}>{const safe=normalizeCredential(credential);if(!safe)throw new Error("Refusing to persist invalid OAuth credential");return await mutateStore(async store=>{await opts.afterPrePersistRead?.();const account=store[provider]?.accounts.find(x=>x.id===accountId);if(!account)throw new Error(`OAuth account disappeared before persist: ${provider}`);if(opts.expectedGeneration!==undefined&&credentialGeneration(account.credential)!==opts.expectedGeneration)return{superseded:true,stored:account.credential};opts.assertOwnership?.(store);account.credential=safe;delete account.needsReauth;return{superseded:false};},[provider,accountId,safe,opts.expectedGeneration]);} -export async function markAccountNeedsReauthIfGeneration(provider:string,accountId:string,generation:string,writerGeneration=captureConfigGeneration()):Promise{const key=oauthAccountKey(provider,accountId);if(writerGeneration{const account=store[provider]?.accounts.find(x=>x.id===accountId);if(!account?.credential||credentialGeneration(account.credential)!==generation)return false;if(writerGeneration{const key=oauthAccountKey(provider,accountId);if(writerGeneration{const account=store[provider]?.accounts.find(x=>x.id===accountId);if(!account?.credential||account.paused||credentialGeneration(account.credential)!==generation)return false;if(writerGeneration { const id = getAccountSet(provider)?.activeAccountId; if (!id) return null; @@ -439,7 +438,6 @@ async function fetchAccountQuota( return provider !== "kiro" || joined.identity === kiroIdentity ? joined : fetchAccountQuota(provider, accountId, true, providerConfig); } - const epoch = explicitAccountEpoch; const probe = (async (): Promise => { let diagnosticIdentity: string | undefined; @@ -469,6 +467,8 @@ async function fetchAccountQuota( quota = result.kind === "available" ? result.quota : null; if (result.kind === "unavailable") quotaFailure = result.failure; } else if (provider === "anthropic") { + const row = getAccountCredentialWithStatus(provider, accountId); + if (!row || row.paused || row.needsReauth || row.credential.access !== token) return { ts: Date.now(), quota: null, unavailable: true }; quota = await fetchAnthropicUsageQuota(token); } else { return { ts: Date.now(), quota: null, unavailable: true }; @@ -524,7 +524,6 @@ async function fetchAccountQuota( accountQuotaInflight.set(key, probe); return probe; } - /** * Per-account quota rows for a provider's logged-in accounts. Probes run in parallel; a * single failing account never blocks the others. diff --git a/src/providers/quota/vendor-probes-oauth.ts b/src/providers/quota/vendor-probes-oauth.ts index 56ad530a7b8..c4c4d006be1 100644 --- a/src/providers/quota/vendor-probes-oauth.ts +++ b/src/providers/quota/vendor-probes-oauth.ts @@ -1,7 +1,7 @@ import { effectiveCodexAuthAccountId, fetchMainAccountInfoSnapshot, listCodexAuthAccountsSnapshot } from "../../codex/auth-api"; import { MAIN_CODEX_ACCOUNT_ID } from "../../codex/main-account"; import { getValidAccessToken } from "../../oauth"; -import { getAccountCredential, getAccountSet } from "../../oauth/store"; +import { captureOAuthAccountSelection, getAccountCredential, getAccountCredentialWithStatus, getAccountSet } from "../../oauth/store"; import { hydrateKiroAccountState, persistKiroAccountState } from "../kiro-account-state-disk"; import { kiroProbeCurrent, kiroProbeIdentity } from "./kiro-account-probe"; import { fetchMuseKeyQuotaSnapshot } from "../muse-key-quota"; @@ -18,6 +18,7 @@ import { aggregateCodexPoolCapacity, CODEX_CAPACITY_MAX_QUOTA_AGE_MS, type Codex import { asRecord, normalizePercent, normalizeResetAt, readQuotaJson, REQUEST_TIMEOUT_MS, toFiniteNumber } from "../quota-wire"; import { providerCodexAccountMode } from "../registry"; import { + accountReportCurrent, TERMINAL_QUOTA_FAILURE, hasQuotaRows, providerLabel, @@ -339,7 +340,8 @@ export async function fetchAnthropicUsageQuota(accessToken: string): Promise { // Capture the account we intend to probe before awaiting — a mid-flight active // switch must not seed the wrong account's cache with this response. - const probedAccountId = getAccountSet("anthropic")?.activeAccountId; + const selection = captureOAuthAccountSelection("anthropic"); + const probedAccountId = selection?.accountId; const probedAccountKey = probedAccountId ? accountCacheKey("anthropic", probedAccountId) : null; const writerGeneration = captureConfigGeneration(); let accessToken: string; @@ -348,6 +350,10 @@ export async function fetchAnthropicQuota(provider: string): Promise getAccountSet("anthropic")?.activeAccountId === probedAccountId); + } + return result; } /** diff --git a/src/server/management/oauth-account-routes.ts b/src/server/management/oauth-account-routes.ts index d0867373dce..836105e1817 100644 --- a/src/server/management/oauth-account-routes.ts +++ b/src/server/management/oauth-account-routes.ts @@ -396,7 +396,8 @@ export async function handleOauthAccountRoutes(ctx: ManagementContext): Promise< const { isGenericFailoverProvider, kiroAutoSelection } = await import("../../oauth/generic-account-failover"); const effectiveProvider = genericOAuthProviderConfig(provider, config); const supportsPause = effectiveProvider !== undefined - && isGenericFailoverProvider(provider, effectiveProvider); + && (provider === "anthropic" && effectiveProvider.authMode === "oauth" + || isGenericFailoverProvider(provider, effectiveProvider)); const { oauthAccountHealthFields, projectOAuthAccountHealth, @@ -504,7 +505,8 @@ export async function handleOauthAccountRoutes(ctx: ManagementContext): Promise< const { isGenericFailoverProvider } = await import("../../oauth/generic-account-failover"); const effectiveProvider = genericOAuthProviderConfig(provider, config); - if (!effectiveProvider || !isGenericFailoverProvider(provider, effectiveProvider)) { + if (!effectiveProvider || !(provider === "anthropic" && effectiveProvider.authMode === "oauth" + || isGenericFailoverProvider(provider, effectiveProvider))) { return jsonResponse({ error: "account pause is not supported for this OAuth provider" }, 400); } @@ -513,8 +515,13 @@ export async function handleOauthAccountRoutes(ctx: ManagementContext): Promise< if (result.status === "not-found") return jsonResponse({ error: "account not found" }, 404); if (result.activeAccountChanged) { - const { genericPoolKey, seedPoolRotationAccount } = await import("../../oauth/pool-kernel"); - seedPoolRotationAccount(genericPoolKey(provider), result.activeAccountId); + if (provider === "anthropic") { + const { resetAnthropicRoutingForManualSelection } = await import("../../oauth/anthropic-routing"); + resetAnthropicRoutingForManualSelection(result.activeAccountId); + } else { + const { genericPoolKey, seedPoolRotationAccount } = await import("../../oauth/pool-kernel"); + seedPoolRotationAccount(genericPoolKey(provider), result.activeAccountId); + } const { clearModelCache } = await import("../../codex/model-cache"); const { clearGatherRoutedModelsInflight } = await import("../../codex/catalog"); clearModelCache(provider); diff --git a/src/server/messages-native-oauth.ts b/src/server/messages-native-oauth.ts index 0369ce55a7b..d6b5f3faa5b 100644 --- a/src/server/messages-native-oauth.ts +++ b/src/server/messages-native-oauth.ts @@ -4,8 +4,8 @@ * * Credential. The account is the one the existing OAuth selection chooses for this request, by * the same steps the Responses pipeline's transport takes for an unpooled Anthropic OAuth route - * (`prepareResponsesTransport`): capture the committed selection, resolve the active account's - * access snapshot (refreshing it through the OAuth owner when it is due), and commit that + * (`prepareResponsesTransport`): capture the committed selection, resolve an eligible account's + * access snapshot (including pause/cooldown recovery), and commit that * proposal against the captured selection so a concurrent manual switch wins. Nothing here picks * an account of its own, and nothing runs at planning time: the planner and eligibility read * config only, and this module is reached from the lane's dispatch alone. @@ -13,7 +13,8 @@ * Pools. A pooled account set (the opt-in Anthropic account pool, or two or more usable accounts, * which turns on reactive 429 rotation) is declined before this lane is chosen * (`oauth-account-pool`). Should one appear between that decision and dispatch, resolution fails - * closed rather than serve a pooled account without the pool's rotation and affinity. + * closed rather than serve a pooled account without the pool's rotation and affinity. This + * lane-change 409 is distinct from the shared local authentication/pause/cooldown refusals. * * Tool names. An OAuth request carries client tool names under the Claude OAuth prefix, as the * adapter sends them; the answer's `tool_use` names are mapped back here, for exactly the names @@ -21,10 +22,12 @@ * * No token, account id or body content is logged or returned in an error message. */ -import { getValidAccessTokenSnapshot, type OAuthAccessSnapshot } from "../oauth"; +import type { OAuthAccessSnapshot } from "../oauth"; import { commitAnthropicSelectionRouting, getAnthropicPoolAccessSnapshot, + getAnthropicAccountHealthSnapshot, + resolveAnthropicDispatchAccountId, hasAnthropicFailoverQuorum, isAnthropicAccountPoolEnabled, } from "../oauth/anthropic-routing"; @@ -72,17 +75,14 @@ function pooled(config: OcxConfig): boolean { * selection could not be committed. */ export async function resolveNativeOAuthBinding(config: OcxConfig): Promise { - if (pooled(config)) throw new NativeOAuthSelectionChangedError(); - let selection: Selection | null = captureOAuthAccountSelection(PROVIDER); - let candidate = await getValidAccessTokenSnapshot(PROVIDER); for (let attempt = 0; attempt < MAX_SELECTION_ATTEMPTS; attempt++) { - if (!selection) break; - if (candidate.accountId !== selection.accountId) { - // The active account moved between capture and resolution: serve the committed one. - selection = captureOAuthAccountSelection(PROVIDER); - if (!selection) break; - candidate = await getAnthropicPoolAccessSnapshot(selection.accountId); - } + const selection = captureOAuthAccountSelection(PROVIDER); + // Resolve eligibility before the lane guard: a local 401/403/429 must not turn + // into a generic lane-change 409 merely because it appeared during pacing. + const accountId = await resolveAnthropicDispatchAccountId(config); + if (pooled(config)) throw new NativeOAuthSelectionChangedError(); + if (!selection) continue; + const candidate = await getAnthropicPoolAccessSnapshot(accountId); const committed = await commitOAuthAccountSelection(PROVIDER, candidate.accountId, { expectedSelection: selection, expectedCredentialGeneration: candidate.generation, @@ -93,14 +93,12 @@ export async function resolveNativeOAuthBinding(config: OcxConfig): Promise Date.now() + && !!row && !row.paused && !row.needsReauth && row.credential.expires > Date.now() + && !getAnthropicAccountHealthSnapshot(binding.snapshot.accountId) && credentialGeneration(row.credential) === binding.snapshot.generation; } diff --git a/src/server/messages-native.ts b/src/server/messages-native.ts index d03b6a33299..b593af2d1b4 100644 --- a/src/server/messages-native.ts +++ b/src/server/messages-native.ts @@ -58,9 +58,9 @@ import { selectProactiveApiKeyTransport, transientRetryPolicyFor, } from "../providers/key-failover"; -import { stampApiKeyAccountLabel, stampOAuthAccountLabel } from "../providers/label"; -import { publicOAuthAuthenticationErrorMessage } from "../oauth"; -import { hasAnthropicFailoverQuorum } from "../oauth/anthropic-routing"; +import { stampApiKeyAccountLabel } from "../providers/label"; +import { OAuthAccountPausedError, OAuthLoginRequiredError, publicOAuthAuthenticationErrorMessage } from "../oauth"; +import { AnthropicAccountCooldownError, formatAnthropicProviderForLog, hasAnthropicFailoverQuorum } from "../oauth/anthropic-routing"; import { resolveProtocolSettings } from "../protocols/settings"; import { addProtocolEntryReason, markProtocolBlocked } from "../protocols/trace"; import type { OcxProviderTransport } from "../providers/xai-transport"; @@ -328,6 +328,16 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) finishLog(status, safeMessage); return anthropicErrorResponse(status, safeMessage, type, code); }; + const localOAuthFailure = (error: unknown): Response | null => { + if (error instanceof AnthropicAccountCooldownError) { + const response = fail(429, error.message, "rate_limit_error"); + if (error.retryAfterSeconds !== null) response.headers.set("retry-after", String(error.retryAfterSeconds)); + return response; + } + if (error instanceof OAuthAccountPausedError) return fail(403, publicOAuthAuthenticationErrorMessage(error), "permission_error"); + if (error instanceof OAuthLoginRequiredError) return fail(401, publicOAuthAuthenticationErrorMessage(error), "authentication_error"); + return null; + }; try { await prepareNativeBody(body, req.signal); @@ -366,6 +376,8 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) cleanupAbort(); upstream.abort(); if (req.signal.aborted) return fail(499, "Client cancelled request", "api_error"); + const localRefusal = localOAuthFailure(error); + if (localRefusal) return localRefusal; if (error instanceof NativeOAuthSelectionChangedError) return fail(409, error.message, "api_error"); return fail(401, publicOAuthAuthenticationErrorMessage(error), "authentication_error"); } @@ -376,7 +388,7 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) } let activeProvider: OcxProviderConfig = oauthBinding ? oauthProvider(oauthBinding) : route.provider; stampApiKeyAccountLabel(logCtx, route.providerName, activeProvider); - if (oauthBinding) stampOAuthAccountLabel(logCtx, route.providerName, activeProvider, oauthBinding.snapshot.accountId); + if (oauthBinding) logCtx.provider = formatAnthropicProviderForLog(route.providerName, oauthBinding.snapshot.accountId, config); const spendTracker = attachRequestSpendTracker(req, logCtx); let activeRequest: AnthropicMessagesPassthroughRequest; let retainedRequestBytes = 0; @@ -460,14 +472,17 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) if (oauthBinding) { // The OAuth twin of the key check below: re-resolve through the same selection // owner when the committed account or its credential moved since the build. - if (!nativeOAuthBindingIsCurrent(oauthBinding)) { + for (let attempt = 0; !nativeOAuthBindingIsCurrent(oauthBinding); attempt++) { + if (attempt >= 3) throw new NativeOAuthSelectionChangedError(); try { oauthBinding = await resolveNativeOAuthBinding(config); - } catch { + } catch (error) { + if (error instanceof OAuthAccountPausedError || error instanceof OAuthLoginRequiredError + || error instanceof AnthropicAccountCooldownError) throw error; throw new NativeOAuthSelectionChangedError(); } rebuildFor(oauthProvider(oauthBinding)); - stampOAuthAccountLabel(logCtx, route.providerName, activeProvider, oauthBinding.snapshot.accountId); + logCtx.provider = formatAnthropicProviderForLog(route.providerName, oauthBinding.snapshot.accountId, config); } } else if (!providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider)) { const current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); @@ -569,6 +584,8 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) finishLog(429); return refusal; } + const localRefusal = localOAuthFailure(sendError); + if (localRefusal) return localRefusal; if (sendError instanceof NativeOAuthSelectionChangedError) return fail(409, sendError.message, "api_error"); if (sendError instanceof NativeOpaqueStateRefusal) { logCtx.errorCode = "unsupported_feature"; diff --git a/src/server/responses/adapter-dispatch.ts b/src/server/responses/adapter-dispatch.ts index f0df55c1bd5..ba7f509eba5 100644 --- a/src/server/responses/adapter-dispatch.ts +++ b/src/server/responses/adapter-dispatch.ts @@ -35,6 +35,7 @@ import { fetchWithResetRetry, applyUpstreamRecoveryInit, SendBudgetExhaustedError, + UpstreamRetryEvidenceError, prepareSameTarget429Wait, sleepWithAbort, } from "../../lib/upstream-retry"; @@ -54,6 +55,7 @@ import { resolveCopilotApiBaseUrl } from "../../oauth/github-copilot"; import { resolveWireProtocolOverride } from "../adapter-resolve"; import { bindRouteReasoningReplayScope } from "./core-replay"; import { + AnthropicAccountCooldownError, ANTHROPIC_POOL_MAX_FAILOVERS_PER_REQUEST, rotateAnthropicAccountOn429, recordAnthropicAccount429, @@ -383,6 +385,18 @@ export async function prepareAdapterExchange( cleanupUpstreamAbort(); upstream.abort(); if (options.abortSignal?.aborted) return clientCancelledResponse(); + const refusal = err instanceof UpstreamRetryEvidenceError ? err.cause : err; + // A pause committed during pacing is local admission policy, not a failed upstream. + if (refusal instanceof OAuthAccountPausedError) { + return formatErrorResponse(403, "permission_error", publicOAuthAuthenticationErrorMessage(refusal)); + } + if (refusal instanceof OAuthLoginRequiredError) { + return formatErrorResponse(401, "authentication_error", publicOAuthAuthenticationErrorMessage(refusal)); + } + if (refusal instanceof AnthropicAccountCooldownError) { + return formatErrorResponse(429, "rate_limit_error", refusal.message, + refusal.retryAfterSeconds === null ? undefined : { retryAfter: String(refusal.retryAfterSeconds) }); + } // A budget refusal is a decision this process made, not an upstream fault. Reporting it as // 502 does more than mislabel it: the Codex client retries 5xx and does not retry a 429, so // blaming the provider makes the caller send the whole turn again -- the amplification this @@ -589,6 +603,17 @@ export async function prepareAdapterExchange( if (options.abortSignal?.aborted) { return { failed: clientCancelledResponse() }; } + const refusal = err instanceof UpstreamRetryEvidenceError ? err.cause : err; + if (refusal instanceof OAuthAccountPausedError) { + return { failed: formatErrorResponse(403, "permission_error", publicOAuthAuthenticationErrorMessage(refusal)) }; + } + if (refusal instanceof OAuthLoginRequiredError) { + return { failed: formatErrorResponse(401, "authentication_error", publicOAuthAuthenticationErrorMessage(refusal)) }; + } + if (refusal instanceof AnthropicAccountCooldownError) { + return { failed: formatErrorResponse(429, "rate_limit_error", refusal.message, + refusal.retryAfterSeconds === null ? undefined : { retryAfter: String(refusal.retryAfterSeconds) }) }; + } // Same rule on the recovery leg: the ladder refused to send again, so the answer names // this proxy rather than the provider it never reached. if (err instanceof SendBudgetExhaustedError) { diff --git a/src/server/responses/passthrough-dispatch.ts b/src/server/responses/passthrough-dispatch.ts index a331cbaeaf8..ca1f92e473c 100644 --- a/src/server/responses/passthrough-dispatch.ts +++ b/src/server/responses/passthrough-dispatch.ts @@ -137,7 +137,8 @@ import { resolveWireProtocolOverride } from "../adapter-resolve"; import { refreshPoolForwardAuth, refreshNativeMainForwardAuth, withClaudeNativeSession } from "./core-auth"; import { bindRouteReasoningReplayScope } from "./core-replay"; import type { OAuthAccessSnapshot } from "../../oauth"; -import { publicOAuthAuthenticationErrorMessage } from "../../oauth"; +import { OAuthAccountPausedError, OAuthLoginRequiredError, publicOAuthAuthenticationErrorMessage } from "../../oauth"; +import { AnthropicAccountCooldownError } from "../../oauth/anthropic-routing"; import { resolveCopilotApiBaseUrl } from "../../oauth/github-copilot"; import { hasEligibleGenericOAuthFailoverTarget, @@ -881,7 +882,23 @@ export async function preparePassthroughExchange( releaseCodexAuthContextProbeLease(admissionState.authCtx); return formatErrorResponse(429, "request_send_budget_exhausted", err.message); } - const localRefusal = mapCodexAuthContextErrorToResponse(unwrapUpstreamRetryEvidenceError(err), { + const refusal = unwrapUpstreamRetryEvidenceError(err); + // Pacing may outlive the selected account's admission. No fetch occurred, so do + // not turn an operator pause into a 502 or charge it to host/account health. + if (refusal instanceof OAuthAccountPausedError || refusal instanceof OAuthLoginRequiredError || refusal instanceof AnthropicAccountCooldownError) { + releaseUpstreamHostAdmission(nativeHostState.lease); + nativeHostState.lease = null; + releaseCodexAuthContextProbeLease(admissionState.authCtx); + if (refusal instanceof OAuthLoginRequiredError) { + return formatErrorResponse(401, "authentication_error", publicOAuthAuthenticationErrorMessage(refusal)); + } + if (refusal instanceof AnthropicAccountCooldownError) { + return formatErrorResponse(429, "rate_limit_error", refusal.message, + refusal.retryAfterSeconds === null ? undefined : { retryAfter: String(refusal.retryAfterSeconds) }); + } + return formatErrorResponse(403, "permission_error", publicOAuthAuthenticationErrorMessage(refusal)); + } + const localRefusal = mapCodexAuthContextErrorToResponse(refusal, { now: Date.now(), accountSelector: route.codexAccountNamespace, }); if (localRefusal) { diff --git a/src/server/responses/request-transport.ts b/src/server/responses/request-transport.ts index 206a923963d..c340c0612a9 100644 --- a/src/server/responses/request-transport.ts +++ b/src/server/responses/request-transport.ts @@ -15,8 +15,10 @@ import type { AdapterEvent, OcxParsedRequest, OcxProviderConfig, OcxUsage } from import type { AnthropicAccountSelectionReason } from "../../oauth/anthropic-routing"; import { resolveAnthropicModelRoute, routeCandidates, type AnthropicRouteDecision } from "../../oauth/anthropic-model-routes"; import { + resolveAnthropicDispatchAccountId, isAnthropicAccountPoolEnabled, getAnthropicPoolAccessSnapshot, + getAnthropicAccountHealthSnapshot, getEligibleAnthropicAccounts, commitAnthropicSelectionRouting, formatAnthropicProviderForLog, @@ -337,7 +339,8 @@ export async function prepareResponsesTransport( const selected = captureOAuthAccountSelection(route.providerName); const row = getAccountCredentialWithStatus(route.providerName, binding.snapshot.accountId); return selected?.accountId === binding.selection.accountId && selected?.revision === binding.selection.revision - && !!row && !row.needsReauth && row.credential.expires > Date.now() + && !!row && !row.paused && !row.needsReauth && row.credential.expires > Date.now() + && (route.providerName !== "anthropic" || !getAnthropicAccountHealthSnapshot(binding.snapshot.accountId)) && credentialGeneration(row.credential) === binding.snapshot.generation; }; const resolveSelectionAdapter = (provider: OcxProviderConfig, retention = config.cacheRetention): ProviderAdapter => { @@ -399,7 +402,16 @@ export async function prepareResponsesTransport( }; const refreshDispatchAdapter = async (requestParsed: OcxParsedRequest): Promise => { if (route.provider.authMode === "oauth") { - if (!servingOAuthSnapshot || !await applyFailoverSnapshot(servingOAuthSnapshot, requestParsed)) { + // Pacing may outlive admission. Preserve local pause/cooldown reasons even when + // a strict route rejects the old proposal before another bearer can be resolved. + let candidate = servingOAuthSnapshot; + if (route.providerName === "anthropic") { + oauthSelection = captureOAuthAccountSelection(route.providerName); + const accountId = await resolveAnthropicDispatchAccountId(config, anthropicSessionKey, anthropicRouteDecision); + candidate = await getAnthropicPoolAccessSnapshot(accountId); + } + if (!candidate || !await applyFailoverSnapshot(candidate, requestParsed)) { + if (route.providerName === "anthropic") await resolveAnthropicDispatchAccountId(config, anthropicSessionKey, anthropicRouteDecision); throw new Error("OAuth account selection changed before dispatch"); } } else { @@ -561,6 +573,7 @@ export async function prepareResponsesTransport( anthropicRouteDecision = routeResult.decision; const selection = resolveAnthropicAccountForSession(anthropicSessionKey, config, Date.now(), anthropicRouteDecision); if (!selection.accountId) { + if (selection.reason === "paused") return formatErrorResponse(403, "permission_error", "Anthropic OAuth accounts are paused. Resume an account in account settings and retry."); // Route names may resemble account IDs; log only the matched rule position. if (anthropicRouteDecision) console.warn(`[anthropic-pool] route:#${anthropicRouteDecision.position} ${selection.reason}; answering locally`); if (selection.reason === "all-cooled") { diff --git a/structure/INDEX.md b/structure/INDEX.md index a0dbbf3ea74..2f5e29efc0a 100644 --- a/structure/INDEX.md +++ b/structure/INDEX.md @@ -57,6 +57,7 @@ Per-vendor contracts and the adapter authority that constructs them. | Doc | Scope | | --- | --- | | [`providers-and-adapters.md`](providers-and-adapters.md) | Provider and adapter selection, the adapter inventory, live model discovery, and the hosted-search continuation bridge. | +| [`providers/anthropic-account-pool.md`](providers/anthropic-account-pool.md) | Anthropic OAuth account pause, model routes, and quota labels. | | [`providers/openai-tiers.md`](providers/openai-tiers.md) | Pool/Direct account modes, API-key separation, and the public provider and quota contract. | | [`providers/openai-accounts.md`](providers/openai-accounts.md) | Migration and restore, wire identity, store concurrency, pool ordering and exclusions, quota observations, and account-bound retention. | | [`providers/cursor.md`](providers/cursor.md) | Cursor native exec, parameterized models, checkpoints, and active-context usage. | @@ -134,10 +135,10 @@ A source area can be described by more than one doc, because these docs are orga | `src/lab/` | [`runtime.md`](runtime.md)
[`adapters/compatibility-lab.md`](adapters/compatibility-lab.md) | | `src/lib/` | [`overview.md`](overview.md)
[`runtime.md`](runtime.md)
[`transports/byte-accounting.md`](transports/byte-accounting.md)
[`transports/responses-wire-shapes.md`](transports/responses-wire-shapes.md)
[`transports/responses-failover.md`](transports/responses-failover.md)
[`transports/responses-spend.md`](transports/responses-spend.md)
[`transports/inventory.md`](transports/inventory.md)
[`gui-and-management-api.md`](gui-and-management-api.md)
[`dashboard-and-usage.md`](dashboard-and-usage.md)
[`clients/integrations.md`](clients/integrations.md)
[`ops/service-and-sidecars.md`](ops/service-and-sidecars.md)
[`ops/docs-and-release.md`](ops/docs-and-release.md) | | `src/link/` | [`remote-link.md`](remote-link.md) | -| `src/oauth/` | [`runtime.md`](runtime.md)
[`transports/inventory.md`](transports/inventory.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`providers/xai-grok.md`](providers/xai-grok.md) | +| `src/oauth/` | [`runtime.md`](runtime.md)
[`transports/inventory.md`](transports/inventory.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`providers/anthropic-account-pool.md`](providers/anthropic-account-pool.md)
[`providers/xai-grok.md`](providers/xai-grok.md) | | `src/plugins/` | [`ops/plugins.md`](ops/plugins.md) | | `src/protocols/` | [`data-planes/protocol-paths.md`](data-planes/protocol-paths.md) | -| `src/providers/` | [`runtime.md`](runtime.md)
[`subagents.md`](subagents.md)
[`transports/inventory.md`](transports/inventory.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`providers/xai-grok.md`](providers/xai-grok.md) | +| `src/providers/` | [`runtime.md`](runtime.md)
[`subagents.md`](subagents.md)
[`transports/inventory.md`](transports/inventory.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`providers/anthropic-account-pool.md`](providers/anthropic-account-pool.md)
[`providers/xai-grok.md`](providers/xai-grok.md) | | `src/quota/` | [`dashboard-and-usage.md`](dashboard-and-usage.md) | | `src/reasoning-effort.ts` | [`runtime.md`](runtime.md) | | `src/remote-control/` | [`remote-workspace.md`](remote-workspace.md) | diff --git a/structure/catalog.md b/structure/catalog.md index 9951e1c979f..930e1410c3d 100644 --- a/structure/catalog.md +++ b/structure/catalog.md @@ -34,7 +34,7 @@ may fill an absent static one only inside that call-local projection — the fro unchanged and never widened — max input never exceeds the resolved context window, and the projection mutates neither input. Gather admission freezes an enriched provider snapshot before discovery; per-model hint projection resolves from that snapshot, so no post-admission -registry read can change a running gather flight. +registry read can change a running gather flight. Anthropic OAuth discovery in `src/codex/catalog/provider-models.ts` rechecks the live pause, selection, and bearer generation at dispatch and after outbound DNS; a superseded flight degrades to configured models without sending. Policy is keyed by the final upstream wire model. Public alias and virtual-model identities remain diagnostic/catalog provenance and must be resolved before policy capture. Exact nonempty explicit diff --git a/structure/decisions/ADR-6013-anthropic-account-pause.md b/structure/decisions/ADR-6013-anthropic-account-pause.md new file mode 100644 index 00000000000..ce0d904d457 --- /dev/null +++ b/structure/decisions/ADR-6013-anthropic-account-pause.md @@ -0,0 +1,44 @@ +# ADR-6013 — decision recorded under "Anthropic account pause" + +- Contract owner: [Anthropic account pool](../providers/anthropic-account-pool.md#anthropic-account-pause) + +## Decision Log + +- Purpose and intent: Temporarily exclude one Claude OAuth account through the API, CLI and + dashboard without deleting its login or conflating operator intent with runtime health. +- Existing implementation and constraints: Anthropic has its own manual/affinity/quota/model + route selector; generic OAuth pause already owns persistence, DTOs and dashboard controls. + Request preparation and refresh can await while the operator changes the selection. +- Alternatives considered: Add `pausedAnthropicAccountIds` to config, implement a second + mutation endpoint, or merely expose the generic pause switch without selector changes. +- Selected approach: Reuse the protected auth store's optional `paused: true`, shared locked + mutation, endpoint and translated controls. Filter Anthropic eligibility at each selector + and validate live pause state at credential, commit and physical-send boundaries. Persist + successful in-flight refresh rotation, but leave paused health unchanged on late failure. + Keep typed local refusals across adapter/passthrough error projection after pacing, and + classify routed cooldown using only usable unpaused members so resume and retry guidance + remain distinct from login errors. + Cooldown refusals carry the route position and retry seconds across pacing and dispatch, + including disabled proactive pools; they never count as an upstream reachability failure. + If only reauthentication-required or unusable credentials remain after pause, dispatch + matches fresh admission: 401 with pooling enabled, or the existing 403 paused-active + refusal when pooling is disabled. Neither refusal changes account or host health. + Native Messages uses the same dispatch eligibility resolver, checks cooldown after each + await, and preserves typed 401/403/429 responses. Its existing 409 remains reserved for a + healthy roster becoming pooled and requiring the bridge lane, or repeated selection races. +- Pool-off policy evidence: #6013's accepted contract retains reactive 429 successors with + pooling disabled. The existing `anthropicAccountPool.enabled` reference likewise gates + only proactive routing, not 429 recovery. Cooldown is recorded from an upstream 429; using + a healthy successor for a paused or already-cooled active account is recovery, not quota + rotation. A healthy active account remains selected regardless of quota/strategy settings. + Recovery proposals say `only-eligible`; native request logs carry the committed account's + ordinal rather than a provider-only label. The native/bridge pacing matrix pins this policy. +- Why this approach: One authoritative row avoids config/auth split-write races and automatic + cleanup follows account deletion. A UI-only toggle would still allow affinity, pool-off + failover or a pre-wait bearer to select the paused account. +- Benefits, tradeoffs, and impact: No new config migration or translation keys; restart and + reauthentication preserve pause. Already-sent turns finish normally. Additional store reads + are limited to existing admission boundaries, and all-paused pools fail locally with 403. + Per-account thresholds remain a separate slice of #6013. This is the TypeScript `dev` + implementation; the maintainer must assess/port the corresponding Go paths on `dev2-go` + at integration, rather than treating this local commit as cross-branch completion. diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index 205de32668e..384fa6fa2cb 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -1,5 +1,13 @@ # GUI And Management API +Anthropic account rows now expose the shared boolean `paused` DTO and use the existing +`PUT /api/oauth/accounts/pause` body `{ provider, accountId, paused }`. The dashboard's +`ProviderAuthPanel` and `useProviderAccountPools` reuse the translated pause/resume actions, +disabled manual selection, and separate mutation-versus-refresh failure notices. CLI +`ocx account pause|resume anthropic [--json]` uses that same endpoint. +Only OAuth routes support this operation; API-key routes remain outside this switch. +Selection and persistence semantics: [Anthropic account pause](providers/anthropic-account-pool.md#anthropic-account-pause). + `src/server/management/oauth-account-routes.ts` exposes Anthropic `routes` through both unified `/api/pool/settings` and legacy `/api/oauth/accounts/pool`. Omitted rules survive other setting writes, `null` clears them, and other pool kinds reject supplied rules. The unified DTO declares `routes` supported only for Anthropic and reports null otherwise. Both Anthropic settings GETs validate saved rules before projection: malformed hand edits yield `routes: null` plus `routesError` without changing the stored value; valid and absent rules omit that diagnostic. Config and management responses retain route names; request logs use only the rule’s 1-based `route:#` position. `src/cli/account-extended.ts` reads, replaces and clears these rules with `ocx account routes anthropic`; the server validates content. The provider management API validates `modelContextTiers` as a strict per-model map, diff --git a/structure/manifest.json b/structure/manifest.json index 0057a6464e4..d200b1c1e29 100644 --- a/structure/manifest.json +++ b/structure/manifest.json @@ -284,6 +284,16 @@ "src/web-search/" ] }, + { + "path": "providers/anthropic-account-pool.md", + "tier": 4, + "title": "Anthropic Account Pool", + "scope": "Anthropic OAuth account pause, model routes, and quota labels.", + "documents": [ + "src/oauth/", + "src/providers/" + ] + }, { "path": "providers/openai-tiers.md", "tier": 4, diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index 89efd30d9d7..3fee43be69d 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -1,6 +1,7 @@ # Providers And Adapters -For Anthropic OAuth, `src/oauth/anthropic-routing.ts` applies the first matching `anthropicAccountPool.routes` rule to every eligible pick. The declared account order is stable while its candidates remain eligible; active, manual, affinity, quota and strategy preferences only choose inside that set. A healthy session affinity outside a model route is ignored for that request and retained for later unrouted or differently routed models; the routed commit does not overwrite it. An explicit fallback widens an empty route to the ordinary pool, and fill-first then advances in ordinary pool order from the active account. A missing eligible route fails locally without that fallback. The rules are operator allowlists, not provider entitlement evidence. Request logs use `route:#` for the 1-based rule position, not the operator name. +Anthropic account pause, model routes, and quota labels follow the +[Anthropic account-pool contract](providers/anthropic-account-pool.md). An Anthropic 429 records the served account's cooldown even when the request has used its allowed retry sends. That final account remains excluded on the next request; combo target cooling is skipped only after the matching account cooldown is present. @@ -74,9 +75,6 @@ OrcaRouter key exchange uses the shared raw-byte reader before returning a durab 64 KiB response ceiling, single 30-second header/body deadline, and cancellation behavior follow the [bounded ingestion contract](transports/inventory.md#bounded-response-ingestion-and-orcarouter-login). -Anthropic model-scoped quota labels in `src/providers/quota/vendor-probes-oauth.ts` publish -only canonical Fable, Opus, or Sonnet labels after removing terminal controls; unknown upstream display names are omitted. - MiniMax and MiniMax CN Coding Plan quota in `src/providers/quota/vendor-probes-key.ts` uses the region-matched `/v1/api/openplatform/coding_plan/remains` endpoint. It publishes the `general` model's consumed 5-hour percentage and, when active, weekly percentage with their reset times; diff --git a/structure/providers/anthropic-account-pool.md b/structure/providers/anthropic-account-pool.md new file mode 100644 index 00000000000..54564e418a0 --- /dev/null +++ b/structure/providers/anthropic-account-pool.md @@ -0,0 +1,40 @@ +# Anthropic Account Pool + +## Anthropic account pause + +Anthropic OAuth shares `ProviderAccount.paused` in the protected auth store with generic +OAuth, not a second list in provider config. `setAccountPaused` serializes pause/resume +with credential and selection writes, advances the selection revision, and only publishes +invalidation after persistence. Removing an account removes its pause; reauthentication +preserves it. The store moves active selection to an unpaused, non-reauth row if available. +Pause does not clear cooldowns, quota, or credentials and does not cancel an already-sent turn. + +`src/oauth/anthropic-routing.ts` excludes paused rows from quota, round-robin, fill-first, +manual, affinity, model-route and reactive 429 candidates, including when proactive pooling +is off. All-paused requests return 403 with resume guidance. Quorum is invalidated on pause +and resume; a sent account paused before its 429 may still recover on its sole unpaused +successor. Credential resolution, refresh-lock acquisition, selection commit and physical +dispatch recheck live eligibility after asynchronous waits. Responses and native Messages +preserve typed 401 authentication, 403 pause and 429 cooldown refusals after pacing, including pool-off recovery; +they do not report local rejection as 502. Only cooled usable survivors of a strict route +produce its scoped 429 and Retry-After, not a login error. An already-dispatched refresh +retains a successful rotated credential without unpausing; a late failure cannot mark the +paused row for reauthentication. Token Guardian and Anthropic quota probes recheck live +pause, selection and bearer ownership after token resolution and before each usage send; +a newly paused account makes no auxiliary request. An account switch during a usage probe still seeds the probed account's quota cache but suppresses its stale provider report and reset observation. Pool-off keeps a healthy active account; +pause/prior-429 recovery uses `only-eligible`, and logs name the committed account. + +> Decision record: [ADR-6013](../decisions/ADR-6013-anthropic-account-pause.md) + +Regression coverage: `tests/adapters/anthropic/anthropic-account-pause.test.ts`, +`tests/adapters/anthropic/anthropic-model-routes.test.ts`, `tests/oauth/oauth-accounts-api.test.ts`, +`tests/cli/cli-account-pool-verbs.test.ts`, and `gui/tests/provider-quota-refresh-controls.test.tsx`. + +## Model routes + +For Anthropic OAuth, `src/oauth/anthropic-routing.ts` applies the first matching `anthropicAccountPool.routes` rule to every eligible pick. The declared account order is stable while its candidates remain eligible; active, manual, affinity, quota and strategy preferences only choose inside that set. A healthy session affinity outside a model route is ignored for that request and retained for later unrouted or differently routed models; the routed commit does not overwrite it. An explicit fallback widens an empty route to the ordinary pool, and fill-first then advances in ordinary pool order from the active account. A missing eligible route fails locally without that fallback. The rules are operator allowlists, not provider entitlement evidence. Request logs use `route:#` for the 1-based rule position, not the operator name. + +## Quota labels + +Anthropic model-scoped quota labels in `src/providers/quota/vendor-probes-oauth.ts` publish +only canonical Fable, Opus, or Sonnet labels after removing terminal controls; unknown upstream display names are omitted. diff --git a/structure/transports/inventory.md b/structure/transports/inventory.md index 18252c7916c..9cdc2a6aa36 100644 --- a/structure/transports/inventory.md +++ b/structure/transports/inventory.md @@ -154,7 +154,7 @@ by transports that can preserve that request-local decision: | Request path | Per-provider route | Current contract | | --- | --- | --- | | Main routed inference through `providerFetch` in `src/server/responses/fetch-helpers.ts` | Honoured | Applied at the physical send in `sendWithConnectionPolicy`, so a request rebuilt against a different destination or a reselected provider transport resolves its route against the destination actually used. The built-in executor passes direct, HTTP(S)-proxy, and SOCKS5(H)-proxy choices through `configuredOutboundFetch` in `src/lib/proxy-env.ts`; an inherited route leaves the global decision unchanged. Native Chat in `src/server/chat-native.ts` and the Responses transport in `src/server/responses/request-transport.ts` bind their own sends. | -| Every caller of `providerOutboundGet` or `providerOutboundPost` in `src/lib/provider-outbound.ts` | Honoured | This includes provider discovery and model-catalog gathering in `src/codex/catalog/provider-models.ts`, management provider tests in `src/server/management/provider-routes.ts`, and the Ollama show probe in `src/providers/ollama-show.ts`. | +| Every caller of `providerOutboundGet` or `providerOutboundPost` in `src/lib/provider-outbound.ts` | Honoured | This includes provider discovery and model-catalog gathering in `src/codex/catalog/provider-models.ts`, management provider tests in `src/server/management/provider-routes.ts`, and the Ollama show probe in `src/providers/ollama-show.ts`. An optional caller authorization predicate is rechecked after DNS and immediately before each physical send, including injected executors and proxy fallback. | | API-key quota probes in `src/providers/quota/vendor-probes-key.ts` | Honoured | Each probe receives its provider config and sends through `configuredOutboundFetch` with the resolved route, so a quota reading and the inference it describes leave by the same exit. | | OAuth token exchange and refresh under `src/oauth/` | Not honoured | These reach fixed vendor endpoints from modules that hold no provider config, so no provider route is in scope at the call site. A provider pinned to its own proxy or to direct still refreshes credentials by the process-wide route. | | OAuth-backed quota probes in `src/providers/quota/vendor-probes-oauth.ts` and `src/providers/quota/devin.ts` | Not honoured | `fetchXaiQuota`, `fetchAnthropicQuota`, `fetchCursorQuota`, `fetchDevinQuota` and their neighbours receive a provider name and a token rather than a provider config. | diff --git a/tests/adapters/anthropic/anthropic-account-pause-outbound.test.ts b/tests/adapters/anthropic/anthropic-account-pause-outbound.test.ts new file mode 100644 index 00000000000..5c6986207fb --- /dev/null +++ b/tests/adapters/anthropic/anthropic-account-pause-outbound.test.ts @@ -0,0 +1,127 @@ +import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import * as oauth from "../../../src/oauth"; +import { captureProviderGather } from "../../../src/codex/catalog/gather-capture"; +import { fetchProviderModelsWithAuth } from "../../../src/codex/catalog/provider-models"; +import { ProviderOutboundSendCancelledError, providerOutboundGet } from "../../../src/lib/provider-outbound"; +import { clearAnthropicAccountPoolState } from "../../../src/oauth/anthropic-routing"; +import { credentialGeneration, getAccountCredential, getAccountSet, saveAccountCredential, saveCredential, setAccountPaused } from "../../../src/oauth/store"; +import { clearAccountQuotaCache, fetchProviderAccountQuotas } from "../../../src/providers/quota"; +import * as accountCache from "../../../src/providers/quota/account-cache"; +import { fetchAnthropicQuota } from "../../../src/providers/quota/vendor-probes-oauth"; +import { removeTreeWithRetry } from "../../helpers/remove-tree"; + +const previousHome = process.env.OPENCODEX_HOME; +const originalFetch = globalThis.fetch; +let home: string; +let accountId: string; +let sends: number; +beforeEach(async () => { + home = mkdtempSync(join(tmpdir(), "ocx-anthropic-pause-outbound-")); + process.env.OPENCODEX_HOME = home; + clearAnthropicAccountPoolState(); + clearAccountQuotaCache(); + await saveCredential("anthropic", { + access: "synthetic-access", refresh: "synthetic-refresh", + expires: Date.now() + 3_600_000, accountId: "outbound-pause", + }); + accountId = getAccountSet("anthropic")!.activeAccountId; + sends = 0; + globalThis.fetch = (async () => { sends++; return new Response("{}", { status: 200 }); }) as typeof fetch; +}); +afterEach(() => { + globalThis.fetch = originalFetch; + clearAnthropicAccountPoolState(); + clearAccountQuotaCache(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + removeTreeWithRetry(home); +}); + +test("per-account quota does not send when pause commits while token resolution waits", async () => { + const started = Promise.withResolvers(); + const token = Promise.withResolvers(); + const read = spyOn(accountCache, "getTokenForAccountQuotaProbe").mockImplementation(async () => { + started.resolve(); + return token.promise; + }); + try { + const pending = fetchProviderAccountQuotas("anthropic", true); + await started.promise; + await setAccountPaused("anthropic", accountId, true); + token.resolve("synthetic-access"); + expect((await pending)[0]?.unavailable).toBe(true); + expect(sends).toBe(0); + } finally { read.mockRestore(); } +}); + +test("provider quota does not send when pause commits while token resolution waits", async () => { + const started = Promise.withResolvers(); + const token = Promise.withResolvers(); + const read = spyOn(oauth, "getValidAccessToken").mockImplementation(async () => { + started.resolve(); + return token.promise; + }); + try { + const pending = fetchAnthropicQuota("anthropic"); + await started.promise; + await setAccountPaused("anthropic", accountId, true); + token.resolve("synthetic-access"); + expect(await pending).toBeNull(); + expect(sends).toBe(0); + } finally { read.mockRestore(); } +}); + +test("model discovery rejects a bearer captured before pause", async () => { + const credential = getAccountCredential("anthropic", accountId)!; + const resolver = { kind: "observed" as const, resolve: () => ({ + apiKey: credential.access, observed: true, + oauthAccountId: accountId, oauthGeneration: credentialGeneration(credential), + }) }; + const captured = captureProviderGather("anthropic", { + adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "oauth", + liveModels: true, models: ["configured-model"], + }, resolver); + await setAccountPaused("anthropic", accountId, true); + const result = await fetchProviderModelsWithAuth(captured, 0, undefined, resolver); + expect(result.outcome.state).toBe("degraded"); + expect(sends).toBe(0); +}); + +test("model discovery rejects a replaced credential generation even when its access token matches", async () => { + const credential = getAccountCredential("anthropic", accountId)!; + const resolver = { kind: "observed" as const, resolve: () => ({ + apiKey: credential.access, observed: true, + oauthAccountId: accountId, oauthGeneration: credentialGeneration(credential), + }) }; + const captured = captureProviderGather("anthropic", { + adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "oauth", + liveModels: true, models: ["configured-model"], + }, resolver); + await saveAccountCredential("anthropic", accountId, { ...credential, refresh: "rotated-refresh" }); + const result = await fetchProviderModelsWithAuth(captured, 0, undefined, resolver); + expect(result.outcome.state).toBe("degraded"); + expect(sends).toBe(0); +}); + +test("model discovery transport cancels after DNS when pause commits before physical send", async () => { + const started = Promise.withResolvers(); + const releaseDns = Promise.withResolvers(); + const pending = providerOutboundGet("anthropic", { baseUrl: "https://api.anthropic.com" }, + "https://api.anthropic.com/v1/models", { headers: { Authorization: "Bearer synthetic-access" } }, { + resolveAddresses: async () => { + started.resolve(); + await releaseDns.promise; + return { hostname: "api.anthropic.com", addresses: [{ address: "93.184.216.34", family: 4 }], privateNetwork: false }; + }, + pinnedGet: async () => { sends++; return new Response("{}"); }, + beforeSend: () => getAccountSet("anthropic")?.accounts.find(row => row.id === accountId)?.paused !== true, + }); + await started.promise; + await setAccountPaused("anthropic", accountId, true); + releaseDns.resolve(); + await expect(pending).rejects.toBeInstanceOf(ProviderOutboundSendCancelledError); + expect(sends).toBe(0); +}); diff --git a/tests/adapters/anthropic/anthropic-account-pause.test.ts b/tests/adapters/anthropic/anthropic-account-pause.test.ts new file mode 100644 index 00000000000..631ecb15e62 --- /dev/null +++ b/tests/adapters/anthropic/anthropic-account-pause.test.ts @@ -0,0 +1,175 @@ +import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; +import { mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { clearPoolRotationState } from "../../../src/codex/pool-rotation"; +import { OAuthAccountPausedError, OAuthLoginRequiredError, OAUTH_PROVIDERS, refreshAnthropicAccountWithLock } from "../../../src/oauth"; +import { AnthropicTokenError } from "../../../src/oauth/anthropic"; +import { + bindAnthropicSessionAffinity, clearAnthropicAccountPoolState, getAnthropicAccountHealthSnapshot, + getAnthropicPoolAccessSnapshot, getEligibleAnthropicAccounts, hasAnthropicFailoverQuorum, + promoteAnthropicActiveAccount, resolveAnthropicAccountForSession, rotateAnthropicAccountOn429, +} from "../../../src/oauth/anthropic-routing"; +import { + captureOAuthAccountSelection, createOAuthRefreshIntentLock, getAccountCredential, getAccountSet, + removeAccount, saveAccountCredential, saveCredential, setAccountPaused, setActiveAccount, +} from "../../../src/oauth/store"; +import { clearAccountQuotaCache, setCachedProviderAccountQuotaForTests } from "../../../src/providers/quota"; +import type { OcxAccountPoolQuotaWindow, OcxAccountPoolRotationStrategy, OcxConfig } from "../../../src/types"; +import { removeTreeWithRetry } from "../../helpers/remove-tree"; + +const previousHome = process.env.OPENCODEX_HOME; +let home: string; +let ids: string[]; +beforeEach(async () => { + home = mkdtempSync(join(tmpdir(), "ocx-anthropic-pause-")); + process.env.OPENCODEX_HOME = home; + clearAnthropicAccountPoolState(); clearPoolRotationState(); clearAccountQuotaCache(); + for (let i = 0; i < 3; i++) await saveCredential("anthropic", { + access: `synthetic-access-${i}`, refresh: `synthetic-refresh-${i}`, + expires: Date.now() + 3_600_000, accountId: `pause-${i}`, + }); + ids = getAccountSet("anthropic")!.accounts.map(account => account.id); + await setActiveAccount("anthropic", ids[0]!); +}); +afterEach(() => { + clearAnthropicAccountPoolState(); clearPoolRotationState(); clearAccountQuotaCache(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + removeTreeWithRetry(home); +}); +function config(enabled = true, strategy: OcxAccountPoolRotationStrategy = "quota", quotaWindow: OcxAccountPoolQuotaWindow = "five-hour"): OcxConfig { + return { port: 0, defaultProvider: "anthropic", providers: { + anthropic: { adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "oauth" }, + }, anthropicAccountPool: { enabled, strategy, quotaWindow } }; +} + +for (const strategy of ["quota", "round-robin", "fill-first"] as const) { + for (const window of ["five-hour", "weekly", "max-utilization"] as const) { + test(`${strategy}/${window}: paused active, manual and affined accounts cannot win`, async () => { + const [a, b, c] = ids as [string, string, string]; + const cfg = config(true, strategy, window); + resolveAnthropicAccountForSession("session", cfg); // Cache the initial manual preference. + bindAnthropicSessionAffinity("session", a); + setCachedProviderAccountQuotaForTests("anthropic", a, { fiveHourPercent: 0, weeklyPercent: 0, updatedAt: Date.now() }); + await setAccountPaused("anthropic", a, true); + await setAccountPaused("anthropic", b, true); + expect(getEligibleAnthropicAccounts()).toEqual([c]); + expect(resolveAnthropicAccountForSession("session", cfg).accountId).toBe(c); + expect(resolveAnthropicAccountForSession(null, cfg).accountId).toBe(c); + expect(rotateAnthropicAccountOn429(cfg, b, "60")).toBe(c); + }); + } +} + +test("pause invalidates quorum immediately; disabled-pool reactive failover skips paused successors", async () => { + const [a, b, c] = ids as [string, string, string]; + expect(hasAnthropicFailoverQuorum()).toBe(true); + await setAccountPaused("anthropic", b, true); + expect(rotateAnthropicAccountOn429(config(false), a, "60")).toBe(c); + await setAccountPaused("anthropic", a, true); + expect(hasAnthropicFailoverQuorum()).toBe(false); + // The turn on A was already sent when it was paused; B stays excluded, C can recover it. + expect(rotateAnthropicAccountOn429(config(false), a, "60")).toBe(c); + await setAccountPaused("anthropic", b, false); + expect(hasAnthropicFailoverQuorum()).toBe(true); +}); + +test("all-paused refusal and resume preserve credentials and cooldown, independent of pool enable", async () => { + const a = ids[0]!; + const credential = getAccountCredential("anthropic", a); + rotateAnthropicAccountOn429(config(), a, "60"); + const health = getAnthropicAccountHealthSnapshot(a); + for (const id of ids) await setAccountPaused("anthropic", id, true); + for (const enabled of [true, false]) expect(resolveAnthropicAccountForSession("old", config(enabled))).toMatchObject({ accountId: null, reason: "paused" }); + await expect(getAnthropicPoolAccessSnapshot(a)).rejects.toBeInstanceOf(OAuthAccountPausedError); + expect(await setActiveAccount("anthropic", a)).toBe(false); + await setAccountPaused("anthropic", a, false); + expect(getAccountCredential("anthropic", a)).toEqual(credential); + expect(getAnthropicAccountHealthSnapshot(a)).toEqual(health); + expect(getAccountSet("anthropic")!.activeAccountId).toBe(a); +}); + +test("model routes cannot restore a paused account, but explicit fallback can widen", async () => { + const [a, b, c] = ids as [string, string, string]; + await setAccountPaused("anthropic", a, true); + await setAccountPaused("anthropic", b, true); + const route = { position: 1, accounts: [a, b], fallback: false }; + expect(resolveAnthropicAccountForSession("", config(), Date.now(), route).reason).toBe("paused"); + expect(resolveAnthropicAccountForSession("", config(), Date.now(), { ...route, fallback: true }).accountId).toBe(c); +}); + +test("pause and pause-resume ABA both invalidate an earlier credential proposal", async () => { + const a = ids[0]!; + const selection = captureOAuthAccountSelection("anthropic"); + const snapshot = await getAnthropicPoolAccessSnapshot(a); + await setAccountPaused("anthropic", a, true); + expect(await promoteAnthropicActiveAccount(a, selection, { config: config(), expectedCredentialGeneration: snapshot.generation })).toBeNull(); + await setAccountPaused("anthropic", a, false); + // Resume preserves the credential but cannot authorize a proposal captured before pause. + expect(await promoteAnthropicActiveAccount(a, selection, { config: config() })).toBeNull(); + expect(await promoteAnthropicActiveAccount(a, captureOAuthAccountSelection("anthropic"), { config: config() })).not.toBeNull(); +}); + +test("pause during an asynchronous credential refresh prevents returning its bearer", async () => { + const a = ids[0]!; + const credential = getAccountCredential("anthropic", a)!; + await saveAccountCredential("anthropic", a, { ...credential, expires: 0 }); + const refresh = spyOn(OAUTH_PROVIDERS.anthropic!, "refresh").mockImplementation(async () => { + await setAccountPaused("anthropic", a, true); + return { ...credential, access: "synthetic-after-wait", refresh: "synthetic-after-wait-refresh" }; + }); + try { + await expect(getAnthropicPoolAccessSnapshot(a)).rejects.toBeInstanceOf(OAuthAccountPausedError); + expect(refresh).toHaveBeenCalledTimes(1); + // Keep an already-rotated refresh token for resume without authorizing this request. + expect(getAccountCredential("anthropic", a)?.access).toBe("synthetic-after-wait"); + } finally { refresh.mockRestore(); } +}); + +test("pause is durable, idempotent, preserved on relogin and removed with its account", async () => { + const a = ids[0]!; + const credential = getAccountCredential("anthropic", a)!; + await Promise.all([setAccountPaused("anthropic", a, true), setAccountPaused("anthropic", ids[1]!, true)]); + expect((await setAccountPaused("anthropic", a, true)).status).toBe("unchanged"); + clearAnthropicAccountPoolState(); + expect(JSON.parse(readFileSync(join(home, "auth.json"), "utf8")).anthropic.accounts.filter((row: { paused?: boolean }) => row.paused)).toHaveLength(2); + await saveCredential("anthropic", credential); + expect(getAccountSet("anthropic")!.accounts.find(row => row.id === a)?.paused).toBe(true); + await removeAccount("anthropic", a); + await saveCredential("anthropic", credential); + expect(getAccountSet("anthropic")!.accounts.find(row => row.credential.accountId === credential.accountId)?.paused).toBeUndefined(); +}); + +test("refresh waiting for its lock observes a newly persisted pause without upstream use", async () => { + const a = ids[0]!; + const lock = createOAuthRefreshIntentLock("anthropic", a); + const originalAcquire = lock.acquire.bind(lock); + lock.acquire = async () => { await setAccountPaused("anthropic", a, true); return originalAcquire(); }; + let sends = 0; + await expect(refreshAnthropicAccountWithLock("anthropic", a, { + ...OAUTH_PROVIDERS.anthropic!, refresh: async () => { sends++; throw new Error("unexpected refresh"); }, + }, getAccountCredential("anthropic", a)!, { intentLock: lock })).rejects.toBeInstanceOf(OAuthAccountPausedError); + expect(sends).toBe(0); +}); + +test("refresh finishing after pause preserves rotated credentials; late failure preserves health", async () => { + const a = ids[0]!; + const original = getAccountCredential("anthropic", a)!; + await refreshAnthropicAccountWithLock("anthropic", a, { + ...OAUTH_PROVIDERS.anthropic!, refresh: async () => { + await setAccountPaused("anthropic", a, true); + return { ...original, access: "synthetic-rotated", refresh: "synthetic-rotated-refresh" }; + }, + }, original); + expect(getAccountCredential("anthropic", a)?.access).toBe("synthetic-rotated"); + expect(getAccountSet("anthropic")!.accounts.find(row => row.id === a)?.paused).toBe(true); + await setAccountPaused("anthropic", a, false); + await expect(refreshAnthropicAccountWithLock("anthropic", a, { + ...OAUTH_PROVIDERS.anthropic!, refresh: async () => { + await setAccountPaused("anthropic", a, true); + throw new AnthropicTokenError("synthetic rejection", 400, "invalid_grant"); + }, + }, getAccountCredential("anthropic", a)!)).rejects.toBeInstanceOf(OAuthLoginRequiredError); + expect(getAccountSet("anthropic")!.accounts.find(row => row.id === a)?.needsReauth).not.toBe(true); +}); diff --git a/tests/adapters/anthropic/anthropic-account-pool.test.ts b/tests/adapters/anthropic/anthropic-account-pool.test.ts index 61396bd71e9..d56f869a070 100644 --- a/tests/adapters/anthropic/anthropic-account-pool.test.ts +++ b/tests/adapters/anthropic/anthropic-account-pool.test.ts @@ -252,7 +252,7 @@ describe("anthropic account pool", () => { expect(resolveAnthropicAccountForSession("uncommitted", quota).accountId).toBe(bId); }); - test("default off always returns the active account", async () => { + test("default off retains the healthy active account", async () => { const { aId, bId } = await seedTwoAccounts(); expect(isAnthropicAccountPoolEnabled(cfg(false))).toBe(false); const sel = resolveAnthropicAccountForSession("session-1", cfg(false)); @@ -261,6 +261,17 @@ describe("anthropic account pool", () => { expect(sel.accountId).not.toBe(bId); }); + test("pool-off ignores proactive preferences but names recovery from an upstream 429", async () => { + const { aId, bId } = await seedTwoAccounts(); + const config = cfg(false, 80, { strategy: "round-robin" }); + setCachedProviderAccountQuotaForTests("anthropic", aId, { fiveHourPercent: 95 }); + setCachedProviderAccountQuotaForTests("anthropic", bId, { fiveHourPercent: 5 }); + expect(resolveAnthropicAccountForSession("new", config)).toMatchObject({ accountId: aId, reason: "pool-disabled" }); + expect(rotateAnthropicAccountOn429(config, aId, "60")).toBe(bId); + expect(resolveAnthropicAccountForSession("new", config)).toMatchObject({ accountId: bId, reason: "only-eligible" }); + expect(getAccountSet("anthropic")!.activeAccountId).toBe(aId); // A proposal is not a committed send. + }); + test("affinity sticks across resolves until cooled", async () => { const { aId, bId } = await seedTwoAccounts(); // Force lowest-usage toward B for a cold start with high active usage. diff --git a/tests/adapters/anthropic/anthropic-model-routes.test.ts b/tests/adapters/anthropic/anthropic-model-routes.test.ts index 9daa0d7c316..faaae0d123f 100644 --- a/tests/adapters/anthropic/anthropic-model-routes.test.ts +++ b/tests/adapters/anthropic/anthropic-model-routes.test.ts @@ -4,9 +4,11 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { acquireOwnedSpendHome } from "../../helpers/owned-spend-home"; import { removeTreeWithRetry } from "../../helpers/remove-tree"; -import { clearAnthropicAccountPoolState, bindAnthropicSessionAffinity, getAnthropicPoolAccessSnapshot, getAnthropicPoolRetryAfterSeconds, promoteAnthropicActiveAccount, resolveAnthropicAccountForSession, rotateAnthropicAccountOn429 } from "../../../src/oauth/anthropic-routing"; +import { clearAnthropicAccountPoolState, bindAnthropicSessionAffinity, getAnthropicAccountHealthSnapshot, getAnthropicPoolAccessSnapshot, getAnthropicPoolRetryAfterSeconds, promoteAnthropicActiveAccount, resolveAnthropicAccountForSession, rotateAnthropicAccountOn429 } from "../../../src/oauth/anthropic-routing"; import { parseAnthropicModelRoutes, resolveAnthropicModelRoute } from "../../../src/oauth/anthropic-model-routes"; -import { captureOAuthAccountSelection, getAccountSet, saveCredential, setActiveAccount } from "../../../src/oauth/store"; +import { captureOAuthAccountSelection, getAccountSet, markAccountNeedsReauth, replaceProviderAccountSet, saveAccountCredential, saveCredential, setAccountPaused, setActiveAccount } from "../../../src/oauth/store"; +import { clearUpstreamHostHealth, getUpstreamHostHealth, upstreamHostHealthKey } from "../../../src/codex/upstream-host-health"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../../src/providers/request-pacing"; import { clearAccountQuotaCache, setCachedProviderAccountQuotaForTests } from "../../../src/providers/quota"; import { clearResponseStateForTests } from "../../../src/responses/state"; import { handleResponses } from "../../../src/server/responses"; @@ -26,10 +28,13 @@ beforeEach(() => { releaseSpend = acquireOwnedSpendHome(); sends = []; clearAnthropicAccountPoolState(); + clearUpstreamHostHealth(); clearAccountQuotaCache(); clearResponseStateForTests(); }); afterEach(() => { + clearUpstreamHostHealth(); + resetProviderRequestPacingForTest(); releaseSpend(); clearAnthropicAccountPoolState(); clearAccountQuotaCache(); @@ -49,7 +54,7 @@ async function seed(): Promise { await setActiveAccount("anthropic", ids[0]!); return ids; } -function config(ids: string[], reply: (token: string) => Response): OcxConfig { +function config(ids: string[], reply: (token: string) => Response | Promise): OcxConfig { const fetcher = (async (_url, init) => { const token = new Headers(init?.headers).get("authorization") ?? new Headers(init?.headers).get("x-api-key") ?? ""; sends.push(token); @@ -87,6 +92,198 @@ test("bounded first-match globs and invalid rules", () => { expect(parseAnthropicModelRoutes([{ name: "a", match: "[bad]", accounts: ["a"] }]).ok).toBe(false); }); +test.each([true, false])("all-paused pool returns 403 without any send (enabled=%s)", async enabled => { + const ids = await seed(); + for (const id of ids) await setAccountPaused("anthropic", id, true); + const cfg = config(ids, () => answer()); + cfg.anthropicAccountPool!.enabled = enabled; + const response = await post(cfg); + expect(response.status).toBe(403); + expect(await response.text()).toContain("Resume"); + expect(sends).toEqual([]); +}); + +test("a paused route successor is skipped on disabled-pool 429 failover", async () => { + const ids = await seed(); + await setAccountPaused("anthropic", ids[1]!, true); + const cfg = config(ids, async token => { + if (token.includes("synthetic-access-0")) { + await setAccountPaused("anthropic", ids[0]!, true); + return Response.json({ error: { type: "rate_limit_error", message: "synthetic refusal" } }, { status: 429, headers: { "retry-after": "60" } }); + } + return answer(); + }); + cfg.anthropicAccountPool!.enabled = false; + expect((await post(cfg)).status).toBe(200); + expect(sends).toHaveLength(2); + expect(sends[0]).toContain("synthetic-access-0"); + expect(sends[1]).toContain("synthetic-access-2"); +}); + +test("pausing an already-sent successful turn does not cancel its result", async () => { + const ids = await seed(); + const cfg = config(ids, async () => { + for (const id of ids) await setAccountPaused("anthropic", id, true); + return answer(); + }); + expect((await post(cfg)).status).toBe(200); + expect(sends).toHaveLength(1); +}); + +test.each([true, false])("pause while queued for pacing never sends the cached bearer (enabled=%s)", async enabled => { + const ids = await seed(); + const cfg = config(ids, () => answer()); + cfg.anthropicAccountPool = { enabled }; + cfg.providers.anthropic!.requestPacing = { enabled: true, maxConcurrentRequests: 1 }; + const slot = await waitForProviderRequestSlot("anthropic", cfg.providers.anthropic!, "claude-sonnet-4-5"); + const pending = post(cfg); + try { + for (let i = 0; i < 100 && providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued === 0; i++) { + await new Promise(resolve => setTimeout(resolve, 5)); + } + expect(providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued).toBe(1); + await setAccountPaused("anthropic", ids[0]!, true); + } finally { slot.release(); } + expect((await pending).status).toBe(200); + expect(sends).toHaveLength(1); + expect(sends[0]).not.toContain("synthetic-access-0"); +}); + +for (const adapter of ["anthropic", "openai-responses"] as const) { + for (const enabled of [true, false]) { + for (const remaining of ["needs-reauth", "unusable"] as const) { + test(`${adapter}: pause with ${remaining} survivors during pacing matches fresh admission, pool enabled=${enabled}`, async () => { + const ids = await seed(); + await setAccountPaused("anthropic", ids[2]!, true); + const cfg = config(ids, () => answer()); + cfg.anthropicAccountPool = { enabled, routes: [{ name: "private-auth-scope", match: "claude-*", accounts: [ids[0]!, ids[1]!] }] }; + cfg.providers.anthropic!.adapter = adapter; + cfg.providers.anthropic!.requestPacing = { enabled: true, maxConcurrentRequests: 1 }; + const hostKey = upstreamHostHealthKey("anthropic", "anthropic-routes.test"); + const slot = await waitForProviderRequestSlot("anthropic", cfg.providers.anthropic!, "claude-sonnet-4-5"); + const pending = post(cfg); + let rosterBeforeDispatch: ReturnType; + try { + for (let i = 0; i < 100 && providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued === 0; i++) { + await new Promise(resolve => setTimeout(resolve, 5)); + } + expect(providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued).toBe(1); + if (remaining === "needs-reauth") await markAccountNeedsReauth("anthropic", ids[1]!, true); + else { + const credential = getAccountSet("anthropic")!.accounts.find(row => row.id === ids[1])!.credential; + await saveAccountCredential("anthropic", ids[1]!, { ...credential, source: "local-cli", expires: 0 }); + } + await setAccountPaused("anthropic", ids[0]!, true); + if (remaining === "unusable") { + // A persisted background local-CLI slot must not adopt the foreground CLI identity. + await replaceProviderAccountSet("anthropic", { ...getAccountSet("anthropic")!, activeAccountId: ids[0]! }); + } + rosterBeforeDispatch = getAccountSet("anthropic"); + expect(getUpstreamHostHealth(hostKey)).toBeNull(); + } finally { slot.release(); } + const response = await pending; + const initial = await post(cfg); + expect(initial.status).toBe(enabled ? 401 : 403); + expect(response.status).toBe(initial.status); + const body = await response.json() as { error: { type: string; message: string } }; + const initialBody = await initial.json() as { error: { type: string } }; + expect(body.error.type).toBe(initialBody.error.type); + expect(body.error.message).not.toContain("private-auth-scope"); + expect(sends).toEqual([]); + expect(getUpstreamHostHealth(hostKey)).toBeNull(); + expect(ids.map(id => getAnthropicAccountHealthSnapshot(id))).toEqual([null, null, null]); + expect(getAccountSet("anthropic")).toEqual(rosterBeforeDispatch!); + }); + } + + test(`${adapter}: a pacing-time pause skips a cooled successor for a healthy account, pool enabled=${enabled}`, async () => { + const ids = await seed(); + const cfg = config(ids, () => answer()); + cfg.anthropicAccountPool = { enabled, routes: [{ name: "all", match: "claude-*", accounts: ids }] }; + cfg.providers.anthropic!.adapter = adapter; + cfg.providers.anthropic!.requestPacing = { enabled: true, maxConcurrentRequests: 1 }; + const slot = await waitForProviderRequestSlot("anthropic", cfg.providers.anthropic!, "claude-sonnet-4-5"); + const pending = post(cfg); + try { + for (let i = 0; i < 100 && providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued === 0; i++) { + await new Promise(resolve => setTimeout(resolve, 5)); + } + expect(providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued).toBe(1); + rotateAnthropicAccountOn429(cfg, ids[1]!, "60"); + await setAccountPaused("anthropic", ids[0]!, true); + } finally { slot.release(); } + expect((await pending).status).toBe(200); + expect(sends).toHaveLength(1); + expect(sends[0]).toContain("synthetic-access-2"); + }); + + test(`${adapter}: paused plus cooled accounts during pacing return scoped 429, pool enabled=${enabled}`, async () => { + const ids = await seed(); + const cfg = config(ids, () => answer()); + cfg.anthropicAccountPool = { enabled, routes: [{ name: "private-scope", match: "claude-*", accounts: [ids[0]!, ids[1]!] }] }; + cfg.providers.anthropic!.adapter = adapter; + cfg.providers.anthropic!.requestPacing = { enabled: true, maxConcurrentRequests: 1 }; + if (!enabled) await setAccountPaused("anthropic", ids[2]!, true); + const slot = await waitForProviderRequestSlot("anthropic", cfg.providers.anthropic!, "claude-sonnet-4-5"); + const pending = post(cfg); + try { + for (let i = 0; i < 100 && providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued === 0; i++) { + await new Promise(resolve => setTimeout(resolve, 5)); + } + expect(providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued).toBe(1); + rotateAnthropicAccountOn429(cfg, ids[1]!, "60"); + // A shorter outsider cooldown must not change the strict route's Retry-After. + if (enabled) rotateAnthropicAccountOn429(cfg, ids[2]!, "5"); + await setAccountPaused("anthropic", ids[0]!, true); + } finally { slot.release(); } + const response = await pending; + expect(response.status).toBe(429); + expect(Number(response.headers.get("retry-after"))).toBeGreaterThan(50); + expect(Number(response.headers.get("retry-after"))).toBeLessThanOrEqual(60); + const body = await response.json() as { error: { type: string; message: string } }; + expect(body.error.type).toBe("rate_limit_error"); + expect(body.error.message.includes("model route")).toBe(enabled); + expect(body.error.message).not.toContain("private-scope"); + expect(sends).toEqual([]); + }); + + test(`${adapter}: pausing every account during pacing returns 403, pool enabled=${enabled}`, async () => { + const ids = await seed(); + const cfg = config(ids, () => answer()); + cfg.anthropicAccountPool!.enabled = enabled; + cfg.providers.anthropic!.adapter = adapter; + cfg.providers.anthropic!.requestPacing = { enabled: true, maxConcurrentRequests: 1 }; + const slot = await waitForProviderRequestSlot("anthropic", cfg.providers.anthropic!, "claude-sonnet-4-5"); + const pending = post(cfg); + try { + for (let i = 0; i < 100 && providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued === 0; i++) { + await new Promise(resolve => setTimeout(resolve, 5)); + } + expect(providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued).toBe(1); + for (const id of ids) await setAccountPaused("anthropic", id, true); + } finally { slot.release(); } + const response = await pending; + expect(response.status).toBe(403); + expect(await response.json()).toMatchObject({ error: { type: "permission_error", message: expect.stringContaining("Resume") } }); + expect(sends).toEqual([]); + }); + } +} + +test("strict route with paused and cooled members returns 429 from its remaining usable member", async () => { + const ids = await seed(); + const cfg = config(ids, () => answer()); + const decision = resolveAnthropicModelRoute(cfg, "claude-sonnet-4-5").decision!; + await setAccountPaused("anthropic", ids[1]!, true); + rotateAnthropicAccountOn429(cfg, ids[2]!, "60", null, Date.now(), null, decision); + expect(resolveAnthropicAccountForSession("", cfg, Date.now(), decision).reason).toBe("all-cooled"); + const response = await post(cfg); + expect(response.status).toBe(429); + expect(Number(response.headers.get("retry-after"))).toBeGreaterThan(0); + expect(Number(response.headers.get("retry-after"))).toBeLessThanOrEqual(60); + expect(sends).toEqual([]); +}); + test("matched route excludes active outsider before an upstream send", async () => { const ids = await seed(); const cfg = config(ids, () => answer()); diff --git a/tests/claude-integration/messages-native-oauth.test.ts b/tests/claude-integration/messages-native-oauth.test.ts index 5d86a9001bf..18f03268a78 100644 --- a/tests/claude-integration/messages-native-oauth.test.ts +++ b/tests/claude-integration/messages-native-oauth.test.ts @@ -12,10 +12,12 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { saveConfig } from "../../src/config"; import { ANTHROPIC_OAUTH_BETA, CLAUDE_CODE_SYSTEM_INSTRUCTION } from "../../src/oauth/anthropic"; -import { clearAnthropicAccountPoolState, forgetAnthropicFailoverQuorum } from "../../src/oauth/anthropic-routing"; -import { getAccountSet, markAccountNeedsReauth, saveCredential, setActiveAccount } from "../../src/oauth/store"; +import { clearAnthropicAccountPoolState, forgetAnthropicFailoverQuorum, formatAnthropicProviderForLog, getAnthropicAccountHealthSnapshot, rotateAnthropicAccountOn429 } from "../../src/oauth/anthropic-routing"; +import { getAccountSet, markAccountNeedsReauth, replaceProviderAccountSet, saveAccountCredential, saveCredential, setAccountPaused, setActiveAccount } from "../../src/oauth/store"; +import { clearUpstreamHostHealth, getUpstreamHostHealth, upstreamHostHealthKey } from "../../src/codex/upstream-host-health"; import { handleClaudeMessages } from "../../src/server/claude-messages"; import { getRequestLogEntries } from "../../src/server/request-log"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; import type { OcxConfig, OcxProviderConfig } from "../../src/types"; import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; import { removeTreeWithRetry } from "../helpers/remove-tree"; @@ -49,11 +51,14 @@ beforeEach(() => { throw new Error("unexpected global fetch in the native OAuth Messages test"); }) as unknown as typeof fetch; clearAnthropicAccountPoolState(); + clearUpstreamHostHealth(); forgetAnthropicFailoverQuorum(); releaseSpendHome = acquireOwnedSpendHome(); }); afterEach(() => { + clearUpstreamHostHealth(); + resetProviderRequestPacingForTest(); releaseSpendHome?.(); releaseSpendHome = undefined; try { @@ -169,6 +174,107 @@ async function send(config: OcxConfig, body: Record) { } describe("managed native Messages over Anthropic OAuth", () => { + for (const native of [true, false]) for (const enabled of [true, false]) { + for (const state of ["needs-reauth", "unusable", "cooled", "paused-cooled", "healthy"] as const) { + test(`Messages native=${native}, pool=${enabled}: pacing ${state} preserves fresh admission`, async () => { + const ids = await seed(2); + await markAccountNeedsReauth("anthropic", ids[1]!, true); + forgetAnthropicFailoverQuorum(); + const cfg = fixtureConfig({ oauthSwitch: native }); + cfg.anthropicAccountPool = { enabled }; + cfg.providers.anthropic!.requestPacing = { enabled: true, maxConcurrentRequests: 1 }; + const slot = await waitForProviderRequestSlot("anthropic", cfg.providers.anthropic!, "claude-sonnet-4-5"); + const pending = send(cfg, { ...BODY, stream: false }); + let setupFailed = false; + let setupError: unknown; + let health: ReturnType[] = []; + let roster: ReturnType; + try { + for (let i = 0; i < 100 && providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued === 0; i++) { + await new Promise(resolve => setTimeout(resolve, 5)); + } + expect(providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued).toBe(1); + const coolingConfig = { ...cfg, anthropicAccountPool: { enabled: true } }; + if (state === "cooled") rotateAnthropicAccountOn429(coolingConfig, ids[0]!, "60"); + else { + await markAccountNeedsReauth("anthropic", ids[1]!, false); + await setAccountPaused("anthropic", ids[0]!, true); + if (state === "needs-reauth") await markAccountNeedsReauth("anthropic", ids[1]!, true); + if (state === "paused-cooled") rotateAnthropicAccountOn429(coolingConfig, ids[1]!, "60"); + if (state === "unusable") { + await saveAccountCredential("anthropic", ids[1]!, { ...credential(1), source: "local-cli", expires: 0 }); + await replaceProviderAccountSet("anthropic", { ...getAccountSet("anthropic")!, activeAccountId: ids[0]! }); + } + } + health = ids.map(id => getAnthropicAccountHealthSnapshot(id)); + roster = getAccountSet("anthropic"); + } catch (error) { setupFailed = true; setupError = error; } + finally { slot.release(); if (setupFailed) await pending.catch(() => undefined); } + if (setupFailed) throw setupError; + const result = await pending; + const initial = await send(cfg, { ...BODY, stream: false }); + const expected = state === "healthy" ? 200 : state === "cooled" || state === "paused-cooled" ? 429 + : state === "unusable" && !enabled ? 403 : 401; + expect(initial.response.status).toBe(expected); + expect(result.response.status).toBe(initial.response.status); + if (expected !== 200) { + expect(JSON.parse(result.text).error.type).toBe(JSON.parse(initial.text).error.type); + expect(sent).toEqual([]); + expect(getAccountSet("anthropic")).toEqual(roster!); + } else { + expect(sent).toHaveLength(2); + expect(sent.every(entry => entry.headers.get("authorization") === `Bearer ${credential(1).access}`)).toBe(true); + expect(result.row.provider).toBe(formatAnthropicProviderForLog("anthropic", ids[1])); + } + if (expected === 429) { + expect(Number(result.response.headers.get("retry-after"))).toBeGreaterThan(50); + expect(Number(result.response.headers.get("retry-after"))).toBeLessThanOrEqual(60); + } + expect(ids.map(id => getAnthropicAccountHealthSnapshot(id))).toEqual(health); + expect(getUpstreamHostHealth(upstreamHostHealthKey("anthropic", "api.anthropic.com"))).toBeNull(); + }); + } + } + + test("a singleton paused during native pacing returns 403 without dispatch", async () => { + const [id] = await seed(1); + const cfg = fixtureConfig(); + cfg.providers.anthropic!.requestPacing = { enabled: true, maxConcurrentRequests: 1 }; + const slot = await waitForProviderRequestSlot("anthropic", cfg.providers.anthropic!, "claude-sonnet-4-5"); + const pending = send(cfg, { ...BODY, stream: false }); + let setupFailed = false; + let setupError: unknown; + try { + for (let i = 0; i < 100 && providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued === 0; i++) { + await new Promise(resolve => setTimeout(resolve, 5)); + } + expect(providerRequestPacingStatus("anthropic", cfg.providers.anthropic!).queued).toBe(1); + await setAccountPaused("anthropic", id!, true); + } catch (error) { + setupFailed = true; + setupError = error; + } finally { + slot.release(); + // A failed queue assertion must not leave this request running into afterEach. Drain it, + // but keep the assertion as the useful failure if teardown also rejects. + if (setupFailed) await pending.catch(() => undefined); + } + if (setupFailed) throw setupError; + const { response, text } = await pending; + expect(response.status).toBe(403); + expect(text).toContain("Resume"); + expect(sent).toEqual([]); + }); + + test("an operator-paused singleton returns 403 and never sends", async () => { + const [id] = await seed(1); + await setAccountPaused("anthropic", id!, true); + const { response, text } = await send(fixtureConfig(), { ...BODY, stream: false }); + expect(response.status).toBe(403); + expect(text).toContain("Resume"); + expect(sent).toEqual([]); + }); + test("sends with the selected account's token and the OAuth shape, never the caller's credential", async () => { await seed(1); const { response, text, row } = await send(fixtureConfig(), { ...BODY, stream: true }); diff --git a/tests/cli/cli-account-pool-verbs.test.ts b/tests/cli/cli-account-pool-verbs.test.ts index 7d7542e6b92..a6b2613fbcf 100644 --- a/tests/cli/cli-account-pool-verbs.test.ts +++ b/tests/cli/cli-account-pool-verbs.test.ts @@ -57,6 +57,37 @@ function capture(): { lines: string[]; errors: string[]; restore: () => void } { } describe("ocx account pause / resume", () => { + test("Anthropic ambiguous aliases fail without a write; exact ids still win", async () => { + const out = capture(); + const writes: unknown[] = []; + const d: AccountDeps = { + baseUrl: "http://127.0.0.1:10100", + loadConfigImpl: () => ({ providers: { anthropic: { adapter: "anthropic", authMode: "oauth" } } }) as never, + fetchImpl: (async (_url, init) => { + if (init?.method === "PUT") { writes.push(JSON.parse(String(init.body))); return Response.json({ ok: true }); } + return Response.json({ accounts: [{ id: "a", alias: "reserve" }, { id: "b", alias: "reserve" }] }); + }) as typeof fetch, + }; + try { + expect(await cmdPause(["anthropic", "reserve"], d, true)).toBe(1); + expect(writes).toEqual([]); + expect(await cmdPause(["anthropic", "a"], d, true)).toBe(0); + expect(writes).toEqual([{ provider: "anthropic", accountId: "a", paused: true }]); + } finally { out.restore(); } + }); + + test.each([true, false])("Anthropic pause=%s resolves a unique alias and preserves the JSON contract", async paused => { + const calls: Captured[] = []; + const out = capture(); + const d = deps(() => ({ json: { ok: true, activeAccountId: "acct_2" } }), calls); + d.loadConfigImpl = () => ({ providers: { anthropic: { adapter: "anthropic", authMode: "oauth" } } }) as never; + try { + expect(await cmdPause(["anthropic", "GEM-PRO", "--json"], d, paused)).toBe(0); + expect(calls.find(call => call.method === "PUT")?.body).toEqual({ provider: "anthropic", accountId: "acct_1", paused }); + expect(JSON.parse(out.lines.join("\n"))).toEqual({ ok: true, provider: "anthropic", id: "acct_1", paused, activeAccountId: "acct_2" }); + } finally { out.restore(); } + }); + test("generic OAuth pause resolves aliases and uses the OAuth account route", async () => { const calls: Captured[] = []; const out = capture(); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index f2cb4ae134a..5c52800bcbc 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -107,6 +107,8 @@ "ambiguous-resend-composition.test.ts": "lib", "ambiguous-resend-gate.test.ts": "lib", "anthropic-account-pool.test.ts": "adapters/anthropic", + "anthropic-account-pause-outbound.test.ts": "adapters/anthropic", + "anthropic-account-pause.test.ts": "adapters/anthropic", "anthropic-combo-account-cooldown.test.ts": "adapters/anthropic", "anthropic-model-routes.test.ts": "adapters/anthropic", "anthropic-agentrouter-language-framing.test.ts": "adapters/anthropic", diff --git a/tests/oauth/oauth-accounts-api.test.ts b/tests/oauth/oauth-accounts-api.test.ts index d1c46e0ede8..c1d4a564d74 100644 --- a/tests/oauth/oauth-accounts-api.test.ts +++ b/tests/oauth/oauth-accounts-api.test.ts @@ -512,7 +512,32 @@ describe("multiauth accounts API", () => { } }); - test("pause API rejects Anthropic and a generic OAuth account behind an API-key route", async () => { + test("Anthropic pause validates input, persists resume and never alters credentials", async () => { + const before = getAccountSet("anthropic")!; + const server = startServer(0); + const pause = (accountId: string, paused: unknown) => fetch(new URL("/api/oauth/accounts/pause", server.url), { + method: "PUT", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ provider: "anthropic", accountId, paused }), + }); + try { + expect((await pause("aaaa1111", "true")).status).toBe(400); + expect((await pause("missing", true)).status).toBe(404); + expect(getAccountSet("anthropic")).toEqual(before); + expect((await pause("aaaa1111", true)).status).toBe(200); + expect((await pause("bbbb2222", true)).status).toBe(200); + const disk = JSON.parse(readFileSync(join(testDir, "auth.json"), "utf8")); + expect(disk.anthropic.accounts.every((row: { paused: boolean }) => row.paused)).toBe(true); + const resumed = await pause("aaaa1111", false); + expect(await resumed.json()).toMatchObject({ paused: false, activeAccountId: "aaaa1111", activeAccountChanged: true }); + const after = getAccountSet("anthropic")!; + expect(after.accounts.map(row => row.credential)).toEqual(before.accounts.map(row => row.credential)); + expect(after.accounts.find(row => row.id === "aaaa1111")?.paused).toBeUndefined(); + const serialized = await fetch(new URL("/api/oauth/accounts?provider=anthropic", server.url)).then(response => response.text()); + expect(serialized).not.toContain('"access"'); + expect(serialized).not.toContain('"refresh"'); + } finally { await server.stop(true); } + }); + + test("pause API supports Anthropic but rejects a generic OAuth account behind an API-key route", async () => { enableGoogleAntigravityAccounts(); const keyRouteConfig = baseConfig(); keyRouteConfig.providers["google-antigravity"] = { @@ -525,7 +550,12 @@ describe("multiauth accounts API", () => { method: "PUT", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ provider: "anthropic", accountId: "aaaa1111", paused: true }), }); - expect(anthropicPause.status).toBe(400); + expect(anthropicPause.status).toBe(200); + expect(await anthropicPause.json()).toMatchObject({ paused: true, activeAccountId: "bbbb2222" }); + const listed = await fetch(new URL("/api/oauth/accounts?provider=anthropic", server.url)); + const rows = await listed.json() as { accounts: Array<{ id: string; paused: boolean }> }; + expect(rows.accounts.find(row => row.id === "aaaa1111")?.paused).toBe(true); + expect(rows.accounts.find(row => row.id === "bbbb2222")?.paused).toBe(false); const keyRoutePause = await fetch(new URL("/api/oauth/accounts/pause", server.url), { method: "PUT", headers: { "Content-Type": "application/json" }, diff --git a/tests/providers/provider-account-quota.test.ts b/tests/providers/provider-account-quota.test.ts index 25c6ae74bb1..e0f1dff3241 100644 --- a/tests/providers/provider-account-quota.test.ts +++ b/tests/providers/provider-account-quota.test.ts @@ -367,7 +367,10 @@ describe("fetchProviderAccountQuotas", () => { // Switch active mid-flight before Anthropic responds. await setActiveAccount("anthropic", second!.id); releaseUsage(); - await reportPromise; + const switchedReport = await reportPromise; + expect(switchedReport.reports).toEqual([]); + expect(getCachedProviderAccountQuota("anthropic", first!.id)?.fiveHourPercent).toBe(70); + expect(getCachedProviderAccountQuota("anthropic", second!.id)).toBeNull(); // First account still owns token-first — seed must land on first, not second. clearProviderQuotaCache();