diff --git a/README.ja.md b/README.ja.md index ea575a23e..4a958779d 100644 --- a/README.ja.md +++ b/README.ja.md @@ -200,6 +200,7 @@ OpenAI の Thibault Sottiaux は、他のコーディングハーネスを通じ | Claude アプリ向けゲートウェイログイン — OAuth デバイスフロー、managed settings、ユーザー単位のポリシー | `public_url`、32 バイト以上の JWT シークレット、静的ユーザーまたは `[server.gateway.oidc]` を備えた `[server.gateway]` | [ガイド](https://shunt.sh/ja/guides/gateway-login/) | | ゲートウェイテレメトリの受信 — 管理対象クライアントの OTLP をそのままリレー | 構成済みの `[server.gateway]` と、`forward_to` が空でない `[server.gateway.telemetry]` | [リファレンス](https://shunt.sh/ja/reference/configuration/#servergatewaytelemetryオプション) | | 管理 Web 画面 — アカウントと使用量のダッシュボード、ブラウザーからのプロビジョニング | `[server.admin]` に管理者資格情報（`tokens_env`、`tokens_file`、または `write_keys` エントリ。`read_keys` エントリだけでもダッシュボードは読み取り専用で起動します — サインインと全ての閲覧はできますが、プロビジョニングには write が必要です）を自分で書くか、**または** `shunt dashboard setup` がテーブルの作成とトークンの発行をまとめて行います。ただしテーブルの作成とトークンの発行が起きるのは `[server.admin]` が存在しないときだけで、すでにある場合は既存の資格情報をそのままにし、欠けている `[server.oauth_usage]` だけを追加します。ダッシュボード本体は `--features ui` ビルドだけが埋め込むバンドルから配信されます — ビルド済みリリースバイナリと Homebrew formula には含まれ、素の `cargo build`/`cargo install` には含まれません | [ガイド](https://shunt.sh/ja/guides/admin-remote-provisioning/) | +| プールアカウント制御 — アカウントの一時停止、または利用可能なアカウントをクォータのリセットが最も早い順に並べ替え — ダッシュボードまたは admin API からランタイムに、`shunt.toml` の編集や再起動なしで | `[server.admin]` write 資格情報。`sort_by_reset` には対応する `[server.pool]` 設定キーもある | [ガイド](https://shunt.sh/ja/guides/pool-account-controls/) | | 支出上限 Admin API — 組織単位・ユーザー単位の上限（ステージ 1 は保存のみで、まだ適用しません） | `[server.admin]`（管理者資格情報が必須: `tokens_env`、`tokens_file`、または `write_keys`/`read_keys` エントリ — read 階層は GET のみ処理） + `[server.spend]` | [リファレンス](https://shunt.sh/ja/reference/configuration/#serverspendオプション) | | クライアント向け使用量エンドポイント — `GET /usage` がサニタイズ・集計されたプールの余裕を返す | `[server.auth]`（`tokens_env` にクライアントトークンが必須、既定値 `SHUNT_CLIENT_TOKENS`） + `[server.usage]` | [リファレンス](https://shunt.sh/ja/reference/configuration/#serverusageオプション) | | Claude Code CLI ネイティブ使用量バー — `GET /api/oauth/usage` を提供 | `[server.oauth_usage]`。ループバック以外の bind では `[server.auth]`（`tokens_env` にクライアントトークンが必須、既定値 `SHUNT_CLIENT_TOKENS`）または `[server.gateway]` も必要 | [リファレンス（英語）](https://shunt.sh/reference/configuration/#serveroauth_usage-optional) | diff --git a/README.ko.md b/README.ko.md index 8244ca10c..4062bf5ad 100644 --- a/README.ko.md +++ b/README.ko.md @@ -201,6 +201,7 @@ OpenAI의 Thibault Sottiaux는 다른 코딩 하네스를 통해 Codex를 실행 | Claude 앱 게이트웨이 로그인 — OAuth device flow, managed settings, 사용자별 정책 | `public_url`, 32바이트 이상 JWT 시크릿, 정적 사용자 또는 `[server.gateway.oidc]`를 갖춘 `[server.gateway]` | [가이드](https://shunt.sh/ko/guides/gateway-login/) | | 게이트웨이 텔레메트리 인제스트 — 관리 클라이언트의 OTLP를 그대로 릴레이 | 구성된 `[server.gateway]`와 `forward_to`가 비어 있지 않은 `[server.gateway.telemetry]` | [레퍼런스](https://shunt.sh/ko/reference/configuration/#servergatewaytelemetry-선택) | | 관리자 웹 화면 — 계정·사용량 대시보드, 브라우저 프로비저닝 | `[server.admin]`에 관리자 자격 증명(`tokens_env`, `tokens_file`, 또는 `write_keys` 항목)을 직접 작성하거나(`read_keys` 항목만 있어도 대시보드는 읽기 전용으로 뜹니다 — 로그인과 모든 조회는 되지만 프로비저닝에는 write가 필요합니다), **또는** `shunt dashboard setup`으로 테이블 작성과 토큰 발급을 한 번에 처리합니다. 단 테이블 작성과 토큰 발급은 `[server.admin]`이 없을 때만 일어납니다 — 이미 있으면 기존 자격 증명을 그대로 두고 빠진 `[server.oauth_usage]`만 추가합니다. 대시보드 자체는 `--features ui` 빌드만 임베드하는 번들에서 제공됩니다 — 사전 빌드 릴리스 바이너리와 Homebrew 포뮬러에는 포함되어 있고, 그냥 `cargo build`/`cargo install`로 빌드하면 포함되지 않습니다 | [가이드](https://shunt.sh/ko/guides/admin-remote-provisioning/) | +| 풀 계정 제어 — 계정을 일시정지하거나, 사용 가능한 계정을 가장 빨리 쿼터가 리셋되는 순으로 정렬 — 대시보드나 admin API에서 런타임에, `shunt.toml` 편집이나 재시작 없이 | `[server.admin]` write 자격 증명; `sort_by_reset`은 대응하는 `[server.pool]` 설정 키도 있음 | [가이드](https://shunt.sh/ko/guides/pool-account-controls/) | | 지출 한도 Admin API — 조직·사용자 단위 상한(1단계는 저장만 하고 아직 적용하지 않음) | `[server.admin]`(관리자 자격 증명 필요: `tokens_env`, `tokens_file`, 또는 `write_keys`/`read_keys` 항목 — read 등급은 GET만 처리) + `[server.spend]` | [레퍼런스](https://shunt.sh/ko/reference/configuration/#serverspend-선택) | | 클라이언트 사용량 엔드포인트 — `GET /usage`가 정제·집계된 풀 여유를 반환 | `[server.auth]`(`tokens_env`에 클라이언트 토큰 필요, 기본값 `SHUNT_CLIENT_TOKENS`) + `[server.usage]` | [레퍼런스](https://shunt.sh/ko/reference/configuration/#serverusage-선택) | | Claude Code CLI 네이티브 사용량 막대 — `GET /api/oauth/usage` 제공 | `[server.oauth_usage]`, 루프백이 아닌 bind에서는 `[server.auth]`(`tokens_env`에 클라이언트 토큰 필요, 기본값 `SHUNT_CLIENT_TOKENS`) 또는 `[server.gateway]` 추가 필요 | [레퍼런스(영문)](https://shunt.sh/reference/configuration/#serveroauth_usage-optional) | diff --git a/README.md b/README.md index fd52ac20b..9138b877c 100644 --- a/README.md +++ b/README.md @@ -200,6 +200,7 @@ Unless a row says otherwise, these are **off by default** — absent its config | Claude apps gateway login — OAuth device flow, managed settings, per-user policy | `[server.gateway]` with `public_url`, a 32-byte-or-longer JWT secret, and static users or `[server.gateway.oidc]` | [How-to](https://shunt.sh/guides/gateway-login/) | | Gateway telemetry ingest — verbatim OTLP relay for managed clients | a configured `[server.gateway]`, plus `[server.gateway.telemetry]` with a non-empty `forward_to` | [Reference](https://shunt.sh/reference/configuration/#servergatewaytelemetry-optional) | | Admin web surface — accounts and usage dashboard, browser provisioning | `[server.admin]` with an admin credential (`tokens_env`, `tokens_file`, or a `write_keys` entry; a `read_keys` entry alone brings the dashboard up read-only — it signs in and serves every view, but provisioning needs write) — **or** `shunt dashboard setup`, which writes the table and mints a token, but only when `[server.admin]` is absent: against an existing block it leaves your credential untouched and only adds a missing `[server.oauth_usage]`. The dashboard itself is served from a bundle only a `--features ui` build embeds — prebuilt release binaries and the Homebrew formula have it, a plain `cargo build`/`cargo install` does not | [How-to](https://shunt.sh/guides/admin-remote-provisioning/) | +| Pool account controls — pause an account, or rank available accounts by soonest quota reset, from the dashboard or admin API, at runtime, with no `shunt.toml` edit or restart | `[server.admin]` write credential; `sort_by_reset` also has an equivalent `[server.pool]` config key | [How-to](https://shunt.sh/guides/pool-account-controls/) | | Spend-limit Admin API — organization- and user-scoped caps (stage 1 stores, does not enforce) | `[server.admin]` with an admin credential (`tokens_env`, `tokens_file`, or a `write_keys`/`read_keys` entry — read-tier serves the GETs) + `[server.spend]` | [Reference](https://shunt.sh/reference/configuration/#serverspend-optional) | | Client usage endpoint — sanitized, aggregated pool headroom at `GET /usage` | `[server.auth]` with client tokens in `tokens_env` (default `SHUNT_CLIENT_TOKENS`) + `[server.usage]` | [Reference](https://shunt.sh/reference/configuration/#serverusage-optional) | | Claude Code CLI native usage bars — serves `GET /api/oauth/usage` | `[server.oauth_usage]`, plus `[server.auth]` (client tokens in `tokens_env`, default `SHUNT_CLIENT_TOKENS`) or `[server.gateway]` on a non-loopback bind | [Reference](https://shunt.sh/reference/configuration/#serveroauth_usage-optional) | diff --git a/README.zh-CN.md b/README.zh-CN.md index aa34ae8c4..4d591614e 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -198,6 +198,7 @@ OpenAI 的 Thibault Sottiaux 已公开欢迎通过其他编码 harness 运行 Co | Claude 应用网关登录 —— OAuth 设备流、managed settings、按用户策略 | 具备 `public_url`、不少于 32 字节的 JWT 密钥,以及静态用户或 `[server.gateway.oidc]` 的 `[server.gateway]` | [指南](https://shunt.sh/zh-cn/guides/gateway-login/) | | 网关遥测接收 —— 原样转发受管客户端的 OTLP | 已配置的 `[server.gateway]`,以及 `forward_to` 非空的 `[server.gateway.telemetry]` | [参考](https://shunt.sh/zh-cn/reference/configuration/#servergatewaytelemetry可选) | | 管理 Web 界面 —— 账号与用量看板、浏览器预配 | 手写 `[server.admin]` 并提供管理员凭据（`tokens_env`、`tokens_file` 或一条 `write_keys`；仅有一条 `read_keys` 也能让看板以只读方式启动 —— 可以登录并查看全部视图，但预配需要 write），**或者**用 `shunt dashboard setup` 一次性写入配置表并签发令牌 —— 但写入配置表和签发令牌仅发生在 `[server.admin]` 不存在时；若已存在，它会保留现有凭据，只补上缺失的 `[server.oauth_usage]`。看板本身由只有 `--features ui` 构建才会内嵌的前端包提供 —— 预构建的发布二进制和 Homebrew formula 已包含，普通的 `cargo build`/`cargo install` 则没有 | [指南](https://shunt.sh/zh-cn/guides/admin-remote-provisioning/) | +| 账户池控制 —— 暂停某个账户,或按配额最快重置的顺序对可用账户排序 —— 在仪表盘或 admin API 中运行时完成,无需编辑 `shunt.toml` 或重启 | `[server.admin]` write 凭据;`sort_by_reset` 也有对应的 `[server.pool]` 配置键 | [指南](https://shunt.sh/zh-cn/guides/pool-account-controls/) | | 支出上限 Admin API —— 组织级和用户级上限(stage 1 只存储,尚未实施) | `[server.admin]`（必须提供管理员凭据: `tokens_env`、`tokens_file` 或一条 `write_keys`/`read_keys` —— read 级别只服务 GET） + `[server.spend]` | [参考](https://shunt.sh/zh-cn/reference/configuration/#serverspend可选) | | 客户端用量端点 —— `GET /usage` 返回脱敏聚合后的池余量 | `[server.auth]`（必须在 `tokens_env` 中提供客户端令牌,默认 `SHUNT_CLIENT_TOKENS`） + `[server.usage]` | [参考](https://shunt.sh/zh-cn/reference/configuration/#serverusage可选) | | Claude Code CLI 原生用量条 —— 提供 `GET /api/oauth/usage` | `[server.oauth_usage]`;非回环 bind 还需 `[server.auth]`（必须在 `tokens_env` 中提供客户端令牌,默认 `SHUNT_CLIENT_TOKENS`）或 `[server.gateway]` | [参考(英文)](https://shunt.sh/reference/configuration/#serveroauth_usage-optional) | diff --git a/site/src/content/docs/guides/pool-account-controls.md b/site/src/content/docs/guides/pool-account-controls.md new file mode 100644 index 000000000..e850693a8 --- /dev/null +++ b/site/src/content/docs/guides/pool-account-controls.md @@ -0,0 +1,68 @@ +--- +title: Pool Account Controls +description: Pause an individual pool account and rank available accounts by soonest quota reset — both from the admin dashboard, at runtime, with no config edit or restart. +--- + +Two runtime controls sit on top of account-pool selection ([Anthropic Multi-Account](/guides/anthropic-multi-account/), [Codex Multi-Account](/guides/codex-multi-account/)): pausing a single account, and ranking available accounts by soonest quota reset instead of burn-rate headroom. Both are operated from the admin dashboard's "Managed pool health" table, or directly against the admin API. Both are memory-only — a restart clears them — so neither touches `shunt.toml`. + +## Pausing an account + +Pausing excludes one account from selection exactly as its config-side `disabled = true` would, without editing `shunt.toml` or signing the account out. The credential and its quota history are untouched; a paused account simply never appears as a selection candidate until resumed. + +This is different from `disabled`: + +| | `disabled` (config) | `paused` (runtime) | +| :-- | :-- | :-- | +| Set via | `shunt.toml`, reloaded | Admin dashboard or `PATCH /admin/api/pool/{provider}/accounts/{account_ref}` | +| Survives a restart | Yes | No | +| Use case | Permanently remove an account from the deployment | Temporary operator intervention — pull an account aside for a few minutes without a config round-trip | + +From the dashboard: open **Manage pool accounts → Managed pool health**, and click **Pause** on the account's row. Its state shows as `paused`; click **Resume** to bring it back. Both buttons require a write-tier admin session. + +Directly against the API (write-tier credential required): + +The `account_ref` is the opaque identifier returned on each account object by `GET /admin/api/pool`. Use it rather than the display `name`, so distinct accounts that share a name remain independently addressable. + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool/anthropic/accounts/$ACCOUNT_REF" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"paused": true}' +``` + +Set `"paused": false` to resume. See [`PATCH /admin/api/pool/{provider}/accounts/{account_ref}`](/reference/endpoints/) for the full endpoint reference. + +## Ranking by soonest reset + +`[server.pool] sort_by_reset` (default `false`) changes how the *available* tier is ordered: instead of largest projected burn-rate headroom, accounts sort by their earliest known quota reset (ascending — the account that recovers soonest is tried first; an account with no reset signal sorts last). The idea is to drain the account that will replenish earliest, keeping accounts with later resets in reserve as buffers. + +`[server.pool]` — and therefore this setting — is process-wide, not per-provider, so toggling it affects every pooled provider at once. + +Set it in `shunt.toml`: + +```toml +[server.pool] +sort_by_reset = true +``` + +Or toggle it at runtime from the dashboard's checkbox above the pool table ("Rank available accounts by soonest quota reset instead of burn-rate headroom"), or directly: + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"sort_by_reset": true}' +``` + +A runtime toggle overrides the config file's value until cleared or the process restarts, at which point the config file's own value applies again. To clear an override explicitly without restarting, send `null`: + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"sort_by_reset": null}' +``` + +Omitting the field entirely is a no-op — it leaves the current override (or its absence) untouched; only an explicit `null` clears it. + +`GET /admin/api/pool` reports the effective value (override or config) as a top-level `sort_by_reset` boolean. This setting has no effect at all while `[server.pool]` itself is absent — the legacy selection path it would otherwise change never runs — so both the runtime toggle and `GET /admin/api/pool`'s reported value are inert until `[server.pool]` exists. diff --git a/site/src/content/docs/ja/guides/pool-account-controls.md b/site/src/content/docs/ja/guides/pool-account-controls.md new file mode 100644 index 000000000..3294ebe02 --- /dev/null +++ b/site/src/content/docs/ja/guides/pool-account-controls.md @@ -0,0 +1,68 @@ +--- +title: プールアカウント制御 +description: 個々のプールアカウントを一時停止し、利用可能なアカウントをクォータのリセットが最も早い順に並べ替える — どちらも管理ダッシュボードからランタイムで、設定変更や再起動なしに。 +--- + +アカウントプールの選択([Anthropic マルチアカウント](/ja/guides/anthropic-multi-account/)、[Codex マルチアカウント](/ja/guides/codex-multi-account/))の上に、2 つのランタイム制御があります。単一アカウントの一時停止と、利用可能なアカウントをバーンレートのヘッドルームではなくクォータのリセットが最も早い順に並べ替えることです。どちらも管理ダッシュボードの「Managed pool health」テーブルから、または管理 API を直接呼び出して操作します。どちらもメモリ上のみの状態です — 再起動すると消えるため、`shunt.toml` には一切触れません。 + +## アカウントの一時停止 + +一時停止は、設定側の `disabled = true` と同様にアカウントを選択対象から除外しますが、`shunt.toml` を編集したりアカウントをサインアウトさせたりしません。資格情報とクォータ履歴はそのまま維持され、一時停止されたアカウントは再開されるまで選択候補として現れません。 + +`disabled` との違い: + +| | `disabled`(設定) | `paused`(ランタイム) | +| :-- | :-- | :-- | +| 設定方法 | `shunt.toml`、リロードが必要 | 管理ダッシュボードまたは `PATCH /admin/api/pool/{provider}/accounts/{account_ref}` | +| 再起動後も維持 | はい | いいえ | +| ユースケース | デプロイからアカウントを恒久的に除外する | 設定の往復なしに、しばらくの間アカウントを外しておく一時的な運用介入 | + +ダッシュボードから: **Manage pool accounts → Managed pool health** を開き、該当アカウントの行の **Pause** をクリックします。状態が `paused` と表示され、**Resume** をクリックすると復帰します。どちらのボタンも write 権限の管理セッションが必要です。 + +API を直接呼び出す場合(write 権限の資格情報が必要): + +`account_ref` は `GET /admin/api/pool` の各 account オブジェクトに含まれる不透明な識別子です。表示名 `name` ではなくこの値を使うため、同名の別アカウントも個別に操作できます。 + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool/anthropic/accounts/$ACCOUNT_REF" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"paused": true}' +``` + +再開するには `"paused": false` を設定します。エンドポイントの完全なリファレンスは [`PATCH /admin/api/pool/{provider}/accounts/{account_ref}`](/ja/reference/endpoints/) を参照してください。 + +## リセットが最も早い順に並べ替える + +`[server.pool] sort_by_reset`(デフォルト `false`)は *available* 階層の並べ替え方法を変更します。予測されるバーンレートのヘッドルームが最大のものではなく、既知のクォータリセットが最も早いアカウントから順に並べます(昇順 — 最も早く回復するアカウントが最初に試され、リセットの兆候がないアカウントは最後に並びます)。狙いは、最も早く補充されるアカウントから消費し、リセットが遅いアカウントは予備として残しておくことです。 + +`[server.pool]` は — したがってこの設定も — プロバイダー単位ではなくプロセス全体に適用されるため、切り替えるとプールされたすべてのプロバイダーに同時に影響します。 + +`shunt.toml` で設定する場合: + +```toml +[server.pool] +sort_by_reset = true +``` + +または、プールテーブルの上にあるダッシュボードのチェックボックス(「Rank available accounts by soonest quota reset instead of burn-rate headroom」)でランタイムに切り替えるか、直接呼び出します: + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"sort_by_reset": true}' +``` + +ランタイムの切り替えは、解除されるかプロセスが再起動されるまで設定ファイルの値を上書きし、その後は再び設定ファイル自身の値が適用されます。再起動せずに明示的にオーバーライドを解除するには `null` を送信します: + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"sort_by_reset": null}' +``` + +フィールドを完全に省略すると何もしません — 現在のオーバーライド(またはその不在)をそのまま残します。明示的な `null` だけが解除します。 + +`GET /admin/api/pool` は有効な値(オーバーライドまたは設定値)をトップレベルの `sort_by_reset` ブール値として報告します。`[server.pool]` 自体が存在しない場合、この設定はまったく効果を持ちません — この設定が変更するはずのレガシー選択パス自体が実行されないためです — したがって `[server.pool]` が存在するまでは、ランタイムの切り替えも `GET /admin/api/pool` が報告する値も意味を持ちません。 diff --git a/site/src/content/docs/ja/reference/configuration.md b/site/src/content/docs/ja/reference/configuration.md index b2e5f659a..2c4a35bb2 100644 --- a/site/src/content/docs/ja/reference/configuration.md +++ b/site/src/content/docs/ja/reference/configuration.md @@ -232,6 +232,7 @@ headers = { "x-api-key" = "..." } | `default_threshold_7d` | 未設定 | 共有の週次（`7d`）ウィンドウのソフトなデフォルト | | `default_threshold_fable` | 未設定 | fable 専用の週次（`7d_oi`）ウィンドウのソフトなデフォルト | | `burn_rate_avoidance` | `false` | ウィンドウのリセット前にソフトしきい値を使い切ると予測されるアカウントも回避する | +| `sort_by_reset` | `false` | バーンレートのヘッドルームではなく、利用可能なアカウントをクォータのリセットが最も早い順(昇順、不明なリセットは最後)に並べ替える。`shunt.toml` を編集せずに、管理ダッシュボードまたは `PATCH /admin/api/pool` でランタイムに切り替え可能 — [プールアカウント制御](/ja/guides/pool-account-controls/)を参照 | | `usage_refresh_seconds` | 無効（`0`/未設定） | Claude `GET /api/oauth/usage`、Codex `GET /wham/usage`、Antigravity `POST :retrieveUserQuotaSummary` のポーリング間隔（秒）。60 未満の正の値は 60 秒の下限に切り上げられます | | `state_path` | 未設定 | プールのアカウント単位のクォータ状態を保存するファイル。再起動時に空のプールではなく、最後に観測された使用率からウォームスタートします。未設定で永続化は無効（デフォルト） | | `ramp_initial_concurrency` | 無効（`0`/未設定） | ストーム制御: トラフィックを受け始めたばかりのアカウントアイデンティティに対する初期の並行受け入れ許容量。`0` または未設定で受け入れゲーティングは無効 | diff --git a/site/src/content/docs/ja/reference/endpoints.md b/site/src/content/docs/ja/reference/endpoints.md index 5d469d92e..024ffdc5f 100644 --- a/site/src/content/docs/ja/reference/endpoints.md +++ b/site/src/content/docs/ja/reference/endpoints.md @@ -28,7 +28,9 @@ description: shunt が Claude Code LLM ゲートウェイとして提供する | `GET` | `/admin/api/accounts/codex` | Codex アカウントストアのメタデータ: 名前、有効期限、ChatGPT アカウント ID。トークン本体は決して返さない | | `GET` | `/admin/api/accounts/antigravity` | Antigravity アカウントストアのメタデータ: 名前、有効期限、メールアドレス（Google が返した場合）。トークン本体は決して返さない | | `GET` | `/admin/api/observed` | ローカルの Claude Code、Codex、Gemini、Kimi、Grok、Cursor のアイデンティティとプロバイダー固有の使用量を読み取り専用で返す。トークン本体は決して返さず、元の認証情報をリフレッシュすることもない。[`[server.admin].hide_observed`](/ja/reference/configuration/#serveradminオプション) が true の場合は、これらのソースを一切読まずに `{ "accounts": [] }` を返す | -| `GET` | `/admin/api/pool` | `claude_oauth` / `chatgpt_oauth` / `kimi_oauth` / `antigravity_oauth` provider ごとのプール状態。各 account オブジェクトには任意の `plan` 文字列が含まれることがあり、ファイルから読んだ値は後の profile 照会でより精密な値に補正されることがあり、Codex の行には報告された 5h/7d 使用量が含まれる(`7d_oi` に対応する Codex の項目はない)。各 account には真偽値 `needs_relogin` も含まれる。クレデンシャルが終端的に拒否された(`invalid_grant`)か、リフレッシュトークンをそもそも持たないか、ローテーションされたトークン対を保存できずに失った場合で、どのリトライでも回復せず、オペレーターの再ログインだけが解決策となる。クールダウンのフィールドとは**独立に**報告される — クールダウンは自然に失効するが、この印は残る — ダッシュボードの二つの表はいずれもクォータ一時停止の `cooling` ではなく **needs re-login** と表示する。メモリ上のみで保持されるため、再起動でクリアされ、そのアカウントの次の終端的な失敗で再び立つ。どの provider テーブルも一度も選択したことのないアカウントについても — `has_state: false` と並んで — 報告される。admin の refresh プローブが判定をストア名で記録するためである。 | +| `GET` | `/admin/api/pool` | `claude_oauth` / `chatgpt_oauth` / `kimi_oauth` / `antigravity_oauth` provider ごとのプール状態。各 account オブジェクトには任意の `plan` 文字列が含まれることがあり、ファイルから読んだ値は後の profile 照会でより精密な値に補正されることがあり、Codex の行には報告された 5h/7d 使用量が含まれる(`7d_oi` に対応する Codex の項目はない)。各 account には真偽値 `needs_relogin` も含まれる。クレデンシャルが終端的に拒否された(`invalid_grant`)か、リフレッシュトークンをそもそも持たないか、ローテーションされたトークン対を保存できずに失った場合で、どのリトライでも回復せず、オペレーターの再ログインだけが解決策となる。クールダウンのフィールドとは**独立に**報告される — クールダウンは自然に失効するが、この印は残る — ダッシュボードの二つの表はいずれもクォータ一時停止の `cooling` ではなく **needs re-login** と表示する。メモリ上のみで保持されるため、再起動でクリアされ、そのアカウントの次の終端的な失敗で再び立つ。どの provider テーブルも一度も選択したことのないアカウントについても — `has_state: false` と並んで — 報告される。admin の refresh プローブが判定をストア名で記録するためである。各 account には真偽値 `paused` も含まれる。これはオペレーターが `shunt.toml` に触れずに切り替えられるランタイム限定の除外フラグである(下の `PATCH .../accounts/{account_ref}` を参照)。レスポンス最上位の `sort_by_reset` は `[server.pool] sort_by_reset` またはそのランタイムオーバーライドを反映し(下のボディなしの `PATCH` を参照)、provider ごとではなくプロセス全体に適用される。 各 account には不透明な `account_ref` も含まれ、provider ごとのプール識別子から導出される。ミューテーションは表示名 `name` ではなくこの値を使うため、同名の別アカウントも安全に個別操作できる。 | +| `PATCH` | `/admin/api/pool/{provider}/accounts/{account_ref}` | write 権限限定。ボディ `{"paused": true\|false}` で指定したアカウントのランタイム一時停止を切り替える。`disabled` と同様に選択から除外されるが、メモリ上のみで(再起動でクリアされる)、`shunt.toml` とは独立している。不明な provider または account には `404` `{account_ref}` は `GET /admin/api/pool` の対応する account オブジェクトから取得する。表示名はミューテーションの対象指定には使わない。 | +| `PATCH` | `/admin/api/pool` | write 権限限定。ボディ `{"sort_by_reset": true\|false}` でプール全体のリセット優先ソートをランタイムに切り替える(`[server.pool]` は provider ごとではなくプロセス全体であるため)。利用可能なアカウントはバーンレートのヘッドルームではなくクォータのリセットが最も早い順に並ぶ。`{"sort_by_reset": null}` は以前に設定したオーバーライドを明示的に解除し、設定ファイル自身の値に戻す(フィールドの省略は何もしない — 解除ではない)。オーバーライドはメモリ上のみで、再起動でも戻る。`[server.pool]` 自体が存在しない場合 — そのレガシー選択パスはこの値を一切参照しないため — オーバーライドの有無にかかわらず効果はなく、`GET /admin/api/pool` も `sort_by_reset: false` を報告する | | `GET` | `/admin/api/routes` | 管理者資格情報で保護された解決済みのルーティング表で、ダッシュボードから利用できます — 上の `GET /routes` が返すものと同じボディで、同じスナップショットから構築されるため、運用者が見る表とクライアントが実際に解決する表が食い違うことはありません。リダイレクトではなく個別に登録しているのは、管理面が自前の認証を保つためです。公開の `/routes` は意図的に認証を持たないので、こちらをそちらへ向けると両者の認証が結び付いてしまいます。ここで管理者資格情報を要求するため、`/routes` を開放していても当該資格情報が守る範囲は広がりません | | `POST` | `/admin/api/accounts/claude` | `{name, mode}` で Claude のブラウザープロビジョニングを開始。`mode` は `oauth` または `setup_token` で、省略時は `setup_token`。`{authorize_url}` を返す | | `POST` | `/admin/api/accounts/claude/{name}/complete` | `#` を含む `{code}` で Claude プロビジョニングを完了。アカウントを保存し、有効（live）かどうかを報告 | diff --git a/site/src/content/docs/ko/guides/pool-account-controls.md b/site/src/content/docs/ko/guides/pool-account-controls.md new file mode 100644 index 000000000..94da109b4 --- /dev/null +++ b/site/src/content/docs/ko/guides/pool-account-controls.md @@ -0,0 +1,68 @@ +--- +title: 풀 계정 제어 +description: 개별 풀 계정을 일시 정지하고, 사용 가능한 계정을 가장 빨리 리셋되는 순서로 정렬하기 — 둘 다 관리자 대시보드에서 런타임에, 설정 변경이나 재시작 없이. +--- + +계정 풀 선택([Anthropic 멀티 계정](/ko/guides/anthropic-multi-account/), [Codex 멀티 계정](/ko/guides/codex-multi-account/)) 위에는 두 가지 런타임 제어가 있습니다: 단일 계정 일시 정지, 그리고 사용 가능한 계정을 번-레이트 헤드룸 대신 가장 빨리 쿼터가 리셋되는 순서로 정렬하는 것입니다. 두 기능 모두 관리자 대시보드의 "Managed pool health" 표에서, 또는 관리자 API를 직접 호출해서 조작합니다. 둘 다 메모리 전용입니다 — 재시작하면 초기화되므로 `shunt.toml`은 전혀 건드리지 않습니다. + +## 계정 일시 정지 + +일시 정지는 설정 쪽의 `disabled = true`와 마찬가지로 계정을 선택 대상에서 제외하지만, `shunt.toml`을 편집하거나 계정을 로그아웃시키지 않습니다. 자격 증명과 쿼터 이력은 그대로 유지되며, 일시 정지된 계정은 재개될 때까지 단순히 선택 후보로 나타나지 않습니다. + +`disabled`와는 다릅니다: + +| | `disabled` (설정) | `paused` (런타임) | +| :-- | :-- | :-- | +| 설정 방법 | `shunt.toml`, 리로드 필요 | 관리자 대시보드 또는 `PATCH /admin/api/pool/{provider}/accounts/{account_ref}` | +| 재시작 후 유지 | 예 | 아니오 | +| 사용 사례 | 배포에서 계정을 영구적으로 제외 | 설정 왕복 없이 잠시 계정을 빼두는 임시 운영 개입 | + +대시보드에서: **Manage pool accounts → Managed pool health**를 열고 해당 계정 행의 **Pause**를 클릭합니다. 상태가 `paused`로 표시되며, **Resume**을 클릭하면 다시 복귀합니다. 두 버튼 모두 write 등급 관리자 세션이 필요합니다. + +API를 직접 호출하는 경우 (write 등급 자격 증명 필요): + +`account_ref`는 `GET /admin/api/pool`의 각 account 객체가 반환하는 불투명 식별자입니다. 표시 이름 `name` 대신 이 값을 사용하므로 같은 이름의 서로 다른 계정도 개별적으로 제어할 수 있습니다. + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool/anthropic/accounts/$ACCOUNT_REF" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"paused": true}' +``` + +재개하려면 `"paused": false`로 설정하세요. 전체 엔드포인트 참조는 [`PATCH /admin/api/pool/{provider}/accounts/{account_ref}`](/ko/reference/endpoints/)를 참고하세요. + +## 가장 빨리 리셋되는 순서로 정렬하기 + +`[server.pool] sort_by_reset`(기본값 `false`)은 *available* 계층의 정렬 방식을 바꿉니다: 가장 큰 예상 번-레이트 헤드룸 대신, 계정을 알려진 쿼터 리셋 시각이 가장 이른 순서(오름차순 — 가장 먼저 회복되는 계정을 먼저 시도하고, 리셋 신호가 없는 계정은 맨 뒤로 정렬)로 정렬합니다. 아이디어는 가장 먼저 보충될 계정을 먼저 소진시켜, 리셋이 늦은 계정은 예비로 남겨두는 것입니다. + +`[server.pool]`은 — 따라서 이 설정도 — 프로바이더별이 아니라 프로세스 전체에 적용되므로, 이를 토글하면 풀링된 모든 프로바이더에 동시에 영향을 줍니다. + +`shunt.toml`에서 설정: + +```toml +[server.pool] +sort_by_reset = true +``` + +또는 풀 표 위의 대시보드 체크박스("Rank available accounts by soonest quota reset instead of burn-rate headroom")로 런타임에 토글하거나, 직접 호출: + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"sort_by_reset": true}' +``` + +런타임 토글은 해제되거나 프로세스가 재시작될 때까지 설정 파일의 값을 덮어쓰며, 그 이후에는 다시 설정 파일의 값이 적용됩니다. 재시작 없이 명시적으로 오버라이드를 해제하려면 `null`을 보내세요: + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"sort_by_reset": null}' +``` + +필드를 아예 생략하면 아무 효과가 없습니다 — 현재 오버라이드(또는 그 부재)를 그대로 둡니다; 명시적인 `null`만이 해제합니다. + +`GET /admin/api/pool`은 유효한 값(오버라이드 또는 설정값)을 최상위 `sort_by_reset` 불리언으로 보고합니다. `[server.pool]` 자체가 없으면 이 설정은 전혀 효과가 없습니다 — 이 설정이 바꾸려는 레거시 선택 경로 자체가 실행되지 않기 때문입니다 — 따라서 `[server.pool]`이 존재하기 전까지는 런타임 토글과 `GET /admin/api/pool`이 보고하는 값 모두 아무 의미가 없습니다. diff --git a/site/src/content/docs/ko/reference/configuration.md b/site/src/content/docs/ko/reference/configuration.md index a596b0dc7..b72f0950a 100644 --- a/site/src/content/docs/ko/reference/configuration.md +++ b/site/src/content/docs/ko/reference/configuration.md @@ -231,6 +231,7 @@ headers = { "x-api-key" = "..." } | `default_threshold_7d` | 미설정 | 공유 주간(`7d`) 창의 소프트 기본값 | | `default_threshold_fable` | 미설정 | fable 전용 주간(`7d_oi`) 창의 소프트 기본값 | | `burn_rate_avoidance` | `false` | 창이 리셋되기 전에 소프트 임계값을 소진할 것으로 예측되는 계정도 함께 회피 | +| `sort_by_reset` | `false` | 번-레이트 헤드룸 대신 사용 가능한 계정을 쿼터 리셋이 가장 빠른 순(오름차순; 알 수 없는 리셋은 맨 뒤)으로 정렬. `shunt.toml`을 편집하지 않고 관리자 대시보드나 `PATCH /admin/api/pool`로 런타임에 토글 가능 — [풀 계정 제어](/ko/guides/pool-account-controls/) 참고 | | `usage_refresh_seconds` | 비활성(`0`/미설정) | Claude `GET /api/oauth/usage`, Codex `GET /wham/usage`, Antigravity `POST :retrieveUserQuotaSummary`의 폴링 간격(초); 60 미만의 양수 값은 60초 하한으로 올림 | | `state_path` | 미설정 | 풀의 계정별 쿼터 상태를 저장할 파일; 재시작 시 빈 풀 대신 마지막으로 관측된 사용률에서 워밍업. 미설정이면 영속화 비활성(기본값) | | `ramp_initial_concurrency` | 비활성(`0`/미설정) | 폭주 제어: 방금 트래픽을 받기 시작한 계정 아이덴티티의 초기 동시 허용치. `0` 또는 미설정이면 허용 게이팅 비활성 | diff --git a/site/src/content/docs/ko/reference/endpoints.md b/site/src/content/docs/ko/reference/endpoints.md index 1fadcd0bc..68d5d47da 100644 --- a/site/src/content/docs/ko/reference/endpoints.md +++ b/site/src/content/docs/ko/reference/endpoints.md @@ -28,7 +28,9 @@ description: shunt가 Claude Code LLM 게이트웨이로서 제공하는 엔드 | `GET` | `/admin/api/accounts/codex` | Codex 계정 스토어 메타데이터: 이름, 만료, ChatGPT 계정 ID; 토큰 자체는 절대 반환하지 않음 | | `GET` | `/admin/api/accounts/antigravity` | Antigravity 계정 스토어 메타데이터: 이름, 만료, 이메일(Google이 반환한 경우); 토큰 자체는 절대 반환하지 않음 | | `GET` | `/admin/api/observed` | 로컬 Claude Code, Codex, Gemini, Kimi, Grok, Cursor 신원과 프로바이더 자체 사용량을 읽기 전용으로 제공; 토큰 자체는 절대 반환하지 않으며 원본 자격 증명을 갱신하지도 않음. [`[server.admin].hide_observed`](/ko/reference/configuration/#serveradmin-선택)가 true이면 이 소스들을 전혀 읽지 않고 `{ "accounts": [] }`를 반환 | -| `GET` | `/admin/api/pool` | `claude_oauth`, `chatgpt_oauth`, `kimi_oauth`, `antigravity_oauth` 프로바이더별 풀 상태; 각 account 객체에는 선택적인 `plan` 문자열이 포함될 수 있고 파일에서 읽은 값은 이후 profile 조회로 더 정밀하게 보정될 수 있으며, Codex 행은 보고된 5시간/7일 사용량을 담으며 `7d_oi`에는 Codex 대응 항목이 없음; 각 account에는 불리언 `needs_relogin`도 실린다: 크리덴셜이 종결적으로 거부되었거나(`invalid_grant`), 리프레시 토큰이 아예 없거나, 회전된 토큰 쌍을 저장하지 못해 잃어버린 경우로, 어떤 재시도로도 되살릴 수 없고 운영자 재로그인만이 해결한다. 쿨다운 필드와 **독립적으로** 보고된다 — 쿨다운은 저절로 만료되지만 이 표식은 남는다 — 그리고 대시보드의 두 표 모두 쿼터 일시정지의 `cooling`이 아니라 **needs re-login**으로 표시한다. 인메모리라 재시작하면 초기화되고, 해당 계정의 다음 종결 실패에서 다시 세워진다. 어떤 provider 테이블도 선택한 적 없는 계정에도 — `has_state: false`와 함께 — 보고된다. admin refresh 프로브가 판정을 스토어 이름으로 기록하기 때문이다. | +| `GET` | `/admin/api/pool` | `claude_oauth`, `chatgpt_oauth`, `kimi_oauth`, `antigravity_oauth` 프로바이더별 풀 상태; 각 account 객체에는 선택적인 `plan` 문자열이 포함될 수 있고 파일에서 읽은 값은 이후 profile 조회로 더 정밀하게 보정될 수 있으며, Codex 행은 보고된 5시간/7일 사용량을 담으며 `7d_oi`에는 Codex 대응 항목이 없음; 각 account에는 불리언 `needs_relogin`도 실린다: 크리덴셜이 종결적으로 거부되었거나(`invalid_grant`), 리프레시 토큰이 아예 없거나, 회전된 토큰 쌍을 저장하지 못해 잃어버린 경우로, 어떤 재시도로도 되살릴 수 없고 운영자 재로그인만이 해결한다. 쿨다운 필드와 **독립적으로** 보고된다 — 쿨다운은 저절로 만료되지만 이 표식은 남는다 — 그리고 대시보드의 두 표 모두 쿼터 일시정지의 `cooling`이 아니라 **needs re-login**으로 표시한다. 인메모리라 재시작하면 초기화되고, 해당 계정의 다음 종결 실패에서 다시 세워진다. 어떤 provider 테이블도 선택한 적 없는 계정에도 — `has_state: false`와 함께 — 보고된다. admin refresh 프로브가 판정을 스토어 이름으로 기록하기 때문이다. 각 account에는 불리언 `paused`도 실리는데, 운영자가 `shunt.toml`을 건드리지 않고 토글하는 런타임 전용 제외 플래그다(아래 `PATCH .../accounts/{account_ref}` 참고); 응답 최상위의 `sort_by_reset`은 `[server.pool] sort_by_reset` 값이나 그 런타임 오버라이드를 반영하며(아래 바디 없는 `PATCH` 참고), 프로바이더별이 아니라 프로세스 전체에 적용된다. 각 account에는 provider 범위의 풀 식별자에서 파생된 불투명 `account_ref`도 포함된다. mutation은 표시 이름 `name` 대신 이 값을 사용하므로 같은 이름의 서로 다른 계정도 안전하게 개별 제어할 수 있다. | +| `PATCH` | `/admin/api/pool/{provider}/accounts/{account_ref}` | write 등급 전용. 바디 `{"paused": true\|false}`로 해당 계정의 런타임 일시정지를 토글한다: `disabled`와 동일하게 선택 대상에서 제외되지만 인메모리라(재시작 시 초기화) `shunt.toml`과 무관하다. 알 수 없는 provider나 account에는 `404` `{account_ref}`는 `GET /admin/api/pool`의 해당 account 객체에서 가져온다. 표시 이름은 mutation 대상을 지정하는 데 사용하지 않는다. | +| `PATCH` | `/admin/api/pool` | write 등급 전용. 바디 `{"sort_by_reset": true\|false}`로 풀 전체의 리셋 우선 정렬을 런타임에 토글한다(`[server.pool]`은 프로바이더별이 아니라 프로세스 전체이므로): 사용 가능한 계정이 번-레이트 헤드룸 대신 쿼터 리셋이 가장 빠른 순으로 정렬된다. `{"sort_by_reset": null}`은 이전에 설정한 오버라이드를 명시적으로 해제하여 설정 파일 자체 값으로 되돌린다(필드 생략은 아무 효과 없음 — 해제가 아님); 오버라이드는 인메모리이며 재시작 시에도 되돌아간다. `[server.pool]` 자체가 없으면 — 그 레거시 선택 경로는 이 값을 전혀 참조하지 않으므로 — 오버라이드와 무관하게 아무 효과가 없고 `GET /admin/api/pool`도 `sort_by_reset: false`를 보고한다 | | `GET` | `/admin/api/routes` | 관리자 자격 증명 뒤에서 제공하는 해석된 라우팅 표로, 대시보드가 사용할 수 있습니다 — 위의 `GET /routes`가 반환하는 것과 같은 본문이며, 같은 스냅샷에서 만들어지므로 운영자가 보는 표와 클라이언트가 실제로 해석하는 표가 어긋날 수 없습니다. 리다이렉트가 아니라 별도로 등록하는 이유는 관리자 네임스페이스가 자체 인증을 유지하기 위해서입니다. 공개 `/routes`는 의도적으로 인증을 두지 않으므로, 이 엔드포인트를 그쪽으로 넘기면 두 표면의 인증이 한데 묶입니다. 여기서 관리자 자격 증명을 요구하므로 `/routes`를 열어 두어도 그 자격 증명이 가리는 범위가 넓어지지 않습니다 | | `POST` | `/admin/api/accounts/claude` | `{name, mode}`로 Claude 브라우저 프로비저닝 시작. `mode`는 `oauth` 또는 `setup_token`이며, 생략하면 `setup_token`; `{authorize_url}` 반환 | | `POST` | `/admin/api/accounts/claude/{name}/complete` | `#`가 담긴 `{code}`로 Claude 프로비저닝 완료; 계정을 저장하고 실제 사용 여부(live)를 보고 | diff --git a/site/src/content/docs/reference/configuration.md b/site/src/content/docs/reference/configuration.md index 250f16dd8..eb66a8fae 100644 --- a/site/src/content/docs/reference/configuration.md +++ b/site/src/content/docs/reference/configuration.md @@ -335,6 +335,7 @@ Quota-aware load-balancing tuning for the account pools — Claude (Anthropic) ( | `default_threshold_7d` | unset | Soft default for the shared weekly (`7d`) window | | `default_threshold_fable` | unset | Soft default for the fable-only weekly (`7d_oi`) window | | `burn_rate_avoidance` | `false` | Also avoid accounts projected to exhaust a window's soft threshold before that window resets | +| `sort_by_reset` | `false` | Rank available accounts by soonest quota reset (ascending; unknown resets sort last) instead of burn-rate headroom. Toggleable at runtime from the admin dashboard or `PATCH /admin/api/pool` without editing this file — see [Pausing a pool account](/guides/pool-account-controls/) | | `usage_refresh_seconds` | disabled (`0`/absent) | Poll interval, in seconds, for Claude `GET /api/oauth/usage`, Codex `GET /wham/usage`, and Antigravity `POST :retrieveUserQuotaSummary`; a positive value below 60 is clamped up to a 60-second floor | | `state_path` | unset | File the pool's per-account quota state is persisted to, so a restart warm-starts from the last observed utilization instead of an empty pool. Absent disables persistence (the default) | | `ramp_initial_concurrency` | disabled (`0`/absent) | Storm control: initial concurrent-admission allowance for an account identity that just started taking traffic. `0` or absent disables admission gating | diff --git a/site/src/content/docs/reference/endpoints.md b/site/src/content/docs/reference/endpoints.md index e2852ac42..083eac58a 100644 --- a/site/src/content/docs/reference/endpoints.md +++ b/site/src/content/docs/reference/endpoints.md @@ -30,7 +30,9 @@ description: The endpoints shunt serves as a Claude Code LLM gateway. | `GET` | `/admin/api/accounts/codex` | Codex account-store metadata: name, expiry, and ChatGPT account ID; never token material | | `GET` | `/admin/api/accounts/antigravity` | Antigravity account-store metadata: name, expiry, and email (when Google returned one); never token material | | `GET` | `/admin/api/observed` | Read-only local Claude Code, Codex, Gemini, Kimi, Grok, and Cursor identity plus provider-native usage; never returns token material or refreshes source credentials. The Claude and Codex rows also carry the account `uuid` (the Claude account uuid or the ChatGPT account id; null when it cannot be established) so the dashboard can tell an observation and a managed pool account holding the same subscription apart from two different accounts. Returns `{ "accounts": [] }` without reading any of those sources when [`[server.admin].hide_observed`](/reference/configuration/#serveradmin-optional) is true | -| `GET` | `/admin/api/pool` | Per-`claude_oauth`/`chatgpt_oauth`/`kimi_oauth`/`antigravity_oauth`-provider managed-pool state; account objects may include an optional `plan` string; a file-derived value can later be refined toward a more precise one via a profile lookup; Codex rows include reported 5h/7d usage (`7d_oi` has no Codex analog); each account also carries a boolean `needs_relogin`: the credential was terminally rejected (`invalid_grant`), carries no refresh token at all, or had a rotated pair lost before it could be stored, so no retry can revive it and only an operator re-login will. It is reported independently of the cooldown fields — a cooldown expires by itself, this does not — and both dashboard tables show it as **needs re-login** rather than the `cooling` a quota pause produces. Memory-only: a restart clears it, and the account's next terminal failure re-establishes it. It is reported even for an account no provider table has ever selected — alongside `has_state: false` — because the admin refresh probe records its verdict by store name. | +| `GET` | `/admin/api/pool` | Per-`claude_oauth`/`chatgpt_oauth`/`kimi_oauth`/`antigravity_oauth`-provider managed-pool state; account objects may include an optional `plan` string; a file-derived value can later be refined toward a more precise one via a profile lookup; Codex rows include reported 5h/7d usage (`7d_oi` has no Codex analog); each account also carries a boolean `needs_relogin`: the credential was terminally rejected (`invalid_grant`), carries no refresh token at all, or had a rotated pair lost before it could be stored, so no retry can revive it and only an operator re-login will. It is reported independently of the cooldown fields — a cooldown expires by itself, this does not — and both dashboard tables show it as **needs re-login** rather than the `cooling` a quota pause produces. Memory-only: a restart clears it, and the account's next terminal failure re-establishes it. It is reported even for an account no provider table has ever selected — alongside `has_state: false` — because the admin refresh probe records its verdict by store name. Each account also carries a boolean `paused`, a runtime-only exclusion an operator toggles without touching `shunt.toml` (see `PATCH .../accounts/{account_ref}` below); the response's top-level `sort_by_reset` reflects `[server.pool] sort_by_reset` or its runtime override (see the bare `PATCH` below), and is process-wide rather than per-provider. Each account also carries an opaque `account_ref`, derived from its provider-scoped pool identity and used by mutations so two distinct accounts may safely share the same display `name`. | +| `PATCH` | `/admin/api/pool/{provider}/accounts/{account_ref}` | Write-tier only. Body `{"paused": true\|false}` toggles the named account's runtime pause: it is excluded from selection exactly as `disabled` would, but memory-only (cleared on restart) and independent of `shunt.toml`. `404` for an unknown provider or account The `{account_ref}` comes from the matching account object in `GET /admin/api/pool`; display names are not used for mutation targeting. | +| `PATCH` | `/admin/api/pool` | Write-tier only. Body `{"sort_by_reset": true\|false}` toggles the reset-priority sort at runtime for the whole pool (`[server.pool]` is process-wide, not per-provider): available accounts then rank by soonest quota reset instead of burn-rate headroom. `{"sort_by_reset": null}` explicitly clears a previously set override, reverting to the config file's own value (an omitted field is a no-op, not a clear); the override is also memory-only, reverting on restart. Has no effect — and `GET /admin/api/pool` reports `sort_by_reset: false` regardless of the override — while `[server.pool]` itself is absent, since that legacy selection path never consults it | | `GET` | `/admin/api/status` | Observation-only view of [`[server.status]`](/reference/configuration/#serverstatus-optional) polling: each configured source's most recently observed Statuspage indicator, description, incidents, and observed timestamp. Empty `sources` when `[server.status]` is absent or unconfigured — never consulted by routing or failover | | `GET` | `/admin/api/routes` | The resolved routing table behind the admin credential, available to the dashboard — the same body `GET /routes` above returns, built from the same snapshot so the operator's view and the table a client resolves against cannot drift. It is registered separately rather than redirected so the admin namespace keeps its own authentication: the public `/routes` is unauthenticated on purpose, and pointing this one at it would tie the two together. Requiring the admin credential here means leaving `/routes` open does not widen what that credential gates | | `POST` | `/admin/api/accounts/claude` | Start Claude browser provisioning with `{name, mode}` where `mode` is `oauth` or `setup_token` (omitted defaults to `setup_token`); returns `{authorize_url}` | diff --git a/site/src/content/docs/zh-cn/guides/pool-account-controls.md b/site/src/content/docs/zh-cn/guides/pool-account-controls.md new file mode 100644 index 000000000..854f20b71 --- /dev/null +++ b/site/src/content/docs/zh-cn/guides/pool-account-controls.md @@ -0,0 +1,68 @@ +--- +title: 账户池控制 +description: 暂停单个账户池账户,并按配额最快重置的顺序对可用账户排序 —— 两者都可在管理仪表盘中于运行时完成,无需改配置或重启。 +--- + +在账户池选择([Anthropic 多账户](/zh-cn/guides/anthropic-multi-account/)、[Codex 多账户](/zh-cn/guides/codex-multi-account/))之上,有两个运行时控制:暂停单个账户,以及让可用账户按配额最快重置的顺序排序,而不是按燃烧速率余量排序。两者都通过管理仪表盘的 "Managed pool health" 表格操作,或直接调用管理 API。两者都只存在于内存中 —— 重启即清空,因此都不会改动 `shunt.toml`。 + +## 暂停一个账户 + +暂停会像配置侧的 `disabled = true` 一样把账户排除在选择之外,但不会编辑 `shunt.toml`,也不会把账户登出。凭证和配额历史都原样保留;被暂停的账户在恢复之前不会出现在候选列表中。 + +它与 `disabled` 不同: + +| | `disabled`(配置) | `paused`(运行时) | +| :-- | :-- | :-- | +| 设置方式 | `shunt.toml`,需要重载 | 管理仪表盘或 `PATCH /admin/api/pool/{provider}/accounts/{account_ref}` | +| 重启后是否保留 | 是 | 否 | +| 使用场景 | 从部署中永久移除一个账户 | 无需往返配置文件,临时把某个账户挪开一会儿 | + +在仪表盘中:打开 **Manage pool accounts → Managed pool health**,点击该账户行的 **Pause**。其状态会显示为 `paused`;点击 **Resume** 即可恢复。这两个按钮都需要 write 级别的管理会话。 + +直接调用 API(需要 write 级别凭证): + +`account_ref` 是 `GET /admin/api/pool` 每个 account 对象返回的不透明标识符。使用它而不是显示名称 `name`，可以分别控制同名但不同的账户。 + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool/anthropic/accounts/$ACCOUNT_REF" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"paused": true}' +``` + +设置 `"paused": false` 即可恢复。完整端点参考见 [`PATCH /admin/api/pool/{provider}/accounts/{account_ref}`](/zh-cn/reference/endpoints/)。 + +## 按最快重置排序 + +`[server.pool] sort_by_reset`(默认 `false`)会改变 *available* 层的排序方式:不再按预计的燃烧速率余量最大排序,而是按已知的最早配额重置时间升序排序(最先恢复的账户被最先尝试;没有重置信号的账户排在最后)。这样做的思路是先耗尽最快恢复的账户,把重置更晚的账户留作缓冲。 + +`[server.pool]` —— 以及这个设置 —— 是进程级的,不是按 provider 划分的,因此切换它会同时影响池中的所有 provider。 + +在 `shunt.toml` 中设置: + +```toml +[server.pool] +sort_by_reset = true +``` + +或者通过账户池表格上方的仪表盘复选框("Rank available accounts by soonest quota reset instead of burn-rate headroom")在运行时切换,或直接调用: + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"sort_by_reset": true}' +``` + +运行时切换会覆盖配置文件中的值,直到被清除或进程重启,此后又会恢复应用配置文件自身的值。要在不重启的情况下显式清除覆盖值,发送 `null`: + +```bash +curl -X PATCH "$SHUNT_URL/admin/api/pool" \ + -H "x-shunt-admin-token: $ADMIN_TOKEN" \ + -H "content-type: application/json" \ + -d '{"sort_by_reset": null}' +``` + +完全省略该字段不会有任何效果 —— 会保留当前的覆盖值(或其缺失状态)不变;只有显式的 `null` 才会清除它。 + +`GET /admin/api/pool` 会以顶层的 `sort_by_reset` 布尔值报告当前生效的值(覆盖值或配置值)。若 `[server.pool]` 本身不存在,这个设置完全不起作用 —— 它原本要改变的旧版选择路径根本不会运行 —— 因此在 `[server.pool]` 存在之前,运行时切换和 `GET /admin/api/pool` 报告的值都没有意义。 diff --git a/site/src/content/docs/zh-cn/reference/configuration.md b/site/src/content/docs/zh-cn/reference/configuration.md index ed562c688..4f6b049a1 100644 --- a/site/src/content/docs/zh-cn/reference/configuration.md +++ b/site/src/content/docs/zh-cn/reference/configuration.md @@ -232,6 +232,7 @@ headers = { "x-api-key" = "..." } | `default_threshold_7d` | 未设置 | 共享周(`7d`)窗口的软默认值 | | `default_threshold_fable` | 未设置 | 仅 fable 的周(`7d_oi`)窗口的软默认值 | | `burn_rate_avoidance` | `false` | 同时避开按预测会在窗口重置之前耗尽其软阈值的账户 | +| `sort_by_reset` | `false` | 按配额重置时间最早排序(升序;未知重置排最后)可用账户,而不是按燃烧速率余量排序。可在不编辑 `shunt.toml` 的情况下,通过管理仪表盘或 `PATCH /admin/api/pool` 在运行时切换 —— 参见[账户池控制](/zh-cn/guides/pool-account-controls/) | | `usage_refresh_seconds` | 禁用(`0`/未设置) | Claude `GET /api/oauth/usage`、Codex `GET /wham/usage` 和 Antigravity `POST :retrieveUserQuotaSummary` 的轮询间隔(秒);低于 60 的正值会向上取到 60 秒下限 | | `state_path` | 未设置 | 用于持久化池中按账户配额状态的文件;重启时从最后观测到的使用率热启动,而非从空池开始。未设置则禁用持久化(默认) | | `ramp_initial_concurrency` | 禁用(`0`/未设置) | 风暴控制:对刚开始承接流量的账户身份的初始并发准入额度。`0` 或未设置则禁用准入门控 | diff --git a/site/src/content/docs/zh-cn/reference/endpoints.md b/site/src/content/docs/zh-cn/reference/endpoints.md index 82b47cb69..abcebc971 100644 --- a/site/src/content/docs/zh-cn/reference/endpoints.md +++ b/site/src/content/docs/zh-cn/reference/endpoints.md @@ -28,7 +28,9 @@ description: shunt 作为 Claude Code LLM 网关所提供的端点。 | `GET` | `/admin/api/accounts/codex` | Codex 账户存储元数据:名称、过期时间和 ChatGPT 账户 ID;绝不返回 token 材料 | | `GET` | `/admin/api/accounts/antigravity` | Antigravity 账户存储元数据:名称、过期时间和邮箱(如果 Google 返回了);绝不返回 token 材料 | | `GET` | `/admin/api/observed` | 只读的本地 Claude Code、Codex、Gemini、Kimi、Grok 与 Cursor 身份及提供方原生用量;绝不返回 token 材料,也不刷新源凭据。当 [`[server.admin].hide_observed`](/zh-cn/reference/configuration/#serveradmin可选) 为 true 时,不读取任何这些来源,直接返回 `{ "accounts": [] }` | -| `GET` | `/admin/api/pool` | `claude_oauth` / `chatgpt_oauth` / `kimi_oauth` / `antigravity_oauth` provider 的池状态;每个 account 对象可能包含可选的 `plan` 字符串;文件中读取的值之后可能通过 profile 查询被修正为更精确的值;Codex 行包含已上报的 5h/7d 用量,`7d_oi` 没有对应的 Codex 字段;每个 account 还带有布尔字段 `needs_relogin`:凭据被终结性拒绝(`invalid_grant`)、根本不带刷新令牌,或轮换出的令牌对未能写入而丢失 —— 任何重试都无法恢复,只有运维人员重新登录才行。它与冷却字段**相互独立**上报 —— 冷却会自行到期,而该标记不会 —— 仪表盘的两个表格都会显示为 **needs re-login**,而不是配额暂停时的 `cooling`。仅存于内存:重启后清空,该账户的下一次终结性失败会重新置位。即使某个账户从未被任何 provider 表选中过,它也会被上报 —— 与 `has_state: false` 并列 —— 因为 admin 的 refresh 探测按存储名记录其判定。 | +| `GET` | `/admin/api/pool` | `claude_oauth` / `chatgpt_oauth` / `kimi_oauth` / `antigravity_oauth` provider 的池状态;每个 account 对象可能包含可选的 `plan` 字符串;文件中读取的值之后可能通过 profile 查询被修正为更精确的值;Codex 行包含已上报的 5h/7d 用量,`7d_oi` 没有对应的 Codex 字段;每个 account 还带有布尔字段 `needs_relogin`:凭据被终结性拒绝(`invalid_grant`)、根本不带刷新令牌,或轮换出的令牌对未能写入而丢失 —— 任何重试都无法恢复,只有运维人员重新登录才行。它与冷却字段**相互独立**上报 —— 冷却会自行到期,而该标记不会 —— 仪表盘的两个表格都会显示为 **needs re-login**,而不是配额暂停时的 `cooling`。仅存于内存:重启后清空,该账户的下一次终结性失败会重新置位。即使某个账户从未被任何 provider 表选中过,它也会被上报 —— 与 `has_state: false` 并列 —— 因为 admin 的 refresh 探测按存储名记录其判定。每个 account 还带有布尔字段 `paused`,这是运维人员无需改动 `shunt.toml` 即可切换的运行时专属排除标记(见下方 `PATCH .../accounts/{account_ref}`);响应顶层的 `sort_by_reset` 反映 `[server.pool] sort_by_reset` 或其运行时覆盖值(见下方无路径参数的 `PATCH`),它是进程级的,不是按 provider 划分的。 每个 account 还包含一个由 provider 范围内的池身份派生出的不透明 `account_ref`。变更操作使用该值而不是显示名称 `name`，因此同名的不同账户也能被安全地分别控制。 | +| `PATCH` | `/admin/api/pool/{provider}/accounts/{account_ref}` | 仅限 write 级别。请求体 `{"paused": true\|false}` 切换指定账户的运行时暂停状态:效果等同于 `disabled` 将其排除在选择之外,但只存在于内存中(重启即清空),与 `shunt.toml` 无关。provider 或 account 不存在时返回 `404` `{account_ref}` 来自 `GET /admin/api/pool` 中对应的 account 对象；显示名称不用于定位变更目标。 | +| `PATCH` | `/admin/api/pool` | 仅限 write 级别。请求体 `{"sort_by_reset": true\|false}` 在运行时为整个账户池切换重置优先排序(`[server.pool]` 是进程级的,不是按 provider 划分):可用账户随后按配额最快重置排序,而不是按燃烧速率余量排序。`{"sort_by_reset": null}` 会显式清除之前设置的覆盖值,恢复为配置文件自身的值(省略该字段不会清除,只是空操作);覆盖值只存在于内存中,重启后也会恢复。若 `[server.pool]` 本身不存在 —— 那条旧版选择路径完全不会读取这个值 —— 覆盖值不会产生任何效果,`GET /admin/api/pool` 也会报告 `sort_by_reset: false` | | `GET` | `/admin/api/routes` | 位于管理员凭据之后、可供仪表板使用的已解析路由表 —— 与上面 `GET /routes` 返回的响应体相同，且由同一快照构建，因此运维人员看到的表与客户端实际解析的表不会出现偏差。之所以单独注册而不是重定向，是为了让管理命名空间保有自己的认证：公开的 `/routes` 有意不做认证，若把此端点指向它，两者的认证就会被绑在一起。此端点要求管理员凭据，因此即使 `/routes` 保持开放，也不会扩大该凭据所保护的范围 | | `POST` | `/admin/api/accounts/claude` | 用 `{name, mode}` 开始 Claude 浏览器预配;`mode` 为 `oauth` 或 `setup_token`,省略时默认为 `setup_token`;返回 `{authorize_url}` | | `POST` | `/admin/api/accounts/claude/{name}/complete` | 用包含 `#` 的 `{code}` 完成 Claude 预配;存储账户并报告其是否生效 | diff --git a/site/src/lib/i18n.ts b/site/src/lib/i18n.ts index eaae32435..298fcc07b 100644 --- a/site/src/lib/i18n.ts +++ b/site/src/lib/i18n.ts @@ -76,6 +76,7 @@ export const NAVIGATION: NavigationEntry[] = [ { label: "Configuration", translations: { ko: "설정", ja: "設定", "zh-cn": "配置" }, slug: "guides/configuration" }, { label: "Anthropic Multi-Account", translations: { ko: "Anthropic 멀티 계정", ja: "Anthropic マルチアカウント", "zh-cn": "Anthropic 多账户" }, slug: "guides/anthropic-multi-account" }, { label: "Codex Multi-Account", translations: { ko: "Codex 멀티 계정", ja: "Codex マルチアカウント", "zh-cn": "Codex 多账户" }, slug: "guides/codex-multi-account" }, + { label: "Pool Account Controls", translations: { ko: "풀 계정 제어", ja: "プールアカウント制御", "zh-cn": "账户池控制" }, slug: "guides/pool-account-controls" }, { label: "Inbound Codex Endpoint", translations: { ko: "인바운드 Codex 엔드포인트", ja: "インバウンド Codex エンドポイント", "zh-cn": "入站 Codex 端点" }, slug: "guides/inbound-codex-endpoint" }, { label: "Admin & Remote Provisioning", translations: { ko: "관리자 & 원격 프로비저닝", ja: "管理とリモートプロビジョニング", "zh-cn": "管理与远程预配" }, slug: "guides/admin-remote-provisioning" }, { label: "Gateway Login", translations: { ko: "게이트웨이 로그인", ja: "ゲートウェイログイン", "zh-cn": "网关登录" }, slug: "guides/gateway-login" }, diff --git a/src/accounts.rs b/src/accounts.rs index 9b443da2b..a1fa0c489 100644 --- a/src/accounts.rs +++ b/src/accounts.rs @@ -405,6 +405,10 @@ struct AccountHealth { /// state_path` persists quota alone, so a restart clears this and the /// account's next terminal failure re-establishes it. needs_relogin: Option, + /// Provider-scoped operator pauses. Quota/cooldown state is shared by physical + /// account identity, but an operator pause belongs to the configured provider + /// lane that was paused. Memory-only and cleared on restart. + paused_providers: HashSet, /// Per-model cooldowns, keyed by the ASCII-lowercased upstream model: the /// account is healthy, but the upstream refused this one model for it /// (see [`is_codex_model_unsupported`]). Selection folds the entry for @@ -428,7 +432,7 @@ pub struct AccountSnapshot { /// Whether the pool has recorded at least one upstream response for this /// account. When `false`, the quota/cooldown fields are all absent. pub has_state: bool, - /// Derived: not disabled, not cooling down, and not near quota. + /// Derived: not disabled or paused for this provider, not cooling down, and not near quota. pub available: bool, pub near_quota: bool, /// Seconds until the account-wide cooldown expires, when active. @@ -439,6 +443,10 @@ pub struct AccountSnapshot { pub priority: u32, /// Configured exclusion from pool selection. pub disabled: bool, + /// Whether the operator has manually paused this account for the provider + /// whose snapshot is being read. Runtime-only: survives config reloads but + /// not process restarts. + pub paused: bool, /// Burn-rate headroom in seconds across the governing quota windows, when /// `[server.pool]` is configured and the projection is finite: positive /// means the account survives to its tightest reset at the current pace. @@ -469,16 +477,17 @@ impl AccountSnapshot { /// records a terminal verdict by store name in the pool's side table, and /// such an account has no health entry to carry it (see /// [`AccountPool::store_relogin`]). - fn unseen(account: &AccountConfig, needs_relogin: bool) -> Self { + fn unseen(account: &AccountConfig, needs_relogin: bool, paused: bool) -> Self { Self { name: account.name.clone(), has_state: false, - available: !account.disabled, + available: !account.disabled && !paused, near_quota: false, cooldown_secs_remaining: None, cooldown_fable_secs_remaining: None, priority: account.priority, disabled: account.disabled, + paused, headroom_secs: None, utilization_5h: None, reset_5h: None, @@ -536,6 +545,11 @@ pub struct AccountPool { /// only pattern needed; `forget_identity` takes this one alone, after it has /// released `entries`, which is still consistent with that order. store_relogin: Mutex>, + /// Runtime override for `[server.pool] sort_by_reset`, set via + /// `PATCH /admin/api/pool`. `[server.pool]` is process-wide (there is one, + /// not one per provider), so this override is too. `None` defers to the + /// config file's own value; memory-only, cleared on restart. + sort_by_reset_override: Mutex>, } #[derive(Debug)] @@ -780,13 +794,14 @@ impl AccountPool { // adding or removing an alias cannot move an existing session. Disabled // aliases yield to an enabled representative; fully disabled identities // are then dropped from the rotation entirely. `collapse_representatives` - // and `rotation` need no lock, so both are computed before the entries - // lock below — the opportunistic re-probe candidate (Change B) is - // selected only from these final representatives. - let rotation = (0..distinct) - .map(|offset| ident_reps[(start_slot + offset) % distinct]) - .filter(|&index| !accounts[index].disabled) - .collect::>(); + // needs no lock, so it is computed before the entries lock below. + // `rotation` also needs each account's `paused` bit, which only the + // lock guards; it is built just inside the lock, right after the + // per-account loop that already computes `account_key` and fetches + // `health` for every account, so its pause bit falls out for free — + // a separate pre-pass locking the mutex once per account + // (`account_pool_mixed_cycles` regressed ~13% when this first shipped + // that way) is what this avoids. let now = Instant::now(); let unix_now = SystemTime::now() @@ -796,25 +811,37 @@ impl AccountPool { let is_fable = is_fable_model(model); let model_key = model.map(str::to_ascii_lowercase); let reprobe = allow_reprobe.then(|| reprobe_interval(pool)).flatten(); - let (snapshots, pending_reprobe, quota_expired) = { + let (snapshots, rotation, pending_reprobe, quota_expired) = { let mut entries = self.entries.lock().expect("account health lock poisoned"); let mut snapshots = Vec::with_capacity(accounts.len()); + let mut paused = vec![false; accounts.len()]; let mut quota_expired = false; - for account in accounts { + for (index, account) in accounts.iter().enumerate() { let health = entries.entry(account_key(&provider, account)).or_default(); health.enabled |= !account.disabled; + paused[index] = health.paused_providers.contains(&provider); quota_expired |= expire_stale_quota(&mut health.quota, unix_now); // Assessing under the lock is pure CPU work and avoids cloning // each account's QuotaState just to assess it after release. let assessment = assess_quota(&health.quota, account, is_fable, pool, unix_now); let weekly_reset = governing_weekly_reset(&health.quota, is_fable); + // Match quota assessment's request-aware weekly window so a + // stale Fable-only reset cannot reorder ordinary traffic. + let min_reset = [health.quota.reset_5h, weekly_reset] + .into_iter() + .flatten() + .min(); // Prune expired per-model refusals here as well as on insert: // the key can be client-supplied, so without a later refusal // an expired entry would otherwise outlive its cooldown. health.model_cooldowns.retain(|_, until| *until > now); let cooldown_until = governing_cooldown(health, is_fable, model_key.as_deref()); - snapshots.push((cooldown_until, assessment, weekly_reset)); + snapshots.push((cooldown_until, assessment, weekly_reset, min_reset)); } + let rotation = (0..distinct) + .map(|offset| ident_reps[(start_slot + offset) % distinct]) + .filter(|&index| !accounts[index].disabled && !paused[index]) + .collect::>(); // Opportunistic re-probe (Change B): among the final rotation // representatives, find the single stale near-quota ChatGPT-family // account and reserve it while still holding the entries lock. @@ -835,7 +862,7 @@ impl AccountPool { if account.store_family != Some(StoreFamily::Chatgpt) { continue; } - let (cooldown_until, ref assessment, _) = snapshots[index]; + let (cooldown_until, ref assessment, _, _) = snapshots[index]; if cooldown_until.is_some_and(|until| until > now) || !assessment.near { continue; } @@ -905,7 +932,7 @@ impl AccountPool { } }); - (snapshots, pending_reprobe, quota_expired) + (snapshots, rotation, pending_reprobe, quota_expired) }; if quota_expired { @@ -931,8 +958,10 @@ impl AccountPool { }; let sticky = ident_reps[start_slot]; - let (sticky_cooldown, ref sticky_quota, _) = snapshots[sticky]; - if !accounts[sticky].disabled + let (sticky_cooldown, ref sticky_quota, _, _) = snapshots[sticky]; + // `rotation` already excludes a disabled or paused sticky account, so + // membership in it covers both checks the fast path needs. + if rotation.contains(&sticky) && sticky_cooldown.is_none_or(|until| until <= now) && !sticky_quota.near { @@ -951,17 +980,32 @@ impl AccountPool { match pool { // Priority beats headroom; ties prefer the account projected to // keep the most margin before its tightest window resets. - Some(_) => available_under.sort_by(|&left, &right| { - accounts[left] - .priority - .cmp(&accounts[right].priority) - .then_with(|| { - snapshots[right] - .1 - .headroom - .total_cmp(&snapshots[left].1.headroom) - }) - }), + Some(pool_cfg) => { + let sort_by_reset = self.effective_sort_by_reset(Some(pool_cfg)); + available_under.sort_by(|&left, &right| { + accounts[left] + .priority + .cmp(&accounts[right].priority) + .then_with(|| { + if sort_by_reset { + // Soonest reset first; None (unknown) sorts last. + let l = snapshots[left].3; + let r = snapshots[right].3; + match (l, r) { + (Some(lt), Some(rt)) => lt.cmp(&rt), + (Some(_), None) => std::cmp::Ordering::Less, + (None, Some(_)) => std::cmp::Ordering::Greater, + (None, None) => std::cmp::Ordering::Equal, + } + } else { + snapshots[right] + .1 + .headroom + .total_cmp(&snapshots[left].1.headroom) + } + }) + }) + } // Legacy: `Option` orders `None` before `Some`, so accounts with // an unknown weekly reset sort first. None => available_under.sort_by(|&left, &right| { @@ -1526,6 +1570,54 @@ impl AccountPool { ); } + /// Pause or resume one provider lane for an account. The pause persists in + /// memory until toggled again or the process restarts, without affecting a + /// different provider that resolves to the same physical identity. Inserts + /// a default health entry if the account has not been selected yet. + pub fn set_paused(&self, provider: &str, account: &AccountConfig, paused: bool) { + let mut entries = self.entries.lock().expect("account health lock poisoned"); + let key = account_key(provider, account); + let health = entries.entry(key).or_default(); + if paused { + health.paused_providers.insert(provider.to_string()); + } else { + health.paused_providers.remove(provider); + } + } + + /// Whether an account is currently paused. + pub fn is_paused(&self, provider: &str, account: &AccountConfig) -> bool { + let entries = self.entries.lock().expect("account health lock poisoned"); + entries + .get(&account_key(provider, account)) + .is_some_and(|h| h.paused_providers.contains(provider)) + } + + /// Set (or clear, with `None`) the runtime override of `sort_by_reset`. + /// `[server.pool]` is process-wide, so this override applies to every + /// provider using the pool tier. + pub fn set_sort_by_reset_override(&self, value: Option) { + *self + .sort_by_reset_override + .lock() + .expect("account health lock poisoned") = value; + } + + /// Effective `sort_by_reset`: the runtime override when set, else the + /// config file's own `[server.pool] sort_by_reset`. Always `false` when + /// `[server.pool]` is absent, whatever the override says — select_order_inner's + /// legacy (no-pool) branch never consults this, so reporting the override's + /// value there would claim an effect selection does not actually have. + pub fn effective_sort_by_reset(&self, pool: Option<&PoolConfig>) -> bool { + let Some(pool) = pool else { + return false; + }; + self.sort_by_reset_override + .lock() + .expect("account health lock poisoned") + .unwrap_or(pool.sort_by_reset) + } + /// Set or clear the needs-re-login mark on every pool entry backed by one /// store account, whatever provider table it is reachable through. Used by /// the admin re-login and refresh-probe paths, which know an account by its @@ -2054,8 +2146,14 @@ impl AccountPool { // with no entry at all, the side table's verdict. `has_state` // stays `false`, which is still true and which both dashboard // tables already read *after* `needs_relogin`, so the row - // renders "needs re-login" rather than "unseen". - return AccountSnapshot::unseen(account, store_condemned); + // renders "needs re-login" rather than "unseen". `paused` is + // read from any entry that exists regardless of `observed`, so + // an operator's pause shows up immediately even before the + // account is ever selected. + let paused = entries + .get(&key) + .is_some_and(|health| health.paused_providers.contains(provider)); + return AccountSnapshot::unseen(account, store_condemned, paused); }; quota_expired |= expire_stale_quota(&mut health.quota, unix_now); let quota = assess_quota(&health.quota, account, is_fable, pool, unix_now); @@ -2072,12 +2170,16 @@ impl AccountPool { AccountSnapshot { name: account.name.clone(), has_state: true, - available: !account.disabled && !cooling && !quota.near, + available: !account.disabled + && !health.paused_providers.contains(provider) + && !cooling + && !quota.near, near_quota: quota.near, cooldown_secs_remaining, cooldown_fable_secs_remaining, priority: account.priority, disabled: account.disabled, + paused: health.paused_providers.contains(provider), headroom_secs: (pool.is_some() && quota.headroom.is_finite()) .then_some(quota.headroom as i64), utilization_5h: health.quota.utilization_5h, @@ -2569,6 +2671,29 @@ pub(crate) fn account_key(upstream: &str, account: &AccountConfig) -> AccountKey } } +/// Opaque provider-scoped reference for one resolved pool identity. +/// +/// The admin API returns this value and accepts it back on mutations. It is +/// deliberately distinct from the display name because different credentials +/// may legitimately share one name. +pub(crate) fn account_ref(upstream: &str, account: &AccountConfig) -> String { + let key = account_key(upstream, account); + let encoded = serde_json::to_vec(&key).expect("account key is serializable"); + let mut hasher = Sha256::new(); + hasher.update(upstream.as_bytes()); + hasher.update([0]); + hasher.update(encoded); + let digest = hasher.finalize(); + const HEX: &[u8; 16] = b"0123456789abcdef"; + let mut value = String::with_capacity(5 + digest.len() * 2); + value.push_str("acct_"); + for byte in digest { + value.push(HEX[(byte >> 4) as usize] as char); + value.push(HEX[(byte & 0x0f) as usize] as char); + } + value +} + /// Collapse accounts sharing a stable upstream identity ([`account_identity`]) /// down to one representative per identity, keeping the enabled (or, among /// equally-disabled duplicates, the lowest-priority) account as the @@ -9951,4 +10076,420 @@ mod tests { ); assert!(reservation.is_some()); } + #[test] + fn paused_account_is_excluded_from_select_order() { + let pool = AccountPool::new(); + let accounts = vec![account("a"), account("b")]; + pool.set_paused("anthropic", &accounts[0], true); + for _ in 0..10 { + let order = pool.select_order("anthropic", &accounts, None, None, None); + assert!( + !order.contains(&0), + "paused account must not appear in selection" + ); + assert_eq!(order, vec![1]); + } + } + + #[test] + fn paused_account_shows_in_snapshot_as_paused_and_unavailable() { + // A pause issued before the account was ever selected must still be + // visible on the dashboard, not just enforced in `select_order`. + let pool = AccountPool::new(); + let accounts = vec![account("a")]; + pool.set_paused("anthropic", &accounts[0], true); + let snaps = pool.snapshot("anthropic", &accounts, None, None); + assert!(snaps[0].paused); + assert!(!snaps[0].available, "a paused account is not available"); + } + + #[test] + fn unpause_restores_selection() { + let pool = AccountPool::new(); + let accounts = vec![account("a")]; + pool.set_paused("anthropic", &accounts[0], true); + assert!(pool.is_paused("anthropic", &accounts[0])); + assert!(pool + .select_order("anthropic", &accounts, None, None, None) + .is_empty()); + + pool.set_paused("anthropic", &accounts[0], false); + assert!(!pool.is_paused("anthropic", &accounts[0])); + assert_eq!( + pool.select_order("anthropic", &accounts, None, None, None), + vec![0] + ); + } + + #[test] + fn paused_sticky_still_ranks_the_remaining_accounts() { + // A session-sticky account that is otherwise healthy (no cooldown, not + // near quota) used to take the fast path regardless of `paused`: the + // paused account itself never appeared (it is filtered out of + // `rotation` earlier), but the *other* accounts came back in raw + // rotation order instead of properly ranked by headroom. + let pool = AccountPool::new(); + let accounts = vec![account("a"), account("b"), account("c"), account("d")]; + let cfg = PoolConfig::default(); + let session = "paused-sticky-ranks"; + let rotation = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + let sticky = rotation[0]; + let others: Vec = rotation[1..].to_vec(); + let now = unix_now(); + // Same pattern as `all_near_accounts_fall_back_to_headroom_order`: + // equal utilization, decreasing reset distance across `others` in + // rotation order, so the headroom order is exactly reversed. + for (offset, &index) in others.iter().enumerate() { + let reset_in = [16_200u64, 9_000, 3_600][offset]; + pool.note_quota( + "anthropic", + &accounts[index], + "a_headers(&[ + ( + "anthropic-ratelimit-unified-5h-utilization", + "0.3".to_string(), + ), + ( + "anthropic-ratelimit-unified-5h-reset", + (now + reset_in).to_string(), + ), + ]), + ); + } + + // Sanity: while the sticky account is healthy and un-paused, it still + // takes the fast path, so `others` stay in raw rotation order. + let baseline = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + assert_eq!( + baseline, rotation, + "a healthy sticky account takes the fast path" + ); + + pool.set_paused("anthropic", &accounts[sticky], true); + let order = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + assert!(!order.contains(&sticky), "the paused account never appears"); + let expected: Vec = others.iter().rev().copied().collect(); + assert_eq!( + order, expected, + "pausing the sticky account must not skip ranking the remaining ones by headroom" + ); + } + + #[test] + fn effective_sort_by_reset_is_false_without_pool_config_even_if_overridden() { + // `[server.pool]` absent means the legacy branch of `select_order_inner` + // runs, which never consults `sort_by_reset` at all. Reporting the + // runtime override as active in that state (e.g. on `GET + // /admin/api/pool`) would claim an effect selection does not have. + let pool = AccountPool::new(); + pool.set_sort_by_reset_override(Some(true)); + assert!(!pool.effective_sort_by_reset(None)); + assert!(pool.effective_sort_by_reset(Some(&PoolConfig { + sort_by_reset: true, + ..Default::default() + }))); + } + + #[test] + fn sort_by_reset_ranks_soonest_reset_first() { + // Mirrors `available_accounts_order_by_burn_rate_headroom`, but with + // `sort_by_reset` on: ordering follows the nearest reset instead of + // burn-rate headroom, even though headroom here would rank the + // opposite way (the soonest-reset account also has the least margin). + let pool = AccountPool::new(); + let accounts = vec![account("a"), account("b"), account("c")]; + let cfg = PoolConfig { + sort_by_reset: true, + ..Default::default() + }; + let session = "sort-by-reset"; + let now = unix_now(); + let rotation = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + let sticky = rotation[0]; + // Push the sticky account near quota so the available_under sort runs. + pool.note_quota( + "anthropic", + &accounts[sticky], + "a_headers(&[( + "anthropic-ratelimit-unified-5h-utilization", + "0.99".to_string(), + )]), + ); + let others: Vec = (0..accounts.len()).filter(|&i| i != sticky).collect(); + let (soonest, latest) = (others[0], others[1]); + pool.note_quota( + "anthropic", + &accounts[soonest], + "a_headers(&[ + ( + "anthropic-ratelimit-unified-5h-utilization", + "0.3".to_string(), + ), + ( + "anthropic-ratelimit-unified-5h-reset", + (now + 3_600).to_string(), + ), + ]), + ); + pool.note_quota( + "anthropic", + &accounts[latest], + "a_headers(&[ + ( + "anthropic-ratelimit-unified-5h-utilization", + "0.3".to_string(), + ), + ( + "anthropic-ratelimit-unified-5h-reset", + (now + 16_200).to_string(), + ), + ]), + ); + let order = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + assert_eq!(order[0], soonest, "soonest-resetting account sorts first"); + assert_eq!(order[1], latest, "later-resetting account sorts after"); + assert_eq!(order.last(), Some(&sticky), "near sticky sorts last"); + } + + #[test] + fn sort_by_reset_ignores_a_stale_fable_reset_on_a_non_fable_request() { + // An account that previously served Fable traffic can carry a stale, + // near `reset_7d_oi` alongside an unrelated (and later) shared weekly + // `reset_7d`. For an ordinary (non-Fable) request, `min_reset` must + // track the same governing window `assess_quota` does — shared + // weekly, not the Fable-only one — or the leftover Fable reset wins + // the sort on a window this request does not consume from. + let pool = AccountPool::new(); + let accounts = vec![account("a"), account("b"), account("c")]; + let cfg = PoolConfig { + sort_by_reset: true, + ..Default::default() + }; + let session = "sort-by-reset-fable-leak"; + let now = unix_now(); + let rotation = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + let sticky = rotation[0]; + pool.note_quota( + "anthropic", + &accounts[sticky], + "a_headers(&[( + "anthropic-ratelimit-unified-5h-utilization", + "0.99".to_string(), + )]), + ); + let others: Vec = (0..accounts.len()).filter(|&i| i != sticky).collect(); + let (nearer_weekly, leaky_fable) = (others[0], others[1]); + pool.note_quota( + "anthropic", + &accounts[nearer_weekly], + "a_headers(&[ + ( + "anthropic-ratelimit-unified-7d-utilization", + "0.3".to_string(), + ), + ( + "anthropic-ratelimit-unified-7d-reset", + (now + 3_600).to_string(), + ), + ]), + ); + // A far shared-weekly reset, but a stale, very-soon Fable-only reset + // left over from earlier Fable traffic on this account. + pool.note_quota( + "anthropic", + &accounts[leaky_fable], + "a_headers(&[ + ( + "anthropic-ratelimit-unified-7d-utilization", + "0.3".to_string(), + ), + ( + "anthropic-ratelimit-unified-7d-reset", + (now + 16_200).to_string(), + ), + ( + "anthropic-ratelimit-unified-7d_oi-reset", + (now + 100).to_string(), + ), + ]), + ); + // `None` model: not Fable, so only the shared `7d` reset governs. + let order = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + assert_eq!( + order[0], nearer_weekly, + "the account with the nearer shared-weekly reset sorts first, \ + regardless of the other account's stale, irrelevant Fable reset" + ); + assert_eq!(order[1], leaky_fable); + assert_eq!(order.last(), Some(&sticky)); + } + + #[test] + fn sort_by_reset_pushes_unknown_reset_last() { + let pool = AccountPool::new(); + let accounts = vec![account("a"), account("b"), account("c")]; + let cfg = PoolConfig { + sort_by_reset: true, + ..Default::default() + }; + let session = "sort-by-reset-unknown"; + let now = unix_now(); + let rotation = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + let sticky = rotation[0]; + pool.note_quota( + "anthropic", + &accounts[sticky], + "a_headers(&[( + "anthropic-ratelimit-unified-5h-utilization", + "0.99".to_string(), + )]), + ); + let others: Vec = (0..accounts.len()).filter(|&i| i != sticky).collect(); + let (known, unknown) = (others[0], others[1]); + // `known` gets a reset timestamp; `unknown` stays under quota with no + // reset signal at all. + pool.note_quota( + "anthropic", + &accounts[known], + "a_headers(&[ + ( + "anthropic-ratelimit-unified-5h-utilization", + "0.3".to_string(), + ), + ( + "anthropic-ratelimit-unified-5h-reset", + (now + 3_600).to_string(), + ), + ]), + ); + pool.note_quota( + "anthropic", + &accounts[unknown], + "a_headers(&[( + "anthropic-ratelimit-unified-5h-utilization", + "0.3".to_string(), + )]), + ); + let order = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + assert_eq!( + order[0], known, + "the account with a known reset sorts first" + ); + assert_eq!( + order[1], unknown, + "the account with no reset signal sorts after a known one" + ); + assert_eq!(order.last(), Some(&sticky), "near sticky sorts last"); + } + + #[test] + fn sort_by_reset_runtime_override_wins_over_config_and_is_reversible() { + let pool = AccountPool::new(); + let accounts = vec![account("a"), account("b"), account("c")]; + let cfg = PoolConfig::default(); // sort_by_reset: false in the file. + let session = "sort-by-reset-override"; + let now = unix_now(); + let rotation = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + let sticky = rotation[0]; + pool.note_quota( + "anthropic", + &accounts[sticky], + "a_headers(&[( + "anthropic-ratelimit-unified-5h-utilization", + "0.99".to_string(), + )]), + ); + let others: Vec = (0..accounts.len()).filter(|&i| i != sticky).collect(); + let (soonest, latest) = (others[0], others[1]); + // Utilizations are deliberately unequal, and chosen so headroom order + // disagrees with reset order: `soonest`'s near reset and high + // utilization make its projected margin negative, while `latest`'s far + // reset and low utilization give it a large positive margin. A + // same-utilization fixture (as in the sibling reset-order tests) would + // have the two orders agree here and prove nothing about the override. + pool.note_quota( + "anthropic", + &accounts[soonest], + "a_headers(&[ + ( + "anthropic-ratelimit-unified-5h-utilization", + "0.9".to_string(), + ), + ( + "anthropic-ratelimit-unified-5h-reset", + (now + 3_600).to_string(), + ), + ]), + ); + pool.note_quota( + "anthropic", + &accounts[latest], + "a_headers(&[ + ( + "anthropic-ratelimit-unified-5h-utilization", + "0.05".to_string(), + ), + ( + "anthropic-ratelimit-unified-5h-reset", + (now + 16_200).to_string(), + ), + ]), + ); + + // Config says headroom order; that is what an un-overridden pool uses. + assert!(!pool.effective_sort_by_reset(Some(&cfg))); + let order = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + assert_eq!( + order[0], latest, + "headroom order: larger margin sorts first" + ); + + // The runtime override flips it without touching the config file. + pool.set_sort_by_reset_override(Some(true)); + assert!(pool.effective_sort_by_reset(Some(&cfg))); + let order = pool.select_order("anthropic", &accounts, Some(session), None, Some(&cfg)); + assert_eq!(order[0], soonest, "override: soonest reset sorts first"); + + // Clearing the override reverts to the config's own value. + pool.set_sort_by_reset_override(None); + assert!(!pool.effective_sort_by_reset(Some(&cfg))); + } + + #[test] + fn pause_is_scoped_to_the_provider_for_a_shared_identity() { + let pool = AccountPool::new(); + let mut shared = account("shared"); + shared.uuid = Some("same-physical-account".to_string()); + let accounts = vec![shared]; + + pool.set_paused("anthropic-primary", &accounts[0], true); + + assert!(pool + .select_order("anthropic-primary", &accounts, None, None, None) + .is_empty()); + assert_eq!( + pool.select_order("anthropic-backup", &accounts, None, None, None), + vec![0], + "pausing one provider lane must not pause another lane using the same identity" + ); + assert!(pool.snapshot("anthropic-primary", &accounts, None, None)[0].paused); + assert!(!pool.snapshot("anthropic-backup", &accounts, None, None)[0].paused); + } + + #[test] + fn account_ref_disambiguates_same_name_and_provider() { + let mut left = account("same"); + left.uuid = Some("left-id".to_string()); + let mut right = account("same"); + right.uuid = Some("right-id".to_string()); + + assert_ne!( + account_ref("anthropic", &left), + account_ref("anthropic", &right) + ); + assert_ne!( + account_ref("anthropic", &left), + account_ref("anthropic-alt", &left) + ); + } } diff --git a/src/admin/mod.rs b/src/admin/mod.rs index 567d8facf..0d4aedfae 100644 --- a/src/admin/mod.rs +++ b/src/admin/mod.rs @@ -39,7 +39,7 @@ use axum::{ extract::{rejection::JsonRejection, Path, State}, http::{header, HeaderMap, HeaderName, StatusCode}, response::{IntoResponse, Response}, - routing::{delete, get, post}, + routing::{delete, get, patch, post}, Form, Json, Router, }; use serde::{Deserialize, Serialize}; @@ -264,7 +264,11 @@ pub fn admin_router() -> Router { .route("/admin/api/session", get(session_bootstrap)) .route("/admin/api/accounts", get(list_accounts)) .route("/admin/api/observed", get(observed_accounts)) - .route("/admin/api/pool", get(pool)) + .route("/admin/api/pool", get(pool).patch(patch_pool_settings)) + .route( + "/admin/api/pool/{provider}/accounts/{account_ref}", + patch(patch_pool_account), + ) .route("/admin/api/status", get(status)) .route("/admin/api/routes", get(routes)) .route("/admin/api/accounts/claude", post(add_account)) @@ -1247,10 +1251,17 @@ async fn pool(State(state): State, headers: HeaderMap) -> Response { let accounts: Vec = snapshots .into_iter() .zip(plans) - .map(|(snapshot, plan)| { + .zip(resolved.iter()) + .map(|((snapshot, plan), account)| { let mut value = serde_json::to_value(&snapshot).unwrap_or(Value::Null); - if let (Value::Object(map), Some(plan)) = (&mut value, plan) { - map.insert("plan".to_string(), Value::String(plan)); + if let Value::Object(map) = &mut value { + map.insert( + "account_ref".to_string(), + Value::String(crate::accounts::account_ref(name, account)), + ); + if let Some(plan) = plan { + map.insert("plan".to_string(), Value::String(plan)); + } } value }) @@ -1264,7 +1275,171 @@ async fn pool(State(state): State, headers: HeaderMap) -> Response { // named "claude". providers.push(json!({ "provider": name, "auth": provider.auth, "accounts": accounts })); } - json_secure(json!({ "providers": providers })) + json_secure(json!({ + "providers": providers, + // `[server.pool]` is process-wide (one, not one per provider), so this + // reflects a runtime `PATCH /admin/api/pool` override when set, else + // the config file's own value. + "sort_by_reset": state + .accounts + .effective_sort_by_reset(state.config.server.pool.as_ref()), + })) +} + +#[derive(serde::Deserialize)] +struct PatchPoolSettingsBody { + /// Double-`Option` so the field's three JSON shapes stay distinguishable: + /// omitted (`None`, outer) is a no-op, `null` (`Some(None)`) clears the + /// runtime override back to config-following, and `true`/`false` + /// (`Some(Some(bool))`) sets it. Collapsing `null` and omitted into one + /// `None` (a plain `Option`, as this used to be) would leave no way + /// to clear an override once set — `serde` gives both the same value by + /// default, so this needs the explicit `deserialize_some` shim below. + #[serde(default, deserialize_with = "deserialize_some")] + sort_by_reset: Option>, +} + +/// Maps a present field (of any value, including `null`) to `Some`, so a +/// `#[serde(default)]` outer `Option` can distinguish "field omitted" from +/// "field present". Standard workaround for serde's lack of a built-in +/// double-`Option` — see . +fn deserialize_some<'de, D, T>(deserializer: D) -> Result, D::Error> +where + T: serde::Deserialize<'de>, + D: serde::Deserializer<'de>, +{ + T::deserialize(deserializer).map(Some) +} + +/// `PATCH /admin/api/pool` — toggle the process-wide reset-priority sort at +/// runtime, without editing `shunt.toml`. Mirrors account pause: memory-only, +/// cleared on restart, and only takes effect where `[server.pool]` is +/// configured (see `AccountPool::effective_sort_by_reset`). `{"sort_by_reset": +/// null}` clears a previously set override back to following the config file; +/// omitting the field entirely leaves the current override untouched. +async fn patch_pool_settings( + State(state): State, + headers: HeaderMap, + Json(body): Json, +) -> Response { + let state = state.refreshed(); + let Some(authok) = authenticate(&state, &headers) else { + return unauthorized(); + }; + if let Some(response) = require_write(&authok) { + return response; + } + if let Some(response) = check_csrf(&authok.kind, &headers) { + return response; + } + if let Some(sort_by_reset) = body.sort_by_reset { + state.accounts.set_sort_by_reset_override(sort_by_reset); + tracing::info!( + sort_by_reset = ?sort_by_reset, + "admin: pool sort_by_reset override updated" + ); + } + json_secure(json!({"ok": true})) +} + +#[derive(serde::Deserialize)] +struct PatchPoolAccountBody { + #[serde(default)] + paused: Option, +} + +async fn patch_pool_account( + State(state): State, + headers: HeaderMap, + Path((provider, account_ref)): Path<(String, String)>, + Json(body): Json, +) -> Response { + let state = state.refreshed(); + let Some(authok) = authenticate(&state, &headers) else { + return unauthorized(); + }; + if let Some(response) = require_write(&authok) { + return response; + } + if let Some(response) = check_csrf(&authok.kind, &headers) { + return response; + } + // Look up the provider. + let Some(provider_cfg) = state.config.providers.get(&provider) else { + return not_found(); + }; + // Resolve pool accounts for this provider. + let resolved = match provider_cfg.auth { + AuthMode::ClaudeOauth => { + crate::auth::shared::resolve_pool_accounts( + "Claude", + &provider_cfg.accounts, + &provider_cfg.account_scope, + crate::accounts::StoreFamily::Claude, + crate::auth::claude::store::default_accounts_dir(), + crate::auth::claude::store::scan_accounts, + ) + .await + } + AuthMode::ChatgptOauth => { + crate::auth::shared::resolve_pool_accounts( + "codex", + &provider_cfg.accounts, + &provider_cfg.account_scope, + crate::accounts::StoreFamily::Chatgpt, + crate::auth::codex::store::default_accounts_dir(), + crate::auth::codex::store::scan_accounts, + ) + .await + } + AuthMode::KimiOauth => { + crate::auth::shared::resolve_pool_accounts( + "Kimi", + &provider_cfg.accounts, + &provider_cfg.account_scope, + crate::accounts::StoreFamily::Kimi, + crate::auth::kimi::store::default_accounts_dir(), + crate::auth::kimi::store::scan_accounts, + ) + .await + } + AuthMode::AntigravityOauth => { + crate::auth::shared::resolve_pool_accounts( + "Antigravity", + &provider_cfg.accounts, + &provider_cfg.account_scope, + crate::accounts::StoreFamily::Antigravity, + crate::auth::antigravity::store::default_accounts_dir(), + crate::auth::antigravity::store::scan_accounts, + ) + .await + } + _ => return bad_request("provider does not use a managed pool"), + }; + let accounts = match resolved { + Ok(accounts) => accounts, + Err(error) => { + tracing::error!(provider = %provider, %error, "admin: failed to resolve pool accounts for patch"); + return internal("failed to read pool state"); + } + }; + let Some(account) = accounts + .iter() + .find(|account| crate::accounts::account_ref(&provider, account) == account_ref) + else { + return not_found(); + }; + if let Some(paused) = body.paused { + state.accounts.set_paused(&provider, account, paused); + tracing::info!( + provider = %provider, + account = %account.name, + account_ref = %account_ref, + paused, + "admin: pool account pause flag updated", + ); + } + json_secure(json!({"ok": true})) } /// `GET /admin/api/routes` — the resolved routing table, for the dashboard. diff --git a/src/auth/claude/usage.rs b/src/auth/claude/usage.rs index df7be27db..541215164 100644 --- a/src/auth/claude/usage.rs +++ b/src/auth/claude/usage.rs @@ -364,6 +364,7 @@ mod tests { cooldown_fable_secs_remaining: None, priority: 100, disabled: false, + paused: false, headroom_secs: None, utilization_5h: Some(0.4237), reset_5h: Some(reset_5h), diff --git a/src/config.rs b/src/config.rs index 0efbed46f..50648c694 100644 --- a/src/config.rs +++ b/src/config.rs @@ -233,6 +233,12 @@ pub struct PoolConfig { /// Avoid an account projected to exhaust a soft threshold before reset. #[serde(default)] pub burn_rate_avoidance: bool, + /// When true, available accounts in the pool tier sort by their earliest + /// known rate-limit reset timestamp (ascending — soonest-reset first) rather + /// than by burn-rate headroom. Accounts with no reset signal sort last + /// within their priority tier. Off by default. + #[serde(default)] + pub sort_by_reset: bool, /// Poll Claude's `/api/oauth/usage`, Codex's `/wham/usage`, and /// Antigravity's `retrieveUserQuotaSummary` every N seconds for refreshable /// accounts. Unset or `0` disables polling; positive values below 60 are @@ -286,6 +292,7 @@ impl Default for PoolConfig { default_threshold_7d: None, default_threshold_fable: None, burn_rate_avoidance: false, + sort_by_reset: false, usage_refresh_seconds: None, state_path: None, ramp_initial_concurrency: None, diff --git a/src/oauth_usage/tests.rs b/src/oauth_usage/tests.rs index 43b829a17..9dbb65dff 100644 --- a/src/oauth_usage/tests.rs +++ b/src/oauth_usage/tests.rs @@ -27,6 +27,7 @@ fn snapshot( cooldown_fable_secs_remaining: None, priority, disabled: false, + paused: false, headroom_secs: None, utilization_5h: util_5h, reset_5h, diff --git a/src/usage/tests.rs b/src/usage/tests.rs index 0ed532fc2..52e73c5e5 100644 --- a/src/usage/tests.rs +++ b/src/usage/tests.rs @@ -26,6 +26,7 @@ fn snapshot( cooldown_fable_secs_remaining: None, priority: 100, disabled: false, + paused: false, headroom_secs: None, utilization_5h: util_5h, reset_5h, diff --git a/tests/pool_pause_api.rs b/tests/pool_pause_api.rs new file mode 100644 index 000000000..e16ad98fe --- /dev/null +++ b/tests/pool_pause_api.rs @@ -0,0 +1,467 @@ +//! `PATCH /admin/api/pool/{provider}/accounts/{account_ref}` — runtime account pause. +//! +//! Covers the admin-surface half of pool pause: a write-tier credential can +//! toggle `paused` and see it reflected on `GET /admin/api/pool` immediately +//! (even before the account was ever selected), a read-tier credential is +//! refused, and the paused account drops out of `select_order`. + +use std::net::SocketAddr; + +use reqwest::StatusCode; +use shunt::{ + config::{AccountConfig, AdminConfig, AdminKey, AuthMode, Config, PoolConfig}, + server, +}; +use tokio::task::JoinHandle; + +mod common; + +struct Gateway { + base_url: String, + task: JoinHandle<()>, +} + +impl Drop for Gateway { + fn drop(&mut self) { + self.task.abort(); + } +} + +fn can_bind_loopback() -> bool { + match std::net::TcpListener::bind("127.0.0.1:0") { + Ok(listener) => { + drop(listener); + true + } + Err(error) if error.kind() == std::io::ErrorKind::PermissionDenied => { + eprintln!("skipping network integration test: loopback bind is not permitted"); + false + } + Err(error) => panic!("unexpected loopback bind failure: {error}"), + } +} + +/// A path that is guaranteed not to exist, so an explicit account entry never +/// falls back to scanning the real on-disk credential store. +fn nonexistent_credentials_path() -> String { + std::env::temp_dir() + .join(format!( + "shunt-pool-pause-test-{}-missing.json", + std::process::id() + )) + .to_string_lossy() + .into_owned() +} + +const ADMIN_WRITE_KEY: &str = "admin-write-0123456789abcdef012345"; +const ADMIN_READ_KEY: &str = "admin-read-0123456789abcdef0123456"; + +fn admin_config() -> Config { + let mut config = Config::default(); + let anthropic = config.providers.get_mut("anthropic").unwrap(); + anthropic.auth = AuthMode::ClaudeOauth; + anthropic.accounts = vec![AccountConfig { + name: "pause-me".to_string(), + credentials: Some(nonexistent_credentials_path()), + uuid: Some("pause-me-uuid".to_string()), + ..Default::default() + }]; + config.server.admin = Some(AdminConfig { + header: "x-shunt-admin-token".to_string(), + tokens_env: "SHUNT_TEST_ADMIN_TOKENS_POOL_PAUSE".to_string(), + tokens_file: None, + write_keys: vec![AdminKey { + id: "terraform".to_string(), + key: ADMIN_WRITE_KEY.into(), + }], + read_keys: vec![AdminKey { + id: "reporting".to_string(), + key: ADMIN_READ_KEY.into(), + }], + session_ttl_secs: 3600, + pending_ttl_secs: 600, + hide_observed: false, + oidc: None, + }); + config +} + +/// `admin_config` plus `[server.pool]`, so `sort_by_reset` (config value or +/// runtime override) actually governs selection — required to observe the +/// override end to end, since `effective_sort_by_reset` is always `false` +/// without a pool table (see `AccountPool::effective_sort_by_reset`). +fn admin_config_with_pool() -> Config { + let mut config = admin_config(); + config.server.pool = Some(PoolConfig::default()); + config +} + +async fn start(mut config: Config) -> (Gateway, shunt::server::AppState) { + config.server.bind = "127.0.0.1:0".to_string(); + let listener = tokio::net::TcpListener::bind(config.server.bind_addr().unwrap()) + .await + .unwrap(); + let addr: SocketAddr = listener.local_addr().unwrap(); + let (app, _shared, state) = server::build_router(config).unwrap(); + let task = tokio::spawn(async move { + axum::serve(listener, app).await.unwrap(); + }); + ( + Gateway { + base_url: format!("http://{addr}"), + task, + }, + state, + ) +} + +async fn pool_accounts( + client: &reqwest::Client, + gateway: &Gateway, + provider: &str, +) -> Vec { + let response = client + .get(format!("{}/admin/api/pool", gateway.base_url)) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + let body: serde_json::Value = response.json().await.unwrap(); + body["providers"] + .as_array() + .unwrap() + .iter() + .find(|table| table["provider"] == provider) + .and_then(|table| table["accounts"].as_array()) + .cloned() + .expect("provider accounts present") +} + +async fn pool_account_ref( + client: &reqwest::Client, + gateway: &Gateway, + provider: &str, + name: &str, +) -> String { + pool_accounts(client, gateway, provider) + .await + .into_iter() + .find(|account| account["name"].as_str() == Some(name)) + .and_then(|account| account["account_ref"].as_str().map(str::to_string)) + .expect("account_ref present") +} + +#[tokio::test] +async fn patch_pool_account_pauses_and_reflects_in_snapshot_and_selection() { + if !can_bind_loopback() { + return; + } + let mut vars = common::env_lock().await; + vars.set("SHUNT_TEST_ADMIN_TOKENS_POOL_PAUSE", "ops:unused"); + let (gateway, state) = start(admin_config()).await; + let client = reqwest::Client::new(); + let account_ref = pool_account_ref(&client, &gateway, "anthropic", "pause-me").await; + + let response = client + .patch(format!( + "{}/admin/api/pool/anthropic/accounts/{account_ref}", + gateway.base_url + )) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .header("content-type", "application/json") + .body(r#"{"paused":true}"#) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + + // Reflected on the dashboard read immediately -- no traffic needed first. + let response = client + .get(format!("{}/admin/api/pool", gateway.base_url)) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + let body: serde_json::Value = response.json().await.unwrap(); + let accounts = body["providers"][0]["accounts"].as_array().unwrap(); + let account = accounts + .iter() + .find(|a| a["name"] == "pause-me") + .expect("patched account present in pool snapshot"); + assert_eq!(account["paused"], true); + assert_eq!(account["available"], false); + + // And enforced at selection time. + let config = admin_config(); + let accounts = config.providers.get("anthropic").unwrap().accounts.clone(); + assert!(state + .accounts + .select_order("anthropic", &accounts, None, None, None) + .is_empty()); + + drop(gateway); +} + +#[tokio::test] +async fn patch_pool_account_is_refused_for_a_read_key() { + if !can_bind_loopback() { + return; + } + let mut vars = common::env_lock().await; + vars.set("SHUNT_TEST_ADMIN_TOKENS_POOL_PAUSE", "ops:unused-2"); + let (gateway, _state) = start(admin_config()).await; + let client = reqwest::Client::new(); + let account_ref = pool_account_ref(&client, &gateway, "anthropic", "pause-me").await; + + let response = client + .patch(format!( + "{}/admin/api/pool/anthropic/accounts/{account_ref}", + gateway.base_url + )) + .header("x-shunt-admin-token", ADMIN_READ_KEY) + .header("content-type", "application/json") + .body(r#"{"paused":true}"#) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::FORBIDDEN); + + drop(gateway); +} + +#[tokio::test] +async fn patch_pool_account_404s_for_an_unknown_account() { + if !can_bind_loopback() { + return; + } + let mut vars = common::env_lock().await; + vars.set("SHUNT_TEST_ADMIN_TOKENS_POOL_PAUSE", "ops:unused-3"); + let (gateway, _state) = start(admin_config()).await; + let client = reqwest::Client::new(); + + let response = client + .patch(format!( + "{}/admin/api/pool/anthropic/accounts/acct_does_not_exist", + gateway.base_url + )) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .header("content-type", "application/json") + .body(r#"{"paused":true}"#) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::NOT_FOUND); + + drop(gateway); +} + +fn admin_config_with_two_accounts() -> Config { + let mut config = admin_config(); + let anthropic = config.providers.get_mut("anthropic").unwrap(); + anthropic.accounts = vec![ + AccountConfig { + name: "first".to_string(), + credentials: Some(nonexistent_credentials_path()), + uuid: Some("first-identity".to_string()), + ..Default::default() + }, + AccountConfig { + name: "second".to_string(), + credentials: Some(nonexistent_credentials_path()), + uuid: Some("second-identity".to_string()), + ..Default::default() + }, + ]; + config +} + +#[tokio::test] +async fn patch_pool_account_targets_selected_account_ref() { + if !can_bind_loopback() { + return; + } + let mut vars = common::env_lock().await; + vars.set("SHUNT_TEST_ADMIN_TOKENS_POOL_PAUSE", "ops:account-ref"); + let config = admin_config_with_two_accounts(); + let configured_accounts = config.providers.get("anthropic").unwrap().accounts.clone(); + let (gateway, state) = start(config).await; + let client = reqwest::Client::new(); + + let before = pool_accounts(&client, &gateway, "anthropic").await; + let first_ref = before + .iter() + .find(|account| account["name"].as_str() == Some("first")) + .and_then(|account| account["account_ref"].as_str()) + .unwrap() + .to_string(); + let second_ref = before + .iter() + .find(|account| account["name"].as_str() == Some("second")) + .and_then(|account| account["account_ref"].as_str()) + .unwrap() + .to_string(); + assert_ne!(first_ref, second_ref); + + let response = client + .patch(format!( + "{}/admin/api/pool/anthropic/accounts/{second_ref}", + gateway.base_url + )) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .header("content-type", "application/json") + .body(r#"{"paused":true}"#) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + + let after = pool_accounts(&client, &gateway, "anthropic").await; + let first = after + .iter() + .find(|account| account["account_ref"].as_str() == Some(first_ref.as_str())) + .unwrap(); + let second = after + .iter() + .find(|account| account["account_ref"].as_str() == Some(second_ref.as_str())) + .unwrap(); + assert_eq!(first["paused"], false); + assert_eq!(second["paused"], true); + + let order = state + .accounts + .select_order("anthropic", &configured_accounts, None, None, None); + assert_eq!(order.len(), 1, "only the targeted identity is excluded"); + + drop(gateway); +} + +/// `[server.pool]` is process-wide, so `PATCH /admin/api/pool` toggles the +/// reset-priority sort for every provider at once, with no `shunt.toml` edit +/// or restart. Also covers clearing the override back to config-following +/// with an explicit `{"sort_by_reset": null}` (distinct from omitting the +/// field, which is a no-op). +#[tokio::test] +async fn patch_pool_sort_by_reset_flips_the_process_wide_setting() { + if !can_bind_loopback() { + return; + } + let mut vars = common::env_lock().await; + vars.set("SHUNT_TEST_ADMIN_TOKENS_POOL_PAUSE", "ops:unused-4"); + let (gateway, state) = start(admin_config_with_pool()).await; + let client = reqwest::Client::new(); + + // Starts unset: falls back to the config file's own value (false, the + // `PoolConfig` default). + let pool_cfg = PoolConfig::default(); + assert!(!state.accounts.effective_sort_by_reset(Some(&pool_cfg))); + + let response = client + .get(format!("{}/admin/api/pool", gateway.base_url)) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .send() + .await + .unwrap(); + let body: serde_json::Value = response.json().await.unwrap(); + assert_eq!(body["sort_by_reset"], false); + + let response = client + .patch(format!("{}/admin/api/pool", gateway.base_url)) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .header("content-type", "application/json") + .body(r#"{"sort_by_reset":true}"#) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + assert!(state.accounts.effective_sort_by_reset(Some(&pool_cfg))); + + let response = client + .get(format!("{}/admin/api/pool", gateway.base_url)) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .send() + .await + .unwrap(); + let body: serde_json::Value = response.json().await.unwrap(); + assert_eq!(body["sort_by_reset"], true); + + // A read key may see the setting but not change it. + let response = client + .patch(format!("{}/admin/api/pool", gateway.base_url)) + .header("x-shunt-admin-token", ADMIN_READ_KEY) + .header("content-type", "application/json") + .body(r#"{"sort_by_reset":false}"#) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::FORBIDDEN); + assert!(state.accounts.effective_sort_by_reset(Some(&pool_cfg))); + + // Omitting the field entirely is a no-op: the override stays set. + let response = client + .patch(format!("{}/admin/api/pool", gateway.base_url)) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .header("content-type", "application/json") + .body(r#"{}"#) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + assert!(state.accounts.effective_sort_by_reset(Some(&pool_cfg))); + + // An explicit `null` clears the override, reverting to the config file's + // own value (`false`, unchanged in this test). + let response = client + .patch(format!("{}/admin/api/pool", gateway.base_url)) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .header("content-type", "application/json") + .body(r#"{"sort_by_reset":null}"#) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + assert!(!state.accounts.effective_sort_by_reset(Some(&pool_cfg))); + + drop(gateway); +} + +/// `effective_sort_by_reset` (and therefore `GET /admin/api/pool`) must report +/// `false` when `[server.pool]` is absent, even with the runtime override set +/// — that legacy branch never consults `sort_by_reset`, so reporting the +/// override as active there would claim an effect selection does not have. +#[tokio::test] +async fn patch_pool_sort_by_reset_has_no_effect_without_a_pool_table() { + if !can_bind_loopback() { + return; + } + let mut vars = common::env_lock().await; + vars.set("SHUNT_TEST_ADMIN_TOKENS_POOL_PAUSE", "ops:unused-5"); + let (gateway, state) = start(admin_config()).await; + let client = reqwest::Client::new(); + + let response = client + .patch(format!("{}/admin/api/pool", gateway.base_url)) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .header("content-type", "application/json") + .body(r#"{"sort_by_reset":true}"#) + .send() + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + assert!( + !state.accounts.effective_sort_by_reset(None), + "no [server.pool] table means the override has no effect" + ); + + let response = client + .get(format!("{}/admin/api/pool", gateway.base_url)) + .header("x-shunt-admin-token", ADMIN_WRITE_KEY) + .send() + .await + .unwrap(); + let body: serde_json::Value = response.json().await.unwrap(); + assert_eq!(body["sort_by_reset"], false); + + drop(gateway); +} diff --git a/tests/router_surface.rs b/tests/router_surface.rs index b4db9e2ba..573d35aaf 100644 --- a/tests/router_surface.rs +++ b/tests/router_surface.rs @@ -47,8 +47,11 @@ use tower::ServiceExt; mod common; /// A method no route in the crate registers, so a response distinguishes -/// "path exists" (`405`) from "path does not exist" (`404`). -const UNREGISTERED_METHOD: Method = Method::PATCH; +/// "path exists" (`405`) from "path does not exist" (`404`). `PATCH` no +/// longer qualifies now that `/admin/api/pool` and +/// `/admin/api/pool/{provider}/accounts/{account_ref}` register it (pool pause and +/// reset-priority-sort runtime controls). +const UNREGISTERED_METHOD: Method = Method::TRACE; /// Every `(path, allow)` pair the router registers with all optional surfaces /// enabled, grouped by the config table that gates it. This list **is** the @@ -85,7 +88,7 @@ const BASE_PATHS: [(&str, &str); 7] = [ /// method sets are unchanged by the move — the same handlers are registered at /// new paths — and `every_registered_method_set_matches_the_inventory` proves it /// against the live router rather than taking it on trust. -const ADMIN_PATHS: [(&str, &str); 23] = [ +const ADMIN_PATHS: [(&str, &str); 24] = [ ("/admin", "GET,HEAD"), // The same handler under the spelling a browser or proxy produces by // appending a slash. A `{*path}` segment cannot match the empty string, so @@ -99,7 +102,10 @@ const ADMIN_PATHS: [(&str, &str); 23] = [ ("/admin/api/session", "GET,HEAD"), ("/admin/api/accounts", "GET,HEAD"), ("/admin/api/observed", "GET,HEAD"), - ("/admin/api/pool", "GET,HEAD"), + // `PATCH` toggles the process-wide `sort_by_reset` runtime override + // (pool pause / reset-priority-sort admin controls). + ("/admin/api/pool", "GET,HEAD,PATCH"), + ("/admin/api/pool/{provider}/accounts/{account_ref}", "PATCH"), ("/admin/api/status", "GET,HEAD"), ("/admin/api/routes", "GET,HEAD"), ("/admin/api/accounts/claude", "POST"), @@ -197,6 +203,8 @@ const USAGE_PATHS: [(&str, &str); 2] = [("/usage", "GET,HEAD"), ("/api/oauth/usa /// the inventory can be sent as a real request. fn probe_path(template: &str) -> String { template + .replace("{provider}", "anthropic") + .replace("{account_ref}", "acct-ref") .replace("{name}", "acct") .replace("{id}", "spl_1") .replace("{*path}", "probe") @@ -685,14 +693,16 @@ fn the_source_scan_finds_every_literal_registration() { .iter() .map(|(_, source)| registered_literal_paths(source).len()) .sum(); - // 9 in `server.rs` (7 base + `/usage` + `/api/oauth/usage`), 23 admin plus - // the 5 UI routes, 7 gateway (its 3 OTLP paths come from `Signal::path()`), - // 2 spend. The UI five are counted unconditionally: this scan reads source - // text, and `#[cfg(feature = "ui")]` does not remove the `.route("…"` - // literals from it. + // 9 in `server.rs` (7 base + `/usage` + `/api/oauth/usage`), 24 admin + // (`/admin/api/pool` merges its `PATCH` onto the existing `.route(` call, + // so only the new `/admin/api/pool/{provider}/accounts/{account_ref}` literal + // adds one) plus the 5 UI routes, 7 gateway (its 3 OTLP paths come from + // `Signal::path()`), 2 spend. The UI five are counted unconditionally: + // this scan reads source text, and `#[cfg(feature = "ui")]` does not + // remove the `.route("…"` literals from it. assert_eq!( - found, 46, - "the literal-path scan found {found} registrations, not 46; either a route was added or \ + found, 47, + "the literal-path scan found {found} registrations, not 47; either a route was added or \ removed, or `.route(\"…\"` is no longer how they are spelled" ); } diff --git a/ui/src/Dashboard.tsx b/ui/src/Dashboard.tsx index c4552ddc3..94f826342 100644 --- a/ui/src/Dashboard.tsx +++ b/ui/src/Dashboard.tsx @@ -104,7 +104,7 @@ export function Dashboard(): ReactElement { onMutated={afterAntigravityMutation} onMessage={(text, ok) => antigravityForm.current?.report(text, ok)} /> - + void reloadPool()} /> diff --git a/ui/src/__tests__/mutations.test.tsx b/ui/src/__tests__/mutations.test.tsx index face9147f..0c14d733e 100644 --- a/ui/src/__tests__/mutations.test.tsx +++ b/ui/src/__tests__/mutations.test.tsx @@ -149,6 +149,40 @@ describe('a store mutation re-reads the grouped table too', () => { }); }); +describe('pool pause identity', () => { + it('targets duplicate display names by account_ref', async () => { + const user = userEvent.setup(); + const api = await renderDashboard( + { + pool: [ + { + provider: 'anthropic', + auth: 'claude_oauth', + accounts: [ + { name: 'same-name', account_ref: 'acct_first' }, + { name: 'same-name', account_ref: 'acct_second' }, + ], + }, + ], + }, + { + 'PATCH /admin/api/pool/anthropic/accounts/acct_second': () => reply({ ok: true }), + }, + ); + + const names = tbody('pool').getAllByText('same-name'); + const secondRow = rowOf(names[1]); + await user.click(within(secondRow).getByRole('button', { name: 'Pause' })); + + expect( + api.callsTo('PATCH', '/admin/api/pool/anthropic/accounts/acct_second'), + ).toHaveLength(1); + expect( + api.callsTo('PATCH', '/admin/api/pool/anthropic/accounts/acct_first'), + ).toHaveLength(0); + }); +}); + describe('the session bootstrap', () => { /** * The shell is served to anyone — it carries no operator data — so an diff --git a/ui/src/api.ts b/ui/src/api.ts index 88c68c790..99e0f8a9e 100644 --- a/ui/src/api.ts +++ b/ui/src/api.ts @@ -111,3 +111,30 @@ export async function mutate( } return { ok: response.ok, message: errorMessage(payload), payload, answered }; } + +export async function patchPoolAccount( + csrf: string, + provider: string, + accountRef: string, + paused: boolean +): Promise { + const path = `${API}/pool/${encodeURIComponent(provider)}/accounts/${encodeURIComponent(accountRef)}`; + return mutate(path, csrf, { + method: 'PATCH', + body: JSON.stringify({ paused }), + }); +} + +/** + * `[server.pool]` is process-wide, not per-provider, so this toggles the + * reset-priority sort for the whole pool at once. + */ +export async function patchPoolSortByReset( + csrf: string, + sortByReset: boolean +): Promise { + return mutate(`${API}/pool`, csrf, { + method: 'PATCH', + body: JSON.stringify({ sort_by_reset: sortByReset }), + }); +} diff --git a/ui/src/components/PoolHealth.tsx b/ui/src/components/PoolHealth.tsx index 4f4860b01..2574ecf47 100644 --- a/ui/src/components/PoolHealth.tsx +++ b/ui/src/components/PoolHealth.tsx @@ -1,7 +1,9 @@ -import type { ReactElement } from 'react'; +import { useState, type ReactElement } from 'react'; +import { patchPoolAccount, patchPoolSortByReset } from '../api'; import { pctReset, titleCase } from '../format'; -import type { PoolAccount, PoolProvider } from '../types'; +import { useCanWrite, useSession } from '../session'; +import type { PoolAccount, PoolData } from '../types'; import type { Loadable } from '../useDashboard'; function poolState(account: PoolAccount): string { @@ -9,7 +11,10 @@ function poolState(account: PoolAccount): string { // credential is *also* cooling down, and reporting only "cooling" is what made // a permanently dead account indistinguishable from a quota pause. if (account.disabled) return 'disabled'; + // A permanently dead credential remains the most actionable state even if + // the operator has also paused this provider lane. if (account.needs_relogin) return 'needs re-login'; + if (account.paused) return 'paused'; if (!account.has_state) return 'unseen'; // The cooldown is checked before `near_quota`, matching `managedState` in // `accounts.ts` so the two tables cannot report the same account differently: @@ -37,17 +42,71 @@ function cooldownText(account: PoolAccount): string { ); } -/** The shunt-owned credential lane, read-only. */ -export function PoolHealth({ pool }: { pool: Loadable }): ReactElement { +export interface PoolHealthProps { + pool: Loadable; + onMutated: () => void; +} + +/** + * The shunt-owned credential lane. Read-only, except pausing an account and + * toggling the reset-priority sort — both runtime overrides that take effect + * without editing `shunt.toml`. + */ +export function PoolHealth({ pool, onMutated }: PoolHealthProps): ReactElement { + const { csrf } = useSession(); + const canWrite = useCanWrite(); + const [message, setMessage] = useState(null); + const columns = canWrite ? 10 : 9; + + async function togglePause(provider: string, accountRef: string, paused: boolean): Promise { + try { + const result = await patchPoolAccount(csrf, provider, accountRef, paused); + if (!result.ok) { + setMessage(result.message ?? `Failed to ${paused ? 'pause' : 'resume'} account`); + return; + } + setMessage(null); + onMutated(); + } catch { + setMessage('Request failed'); + } + } + + async function toggleSortByReset(sortByReset: boolean): Promise { + try { + const result = await patchPoolSortByReset(csrf, sortByReset); + if (!result.ok) { + setMessage(result.message ?? 'Failed to change pool sort order'); + return; + } + setMessage(null); + onMutated(); + } catch { + setMessage('Request failed'); + } + } + const rows = pool.status === 'ready' - ? pool.data.flatMap((table) => + ? pool.data.providers.flatMap((table) => (table.accounts ?? []).map((account) => ({ provider: table.provider, account })), ) : []; return ( <>

Managed pool health

+ {canWrite && pool.status === 'ready' ? ( +

+ +

+ ) : null}
@@ -61,32 +120,33 @@ export function PoolHealth({ pool }: { pool: Loadable }): ReactE + {canWrite ? {pool.status === 'loading' ? ( - ) : null} {pool.status === 'error' ? ( - + ) : null} {pool.status === 'ready' && !rows.length ? ( - ) : null} - {rows.map(({ provider, account }) => { + {rows.map(({ provider, account }, rowIndex) => { const state = poolState(account); return ( - + @@ -112,12 +172,34 @@ export function PoolHealth({ pool }: { pool: Loadable }): ReactE + {canWrite ? ( + + ) : null} ); })}
7d_oi Status Cooldown : null}
+ Loading…
{pool.message}{pool.message}
+ No pooled accounts configured
{provider} {account.name} {titleCase(account.plan) || '—'}{account.status || '—'} {cooldownText(account)} + +
+ {message ?

{message}

: null} ); } diff --git a/ui/src/types.ts b/ui/src/types.ts index e8181f0d6..0ec90ba41 100644 --- a/ui/src/types.ts +++ b/ui/src/types.ts @@ -52,9 +52,12 @@ export interface ObservedAccount { export interface PoolAccount { name: string; + /** Opaque provider-scoped identity used by runtime account mutations. */ + account_ref?: string | null; plan?: string | null; status?: string | null; disabled?: boolean; + paused?: boolean; needs_relogin?: boolean; has_state?: boolean; near_quota?: boolean; @@ -76,6 +79,15 @@ export interface PoolProvider { accounts?: PoolAccount[]; } +/** + * `sortByReset` is process-wide (`[server.pool]` is not per-provider), so it + * rides alongside the provider list rather than inside it. + */ +export interface PoolData { + providers: PoolProvider[]; + sortByReset: boolean; +} + export interface ClaudeStoreAccount { name: string; kind: string; diff --git a/ui/src/useDashboard.ts b/ui/src/useDashboard.ts index 486baa4f7..28e90abd1 100644 --- a/ui/src/useDashboard.ts +++ b/ui/src/useDashboard.ts @@ -9,6 +9,7 @@ import type { ClaudeStoreAccount, CodexStoreAccount, ObservedAccount, + PoolData, PoolProvider, StatusSource, } from './types'; @@ -47,7 +48,7 @@ export interface Dashboard { accounts: Loadable; codexAccounts: Loadable; antigravityAccounts: Loadable; - pool: Loadable; + pool: Loadable; /** `null` means the section is hidden: `[server.status]` is opt-in. */ status: StatusSource[] | null; reloadObserved: () => Promise; @@ -128,10 +129,16 @@ export function useDashboard(): Dashboard { : { status: 'error', message: result.message }; }, []); - const loadPool = useCallback(async (): Promise> => { - const result = await readJson<{ providers?: PoolProvider[] }>(`${API}/pool`, 'Failed to load pool'); + const loadPool = useCallback(async (): Promise> => { + const result = await readJson<{ providers?: PoolProvider[]; sort_by_reset?: boolean }>( + `${API}/pool`, + 'Failed to load pool', + ); return result.ok - ? { status: 'ready', data: result.data.providers ?? [] } + ? { + status: 'ready', + data: { providers: result.data.providers ?? [], sortByReset: result.data.sort_by_reset ?? false }, + } : { status: 'error', message: result.message }; }, []);