diff --git a/docs-site/src/content/docs/fr/reference/management-api.md b/docs-site/src/content/docs/fr/reference/management-api.md index 9e7a6ec106f..1ee13d85e11 100644 --- a/docs-site/src/content/docs/fr/reference/management-api.md +++ b/docs-site/src/content/docs/fr/reference/management-api.md @@ -345,6 +345,8 @@ lui-même s'il souhaite ajouter une étoile au dépôt. ### Cycle de vie du système +`POST /api/system/restart` conserve par défaut un délai de 60 secondes lorsque le corps est absent ou vaut `{}`. Une session de gestion ou un jeton administrateur peut choisir explicitement un délai plus court avec `{"drainGraceMs":2000}` : la valeur doit être un entier compris entre 1 et 60000 millisecondes. Un corps ou une valeur invalide renvoie 400 sans lancer le redémarrage ; la capacité locale liée à la cible ne permet pas cette option (403). `drainTimeoutMs` indique le délai accepté, fixé par le premier appel et inchangé lors des appels suivants. Il inclut le temps de transmission de la réponse ; les budgets de nettoyage et de préparation du processus de remplacement restent respectivement de 60 et 70 secondes. Une requête interrompue peut déjà avoir été exécutée : vérifiez son résultat avant de la renvoyer, sans répétition automatique. + | Méthode et chemin | Objectif | Erreurs notables | | --- | --- | --- | | `GET /api/system/memory` | Renvoyer les mesures scalaires du processus, du tas, des flux, de l'état des réponses, du mécanisme de surveillance et des tours actifs | — | diff --git a/docs-site/src/content/docs/ja/reference/management-api.md b/docs-site/src/content/docs/ja/reference/management-api.md index 25eb7807eab..733c910157c 100644 --- a/docs-site/src/content/docs/ja/reference/management-api.md +++ b/docs-site/src/content/docs/ja/reference/management-api.md @@ -301,6 +301,8 @@ CLI コマンドは Anthropic OAuth アカウントを id または一意の別 ### システムのライフサイクル +`POST /api/system/restart` は、本文を省略した場合や `{}` の場合、既定の 60 秒のドレイン猶予を維持します。管理セッションまたは管理者トークンを使う呼び出し元は、`{"drainGraceMs":2000}` で短い猶予を明示的に選択できます。値は 1~60000 ミリ秒の整数です。不正な本文や値は再起動を開始せずに 400 を返します。対象プロセスに限定されたローカル再起動 capability では、このオプションを設定できません(403)。応答の `drainTimeoutMs` は最初に受け入れた猶予を示し、繰り返しの呼び出しでも期限は変わりません。猶予には応答送信までの遅延を含み、クリーンアップと置換プロセスの準備には別々の 60 秒・70 秒の予算があります。中断された処理が既に実行された可能性があるため、再送前に結果を確認してください。自動再送は追加されません。 + |メソッドとパス |目的 |注目すべきエラー | | --- | --- | --- | | `GET /api/system/memory` |スカラー プロセス、ヒープ、ストリーム、応答状態、ウォッチドッグ、およびアクティブ ターン メトリックを返します。 — | diff --git a/docs-site/src/content/docs/ko/reference/management-api.md b/docs-site/src/content/docs/ko/reference/management-api.md index ed7d0292fa8..f9dd67d9f89 100644 --- a/docs-site/src/content/docs/ko/reference/management-api.md +++ b/docs-site/src/content/docs/ko/reference/management-api.md @@ -313,6 +313,8 @@ OpenAI도 같은 규칙을 따르며, 스위치를 켠다고 별도의 922k 모 ### 시스템 수명 주기 +`POST /api/system/restart`는 본문이 없거나 `{}`이면 기본 60초 drain 유예 시간을 유지합니다. 관리 세션 또는 관리자 토큰을 사용하는 호출자는 `{"drainGraceMs":2000}`으로 짧은 유예 시간을 명시적으로 선택할 수 있습니다. 값은 1~60000밀리초의 정수여야 합니다. 잘못된 본문이나 값은 재시작을 시작하지 않고 400을 반환하며, 대상 프로세스에 한정된 로컬 재시작 capability는 이 옵션을 설정할 수 없습니다(403). 응답의 `drainTimeoutMs`는 처음 수락한 유예 시간을 나타내며 반복 호출은 기존 기한을 바꾸지 않습니다. 유예 시간에는 응답 전송 지연이 포함됩니다. 정리와 대체 프로세스 준비에는 각각 별도의 60초와 70초 예산이 유지됩니다. 중단된 요청이 이미 실행되었을 수 있으므로 재전송 전에 결과를 확인하세요. 자동 재전송은 추가되지 않습니다. + | HTTP 메서드와 경로 | 목적 | 주요 오류 | | --- | --- | --- | | `GET /api/system/memory` | 프로세스, heap, stream, response-state, watchdog, active-turn의 스칼라 메트릭을 반환합니다 | — | diff --git a/docs-site/src/content/docs/reference/management-api.md b/docs-site/src/content/docs/reference/management-api.md index 3bd3700766c..5b7e28afe56 100644 --- a/docs-site/src/content/docs/reference/management-api.md +++ b/docs-site/src/content/docs/reference/management-api.md @@ -662,6 +662,23 @@ whether to star the repository. ### System lifecycle +`POST /api/system/restart` keeps the 60-second drain by default, including requests +with no body or `{}`. An authenticated caller can explicitly send a JSON body such +as `{"drainGraceMs":2000}` to choose a shorter active-request/scoped-drain grace. +The value must be an integer from 1 to 60000 milliseconds. Invalid bodies or values +return 400 before starting a drain; the standard management-body limit still applies. +The target-bound local restart capability retains the default grace; setting this +body option requires a management session or admin token (otherwise 403). +The response's `drainTimeoutMs` reports the accepted grace. Repeated calls retain +the first accepted restart's grace and do not change its deadline. + +Grace is measured from acceptance, including the response-flush delay. The cleanup +watchdog remains 60 seconds and replacement readiness retains its separate 70-second +budget. A short grace can interrupt a turn that already executed upstream. Check its +outcome before resubmitting; OpenCodex adds no automatic replay of ambiguous work, +and a healthy replacement does not prove resending is safe. The dashboard, CLI, tray +and automatic restarts continue using the 60-second default. + During a restart drain, new data-plane requests receive HTTP 503 with JSON `error.type: "server_error"`, `error.code: "server_restarting"`, and the message "OpenCodex is restarting; retry this request." Responses retain `Retry-After: 5` @@ -671,7 +688,7 @@ the 503 without reporting model capacity; provider overload errors retain their | Method and path | Purpose | Notable errors | | --- | --- | --- | | `GET /api/system/memory` | Return scalar process, heap, stream, response-state, watchdog, and active-turn metrics. Response-state diagnostics include spill-write status, consecutive failures, fixed privacy-safe failure class, and last failure/success timestamps. `spillLastWriteFailureOrigin` is `retry_returned_timeout`, `timeout_memo_refusal`, or null; cumulative `spillAclRetryReturnedTimeouts` and `spillAclTimeoutMemoRefusals` count terminal failed publications. See [Windows spill diagnostics](/troubleshooting/windows-memory/) for process-local semantics. Raw errors and paths are never returned. | — | -| `POST /api/system/restart` | Begin a drain-aware process restart without removing client injection | Returns 202; repeated calls report the existing drain | +| `POST /api/system/restart` | Begin a drain-aware process restart without removing client injection; optional JSON `drainGraceMs` explicitly shortens the default 60s grace | Returns 202; repeated calls report the existing grace; 400 invalid body or grace | | `POST /api/stop` | Stop the service, restore native Codex, remove managed Grok injection, and drain the proxy | 409 service ownership conflict; 409 `respawnable_service` when a Windows Task Scheduler wrapper could respawn the proxy and the caller is not `ocx stop` (nothing is changed); 409 `self_unload_service` when this proxy is running as the installed launchd/systemd service, because stopping the manager from inside it would end the process before native Codex is restored — run `ocx stop` instead (nothing is changed); 409 when the installed manager refuses to stop; 409 `service_state_unknown` when the Task Scheduler state cannot be read (nothing is changed; repair the query and retry) | | `GET /api/system/codex-app-server` | Report whether running Codex app-servers predate the current model catalog | — | | `POST /api/system/codex-restart` | Refresh the catalog, then restart stale Codex app-servers and fully quit and relaunch the Codex desktop app so the model picker reloads. When the proxy itself is running inside the Codex app, the desktop restart is refused rather than handed off. | Returns 200 with `code: partially_stopped` when a target survives | diff --git a/docs-site/src/content/docs/ru/reference/management-api.md b/docs-site/src/content/docs/ru/reference/management-api.md index 77b456ab70c..209c7ffdf05 100644 --- a/docs-site/src/content/docs/ru/reference/management-api.md +++ b/docs-site/src/content/docs/ru/reference/management-api.md @@ -338,6 +338,8 @@ Management-аутентификация доказывает доступ к п ### Жизненный цикл системы +`POST /api/system/restart` сохраняет стандартный период drain в 60 секунд, если тело отсутствует или равно `{}`. Сессия управления или токен администратора позволяют явно выбрать более короткий период через `{"drainGraceMs":2000}`. Значение должно быть целым числом от 1 до 60000 миллисекунд. Некорректное тело или значение возвращает 400 без запуска перезапуска; локальная capability, привязанная к целевому процессу, не разрешает этот параметр (403). Поле ответа `drainTimeoutMs` сообщает первый принятый период, который повторные вызовы не меняют. Отсчёт включает задержку отправки ответа; отдельные бюджеты очистки и готовности замены остаются равными 60 и 70 секундам. Прерванный запрос мог уже выполниться: проверьте результат перед повторной отправкой. Автоматическая повторная отправка не добавляется. + | Метод и путь | Назначение | Особые ошибки | | --- | --- | --- | | `GET /api/system/memory` | Вернуть скалярные метрики процесса, heap, stream, response-state, watchdog и active-turn | — | diff --git a/docs-site/src/content/docs/tr/reference/management-api.md b/docs-site/src/content/docs/tr/reference/management-api.md index 2199bce119f..5233d4ce458 100644 --- a/docs-site/src/content/docs/tr/reference/management-api.md +++ b/docs-site/src/content/docs/tr/reference/management-api.md @@ -363,6 +363,8 @@ dolaşmamalıdır. Depoya yıldız verip vermeyeceğini kullanıcı seçmelidir. ### Sistem yaşam döngüsü +`POST /api/system/restart`, gövde yoksa veya `{}` ise varsayılan 60 saniyelik boşaltma süresini korur. Bir yönetim oturumu veya yönetici token'ı, `{"drainGraceMs":2000}` ile daha kısa süreyi açıkça seçebilir. Değer 1 ile 60000 milisaniye arasında bir tam sayı olmalıdır. Geçersiz gövde veya değer, yeniden başlatmayı başlatmadan 400 döndürür; hedef sürece bağlı yerel yeniden başlatma capability'si bu seçeneğe izin vermez (403). Yanıttaki `drainTimeoutMs`, ilk kabul edilen süreyi bildirir; tekrarlanan çağrılar mevcut süreyi veya son tarihi değiştirmez. Yanıt gönderme gecikmesi bu süreye dahildir; temizleme ve yeni sürecin hazır olması için ayrı 60 ve 70 saniyelik bütçeler korunur. Kesilen bir istek zaten yürütülmüş olabilir: yeniden göndermeden önce sonucunu kontrol edin. Otomatik yeniden gönderme eklenmez. + | Yöntem ve yol | Amaç | Dikkate değer hatalar | | --- | --- | --- | | `GET /api/system/memory` | Skaler süreç, yığın (heap), akış, yanıt durumu, denetleyici ve aktif tur metriklerini döndürün | — | diff --git a/docs-site/src/content/docs/zh-cn/reference/management-api.md b/docs-site/src/content/docs/zh-cn/reference/management-api.md index abcdb26b4f9..a590806cd01 100644 --- a/docs-site/src/content/docs/zh-cn/reference/management-api.md +++ b/docs-site/src/content/docs/zh-cn/reference/management-api.md @@ -292,6 +292,8 @@ OpenAI 也遵循此规则:开关不会选择特殊的 922k 模式。有效上 ### 系统生命周期 +`POST /api/system/restart` 在没有请求体或请求体为 `{}` 时保留默认的 60 秒排空等待。使用管理会话或管理员令牌的调用方可通过 `{"drainGraceMs":2000}` 显式选择短等待,取值须为 1–60000 毫秒的整数。无效 JSON 或参数返回 400,且不会开始重启;绑定目标进程的本机 restart capability 不允许设置该参数(403)。响应中的 `drainTimeoutMs` 表示首次接受的等待时间,重复调用不会改变已接受的时间或截止点。等待时间包含发送接受响应前的延迟,清理和替代进程就绪仍分别保留独立的 60 秒、70 秒预算。被中断的请求可能已经执行,重发前应核对结果;此选项不会增加自动重发。 + | 方法和路径 | 用途 | 典型错误 | | --- | --- | --- | | `GET /api/system/memory` | 返回标量级的进程、堆、流、响应状态、看门狗和活跃回合指标 | — | diff --git a/docs-site/src/content/docs/zh-tw/reference/management-api.md b/docs-site/src/content/docs/zh-tw/reference/management-api.md index 8bbfae3db3c..a67def0e48a 100644 --- a/docs-site/src/content/docs/zh-tw/reference/management-api.md +++ b/docs-site/src/content/docs/zh-tw/reference/management-api.md @@ -289,6 +289,8 @@ OpenAI 也遵循此規則:開關不會選擇特殊的 922k 模式。生效中 ### 系統生命週期 +`POST /api/system/restart` 在沒有請求本文或本文為 `{}` 時,保留預設的 60 秒排空等待。使用管理工作階段或管理員權杖的呼叫端可透過 `{"drainGraceMs":2000}` 明確選擇短等待;值必須是 1–60000 毫秒的整數。無效 JSON 或參數回傳 400,且不會開始重新啟動;綁定目標行程的本機 restart capability 不允許設定此參數(403)。回應中的 `drainTimeoutMs` 表示第一次接受的等待時間,重複呼叫不會變更已接受的時間或截止點。等待時間包含傳送接受回應前的延遲;清理與替代行程就緒仍分別保留獨立的 60 秒及 70 秒預算。被中斷的請求可能已經執行,重新傳送前應核對結果;此選項不會增加自動重新傳送。 + | 方法與路徑 | 用途 | 主要錯誤 | | --- | --- | --- | | `GET /api/system/memory` | 回傳純量行程、heap、串流、回應狀態、看門狗與活躍回合指標 | — | diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 05cae70aa51..8d1f0b60845 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1807,6 +1807,7 @@ "system-restart-client-package-tree.test.ts": "cli", "system-restart-client.test.ts": "cli", "system-restart-contract-security.test.ts": "lib", + "system-restart-admission.test.ts": "server", "system-restart.test.ts": "server", "system-routes.test.ts": "server", "systemd-install-cleanup-hardening.test.ts": "service", diff --git a/src/server/lifecycle.ts b/src/server/lifecycle.ts index e36bc460682..c4da2c28a16 100644 --- a/src/server/lifecycle.ts +++ b/src/server/lifecycle.ts @@ -479,7 +479,7 @@ export async function drainAndShutdown( beginBackgroundShellShutdown(); try { while (admittedTurns.size > 0 && Date.now() < deadline) { - await Bun.sleep(100); + await Bun.sleep(Math.min(100, Math.max(0, deadline - Date.now()))); } if (admittedTurns.size > 0) { console.warn(`⚠️ Aborting ${admittedTurns.size} in-flight turn(s) after ${timeoutMs}ms deadline`); diff --git a/src/server/management/system-restart.ts b/src/server/management/system-restart.ts index 5f7ff8e3f7e..780fbc94e3f 100644 --- a/src/server/management/system-restart.ts +++ b/src/server/management/system-restart.ts @@ -1,8 +1,8 @@ /** * Dashboard memory-card drain-and-restart (#563). * - * Longer than POST /api/stop's short drain: waits up to 60s for active turns, - * then respawns. Never runs restoreNativeCodex / stripGrokConfig — this is a + * Waits up to 60s for active turns unless a manual API caller explicitly opts + * into a shorter grace, with the existing 60s cleanup watchdog, then respawns. Never runs restoreNativeCodex / stripGrokConfig — this is a * recycle to reclaim RSS, not a teardown. * * Respawn policy (matches real supervisor configs in src/service.ts): @@ -92,6 +92,11 @@ export interface SystemRestartIo { now?: () => number; } +export interface SystemRestartOptions { + /** Explicit API opt-in; omission preserves the existing 60s drain. */ + drainGraceMs?: number; +} + export interface SystemRestartAdmission { /** Only the caller that created this pending restart receives its veto. */ onAccepted?: (veto: () => void) => void; @@ -102,6 +107,7 @@ export interface SystemRestartAdmission { let restartIo: SystemRestartIo = {}; /** Prevents double-scheduling in the 200ms window before drainAndShutdown sets draining. */ let restartAccepted = false; +let acceptedDrainGraceMs = MEMORY_DRAIN_RESTART_MS; type RestartDrainOutcome = "completed" | "failed" | "rejected" | "deadline"; type BoundedSettlementOutcome = "completed" | "rejected" | "deadline"; @@ -173,6 +179,7 @@ export function noteExplicitShutdownRequested(): void { export function setSystemRestartIoForTests(io: SystemRestartIo = {}): void { restartIo = io; restartAccepted = false; + acceptedDrainGraceMs = MEMORY_DRAIN_RESTART_MS; explicitShutdownRequested = false; } @@ -327,7 +334,7 @@ async function completeDeadlineRestartHandoff( * respawn runs on a short timer so the HTTP response can flush first. * Idempotent while already draining: returns the accepted shape again. */ -export function acceptSystemRestart(io: SystemRestartIo = restartIo, admission: SystemRestartAdmission = {}): { +export function acceptSystemRestart(io: SystemRestartIo = restartIo, admission: SystemRestartAdmission = {}, options: SystemRestartOptions = {}): { accepted: true; alreadyDraining: boolean; activeTurnCount: number; @@ -337,11 +344,15 @@ export function acceptSystemRestart(io: SystemRestartIo = restartIo, admission: ?? io.isDraining ?? isShutdownDraining; const alreadyDraining = restartAccepted || shutdownActive(); + const drainGraceMs = alreadyDraining + ? (restartAccepted ? acceptedDrainGraceMs : MEMORY_DRAIN_RESTART_MS) + : (options.drainGraceMs ?? MEMORY_DRAIN_RESTART_MS); const activeTurnCount = (io.getActiveTurnCount ?? getActiveTurnCount)(); const schedule = io.schedule ?? ((fn, ms) => { setTimeout(() => { void fn(); }, ms); }); if (!alreadyDraining) { restartAccepted = true; + acceptedDrainGraceMs = drainGraceMs; const automatic = Boolean(admission.onAccepted); const temporaryDrain = automatic ? (io.acquireTemporaryDrain ?? (() => acquireTemporaryDrain("automatic-restart")))() @@ -363,7 +374,9 @@ export function acceptSystemRestart(io: SystemRestartIo = restartIo, admission: releasePending(); }); const now = io.now ?? Date.now; - const restartDeadlineMs = now() + MEMORY_DRAIN_RESTART_MS; + const acceptedAtMs = now(); + const restartDeadlineMs = acceptedAtMs + MEMORY_DRAIN_RESTART_MS; + const drainDeadlineMs = acceptedAtMs + drainGraceMs; // Reject new data-plane traffic immediately (503), before the 200ms response-flush delay. if (!automatic) { if (io.beginShutdownDrain) io.beginShutdownDrain(); @@ -390,8 +403,11 @@ export function acceptSystemRestart(io: SystemRestartIo = restartIo, admission: // closes the listener and makes both the server ref and runtime metadata stale. const restartPort = (io.listenPort ?? resolveListenPort)(); const drain = io.drainAndShutdown ?? drainAndShutdown; - const remainingMs = Math.max(0, restartDeadlineMs - now()); - const drainPromise = Promise.resolve().then(() => drain(undefined, remainingMs)); + // Bound active/scoped-drain waiting independently of cleanup and readiness. + // A cut turn may already have executed upstream; never replay it automatically. + const drainPromise = Promise.resolve().then( + () => drain(undefined, Math.max(0, drainDeadlineMs - now())), + ); const scheduleDeadline = io.scheduleDeadline ?? ((fn, ms) => { const timer = setTimeout(fn, ms); return () => clearTimeout(timer); @@ -458,6 +474,6 @@ export function acceptSystemRestart(io: SystemRestartIo = restartIo, admission: accepted: true, alreadyDraining, activeTurnCount, - drainTimeoutMs: MEMORY_DRAIN_RESTART_MS, + drainTimeoutMs: drainGraceMs, }; } diff --git a/src/server/management/system-routes.ts b/src/server/management/system-routes.ts index 849cde12540..7ee0376aa72 100644 --- a/src/server/management/system-routes.ts +++ b/src/server/management/system-routes.ts @@ -30,6 +30,7 @@ import { inspectResponseSpillStorage, responseStateMetrics, type ResponseSpillDi import { appOwnedBytesSnapshot } from "../../lib/app-owned-memory"; import { readWindowsReplaceRetryCounters } from "../../lib/windows-atomic-replace"; import { + MEMORY_DRAIN_RESTART_MS, SYSTEM_RESTART_EXPECTED_PID_HEADER, parseExpectedSystemRestartPid, } from "../../lib/system-restart-contract"; @@ -46,6 +47,7 @@ import type { } from "../../codex/app-server-restart-service"; import type { ManagementContext } from "./context"; import { acceptSystemRestart } from "./system-restart"; +import { readOptionalManagementJsonBody, rethrowManagementBodyTooLarge } from "./body"; const ENDPOINT_SAMPLE_LIMIT = 60; // The spill report reads the snapshot and walks the spill directory synchronously, @@ -179,8 +181,28 @@ export async function handleSystemRoutes(ctx: ManagementContext): Promise).drainGraceMs; + if (drainGraceMs !== undefined && ( + typeof drainGraceMs !== "number" || !Number.isSafeInteger(drainGraceMs) + || drainGraceMs < 1 || drainGraceMs > MEMORY_DRAIN_RESTART_MS + )) { + return jsonResponse({ success: false, error: "drainGraceMs must be an integer from 1 to 60000." }, 400, req, config); + } + // The process capability signs the restart target, not this new body option. + if (drainGraceMs !== undefined && ctx.principal === "system-restart-capability") { + return jsonResponse({ success: false, error: "Short grace requires management authentication." }, 403, req, config); + } + // Omission retains the informed 60s drain; only an explicit API request shortens it. + const result = acceptSystemRestart(undefined, {}, { drainGraceMs }); return jsonResponse({ success: true, message: result.alreadyDraining diff --git a/structure/ops/service-and-sidecars.md b/structure/ops/service-and-sidecars.md index 64cc27a2950..c78d77a120e 100644 --- a/structure/ops/service-and-sidecars.md +++ b/structure/ops/service-and-sidecars.md @@ -466,6 +466,15 @@ clients do not retry it at all. `server_restarting` falls through to retryable response bypasses the shared provider-overload mapping in `src/lib/errors.ts`. `tests/codex-integration/issue-452-empty-503.test.ts` pins its body and both listeners' CORS. +`src/server/management/system-routes.ts` accepts an optional JSON `drainGraceMs` on the authenticated +restart API: an integer from 1 to 60000, with omission retaining the 60-second default. Invalid +input starts no drain. `src/server/management/system-restart.ts` fixes the selected grace at first +acceptance, including the response-flush delay; repeated requests cannot shorten it. The cleanup +watchdog stays 60 seconds and replacement readiness stays 70 seconds. Short grace is an explicit +API opt-in; dashboard, CLI, tray, join and automatic restarts keep their default. An interrupted +turn may have an unknown upstream outcome and is not automatically replayed. +`tests/server/system-restart-admission.test.ts` covers this contract and active-turn preservation. + A dashboard drain-and-restart (`src/server/management/system-restart.ts`, which is also the restart after a join into a Child) and the client runtime's standalone recycle (`src/client/runtime.ts`) replace their process through `src/server/restart-replacement.ts`. Every replacement `ocx start` diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index b41a1beb973..e6b8525744a 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1023,7 +1023,7 @@ "sync-client-integrations.test.ts": "clients", "synced-state-location.test.ts": "lib", "synthetic-tool.test.ts": "images", "system-env.test.ts": "server", "system-restart-client-package-tree.test.ts": "cli", "system-restart-client.test.ts": "cli", - "system-restart-contract-security.test.ts": "lib", "system-restart.test.ts": "server", + "system-restart-contract-security.test.ts": "lib", "system-restart-admission.test.ts": "server", "system-restart.test.ts": "server", "system-routes.test.ts": "server", "systemd-install-cleanup-hardening.test.ts": "service", "tencent-siliconflow-providers.test.ts": "gui", "terminal-continuation-owner-rotation.test.ts": "adapters", "terminal-guard-server.test.ts": "server", "terminal-guard.test.ts": "server", diff --git a/tests/server/system-restart-admission.test.ts b/tests/server/system-restart-admission.test.ts new file mode 100644 index 00000000000..72c0f037a80 --- /dev/null +++ b/tests/server/system-restart-admission.test.ts @@ -0,0 +1,426 @@ +/** #6643: an explicit short grace preserves the default drain and cleanup budget. */ +import { afterEach, expect, spyOn, test } from "bun:test"; +import { + acceptSystemRestart, + setSystemRestartIoForTests, + noteExplicitShutdownRequested, +} from "../../src/server/management/system-restart"; + +import { + abortAndReleaseAllTurns, + acquireTemporaryDrain, + drainAndShutdown, + getActiveTurnCount, + isDraining, + registerTurn, + resetLifecycleDrainStateForTests, + stopServerListener, + tryAdmitTurn, +} from "../../src/server/lifecycle"; +import { handleManagementAPI } from "../../src/server/management-api"; +import { handleSystemRoutes } from "../../src/server/management/system-routes"; +import { ManagementRequest } from "../helpers/management-auth"; +import type { OcxConfig } from "../../src/types"; + +afterEach(() => setSystemRestartIoForTests()); + +const apiConfig: OcxConfig = { + port: 10100, + defaultProvider: "openai", + providers: { openai: { adapter: "openai-chat", baseUrl: "https://api.example.test/v1", apiKey: "fixture", defaultModel: "fixture" } }, +}; + +function apiRequest(body?: string) { + return new ManagementRequest("http://127.0.0.1:10100/api/system/restart", { + method: "POST", body, headers: body === undefined ? undefined : { "Content-Type": "application/json" }, + }); +} + +test("empty API requests and empty objects retain the 60-second default", async () => { + for (const payload of [undefined, "{}"]) { + setSystemRestartIoForTests({ isDraining: () => false, schedule: () => {}, setDraining: () => {} }); + const req = apiRequest(payload); + const res = await handleManagementAPI(req, new URL(req.url), apiConfig); + expect(res?.status).toBe(202); + expect((await res!.json()).drainTimeoutMs).toBe(60_000); + } +}); + +test("the API explicitly opts into short grace and repeated requests retain the accepted grace", async () => { + let scheduled = 0; + setSystemRestartIoForTests({ isDraining: () => false, schedule: () => { scheduled += 1; }, setDraining: () => {} }); + for (const payload of ['{"drainGraceMs":2000}', undefined, '{"drainGraceMs":1000}']) { + const req = apiRequest(payload); + const res = await handleManagementAPI(req, new URL(req.url), apiConfig); + expect(res?.status).toBe(202); + expect((await res!.json()).drainTimeoutMs).toBe(2_000); + } + expect(scheduled).toBe(1); +}); + +test("a repeated API opt-in cannot shorten an existing default restart", async () => { + setSystemRestartIoForTests({ isDraining: () => false, schedule: () => {}, setDraining: () => {} }); + for (const payload of [undefined, '{"drainGraceMs":2000}']) { + const req = apiRequest(payload); + const res = await handleManagementAPI(req, new URL(req.url), apiConfig); + expect((await res!.json()).drainTimeoutMs).toBe(60_000); + } +}); + +test("invalid API grace or malformed JSON is rejected without beginning a drain", async () => { + let scheduled = 0; + let drains = 0; + setSystemRestartIoForTests({ isDraining: () => false, schedule: () => { scheduled += 1; }, setDraining: () => { drains += 1; } }); + const invalid = [0, -1, 60_001, 1.5, null, "2000", true]; + for (const payload of [...invalid.map(drainGraceMs => JSON.stringify({ drainGraceMs })), "{", "null", "[]", '"2000"']) { + const req = apiRequest(payload); + const res = await handleManagementAPI(req, new URL(req.url), apiConfig); + expect(res?.status).toBe(400); + } + expect(scheduled).toBe(0); + expect(drains).toBe(0); +}); + +test("the API accepts both supported grace boundaries", async () => { + for (const drainGraceMs of [1, 60_000]) { + setSystemRestartIoForTests({ isDraining: () => false, schedule: () => {}, setDraining: () => {} }); + const req = apiRequest(JSON.stringify({ drainGraceMs })); + const res = await handleManagementAPI(req, new URL(req.url), apiConfig); + expect(res?.status).toBe(202); + expect((await res!.json()).drainTimeoutMs).toBe(drainGraceMs); + } +}); + +test("a target-bound restart capability cannot opt into an unsigned grace option", async () => { + let scheduled = 0; + setSystemRestartIoForTests({ isDraining: () => false, schedule: () => { scheduled += 1; }, setDraining: () => {} }); + const req = apiRequest('{"drainGraceMs":2000}'); + const res = await handleSystemRoutes({ req, url: new URL(req.url), config: apiConfig, deps: {}, version: "fixture", principal: "system-restart-capability" }); + expect(res?.status).toBe(403); + expect(scheduled).toBe(0); + const defaultReq = apiRequest(); + const accepted = await handleSystemRoutes({ req: defaultReq, url: new URL(defaultReq.url), config: apiConfig, deps: {}, version: "fixture", principal: "system-restart-capability" }); + expect(accepted?.status).toBe(202); + expect((await accepted!.json()).drainTimeoutMs).toBe(60_000); + expect(scheduled).toBe(1); +}); + +test("restart spends at most two seconds on active turns, including the response-flush delay", async () => { + let now = 10_000; + let scheduled!: () => void | Promise; + let turnWaitMs = -1; + let cleanupWaitMs = -1; + acceptSystemRestart({ + now: () => now, + isDraining: () => false, + getActiveTurnCount: () => 1, + setDraining: () => {}, + schedule: fn => { scheduled = fn; }, + scheduleDeadline: (_fn, ms) => { cleanupWaitMs = ms; return () => {}; }, + drainAndShutdown: async (_server, ms) => { turnWaitMs = ms; }, + isDesktopSupervised: () => false, + isSupervisedServiceChild: () => false, + listenPort: () => 10123, + stopListener: () => {}, + spawnStart: () => {}, + markRecycling: () => {}, + exitProcess: () => {}, + }, {}, { drainGraceMs: 2_000 }); + now += 200; + await scheduled(); + expect(turnWaitMs).toBe(1_800); + expect(cleanupWaitMs).toBe(59_800); +}); + + +class RestartClock { + now = 10_000; + private timers = new Set<{ at: number; run: () => void }>(); + schedule = (run: () => void, ms: number) => { + const timer = { at: this.now + ms, run }; + this.timers.add(timer); + return () => { this.timers.delete(timer); }; + }; + async settle() { + // Let real shutdown cleanup promises settle without advancing virtual time. + for (let i = 0; i < 40; i += 1) await Promise.resolve(); + } + async advance(ms: number) { + const until = this.now + ms; + while (true) { + const next = [...this.timers].filter(timer => timer.at <= until) + .sort((a, b) => a.at - b.at)[0]; + if (!next) break; + this.now = next.at; + this.timers.delete(next); + next.run(); + await this.settle(); + } + this.now = until; + await this.settle(); + } +} + +const restore: Array<() => void> = []; +afterEach(() => { + for (const reset of restore.splice(0)) reset(); + abortAndReleaseAllTurns(); + resetLifecycleDrainStateForTests(); +}); + +function restartFixture(options: { automatic?: boolean; holdCleanup?: boolean; holdReadiness?: boolean; failSpawn?: boolean; flushDelayMs?: number; useDefaultGrace?: boolean } = {}) { + const clock = new RestartClock(); + const date = spyOn(Date, "now").mockImplementation(() => clock.now); + const sleep = spyOn(Bun, "sleep").mockImplementation((ms) => new Promise(resolve => { + clock.schedule(resolve, typeof ms === "number" ? ms : ms.getTime() - clock.now); + })); + restore.push(() => date.mockRestore(), () => sleep.mockRestore()); + const calls: string[] = []; + let finishCleanup!: () => void; + const cleanup = new Promise(resolve => { finishCleanup = resolve; }); + let finishReadiness!: () => void; + const readiness = new Promise(resolve => { finishReadiness = resolve; }); + let veto!: () => void; + let running: void | Promise; + const server = { + stop: () => { calls.push(`stop:${clock.now}`); }, + } as unknown as ReturnType; + const accept = () => acceptSystemRestart({ + now: () => clock.now, + getActiveTurnCount, + schedule: (fn, ms) => { clock.schedule(() => { running = fn(); }, options.flushDelayMs ?? ms); }, + scheduleDeadline: clock.schedule, + drainAndShutdown: async (_server, ms) => { + calls.push(`drain:${ms}`); + const result = await drainAndShutdown(server, ms); + if (options.holdCleanup) await cleanup; + return result; + }, + stopListener: () => stopServerListener(server), + listenPort: () => 10123, + isDesktopSupervised: () => false, + isSupervisedServiceChild: () => false, + spawnStart: async (port, waitForHealth) => { + calls.push(`spawn:${port}:${waitForHealth}:${clock.now}`); + if (options.failSpawn) throw Object.assign(new Error("fixture"), { code: "EACCES" }); + if (options.holdReadiness) await readiness; + }, + markRecycling: () => { calls.push("recycle"); }, + isClientConnected: () => false, + exitProcess: code => { calls.push(`exit:${code}`); }, + }, options.automatic ? { onAccepted: callback => { veto = callback; } } : {}, options.useDefaultGrace ? {} : { drainGraceMs: 2_000 }); + return { clock, calls, accept, finishCleanup, finishReadiness, veto: () => veto(), running: () => running }; +} + +function upstreamTurn(bindController = true) { + const lease = tryAdmitTurn(); + expect(lease).not.toBeNull(); + const controller = new AbortController(); + if (bindController) registerTurn(controller, lease!); + // The admitted handler may have sent work whose upstream outcome is not yet known. + return { lease: lease!, controller }; +} + +for (const automatic of [false, true]) { + const mode = automatic ? "automatic" : "manual"; + test(`${mode} default restart preserves active work beyond two seconds`, async () => { + const fixture = restartFixture({ automatic, useDefaultGrace: true }); + const turn = upstreamTurn(); + expect(fixture.accept().drainTimeoutMs).toBe(60_000); + await fixture.clock.advance(2_000); + expect(turn.controller.signal.aborted).toBe(false); + expect(getActiveTurnCount()).toBe(1); + turn.lease.release(); + await fixture.clock.advance(100); + await fixture.running(); + expect(turn.controller.signal.aborted).toBe(false); + expect(fixture.calls).toContain("drain:59800"); + }); + test(`${mode} restart preserves a completed turn and hands off without spending the grace`, async () => { + const fixture = restartFixture({ automatic }); + const turn = upstreamTurn(); + turn.lease.release(); + expect(fixture.accept().drainTimeoutMs).toBe(2_000); + expect(tryAdmitTurn()).toBeNull(); + await fixture.clock.advance(200); + await fixture.running(); + expect(fixture.calls).toContain("spawn:10123:true:10200"); + expect(turn.controller.signal.aborted).toBe(false); + }); + + test(`${mode} restart lets a turn finish inside the grace without abort`, async () => { + const fixture = restartFixture({ automatic }); + const turn = upstreamTurn(); + fixture.accept(); + await fixture.clock.advance(700); + expect(turn.controller.signal.aborted).toBe(false); + expect(fixture.calls.some(call => call.startsWith("spawn:"))).toBe(false); + turn.lease.release(); + await fixture.clock.advance(100); + await fixture.running(); + expect(fixture.calls).toContain("spawn:10123:true:10800"); + expect(turn.controller.signal.aborted).toBe(false); + }); + + test(`${mode} restart cuts an ambiguous active turn at grace expiry`, async () => { + const fixture = restartFixture({ automatic }); + const turn = upstreamTurn(); + fixture.accept(); + await fixture.clock.advance(1_999); + expect(turn.controller.signal.aborted).toBe(false); + expect(getActiveTurnCount()).toBe(1); + await fixture.clock.advance(1); + await fixture.running(); + expect(turn.controller.signal.aborted).toBe(true); + expect(getActiveTurnCount()).toBe(0); + expect(fixture.calls).toContain("spawn:10123:true:12000"); + // Successful cleanup/readiness does not reopen admission in the old process. + expect(tryAdmitTurn()).toBeNull(); + }); +} + +test("an admitted turn without a controller is released at the same deadline", async () => { + const fixture = restartFixture(); + upstreamTurn(false); + fixture.accept(); + await fixture.clock.advance(2_000); + await fixture.running(); + expect(getActiveTurnCount()).toBe(0); + expect(fixture.calls).toContain("exit:0"); +}); + +test("client cancellation finishes the drain without waiting out the grace", async () => { + const fixture = restartFixture(); + const turn = upstreamTurn(); + turn.controller.signal.addEventListener("abort", () => turn.lease.release(), { once: true }); + fixture.accept(); + await fixture.clock.advance(300); + turn.controller.abort(new Error("client cancelled")); + await fixture.clock.advance(100); + await fixture.running(); + expect(fixture.calls).toContain("spawn:10123:true:10400"); +}); + +test("held cleanup retains the original 60s watchdog and ignores late completion", async () => { + const fixture = restartFixture({ holdCleanup: true }); + fixture.accept(); + await fixture.clock.advance(2_000); + expect(fixture.calls.some(call => call.startsWith("spawn:"))).toBe(false); + expect(isDraining()).toBe(true); + await fixture.clock.advance(58_000); + await fixture.running(); + expect(fixture.calls).toContain("spawn:10123:false:70000"); + fixture.finishCleanup(); + await fixture.clock.settle(); + expect(fixture.calls.filter(call => call.startsWith("spawn:"))).toHaveLength(1); + expect(fixture.calls.filter(call => call.startsWith("exit:"))).toEqual(["exit:0"]); +}); + +test("delayed replacement readiness does not reopen the old process while a cut turn stays aborted", async () => { + const fixture = restartFixture({ holdReadiness: true }); + const turn = upstreamTurn(); + fixture.accept(); + await fixture.clock.advance(2_000); + expect(fixture.calls).toContain("spawn:10123:true:12000"); + await fixture.clock.advance(65_000); + expect(fixture.calls.some(call => call.startsWith("exit:"))).toBe(false); + expect(tryAdmitTurn()).toBeNull(); + expect(turn.controller.signal.aborted).toBe(true); + fixture.finishReadiness(); + await fixture.running(); + expect(fixture.calls).toContain("exit:0"); +}); + +test("automatic pending cancellation releases only its admission fence", async () => { + const fixture = restartFixture({ automatic: true }); + fixture.accept(); + expect(isDraining()).toBe(true); + fixture.veto(); + await fixture.clock.advance(2_000); + expect(fixture.calls).toEqual([]); + const admitted = tryAdmitTurn(); + expect(admitted).not.toBeNull(); + admitted?.release(); +}); + +for (const automatic of [false, true]) { + test(`explicit stop during ${automatic ? "automatic" : "manual"} grace preserves restart ownership`, async () => { + const fixture = restartFixture({ automatic }); + upstreamTurn(); + fixture.accept(); + await fixture.clock.advance(200); + noteExplicitShutdownRequested(); + await fixture.clock.advance(1_800); + await fixture.running(); + expect(fixture.calls.filter(call => call.startsWith("spawn:"))).toHaveLength(automatic ? 0 : 1); + expect(isDraining()).toBe(true); + }); +} + +test("spawn failure after a cut turn exits once without recycling", async () => { + const fixture = restartFixture({ failSpawn: true }); + upstreamTurn(); + fixture.accept(); + await fixture.clock.advance(2_000); + await fixture.running(); + expect(fixture.calls.filter(call => call.startsWith("exit:"))).toEqual(["exit:1"]); + expect(fixture.calls).not.toContain("recycle"); +}); + +test("waiting to enter the drain microtask does not renew the acceptance-based grace", async () => { + let now = 10_000; + let scheduled!: () => void | Promise; + let turnWaitMs = -1; + acceptSystemRestart({ + now: () => now, + isDraining: () => false, + getActiveTurnCount: () => 1, + setDraining: () => {}, + schedule: fn => { scheduled = fn; }, + scheduleDeadline: () => () => {}, + drainAndShutdown: async (_server, ms) => { turnWaitMs = ms; }, + isDesktopSupervised: () => false, + isSupervisedServiceChild: () => false, + listenPort: () => 10123, + stopListener: () => {}, + spawnStart: () => {}, + markRecycling: () => {}, + exitProcess: () => {}, + }, {}, { drainGraceMs: 2_000 }); + now += 200; + const running = scheduled(); + now += 350; + await running; + expect(turnWaitMs).toBe(1_450); +}); + +test("a non-round response-flush delay does not extend grace by a full lifecycle poll", async () => { + const fixture = restartFixture({ flushDelayMs: 275 }); + const turn = upstreamTurn(); + fixture.accept(); + await fixture.clock.advance(1_999); + expect(turn.controller.signal.aborted).toBe(false); + await fixture.clock.advance(1); + await fixture.running(); + expect(turn.controller.signal.aborted).toBe(true); + expect(fixture.calls).toContain("drain:1725"); + expect(fixture.calls).toContain("spawn:10123:true:12000"); +}); + +test("an existing scoped drain and active turn share the same grace", async () => { + const fixture = restartFixture(); + const turn = upstreamTurn(); + const scoped = acquireTemporaryDrain("fixture-scoped-drain"); + expect(scoped).not.toBeNull(); + fixture.accept(); + await fixture.clock.advance(500); + expect(fixture.calls).toEqual(["drain:1800"]); + scoped!.release(); + await fixture.clock.settle(); + await fixture.clock.advance(1_499); + expect(turn.controller.signal.aborted).toBe(false); + await fixture.clock.advance(1); + await fixture.running(); + expect(turn.controller.signal.aborted).toBe(true); + expect(fixture.calls).toContain("spawn:10123:true:12000"); +});