diff --git a/docs-site/src/content/docs/fr/guides/remote-link.md b/docs-site/src/content/docs/fr/guides/remote-link.md index 52f45296021..ed47bd29ae4 100644 --- a/docs-site/src/content/docs/fr/guides/remote-link.md +++ b/docs-site/src/content/docs/fr/guides/remote-link.md @@ -55,7 +55,7 @@ Le rôle **Child** exige également qu’OpenCodex fonctionne en mode autonome s - **Connected** signifie que le tunnel SSH est prêt et que Child peut utiliser la liaison Home. - **Reconnecting** signifie que le tunnel est réessayé. Les requêtes peuvent temporairement renvoyer `503` avec `Retry-After`. Sur un Child connecté depuis son propre tableau de bord, une requête attend d’abord jusqu’à 15 secondes le retour du tunnel. -- **Failed** signifie que la liaison nécessite une intervention. Vérifiez l’authentification SSH, la clé d’hôte confirmée, la redirection ou le délai indiqué. Un Child connecté depuis son propre tableau de bord réessaie de lui-même après une mise en veille, une panne ou un redémarrage : environ une fois par minute après un délai dépassé ou une erreur de redirection, et toutes les cinq minutes après une erreur d’authentification. Une clé d’hôte modifiée n’est jamais réessayée. +- **Failed** signifie que la liaison nécessite une intervention. Vérifiez l’authentification SSH, la clé d’hôte confirmée, la redirection ou le délai indiqué. Le côté qui ouvre le tunnel SSH réessaie de lui-même après une mise en veille, une panne ou un redémarrage (Home pour un Child qu’il a ajouté, le Child pour une liaison qu’il a rejointe depuis son propre tableau de bord) : environ une fois par minute après un délai dépassé ou une erreur de redirection, et toutes les cinq minutes après une erreur d’authentification. Une clé d’hôte modifiée n’est jamais réessayée. Pour réessayer tout de suite, choisissez **Retry** dans le tableau de bord du Home ou exécutez `ocx link reconnect --link-id ` sur le Home. Le tunnel de cette liaison redémarre sans redémarrer OpenCodex. Une liaison en échec ne bascule pas silencieusement vers un fournisseur local. @@ -82,6 +82,7 @@ ocx link port [--json] ocx link issue --alias --tunnel-port [--json] ocx link status [--json] ocx link revoke --link-id [--json] +ocx link reconnect --link-id [--json] ``` ## Guides associés diff --git a/docs-site/src/content/docs/guides/remote-link.md b/docs-site/src/content/docs/guides/remote-link.md index 0d40a64cd66..e4788e1c3a4 100644 --- a/docs-site/src/content/docs/guides/remote-link.md +++ b/docs-site/src/content/docs/guides/remote-link.md @@ -57,7 +57,7 @@ The **Child** role also requires a standalone OpenCodex runtime running on its c - **Connected** means the SSH tunnel is ready and the Child can use the Home link. - **Reconnecting** means the tunnel is being retried. Requests can temporarily return `503` with `Retry-After` while the retry is in progress. On a Child that connected from its own dashboard, a request first waits up to 15 seconds for the tunnel to come back. -- **Failed** means the link needs attention. Check SSH authentication, the confirmed host key, forwarding, or the timeout reason shown in the dashboard. A Child that connected from its own dashboard keeps retrying by itself, after sleep, an outage or a restart: about once a minute after a timeout or forwarding error, and every five minutes after an authentication error. A changed host key is never retried. +- **Failed** means the link needs attention. Check SSH authentication, the confirmed host key, forwarding, or the timeout reason shown in the dashboard. The side that runs the SSH tunnel keeps retrying by itself, after sleep, an outage or a restart: Home for a Child it added, and the Child for a link it joined from its own dashboard. It retries about once a minute after a timeout or forwarding error, and every five minutes after an authentication error. A changed host key is never retried. To try again at once, select **Retry** in the Home's dashboard or run `ocx link reconnect --link-id ` on the Home. It restarts that link's tunnel without restarting OpenCodex. A failed link does not silently switch to a local provider. The Child returns a retryable `503` without forwarding the link key or request when its tunnel is failed, stopped, or not supervised. @@ -95,6 +95,7 @@ ocx link port [--json] ocx link issue --alias --tunnel-port [--json] ocx link status [--json] ocx link revoke --link-id [--json] +ocx link reconnect --link-id [--json] ``` ## Related guides diff --git a/docs-site/src/content/docs/ja/guides/remote-link.md b/docs-site/src/content/docs/ja/guides/remote-link.md index 770e9a0ec5c..504ebce54c1 100644 --- a/docs-site/src/content/docs/ja/guides/remote-link.md +++ b/docs-site/src/content/docs/ja/guides/remote-link.md @@ -55,7 +55,7 @@ Home のプロバイダーを使うコンピューターで次の操作を行い - **Connected** は SSH トンネルが準備でき、Child が Home のリンクを使える状態です。 - **Reconnecting** はトンネルを再試行している状態です。再試行中はリクエストが `Retry-After` 付きの `503` を一時的に返すことがあります。自分のダッシュボードから接続した Child では、リクエストはまずトンネルの復帰を最大 15 秒待ちます。 -- **Failed** は対応が必要な状態です。SSH 認証、確認済みのホストキー、転送、タイムアウトの理由を確認してください。自分のダッシュボードから接続した Child は、スリープ、障害、再起動の後も自動で再試行します。タイムアウトや転送エラーの後は約 1 分ごと、認証エラーの後は 5 分ごとです。ホストキーが変わった場合は再試行しません。 +- **Failed** は対応が必要な状態です。SSH 認証、確認済みのホストキー、転送、タイムアウトの理由を確認してください。SSH トンネルを張る側が、スリープ、障害、再起動の後も自動で再試行します。Home が追加した Child では Home が、自分のダッシュボードから接続した Child では Child が再試行します。タイムアウトや転送エラーの後は約 1 分ごと、認証エラーの後は 5 分ごとです。ホストキーが変わった場合は再試行しません。すぐに再接続するには、Home のダッシュボードで **Retry** を選ぶか、Home で `ocx link reconnect --link-id ` を実行します。OpenCodex を再起動せずに、そのリンクのトンネルだけを張り直します。 リンクが失敗しても、ローカルプロバイダーへ自動的に切り替わることはありません。 @@ -82,6 +82,7 @@ ocx link port [--json] ocx link issue --alias --tunnel-port [--json] ocx link status [--json] ocx link revoke --link-id [--json] +ocx link reconnect --link-id [--json] ``` ## 関連ガイド diff --git a/docs-site/src/content/docs/ko/guides/remote-link.md b/docs-site/src/content/docs/ko/guides/remote-link.md index f22fad49030..6bcaba82a9a 100644 --- a/docs-site/src/content/docs/ko/guides/remote-link.md +++ b/docs-site/src/content/docs/ko/guides/remote-link.md @@ -55,7 +55,7 @@ Home의 프로바이더를 사용할 컴퓨터에서 다음을 진행합니다. - **Connected**는 SSH 터널이 준비되어 Child가 Home 링크를 사용할 수 있다는 뜻입니다. - **Reconnecting**은 터널을 다시 연결하는 중이라는 뜻입니다. 재시도 중에는 요청이 일시적으로 `Retry-After`와 함께 `503`을 반환할 수 있습니다. 자기 대시보드에서 연결한 Child에서는 요청이 먼저 터널이 돌아오기를 최대 15초 기다립니다. -- **Failed**는 조치가 필요하다는 뜻입니다. SSH 인증, 확인한 호스트 키, 포워딩 또는 타임아웃 사유를 확인하세요. 자기 대시보드에서 연결한 Child는 절전, 장애, 재시작 뒤에도 스스로 다시 시도합니다. 타임아웃이나 포워딩 오류 뒤에는 약 1분마다, 인증 오류 뒤에는 5분마다 시도합니다. 호스트 키가 바뀐 경우에는 다시 시도하지 않습니다. +- **Failed**는 조치가 필요하다는 뜻입니다. SSH 인증, 확인한 호스트 키, 포워딩 또는 타임아웃 사유를 확인하세요. SSH 터널을 띄운 쪽이 절전, 장애, 재시작 뒤에도 스스로 다시 시도합니다. Home이 추가한 Child라면 Home이, 자기 대시보드에서 연결한 Child라면 Child가 시도합니다. 타임아웃이나 포워딩 오류 뒤에는 약 1분마다, 인증 오류 뒤에는 5분마다 시도합니다. 호스트 키가 바뀐 경우에는 다시 시도하지 않습니다. 바로 다시 연결하려면 Home 대시보드에서 **Retry**를 누르거나 Home에서 `ocx link reconnect --link-id `를 실행하세요. OpenCodex를 재시작하지 않고 그 링크의 터널만 다시 띄웁니다. 링크가 실패해도 로컬 프로바이더로 조용히 전환하지 않습니다. @@ -82,6 +82,7 @@ ocx link port [--json] ocx link issue --alias --tunnel-port [--json] ocx link status [--json] ocx link revoke --link-id [--json] +ocx link reconnect --link-id [--json] ``` ## 관련 가이드 diff --git a/docs-site/src/content/docs/ru/guides/remote-link.md b/docs-site/src/content/docs/ru/guides/remote-link.md index b88687261d5..1565d8f0443 100644 --- a/docs-site/src/content/docs/ru/guides/remote-link.md +++ b/docs-site/src/content/docs/ru/guides/remote-link.md @@ -55,7 +55,7 @@ Host devbox - **Connected** означает, что SSH-туннель готов и Child может использовать связь Home. - **Reconnecting** означает, что туннель переподключается. Во время повторных попыток запросы могут временно получать `503` с `Retry-After`. На Child, подключённом из собственной панели, запрос сначала до 15 секунд ждёт восстановления туннеля. -- **Failed** означает, что связь требует действий. Проверьте SSH-аутентификацию, подтверждённый ключ хоста, перенаправление или причину тайм-аута. Child, подключённый из собственной панели, сам повторяет попытки после сна, сбоя или перезапуска: примерно раз в минуту после тайм-аута или ошибки перенаправления и каждые пять минут после ошибки аутентификации. Изменённый ключ хоста никогда не повторяется. +- **Failed** означает, что связь требует действий. Проверьте SSH-аутентификацию, подтверждённый ключ хоста, перенаправление или причину тайм-аута. Сторона, которая держит SSH-туннель, сама повторяет попытки после сна, сбоя или перезапуска (Home — для добавленного им Child, Child — для связи, к которой он подключился из собственной панели): примерно раз в минуту после тайм-аута или ошибки перенаправления и каждые пять минут после ошибки аутентификации. Изменённый ключ хоста никогда не повторяется. Чтобы переподключиться сразу, нажмите **Retry** в панели Home или выполните `ocx link reconnect --link-id ` на Home. Туннель этой связи перезапустится без перезапуска OpenCodex. При сбое связи система молча не переключается на локального провайдера. @@ -82,6 +82,7 @@ ocx link port [--json] ocx link issue --alias --tunnel-port [--json] ocx link status [--json] ocx link revoke --link-id [--json] +ocx link reconnect --link-id [--json] ``` ## Связанные руководства diff --git a/docs-site/src/content/docs/tr/guides/remote-link.md b/docs-site/src/content/docs/tr/guides/remote-link.md index 9a866de7bd4..df3f77c0679 100644 --- a/docs-site/src/content/docs/tr/guides/remote-link.md +++ b/docs-site/src/content/docs/tr/guides/remote-link.md @@ -55,7 +55,7 @@ Bağlanmak bu bilgisayardaki OpenCodex'i yeniden başlatır. Zaten çalışan Co - **Connected**, SSH tünelinin hazır ve Child'ın Home bağlantısını kullanabilir olduğu anlamına gelir. - **Reconnecting**, tünelin yeniden denendiği anlamına gelir. Yeniden deneme sırasında istekler geçici olarak `Retry-After` ile birlikte `503` döndürebilir. Kendi panosundan bağlanan bir Child üzerinde istek önce tünelin geri gelmesi için en fazla 15 saniye bekler. -- **Failed**, bağlantının ilgilenilmesi gerektiği anlamına gelir. SSH kimlik doğrulamasını, onaylanan ana bilgisayar anahtarını, yönlendirmeyi veya zaman aşımı nedenini kontrol edin. Kendi panosundan bağlanan bir Child; uyku, kesinti veya yeniden başlatmadan sonra kendiliğinden yeniden dener: zaman aşımı veya yönlendirme hatasından sonra yaklaşık dakikada bir, kimlik doğrulama hatasından sonra beş dakikada bir. Değişmiş bir ana bilgisayar anahtarı asla yeniden denenmez. +- **Failed**, bağlantının ilgilenilmesi gerektiği anlamına gelir. SSH kimlik doğrulamasını, onaylanan ana bilgisayar anahtarını, yönlendirmeyi veya zaman aşımı nedenini kontrol edin. SSH tünelini açan taraf (eklediği Child için Home, kendi panosundan katıldığı bağlantı için Child); uyku, kesinti veya yeniden başlatmadan sonra kendiliğinden yeniden dener: zaman aşımı veya yönlendirme hatasından sonra yaklaşık dakikada bir, kimlik doğrulama hatasından sonra beş dakikada bir. Değişmiş bir ana bilgisayar anahtarı asla yeniden denenmez. Hemen yeniden denemek için Home panosunda **Retry** düğmesini seçin veya Home üzerinde `ocx link reconnect --link-id ` komutunu çalıştırın. Bu, OpenCodex'i yeniden başlatmadan yalnızca o bağlantının tünelini yeniden başlatır. Bağlantı başarısız olduğunda sistem sessizce yerel bir sağlayıcıya geçmez. @@ -82,6 +82,7 @@ ocx link port [--json] ocx link issue --alias --tunnel-port [--json] ocx link status [--json] ocx link revoke --link-id [--json] +ocx link reconnect --link-id [--json] ``` ## İlgili kılavuzlar diff --git a/docs-site/src/content/docs/zh-cn/guides/remote-link.md b/docs-site/src/content/docs/zh-cn/guides/remote-link.md index a59e2a57fb1..ce10f3d5259 100644 --- a/docs-site/src/content/docs/zh-cn/guides/remote-link.md +++ b/docs-site/src/content/docs/zh-cn/guides/remote-link.md @@ -55,7 +55,7 @@ Host devbox - **Connected** 表示 SSH 隧道已就绪，子机可以使用主机链接。 - **Reconnecting** 表示正在重试隧道。重试期间请求可能暂时返回带有 `Retry-After` 的 `503`。在从自己的仪表板连接的 Child 上，请求会先最多等待 15 秒让隧道恢复。 -- **Failed** 表示链接需要处理。请检查 SSH 身份验证、已确认的主机密钥、转发或超时原因。从自己的仪表板连接的 Child 会在睡眠、故障或重启后自动重试：超时或转发错误后大约每分钟一次，身份验证错误后每五分钟一次。主机密钥变更时不会重试。 +- **Failed** 表示链接需要处理。请检查 SSH 身份验证、已确认的主机密钥、转发或超时原因。建立 SSH 隧道的一方会在睡眠、故障或重启后自动重试（Home 添加的 Child 由 Home 重试，从自己的仪表板连接的 Child 由 Child 重试）：超时或转发错误后大约每分钟一次，身份验证错误后每五分钟一次。主机密钥变更时不会重试。要立即重连，请在 Home 的仪表板中选择 **Retry**，或在 Home 上运行 `ocx link reconnect --link-id `。这只会重启该链接的隧道，不会重启 OpenCodex。 链接失败时不会静默切换到本地提供商。 @@ -82,6 +82,7 @@ ocx link port [--json] ocx link issue --alias --tunnel-port [--json] ocx link status [--json] ocx link revoke --link-id [--json] +ocx link reconnect --link-id [--json] ``` ## 相关指南 diff --git a/docs-site/src/content/docs/zh-tw/guides/remote-link.md b/docs-site/src/content/docs/zh-tw/guides/remote-link.md index 87233145b80..406c8f53f47 100644 --- a/docs-site/src/content/docs/zh-tw/guides/remote-link.md +++ b/docs-site/src/content/docs/zh-tw/guides/remote-link.md @@ -55,7 +55,7 @@ Host devbox - **Connected** 表示 SSH 通道已準備好，Child 可以使用 Home 連結。 - **Reconnecting** 表示正在重試通道。重試期間請求可能暫時回傳帶有 `Retry-After` 的 `503`。在從自己的儀表板連線的 Child 上，請求會先最多等待 15 秒讓通道恢復。 -- **Failed** 表示連結需要處理。請檢查 SSH 驗證、已確認的主機金鑰、轉送或逾時原因。從自己的儀表板連線的 Child 會在睡眠、故障或重新啟動後自動重試：逾時或轉送錯誤後大約每分鐘一次，驗證錯誤後每五分鐘一次。主機金鑰變更時不會重試。 +- **Failed** 表示連結需要處理。請檢查 SSH 驗證、已確認的主機金鑰、轉送或逾時原因。建立 SSH 通道的一方會在睡眠、故障或重新啟動後自動重試（Home 新增的 Child 由 Home 重試，從自己的儀表板連線的 Child 由 Child 重試）：逾時或轉送錯誤後大約每分鐘一次，驗證錯誤後每五分鐘一次。主機金鑰變更時不會重試。要立即重新連線，請在 Home 的儀表板中選擇 **Retry**，或在 Home 上執行 `ocx link reconnect --link-id `。這只會重新啟動該連結的通道，不會重新啟動 OpenCodex。 連結失敗時不會靜默切換到本機供應商。 @@ -82,6 +82,7 @@ ocx link port [--json] ocx link issue --alias --tunnel-port [--json] ocx link status [--json] ocx link revoke --link-id [--json] +ocx link reconnect --link-id [--json] ``` ## 相關指南 diff --git a/gui/src/pages/RemoteLink.tsx b/gui/src/pages/RemoteLink.tsx index c29708156b1..a07c4194b7c 100644 --- a/gui/src/pages/RemoteLink.tsx +++ b/gui/src/pages/RemoteLink.tsx @@ -61,6 +61,8 @@ const ERROR_TKEY: Record = { link_apply_failed: "remoteLink.error.link_apply_failed", link_exists: "remoteLink.error.link_exists", link_not_found: "remoteLink.error.link_not_found", + // Only `ocx link reconnect` can reach it: Retry restarts Home-initiated rows only. + link_not_home_initiated: "remoteLink.error.generic", link_remove_failed: "remoteLink.error.link_remove_failed", link_unavailable: "remoteLink.error.link_unavailable", listener_unavailable: "remoteLink.error.listener_unavailable", @@ -140,7 +142,7 @@ export default function RemoteLink({ apiBase, sessionReady, workspaceAvailable = const [probe, setProbe] = useState(null); const [confirmation, setConfirmation] = useState(null); const [checkedFingerprint, setCheckedFingerprint] = useState(false); - const [busy, setBusy] = useState<"candidates" | "probe" | "confirm" | "apply" | "join" | "remove" | null>(null); + const [busy, setBusy] = useState<"candidates" | "probe" | "confirm" | "apply" | "join" | "remove" | "reconnect" | null>(null); const [actionError, setActionError] = useState(null); const [failedAction, setFailedAction] = useState(null); const [confirming, setConfirming] = useState<{ row: LinkRowWire; force: boolean } | null>(null); @@ -356,8 +358,24 @@ export default function RemoteLink({ apiBase, sessionReady, workspaceAvailable = } }; + // Retry on a failed or reconnecting Home link restarts its tunnel now instead of only re-reading + // status, which never changed a link the supervisor had given up on. + const reconnectLinks = async () => { + const rows = (status?.links ?? []).filter(row => row.direction === "hub-initiated" && (row.state === "failed" || row.state === "reconnecting")); + if (rows.length === 0) { await refreshStatus(); return; } + setBusy("reconnect"); setActionError(null); + try { + for (const row of rows) await requestLinkJson<{ linkId: string }>(apiBase, `/api/link/${encodeURIComponent(row.id)}/reconnect`, { method: "POST" }); + } catch (error) { + setActionError(linkActionError(error)); + } finally { + setBusy(null); + await refreshStatus(); + } + }; + const retryFailedAction = () => { - if (!failedAction) { void refreshStatus(); return; } + if (!failedAction) { void reconnectLinks(); return; } const attempt = startLinkAttempt(); if (failedAction.phase === "probe") { setAlias(failedAction.alias); void runProbe(failedAction.alias, attempt); return; } if (failedAction.phase === "join") { void joinLink(failedAction.alias, attempt); return; } @@ -408,7 +426,7 @@ export default function RemoteLink({ apiBase, sessionReady, workspaceAvailable = {uiState === "role-select" && canPairLocally && status?.joinAvailable === false && status.joinDenied === "pairing_required" && ( { void refreshStatus(); }} /> )} - {(uiState === "connected" || uiState === "reconnecting" || uiState === "failed" || uiState === "restart-waiting" || status?.role === "child" || statusRows.length > 0 || uiState === "adding-child" || uiState === "confirming-host" || uiState === "applying" || uiState === "joining") &&

{role === "child" && standaloneRuntime ? t("remoteLink.findHome.title") : t("link.children")}

{uiState === "restart-waiting" ? t("remoteLink.restart.waiting") : t(roleLabel)}

{uiState === "restart-waiting" ?
{t("remoteLink.restart.title")}

{t("remoteLink.restart.body")}

{restartSlow && {t("remoteLink.restart.slow")}}
: status?.role === "child" ?
{status.child?.alias ?? t("remoteLink.role.child")}{status.child &&
{t(STATUS_LABEL[status.child.state])}
}
: statusRows.length > 0 ?
{statusRows.map(row =>
{row.alias}
{t(STATUS_LABEL[row.state])}{row.direction === "hub-initiated" ? t("remoteLink.direction.hub") : t("remoteLink.direction.client")}{row.reason && {row.reason in REASON_TKEY ? t(REASON_TKEY[row.reason]) : <>{t("remoteLink.reason.generic")} {row.reason}}}
)}
:

{t(role === "child" && standaloneRuntime ? "remoteLink.findHome.empty" : "link.noChildren")}

}{(!sheetOpen && (uiState === "reconnecting" || (uiState === "failed" && ((failedAction !== null && actionError?.key !== "remoteLink.error.join_restart_failed") || statusRows.some(row => row.state === "failed"))))) &&
{t(STATUS_LABEL[uiState === "failed" ? "failed" : "reconnecting"])}
}{uiState === "joining" &&

{t("remoteLink.joining")}

}{!sheetOpen && actionError && }
} + {(uiState === "connected" || uiState === "reconnecting" || uiState === "failed" || uiState === "restart-waiting" || status?.role === "child" || statusRows.length > 0 || uiState === "adding-child" || uiState === "confirming-host" || uiState === "applying" || uiState === "joining") &&

{role === "child" && standaloneRuntime ? t("remoteLink.findHome.title") : t("link.children")}

{uiState === "restart-waiting" ? t("remoteLink.restart.waiting") : t(roleLabel)}

{uiState === "restart-waiting" ?
{t("remoteLink.restart.title")}

{t("remoteLink.restart.body")}

{restartSlow && {t("remoteLink.restart.slow")}}
: status?.role === "child" ?
{status.child?.alias ?? t("remoteLink.role.child")}{status.child &&
{t(STATUS_LABEL[status.child.state])}
}
: statusRows.length > 0 ?
{statusRows.map(row =>
{row.alias}
{t(STATUS_LABEL[row.state])}{row.direction === "hub-initiated" ? t("remoteLink.direction.hub") : t("remoteLink.direction.client")}{row.reason && {row.reason in REASON_TKEY ? t(REASON_TKEY[row.reason]) : <>{t("remoteLink.reason.generic")} {row.reason}}}
)}
:

{t(role === "child" && standaloneRuntime ? "remoteLink.findHome.empty" : "link.noChildren")}

}{(!sheetOpen && (uiState === "reconnecting" || (uiState === "failed" && ((failedAction !== null && actionError?.key !== "remoteLink.error.join_restart_failed") || statusRows.some(row => row.state === "failed"))))) &&
{t(STATUS_LABEL[uiState === "failed" ? "failed" : "reconnecting"])}
}{uiState === "joining" &&

{t("remoteLink.joining")}

}{!sheetOpen && actionError && }
} { event.preventDefault(); closeSheet(); }}>
diff --git a/gui/src/remote-link-api.ts b/gui/src/remote-link-api.ts index 9dec2e1cea6..f61911555ea 100644 --- a/gui/src/remote-link-api.ts +++ b/gui/src/remote-link-api.ts @@ -23,6 +23,7 @@ export const LINK_ERROR_CODES = [ "link_apply_failed", "link_exists", "link_not_found", + "link_not_home_initiated", "link_remove_failed", "link_unavailable", "listener_unavailable", diff --git a/gui/tests/remote-link.test.tsx b/gui/tests/remote-link.test.tsx index b422360d4d7..c5054bb860a 100644 --- a/gui/tests/remote-link.test.tsx +++ b/gui/tests/remote-link.test.tsx @@ -535,6 +535,28 @@ test("disconnect failure opens force confirmation and sends force body", async ( expect(calls.some(call => call.method === "DELETE" && call.body === JSON.stringify({ force: true }))).toBe(true); }); +test("Retry on a failed Home link restarts its tunnel instead of only re-reading status", async () => { + const calls: Array<{ method: string; path: string }> = []; + let state: "failed" | "connecting" = "failed"; + globalThis.fetch = (async (input, init) => { + const path = new URL(String(input)).pathname; + calls.push({ method: init?.method ?? "GET", path }); + if (path === "/api/link/status") return response({ ...baseStatus, links: [ + { id: "link-1", alias: "child", direction: "hub-initiated", state, since: "now", reason: state === "failed" ? "timeout" : null, tunnelPort: 43110 }, + { id: "link-2", alias: "home", direction: "client-initiated", state: "failed", since: "now", reason: "timeout", tunnelPort: 43111 }, + ] }); + if (path === "/api/link/link-1/reconnect" && init?.method === "POST") { state = "connecting"; return response({ linkId: "link-1" }); } + return response({ error: { code: "link_not_home_initiated" } }, 409); + }) as typeof fetch; + const host = await mount(); + const retry = [...host.querySelectorAll("button")].find(button => button.textContent === "Retry") as HTMLButtonElement; + await act(async () => { retry.click(); }); + await settleUntil(() => host.querySelector(".remote-link-status[data-state='connecting']") !== null); + expect(calls.filter(call => call.method === "POST").map(call => call.path)).toEqual(["/api/link/link-1/reconnect"]); + expect(calls.at(-1)).toEqual({ method: "GET", path: "/api/link/status" }); + expect(host.querySelector(".remote-link-status[data-state='connecting']")).not.toBeNull(); +}); + test("readLinkJson preserves unknown server codes and status", async () => { let caught: unknown; try { await readLinkJson(new Response(JSON.stringify({ error: { code: "future_code" } }), { status: 418 })); } catch (error) { caught = error; } diff --git a/skills/ocx/references/01_management_surface.md b/skills/ocx/references/01_management_surface.md index a7ee4e51093..e861be048dd 100644 --- a/skills/ocx/references/01_management_surface.md +++ b/skills/ocx/references/01_management_surface.md @@ -31,7 +31,7 @@ These answer in the CLI head and never reach the proxy, so they work with nothin | [agents-routing](01_surface_agents-routing.md) | 50 | | [integrations](01_surface_integrations.md) | 41 | | [observe-system](01_surface_observe-system.md) | 92 | -| [access-remote](01_surface_access-remote.md) | 28 | +| [access-remote](01_surface_access-remote.md) | 29 | | [lab](01_surface_lab.md) | 21 | ## Read-only capabilities @@ -67,6 +67,10 @@ Original invocation order. These headings preserve links to the previous single- [State-changing task](01_surface_access-remote.md#ocx-link-revoke) +### `ocx link reconnect` + +[State-changing task](01_surface_access-remote.md#ocx-link-reconnect) + ### `ocx remote-workspace pair` [State-changing task](01_surface_access-remote.md#ocx-remote-workspace-pair) @@ -1373,6 +1377,6 @@ Original invocation order. These headings preserve links to the previous single- ## Counts -- declared capabilities: 331 -- of those, state-changing: 200 +- declared capabilities: 332 +- of those, state-changing: 201 - head-resolved invocations: 2 diff --git a/skills/ocx/references/01_surface_access-remote.md b/skills/ocx/references/01_surface_access-remote.md index 19484c38e3c..9dc6ac83595 100644 --- a/skills/ocx/references/01_surface_access-remote.md +++ b/skills/ocx/references/01_surface_access-remote.md @@ -8,7 +8,7 @@ Use these declarations to choose a task, then check its flags and authority before execution. Non-mutating probes may still contact providers, consume quota or refresh caches. -Declared capabilities: 28. +Declared capabilities: 29. ### `ocx link port` @@ -97,6 +97,29 @@ JSON mode: `envelope`. - Ordinary revoke retains its no-body and idempotent behavior. Forced success reports remoteCleanup:skipped from invocation intent; already-missing link reports unverified. Neither proves the remote client disconnected. - If forced cleanup was needed, run ocx disconnect on the remote client. No new enrollment, SSH trust or credential action is implied. +### `ocx link reconnect` + +Usage: `ocx link reconnect --link-id [--json]` + +Restart a Home-initiated link's SSH tunnel now. + +State-changing: yes. + +| Method | Route | +|---|---| +| POST | `/api/link/{id}/reconnect` | + +| Flag | Value | Meaning | +|---|---|---| +| `--link-id` | string | Home-initiated link id whose tunnel to restart. | +| `--json` | boolean | Emit the restarted link id as JSON. | + +JSON mode: `envelope`. + +- Requires a valid link ID and loopback admin authority. Stops the current ssh child, clears any backoff or failed state and starts a fresh attempt; read ocx link status for the outcome. +- A Child-initiated link is refused with link_not_home_initiated: its own computer owns and retries that tunnel. +- Always emits {linkId} JSON. + ### `ocx remote-workspace pair` Usage: `ocx remote-workspace pair --pairing-code-stdin --root [--root ...] [--toolchain-root ...] [--executor-helper ] [--name ] [--json]` diff --git a/src/cli/capabilities-base.ts b/src/cli/capabilities-base.ts index 6a03564625c..15e07b87ad9 100644 --- a/src/cli/capabilities-base.ts +++ b/src/cli/capabilities-base.ts @@ -65,6 +65,16 @@ export const CAPABILITIES: readonly Capability[] = [ details: ["Requires a valid link ID and loopback admin authority. Explicit force additionally requires --yes.","Ordinary revoke always emits {linkId} JSON; forced revoke has human output or a JSON cleanup receipt. Already-revoked link_not_found is idempotent success.","Ordinary revoke retains its no-body and idempotent behavior. Forced success reports remoteCleanup:skipped from invocation intent; already-missing link reports unverified. Neither proves the remote client disconnected.","If forced cleanup was needed, run ocx disconnect on the remote client. No new enrollment, SSH trust or credential action is implied."], usage: "ocx link revoke --link-id [--force --yes] [--json]", }, + { + command: ["link","reconnect"], + summary: "Restart a Home-initiated link's SSH tunnel now.", + routes: [{"method":"POST","path":"/api/link/{id}/reconnect"}], + flags: [{"name":"--link-id","value":"string","required":true,"summary":"Home-initiated link id whose tunnel to restart."},{"name":"--json","value":"boolean","summary":"Emit the restarted link id as JSON."}], + mutates: true, + json: "envelope", + usage: "ocx link reconnect --link-id [--json]", + details: ["Requires a valid link ID and loopback admin authority. Stops the current ssh child, clears any backoff or failed state and starts a fresh attempt; read ocx link status for the outcome.","A Child-initiated link is refused with link_not_home_initiated: its own computer owns and retries that tunnel.","Always emits {linkId} JSON."], + }, { command: ["remote-workspace","pair"], summary: "Enroll this executor with one Hub using a one-time code from stdin and locally approved roots.", diff --git a/src/cli/link.ts b/src/cli/link.ts index 4b4894a74b8..653e5a44377 100644 --- a/src/cli/link.ts +++ b/src/cli/link.ts @@ -23,6 +23,7 @@ export const LINK_USAGE = `Usage: ocx link issue --alias --tunnel-port [--json] ocx link status [--json] ocx link revoke --link-id [--force --yes] [--json] + ocx link reconnect --link-id [--json] Forced revoke skips remote cleanup. Run ocx disconnect on the remote client.`; @@ -303,6 +304,18 @@ async function runRevoke(args: string[], deps: LinkCliDeps): Promise { console.log(JSON.stringify({ linkId })); } +async function runReconnect(args: string[], deps: LinkCliDeps): Promise { + takeJsonFlag(args); + const linkId = takeOption(args, "--link-id"); + if (!linkId || !LINK_ID.test(linkId)) throw new CliUsageError("reconnect requires a valid --link-id", LINK_USAGE); + rejectArgs(args, LINK_USAGE); + const response = await linkRequest(`/api/link/${encodeURIComponent(linkId)}/reconnect`, { method: "POST" }, deps); + if (!isRecord(response)) throw new Error("invalid link API response: reconnect object"); + assertExactKeys(response, ["linkId"], "reconnect"); + if (response.linkId !== linkId) throw new Error("invalid link API response: reconnect link id"); + console.log(JSON.stringify({ linkId })); +} + export async function runLinkCommand(rawArgs: string[], deps: LinkCliDeps = {}): Promise { return runCliAction(async () => { const args = [...rawArgs]; @@ -311,6 +324,7 @@ export async function runLinkCommand(rawArgs: string[], deps: LinkCliDeps = {}): else if (command === "issue") await runIssue(args, deps); else if (command === "status") await runStatus(args, deps); else if (command === "revoke") await runRevoke(args, deps); - else throw new CliUsageError("link requires one of: port, issue, status, revoke", LINK_USAGE); + else if (command === "reconnect") await runReconnect(args, deps); + else throw new CliUsageError("link requires one of: port, issue, status, revoke, reconnect", LINK_USAGE); }); } diff --git a/src/cli/registry.ts b/src/cli/registry.ts index bf1b2660bdc..8a1fa270c83 100644 --- a/src/cli/registry.ts +++ b/src/cli/registry.ts @@ -144,14 +144,15 @@ export const CLI_COMMANDS: CliCommandEntry[] = [ }, { name: "link", - usage: "ocx link ", + usage: "ocx link ", summary: "Allocate and manage a loopback remote home link.", details: [ "Port: ocx link port [--json]", "Issue: ocx link issue --alias --tunnel-port [--json]", "Status: ocx link status [--json]", "Revoke: ocx link revoke --link-id [--force --yes] [--json]", - "Issue, status, and revoke use the running proxy's loopback management API and admin token.", + "Reconnect: ocx link reconnect --link-id [--json]", + "Issue, status, revoke, and reconnect use the running proxy's loopback management API and admin token.", ], }, { diff --git a/src/link/supervisor.ts b/src/link/supervisor.ts index c4485ee2a1f..083634b36a1 100644 --- a/src/link/supervisor.ts +++ b/src/link/supervisor.ts @@ -7,6 +7,8 @@ import { createSshRunner, type SshChild, type SshRunner } from "./ssh-runner"; import { classifySshStderr, dueForSpawn, + failedTunnel, + HOME_TUNNEL_RETRY_POLICY, IDLE, reduceTunnel, type TunnelState, @@ -33,6 +35,11 @@ export interface LinkSupervisor { ensureStarted(): Promise; reload(): Promise; stopLink(linkId: string): Promise; + /** + * Restart one Home-initiated tunnel now: stop its child, forget the backoff or failed state, + * and spawn a fresh attempt. False when the link is not a Home-initiated link of this store. + */ + reconnect(linkId: string): Promise; status(): readonly LinkTunnelStatus[]; notifyAuthenticatedRequest?(apiKeyId: string): void; stop(): Promise; @@ -67,6 +74,20 @@ export interface LinkSupervisorDeps { const TIMER_MS = 1_000; const SPAWN_GRACE_MS = 5_000; +/** + * A retry runs while the network may still be down, and ssh can wait out its whole ConnectTimeout + * (10 s for the TCP connect, the SSH handshake and key exchange, see ssh-argv.ts) before it exits. + * A retry is promoted only once it has outlived that, so an attempt that never reached the Child + * does not read connected and restart the outage clock. Like the first attempt's five seconds, + * liveness is a heuristic: authentication and the forward request come after ConnectTimeout, so + * one that stalls past the grace reads connected until ssh exits. + */ +const RETRY_GRACE_MS = 15_000; + +/** A retry whose ssh child is still running: reconnecting, or failed under the retry policy. */ +function retryInFlight(state: TunnelState): boolean { + return (state.kind === "reconnecting" || state.kind === "failed") && state.inFlight === true; +} function sameArgv(left: readonly string[], right: readonly string[]): boolean { return left.length === right.length && left.every((value, index) => value === right[index]); @@ -131,8 +152,14 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi let lifecycleFlight: Promise | undefined; let timer: ReturnType | undefined; const states = new Map(); - const children = new Map(); + const children = new Map(); + /** Per-link spawn counter. An exit or stop that resumes after a newer spawn is stale. */ + const generations = new Map(); + /** Reconnects in flight, so overlapping requests for one link restart it once. */ + const reconnects = new Map>(); const orphanUnverified = new Set(); + /** The generation of the child whose exit is still being classified, per link. */ + const exiting = new Map(); const recordInstances = new Map(); const reconcileApiKeys = (): void => { @@ -187,14 +214,17 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi }; const setEvent = (linkId: string, event: Parameters[1]): TunnelState => { - const next = reduceTunnel(states.get(linkId) ?? IDLE, event, deps.random); + const next = reduceTunnel(states.get(linkId) ?? IDLE, event, deps.random, HOME_TUNNEL_RETRY_POLICY); states.set(linkId, next); return next; }; const spawnFor = (record: LinkRecord): void => { if (stopping || record.direction !== "hub-initiated" || children.has(record.id)) return; - if (states.get(record.id)?.kind === "failed") return; + // Only a fresh link or a due attempt spawns. A reload must neither skip the backoff nor start + // a second child while an exited one is still reporting its stderr. + const prior = states.get(record.id); + if (prior && prior.kind !== "idle" && !dueForSpawn(prior, now())) return; if (store.listenerPort === null) { states.set(record.id, { kind: "failed", since: now(), reason: "forward" }); return; @@ -209,7 +239,11 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi knownHostsFile: linkKnownHostsPath(), }); const child = runner.spawnTunnel(argv); - children.set(record.id, { child, argv }); + const generation = (generations.get(record.id) ?? 0) + 1; + generations.set(record.id, generation); + // An older child's exit still being classified no longer speaks for this link. + exiting.delete(record.id); + children.set(record.id, { child, argv, spawnedAt: now() }); orphanUnverified.delete(record.id); setEvent(record.id, { type: "spawn", now: now() }); writePidfile(pidfilePath(record.id), { version: 1, linkId: record.id, pid: child.pid, argv }); @@ -218,9 +252,17 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi children.delete(record.id); conditionalRemovePidfile(record.id, child.pid); if (stopping) return; - const stderr = child.stderr ? await child.stderr : ""; - const next = setEvent(record.id, { type: "exit", now: now(), stderrClass: classifySshStderr(stderr) }); - if (next.kind === "failed") return; + exiting.set(record.id, generation); + try { + let stderr = ""; + // An unreadable stderr (the runner rejects it past its size cap) still reports the exit. + try { stderr = child.stderr ? await child.stderr : ""; } catch { /* classified as unknown */ } + // A reconnect may have started a fresh child meanwhile; its own lifecycle owns the state. + if (generations.get(record.id) !== generation) return; + setEvent(record.id, { type: "exit", now: now(), stderrClass: classifySshStderr(stderr) }); + } finally { + if (exiting.get(record.id) === generation) exiting.delete(record.id); + } }).catch(() => { if (children.get(record.id)?.child !== child) return; children.delete(record.id); @@ -228,7 +270,8 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi if (!stopping) setEvent(record.id, { type: "exit", now: now(), stderrClass: "network" }); }); } catch { - states.set(record.id, { kind: "failed", since: now(), reason: "forward" }); + const at = now(); + states.set(record.id, failedTunnel("forward", at, HOME_TUNNEL_RETRY_POLICY, prior?.kind === "failed" ? prior.since : at)); } }; @@ -237,14 +280,29 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi const current = now(); for (const record of store.links) { if (record.direction !== "hub-initiated") continue; - const next = setEvent(record.id, { type: "tick", now: current }); + // The exit is still being classified. Failing the link on the outage clock now would make + // a later host-key or auth exit land on a failed state that ignores it. + if (exiting.has(record.id)) continue; + const before = states.get(record.id) ?? IDLE; const child = children.get(record.id); - if (next.kind === "failed" && child) { + // Promotion is checked before the outage limit, so a child that has outlived its grace by the + // time this tick runs (at the limit, or late after a sleep) is promoted rather than killed. + // An idle Child may send no catalog request for hours; for a retry, staying up past the + // retry grace is the proof instead, as five seconds is for a first attempt. + if (child && ((before.kind === "connecting" && current - before.since >= SPAWN_GRACE_MS) + || (retryInFlight(before) && current - child.spawnedAt >= RETRY_GRACE_MS))) { + setEvent(record.id, { type: "ready", now: current }); + continue; + } + const next = setEvent(record.id, { type: "tick", now: current }); + if (next.kind === "failed" && !next.inFlight && child && retryInFlight(before)) { + // A retry started just before the outage limit has not had its grace yet. It carries on as + // the failed state's retry instead of being killed with a link that may already be back. + states.set(record.id, { ...next, inFlight: true }); + } else if (next.kind === "failed" && !next.inFlight && child) { child.child.kill("SIGTERM"); children.delete(record.id); conditionalRemovePidfile(record.id, child.child.pid); - } else if (next.kind === "connecting" && child && current - next.since >= SPAWN_GRACE_MS) { - setEvent(record.id, { type: "ready", now: current }); } else if (dueForSpawn(next, current)) { spawnFor(record); } @@ -299,11 +357,13 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi states.set(linkId, IDLE); return; } + const generation = generations.get(linkId); children.delete(linkId); current.child.kill("SIGTERM"); await current.child.exited; conditionalRemovePidfile(linkId, current.child.pid); - states.set(linkId, IDLE); + // A reconnect may have spawned a replacement while this child was exiting; keep its state. + if (generations.get(linkId) === generation) states.set(linkId, IDLE); }; return { @@ -354,11 +414,31 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi }); }, stopLink, + reconnect(linkId: string) { + const inFlight = reconnects.get(linkId); + if (inFlight) return inFlight; + const flight = (async () => { + // runLifecycle keeps only the latest queued operation, so a reconnect waits for a running + // reload instead of queueing behind it where a second reload could replace it. + if (lifecycleFlight) await lifecycleFlight; + if (stopping || !started) return false; + const record = store.links.find(link => link.id === linkId && link.direction === "hub-initiated"); + if (!record) return false; + await stopLink(linkId); + if (stopping) return false; + spawnFor(record); + return true; + })().finally(() => { reconnects.delete(linkId); }); + reconnects.set(linkId, flight); + return flight; + }, notifyAuthenticatedRequest(apiKeyId: string) { for (const record of store.links) { if (record.direction === "hub-initiated" && record.apiKeyId === apiKeyId && children.has(record.id)) { const state = states.get(record.id); - if (state?.kind === "connecting" || state?.kind === "reconnecting") setEvent(record.id, { type: "ready", now: now() }); + if (state?.kind === "connecting" || state?.kind === "reconnecting" || (state && retryInFlight(state))) { + setEvent(record.id, { type: "ready", now: now() }); + } } } }, diff --git a/src/link/tunnel-state.ts b/src/link/tunnel-state.ts index 1df2c703009..804881307d7 100644 --- a/src/link/tunnel-state.ts +++ b/src/link/tunnel-state.ts @@ -6,10 +6,10 @@ * - reconnecting: a transient failure; requests through the link fail with 503 meanwhile, and a * new attempt is due at `retryAt`. * - failed: auth, host key and forward failures, and a link that stayed down for FAILED_AFTER_MS. - * Without a retry policy (the Home's `-R` supervisor) it needs the user. With one (the Child's - * own `-L` tunnel) a reason that has a delay is tried again at `retryAt`, and the retry attempt - * runs with `inFlight` while the state still reads failed; a reason without a delay stays - * terminal. + * Without a retry policy it needs the user. With one (the Home's `-R` supervisor and the Child's + * own `-L` tunnel both run with one) a reason that has a delay is tried again at `retryAt`, and + * the retry attempt runs with `inFlight` while the state still reads failed; a reason without a + * delay stays terminal. */ export type TunnelFailure = "auth" | "hostkey" | "forward" | "timeout"; @@ -48,6 +48,13 @@ export const CLIENT_TUNNEL_RETRY_POLICY: TunnelRetryPolicy = { retryFailedAfterMs: { timeout: 60_000, forward: 60_000, auth: 5 * 60_000 }, }; +/** + * The Home's `-R` supervisor dials the Child's sshd on the same cadence and limits. A sleep, an + * outage longer than FAILED_AFTER_MS, or a forward the Child's sshd still holds for the session + * that just died heals about a minute after it clears, instead of waiting for a proxy restart. + */ +export const HOME_TUNNEL_RETRY_POLICY: TunnelRetryPolicy = CLIENT_TUNNEL_RETRY_POLICY; + export const IDLE: TunnelState = { kind: "idle" }; /** Capped exponential backoff with ±20% jitter. `random` is injectable for tests. */ diff --git a/src/server/management/link-routes.ts b/src/server/management/link-routes.ts index 9d09b8e86f8..e52d6ab0c40 100644 --- a/src/server/management/link-routes.ts +++ b/src/server/management/link-routes.ts @@ -525,6 +525,30 @@ async function handleJoin(ctx: ManagementContext, state: LinkRouteState): Promis } } +/** + * Restart a Home-initiated link's tunnel now: the dashboard's Retry and `ocx link reconnect`. + * The supervisor stops the current ssh child, if any, forgets its backoff or failed state and + * spawns a fresh attempt; status then reports it like any other attempt. + */ +async function reconnect(ctx: ManagementContext, state: LinkRouteState, id: string): Promise { + if (!LINK_ID.test(id)) return fail("invalid_link_id", "The link id is invalid.", 400); + const record = readStoreFor(ctx).links.find(link => link.id === id); + if (!record) return fail("link_not_found", "The link was not found.", 404); + if (record.direction !== "hub-initiated") { + return fail("link_not_home_initiated", "Only a Home-initiated link has a tunnel this computer can restart.", 409); + } + try { + // A reverse forward may only target a port the link listener owns, as when a link is applied. + await state.listener.ensureStarted(); + if (state.listener.status().state !== "listening") return fail("listener_unavailable", "The link listener is unavailable.", 503); + await state.supervisor.ensureStarted(); + if (!await state.supervisor.reconnect(id)) return fail("link_unavailable", "The link tunnel could not be restarted.", 503); + } catch { + return fail("link_unavailable", "The link tunnel could not be restarted.", 503); + } + return Response.json({ linkId: id }); +} + async function remove(ctx: ManagementContext, state: LinkRouteState, id: string): Promise { if (!LINK_ID.test(id)) return fail("invalid_link_id", "The link id is invalid.", 400); const current = readStoreFor(ctx); @@ -637,6 +661,14 @@ export async function handleLinkRoutes(ctx: ManagementContext, suppliedState?: L if (denied) return denied; return issue(ctx); } + const reconnectPath = req.method === "POST" ? /^\/api\/link\/([^/]+)\/reconnect$/.exec(path) : null; + if (reconnectPath) { + const denied = auth(ctx, "either"); + if (denied) return denied; + let id: string; + try { id = decodeURIComponent(reconnectPath[1]!); } catch { return fail("invalid_link_id", "The link id is invalid.", 400); } + return reconnect(ctx, state, id); + } if (req.method === "DELETE" && path.startsWith("/api/link/")) { const denied = auth(ctx, "either"); if (denied) return denied; diff --git a/src/server/management/route-registry.ts b/src/server/management/route-registry.ts index 17e7932739b..9496308482a 100644 --- a/src/server/management/route-registry.ts +++ b/src/server/management/route-registry.ts @@ -395,6 +395,7 @@ export const MANAGEMENT_ROUTES: readonly ManagementRoute[] = [ { method: "POST", path: "/api/link/join", module: "server/management/link-routes", mutates: true, exempt: { reason: "session-only", why: "Joining a confirmed Home issues a link credential and restarts this standalone runtime as a client on its configured port, which briefly interrupts running Codex turns. It requires an operator-paired dashboard session and a pre-join notice; unpaired standalone loopback, admin-token and Tailscale identity sessions are refused." } }, { method: "POST", path: "/api/link/apply", module: "server/management/link-routes", mutates: true, exempt: { reason: "session-only", why: "Applying a link issues a data key and starts a remote tunnel, so it requires a dashboard session (paired, or standalone loopback dashboard on trusted loopback ingress)." } }, { method: "DELETE", path: "/api/link/{id}", module: "server/management/link-routes", mutates: true, mechanism: "regex" }, + { method: "POST", path: "/api/link/{id}/reconnect", module: "server/management/link-routes", mutates: true, mechanism: "regex" }, { method: "POST", path: "/api/link/issue", module: "server/management/link-routes", mutates: true }, // server/management/remote-workspace-routes { method: "GET", path: "/api/remote-workspace", module: "server/management/remote-workspace-routes", mutates: false }, diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index 6875ea17229..cafff4f5090 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -278,7 +278,7 @@ per-request first-party callback reads that live object; a failed write leaves i | Providers | Create/update/delete ordinary provider configs and enrich registry metadata. A `POST /api/providers` overwrite of an existing name keeps the eight operator compatibility settings (`PROVIDER_COMPAT_CARRY_FIELDS` in `src/server/management/provider-overwrite-carry.ts`) and the stored key pool only while the destination (adapter, normalized base URL, auth mode when named) is unchanged. It keeps omitted `hideRawReasoning` on every overwrite of the same name, including destination moves, because it is a display policy; a submitted boolean wins. It never merges the rest of the old row. `PATCH` is a field mask and keeps every field it does not name; `hideRawReasoning` accepts a boolean or `null` to clear it. The reserved `openai` card exposes Pool(default)/Direct account mode; `openai-apikey` remains the separate API route. | | Models | Fetch routed model lists, disabled model visibility, and catalog-facing ids. New non-OAuth registration holds exposure until authoritative discovery; 20 or more distinct switch rows start OFF without disabling the provider. Pending rows cannot accept visibility changes. Ordinary model reads also apply the [new-arrival catalog contract](catalog.md#shared-catalog) before returning models. With a matching persisted configuration, discovery records the baseline, NEW badges, and automatic disables together; HTTP listing does not wait for a separate Codex sync. Inventory already drifted from disk uses a detached policy projection: arrivals remain disabled in management rows without changing live config or disk. | | OAuth | Login/status/logout for OAuth-backed providers, plus multiauth account management: `GET /api/oauth/accounts`, `PUT /api/oauth/accounts/active`, `PUT /api/oauth/accounts/alias`, `PUT /api/oauth/accounts/pause`, and `DELETE /api/oauth/accounts` list masked accounts per provider, switch the active one, edit its display-only alias, pause/resume a generic OAuth account, and remove one. Paused generic OAuth accounts are excluded from request selection, 429 failover, and proactive token refresh; pausing an active account selects the next usable account when available, and resuming an account restores an active selection if the current one remains paused. With no unpaused account, requests return 403 rather than a login error. Kiro account-list rows include the current automatic-selection projection and closed exclusion reason; an active singleton may still send. The login flow itself is `GET /api/oauth/providers`, `POST /api/oauth/login`, `POST /api/oauth/login/code`, `POST /api/oauth/login/cancel`, `POST /api/oauth/logout`, and `GET /api/oauth/status`; pool controls are `GET/PUT/PATCH /api/oauth/accounts/pool` and `POST /api/oauth/accounts/clear-cooldown`. Login accepts `addAccount: true` to force a fresh browser identity. Meta Muse login start and manual-code continuation require the server-resolved `gui-session` principal before credential acquisition or code submission (including reauth); see the [provider contract](providers-and-adapters.md). Device flows return a structured `deviceCode`; the GUI highlights and copies it before the user opens the verification page. | -| Key providers | `GET /api/key-providers` exposes API-key provider presets for setup and dashboard flows, and `GET/POST/DELETE /api/keys` owns the proxy's own admission keys. Machine links live in `src/server/management/link-routes.ts`: dashboard sessions reach `GET /api/link/candidates`, `POST /api/link/probe`, `POST /api/link/confirm-host` and `POST /api/link/apply`, meaning a paired session or, on a standalone runtime, the current loopback-issued session on trusted loopback ingress; `POST /api/link/join` instead requires an operator-paired session because it durably redirects local client traffic, and also refuses a runtime that is not standalone (`409 standalone_required`) or not listening on its configured port (`409 join_port_mismatch`) before any SSH, then restarts this runtime as a client; `GET /api/link/status` reports `joinAvailable` and typed `joinDenied` causes only to GUI-session callers, distinguishing pairing, runtime-role and port refusals without changing the admin/CLI DTO, so the dashboard enables the Child role only when the join gates pass; `GET /api/link/status` and `DELETE /api/link/{id}` also accept the admin token on a trusted loopback ingress; `POST /api/link/issue` accepts only that admin token. Tailscale-identity sessions are refused on every link route. See [Remote Link](remote-link.md). Multi-key pool per key-auth provider: `GET /api/providers/keys`, `POST /api/providers/keys`, `PUT /api/providers/keys/active`, `PUT /api/providers/keys/alias`, `DELETE /api/providers/keys` masked list, add (upsert + activate), switch, rename, and remove keys. `provider.apiKey` always mirrors the active pool entry so routing stays single-key. | +| Key providers | `GET /api/key-providers` exposes API-key provider presets for setup and dashboard flows, and `GET/POST/DELETE /api/keys` owns the proxy's own admission keys. Machine links live in `src/server/management/link-routes.ts`: dashboard sessions reach `GET /api/link/candidates`, `POST /api/link/probe`, `POST /api/link/confirm-host` and `POST /api/link/apply`, meaning a paired session or, on a standalone runtime, the current loopback-issued session on trusted loopback ingress; `POST /api/link/join` instead requires an operator-paired session because it durably redirects local client traffic, and also refuses a runtime that is not standalone (`409 standalone_required`) or not listening on its configured port (`409 join_port_mismatch`) before any SSH, then restarts this runtime as a client; `GET /api/link/status` reports `joinAvailable` and typed `joinDenied` causes only to GUI-session callers, distinguishing pairing, runtime-role and port refusals without changing the admin/CLI DTO, so the dashboard enables the Child role only when the join gates pass; `GET /api/link/status`, `POST /api/link/{id}/reconnect` and `DELETE /api/link/{id}` also accept the admin token on a trusted loopback ingress, and reconnect restarts a Home-initiated link's tunnel now; `POST /api/link/issue` accepts only that admin token. Tailscale-identity sessions are refused on every link route. See [Remote Link](remote-link.md). Multi-key pool per key-auth provider: `GET /api/providers/keys`, `POST /api/providers/keys`, `PUT /api/providers/keys/active`, `PUT /api/providers/keys/alias`, `DELETE /api/providers/keys` masked list, add (upsert + activate), switch, rename, and remove keys. `provider.apiKey` always mirrors the active pool entry so routing stays single-key. | | OpenAI account mode | Report one OpenAI Codex card with Pool/Direct controls and one API-key card. Mode PATCH persists live without restart or catalog identity changes; Pool owns account/quota controls and Direct uses caller/main login only. Main-account DTOs report real credential presence and terminal `needsReauth` state instead of treating missing/invalid native auth as an unknown quota. Selection order has its own route: `PUT /api/codex-auth/accounts/priority` takes `{ id, priority }`, where `priority` is an integer -100..100 or `null` to restore the default, accepts `__main__`, 404s an unknown id, and echoes the stored value. Re-ordering never clears thread affinity, so the response carries no `appliesImmediately`, but it does release any pin — see [`openai-tiers.md`](providers/openai-tiers.md) for why. `PUT /api/codex-auth/active` with a null id releases one too, but that drops the operator's account selection along with it, so this route is the only operator-facing way to clear a pin while leaving the selected account in place. `GET /api/codex-auth/active` reports `pinned`, true only while the manually selected account is still the effective active one, plus `pinnedAccountId`, which names the pinned account whether or not it is the active one. Surfaces should render `pinnedAccountId`: under round-robin and fill-first the pin caps the tier ceiling at its own tier while the strategy cursor moves freely inside that tier, so `pinned` goes false on a sibling's turn even though the pin is still suppressing every higher tier — which is why the dashboard badges `pinnedAccountId` and the GUI controller tracks only the id. `pinned` answers the narrower question of whether routing is *currently* on the operator's choice; no surface in this repo asks it, and a new one almost certainly wants the id instead. | | Subagents | Read/write the featured `subagentModels` list capped at five ids. `GET/PUT /api/injection-model` manages the shared delegation model/effort selection, the independent OpenCodex guidance switch, and the default-off `syncCodexSubagentDefaults` opt-in for native Codex subagent defaults. When OpenCodex owns the active Codex routing, native `[agents]` defaults apply to newly created Codex tasks after sync/restart; external user-managed provider configs remain untouched. The defaults do not cause delegation and preserve existing user-owned defaults rather than overwriting them. PUT is partial-update: absent keys are unchanged, `null` clears, and non-object bodies are rejected with 400 before field validation. `syncCodexSubagentDefaults: true` requires a nonblank `model` and a supported Codex reasoning effort when effort is set; clearing `model` (null/empty) always clears effort and disables native-default sync even when the stored effort was invalid. | | V2 / Multi-agent mode | `GET/PUT /api/v2` — reports/sets the codex `multi_agent_v2` feature flag, the 3-state `multiAgentMode` override (`v1`/`default`/`v2`), the `keepNativeChatGptOnV1` hybrid pin, and the logical maximum thread count. Selecting `v2` normally enables the native flag; with the hybrid pin it disables that global override so native rows can resolve to v1 while routed rows resolve to v2. Selecting `v1` disables the flag; `default` leaves it unchanged. PUT rejects an explicit enabled flag that conflicts with the selected mode or hybrid pin. Every transition preserves the logical thread limit, is rollback-safe, and resyncs the catalog. GET and successful PUT also return stored `multiAgentModeHintText` plus response-only `multiAgentModeHintRecommendation: { text, revision }`; the recommendation is not a writable or persisted config field. Both also return response-only `multiAgentSurfaceAdvisory: { required, mode, recommended, version, docsUrl }`, true while the resolved mode is not v1 and the stored acknowledgement version is behind; PUT accepts `multiAgentSurfaceAdvisoryAcknowledged`, where only `true` stores the current version and `false` is an explicit no-op, and it composes with a `multiAgentMode` write in the same body so the dialog's recommended answer is one request. | diff --git a/structure/remote-link.md b/structure/remote-link.md index d812d65cde2..8f2c0fb1eed 100644 --- a/structure/remote-link.md +++ b/structure/remote-link.md @@ -8,7 +8,7 @@ Every remote `ocx` call goes through `remoteOcxArgv`, which runs `sh -c` with a `src/link/ssh-config.ts` lists host candidates from `~/.ssh/config`. Arguments are split with the rules of OpenSSH's `argv_split`. Pattern hosts, `Match` blocks and aliases that fail the alias check produce no candidates, and only top-level `Include` directives are followed, because an include inside a `Host` or `Match` block is conditional. A candidate is an offer, not trust. -`src/link/tunnel-state.ts` is the tunnel lifecycle reducer: connecting, connected, reconnecting with capped jittered backoff, and failed for auth, host-key and forward errors or after five minutes without a connection, whether or not an attempt is in flight. Without a retry policy, failed is terminal; that is how `src/link/supervisor.ts` runs the Home's `-R` tunnels, and it keeps a spawned tunnel in connecting until its link key authenticates a catalog request or the child remains alive for five seconds. `CLIENT_TUNNEL_RETRY_POLICY`, which only the Child's own tunnel uses, gives failed a `retryAt`: timeout and forward failures are retried after 60 seconds and auth failures after five minutes, so no more than 12 auth attempts reach the Home's sshd in an hour; a host-key failure is never retried. A retry runs while the state still reads failed (`inFlight`), a ready event promotes it to connected, and an exit returns it to failed with the original `since` and the next `retryAt`, so a long outage keeps the one-a-minute cadence instead of restarting fast backoff. +`src/link/tunnel-state.ts` is the tunnel lifecycle reducer: connecting, connected, reconnecting with capped jittered backoff, and failed for auth, host-key and forward errors or after five minutes without a connection, whether or not an attempt is in flight. Without a retry policy, failed is terminal. `CLIENT_TUNNEL_RETRY_POLICY` gives failed a `retryAt`: timeout and forward failures are retried after 60 seconds and auth failures after five minutes, so no more than 12 auth attempts reach the peer's sshd in an hour; a host-key failure is never retried. The Child's own tunnel runs with it, and `src/link/supervisor.ts` runs the Home's `-R` tunnels with `HOME_TUNNEL_RETRY_POLICY`, the same policy. A retry runs while the state still reads failed (`inFlight`), a ready event promotes it to connected, and an exit returns it to failed with the original `since` and the next `retryAt`, so a long outage keeps the one-a-minute cadence instead of restarting fast backoff. The Home's supervisor promotes a first attempt to connected when its link key authenticates a catalog request or the child stays alive for five seconds. It promotes a retry (reconnecting, or failed with `inFlight`) on the same catalog request or once the child outlives a 15-second retry grace, longer than the ten seconds ssh's `ConnectTimeout` allows for the TCP connect, handshake and key exchange, because an idle Child may send no catalog request at all. Liveness is a heuristic here, as it is for a first attempt: authentication and the remote forward request come after `ConnectTimeout`, so one that stalls past the grace reads connected until ssh exits, and the exit returns the link to the retry path. Promotion is checked before the outage limit, so a child that has outlived its grace by the time a tick runs, at the limit or late after a sleep, is promoted. At the outage limit it kills the child, except a retry still inside its grace, which carries on as the failed state's retry. A reload spawns a tunnel only for a fresh link or a due attempt, so it neither skips the backoff nor starts a second child while an exited one is still reporting. `reconnect(linkId)` is the deliberate exception: it waits for a running reload, stops that link's child, forgets its backoff or failed state and spawns a fresh attempt at once. Overlapping reconnects for one link share one restart, and every spawn bumps a per-link generation, so an exit still being classified or a stop still waiting for its child never overwrites the state of a newer child. `POST /api/link/{id}/reconnect` (the dashboard's **Retry** on a failed or reconnecting Home link, and `ocx link reconnect --link-id `) requires the link listener to be listening (`503 listener_unavailable` otherwise, as for apply), then calls it after `ensureStarted`, answers `{ linkId }`, and refuses a Child-initiated link with `409 link_not_home_initiated` because the Child owns that tunnel. `src/link/routes.ts` holds the one route and method table for linked traffic; the hub-link listener and the client relay both decide admission from it. @@ -36,7 +36,7 @@ The client tunnel pidfile is `/link/client-tunnel.pid` with `{ versio ## Dashboard admission -The dashboard link routes (`GET /api/link/status`, `GET /api/link/candidates`, `POST /api/link/probe`, `POST /api/link/confirm-host`, `POST /api/link/apply` and `DELETE /api/link/{id}`) admit a paired GUI session, or, on a standalone runtime only, the current loopback-issued GUI session that reached the public listener bound to a loopback hostname. `POST /api/link/join` is stricter and requires the paired session because it durably redirects this machine's client traffic. The hub-link, hub-management and claude-intercept ingresses are never trusted loopback ingress, a stale session is refused, and a hub keeps the paired-only rule. The Tailscale identity refusal runs before either check. A loopback session is minted by the loopback dashboard bootstrap without a credential, so it proves possession, not user presence: any local process can fetch the bootstrap and replay its token and CSRF value, which is why it cannot authorize joining and why `src/client/machine-listener.ts` keeps durable machine changes on the connected listener away from that session. The remaining link routes accept this casual-path trade, the same as `POST /api/github/star` in `src/server/management/sidebar-routes.ts`; it is not a secret-backed boundary like an operator-created pairing grant or the admin token. +The dashboard link routes (`GET /api/link/status`, `GET /api/link/candidates`, `POST /api/link/probe`, `POST /api/link/confirm-host`, `POST /api/link/apply`, `POST /api/link/{id}/reconnect` and `DELETE /api/link/{id}`) admit a paired GUI session, or, on a standalone runtime only, the current loopback-issued GUI session that reached the public listener bound to a loopback hostname. `POST /api/link/join` is stricter and requires the paired session because it durably redirects this machine's client traffic. The hub-link, hub-management and claude-intercept ingresses are never trusted loopback ingress, a stale session is refused, and a hub keeps the paired-only rule. The Tailscale identity refusal runs before either check. A loopback session is minted by the loopback dashboard bootstrap without a credential, so it proves possession, not user presence: any local process can fetch the bootstrap and replay its token and CSRF value, which is why it cannot authorize joining and why `src/client/machine-listener.ts` keeps durable machine changes on the connected listener away from that session. The remaining link routes accept this casual-path trade, the same as `POST /api/github/star` in `src/server/management/sidebar-routes.ts`; it is not a secret-backed boundary like an operator-created pairing grant or the admin token. A successful join restarts this proxy (a 503 drain of up to a minute while running turns finish, then a closed listener) into the client runtime on the configured port. `GET /api/link/status` tells a GUI-session caller whether it may join: the response gains `joinAvailable`, true only for a paired dashboard session on a standalone runtime whose live port is its configured port, and `joinDenied`, which is `null` when allowed or the first failed gate (`pairing_required`, `standalone_required`, then `join_port_mismatch`). An admin-token caller gets the exact K16 document without either field, because `ocx link status` validates it key by key. The dashboard reads an absent availability field as false and an absent or unknown denial as generic unavailability; while unavailable the Child role card cannot be selected by pointer or keyboard. A pairing refusal says to pair this machine first and offers code entry only on the existing same-origin literal-loopback transport. A port refusal instead asks the operator to restart on the configured port; neither status reads nor displaying this guidance mint a grant or change pairing, keys, routing, or runtime state. Before **Connect as Child** the confirmation panel says that the restart briefly interrupts Codex and that Codex keeps its local address. The dashboard reads the standalone's pid from same-origin `/healthz`, sends the join, then reads `/healthz` once a second, skipping status polls meanwhile, and reloads only when it reports `role: "client"` under another pid, so the reloaded document carries the client role and a fresh session. It never gives up by itself: past the server's own handoff budget (60 s drain plus 70 s replacement readiness, plus a margin: 145 seconds) it says the restart is slow and keeps reading `/healthz` every 5 seconds until the Child answers or the page is left. @@ -48,7 +48,7 @@ A connected Child answers `GET` and `HEAD /api/link/status` on its own listener `src/link/ssh-runner.ts` runs every OpenSSH and `ssh-keygen` argv without a shell and caps captured output. On POSIX it spawns them with the inherited PATH plus `/opt/homebrew/bin`, `/usr/local/bin`, `$HOME/.bun/bin` and `$HOME/.local/bin` appended once, so a ProxyCommand helper resolves inside a desktop sidecar that inherited a minimal PATH; Windows keeps the inherited environment. `src/link/supervisor.ts` keeps one `ssh -R` child per hub-initiated link, drives it with the tunnel reducer, coalesces reloads without dropping a later request, reconciles unowned `link:` API keys at startup, and stops children before the hub-link listener on shutdown. Automatic startup begins the supervisor only after the hub-link listener owns its socket; a bind failure must never leave a reverse forward targeting the persisted port. It reaps a leftover tunnel only when Linux `/proc//cmdline` matches the recorded argv exactly; on other platforms a leftover is reported, never killed. `src/link/status-projection.ts` builds the status document the dashboard and `ocx link status` read, including persisted compensation failures, and `src/link/admission-wait.ts` waits for the first key-authenticated `/v1/catalog` read that proves a new link works. -Applying a link probes the host key into a temporary file, waits for the operator to confirm the fingerprint, issues a data key, records the link, starts the tunnel and runs the client's `ocx connect --link --key-stdin` over SSH with the key on standard input. A failed step revokes the new key first and removes the record only after revocation succeeds; if revocation fails the `src/link/` state persists a `compensation_failed` marker, and status reports the failed compensation after restart. A listener that is not listening fails the request instead of handing out a key. Removing a link stops its tunnel, disconnects the client, revokes the key and deletes the record; a failed client disconnect restarts the tunnel and keeps the record and key unless removal is forced. `src/cli/link.ts` provides `ocx link port|issue|revoke|status`. `ocx link revoke` is idempotent: a `404 link_not_found` answer exits 0, because removal revokes the key before it deletes the record, so a missing record means the key is already gone. A 404 without that code still fails. +Applying a link probes the host key into a temporary file, waits for the operator to confirm the fingerprint, issues a data key, records the link, starts the tunnel and runs the client's `ocx connect --link --key-stdin` over SSH with the key on standard input. A failed step revokes the new key first and removes the record only after revocation succeeds; if revocation fails the `src/link/` state persists a `compensation_failed` marker, and status reports the failed compensation after restart. A listener that is not listening fails the request instead of handing out a key. Removing a link stops its tunnel, disconnects the client, revokes the key and deletes the record; a failed client disconnect restarts the tunnel and keeps the record and key unless removal is forced. `src/cli/link.ts` provides `ocx link port|issue|revoke|status|reconnect`. `ocx link revoke` is idempotent: a `404 link_not_found` answer exits 0, because removal revokes the key before it deletes the record, so a missing record means the key is already gone. A 404 without that code still fails. ## Client link transport diff --git a/tests/cli/cli-link.test.ts b/tests/cli/cli-link.test.ts index da86a4da416..bd3d968614b 100644 --- a/tests/cli/cli-link.test.ts +++ b/tests/cli/cli-link.test.ts @@ -212,6 +212,32 @@ describe("ocx link", () => { } }); + test("reconnect posts to the link's reconnect route and prints only the link id", async () => { + const calls: RequestCall[] = []; + const output = captureOutput(); + try { + const code = await runLinkCommand(["reconnect", "--link-id", issueResponse.linkId, "--json"], { + baseUrl: "http://127.0.0.1:19101", + fetchImpl: fakeFetch({ linkId: issueResponse.linkId }, calls), + readAdminToken: () => "ocx_admin_test-token", + }); + expect(code).toBe(0); + expect(JSON.parse(output.stdout[0]!)).toEqual({ linkId: issueResponse.linkId }); + expect(calls[0]?.url).toBe(`http://127.0.0.1:19101/api/link/${issueResponse.linkId}/reconnect`); + expect(calls[0]?.init?.method).toBe("POST"); + const refused = await runLinkCommand(["reconnect", "--link-id", issueResponse.linkId], { + baseUrl: "http://127.0.0.1:19101", + fetchImpl: fakeFetch({ error: { code: "link_not_home_initiated" } }, [], 409), + readAdminToken: () => "ocx_admin_test-token", + }); + expect(refused).not.toBe(0); + expect(await runLinkCommand(["reconnect", "--link-id", "not-a-link"], { baseUrl: "http://127.0.0.1:19101" })).not.toBe(0); + expect(output.stdout).toHaveLength(1); + } finally { + output.restore(); + } + }); + test("rejects malformed status responses", async () => { const output = captureOutput(); try { @@ -228,7 +254,7 @@ describe("ocx link", () => { } }); - test("registers the canonical command and all four capability leaves", () => { + test("registers the canonical command and all five capability leaves", () => { expect(findCommand("link")?.name).toBe("link"); expect(DISPATCH_COMMANDS).toContain("link"); expect(CAPABILITIES.filter(cap => cap.command[0] === "link").map(cap => cap.command.join(" "))).toEqual([ @@ -236,6 +262,7 @@ describe("ocx link", () => { "link issue", "link status", "link revoke", + "link reconnect", ]); }); }); diff --git a/tests/clients/link-supervisor.test.ts b/tests/clients/link-supervisor.test.ts index 3a2c7eb41af..c67d929699b 100644 --- a/tests/clients/link-supervisor.test.ts +++ b/tests/clients/link-supervisor.test.ts @@ -127,21 +127,48 @@ test("marks a live tunnel connected after the grace period or authenticated cata await grace.stop(); }); -test("auth and host ownership failures do not retry, and client links stay client-owned", async () => { +test("auth failures retry every five minutes, a changed host key never does, and client links stay client-owned", async () => { const fake = fakeRunner(); + const timers: Array<() => void> = []; + let current = 0; const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef"), record("client-initiated", "lnk_fedcba9876543210")); - const supervisor = createLinkSupervisor({ readStore: () => store, runner: fake.runner, pidfileDir: "/tmp/opencodex-link-supervisor-test" }); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner: fake.runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + }); supervisor.start(); fake.children[0]!.stderr = "Permission denied"; fake.children[0]!.resolve(1); await Promise.resolve(); await Promise.resolve(); - expect(supervisor.status()[0]!.state).toEqual({ kind: "failed", since: expect.any(Number), reason: "auth" }); + expect(supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 0, reason: "auth", retryAt: 300_000, inFlight: false }); expect(supervisor.status()[1]).toMatchObject({ state: "client-owned", pid: null }); + current = 299_999; + timers[0]!(); + expect(fake.children).toHaveLength(1); + current = 300_000; + timers[0]!(); + expect(fake.children).toHaveLength(2); + fake.children[1]!.stderr = "Host key verification failed."; + fake.children[1]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + const failed = supervisor.status()[0]!.state; + expect(failed).toEqual({ kind: "failed", since: 0, reason: "hostkey" }); + for (let second = 0; second < 2 * 60 * 60; second += 10) { + current += 10_000; + timers[0]!(); + } + expect(fake.children).toHaveLength(2); + expect(supervisor.status()[0]!.state).toEqual(failed); await supervisor.stop(); }); -test("the Home's -R supervisor keeps the old policy: a failed tunnel is not retried and a live one is promoted after five seconds", async () => { +test("the Home's -R supervisor retries a failed forward once a minute and promotes a retry that stays up", async () => { const fake = fakeRunner(); const timers: Array<() => void> = []; let current = 0; @@ -159,21 +186,454 @@ test("the Home's -R supervisor keeps the old policy: a failed tunnel is not retr current = 5_000; timers[0]!(); expect(supervisor.status()[0]!.state).toEqual({ kind: "connected", since: 5_000 }); + // The Child's sshd still holds the forward of the session that just died. fake.children[0]!.stderr = "Error: remote port forwarding failed for listen port 19002"; fake.children[0]!.resolve(255); await Promise.resolve(); await Promise.resolve(); - const failed = supervisor.status()[0]!.state; - expect(failed).toEqual({ kind: "failed", since: 5_000, reason: "forward" }); - for (let second = 0; second < 2 * 60 * 60; second += 10) { - current += 10_000; + expect(supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 5_000, reason: "forward", retryAt: 65_000, inFlight: false }); + current = 64_999; + timers[0]!(); + expect(fake.children).toHaveLength(1); + current = 65_000; + timers[0]!(); + expect(fake.children).toHaveLength(2); + expect(supervisor.status()[0]!.state).toMatchObject({ kind: "failed", reason: "forward", inFlight: true }); + fake.children[1]!.stderr = "Error: remote port forwarding failed for listen port 19002"; + fake.children[1]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + expect(supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 5_000, reason: "forward", retryAt: 125_000, inFlight: false }); + current = 125_000; + timers[0]!(); + expect(fake.children).toHaveLength(3); + // A retry in flight is not killed for reading failed, and staying up past the retry grace proves it. + current = 139_999; + timers[0]!(); + expect(supervisor.status()[0]!.state).toMatchObject({ kind: "failed", inFlight: true }); + current = 140_000; + timers[0]!(); + expect(supervisor.status()[0]!.state).toEqual({ kind: "connected", since: 140_000 }); + expect(supervisor.status()[0]!.pid).toBe(fake.children[2]!.child.pid); + await supervisor.stop(); +}); + +test("the Home's -R supervisor keeps a reconnected tunnel past the outage limit without a catalog request", async () => { + const fake = fakeRunner(); + const timers: Array<() => void> = []; + let current = 0; + const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef")); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner: fake.runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + random: () => 0.5, + }); + supervisor.start(); + current = 5_000; + timers[0]!(); + fake.children[0]!.stderr = "client_loop: send disconnect: Broken pipe"; + fake.children[0]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + expect(supervisor.status()[0]!.state).toMatchObject({ kind: "reconnecting", since: 5_000, retryAt: 6_000 }); + current = 6_000; + timers[0]!(); + expect(fake.children).toHaveLength(2); + // Still inside ssh's ConnectTimeout, a live retry proves nothing yet. + current = 16_000; + timers[0]!(); + expect(supervisor.status()[0]!.state).toMatchObject({ kind: "reconnecting", inFlight: true }); + current = 21_000; + timers[0]!(); + expect(supervisor.status()[0]!.state).toEqual({ kind: "connected", since: 21_000 }); + // Past the five-minute outage limit, where the unpromoted retry used to be killed. + for (; current < 400_000; current += 1_000) timers[0]!(); + expect(fake.children).toHaveLength(2); + expect(supervisor.status()[0]!.state).toEqual({ kind: "connected", since: 21_000 }); + await supervisor.stop(); +}); + +test("the Home's -R supervisor fails a long outage at five minutes, then retries it once a minute", async () => { + const fake = fakeRunner(); + const timers: Array<() => void> = []; + let current = 0; + const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef")); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner: fake.runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + random: () => 0.5, + }); + supervisor.start(); + current = 5_000; + timers[0]!(); + let exited = 0; + // Every attempt hits a dead network until the link has been down for five minutes. + while (current < 5_000 + 5 * 60_000) { + for (; exited < fake.children.length; exited += 1) { + fake.children[exited]!.stderr = "ssh: connect to host child.example port 22: Network is unreachable"; + fake.children[exited]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + } + current += 1_000; timers[0]!(); } - expect(fake.children).toHaveLength(1); + expect(supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 305_000, reason: "timeout", retryAt: 365_000, inFlight: false }); + const attempts = fake.children.length; + current = 364_999; + timers[0]!(); + expect(fake.children).toHaveLength(attempts); + current = 365_000; + timers[0]!(); + expect(fake.children).toHaveLength(attempts + 1); + supervisor.notifyAuthenticatedRequest!("lnk_0123456789abcdef-key"); + expect(supervisor.status()[0]!.state).toEqual({ kind: "connected", since: 365_000 }); + await supervisor.stop(); +}); + +test("the Home's -R supervisor lets a retry started just before the outage limit finish its grace", async () => { + const fake = fakeRunner(); + const timers: Array<() => void> = []; + let current = 0; + const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef")); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner: fake.runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + random: () => 0.5, + }); + supervisor.start(); + current = 5_000; + timers[0]!(); + fake.children[0]!.stderr = "client_loop: send disconnect: Broken pipe"; + fake.children[0]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + // The Home slept through the backoff; the first tick after waking starts the retry. + current = 300_000; + timers[0]!(); + expect(fake.children).toHaveLength(2); + current = 305_000; + timers[0]!(); + expect(supervisor.status()[0]).toMatchObject({ + state: { kind: "failed", reason: "timeout", inFlight: true }, + pid: fake.children[1]!.child.pid, + }); + current = 315_000; + timers[0]!(); + expect(supervisor.status()[0]!.state).toEqual({ kind: "connected", since: 315_000 }); + await supervisor.stop(); +}); + +test("the Home's -R supervisor promotes a retry whose grace ends at the outage limit or before a late tick", async () => { + const fake = fakeRunner(); + const timers: Array<() => void> = []; + let current = 0; + const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef")); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner: fake.runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + random: () => 0.5, + }); + supervisor.start(); + current = 5_000; + timers[0]!(); + fake.children[0]!.stderr = "client_loop: send disconnect: Broken pipe"; + fake.children[0]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + // The retry's grace ends on the very tick that reaches the outage limit. + current = 290_000; + timers[0]!(); + current = 305_000; + timers[0]!(); + expect(supervisor.status()[0]).toMatchObject({ state: { kind: "connected", since: 305_000 }, pid: fake.children[1]!.child.pid }); + fake.children[1]!.stderr = "client_loop: send disconnect: Broken pipe"; + fake.children[1]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + current = 600_000; + timers[0]!(); + expect(fake.children).toHaveLength(3); + // The Home sleeps past both deadlines; the retry that kept running is promoted on waking. + current = 900_000; + timers[0]!(); + expect(supervisor.status()[0]).toMatchObject({ state: { kind: "connected", since: 900_000 }, pid: fake.children[2]!.child.pid }); + await supervisor.stop(); +}); + +test("a host-key exit still draining stderr at the outage limit stays terminal", async () => { + const fake = fakeRunner(); + const timers: Array<() => void> = []; + let current = 0; + const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef")); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner: fake.runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + random: () => 0.5, + }); + supervisor.start(); + current = 5_000; + timers[0]!(); + fake.children[0]!.stderr = "client_loop: send disconnect: Broken pipe"; + fake.children[0]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + current = 300_000; + timers[0]!(); + expect(fake.children).toHaveLength(2); + const stderr = deferred(); + Object.defineProperty(fake.children[1]!.child, "stderr", { value: stderr.promise }); + current = 305_000; + fake.children[1]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + // The outage limit is reached while the exit's stderr is still being read. + timers[0]!(); + expect(supervisor.status()[0]!.state).toMatchObject({ kind: "reconnecting", since: 5_000 }); + stderr.resolve("Host key verification failed."); + await Promise.resolve(); + await Promise.resolve(); + const failed = supervisor.status()[0]!.state; + expect(failed).toEqual({ kind: "failed", since: 305_000, reason: "hostkey" }); + for (; current < 2 * 60 * 60_000; current += 10_000) timers[0]!(); + expect(fake.children).toHaveLength(2); expect(supervisor.status()[0]!.state).toEqual(failed); await supervisor.stop(); }); +test("reconnect restarts a Home tunnel at once, from a failed state or over a live child", async () => { + const fake = fakeRunner(); + const timers: Array<() => void> = []; + let current = 0; + const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef"), record("client-initiated", "lnk_fedcba9876543210")); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner: fake.runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + }); + expect(await supervisor.reconnect("lnk_0123456789abcdef")).toBe(false); + supervisor.start(); + current = 5_000; + timers[0]!(); + fake.children[0]!.stderr = "Error: remote port forwarding failed for listen port 19002"; + fake.children[0]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + expect(supervisor.status()[0]!.state).toMatchObject({ kind: "failed", retryAt: 65_000 }); + current = 10_000; + expect(await supervisor.reconnect("lnk_0123456789abcdef")).toBe(true); + expect(fake.children).toHaveLength(2); + expect(supervisor.status()[0]!.state).toEqual({ kind: "connecting", since: 10_000 }); + current = 12_000; + expect(await supervisor.reconnect("lnk_0123456789abcdef")).toBe(true); + await expect(fake.children[1]!.child.exited).resolves.toBe(143); + expect(fake.children).toHaveLength(3); + expect(supervisor.status()[0]).toMatchObject({ state: { kind: "connecting", since: 12_000 }, pid: fake.children[2]!.child.pid }); + expect(await supervisor.reconnect("lnk_fedcba9876543210")).toBe(false); + expect(await supervisor.reconnect("lnk_0000000000000000")).toBe(false); + expect(fake.children).toHaveLength(3); + await supervisor.stop(); + expect(await supervisor.reconnect("lnk_0123456789abcdef")).toBe(false); +}); + +test("an old exit still draining after a reconnect never rewrites the new child's state", async () => { + const fake = fakeRunner(); + const timers: Array<() => void> = []; + let current = 0; + const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef")); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner: fake.runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + random: () => 0.5, + }); + supervisor.start(); + current = 5_000; + timers[0]!(); + const oldStderr = deferred(); + Object.defineProperty(fake.children[0]!.child, "stderr", { value: oldStderr.promise }); + fake.children[0]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + current = 6_000; + expect(await supervisor.reconnect("lnk_0123456789abcdef")).toBe(true); + expect(fake.children).toHaveLength(2); + fake.children[1]!.stderr = "client_loop: send disconnect: Broken pipe"; + fake.children[1]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + const reconnecting = supervisor.status()[0]!.state; + expect(reconnecting).toMatchObject({ kind: "reconnecting", since: 6_000, retryAt: 7_000 }); + oldStderr.resolve("Host key verification failed."); + await Promise.resolve(); + await Promise.resolve(); + expect(supervisor.status()[0]!.state).toEqual(reconnecting); + current = 7_000; + timers[0]!(); + expect(fake.children).toHaveLength(3); + await supervisor.stop(); +}); + +test("overlapping reconnects for one link restart it once and keep the new child supervised", async () => { + const fake = fakeRunner(); + const timers: Array<() => void> = []; + let current = 0; + const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef")); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner: fake.runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + }); + supervisor.start(); + current = 5_000; + timers[0]!(); + // The live child takes a while to exit after SIGTERM. + (fake.children[0]!.child as { kill: () => void }).kill = () => {}; + current = 8_000; + const first = supervisor.reconnect("lnk_0123456789abcdef"); + const second = supervisor.reconnect("lnk_0123456789abcdef"); + fake.children[0]!.resolve(143); + expect(await Promise.all([first, second])).toEqual([true, true]); + expect(fake.children).toHaveLength(2); + expect(supervisor.status()[0]).toMatchObject({ state: { kind: "connecting", since: 8_000 }, pid: fake.children[1]!.child.pid }); + current = 13_000; + timers[0]!(); + expect(supervisor.status()[0]!.state).toEqual({ kind: "connected", since: 13_000 }); + await supervisor.stop(); +}); + +test("a reload neither skips the reconnect backoff nor a failed link's retry delay", async () => { + const fake = fakeRunner(); + const timers: Array<() => void> = []; + let current = 0; + const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef")); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner: fake.runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + random: () => 0.5, + }); + supervisor.start(); + current = 5_000; + timers[0]!(); + fake.children[0]!.stderr = "client_loop: send disconnect: Broken pipe"; + fake.children[0]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + expect(supervisor.status()[0]!.state).toMatchObject({ kind: "reconnecting", retryAt: 6_000 }); + await supervisor.reload(); + expect(fake.children).toHaveLength(1); + current = 6_000; + timers[0]!(); + expect(fake.children).toHaveLength(2); + fake.children[1]!.stderr = "Error: remote port forwarding failed for listen port 19002"; + fake.children[1]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + expect(supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 6_000, reason: "forward", retryAt: 66_000, inFlight: false }); + await supervisor.reload(); + expect(fake.children).toHaveLength(2); + await supervisor.stop(); +}); + +test("the Home's -R supervisor keeps the retry cadence when ssh cannot be started", async () => { + const fake = fakeRunner(); + let spawnError: Error | null = null; + const runner: SshRunner = { + ...fake.runner, + spawnTunnel(argv) { + if (spawnError) throw spawnError; + return fake.runner.spawnTunnel(argv); + }, + }; + const timers: Array<() => void> = []; + let current = 0; + const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef")); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + }); + supervisor.start(); + current = 5_000; + timers[0]!(); + fake.children[0]!.stderr = "Error: remote port forwarding failed for listen port 19002"; + fake.children[0]!.resolve(255); + await Promise.resolve(); + await Promise.resolve(); + spawnError = new Error("spawn ssh ENOENT"); + current = 65_000; + timers[0]!(); + expect(supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 5_000, reason: "forward", retryAt: 125_000, inFlight: false }); + spawnError = null; + current = 125_000; + timers[0]!(); + expect(fake.children).toHaveLength(2); + await supervisor.stop(); +}); + +test("an exit whose stderr cannot be read still reaches the reducer", async () => { + const fake = fakeRunner(); + const timers: Array<() => void> = []; + let current = 0; + const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef")); + const supervisor = createLinkSupervisor({ + readStore: () => store, + runner: fake.runner, + pidfileDir: "/tmp/opencodex-link-supervisor-test", + now: () => current, + setTimer: callback => { timers.push(callback); return 1 as unknown as ReturnType; }, + clearTimer: () => {}, + random: () => 0.5, + }); + supervisor.start(); + current = 5_000; + timers[0]!(); + // The runner rejects stderr once it passes its size cap. + const unreadable = Promise.reject(new Error("ssh stderr exceeded its limit")); + unreadable.catch(() => {}); + Object.defineProperty(fake.children[0]!.child, "stderr", { value: unreadable }); + fake.children[0]!.resolve(255); + for (let turn = 0; turn < 4; turn += 1) await Promise.resolve(); + expect(supervisor.status()[0]!.state).toMatchObject({ kind: "reconnecting", since: 5_000, retryAt: 6_000 }); + await supervisor.stop(); +}); + test("reaps a pidfile only after an exact Linux argv match", () => { const fake = fakeRunner(); const store = baseStore(record("hub-initiated", "lnk_0123456789abcdef")); diff --git a/tests/server/link-management-routes.test.ts b/tests/server/link-management-routes.test.ts index 8ddd98f18be..5bab8f703d2 100644 --- a/tests/server/link-management-routes.test.ts +++ b/tests/server/link-management-routes.test.ts @@ -489,6 +489,62 @@ describe("link management routes", () => { expect(h.store.links).toHaveLength(1); }); + test("reconnect restarts a Home-initiated tunnel for either principal and refuses everything else", async () => { + temp = mkdtempSync(join(tmpdir(), "ocx-link-reconnect-")); + const h = harness(); + const reconnected: string[] = []; + let restartable = true; + const deps: ManagementApiDeps = { + ...h.deps, + linkSupervisor: () => ({ ...supervisor(h.events), reconnect: async (id: string) => { reconnected.push(id); return restartable; } }), + }; + h.deps.writeLinkStore!({ ...h.store, links: [{ + id: "lnk_0123456789abcdef", + alias: "child", + direction: "hub-initiated", + hostKeyFingerprint: "SHA256:abcdefghijklmnop", + tunnelPort: 2200, + apiKeyId: "key-1", + createdAt: "2026-09-25T00:00:00.000Z", + }, { + id: "lnk_fedcba9876543210", + alias: "home", + direction: "client-initiated", + hostKeyFingerprint: null, + tunnelPort: 2201, + apiKeyId: "key-2", + createdAt: "2026-09-25T00:00:00.000Z", + }] }); + const path = "/api/link/lnk_0123456789abcdef/reconnect"; + const ok = await call(path, "POST", undefined, deps, "admin-token", true, null, true, h.config); + expect(ok?.status).toBe(200); + expect(await ok!.json()).toEqual({ linkId: "lnk_0123456789abcdef" }); + expect(h.events).toContain("supervisor"); + expect((await call(path, "POST", undefined, deps, "gui-session", true, "pairing", true, h.config))?.status).toBe(200); + expect(reconnected).toEqual(["lnk_0123456789abcdef", "lnk_0123456789abcdef"]); + expect((await call(path, "POST", undefined, deps, "admin-token", false, null, true, h.config))?.status).toBe(403); + expect((await call(path, "POST", undefined, deps, "admin-token", true, "tailscale-identity", true, h.config))?.status).toBe(403); + const childOwned = await call("/api/link/lnk_fedcba9876543210/reconnect", "POST", undefined, deps, "admin-token", true, null, true, h.config); + expect(childOwned?.status).toBe(409); + expect(await childOwned!.json()).toMatchObject({ error: { code: "link_not_home_initiated" } }); + const missing = await call("/api/link/lnk_0000000000000000/reconnect", "POST", undefined, deps, "admin-token", true, null, true, h.config); + expect(missing?.status).toBe(404); + expect(await missing!.json()).toMatchObject({ error: { code: "link_not_found" } }); + expect((await call("/api/link/not-a-link/reconnect", "POST", undefined, deps, "admin-token", true, null, true, h.config))?.status).toBe(400); + h.setListenerState("failed"); + const unbound = await call(path, "POST", undefined, deps, "admin-token", true, null, true, h.config); + expect(unbound?.status).toBe(503); + expect(await unbound!.json()).toMatchObject({ error: { code: "listener_unavailable" } }); + expect(reconnected).toHaveLength(2); + h.setListenerState("listening"); + restartable = false; + const refused = await call(path, "POST", undefined, deps, "admin-token", true, null, true, h.config); + expect(refused?.status).toBe(503); + expect(await refused!.json()).toMatchObject({ error: { code: "link_unavailable" } }); + expect(reconnected).toHaveLength(3); + expect(h.store.links).toHaveLength(2); + }); + test("issue starts the supervisor once a recovered listener binds", async () => { temp = mkdtempSync(join(tmpdir(), "ocx-link-issue-recover-")); const h = harness();