Skip to content

[Bug]: Home-initiated Remote Link never recovers after a network drop (failed: timeout until ocx restart) #6899

Closed

Description

@WOULDU-pres

Client or integration

OpenCodex dashboard

Area

Proxy and routing

Summary

A Home-initiated Remote Link (the ssh -R tunnel a Home starts after adding a Child from its dashboard) breaks after a network drop and never comes back by itself. About five minutes after the drop the link reads failed with reason timeout. No ssh child is running and nothing tries again. The dashboard's Retry button only re-reads /api/link/status, so it shows the same failed. The only way out is ocx restart on the Home.

I expected the Home to reconnect once the network returns, the way a Child-initiated link already does since #5973 (CLIENT_TUNNEL_RETRY_POLICY).

Two defects in src/link/supervisor.ts add up to this:

  1. A healthy reconnect gets killed. After a drop the state goes connected → reconnecting. When the retry's ssh comes up, only an authenticated GET /v1/catalog from the Child promotes it (notifyAuthenticatedRequest). The five-second grace applies to connecting only. An idle Child sends no catalog request, so the working tunnel stays reconnecting. Five minutes after the drop (FAILED_AFTER_MS), the tick turns it into failed: timeout and SIGTERMs the live ssh child. A one-second blip is enough.
  2. failed is terminal on the Home. The Home calls reduceTunnel without a retry policy, so a failed link is never retried. That covers timeout, and also forward when the Child's sshd still holds the port of the session that just died (remote port forwarding failed). The design note devlog/_fin/260925_remote_home_child_link/003_decisions.md (K19) deferred the Home side to a later PR, and tests/clients/link-supervisor.test.ts still pins the old behaviour ("the Home's -R supervisor keeps the old policy").

A fix PR follows.

Reproduction

Deterministic with the supervisor's injectable clock and runner, the same harness as tests/clients/link-supervisor.test.ts:

  1. Start a supervisor with one hub-initiated link. At t=5s the grace promotes it to connected.
  2. The ssh child exits with client_loop: send disconnect: Broken pipe. The state becomes reconnecting with the retry due at t=6s.
  3. At t=6s the retry spawns and stays alive. No catalog request arrives.
  4. Tick once a second until t=400s.

Result on 2.82.0 and on dev at 03b865e:

state: {"kind":"failed","since":305000,"reason":"timeout"}  retry child killed: true  children: 2

No further spawn happens after that.

Real setup: Home on macOS, Child on WSL2, reached over Tailscale. Every Wi-Fi drop left the link failed until ocx restart.

Version

2.82.0 on both Home and Child; also reproduced on dev at 03b865e

Operating system

Home: macOS 26.6.2. Child: Ubuntu 24.04.4 on WSL2 (6.6.87.2-microsoft-standard-WSL2)

Provider and model

No response

Logs or error output

# Home, after the network was back
$ ocx link status
{"role":"home","listener":{"state":"listening","port":50665},"links":[{"id":"lnk_…","alias":"wsl","direction":"hub-initiated","state":"failed","since":"2026-10-10T13:06:04.617Z","reason":"timeout","tunnelPort":37369}],"child":null}

# Child: the reverse forward is gone
$ ss -ltn | grep 37369 || echo "37369 not listening"
37369 not listening

# Home: only a restart brings it back
$ ocx restart && ocx link status
{"role":"home",...,"links":[{...,"state":"connected","since":"2026-10-10T14:36:28.277Z","reason":null,"tunnelPort":37369}],"child":null}

Screenshots and supporting files

No response

Redacted configuration

No response

Checks

  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.

Activity

  1. added
    bugSomething isn't working
    cliCLI, config inject, packaging flags
    proxyHTTP proxy, routing, reverse-proxy / management auth
    on Oct 10, 2026
  2. lidge-jun commented on Oct 10, 2026

    @lidge-jun
    Owner

    Fixed on dev by #6915 (squash 09b3e76b97), which carries #6900 by @WOULDU-pres. After a network drop, a Home-initiated Remote Link now recovers, and Retry reconnects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcliCLI, config inject, packaging flagsproxyHTTP proxy, routing, reverse-proxy / management auth

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions