Skip to content

feat(executor): reuse SSH connections instead of dialing per command - #5

Merged
Kritoooo merged 2 commits into
mainfrom
feat/ssh-connection-reuse
Jul 2, 2026
Merged

Kritoooo merged 2 commits into
mainfrom
feat/ssh-connection-reuse

Conversation

@Kritoooo

@Kritoooo Kritoooo commented Jul 2, 2026

Copy link
Copy Markdown
Member

Summary

Previously both transports opened a fresh SSH connection (TCP dial + handshake + auth) for every command and closed it immediately. This PR adds connection reuse to both:

Native transport (golang.org/x/crypto/ssh)

  • Connection pool: concurrency-safe per-executor pool keyed by resolved target shape (host, port, user, ProxyJump, identity files); concurrent dials for the same key are single-flighted, failures are broadcast to all waiters.
  • Keepalive: keepalive@openssh.com heartbeat per pooled connection (default 30s, configurable) under a short timeout; a failed/hung heartbeat evicts and closes the client.
  • Liveness: pooled NewSession is bounded by ConnectTimeout; session.Run is ctx-aware (closes transport on cancel, no command timeout). A stale pooled client is evicted and transparently re-dialed once — never after session.Run has started (double-execution risk).
  • Concurrency safety: in-flight sessions are refcounted per pool entry; suspicious post-Run errors de-pool immediately but defer Close until in-flight sessions drain. MaxSessions channel-open rejections fall back to a one-off unpooled dial.
  • ProxyJump closer chains are preserved; Close() drains the pool; Probe stays one-shot.

Shell transport (system ssh)

  • OpenSSH multiplexing: ControlMaster=auto + ControlPersist=60s + per-target ControlPath under a stable per-user 0700 dir (UserCacheDir/agentssh/ssh-mux), so masters are reused across invocations; Close() leaves live sockets alone.
  • ssh_config alias targets get no injected mux options — the user's ssh_config governs.
  • Falls back to plain argv when the ControlPath would exceed the unix-socket sun_path limit.

Config & lifecycle

  • New config keys: ssh.multiplexing, ssh.control_persist, ssh.keepalive_interval.
  • Normalized OS is cached per host (pool entry / mux key) instead of re-probing uname -s every command.
  • Close() error added to the Executor interface.

Review

The initial implementation went through a high-effort code review (8 finder angles, adversarial verification): 10 findings confirmed (3 CONFIRMED / 7 PLAUSIBLE) — including indefinite hangs on half-open pooled connections, a keepalive that itself blocked forever, evictions killing concurrent in-flight sessions, and orphaned ControlPersist masters. All 10 are fixed in the second commit, plus a ctx-cancel output-buffer data race caught by the race detector.

Testing

  • gofmt -l . clean, go build ./..., go vet ./... pass
  • go test -count=1 ./... — all packages green
  • go test -count=1 -race ./... — full suite green under the race detector
  • 20+ new tests: pool reuse, single-flight + error broadcast, dead-client reconnect, keepalive timeout eviction, in-flight-drain deferred close, MaxSessions fallback, ControlMaster argv, stable control dir surviving Close, alias non-injection, ControlPath length fallback, OS caching, config wiring.

🤖 Generated with Claude Code

Kritoooo and others added 2 commits July 1, 2026 19:51
Both transports previously opened a fresh connection (dial + handshake +
auth) for every command and closed it immediately. Add connection reuse.

Native transport (golang.org/x/crypto/ssh):
- Concurrency-safe per-host connection pool keyed by resolved target shape
  (host, port, user, ProxyJump, identity files); concurrent dials for the
  same key are single-flighted, different keys proceed independently.
- Pooled clients keep the existing nativeClient closer chain, so direct
  clients, ProxyJump tunnels, and jump clients still close in order.
- keepalive@openssh.com heartbeat per pooled connection (default 30s); a
  failed heartbeat evicts and closes the client.
- Transparent one-shot reconnect when a stale pooled client fails
  NewSession. Never retries after session.Run has started, preserving
  remote side-effect and exit-code semantics.
- NativeExecutor.Close() drains the pool; Probe stays one-shot so
  diagnostics never leave a long-lived socket.

Shell transport (system ssh binary):
- Enable OpenSSH multiplexing by default: ControlMaster=auto,
  ControlPersist=60s, stable per-target ControlPath under a 0700 per-run
  temp dir; command and hidden OS probe share the socket.
- SSHOptions allows disabling or customizing persist/dir; BuildSSHArgv
  stays backward-compatible, multiplexing rides on BuildSSHArgvWithOptions.
- SSHExecutor.Close() removes the owned temp dir / tracked sockets.

Tests cover pool reuse, concurrent single-flight dialing, reconnect on a
dead pooled client, keepalive+close, closer-chain shutdown, ControlMaster
argv construction, shared command/probe socket, disable path, and temp-dir
cleanup. gofmt/go build/go vet/go test all pass.

Implemented via Codex.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correctness (native pool):
- Bound pooled NewSession with ConnectTimeout and make session.Run
  ctx-aware (close transport on cancel, no command timeout), so a
  half-open pooled connection fails fast instead of hanging until
  kernel TCP timeout. Wait for the Run goroutine after cancel so
  output buffers are never read while its copies still write.
- Run keepalive SendRequest under a short timeout; timeout/error
  evicts and closes, restoring dead-connection detection.
- Track in-flight sessions per pool entry: suspicious post-Run errors
  de-pool immediately but defer Close until in-flight sessions drain,
  so one command's channel-level error no longer kills concurrent
  sessions. Provably-dead paths still close immediately.
- Broadcast dial errors to all waiters via entry.dialErr, so N
  concurrent commands to an unreachable host fail in parallel.
- Fall back to a one-off unpooled dial when NewSession is rejected
  with Prohibited/ResourceShortage (sshd MaxSessions), without
  evicting the healthy shared client.

Shell transport (multiplexing):
- Use a stable per-user control dir (UserCacheDir/agentssh/ssh-mux,
  0700); Close no longer removes live sockets, so masters are
  actually reused across invocations instead of leaking orphaned
  authenticated masters with unlinked sockets.
- Do not inject ControlMaster/ControlPath/ControlPersist for
  ssh_config alias targets - the user's ssh_config governs.
- Fall back to the non-multiplexed argv when the ControlPath would
  exceed ~90 bytes (sun_path limit), caching the decision.

Config & lifecycle:
- Expose ssh.multiplexing / ssh.control_persist /
  ssh.keepalive_interval in config and wire through newExecutor.
- Cache the normalized OS per host (pool entry / mux key) instead of
  re-probing uname -s on every command.
- Add Close() error to the Executor interface; drop the ad-hoc
  io.Closer type assertion in runDirect.

All fixes verified with gofmt, go vet, and go test -count=1 -race
./... (full suite green). Implemented via Codex; the ctx-cancel
buffer race was found by the race detector and fixed on top.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Kritoooo
Kritoooo merged commit a621545 into main Jul 2, 2026
2 checks passed
@Kritoooo
Kritoooo deleted the feat/ssh-connection-reuse branch August 31, 2026 05:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant