Problem
SshClient (lnvps_api/src/ssh_client.rs) is built on ssh2 = "0.9", a binding to the synchronous C library libssh2. Its methods (connect, execute, ...) are declared async, but the actual SSH protocol work (handshake, channel.exec, read_to_string) is blocking socket I/O — the async is essentially cosmetic (only TcpStream::connect truly yields).
This bit us with OS image downloads: a blocking wget/decompress over SSH held a tokio worker thread for the full duration (~2 min for a FreeBSD image), freezing the entire runtime — concurrent host Proxmox API calls, the nostr relay, and the Telegram poller all timed out.
Current mitigation
SshClient::run_command now runs the blocking connect + exec on the blocking thread pool via tokio::task::spawn_blocking, and download_images_on_hosts spawns one task per host. This fixes the freeze and restores cross-host parallelism, but:
- It's a workaround — every long SSH command still consumes a dedicated OS thread from the blocking pool (max 512).
- Two SSH code paths now coexist: the async-but-blocking
connect/execute (still used by disk import, cloud-init snippet writes, terminal proxy, Linux router driver) and the spawn_blocking run_command.
Proposal
Migrate SshClient to a natively-async SSH library so all SSH I/O is genuinely non-blocking and there is a single, consistent code path.
- Preferred: russh — pure-Rust, tokio-native, no C dependency.
- Alternatives: async-ssh2-lite (async reactor over the same libssh2, smaller migration) or openssh (shells out to system
ssh).
Scope / affected callers
All users of SshClient need porting + testing:
lnvps_api/src/host/proxmox.rs — image download (download_image_ssh), decompression (decompress_image), checksum verification (verify_image_checksum), cloud-init vendor snippet writes, import_disk_image, terminal proxy (connect_terminal / tunnel_unix_socket, non-blocking channel bridge)
- Linux router driver (SSH-managed router,
linux-ssh feature)
scp_upload (SFTP) and set_blocking / tunnel_unix_socket semantics need equivalents in the new lib
Feature flags to update: linux-ssh, proxmox (currently dep:ssh2).
Acceptance criteria
Context
Introduced alongside the compressed-OS-image download fix (SSH-based wget + decompress). See the OS image download changes in lnvps_api/src/host/proxmox.rs and download_images_on_hosts in lnvps_api/src/worker.rs.
Problem
SshClient(lnvps_api/src/ssh_client.rs) is built on ssh2 = "0.9", a binding to the synchronous C library libssh2. Its methods (connect,execute, ...) are declaredasync, but the actual SSH protocol work (handshake,channel.exec,read_to_string) is blocking socket I/O — theasyncis essentially cosmetic (onlyTcpStream::connecttruly yields).This bit us with OS image downloads: a blocking
wget/decompress over SSH held a tokio worker thread for the full duration (~2 min for a FreeBSD image), freezing the entire runtime — concurrent host Proxmox API calls, the nostr relay, and the Telegram poller all timed out.Current mitigation
SshClient::run_commandnow runs the blocking connect + exec on the blocking thread pool viatokio::task::spawn_blocking, anddownload_images_on_hostsspawns one task per host. This fixes the freeze and restores cross-host parallelism, but:connect/execute(still used by disk import, cloud-init snippet writes, terminal proxy, Linux router driver) and thespawn_blockingrun_command.Proposal
Migrate
SshClientto a natively-async SSH library so all SSH I/O is genuinely non-blocking and there is a single, consistent code path.ssh).Scope / affected callers
All users of
SshClientneed porting + testing:lnvps_api/src/host/proxmox.rs— image download (download_image_ssh), decompression (decompress_image), checksum verification (verify_image_checksum), cloud-init vendor snippet writes,import_disk_image, terminal proxy (connect_terminal/tunnel_unix_socket, non-blocking channel bridge)linux-sshfeature)scp_upload(SFTP) andset_blocking/tunnel_unix_socketsemantics need equivalents in the new libFeature flags to update:
linux-ssh,proxmox(currentlydep:ssh2).Acceptance criteria
SshClient(or its replacement) exposes fully-async command execution, SFTP upload, and unix-socket tunnelling (for the terminal proxy) with nospawn_blockingand no blocking socket reads on async worker threads.spawn_blockingshim (run_command) removed once callers are ported.ssh2dependency dropped.Context
Introduced alongside the compressed-OS-image download fix (SSH-based
wget+ decompress). See the OS image download changes inlnvps_api/src/host/proxmox.rsanddownload_images_on_hostsinlnvps_api/src/worker.rs.