Skip to content

Migrate SshClient from blocking ssh2 to an async SSH library (russh) #198

Description

@v0l

Problem

SshClient (lnvps_api/src/ssh_client.rs) is built on ssh2 = "0.9", a binding to the synchronous C library libssh2. Its methods (connect, execute, ...) are declared async, but the actual SSH protocol work (handshake, channel.exec, read_to_string) is blocking socket I/O — the async is essentially cosmetic (only TcpStream::connect truly yields).

This bit us with OS image downloads: a blocking wget/decompress over SSH held a tokio worker thread for the full duration (~2 min for a FreeBSD image), freezing the entire runtime — concurrent host Proxmox API calls, the nostr relay, and the Telegram poller all timed out.

Current mitigation

SshClient::run_command now runs the blocking connect + exec on the blocking thread pool via tokio::task::spawn_blocking, and download_images_on_hosts spawns one task per host. This fixes the freeze and restores cross-host parallelism, but:

  • It's a workaround — every long SSH command still consumes a dedicated OS thread from the blocking pool (max 512).
  • Two SSH code paths now coexist: the async-but-blocking connect/execute (still used by disk import, cloud-init snippet writes, terminal proxy, Linux router driver) and the spawn_blocking run_command.

Proposal

Migrate SshClient to a natively-async SSH library so all SSH I/O is genuinely non-blocking and there is a single, consistent code path.

  • Preferred: russh — pure-Rust, tokio-native, no C dependency.
  • Alternatives: async-ssh2-lite (async reactor over the same libssh2, smaller migration) or openssh (shells out to system ssh).

Scope / affected callers

All users of SshClient need porting + testing:

  • lnvps_api/src/host/proxmox.rs — image download (download_image_ssh), decompression (decompress_image), checksum verification (verify_image_checksum), cloud-init vendor snippet writes, import_disk_image, terminal proxy (connect_terminal / tunnel_unix_socket, non-blocking channel bridge)
  • Linux router driver (SSH-managed router, linux-ssh feature)
  • scp_upload (SFTP) and set_blocking / tunnel_unix_socket semantics need equivalents in the new lib

Feature flags to update: linux-ssh, proxmox (currently dep:ssh2).

Acceptance criteria

  • SshClient (or its replacement) exposes fully-async command execution, SFTP upload, and unix-socket tunnelling (for the terminal proxy) with no spawn_blocking and no blocking socket reads on async worker threads.
  • spawn_blocking shim (run_command) removed once callers are ported.
  • ssh2 dependency dropped.
  • Existing behaviour preserved (image download/verify/decompress, disk import, cloud-init snippets, terminal proxy, Linux router).

Context

Introduced alongside the compressed-OS-image download fix (SSH-based wget + decompress). See the OS image download changes in lnvps_api/src/host/proxmox.rs and download_images_on_hosts in lnvps_api/src/worker.rs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthost:proxmoxrefactorInternal restructuring without behaviour change

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions