Skip to content

Keep reconnecting after stalls and unexpected errors - #2

Open
jaltmayerpizzorno wants to merge 1 commit into
PoolsideStan:mainfrom
jaltmayerpizzorno:reconnect-robustness
Open

jaltmayerpizzorno wants to merge 1 commit into
PoolsideStan:mainfrom
jaltmayerpizzorno:reconnect-robustness

Conversation

@jaltmayerpizzorno

Copy link
Copy Markdown
Contributor

Problem

After running fine for days, the Home Assistant integration lost its connection to my Attendant at 3:40 AM and never got it back. Every entity went unavailable, but the config entry still showed as loaded, and nothing was logged at warning level or above. Reloading the integration reconnected immediately, so the controller itself was reachable.

The connection was lost without a log entry, so I can't tell which path it took. Reading client.py, though, there are two ways the client can end up permanently disconnected without logging anything:

  1. Unbounded connection steps. ws_connect, the Noise handshake (_read_binary waits on ws.receive()), and the initial Device.getStatus in async_refresh_status have no timeout. A controller that accepts the socket but doesn't answer (for example while it restarts) stalls _async_connect_once forever. Because the reconnect loop awaits that call, the loop stalls with it. The ping loop can't help, because it only starts after a successful connect.
  2. The reconnect loop exits on unexpected exceptions. _async_reconnect_loop only catches PoolsideConnectionError and PoolsideAuthError. Any other exception ends the loop, and nothing ever retrieves the error because the task is kept on the client. Examples are a Noise error from read_message during the handshake, a json.loads failure in the receive loop, or a status listener raising. The client is left disconnected for good.

Fix

  • Bound the connect and the handshake by PING_TIMEOUT, and raise PoolsideConnectionError on timeout so the loop retries with backoff.
  • Bound Device.getStatus in async_refresh_status the same way. A timeout is logged like the other errors there, and the ping loop then detects the dead connection.
  • In _async_reconnect_loop, log and survive exceptions from the receive task and from reconnect attempts, and close the old socket before retrying.

Tests

Four new tests. Each fails on main and passes with this change:

  • test_async_connect_times_out_on_silent_controller: a real websocket server that accepts the connection but never answers the handshake.
  • test_refresh_status_times_out
  • test_reconnect_loop_survives_unexpected_error
  • test_reconnect_loop_survives_failed_receive_loop

The full suite passes (36 tests), and ruff check / ruff format --check are clean.

🤖 Generated with Claude Code

Two ways the client could end up disconnected for good, with every entity
unavailable and nothing in the log:

- The websocket connect, the Noise handshake and the initial
  Device.getStatus were unbounded. A controller that accepts the socket but
  never answers (e.g. while it is restarting) stalled the attempt, and with
  it the reconnect loop, forever. Bound each by PING_TIMEOUT; an unanswered
  getStatus is logged and the ping loop then catches a dead connection.

- The reconnect loop only caught PoolsideConnectionError and
  PoolsideAuthError. Any other exception from a reconnect attempt (e.g. a
  malformed handshake message) or from the receive loop ended the loop
  silently. Log it and keep retrying with backoff, closing the old socket
  first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant