Skip to content

Make local sessions recover from lost connections and failed setup - #50

Open
idomp wants to merge 10 commits into
OnFreund:masterfrom
idomp:local-resilience
Open

idomp wants to merge 10 commits into
OnFreund:masterfrom
idomp:local-resilience

Conversation

@idomp

@idomp idomp commented Sep 17, 2026

Copy link
Copy Markdown

Local (RiscoLocal) sessions could fail in ways that only a manual reload of the consumer fixed. A dropped connection left commands waiting and the listener running. A failed setup left the panel session open. A late reply could answer a different command. This PR makes the local client detect lost or unusable sessions, close them cleanly, and report the loss to the consumer. It also paces the consumer's next connect() to what the panel accepts.

It is split into 10 commits, one concern each, and is meant to be reviewed commit by commit. Each behaviour change comes with its tests, and the suite passes at every commit.

# Commit Change
1 Fix local frame boundaries and validate CRC before decoding Preserve frame boundaries; check the CRC on the received bytes before decoding
2 Close failed local connections and bound session cleanup End failed sessions, fail pending commands, clean up failed or cancelled connects, bound disconnect
3 Close local sessions after failed keep-alives or unreadable frames Close a session that stops producing usable replies
4 Report local connection loss and retain errors for consumers Report connection loss consistently; keep errors until a handler is registered
5 Keep command IDs owned until replies or session teardown Keep an unanswered ID reserved so a late reply cannot answer a later command
6 Make local discovery recover lost answers without inventing status Retry lost reads; use reported placeholders for missing details; never make up a status
7 Pace local reconnects per panel across client instances Back off after repeated short sessions, per panel, across client objects
8 Treat only identified N06 login refusals as authorization failures Raise UnauthorizedError only for an N06 that answers the login
9 Run the regression suite on Python 3.11 through 3.14 Add a test workflow
10 Fail CI on leaked resources and unhandled asyncio diagnostics Fail that workflow on -X dev diagnostics

Library: +711 / -149 lines in pyrisco/ and README.md. Tests and CI: +4,053 lines.

Related PRs

This covers the fix proposed in #44 (cancel pending futures and stop the listener at end of stream). Credit to @rhalutz for the original fix; that PR's test file is not included here. The keep-alive changes overlap #22.

Field failures that motivated it

These happened on one RISCO LightSYS 2 (RP432M firmware 6.07, RP512IP), used through Home Assistant 2026.9 at concurrency 1:

  • 2026-09-10, after a Home Assistant restart: the integration reported loaded, but none of its 122 entities worked. Zone and partition status handling raised KeyError, and then the keep-alive got id-less N05 answers, 8,021 times in 15 hours, until a manual reload.
  • 2026-09-11, after another restart: connect() failed when FSVER? was answered N05, and the panel then refused TCP connections for about 30 minutes.

How it fails today

Reproduced on that installation with released 0.8.1 and Home Assistant 2026.9's own risco integration, by black-holing the panel connection at the firewall, so the socket stays open and the panel's answers never arrive:

21:42:22  connection black-holed (no FIN, no RST)
21:42:33  WARNING [homeassistant.components.risco] Risco keep-alive timeout for entry <serial> (host: <panel>)
21:42:48  WARNING [homeassistant.components.risco] Risco keep-alive timeout for entry <serial> (host: <panel>)
21:43:03  WARNING [homeassistant.components.risco] Risco keep-alive timeout for entry <serial> (host: <panel>)
          ... one every 15 s, 42 in the ten minutes watched
21:53:01  config entry still "loaded"; all 122 entities still showing their last values,
          both partitions still "disarmed"; no new connection attempted to the panel

The session is never given up, so the consumer is never told and never reconnects. Only a manual reload or a restart recovers. The 2026-09-10 incident above is the same shape, with N05 answers in place of timeouts.

The other half of that incident also reproduces against released 0.8.1, with a stand-in panel object that answers N05 to one zone's status query during connect():

connect() returned normally; zones discovered: [1, 2, 3]

Traceback (most recent call last):
  File "pyrisco/local/risco_local.py", line 228, in _listen
    self._zone_status(int(command[4:]), result)
  File "pyrisco/local/risco_local.py", line 182, in _zone_status
    z = self._zones[zone_id]
KeyError: 4

Setup reports success without zone 4, and every status the panel pushes for it reaches the consumer as that KeyError. With this PR, the zone is read again; if the panel still refuses it, the zone is left out and its pushes are reported once as a status for an unknown zone (test_a_zone_whose_status_is_refused_twice_is_left_out_and_reported).

Behaviour changes to review

  • Errors (2, 4): CommunicationError (a subclass of OperationError) is raised for lost or malformed answers. ConnectionLostError also subclasses ConnectionResetError, so existing reconnect checks keep working. Communication failures during setup raise CannotConnectError. Errors reported while no error handler is registered are kept (the last 20) and delivered to the first handler added. The backlog is reset at each connect(), and setup errors that the connect recovered from are not reported.
  • Disconnect (2): DCN has a 2 s deadline that includes waiting for a command slot, so a busy or dead session can close without sending it. A 5 s per-panel cooldown moves the wait from disconnect() to the next connect().
  • Session health (3): a session is closed after 3 consecutive unusable keep-alive answers or 2 consecutive unreadable frames. It is closed at once if its first frame is unreadable and already encrypted, which means the panel still holds a previous session's encryption state. Both thresholds are policy choices.
  • Command IDs (5): an unanswered ID stays reserved until its reply arrives or the session ends, with no time limit. If all 49 are held, commands fail until the keep-alive closes the session. Id-less errors no longer rewind the ID. Controls are never resent automatically, because some (such as bypass) toggle state.
  • Discovery (6): a lost answer while reading a zone or partition, or one of its details, is asked again once. A lost answer to PNLCNF, FSVER?, PNLSERD or SSTT? fails the connect with CannotConnectError. A missing label, partition or group gets a placeholder and is reported to the error handlers; a placeholder partition or group reads as "in none", not as "unknown". A zone whose status is refused twice is left out until the next connect. Setup now fails without a usable system status or without an enabled partition.
  • Reconnect pacing (7): pacing is process-wide per host and port, so it also affects configuration checks and other clients of the same panel. After repeated sessions lost within 120 s, the delay doubles from 5 s, capped at 300 s. It resets after a session that lasts 120 s, or when more than 14 minutes pass between closes. A wait longer than the base delay raises a retryable CannotConnectError immediately, instead of holding connect() open.
  • Login (8): only an N06 carrying the login's command ID raises UnauthorizedError. Other refusals, unexpected replies and timeouts are retryable. An id-less N06 therefore times out as a retryable failure.
  • Framing (1): invalid text bytes become replacement characters instead of invalidating the frame. Receiving plaintext no longer disables encryption for later commands.
  • CI (9, 10): a new Tests workflow runs on pull requests and master (Ubuntu, Python 3.11-3.14, -X dev). It fails on asyncio or resource diagnostics (leaked sockets, never-awaited coroutines, unhandled task exceptions) even when unittest passes.

Testing

  • Unit tests, plus end-to-end tests over TCP against a scripted encrypted panel (tests/scripted_panel.py), each added with the commit it tests: 220 tests at the last commit.
  • Every commit: Python 3.12 on Windows. The last commit: Python 3.11, 3.12, 3.13 and 3.14 on Linux and on Windows, each followed by the diagnostics check. The new workflow has not yet run on GitHub.
  • Installed as a package into Home Assistant 2026.9.2, this change still triggers the reload that its released risco integration does on a lost connection, with that integration unchanged. That check needs Home Assistant, so it lives outside this repository.
  • On the installation above, with all commits, through Home Assistant 2026.9. Its risco integration was adapted for these runs, so that it disconnects when Home Assistant stops or setup fails. Like the released integration, it reloads on a reported connection loss.
    • Repeated Home Assistant restarts reconnected with no manual action, with DCN sent at shutdown.
    • Connections black-holed at the firewall recovered with no manual action every time: the integration reloaded on the reported loss, and connect() paced the retries. That includes a cut in the middle of discovery and three cuts in a row, where the reconnect delay grew to 10 s and then 20 s.
    • The keep-alive detected a cut in a running session in 40-45 s. In the timed runs, entities were working again 80-135 s after the cut.
    • After a black-holed connection, the panel ignored new connections until it reset the old one, 40-110 s later.
    • Replies drop the * from the query key (ZSTT*1? is answered ZSTT1=...), and a reply the panel had withheld arrived on the next connection.
  • Separately, a raw protocol probe on the same panel sent one wrong access code. The panel answered N06 carrying the login's command ID.

These results come from one panel model and firmware. They do not validate every policy above on other panels.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant