Skip to content

fix: retry unacked events over an open WebSocket - #83

Merged
Skeyelab merged 1 commit into
masterfrom
fix/ws-redeliver
Sep 22, 2026
Merged

Skeyelab merged 1 commit into
masterfrom
fix/ws-redeliver

Conversation

@Skeyelab

Copy link
Copy Markdown
Member

Problem

Found during the Hermes plugin rollout. A connected node only gets an event when it first arrives (ingress push) or when the node authenticates (pushPendingEventsForNode). If the node doesn't ack (in the rollout, the local Hermes route returned 404), the event stays leased and is never retried while the connection stays open. Two test events sat stuck for 4+ minutes after their 60s lease expired.

Before #81 the relay reconnected every ~45s, which quietly re-sent stuck events on each new connection. The keepalive made connections stable, so the gap is now real.

Change

app/lib/ws/node-socket.ts: on each 30s keepalive tick, call pushPendingEvents(ws, nodeId), the same function used at auth, which selects pending plus lease-expired leased events and re-leases them. So nothing is sent twice inside a lease window. A tick that terminates an unresponsive node doesn't re-send.

Tests (written first)

  • re-sends on each keepalive tick
  • nothing is re-sent to a node being terminated
  • a failed re-send is logged, not thrown

yarn vitest run: 245/245. Lint clean, no type errors outside tests.

A connected node only received events when they first arrived or when it
authenticated. If a delivery failed (the node didn't ack, e.g. its local
webhook returned 404), the event sat in `leased` until the node reconnected.
Before the keepalive (#81) the ~45s reconnect cycle masked this; with stable
connections an unacked event could wait indefinitely.

Re-send pending and lease-expired events on every 30s keepalive tick, using
the existing pushPendingEventsForNode (which re-leases, so nothing is sent
twice inside a lease window). Nothing is re-sent to a node being terminated.
@strix-security

strix-security Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Strix Security Review

No security issues found.

Review summary

The PR adds retry of pending/lease-expired events on each of a node's WebSocket keepalive ticks, reusing the existing pushPendingEvents path rather than introducing new logic. The changed connection handler was reviewed together with the wired dependencies in the server entrypoint. All database interactions remain parameterized and scoped to the authenticated node, no authorization or input-validation controls were removed, the unresponsive-node termination path correctly skips re-sending, and retries are bounded by an attempt count. No security vulnerabilities were identified in the changed code.

Updated for 071a018.


Reviewed by Strix
Re-run review · Configure security review settings

@Skeyelab
Skeyelab merged commit e5e696f into master Sep 22, 2026
3 checks passed
@Skeyelab
Skeyelab deleted the fix/ws-redeliver branch September 22, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant