fix(pegboard-envoy): retry failed actor event dispatch instead of dropping events

[eersnington]

Retry a failed actor event dispatch instead of dropping every later event for that actor.

 channel_handler
   recv batch
-  dispatch_events() fails → log, break     // later events dropped, including "stopped"
+  dispatch_events() fails → log, backoff, retry same batch
+  cancelled on shutdown → return           // envoy resends unacked events on reconnect

• Problem: In actor_event_demuxer.rs, one dispatch_events error (for example UDB "max number of transaction retries reached") ended the handler loop. The actor never got its "stopped" event, so every later wake waited forever.
• Fix: Retry the same batch, in order, with Backoff::new(5, None, 250, 250) until it lands. This is safe because the actor2 workflow ignores events it already received (dedupe on generation and index).
• shutdown() cancels the token after it drops the senders. Handlers still dispatch buffered events, and a handler that is retrying exits.
• dispatch_events takes &[EventWrapper] so the batch can be retried.
• The test runs pegboard-envoy against Postgres through Toxiproxy, cuts storage for 2 seconds while events are sent, and checks that all events reach the actor workflow.

[railway-app]

This PR was not deployed automatically as @eersnington does not have access to the Railway project.

In order to get automatic PR deploys, please add @eersnington to your workspace on Railway.

[claude]

Review: retry failed actor event dispatch

The approach is sound. Retrying the same batch in order fixes the dropped "stopped" bug, and graceful_not_found already covers the "workflow gone" case. A few things to consider:

Potential issues

1. Ordering across handler replacement. The idle-channel cleanup (retain on last_seen) drops the sender. A handler stuck in its retry loop keeps running after that. If the same actor sends another event, ingest creates a new channel and handler, and two handlers can dispatch for one actor concurrently. Before this PR a failed handler exited, so this did not arise. During a long outage this could deliver events out of order. Consider not evicting channels whose handler is mid-retry.
2. Unbounded retry on permanent errors. Every Err is retried forever at 250ms. That is right for transient UDB failures, but a deterministic error would wedge that actor channel and grow its buffer. Consider classifying errors or escalating the log level after N attempts. The warn! repeats every 250ms for the whole outage, so rate-limiting it would help.
3. Backpressure. Check whether the channel behind ingest is bounded. If it is unbounded, memory grows during an outage.
4. Confirm Backoff::new(5, None, 250, 250) gives a flat 250ms and that tick() does not return immediately once its retry count is exhausted.

Tests

• The Toxiproxy/Postgres test is thorough, but it includes ../src/actor_event_demuxer.rs through #[path] and re-creates a metrics shim. That is fragile if the module gains more crate:: dependencies. Exposing the demuxer from the lib would be more robust.
• The 2s sleep is what guarantees event 1 dispatch failed. Asserting on an actual failed attempt would be more deterministic.
• unsafe { set_var } in a multi-thread tokio test is racy if more tests are added to the binary.

Nits

• events.to_vec() clones the batch on every attempt, including the first. This is acceptable.
• In shutdown(), the order of dropping senders and then cancel() matters for the drain semantics, so a short comment saying so would help.

No security concerns. Overall this looks good, and (1) is the one I would address before merging.