Repository navigation
serverconn+round: never park the ingress path, make backpressure visible - #1093
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aaf3c813c5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| a.log.ErrorS(ctx, "Ingress dispatch deferred: target mailbox full, "+ | ||
| "re-pulling after backoff", err, |
There was a problem hiding this comment.
Use non-error logging for dispatch backpressure
A full target mailbox is an expected backpressure condition that this path handles by leaving the envelope unacknowledged and redriving after backoff, so logging every episode with ErrorS promotes a retryable load/slow-consumer condition into error-level alerts. The repository logging guidance reserves error level for internal bugs and not external/operational triggers; the existing metric plus WarnS/InfoS would keep this visible without violating that contract.
AGENTS.md reference: AGENTS.md:L36-L38
Useful? React with 👍 / 👎.
aaf3c81 to
3c3c59d
Compare
|
@claude review this |
|
Claude finished @Roasbeef's task in 4m 2s —— View job Review: serverconn+round backpressureI read the five commits against Correctness I verified (no action needed)
1. Error-level logging for deferrals — I lean toward the Codex suggestion, with nuanceThe Codex bot flagged
The throttling (first + every 30s) is good, but it doesn't distinguish those two cases — the first line fires at That said — the rationale in the doc comment is deliberate and well-argued, and this is ultimately a judgment call the maintainers own. If the team decides sustained visibility outweighs the guideline here, documenting that decision next to the 2.
|
In this commit, we add TellWithoutParking and the NonParkingTeller interface to the actor baselib. An actor or loop that must never block indefinitely (a receive goroutine, or a sender holding a database write transaction) can now hand a message to any TellOnlyRef and get the right primitive per target: TryTell with an ErrMailboxFull sentinel for a fixed-capacity in-memory mailbox, and the ordinary transactional Tell for a durable one, whose enqueue is a DB write that cannot park on a channel. Every reference implementation in the package answers for itself through NonParkingTeller, so the decision happens at send time against the concrete resolved target rather than at wiring time. A Router resolves its round-robin selection first and decides for that one ref, so a mixed key with both durable and in-memory registrations cannot mis-route a durable target through TryTell's background write, which would drop the caller's transaction and misreport a full mailbox as a deadline error. The Map/MapInput/FilterMapInput wrappers delegate the decision inward, so a transform sitting in front of a router does not flatten the per-target answer. A reference the package does not recognise keeps the plain Tell, the direction that preserves a durable enqueue's transaction atomicity. The call returns fn.Result[bool] reporting whether the delivery escaped the caller's transaction: true only for a successful TryTell hand-off, which is visible to the receiver immediately and survives a rollback. The serverconn ingress path reads it to know what must not be replayed when the store retries the transaction body.
In this commit, we make FSM.CurrentState safe to call against a machine whose executor is parked mid-transition. The old implementation sent the query with an unbounded blocking channel send and only applied its one second timeout to the reply half, so a caller could park forever on the request half with no context to cancel it. This was reachable from eight call sites in the round actor's receive loop, making it one of the cheapest ways to wedge that loop. We add CurrentStateWithContext, which selects on the context for both the send and the receive, and turn CurrentState into a wrapper that runs both halves on one DefaultStateQueryTimeout timer of one second. This is a semantic change for the no-context callers in oor, txconfirm, and unroll: a query issued against a machine mid-transition previously parked on the send until the transition finished, and now returns DeadlineExceeded after the timer. The exposure is theoretical for those packages, since their machines are driven by AskEvent from the same goroutine that issues the queries, so a transition cannot be in flight when a query runs; and every caller already handled the reply half's one second timeout, so the error path is not new. The new test drives a query against a machine held mid-transition from a goroutine, so a regression fails the test instead of hanging it.
In this commit, we give the server connection's ingress path a liveness surface. Both production wedges of this subsystem produced zero evidence: no error logs, no metric movement, nothing for an operator to alert on while the client sat deaf to round events for hours. The existing server_connected boolean is set once at startup and never flips on a wedge, so it cannot carry this signal. We add three collectors. The last-poll timestamp says the ingress loop itself is alive and reaching the operator. The last-event timestamp says dispatch is actually landing messages in actor mailboxes. The deferred counter counts redrives refused because a bounded in-memory target was full. The split matters: a deaf client keeps polling, so poll staleness cannot detect a dispatch wedge, and an idle client has a legitimately stale event gauge. The README's new alerting section spells out which expression detects which failure, with the deferral rate as the primary wedge signal. These gauges are stamped directly from the ingress goroutine rather than through the metrics actor. That deviation is deliberate and documented: the goroutine reporting its own liveness must not depend on yet another actor edge that can itself back up.
3c3c59d to
772093b
Compare
|
@claude review this |
|
Claude finished @Roasbeef's task in 3m 48s —— View job Review: serverconn+round — never park the ingress pathI re-read the seven commits with the diff against Correctness I traced independently (no action needed)
1.
|
772093b to
fef8edf
Compare
|
Both items addressed in fef8edf:
Thanks for tracing the clamp/watermark disjoint-window interaction independently; that is the subtlest piece of the cursor arithmetic and it now has three independent confirmations (adversarial review, the P model's non-vacuity mutations, and this pass). |
In this commit, we fix the wedge behind the recurring "client goes deaf to round events" incidents. The ingress loop delivered server events with a blocking, deadline-free Tell into fixed-capacity in-memory mailboxes (round-client and incoming-vtxo-handler, which the package docs wrongly described as durable), and it did so while holding the SQLite global write transaction. Once a target mailbox filled, the sole ingress goroutine parked forever with the writer pinned. WAL readers kept answering RPC, so the pod looked healthy while every write in the process was frozen, and the only log on that path was at trace level. Two production incidents produced no evidence at all. Dispatch now goes through TellWithoutParking. A full in-memory mailbox becomes a deferral instead of a park: the fold commits the cursor at the deferred envelope's own event_seq, so everything below it is fully handled (waiters delivered, hoisted requests served, the durable prefix enqueued in the same transaction) and the undelivered tail is never acked. The pull loop is the redrive. On a redrive the batch is clamped to the deferred seq and a served watermark suppresses re-serving hoisted requests, so a persistent wedge cannot re-answer the same non-transactional request every cycle. Backpressure gets its own backoff with a 500ms ceiling and resets on progress, keeping drain latency well inside the round phase deadlines rather than saturating at the transport schedule's 30s. Because the store retries its transaction body on retryable errors, an in-memory TryTell that already landed must not run again on the next attempt. A per-dispatch record of out-of-transaction deliveries, keyed by event_seq and scoped to a single folded dispatch, makes the replay idempotent. In-memory dispatch stays inside the fold because the single ordered pass keeps per-target delivery order and the serve-before-ack crash safety with no second bookkeeping pass; cross-target ordering is not a consideration either way, since a durable enqueue is invisible to its consumer until commit, which lands after every in-memory hand-off in the batch. Hoisting the in-memory half out of the fold would delete the replay record entirely and is recorded as a follow-up. Deferrals log at error level once per episode with periodic re-logging and increment a counter, so the next wedge pages instead of hiding. One residual is documented in the package docs rather than fixed here: all routes share one strictly ordered cursor, so a stalled target still delays events for the others. This change removes the writer pin and makes that state visible; it does not decouple the lanes. The regression tests wire a real actor system, a real event router, and the writer-lock harness. They fail on the pre-fix code in seconds, one per mechanism: the storm test observes the parked transaction, the redrive test counts unbounded hoisted re-serves, the recovery test times the drain, and the replay test counts duplicate deliveries.
In this commit, we close the round actor's two remaining ways to park its receive loop indefinitely, the loop that gates every inbound round event. The first is the round-to-wallet Ask cycle: the round actor asked the wallet with no deadline while the wallet's own handlers ask the round actor back, so two full mailboxes could produce a circular wait that never resolves. The onchain intent replayer already bounds the identical Ask and documents the hazard; the live paths never got that treatment. The new askWallet helper applies one WithTimeout to both the enqueue and the Await, since a deadline on the Await alone leaves the blocking enqueue unbounded. The 30s default is overridable through the config, and the actor's lifecycle context stays the parent so shutdown still cancels a pending Ask. Bounding one direction is enough to break the cycle; the wallet-to-round direction is left alone deliberately, because an Ask's context becomes the receiver's turn context, and a deadline there would bound money-path FSM registration work. The second is scan cost: state queries now carry a context, and the call sites that scan every tracked round share a single deadline per scan instead of paying the per-query budget once per FSM. With N tracked rounds mid-transition, a single message could previously spend 3N seconds in state queries; OnStop could burn its whole cleanup budget on queries and strand external-signer session cleanup. Every scanning caller already degrades by skipping an unreadable round, so the shared budget changes worst-case latency, not behaviour. The new test parks the wallet double behind a gate that honors the Ask context, so it faithfully models a full mailbox: pre-fix the round actor visibly parks on the stalled wallet, post-fix the Ask times out, logs at error level, and the loop keeps taking turns.
In this commit, we extend the P model suite to cover the delivery half of the connection actor's ingress loop, which the existing cursor-fold model deliberately abstracted away. ingress_fold.p says the persisted cursor may never cover an envelope whose local enqueue did not commit, and it gets there by treating every dispatch as a durable enqueue that lives and dies with the write transaction. The dispatch table does not work that way. It has four kinds of destination and only one of them is transactional: a durable target rolls back with the transaction, a hoisted request has already left the process by the time the transaction opens, a waiter response is handed over in memory ahead of it, and a bounded in-memory target is a fixed-capacity actor mailbox that is delivered to from inside the transaction but is not in it. That last one is also the only destination that can refuse, and refusing is what the whole deferral path exists to handle. The new model, ingress_deferral.p, states what the loop owes each of those four. The committed cursor may only cover envelopes some target actually received, so a deferral commits the prefix and stops at the undelivered envelope. A store that replays its transaction body after a retryable error may not re-send what the previous attempt already handed to a bounded mailbox, because that delivery was never in the transaction to take back. Redelivery across redrives and crashes is expected, but at most once per epoch, so the duplicate count is bounded by how many times the loop went around rather than by how many internal steps each turn took. And a request answered ahead of the commit is answered at most once per process lifetime, however many times a wedged target makes the loop re-pull it. Above all, the ingress goroutine never parks. It is the process's only consumer of the remote mailbox and it holds the database's single writer while it works, so a blocking send into a full fixed-capacity mailbox is not a slowdown but a permanently deaf client that still answers reads, still reports healthy, and logs nothing. A model that cannot detect the bug it exists for is worthless, so the pre-fix implementation stays reachable through IngressPipelineConfig. Each counterexample flips exactly one field of the production profile and fails: ParkingBlockingSend wedges the writer, a cleared track_tx_deliveries duplicates in-memory deliveries across store retries, and a cleared served_watermark re-answers a hoisted request once per backoff cycle. The park is caught twice over, once as a safety violation and once as the starvation an operator actually sees. The model also declines to flatter the implementation on two points. The nonTx serve genuinely can repeat across a crash, because the answer has to precede the commit, so the monitor keys on the process incarnation and rejects only a repeat within one lifetime rather than asserting the residual away. And removing the redrive clamp while keeping the served watermark breaks no property here: the clamp earns its place as a bound on repeated work, not as a second safety net. check.sh picks the new cases up alongside the durable mailbox suite, and while we are here it also picks up the three ingress cursor-fold cases, which have been compiled but never checked since they were added.
In this commit, we add property tests for the invariants the ingress loop's deferral and redrive path has to hold, and for the two pure functions that decide the cursor a deferred cycle commits. The reason for property tests rather than more examples is the size of the state space. One backpressure episode is a cross product of where in the pulled window the deferral falls, how many times the store replays its transaction body after a retryable error, which hoisted serves fail, and whether the target drained between redrives. The invariants are about what survives all of that rather than about any one path through it, and the example tests in ingress_backpressure_test.go each pin a single point in that space. Five things are pinned. The first is cursor safety, which is the one the rest exist to protect: the committed cursor may never cover an envelope that no destination accepted. An exclusive cursor acks everything below it, so a cursor that ran one envelope too far is a silently dropped round event with no local symptom. We check it against a handled set built out of what the real dispatchers returned, not out of a second model of what they ought to have returned. The second is no loss and bounded duplication. Every envelope arrives once the mailboxes drain, and a delivery into a bounded in-memory mailbox happens at most once per folded dispatch however many times the store runs the body, because that delivery is not in the transaction and a rollback does not take it back. The third is ordering: the target's own lane must hold exactly the deliveries the fold made, in the order it made them. The fourth is that a hoisted request is answered exactly once per episode, however many redrives the episode takes, because the answer goes to the operator over the network while the cursor stays put. The fifth is the redrive schedule's ceiling, the minimum of maxDeferralDelay and the configured RetryMaxDelay, together with the cursor advance the loop resets deferFailCount on. The end-to-end property drives the real thing: a real actor system, a real fixed-capacity mailbox, the real EventRouter dispatcher, and runFoldedDispatch itself, with only the loop control around it mirrored from ingressLoop. A property test of a parallel model proves nothing about the code, so the two dispatchers the harness installs wrap the production ones and do no more than record what they did, including asking the fold's own in-flight record whether a delivery was about to be suppressed. clampToDeferred and deferredCursor are small enough to drive directly, so they get their own properties over randomized batch shapes, gap layouts and deferral points. We checked that the properties bite by mutating the code they cover: an off-by-one in deferredCursor, a clamp that returns the pulled window's cursor instead of the deferred envelope's, a replay record that never suppresses, a dropped servedNonTxSeq watermark, and a redrive schedule with no ceiling are each caught. Two thirds of the generated episodes reach a deferral, and the test asserts that at least one did so a generator that drifted cannot make this pass vacuously. Test files only; no production code changes.
fef8edf to
e1f0216
Compare
|
Successfully created backport PR for |
…ranch [v0.1.x-branch] Backport #1093: serverconn+round: never park the ingress path, make backpressure visible
In this PR, we fix the wedge behind the recurring "ark client goes deaf to round events" incidents (lightninglabs/lightning-infra#3675, two production occurrences). The serverconn ingress loop delivered server events with a blocking, deadline-free
Tellinto fixed-capacity in-memory mailboxes, while holding the SQLite global write transaction. Once a target mailbox filled, the sole ingress goroutine parked forever with the writer pinned. WAL mode meant readers kept answering RPC, so the pod looked healthy while every write in the process was frozen, and the only log line on that path was at trace level. The client silently dropped out of every round it had joined ("all clients dropped at seal time" on the operator side) until someone restarted it.This is the client-side sibling of lightninglabs/lumos#734: an actor's receive goroutine (or here, the loop feeding all of them) must never block indefinitely on a peer's bounded mailbox. It builds on the
TryTellprimitive from #1082.What changes
The ingress path now dispatches through a new
TellWithoutParkingprimitive that picks the right send per resolved target:TryTellfor a bounded in-memory mailbox, ordinary transactionalTellfor a durable one. A full mailbox becomes a deferral instead of a park: the fold commits the cursor at the deferred envelope's ownevent_seq(everything below it is fully handled, the undelivered tail is never acked), and the pull loop is the redrive. Redriven batches are clamped and a served watermark prevents re-answering hoisted non-transactional requests every cycle. Backpressure gets its own backoff with a 500ms ceiling that resets on progress, so a drained backlog clears in seconds rather than at the transport-failure schedule's 30s. Since the store retries its transaction body on retryable errors, a per-dispatch record of out-of-transaction deliveries keeps those retries from re-sending messages that already landed in a mailbox.Beyond the ingress path, we close the two other ways the audit found to park the round actor's receive loop: the round-to-wallet Ask cycle gets a bounded
askWallet(both the enqueue and the Await, 30s default), andFSM.CurrentStategets a context-aware variant, with the scanning call sites sharing one deadline per scan instead of paying a per-FSM budget N times.Observability
Both incidents produced zero evidence, so visibility is half this PR. Deferrals now log at error level (once per episode, re-logged every 30s with episode counters) and increment
waved_serverconn_ingress_dispatch_deferred_total; two new gauges split "the ingress loop is alive" (last poll) from "dispatch is landing events" (last event). The metrics README gains an alerting section that maps each expression to the failure it detects; the deferral rate is the primary wedge signal, since a deaf client keeps polling and poll staleness alone can never see this failure.What this does not claim
One residual is documented in
serverconn/doc.gorather than fixed: all sixteen inbound routes share one strictly ordered cursor, so a stalled target still delays events for the other routes until it drains. What this PR removes is the writer pin and the silence. Decoupling the lanes (a per-target deferral queue with its own cursor) is follow-up work.Testing
The regression tests wire a real actor system, the real event router, and the existing writer-lock harness, and each fails on the pre-fix code in seconds, one per mechanism: the storm test observes the loop parked inside the write transaction (the exact production signature), the redrive test counts unbounded hoisted re-serves, the recovery test times the drain on production-shaped config (63s pre-fix, under 4s post), and the replay test counts duplicate in-memory deliveries under transaction retries.
On top of the example-based tests, two further layers pin the invariants. Property tests (rapid) drive the real fold, cursor arithmetic, and backoff across randomized batches, deferral points, and store-retry injections; each property was validated non-vacuous by mutating the production code and watching it fail. And the P model suite gains a machine for this pipeline with six spec monitors (no acked loss, no tx-scoped duplicate, bounded at-least-once, served-once-per-incarnation, writer never parks, eventual drain); each pre-fix bug is a one-field config toggle the checker catches, and crash/restart nondeterminism is part of the schedule space.
The diff went through an adversarial review pass, a verification pass, and a complexity-focused design review; all findings are addressed in-line or documented as residuals. The design review drove one API simplification: the delivery helper is two exported names (
TellWithoutParking+NonParkingTellerreturningfn.Result[bool]), with the capability-query surface deleted rather than exported.See the individual commit messages for the incremental story.