Skip to content

[v0.1.x-branch] Backport #1093: serverconn+round: never park the ingress path, make backpressure visible - #1098

Merged
Roasbeef merged 7 commits into
v0.1.x-branchfrom
backport-1093-to-v0.1.x-branch
Aug 5, 2026
Merged

Roasbeef merged 7 commits into
v0.1.x-branchfrom
backport-1093-to-v0.1.x-branch

Conversation

@github-actions

@github-actions github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown

Backport of #1093


In this PR, we fix the wedge behind the recurring "ark client goes deaf to round events" incidents (lightninglabs/lightning-infra#3675, two production occurrences). The serverconn ingress loop delivered server events with a blocking, deadline-free Tell into fixed-capacity in-memory mailboxes, while holding the SQLite global write transaction. Once a target mailbox filled, the sole ingress goroutine parked forever with the writer pinned. WAL mode meant readers kept answering RPC, so the pod looked healthy while every write in the process was frozen, and the only log line on that path was at trace level. The client silently dropped out of every round it had joined ("all clients dropped at seal time" on the operator side) until someone restarted it.

This is the client-side sibling of lightninglabs/lumos#734: an actor's receive goroutine (or here, the loop feeding all of them) must never block indefinitely on a peer's bounded mailbox. It builds on the TryTell primitive from #1082.

What changes

The ingress path now dispatches through a new TellWithoutParking primitive that picks the right send per resolved target: TryTell for a bounded in-memory mailbox, ordinary transactional Tell for a durable one. A full mailbox becomes a deferral instead of a park: the fold commits the cursor at the deferred envelope's own event_seq (everything below it is fully handled, the undelivered tail is never acked), and the pull loop is the redrive. Redriven batches are clamped and a served watermark prevents re-answering hoisted non-transactional requests every cycle. Backpressure gets its own backoff with a 500ms ceiling that resets on progress, so a drained backlog clears in seconds rather than at the transport-failure schedule's 30s. Since the store retries its transaction body on retryable errors, a per-dispatch record of out-of-transaction deliveries keeps those retries from re-sending messages that already landed in a mailbox.

Beyond the ingress path, we close the two other ways the audit found to park the round actor's receive loop: the round-to-wallet Ask cycle gets a bounded askWallet (both the enqueue and the Await, 30s default), and FSM.CurrentState gets a context-aware variant, with the scanning call sites sharing one deadline per scan instead of paying a per-FSM budget N times.

Observability

Both incidents produced zero evidence, so visibility is half this PR. Deferrals now log at error level (once per episode, re-logged every 30s with episode counters) and increment waved_serverconn_ingress_dispatch_deferred_total; two new gauges split "the ingress loop is alive" (last poll) from "dispatch is landing events" (last event). The metrics README gains an alerting section that maps each expression to the failure it detects; the deferral rate is the primary wedge signal, since a deaf client keeps polling and poll staleness alone can never see this failure.

What this does not claim

One residual is documented in serverconn/doc.go rather than fixed: all sixteen inbound routes share one strictly ordered cursor, so a stalled target still delays events for the other routes until it drains. What this PR removes is the writer pin and the silence. Decoupling the lanes (a per-target deferral queue with its own cursor) is follow-up work.

Testing

The regression tests wire a real actor system, the real event router, and the existing writer-lock harness, and each fails on the pre-fix code in seconds, one per mechanism: the storm test observes the loop parked inside the write transaction (the exact production signature), the redrive test counts unbounded hoisted re-serves, the recovery test times the drain on production-shaped config (63s pre-fix, under 4s post), and the replay test counts duplicate in-memory deliveries under transaction retries.

On top of the example-based tests, two further layers pin the invariants. Property tests (rapid) drive the real fold, cursor arithmetic, and backoff across randomized batches, deferral points, and store-retry injections; each property was validated non-vacuous by mutating the production code and watching it fail. And the P model suite gains a machine for this pipeline with six spec monitors (no acked loss, no tx-scoped duplicate, bounded at-least-once, served-once-per-incarnation, writer never parks, eventual drain); each pre-fix bug is a one-field config toggle the checker catches, and crash/restart nondeterminism is part of the schedule space.

The diff went through an adversarial review pass, a verification pass, and a complexity-focused design review; all findings are addressed in-line or documented as residuals. The design review drove one API simplification: the delivery helper is two exported names (TellWithoutParking + NonParkingTeller returning fn.Result[bool]), with the capability-query surface deleted rather than exported.

See the individual commit messages for the incremental story.

In this commit, we add TellWithoutParking and the NonParkingTeller
interface to the actor baselib. An actor or loop that must never block
indefinitely (a receive goroutine, or a sender holding a database write
transaction) can now hand a message to any TellOnlyRef and get the
right primitive per target: TryTell with an ErrMailboxFull sentinel for
a fixed-capacity in-memory mailbox, and the ordinary transactional Tell
for a durable one, whose enqueue is a DB write that cannot park on a
channel.

Every reference implementation in the package answers for itself
through NonParkingTeller, so the decision happens at send time against
the concrete resolved target rather than at wiring time. A Router
resolves its round-robin selection first and decides for that one ref,
so a mixed key with both durable and in-memory registrations cannot
mis-route a durable target through TryTell's background write, which
would drop the caller's transaction and misreport a full mailbox as a
deadline error. The Map/MapInput/FilterMapInput wrappers delegate the
decision inward, so a transform sitting in front of a router does not
flatten the per-target answer. A reference the package does not
recognise keeps the plain Tell, the direction that preserves a durable
enqueue's transaction atomicity.

The call returns fn.Result[bool] reporting whether the delivery escaped
the caller's transaction: true only for a successful TryTell hand-off,
which is visible to the receiver immediately and survives a rollback.
The serverconn ingress path reads it to know what must not be replayed
when the store retries the transaction body.

(cherry picked from commit 8a44a41)
In this commit, we make FSM.CurrentState safe to call against a machine
whose executor is parked mid-transition. The old implementation sent
the query with an unbounded blocking channel send and only applied its
one second timeout to the reply half, so a caller could park forever on
the request half with no context to cancel it. This was reachable from
eight call sites in the round actor's receive loop, making it one of
the cheapest ways to wedge that loop.

We add CurrentStateWithContext, which selects on the context for both
the send and the receive, and turn CurrentState into a wrapper that
runs both halves on one DefaultStateQueryTimeout timer of one second.
This is a semantic change for the no-context callers in oor, txconfirm,
and unroll: a query issued against a machine mid-transition previously
parked on the send until the transition finished, and now returns
DeadlineExceeded after the timer. The exposure is theoretical for those
packages, since their machines are driven by AskEvent from the same
goroutine that issues the queries, so a transition cannot be in flight
when a query runs; and every caller already handled the reply half's
one second timeout, so the error path is not new. The new test drives a
query against a machine held mid-transition from a goroutine, so a
regression fails the test instead of hanging it.

(cherry picked from commit 9b5d42e)
In this commit, we give the server connection's ingress path a liveness
surface. Both production wedges of this subsystem produced zero
evidence: no error logs, no metric movement, nothing for an operator to
alert on while the client sat deaf to round events for hours. The
existing server_connected boolean is set once at startup and never
flips on a wedge, so it cannot carry this signal.

We add three collectors. The last-poll timestamp says the ingress loop
itself is alive and reaching the operator. The last-event timestamp
says dispatch is actually landing messages in actor mailboxes. The
deferred counter counts redrives refused because a bounded in-memory
target was full. The split matters: a deaf client keeps polling, so
poll staleness cannot detect a dispatch wedge, and an idle client has a
legitimately stale event gauge. The README's new alerting section
spells out which expression detects which failure, with the deferral
rate as the primary wedge signal.

These gauges are stamped directly from the ingress goroutine rather
than through the metrics actor. That deviation is deliberate and
documented: the goroutine reporting its own liveness must not depend on
yet another actor edge that can itself back up.

(cherry picked from commit f66da00)
In this commit, we fix the wedge behind the recurring "client goes deaf
to round events" incidents. The ingress loop delivered server events
with a blocking, deadline-free Tell into fixed-capacity in-memory
mailboxes (round-client and incoming-vtxo-handler, which the package
docs wrongly described as durable), and it did so while holding the
SQLite global write transaction. Once a target mailbox filled, the sole
ingress goroutine parked forever with the writer pinned. WAL readers
kept answering RPC, so the pod looked healthy while every write in the
process was frozen, and the only log on that path was at trace level.
Two production incidents produced no evidence at all.

Dispatch now goes through TellWithoutParking. A full in-memory mailbox
becomes a deferral instead of a park: the fold commits the cursor at
the deferred envelope's own event_seq, so everything below it is fully
handled (waiters delivered, hoisted requests served, the durable prefix
enqueued in the same transaction) and the undelivered tail is never
acked. The pull loop is the redrive. On a redrive the batch is clamped
to the deferred seq and a served watermark suppresses re-serving
hoisted requests, so a persistent wedge cannot re-answer the same
non-transactional request every cycle. Backpressure gets its own
backoff with a 500ms ceiling and resets on progress, keeping drain
latency well inside the round phase deadlines rather than saturating at
the transport schedule's 30s.

Because the store retries its transaction body on retryable errors, an
in-memory TryTell that already landed must not run again on the next
attempt. A per-dispatch record of out-of-transaction deliveries, keyed
by event_seq and scoped to a single folded dispatch, makes the replay
idempotent. In-memory dispatch stays inside the fold because the single
ordered pass keeps per-target delivery order and the serve-before-ack
crash safety with no second bookkeeping pass; cross-target ordering is
not a consideration either way, since a durable enqueue is invisible to
its consumer until commit, which lands after every in-memory hand-off
in the batch. Hoisting the in-memory half out of the fold would delete
the replay record entirely and is recorded as a follow-up.

Deferrals log at error level once per episode with periodic re-logging
and increment a counter, so the next wedge pages instead of hiding. One
residual is documented in the package docs rather than fixed here: all
routes share one strictly ordered cursor, so a stalled target still
delays events for the others. This change removes the writer pin and
makes that state visible; it does not decouple the lanes.

The regression tests wire a real actor system, a real event router, and
the writer-lock harness. They fail on the pre-fix code in seconds, one
per mechanism: the storm test observes the parked transaction, the
redrive test counts unbounded hoisted re-serves, the recovery test
times the drain, and the replay test counts duplicate deliveries.

(cherry picked from commit ea00863)
In this commit, we close the round actor's two remaining ways to park
its receive loop indefinitely, the loop that gates every inbound round
event.

The first is the round-to-wallet Ask cycle: the round actor asked the
wallet with no deadline while the wallet's own handlers ask the round
actor back, so two full mailboxes could produce a circular wait that
never resolves. The onchain intent replayer already bounds the
identical Ask and documents the hazard; the live paths never got that
treatment. The new askWallet helper applies one WithTimeout to both the
enqueue and the Await, since a deadline on the Await alone leaves the
blocking enqueue unbounded. The 30s default is overridable through the
config, and the actor's lifecycle context stays the parent so shutdown
still cancels a pending Ask. Bounding one direction is enough to break
the cycle; the wallet-to-round direction is left alone deliberately,
because an Ask's context becomes the receiver's turn context, and a
deadline there would bound money-path FSM registration work.

The second is scan cost: state queries now carry a context, and the
call sites that scan every tracked round share a single deadline per
scan instead of paying the per-query budget once per FSM. With N
tracked rounds mid-transition, a single message could previously spend
3N seconds in state queries; OnStop could burn its whole cleanup budget
on queries and strand external-signer session cleanup. Every scanning
caller already degrades by skipping an unreadable round, so the shared
budget changes worst-case latency, not behaviour.

The new test parks the wallet double behind a gate that honors the Ask
context, so it faithfully models a full mailbox: pre-fix the round
actor visibly parks on the stalled wallet, post-fix the Ask times out,
logs at error level, and the loop keeps taking turns.

(cherry picked from commit 3131675)
In this commit, we extend the P model suite to cover the delivery half
of the connection actor's ingress loop, which the existing cursor-fold
model deliberately abstracted away.

ingress_fold.p says the persisted cursor may never cover an envelope
whose local enqueue did not commit, and it gets there by treating every
dispatch as a durable enqueue that lives and dies with the write
transaction. The dispatch table does not work that way. It has four
kinds of destination and only one of them is transactional: a durable
target rolls back with the transaction, a hoisted request has already
left the process by the time the transaction opens, a waiter response is
handed over in memory ahead of it, and a bounded in-memory target is a
fixed-capacity actor mailbox that is delivered to from inside the
transaction but is not in it. That last one is also the only destination
that can refuse, and refusing is what the whole deferral path exists to
handle.

The new model, ingress_deferral.p, states what the loop owes each of
those four. The committed cursor may only cover envelopes some target
actually received, so a deferral commits the prefix and stops at the
undelivered envelope. A store that replays its transaction body after a
retryable error may not re-send what the previous attempt already handed
to a bounded mailbox, because that delivery was never in the transaction
to take back. Redelivery across redrives and crashes is expected, but at
most once per epoch, so the duplicate count is bounded by how many times
the loop went around rather than by how many internal steps each turn
took. And a request answered ahead of the commit is answered at most
once per process lifetime, however many times a wedged target makes the
loop re-pull it.

Above all, the ingress goroutine never parks. It is the process's only
consumer of the remote mailbox and it holds the database's single writer
while it works, so a blocking send into a full fixed-capacity mailbox is
not a slowdown but a permanently deaf client that still answers reads,
still reports healthy, and logs nothing.

A model that cannot detect the bug it exists for is worthless, so the
pre-fix implementation stays reachable through IngressPipelineConfig.
Each counterexample flips exactly one field of the production profile
and fails: ParkingBlockingSend wedges the writer, a cleared
track_tx_deliveries duplicates in-memory deliveries across store
retries, and a cleared served_watermark re-answers a hoisted request
once per backoff cycle. The park is caught twice over, once as a safety
violation and once as the starvation an operator actually sees.

The model also declines to flatter the implementation on two points. The
nonTx serve genuinely can repeat across a crash, because the answer has
to precede the commit, so the monitor keys on the process incarnation
and rejects only a repeat within one lifetime rather than asserting the
residual away. And removing the redrive clamp while keeping the served
watermark breaks no property here: the clamp earns its place as a bound
on repeated work, not as a second safety net.

check.sh picks the new cases up alongside the durable mailbox suite, and
while we are here it also picks up the three ingress cursor-fold cases,
which have been compiled but never checked since they were added.

(cherry picked from commit b635691)
In this commit, we add property tests for the invariants the ingress
loop's deferral and redrive path has to hold, and for the two pure
functions that decide the cursor a deferred cycle commits.

The reason for property tests rather than more examples is the size of
the state space. One backpressure episode is a cross product of where in
the pulled window the deferral falls, how many times the store replays
its transaction body after a retryable error, which hoisted serves fail,
and whether the target drained between redrives. The invariants are
about what survives all of that rather than about any one path through
it, and the example tests in ingress_backpressure_test.go each pin a
single point in that space.

Five things are pinned. The first is cursor safety, which is the one the
rest exist to protect: the committed cursor may never cover an envelope
that no destination accepted. An exclusive cursor acks everything below
it, so a cursor that ran one envelope too far is a silently dropped
round event with no local symptom. We check it against a handled set
built out of what the real dispatchers returned, not out of a second
model of what they ought to have returned.

The second is no loss and bounded duplication. Every envelope arrives
once the mailboxes drain, and a delivery into a bounded in-memory
mailbox happens at most once per folded dispatch however many times the
store runs the body, because that delivery is not in the transaction and
a rollback does not take it back. The third is ordering: the target's
own lane must hold exactly the deliveries the fold made, in the order it
made them. The fourth is that a hoisted request is answered exactly once
per episode, however many redrives the episode takes, because the answer
goes to the operator over the network while the cursor stays put. The
fifth is the redrive schedule's ceiling, the minimum of maxDeferralDelay
and the configured RetryMaxDelay, together with the cursor advance the
loop resets deferFailCount on.

The end-to-end property drives the real thing: a real actor system, a
real fixed-capacity mailbox, the real EventRouter dispatcher, and
runFoldedDispatch itself, with only the loop control around it mirrored
from ingressLoop. A property test of a parallel model proves nothing
about the code, so the two dispatchers the harness installs wrap the
production ones and do no more than record what they did, including
asking the fold's own in-flight record whether a delivery was about to
be suppressed. clampToDeferred and deferredCursor are small enough to
drive directly, so they get their own properties over randomized batch
shapes, gap layouts and deferral points.

We checked that the properties bite by mutating the code they cover: an
off-by-one in deferredCursor, a clamp that returns the pulled window's
cursor instead of the deferred envelope's, a replay record that never
suppresses, a dropped servedNonTxSeq watermark, and a redrive schedule
with no ceiling are each caught. Two thirds of the generated episodes
reach a deferral, and the test asserts that at least one did so a
generator that drifted cannot make this pass vacuously. Test files only;
no production code changes.

(cherry picked from commit e1f0216)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant