Skip to content

serverconn+round: never park the ingress path, make backpressure visible - #1093

Merged
Roasbeef merged 7 commits into
mainfrom
fix/serverconn-backpressure
Aug 5, 2026
Merged

Roasbeef merged 7 commits into
mainfrom
fix/serverconn-backpressure

Conversation

@Roasbeef

@Roasbeef Roasbeef commented Aug 4, 2026 •

Copy link
Copy Markdown
Member

In this PR, we fix the wedge behind the recurring "ark client goes deaf to round events" incidents (lightninglabs/lightning-infra#3675, two production occurrences). The serverconn ingress loop delivered server events with a blocking, deadline-free Tell into fixed-capacity in-memory mailboxes, while holding the SQLite global write transaction. Once a target mailbox filled, the sole ingress goroutine parked forever with the writer pinned. WAL mode meant readers kept answering RPC, so the pod looked healthy while every write in the process was frozen, and the only log line on that path was at trace level. The client silently dropped out of every round it had joined ("all clients dropped at seal time" on the operator side) until someone restarted it.

This is the client-side sibling of lightninglabs/lumos#734: an actor's receive goroutine (or here, the loop feeding all of them) must never block indefinitely on a peer's bounded mailbox. It builds on the TryTell primitive from #1082.

What changes

The ingress path now dispatches through a new TellWithoutParking primitive that picks the right send per resolved target: TryTell for a bounded in-memory mailbox, ordinary transactional Tell for a durable one. A full mailbox becomes a deferral instead of a park: the fold commits the cursor at the deferred envelope's own event_seq (everything below it is fully handled, the undelivered tail is never acked), and the pull loop is the redrive. Redriven batches are clamped and a served watermark prevents re-answering hoisted non-transactional requests every cycle. Backpressure gets its own backoff with a 500ms ceiling that resets on progress, so a drained backlog clears in seconds rather than at the transport-failure schedule's 30s. Since the store retries its transaction body on retryable errors, a per-dispatch record of out-of-transaction deliveries keeps those retries from re-sending messages that already landed in a mailbox.

Beyond the ingress path, we close the two other ways the audit found to park the round actor's receive loop: the round-to-wallet Ask cycle gets a bounded askWallet (both the enqueue and the Await, 30s default), and FSM.CurrentState gets a context-aware variant, with the scanning call sites sharing one deadline per scan instead of paying a per-FSM budget N times.

Observability

Both incidents produced zero evidence, so visibility is half this PR. Deferrals now log at error level (once per episode, re-logged every 30s with episode counters) and increment waved_serverconn_ingress_dispatch_deferred_total; two new gauges split "the ingress loop is alive" (last poll) from "dispatch is landing events" (last event). The metrics README gains an alerting section that maps each expression to the failure it detects; the deferral rate is the primary wedge signal, since a deaf client keeps polling and poll staleness alone can never see this failure.

What this does not claim

One residual is documented in serverconn/doc.go rather than fixed: all sixteen inbound routes share one strictly ordered cursor, so a stalled target still delays events for the other routes until it drains. What this PR removes is the writer pin and the silence. Decoupling the lanes (a per-target deferral queue with its own cursor) is follow-up work.

Testing

The regression tests wire a real actor system, the real event router, and the existing writer-lock harness, and each fails on the pre-fix code in seconds, one per mechanism: the storm test observes the loop parked inside the write transaction (the exact production signature), the redrive test counts unbounded hoisted re-serves, the recovery test times the drain on production-shaped config (63s pre-fix, under 4s post), and the replay test counts duplicate in-memory deliveries under transaction retries.

On top of the example-based tests, two further layers pin the invariants. Property tests (rapid) drive the real fold, cursor arithmetic, and backoff across randomized batches, deferral points, and store-retry injections; each property was validated non-vacuous by mutating the production code and watching it fail. And the P model suite gains a machine for this pipeline with six spec monitors (no acked loss, no tx-scoped duplicate, bounded at-least-once, served-once-per-incarnation, writer never parks, eventual drain); each pre-fix bug is a one-field config toggle the checker catches, and crash/restart nondeterminism is part of the schedule space.

The diff went through an adversarial review pass, a verification pass, and a complexity-focused design review; all findings are addressed in-line or documented as residuals. The design review drove one API simplification: the delivery helper is two exported names (TellWithoutParking + NonParkingTeller returning fn.Result[bool]), with the capability-query surface deleted rather than exported.

See the individual commit messages for the incremental story.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aaf3c813c5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread serverconn/dispatch_deferral.go Outdated
Comment on lines +190 to +191
a.log.ErrorS(ctx, "Ingress dispatch deferred: target mailbox full, "+
"re-pulling after backoff", err,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use non-error logging for dispatch backpressure

A full target mailbox is an expected backpressure condition that this path handles by leaving the envelope unacknowledged and redriving after backoff, so logging every episode with ErrorS promotes a retryable load/slow-consumer condition into error-level alerts. The repository logging guidance reserves error level for internal bugs and not external/operational triggers; the existing metric plus WarnS/InfoS would keep this visible without violating that contract.

AGENTS.md reference: AGENTS.md:L36-L38

Useful? React with 👍 / 👎.

@Roasbeef

Roasbeef commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

@claude review this

@claude

claude Bot commented Aug 5, 2026 •

Copy link
Copy Markdown

Claude finished @Roasbeef's task in 4m 2s —— View job


Review: serverconn+round backpressure

I read the five commits against origin/main, focusing on the ingress deferral/cursor arithmetic, the askWallet/FSM-scan changes, and the new CurrentState semantics. This is a genuinely strong PR — the invariants are spelled out at the call sites, the cursor math is internally consistent, and the observability story is coherent. Comments below are one guideline tension, one cross-package behavior change worth confirming, and a few smaller notes. No blocking correctness bug found.


Correctness I verified (no action needed)

  • Deferral cursor math is sound. deferredCursor returns deferral.eventSeq as the exclusive ack watermark and inclusive next-pull position. Against AckState/AckUpTo semantics (AckUpTo(cursor) acks event_seq < cursor; Pull(cursor) returns event_seq >= cursor), committing at deferredSeq acks everything strictly below the undelivered envelope and re-pulls starting exactly at it. The "acks everything before it and nothing at or after it" claim holds. clampToDeferred + servedNonTxSeq correctly prevent re-serving hoisted requests every redrive.
  • deliveredOutsideTx replay guard is correctly scoped. Only DeliveredByTryTell hand-offs are marked (durable Tell enqueues roll back with the tx and must re-run), the record lives outside the ExecTx closure so it survives body retries, and it's rebuilt per runFoldedDispatch call — matching the reasoning in dispatch_replay.go.
  • servedNonTxSeq is never reset across episodes — this is safe (and necessary) because event_seq is monotonic per mailbox, so a stale high-water mark can never suppress a genuinely new request.

1. Error-level logging for deferrals — I lean toward the Codex suggestion, with nuance

The Codex bot flagged dispatch_deferral.go:191 against the repo rule (AGENTS.md/CLAUDE.md): "error log level is only for internal bugs, never external triggers." I think this is a legitimate tension worth resolving explicitly rather than dismissing:

  • A brief, self-clearing deferral (a target momentarily busy under load) is an operational condition — exactly what the metric ServerConnIngressDeferredTotal exists to surface. Emitting ErrorS on the first deferral of every episode risks paging on benign momentary backpressure.
  • A persistent episode (a target that has stopped draining entirely) genuinely is an internal-bug-class signal, and there Error is defensible.

The throttling (first + every 30s) is good, but it doesn't distinguish those two cases — the first line fires at Error even for an episode that clears on the very next redrive. A middle path that keeps your visibility goal while honoring the guideline: log the first/interval lines at WarnS, and reserve ErrorS for an episode that has stayed open past some threshold (e.g. episode.deferrals beyond N, or age beyond a small multiple of deferralLogInterval). The primary alerting signal is the metric regardless, per your own README section.

That said — the rationale in the doc comment is deliberate and well-argued, and this is ultimately a judgment call the maintainers own. If the team decides sustained visibility outweighs the guideline here, documenting that decision next to the ErrorS call (rather than only the "was previously silent" history) would close the loop for the next reader who hits the lint rule. Fix this →


2. CurrentState() now bounds its send half — silent behavior change for oor/txconfirm/unroll

baselib/protofsm/state_machine.go: the no-context CurrentState() was rewritten from fn.SendOrQuit(...) + fn.RecvOrTimeout(1s) to a shared 1s timer across both the request send and the reply. Previously the send half could block for the entire duration of an in-flight ProcessEvent (driveMachine only serves queries from its top-level select). Now it returns context.DeadlineExceeded after 1s.

That's the intended fix for the round actor, but CurrentState() (no ctx) is still called from outside this PR's stated scope:

  • oor/session_actor.go:1190, oor/session_actor_handlers.go, oor/session.go
  • txconfirm/fsm_types.go:273, txconfirm/fee_input_actor.go:147
  • unroll/actor.go:2067

For these, a transition that legitimately runs >1s (signing/network work in txconfirm/unroll is plausible) will now surface a transient DeadlineExceeded where it previously blocked and eventually returned the true state. I spot-checked unroll/actor.go:currentState — it returns the error (with a shutdown special-case), so it degrades rather than misbehaves. Worth confirming the txconfirm and oor call sites treat a transient state-read timeout as acceptable (skip/retry) and won't, e.g., fail a job on it. If they do, this is fine and arguably an improvement; it just deserves a sentence in the commit body since it changes semantics for three packages the PR title doesn't mention.


3. Smaller notes

  • askWallet covers both halves — good. Passing the same askCtx to Ask and Await is the correct fix; a deadline on Await alone would leave the mailbox enqueue unbounded. The asymmetry (bounding only the round→wallet direction) is documented as deliberate.
  • fsmScanContext single-deadline-per-scan is the right call; the OnStop note about the 5s budget being consumed by reads (starving MuSig2 cleanup) is a good catch and a real leak avoided.
  • TellWithoutParking / NonParkingTeller — the Router "ANY vs the ONE chosen" reasoning is subtle and correctly handled by resolving the target first via TellWithoutParkingTo. The compile-time interface assertions cover every in-tree ref type, so the conservative DeliveredByTell fallback only applies to out-of-tree refs, which is the safe direction.
  • Docs — the serverconn/doc.go "Known residual" (one shared ordered cursor still couples all 16 routes) is honestly stated, and the store_impl.go NOTE is correctly updated to reflect that the send is now hoisted rather than sitting inside fn.

Overall: approve-shaped from a correctness standpoint. The only thing I'd want a maintainer decision on before merge is the ErrorS level (#1); #2 is a "please confirm + note it" rather than a defect.
· fix/serverconn-backpressure

In this commit, we add TellWithoutParking and the NonParkingTeller
interface to the actor baselib. An actor or loop that must never block
indefinitely (a receive goroutine, or a sender holding a database write
transaction) can now hand a message to any TellOnlyRef and get the
right primitive per target: TryTell with an ErrMailboxFull sentinel for
a fixed-capacity in-memory mailbox, and the ordinary transactional Tell
for a durable one, whose enqueue is a DB write that cannot park on a
channel.

Every reference implementation in the package answers for itself
through NonParkingTeller, so the decision happens at send time against
the concrete resolved target rather than at wiring time. A Router
resolves its round-robin selection first and decides for that one ref,
so a mixed key with both durable and in-memory registrations cannot
mis-route a durable target through TryTell's background write, which
would drop the caller's transaction and misreport a full mailbox as a
deadline error. The Map/MapInput/FilterMapInput wrappers delegate the
decision inward, so a transform sitting in front of a router does not
flatten the per-target answer. A reference the package does not
recognise keeps the plain Tell, the direction that preserves a durable
enqueue's transaction atomicity.

The call returns fn.Result[bool] reporting whether the delivery escaped
the caller's transaction: true only for a successful TryTell hand-off,
which is visible to the receiver immediately and survives a rollback.
The serverconn ingress path reads it to know what must not be replayed
when the store retries the transaction body.
In this commit, we make FSM.CurrentState safe to call against a machine
whose executor is parked mid-transition. The old implementation sent
the query with an unbounded blocking channel send and only applied its
one second timeout to the reply half, so a caller could park forever on
the request half with no context to cancel it. This was reachable from
eight call sites in the round actor's receive loop, making it one of
the cheapest ways to wedge that loop.

We add CurrentStateWithContext, which selects on the context for both
the send and the receive, and turn CurrentState into a wrapper that
runs both halves on one DefaultStateQueryTimeout timer of one second.
This is a semantic change for the no-context callers in oor, txconfirm,
and unroll: a query issued against a machine mid-transition previously
parked on the send until the transition finished, and now returns
DeadlineExceeded after the timer. The exposure is theoretical for those
packages, since their machines are driven by AskEvent from the same
goroutine that issues the queries, so a transition cannot be in flight
when a query runs; and every caller already handled the reply half's
one second timeout, so the error path is not new. The new test drives a
query against a machine held mid-transition from a goroutine, so a
regression fails the test instead of hanging it.
In this commit, we give the server connection's ingress path a liveness
surface. Both production wedges of this subsystem produced zero
evidence: no error logs, no metric movement, nothing for an operator to
alert on while the client sat deaf to round events for hours. The
existing server_connected boolean is set once at startup and never
flips on a wedge, so it cannot carry this signal.

We add three collectors. The last-poll timestamp says the ingress loop
itself is alive and reaching the operator. The last-event timestamp
says dispatch is actually landing messages in actor mailboxes. The
deferred counter counts redrives refused because a bounded in-memory
target was full. The split matters: a deaf client keeps polling, so
poll staleness cannot detect a dispatch wedge, and an idle client has a
legitimately stale event gauge. The README's new alerting section
spells out which expression detects which failure, with the deferral
rate as the primary wedge signal.

These gauges are stamped directly from the ingress goroutine rather
than through the metrics actor. That deviation is deliberate and
documented: the goroutine reporting its own liveness must not depend on
yet another actor edge that can itself back up.
@Roasbeef
Roasbeef force-pushed the fix/serverconn-backpressure branch from 3c3c59d to 772093b Compare August 5, 2026 02:38
@Roasbeef

Roasbeef commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

@claude review this

@claude

claude Bot commented Aug 5, 2026 •

Copy link
Copy Markdown

Claude finished @Roasbeef's task in 3m 48s —— View job


Review: serverconn+round — never park the ingress path

I re-read the seven commits with the diff against origin/main, tracing the deferral/cursor arithmetic end-to-end, the replay guard, TellWithoutParking, and the askWallet/FSM-scan/CurrentState changes. This holds up: the invariants are argued at the call sites, the cursor math is internally consistent, and the observability story is coherent. No blocking correctness bug found. Details below, then the two things worth a maintainer decision.


Correctness I traced independently (no action needed)

  • Deferral cursor + servedNonTxSeq handoff is sound — including the case where the two guards trade off. I walked a batch [1=nonTx-request, 2=durable→full]:
    • Cycle 1: request 2… (seq 1) served over the wire (servedNonTxSeq=1), durable seq 2 defers, commit lands at deferredCursor(2,·)=2.
    • While wedged: clampToDeferred drops everything > deferredSeq before splitIngressEnvelopes, so the hoisted request is never re-served on a redrive — the clamp is the active guard.
    • After the target drains, on the cursor-advancing cycle the clamp is gone (deferredSeq=0), and it's redrive.servedNonTxSeq that suppresses the re-serve while batchNextCursor still advances the cursor past the already-answered request. So the two mechanisms cover disjoint windows — the clamp during the wedge, the watermark on the drain cycle — and neither is redundant. That's subtle and correct.
  • servedNonTxSeq never resetting is safe, because event_seq is monotonic per mailbox so a stale high-water mark can't suppress a genuinely new request; and it lives only on the ingress goroutine's stack, so a restart redelivers (the documented at-least-once exposure).
  • deliveredOutsideTx replay guard is correctly scoped and correctly narrow. Only DeliveredByTryTell hand-offs are marked (durable Tell rolls back with the tx and must re-run); the record is built outside the ExecTx closure so it survives body retries; and it is per-runFoldedDispatch, not cross-cycle — widening it would suppress the first delivery of an envelope whose earlier cycle rolled back after the TryTell, trading a benign duplicate for a silent loss. The eventSeq==0 opt-out is the right call for the same reason.
  • askWallet bounds both halves. Passing one askCtx to Ask and Await is the fix that matters — a deadline on Await alone leaves the mailbox enqueue (the half a wedged wallet actually stalls) unbounded. Bounding only round→wallet to break the cycle is documented as deliberate.
  • TellWithoutParkingTo resolves the Router target first, so the bounded-vs-durable decision is made against the one actor the message is handed to, not the key's ANY-of set. Compile-time NonParkingTeller assertions cover every in-tree ref, so the conservative Tell fallback only applies to out-of-tree refs — the safe direction.

1. ErrorS for deferrals — I'd act on Codex's flag, but keep the visibility

Codex flagged dispatch_deferral.go:190 against the repo rule ("error level is only for internal bugs, never external triggers"), and I agree there's a real tension. The nuance that resolves it cleanly: a brief deferral (a target momentarily busy) is normal operation the metric already surfaces; a persistent episode (a target that has stopped draining) is the internal-bug-class signal where Error is defensible. The current throttle (first + every 30s) fires Error on the very first deferral even for an episode that clears on the next redrive.

noteDispatchDeferral already carries everything needed to split these: episode.deferrals and episode.startedAt. Concretely — log the first/interval lines at WarnS, and escalate to ErrorS only once an episode has persisted (e.g. deferrals beyond a small N, or age beyond a couple of deferralLogIntervals). That keeps a permanent wedge paging while honoring the guideline for transient backpressure, and the deferral-rate metric remains the primary alert either way (per your own README). If the team instead decides sustained Error visibility outweighs the rule, please record that decision next to the call — the doc comment today argues "was previously silent," which motivates visibility but not the level choice, and the next reader will hit the lint rule cold. Fix this →

2. CurrentState() (no-ctx) now bounds its send half — silent semantic change for three out-of-scope packages

baselib/protofsm/state_machine.go:338 rewrites the no-context CurrentState() to share one 1s timer across both the request send and the reply. Previously the send half could block for the whole of an in-flight ProcessEvent; now it returns context.DeadlineExceeded after 1s. That's the intended round-actor fix, but CurrentState() (no ctx) is still called from packages the PR title doesn't mention:

  • oor/session_actor.go, oor/session.go
  • txconfirm/fsm_types.go, txconfirm/fee_input_actor.go
  • unroll/actor.go

For a transition that legitimately runs >1s (signing/network work in txconfirm/unroll is plausible), these now see a transient DeadlineExceeded where they previously blocked and returned the true state. unroll returns the error (with a shutdown special-case), so it degrades rather than misbehaves — but it's worth confirming the txconfirm/oor sites treat a transient state-read timeout as skip/retry and won't fail a job on it. If they do, this is an improvement; it just deserves a sentence in the commit body, since it changes semantics for three packages beyond serverconn+round.


Smaller notes (non-blocking)

  • fsmScanContext — one deadline per scan is the right call, and the OnStop motivation (per-FSM budget × N rounds starving MuSig2 cleanup within the 5s teardown budget) is a genuine leak avoided. Note replayCheckpointedServerMessages/handleTimeout/processOutbox pass the raw ctx (per-call 1s) rather than a shared scan ctx — correct, since those are single-FSM reads, not scans.
  • Start boot registration now bounded at 30s (shares the steady-state budget). The doc comment already justifies "failed start + restart beats silent hang"; agreed, and the registration is an idempotent read.
  • store_impl.go NOTE is correctly rewritten: the fold now "works at" the out-of-fn contract via the hoisted sends + dispatch_replay.go record, rather than being the one unfixed exception. The serverconn/doc.go "known residual" (16 routes share one ordered cursor, so a stalled target still delays the others) is honestly stated and scoped as follow-up.
  • Metrics split (LastIngressPoll = loop alive vs LastIngressEvent = dispatch landing, plus the deferred counter) is exactly the right shape given a deaf client keeps polling — poll-staleness alone can't see this failure, which is why both prior incidents produced no evidence.

Overall: approve-shaped from a correctness standpoint. #1 is the only pre-merge decision I'd want a maintainer to make; #2 is "confirm + note in the commit body," not a defect.
· fix/serverconn-backpressure

@Roasbeef
Roasbeef force-pushed the fix/serverconn-backpressure branch from 772093b to fef8edf Compare August 5, 2026 02:47
@Roasbeef

Roasbeef commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

Both items addressed in fef8edf:

  1. Deferral log level split by episode age. The opening line of an episode is now WarnS (a brief deferral is ordinary backpressure the deferred counter already captures, and the repo rule is right that error level is for internal bugs), and the interval re-logs escalate to ErrorS: an episode still open a full deferralLogInterval later means the target has stopped draining, which is exactly the stuck-receive-loop bug class this path exists to expose. The decision is recorded in the doc comment next to the call, so the next reader hits the reasoning rather than the lint rule. The deferred-counter rate stays the paging signal either way.

  2. The no-ctx CurrentState semantic change is now stated in the protofsm commit body, with the reasoning for why the exposure is theoretical for oor/txconfirm/unroll: those machines are driven by AskEvent from the same goroutine that issues the state queries, so a transition cannot be in flight when a query runs, and every caller already handled the reply half's one second timeout, so the error path is not new. One related note: txconfirm converting a state-read error into TxStateFailed is a pre-existing issue reachable before this PR (via ErrStateMachineShutdown and the old reply-half timeout) and is tracked separately in txconfirm: state-query errors are laundered into TxStateFailed #1095.

Thanks for tracing the clamp/watermark disjoint-window interaction independently; that is the subtlest piece of the cursor arithmetic and it now has three independent confirmations (adversarial review, the P model's non-vacuity mutations, and this pass).

In this commit, we fix the wedge behind the recurring "client goes deaf
to round events" incidents. The ingress loop delivered server events
with a blocking, deadline-free Tell into fixed-capacity in-memory
mailboxes (round-client and incoming-vtxo-handler, which the package
docs wrongly described as durable), and it did so while holding the
SQLite global write transaction. Once a target mailbox filled, the sole
ingress goroutine parked forever with the writer pinned. WAL readers
kept answering RPC, so the pod looked healthy while every write in the
process was frozen, and the only log on that path was at trace level.
Two production incidents produced no evidence at all.

Dispatch now goes through TellWithoutParking. A full in-memory mailbox
becomes a deferral instead of a park: the fold commits the cursor at
the deferred envelope's own event_seq, so everything below it is fully
handled (waiters delivered, hoisted requests served, the durable prefix
enqueued in the same transaction) and the undelivered tail is never
acked. The pull loop is the redrive. On a redrive the batch is clamped
to the deferred seq and a served watermark suppresses re-serving
hoisted requests, so a persistent wedge cannot re-answer the same
non-transactional request every cycle. Backpressure gets its own
backoff with a 500ms ceiling and resets on progress, keeping drain
latency well inside the round phase deadlines rather than saturating at
the transport schedule's 30s.

Because the store retries its transaction body on retryable errors, an
in-memory TryTell that already landed must not run again on the next
attempt. A per-dispatch record of out-of-transaction deliveries, keyed
by event_seq and scoped to a single folded dispatch, makes the replay
idempotent. In-memory dispatch stays inside the fold because the single
ordered pass keeps per-target delivery order and the serve-before-ack
crash safety with no second bookkeeping pass; cross-target ordering is
not a consideration either way, since a durable enqueue is invisible to
its consumer until commit, which lands after every in-memory hand-off
in the batch. Hoisting the in-memory half out of the fold would delete
the replay record entirely and is recorded as a follow-up.

Deferrals log at error level once per episode with periodic re-logging
and increment a counter, so the next wedge pages instead of hiding. One
residual is documented in the package docs rather than fixed here: all
routes share one strictly ordered cursor, so a stalled target still
delays events for the others. This change removes the writer pin and
makes that state visible; it does not decouple the lanes.

The regression tests wire a real actor system, a real event router, and
the writer-lock harness. They fail on the pre-fix code in seconds, one
per mechanism: the storm test observes the parked transaction, the
redrive test counts unbounded hoisted re-serves, the recovery test
times the drain, and the replay test counts duplicate deliveries.
In this commit, we close the round actor's two remaining ways to park
its receive loop indefinitely, the loop that gates every inbound round
event.

The first is the round-to-wallet Ask cycle: the round actor asked the
wallet with no deadline while the wallet's own handlers ask the round
actor back, so two full mailboxes could produce a circular wait that
never resolves. The onchain intent replayer already bounds the
identical Ask and documents the hazard; the live paths never got that
treatment. The new askWallet helper applies one WithTimeout to both the
enqueue and the Await, since a deadline on the Await alone leaves the
blocking enqueue unbounded. The 30s default is overridable through the
config, and the actor's lifecycle context stays the parent so shutdown
still cancels a pending Ask. Bounding one direction is enough to break
the cycle; the wallet-to-round direction is left alone deliberately,
because an Ask's context becomes the receiver's turn context, and a
deadline there would bound money-path FSM registration work.

The second is scan cost: state queries now carry a context, and the
call sites that scan every tracked round share a single deadline per
scan instead of paying the per-query budget once per FSM. With N
tracked rounds mid-transition, a single message could previously spend
3N seconds in state queries; OnStop could burn its whole cleanup budget
on queries and strand external-signer session cleanup. Every scanning
caller already degrades by skipping an unreadable round, so the shared
budget changes worst-case latency, not behaviour.

The new test parks the wallet double behind a gate that honors the Ask
context, so it faithfully models a full mailbox: pre-fix the round
actor visibly parks on the stalled wallet, post-fix the Ask times out,
logs at error level, and the loop keeps taking turns.
In this commit, we extend the P model suite to cover the delivery half
of the connection actor's ingress loop, which the existing cursor-fold
model deliberately abstracted away.

ingress_fold.p says the persisted cursor may never cover an envelope
whose local enqueue did not commit, and it gets there by treating every
dispatch as a durable enqueue that lives and dies with the write
transaction. The dispatch table does not work that way. It has four
kinds of destination and only one of them is transactional: a durable
target rolls back with the transaction, a hoisted request has already
left the process by the time the transaction opens, a waiter response is
handed over in memory ahead of it, and a bounded in-memory target is a
fixed-capacity actor mailbox that is delivered to from inside the
transaction but is not in it. That last one is also the only destination
that can refuse, and refusing is what the whole deferral path exists to
handle.

The new model, ingress_deferral.p, states what the loop owes each of
those four. The committed cursor may only cover envelopes some target
actually received, so a deferral commits the prefix and stops at the
undelivered envelope. A store that replays its transaction body after a
retryable error may not re-send what the previous attempt already handed
to a bounded mailbox, because that delivery was never in the transaction
to take back. Redelivery across redrives and crashes is expected, but at
most once per epoch, so the duplicate count is bounded by how many times
the loop went around rather than by how many internal steps each turn
took. And a request answered ahead of the commit is answered at most
once per process lifetime, however many times a wedged target makes the
loop re-pull it.

Above all, the ingress goroutine never parks. It is the process's only
consumer of the remote mailbox and it holds the database's single writer
while it works, so a blocking send into a full fixed-capacity mailbox is
not a slowdown but a permanently deaf client that still answers reads,
still reports healthy, and logs nothing.

A model that cannot detect the bug it exists for is worthless, so the
pre-fix implementation stays reachable through IngressPipelineConfig.
Each counterexample flips exactly one field of the production profile
and fails: ParkingBlockingSend wedges the writer, a cleared
track_tx_deliveries duplicates in-memory deliveries across store
retries, and a cleared served_watermark re-answers a hoisted request
once per backoff cycle. The park is caught twice over, once as a safety
violation and once as the starvation an operator actually sees.

The model also declines to flatter the implementation on two points. The
nonTx serve genuinely can repeat across a crash, because the answer has
to precede the commit, so the monitor keys on the process incarnation
and rejects only a repeat within one lifetime rather than asserting the
residual away. And removing the redrive clamp while keeping the served
watermark breaks no property here: the clamp earns its place as a bound
on repeated work, not as a second safety net.

check.sh picks the new cases up alongside the durable mailbox suite, and
while we are here it also picks up the three ingress cursor-fold cases,
which have been compiled but never checked since they were added.
In this commit, we add property tests for the invariants the ingress
loop's deferral and redrive path has to hold, and for the two pure
functions that decide the cursor a deferred cycle commits.

The reason for property tests rather than more examples is the size of
the state space. One backpressure episode is a cross product of where in
the pulled window the deferral falls, how many times the store replays
its transaction body after a retryable error, which hoisted serves fail,
and whether the target drained between redrives. The invariants are
about what survives all of that rather than about any one path through
it, and the example tests in ingress_backpressure_test.go each pin a
single point in that space.

Five things are pinned. The first is cursor safety, which is the one the
rest exist to protect: the committed cursor may never cover an envelope
that no destination accepted. An exclusive cursor acks everything below
it, so a cursor that ran one envelope too far is a silently dropped
round event with no local symptom. We check it against a handled set
built out of what the real dispatchers returned, not out of a second
model of what they ought to have returned.

The second is no loss and bounded duplication. Every envelope arrives
once the mailboxes drain, and a delivery into a bounded in-memory
mailbox happens at most once per folded dispatch however many times the
store runs the body, because that delivery is not in the transaction and
a rollback does not take it back. The third is ordering: the target's
own lane must hold exactly the deliveries the fold made, in the order it
made them. The fourth is that a hoisted request is answered exactly once
per episode, however many redrives the episode takes, because the answer
goes to the operator over the network while the cursor stays put. The
fifth is the redrive schedule's ceiling, the minimum of maxDeferralDelay
and the configured RetryMaxDelay, together with the cursor advance the
loop resets deferFailCount on.

The end-to-end property drives the real thing: a real actor system, a
real fixed-capacity mailbox, the real EventRouter dispatcher, and
runFoldedDispatch itself, with only the loop control around it mirrored
from ingressLoop. A property test of a parallel model proves nothing
about the code, so the two dispatchers the harness installs wrap the
production ones and do no more than record what they did, including
asking the fold's own in-flight record whether a delivery was about to
be suppressed. clampToDeferred and deferredCursor are small enough to
drive directly, so they get their own properties over randomized batch
shapes, gap layouts and deferral points.

We checked that the properties bite by mutating the code they cover: an
off-by-one in deferredCursor, a clamp that returns the pulled window's
cursor instead of the deferred envelope's, a replay record that never
suppresses, a dropped servedNonTxSeq watermark, and a redrive schedule
with no ceiling are each caught. Two thirds of the generated episodes
reach a deferral, and the test asserts that at least one did so a
generator that drifted cannot make this pass vacuously. Test files only;
no production code changes.
@Roasbeef
Roasbeef force-pushed the fix/serverconn-backpressure branch from fef8edf to e1f0216 Compare August 5, 2026 03:02
@Roasbeef Roasbeef added the backport-v0.1.x-branch Backport this merged PR to v0.1.x-branch label Aug 5, 2026
@Roasbeef
Roasbeef merged commit f9d913e into main Aug 5, 2026
37 of 38 checks passed
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-v0.1.x-branch Backport this merged PR to v0.1.x-branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant