Repository navigation
actor: never park a receive turn on a full in-memory mailbox - #1371
Conversation
|
Doc advisory: diff --git a/baselib/actor/AGENTS.md b/baselib/actor/AGENTS.md
index 39008064..12bd4c35 100644
--- a/baselib/actor/AGENTS.md
+++ b/baselib/actor/AGENTS.md
@@ -10,7 +10,10 @@ crash-safe at-least-once delivery with exactly-once deduplication.
- `Actor[M, R]` — Generic actor with typed message `M` and response `R`. Processes messages sequentially from its mailbox.
- `ActorBehavior[M, R]` — Interface that actors implement: `Start`, `Receive`, `Stop`.
-- `ActorConfig[M, R]` — Configuration for actor creation (behavior, mailbox, codec, delivery store).
+- `ActorConfig[M, R]` — Configuration for actor creation (behavior, mailbox,
+ codec, delivery store). `MailboxSize` is the channel capacity;
+ `MailboxOverflowLimit` is the hard cap on the overflow queue behind it
+ (zero selects `DefaultMailboxOverflowLimit`).
- `ActorRef[M, R]` — Typed reference for sending messages to an actor (`Tell`, `TryTell`, `Ask`).
- `TellOnlyRef[M]` — Fire-and-forget reference (no response type). `Tell` blocks for mailbox room unless made with an active turn context (then it overflows instead), `TryTell` never blocks.
- `ActorSystem` — Container managing actor lifecycles, registration, and
@@ -82,8 +85,22 @@ crash-safe at-least-once delivery with exactly-once deduplication.
`CallerCtx` in `context.WithTimeout` itself before handing it to
`OnComplete` — that wrap is the sole bound on the continuation. Returns
false for Tells, DurableAsks, and redelivered asks whose caller is gone.
-- `ChannelMailbox[M, R]` — In-memory channel-based mailbox (non-durable, for lightweight actors). A bounded channel fronted by an overflow queue: `Send` with an active turn context (see `turn.go`) never blocks and spills into the overflow, which the receive loop refills into the channel after every receive (so a non-empty overflow implies a non-empty channel). Lock order is `mu` (read side) then `overflowMu`. `NewChannelMailbox` takes `WithOverflowLimit` and `WithMailboxID` options; `OverflowLen()` reports the backlog.
+- `ChannelMailbox[M, R]` — In-memory channel-based mailbox (non-durable, for lightweight actors). A bounded channel fronted by an overflow queue: `Send` with an active turn context (see `turn.go`) never blocks and spills into the overflow, which the receive loop refills into the channel after every receive (so a non-empty overflow implies a non-empty channel). Lock order is `mu` (read side) then `overflowMu`. `NewChannelMailbox` takes `WithOverflowLimit` and `WithMailboxID` options; `OverflowLen()` reports the backlog. After `Close`, `Drain` yields the channel envelopes first and the overflow second, which is send order because the overflow always trails the channel.
- `Mailbox[M, R]` — Interface for actor message queues: `Send(ctx, env) error` (blocking, except for a `ChannelMailbox` send made with an active turn context; returns `ErrMailboxClosed`, `ErrActorTerminated`, or a context error on failure), `TrySend(env) error` (non-blocking), `Receive(ctx) iter.Seq[envelope]`, `Close()`, `IsClosed() bool`, `Drain() iter.Seq[envelope]`.
+- `DefaultMailboxOverflowLimit` — 100000. The overflow cap a `ChannelMailbox`
+ falls back to when `ActorConfig.MailboxOverflowLimit` / `WithOverflowLimit`
+ is zero or negative.
+- `TurnActor(ctx) (string, bool)` — ID of the actor whose receive turn `ctx`
+ belongs to. Both `Actor.process` and `DurableActor.processDelivery` stamp
+ this marker on the context they hand the behavior and clear it when the
+ behavior returns, so the bool is false outside a turn and for a context that
+ outlived the turn it was created in (a goroutine the behavior spawned is not
+ the actor's own goroutine and may park like any other producer). This is the
+ marker `ChannelMailbox.Send` keys its non-parking path off.
+- `WithTurnForTest(ctx, actorID) (context.Context, func())` — Stamps the same
+ turn marker from outside the package, as the runtime does before invoking a
+ behavior, and returns the function that ends the turn. For tests that
+ exercise turn-sensitive paths without standing up an actor.
- `isExpectedShutdownErr(err) bool` — Internal helper that classifies errors as expected during teardown: context cancellation/deadline, closed DB handle ("sql: database is closed", "sql: connection is already closed", "use of closed network connection"). Used by the lease loop to demote shutdown-path failures to debug instead of warn-flooding test artifacts at itest tail.
- `Message.CorrelationKey() string` — Per-message FIFO key consumed by the
durable mailbox's claim path. Non-empty keys participate in per-key FIFO:
@@ -143,7 +160,8 @@ crash-safe at-least-once delivery with exactly-once deduplication.
a parked external sender is handed each freed slot ahead of the overflow.
`TryTell` is unchanged and still returns `ErrMailboxFull` immediately, and also while
the overflow is non-empty. The overflow has a hard cap
- (`ActorConfig.MailboxOverflowLimit`, default 100000); exceeding it logs at
+ (`ActorConfig.MailboxOverflowLimit`, default
+ `DefaultMailboxOverflowLimit` = 100000); exceeding it logs at
critical severity and panics, since a consumer that cannot catch up is a bug
and a restart that rebuilds from durable state beats a silent deadlock.
`OverflowLen` reports the current backlog, and a warning is logged once per
diff --git a/baselib/actor/CLAUDE.md b/baselib/actor/CLAUDE.md
index 39008064..12bd4c35 100644
--- a/baselib/actor/CLAUDE.md
+++ b/baselib/actor/CLAUDE.md
@@ -10,7 +10,10 @@ crash-safe at-least-once delivery with exactly-once deduplication.
- `Actor[M, R]` — Generic actor with typed message `M` and response `R`. Processes messages sequentially from its mailbox.
- `ActorBehavior[M, R]` — Interface that actors implement: `Start`, `Receive`, `Stop`.
-- `ActorConfig[M, R]` — Configuration for actor creation (behavior, mailbox, codec, delivery store).
+- `ActorConfig[M, R]` — Configuration for actor creation (behavior, mailbox,
+ codec, delivery store). `MailboxSize` is the channel capacity;
+ `MailboxOverflowLimit` is the hard cap on the overflow queue behind it
+ (zero selects `DefaultMailboxOverflowLimit`).
- `ActorRef[M, R]` — Typed reference for sending messages to an actor (`Tell`, `TryTell`, `Ask`).
- `TellOnlyRef[M]` — Fire-and-forget reference (no response type). `Tell` blocks for mailbox room unless made with an active turn context (then it overflows instead), `TryTell` never blocks.
- `ActorSystem` — Container managing actor lifecycles, registration, and
@@ -82,8 +85,22 @@ crash-safe at-least-once delivery with exactly-once deduplication.
`CallerCtx` in `context.WithTimeout` itself before handing it to
`OnComplete` — that wrap is the sole bound on the continuation. Returns
false for Tells, DurableAsks, and redelivered asks whose caller is gone.
-- `ChannelMailbox[M, R]` — In-memory channel-based mailbox (non-durable, for lightweight actors). A bounded channel fronted by an overflow queue: `Send` with an active turn context (see `turn.go`) never blocks and spills into the overflow, which the receive loop refills into the channel after every receive (so a non-empty overflow implies a non-empty channel). Lock order is `mu` (read side) then `overflowMu`. `NewChannelMailbox` takes `WithOverflowLimit` and `WithMailboxID` options; `OverflowLen()` reports the backlog.
+- `ChannelMailbox[M, R]` — In-memory channel-based mailbox (non-durable, for lightweight actors). A bounded channel fronted by an overflow queue: `Send` with an active turn context (see `turn.go`) never blocks and spills into the overflow, which the receive loop refills into the channel after every receive (so a non-empty overflow implies a non-empty channel). Lock order is `mu` (read side) then `overflowMu`. `NewChannelMailbox` takes `WithOverflowLimit` and `WithMailboxID` options; `OverflowLen()` reports the backlog. After `Close`, `Drain` yields the channel envelopes first and the overflow second, which is send order because the overflow always trails the channel.
- `Mailbox[M, R]` — Interface for actor message queues: `Send(ctx, env) error` (blocking, except for a `ChannelMailbox` send made with an active turn context; returns `ErrMailboxClosed`, `ErrActorTerminated`, or a context error on failure), `TrySend(env) error` (non-blocking), `Receive(ctx) iter.Seq[envelope]`, `Close()`, `IsClosed() bool`, `Drain() iter.Seq[envelope]`.
+- `DefaultMailboxOverflowLimit` — 100000. The overflow cap a `ChannelMailbox`
+ falls back to when `ActorConfig.MailboxOverflowLimit` / `WithOverflowLimit`
+ is zero or negative.
+- `TurnActor(ctx) (string, bool)` — ID of the actor whose receive turn `ctx`
+ belongs to. Both `Actor.process` and `DurableActor.processDelivery` stamp
+ this marker on the context they hand the behavior and clear it when the
+ behavior returns, so the bool is false outside a turn and for a context that
+ outlived the turn it was created in (a goroutine the behavior spawned is not
+ the actor's own goroutine and may park like any other producer). This is the
+ marker `ChannelMailbox.Send` keys its non-parking path off.
+- `WithTurnForTest(ctx, actorID) (context.Context, func())` — Stamps the same
+ turn marker from outside the package, as the runtime does before invoking a
+ behavior, and returns the function that ends the turn. For tests that
+ exercise turn-sensitive paths without standing up an actor.
- `isExpectedShutdownErr(err) bool` — Internal helper that classifies errors as expected during teardown: context cancellation/deadline, closed DB handle ("sql: database is closed", "sql: connection is already closed", "use of closed network connection"). Used by the lease loop to demote shutdown-path failures to debug instead of warn-flooding test artifacts at itest tail.
- `Message.CorrelationKey() string` — Per-message FIFO key consumed by the
durable mailbox's claim path. Non-empty keys participate in per-key FIFO:
@@ -143,7 +160,8 @@ crash-safe at-least-once delivery with exactly-once deduplication.
a parked external sender is handed each freed slot ahead of the overflow.
`TryTell` is unchanged and still returns `ErrMailboxFull` immediately, and also while
the overflow is non-empty. The overflow has a hard cap
- (`ActorConfig.MailboxOverflowLimit`, default 100000); exceeding it logs at
+ (`ActorConfig.MailboxOverflowLimit`, default
+ `DefaultMailboxOverflowLimit` = 100000); exceeding it logs at
critical severity and panics, since a consumer that cannot catch up is a bug
and a restart that rebuilds from durable state beats a silent deadlock.
`OverflowLen` reports the current backlog, and a warning is logged once perHow to apply: copy the diff above into a file and https://github.com/lightninglabs/wavelength/actions?query=branch%3Aactor-nonparking-sends |
d0f661a to
23f00e1
Compare
23f00e1 to
85dacc1
Compare
|
Applied the doc advisory to |
85dacc1 to
8de4db3
Compare
|
/gateway review |
There was a problem hiding this comment.
Gateway review — 2 findings
🔴 0 Blocker · 🟠 1 Major · 🟡 1 Minor · 🔵 0 Nit
Summary
This PR stamps a turn marker on the context that Actor.process and DurableActor.processDelivery pass to a behavior. When ChannelMailbox.Send gets a context with an active turn, it sends into the channel if there is room and otherwise appends to an overflow queue, so it never parks. The point is to break the "Tell into a full mailbox" edge that the listed deadlocks share. The mailbox changes hold up under close reading. Lock order is consistently mu (read) then overflowMu. The refill invariant holds: the refill is the receiver's only step after a receive, it takes overflowMu, so it always sees an append that raced with the receive. TrySend reporting full while the overflow is non-empty correctly stops a TryTell overtaking an earlier in-turn Tell. Drain ordering is sound because nothing writes the overflow once closed is set under the write lock.
The main problem is the durable actor side. The PR marks durable turns the same way as channel-actor turns, but durable actors recover panics. So the "loud restart" the overflow cap is designed around never happens for a durable sender. Instead, a durable actor that used to be paced by a slow in-memory consumer now has no backpressure, and its Tell messages get retried and eventually dead-lettered. The serial flag that would separate pool workers from serial turns is set but never read. There is also some leftover guidance in the README that contradicts the new semantics.
Lower-severity findings (1)
🟡 F2 (Minor) — TryTell guidance now contradicts the in-turn Tell semantics · baselib/actor/README.md:223
The TryTell section just below this hunk still says "Reach for this whenever the sender is itself inside a receive goroutine. A blocking Tell there parks the sender's own message processing…". The doc comment on actorRefImpl.TryTell in actor.go makes the same claim ("Callers that run inside another actor's receive loop use this so a backlogged target cannot stall their own message processing"). With this PR, an in-turn Tell no longer parks, and the Tell paragraph you just updated says so. Reword both to the position bounded_delivery.go now takes: inside a turn, TryTell is only for callers that want an ErrMailboxFull signal so they can drop the message instead of queueing it.
Bot commands
/gateway re-review— re-run after pushing changes (maintainers)/gateway dismiss <id>— silence a finding (maintainers)/gateway explain <id>— elaborate on a finding (anyone)
|
|
||
| // Mark the context as this actor's turn. With a single worker a parked | ||
| // turn parks the whole actor; with a pool it parks only this worker. | ||
| processCtx, endTurn := beginTurn( |
There was a problem hiding this comment.
🟠 F1 (Major) — Durable senders lose backpressure and the overflow panic is swallowed · baselib/actor/durable_actor.go:659
Every durable turn is now marked as a turn, so its Tells into a ChannelMailbox never block. If the downstream actor is slow, a durable actor working through a database backlog (for example after a restart) fills the target's overflow at database speed instead of being slowed down to the consumer's pace. When the cap is hit, the panic is caught by executeBehaviorSafely / runExecSafely and turned into an error result. The message is then nacked and retried rather than the process restarting. On each retry the turn re-sends the Tells that had already been queued before the panic, which duplicates them. Once the Tell retry policy gives up, the message is dead-lettered because a downstream actor was slow.
Why this matters
The PR's argument for panicking at the cap ("a loud restart that rebuilds from durable state beats a silent wedge") only holds for Actor.process, which has no recover. On the durable path the result is a Critical log on every attempt, burned attempts, duplicated fan-out, and eventually dead letters. Pool workers are hit hardest. beginTurn(processCtx, a.id, a.numWorkers <= 1) records serial=false precisely because a parked pool worker cannot stall the whole actor, but ChannelMailbox.Send only checks activeTurn(ctx) and never reads serial. So pool workers lose backpressure even though they were never part of the deadlock this PR targets.
I can't tell from the loaded files which durable actors send into channel-mailbox actors at volume. That determines how likely this is to happen in practice.
Fix options:
- Only take the non-parking path for
serialturns, so durable pool workers keep blocking backpressure. - Have the durable panic recovery re-panic on the overflow-limit panic (use a typed sentinel value), so the durable side gets the same fail-loud behavior.
- Leave durable turns unmarked until the follow-up PRs define their semantics.
|
🤖 gateway audit metadata for this PR — auto-generated, please don't edit. |
|
A note on where this series goes after the three PRs, so the scope is on record here. Once #1371 to #1373 land, the guarantee we want is that blocking inside a
Item 1 is the gate for the rest, and it lands as a follow-up stacked on top of this one. Nothing here changes the default, so none of it is needed to merge this stack. |
5d33c93 to
b46eadf
Compare
|
One test change in this PR is worth flagging. If we would rather external sends never overtake an already overflowed message, the mailbox could make a non-turn send wait for the overflow to drain before it takes a slot. That costs some fairness machinery and I have left it out unless someone wants that ordering guarantee. |
In this commit, we have the runtime stamp the context it hands to a behavior with the receive turn it belongs to: the actor's ID, whether the actor is serial (a single goroutine, so a parked turn parks the whole actor), and whether the turn is still running. Both the channel actor loop and the durable actor's processDelivery mark the turn. The marker goes inactive once the behavior returns, so a goroutine spawned during the turn that keeps the context is not mistaken for the actor's own goroutine later on. Nothing reads the marker yet. It's the hook that lets the mailbox and futures tell a send or await made on an actor's own goroutine apart from one made by an external producer, which the following commits use to keep a turn from ever parking on another actor.
In this commit, we give ChannelMailbox an overflow queue behind its channel. A Send made with the context of an active receive turn never blocks: it uses the channel when there is room and nothing is queued ahead of it, and otherwise appends to the overflow. Sends from any other context keep the blocking backpressure. Repeated deadlocks came from one shape. An actor's receive goroutine does a blocking Tell into a peer's full mailbox while that peer waits on the first actor, so both park forever. Removing the parking edge in the framework fixes every call site at once, where TryTell plus stash and retry had to be applied one site at a time. The receive loop refills the channel from the overflow after every receive, so a non-empty overflow implies a non-empty channel and the receiver never sleeps while envelopes wait. Appending whenever the overflow is non-empty keeps each sender's order, and TrySend reports a full mailbox while the overflow is non-empty so it cannot overtake an earlier in-turn send. Drain yields the channel and then the overflow. The overflow has a hard cap, 100000 by default. Exceeding it logs at critical severity and panics, since a consumer that cannot catch up is a bug and a restart is better than a silent deadlock. Episodes are logged at debug, with one warning per episode at a tenth of the cap.
In this commit, we add MailboxOverflowLimit to ActorConfig and pass it, along with the actor ID, to the channel mailbox in NewActor. A zero value selects the default cap, and the ID lets overflow logs name the actor whose consumer is falling behind.
In this commit, we add a regression test using real actors with a one-slot mailbox. Actor A Tells actor B many messages from inside one turn, and B Asks A for every message it handles. Before in-turn sends stopped parking, A blocked on B's full mailbox, B blocked on A's reply, and neither made progress. The test asserts that every message is processed and every Ask answered within a bounded time.
In this commit, we replace the rule against calling Tell from a receive goroutine without a bound with the new contract. A Tell or Ask made with the turn context never parks on a full channel mailbox, while a send made with any other context still can, so the turn context must be passed through. TryTell is unchanged. We also describe the overflow queue, its lock order, and its hard cap.
b46eadf to
c20bc88
Compare
In this PR, we remove one of the two ways an actor can deadlock against another actor: a turn parking on a full in-memory mailbox. A send made with the context of an active receive turn now never blocks on a
ChannelMailbox. It goes into the channel if there's room, and otherwise into an overflow queue behind it. Sends from anything that isn't an actor turn (ingress loops, RPC handlers, background goroutines) keep today's blocking backpressure.This is the first of a short series that addresses the deadlock class at the framework level, rather than at each call site. The plan is laid out in this comment. In short, every mailbox deadlock we've hit so far (#1082, #1093, #1361, #1364, #1368, and the shape #394 described back in May) comes from a receive goroutine picking up one of two wait edges: a plain
Tellinto a full boundedChannelMailbox, or anAwaiton another actor's reply insideReceive. If neither edge can exist inside a turn, there's nothing left to form a cycle. This PR removes the first edge. The follow-ups addAskThenplus an await-in-turn policy for the second, and a wait-for cycle detector as a backstop.The turn marker
The first commit has the runtime stamp the context it hands to a behavior with the turn it belongs to: the actor ID, whether the actor is serial, and an
activebit that's cleared when the behavior returns. Both the channel actor loop and the durable actor'sprocessDeliverymark the turn. Theactivebit matters for goroutines a behavior spawns with its context: once the turn is over they're ordinary producers again, and they get ordinary backpressure. On an Ask turn, the context is built from the caller's context, so the caller's marker rides along.beginTurnalways installs a fresh marker, so the callee's turn shadows it.Overflow instead of parking
ChannelMailbox.Sendwith an active turn takes a non-blocking path (enqueueOverflow). If the overflow is empty and the channel has room, the envelope goes into the channel. Otherwise it's appended to the overflow. "Append whenever the overflow is non-empty" is what preserves per-sender FIFO. For the same reason,TrySendreportsErrMailboxFullwhile the overflow is non-empty, so aTryTellcan't overtake an earlier in-turnTellfrom the same sender.The receive loop refills the channel from the overflow after every receive. That maintains the invariant that a non-empty overflow implies a non-empty channel, so the receiver never sleeps while overflow envelopes wait. The argument is spelled out on
refillFromOverflow. Lock order ismu(read side) thenoverflowMueverywhere, andDrainyields channel items then overflow items, so shutdown routes both to the DLO and fails pending Asks the same way.The overflow is unbounded in normal operation, BEAM style, but it has a hard cap (
ActorConfig.MailboxOverflowLimit, default 100k). Hitting the cap makes an in-turn send returnErrMailboxOverflow(enqueuing nothing) and logs at Critical once per overflow episode. It deliberately does not panic: durable actors recover behavior panics, so a panic there would be swallowed and a send made after the turn committed (arming a retry timer, say) would be lost with no restart. A durable sender instead gets the error as a normal behavior failure, which is nacked and retried, so the cap is also its backpressure signal; its sends must tolerate redelivery, as they already must for at-least-once delivery. There's a Warn once per episode at a tenth of the cap, and an Info when it drains. One caveat, documented at the refill site: Go hands a freed slot straight to a parked external sender, so under sustained external saturation the overflow doesn't drain and the cap is what eventually fires. That trades "slow" for "loud" under overload, which is the direction we want, and the error is what makes it loud without taking the process down.Testing
TestInTurnTellDoesNotDeadlockAskCyclereproduces the classic shape with real actors andMailboxSize: 1: actor A Tells B 50 messages from its turn, while each of B's turns Asks A and awaits the reply. With the in-turn path forced off, the test fails with the actors wedged. With it on, it passes in about a second. The mailbox tests cover per-sender FIFO across channel and overflow, blocking for non-turn and ended-turn contexts,TrySendunder overflow,Close/Drainordering, the cap panic, and a concurrent stress run mixing in-turn and parked external senders. Everything passes under-race, as do thetimeout,round,vtxo,serverconn,oor,unrollandcredittest suites.Follow-ups
With this in place, the per call site
TryTell+ retain + retry patterns (including routing overflow through the timeout actor) stop being necessary for correctness when the sender passes its turn context through. We'll simplify those in follow-ups rather than here, so this PR stays a framework-only change.