Repository navigation
actor: add AskThen and flag Await inside a receive turn - #1372
Conversation
|
This PR's changes to The doc commit ( diff --git a/baselib/actor/AGENTS.md b/baselib/actor/AGENTS.md
index 8e38d136..4a4e1181 100644
--- a/baselib/actor/AGENTS.md
+++ b/baselib/actor/AGENTS.md
@@ -88,20 +88,27 @@ crash-safe at-least-once delivery with exactly-once deduplication.
`wrap(result)` to `self` as an ordinary message handled in a later turn. The
result is a value, the callee's error, a send error, or
`context.DeadlineExceeded`; a non-positive timeout yields the deadline error
- without sending. The Ask keeps ctx's values (including the transaction, like
- a plain `Ask`) but not its cancellation. The reply is delivered with a
- context stripped of the transaction and cancellation, and is dropped at
- debug level if `self` has stopped. `wrap` must be pure: it runs on a helper
- goroutine and must not touch actor state.
+ without sending. The Ask keeps ctx's values but not its cancellation, so it
+ is bounded only by `timeout` and outlives the turn. What the callee sees of
+ the caller's transaction depends on its mailbox: a durable callee enqueues
+ synchronously, before `AskThen` returns, so that enqueue joins the caller's
+ transaction, but the envelope's caller context has the transaction stripped,
+ so no callee ever runs its turn inside it, and a channel-mailbox callee
+ (which runs after the caller's turn has ended) never sees it at all. The
+ reply is delivered with a context stripped of the transaction and
+ cancellation, and is dropped at debug level if `self` has stopped. `wrap`
+ must be pure: it runs on a helper goroutine and must not touch actor state.
- `ErrAwaitInTurn` / `AwaitInTurnPolicy` / `SetAwaitInTurnPolicy(p)` —
Process-wide policy for `Future.Await` on an incomplete future from inside a
receive turn. `AwaitInTurnWarn` (default) logs at info once per call site
(keyed on the caller's PC) and waits as before; `AwaitInTurnAllow` does
nothing; `AwaitInTurnError` returns `ErrAwaitInTurn` without waiting. A
completed future always returns its value, and a context whose turn has
- ended is unaffected. `ThenApply`, `OnComplete` and `MapRef.Ask` await on
- helper goroutines through the unexported `awaitInternal`, so they never trip
- the guard.
+ ended is unaffected. `ThenApply` and `OnComplete` await on helper goroutines
+ through the unexported `awaitInternal`, and `MapRef.Ask` goes through the
+ `awaitFuture` helper, which uses `awaitInternal` for this package's own
+ futures and falls back to the public `Await` for any foreign `Future`
+ implementation, so framework helpers never trip the guard.
- `ChannelMailbox[M, R]` — In-memory channel-based mailbox (non-durable, for lightweight actors). A bounded channel fronted by an overflow queue: `Send` with an active turn context (see `turn.go`) never blocks and spills into the overflow, which the receive loop refills into the channel after every receive (so a non-empty overflow implies a non-empty channel). Lock order is `mu` (read side) then `overflowMu`. `NewChannelMailbox` takes `WithOverflowLimit` and `WithMailboxID` options; `OverflowLen()` reports the backlog.
- `Mailbox[M, R]` — Interface for actor message queues: `Send(ctx, env) error` (blocking, except for a `ChannelMailbox` send made with an active turn context; returns `ErrMailboxClosed`, `ErrActorTerminated`, or a context error on failure), `TrySend(env) error` (non-blocking), `Receive(ctx) iter.Seq[envelope]`, `Close()`, `IsClosed() bool`, `Drain() iter.Seq[envelope]`.
- `isExpectedShutdownErr(err) bool` — Internal helper that classifies errors as expected during teardown: context cancellation/deadline, closed DB handle ("sql: database is closed", "sql: connection is already closed", "use of closed network connection"). Used by the lease loop to demote shutdown-path failures to debug instead of warn-flooding test artifacts at itest tail.
@@ -148,7 +155,7 @@ crash-safe at-least-once delivery with exactly-once deduplication.
because begin/commit failures happen outside those operation-level logs.
- `ServiceKey` lookup via `Receptionist` is type-safe: mismatched types return `ErrServiceKeyTypeMismatch`.
- `RestartMessage` has `RestartPriority` (MaxInt32) ensuring it is processed before all other messages on recovery.
-- Transaction context (`WithTx`/`RequireTx`) enables same-DB-transaction joining between actors and their callers.
+- Transaction context (`WithTx`/`RequireTx`) enables same-DB-transaction joining between actors and their callers. `AskThen` is the deliberate exception on the receiving side: the durable enqueue still joins the caller's transaction, but the transaction is stripped from the envelope's caller context, because that context is the base of the callee's turn context and the caller's turn has committed or rolled back by the time the callee runs. The detached marker is cleared along with it, so the callee's own later `Ask` calls — which do wait for their replies — keep propagating their transaction normally.
- `Mailbox.Send` returns the exact failure error (`ErrMailboxClosed`, `ErrActorTerminated`, `context.Canceled`, `context.DeadlineExceeded`) rather than a boolean; `Tell` and `Ask` propagate this directly to callers.
- **A `Tell` or `Ask` made with the turn's context never parks on a full
channel mailbox.** The runtime marks the context handed to a behavior as an
@@ -181,7 +188,16 @@ crash-safe at-least-once delivery with exactly-once deduplication.
process can switch to `AwaitInTurnError`. Awaiting from outside a turn, from
a goroutine that outlives the turn, or on an already-complete future is
always fine. Awaits that live in this package's own helper goroutines go
- through `awaitInternal` and must keep doing so.
+ through `awaitInternal` (or `awaitFuture`) and must keep doing so.
+- **An `AskThen` reply is delivered exactly once, even when the request never
+ happened, so the handler must tolerate a stale one.** A timeout abandons the
+ wait, not the request: the callee may still process the message, and its late
+ reply is then discarded. A durable caller's turn may also roll back after
+ `AskThen` returned, which rolls the durable Ask row back with it while the
+ helper goroutine still delivers a wrapped `context.DeadlineExceeded`. A
+ behavior must therefore record that a request is pending in the turn that
+ issues it and treat a wrapped result it has no pending entry for as stale and
+ ignore it, rather than assuming every delivered reply answers a live request.
- During daemon teardown, the underlying DB is closed before every actor's lease loop has wound down. The lease loop uses `isExpectedShutdownErr` to demote these "database is closed" errors to debug level; real operational errors still surface as warnings because neither the actor context nor the outer context is done in those cases.
- **Postpone preserves the attempt budget; nack spends it.** A nack increments
`attempts` on every release, and both the claim and the peek queries filter
diff --git a/baselib/actor/CLAUDE.md b/baselib/actor/CLAUDE.md
index 8e38d136..4a4e1181 100644
--- a/baselib/actor/CLAUDE.md
+++ b/baselib/actor/CLAUDE.md
@@ -88,20 +88,27 @@ crash-safe at-least-once delivery with exactly-once deduplication.
`wrap(result)` to `self` as an ordinary message handled in a later turn. The
result is a value, the callee's error, a send error, or
`context.DeadlineExceeded`; a non-positive timeout yields the deadline error
- without sending. The Ask keeps ctx's values (including the transaction, like
- a plain `Ask`) but not its cancellation. The reply is delivered with a
- context stripped of the transaction and cancellation, and is dropped at
- debug level if `self` has stopped. `wrap` must be pure: it runs on a helper
- goroutine and must not touch actor state.
+ without sending. The Ask keeps ctx's values but not its cancellation, so it
+ is bounded only by `timeout` and outlives the turn. What the callee sees of
+ the caller's transaction depends on its mailbox: a durable callee enqueues
+ synchronously, before `AskThen` returns, so that enqueue joins the caller's
+ transaction, but the envelope's caller context has the transaction stripped,
+ so no callee ever runs its turn inside it, and a channel-mailbox callee
+ (which runs after the caller's turn has ended) never sees it at all. The
+ reply is delivered with a context stripped of the transaction and
+ cancellation, and is dropped at debug level if `self` has stopped. `wrap`
+ must be pure: it runs on a helper goroutine and must not touch actor state.
- `ErrAwaitInTurn` / `AwaitInTurnPolicy` / `SetAwaitInTurnPolicy(p)` —
Process-wide policy for `Future.Await` on an incomplete future from inside a
receive turn. `AwaitInTurnWarn` (default) logs at info once per call site
(keyed on the caller's PC) and waits as before; `AwaitInTurnAllow` does
nothing; `AwaitInTurnError` returns `ErrAwaitInTurn` without waiting. A
completed future always returns its value, and a context whose turn has
- ended is unaffected. `ThenApply`, `OnComplete` and `MapRef.Ask` await on
- helper goroutines through the unexported `awaitInternal`, so they never trip
- the guard.
+ ended is unaffected. `ThenApply` and `OnComplete` await on helper goroutines
+ through the unexported `awaitInternal`, and `MapRef.Ask` goes through the
+ `awaitFuture` helper, which uses `awaitInternal` for this package's own
+ futures and falls back to the public `Await` for any foreign `Future`
+ implementation, so framework helpers never trip the guard.
- `ChannelMailbox[M, R]` — In-memory channel-based mailbox (non-durable, for lightweight actors). A bounded channel fronted by an overflow queue: `Send` with an active turn context (see `turn.go`) never blocks and spills into the overflow, which the receive loop refills into the channel after every receive (so a non-empty overflow implies a non-empty channel). Lock order is `mu` (read side) then `overflowMu`. `NewChannelMailbox` takes `WithOverflowLimit` and `WithMailboxID` options; `OverflowLen()` reports the backlog.
- `Mailbox[M, R]` — Interface for actor message queues: `Send(ctx, env) error` (blocking, except for a `ChannelMailbox` send made with an active turn context; returns `ErrMailboxClosed`, `ErrActorTerminated`, or a context error on failure), `TrySend(env) error` (non-blocking), `Receive(ctx) iter.Seq[envelope]`, `Close()`, `IsClosed() bool`, `Drain() iter.Seq[envelope]`.
- `isExpectedShutdownErr(err) bool` — Internal helper that classifies errors as expected during teardown: context cancellation/deadline, closed DB handle ("sql: database is closed", "sql: connection is already closed", "use of closed network connection"). Used by the lease loop to demote shutdown-path failures to debug instead of warn-flooding test artifacts at itest tail.
@@ -148,7 +155,7 @@ crash-safe at-least-once delivery with exactly-once deduplication.
because begin/commit failures happen outside those operation-level logs.
- `ServiceKey` lookup via `Receptionist` is type-safe: mismatched types return `ErrServiceKeyTypeMismatch`.
- `RestartMessage` has `RestartPriority` (MaxInt32) ensuring it is processed before all other messages on recovery.
-- Transaction context (`WithTx`/`RequireTx`) enables same-DB-transaction joining between actors and their callers.
+- Transaction context (`WithTx`/`RequireTx`) enables same-DB-transaction joining between actors and their callers. `AskThen` is the deliberate exception on the receiving side: the durable enqueue still joins the caller's transaction, but the transaction is stripped from the envelope's caller context, because that context is the base of the callee's turn context and the caller's turn has committed or rolled back by the time the callee runs. The detached marker is cleared along with it, so the callee's own later `Ask` calls — which do wait for their replies — keep propagating their transaction normally.
- `Mailbox.Send` returns the exact failure error (`ErrMailboxClosed`, `ErrActorTerminated`, `context.Canceled`, `context.DeadlineExceeded`) rather than a boolean; `Tell` and `Ask` propagate this directly to callers.
- **A `Tell` or `Ask` made with the turn's context never parks on a full
channel mailbox.** The runtime marks the context handed to a behavior as an
@@ -181,7 +188,16 @@ crash-safe at-least-once delivery with exactly-once deduplication.
process can switch to `AwaitInTurnError`. Awaiting from outside a turn, from
a goroutine that outlives the turn, or on an already-complete future is
always fine. Awaits that live in this package's own helper goroutines go
- through `awaitInternal` and must keep doing so.
+ through `awaitInternal` (or `awaitFuture`) and must keep doing so.
+- **An `AskThen` reply is delivered exactly once, even when the request never
+ happened, so the handler must tolerate a stale one.** A timeout abandons the
+ wait, not the request: the callee may still process the message, and its late
+ reply is then discarded. A durable caller's turn may also roll back after
+ `AskThen` returned, which rolls the durable Ask row back with it while the
+ helper goroutine still delivers a wrapped `context.DeadlineExceeded`. A
+ behavior must therefore record that a request is pending in the turn that
+ issues it and treat a wrapped result it has no pending entry for as stale and
+ ignore it, rather than assuming every delivered reply answers a live request.
- During daemon teardown, the underlying DB is closed before every actor's lease loop has wound down. The lease loop uses `isExpectedShutdownErr` to demote these "database is closed" errors to debug level; real operational errors still surface as warnings because neither the actor context nor the outer context is done in those cases.
- **Postpone preserves the attempt budget; nack spends it.** A nack increments
`attempts` on every release, and both the claim and the peek queries filterHow to apply: save the diff above to a file and Note on This check is advisory — it does not block the merge. Workflow run — |
beebe1a to
08c91f3
Compare
|
Applied the doc advisory to |
08c91f3 to
10f070d
Compare
|
/gateway review |
There was a problem hiding this comment.
Gateway review — 3 findings
🔴 0 Blocker · 🟠 0 Major · 🟡 3 Minor · 🔵 0 Nit
Summary
This PR adds AskThen, a pipe-to-self helper. It also adds a process-wide policy for Future.Await calls made inside a receive turn. The default policy only logs, so nothing changes at runtime yet. The core pieces look right:
- The delivery context drops both cancellation and the transaction, so a delivery after a timeout doesn't fail at once.
- The detached marker is reset in the callee's envelope, so it doesn't leak into the callee's own Asks.
ThenApply,OnCompleteandMapRef.Asknow wait throughawaitInternal, so the framework's own helper goroutines don't trip the guard.- The tests cover the A/B deadlock case, the transaction split and the policy modes.
The remaining concerns are about what happens off the happy path. The guard decides "in a turn" from a context value, not from which goroutine is waiting. That causes false reports now and will cause racy failures once the policy is switched to AwaitInTurnError. Separately, AskThen promises to deliver the result exactly once, but a failed delivery is only logged at debug level. A behavior that kept a pending entry for that request would then wait forever. Neither issue changes behavior under the default Warn policy, and both should be fixed before relying on these primitives in production.
Lower-severity findings (3)
🟡 F1 (Minor) — Await guard flags goroutines that only carry a live turn context · baselib/actor/future.go:87
checkAwaitInTurn decides "in a turn" from activeTurn(ctx). A goroutine that holds a context from a turn that is still running gets flagged, even though it is not the actor's goroutine. Under AwaitInTurnError it gets ErrAwaitInTurn instead of a wait, depending on timing.
Concrete cases:
- Goroutine started by a behavior. If a behavior starts a goroutine with its turn
ctxand that goroutine callsAwaitbefore the turn returns, it fails. The same call succeeds if it happens a moment later. The docs (AGENTS.md:224) say such goroutines are "always fine", which isn't true while the turn is still running. - AskThen ctx reaching a callee.
AskThen'saskCtxstill carries the caller's turn marker. InactorRefImpl.Askthat ctx becomes the envelope'scallerCtx, with the transaction and detached marker removed but the turn marker kept. The caller's turn keeps running afterAskThenreturns. Meanwhile, a channel-mailbox callee that usesDetachAskPromisehands thatcallerCtx(asCallerCtx) to its follow-up code. If that code awaits, it is blamed on the caller's actor.
Under Warn this adds wrong entries to the list of call sites to migrate. Under Error it causes intermittent failures in code that isn't on the turn goroutine. Fix options:
- Remove the turn marker in
AskThen(foraskCtxanddeliverCtx) and in the envelopecallerCtx. This matches how the transaction is handled. - Also record a goroutine identity in the turn marker, or document that a turn context must not be awaited from a goroutine started during the turn. Do one of these before switching the default to Error.
🟡 F2 (Minor) — Failed AskThen delivery is dropped at debug, leaving pending state stuck · baselib/actor/ask_then.go:126
AskThen documents that the result is delivered exactly once, and tells behaviors to record a pending entry that the delivered reply clears. If self.Tell fails, the result is logged at debug and dropped, so a pending entry recorded by a still-running behavior is never cleared.
The comment at lines 124–125 assumes the only way delivery can fail is that self has stopped. That holds for a channel mailbox. For a durable self, Tell writes to the database, so a transient store error makes it fail even though the actor is still running. The behavior is then stuck waiting, and nothing above debug level shows it.
I can't see the durable Tell implementation, so I can't confirm which errors it returns. Suggested fix: log at debug only for ErrActorTerminated / ErrMailboxClosed. Log anything else at warn (an operator needs to act) and consider retrying.
🟡 F3 (Minor) — AskThen can block for up to timeout when called outside a turn · baselib/actor/ask_then.go:140
AskThen says it "returns immediately", but ref.Ask(askCtx, msg) runs mailbox.Send synchronously. A send only skips blocking when the context marks an active turn. If AskThen is called with a context from outside a turn, the send blocks on a full channel mailbox for up to timeout. That includes context.Background() (as in TestAskThenDropsResultForStoppedSelf), a context from a goroutine, or a context from a turn that has ended.
Either document that AskThen must be called with the turn context, or start the Ask on the helper goroutine when activeTurn(ctx) is false.
Bot commands
/gateway re-review— re-run after pushing changes (maintainers)/gateway dismiss <id>— silence a finding (maintainers)/gateway explain <id>— elaborate on a finding (anyone)
|
🤖 gateway audit metadata for this PR — auto-generated, please don't edit. |
10f070d to
d06b185
Compare
d06b185 to
ca8257a
Compare
In this commit, we split the wait loop of futureImpl.Await into an unexported awaitInternal, and route the helper goroutines spawned by ThenApply, OnComplete and MapRef.Ask through it. Those goroutines frequently carry a context that still marks an active receive turn, for example when a behavior calls MapRef.Ask with its turn context. They are not the actor's own goroutine, so waiting there cannot stall the actor. A follow-up change makes Await on a turn context observable, and these awaits must not be reported when it does. Behavior is unchanged by this commit.
In this commit, we add a process-wide policy for what Future.Await does when an actor's receive turn waits on a future that has not completed. Such a wait parks the actor's whole mailbox, and when the callee is itself waiting on the caller, both hang. The previous commits stopped in-turn sends from parking, which leaves this as the remaining way for a turn to wait on another actor. SetAwaitInTurnPolicy selects AwaitInTurnWarn (the default), which logs at info level once per call site and then waits as before, AwaitInTurnAllow, or AwaitInTurnError, which returns ErrAwaitInTurn without waiting. We log at info rather than warn because a warn pages a human and this is an enumeration of sites to migrate, not an incident. The call site is keyed on the caller's program counter, so the report names the behavior code that called Await. Only the slow path pays for the check: a completed future returns before the turn is consulted, and under the error policy it still returns its value since reading it cannot block.
In this commit, we add AskThen, the pipe-to-self alternative to awaiting an Ask reply inside Receive. It sends the request bounded by a required timeout, returns at once, and delivers wrap(result) to the caller's own mailbox, so the reply is handled in a later turn and the caller keeps serving other actors in the meantime. That removes the cycle in which two actors each wait on the other's reply. The Ask keeps the caller's context values, as a plain Ask from the same turn does, so a durable callee still enqueues the request in the caller's transaction, but it is detached from the turn's cancellation and bounded only by the timeout. The reply is delivered with a separate context that drops the transaction and the cancellation: on a timeout the Ask context has already expired and a Tell on it would fail at once. If the actor has stopped, the result is dropped at debug level rather than parking a helper goroutine. A timeout that is not positive is reported as context.DeadlineExceeded without sending anything.
In this commit, we document AskThen, ErrAwaitInTurn and SetAwaitInTurnPolicy in the package guide, and add the invariant that a turn should not wait on another actor's reply, with AskThen and DetachAskPromise as the alternatives and the warn-then-enforce rollout. The README now says Await is for code outside an actor.
ca8257a to
7b04dfe
Compare
In this PR, we go after the second way one actor can wait on another: an
Awaiton another actor's reply from insideReceive. If the callee (transitively) needs something from the caller before it can answer, both turns hang until somebody's deadline fires, or forever if nobody set one. This builds on #1371 (sends from a turn never park), and is the second step of the plan in this comment.We borrow the OTP answer. A
gen_serverthat needs someone else's reply doesn't block inhandle_call; it returns{noreply, State}and replies later. Akka Typed calls the same thingpipeToSelf. We already have the forwarding special case inDetachAskPromise. What was missing is the general form.AskThen
AskThensends the Ask bounded by a requiredtimeout, returns right away, and once the reply (or the timeout) lands, deliverswrap(result)back toselfas an ordinary message. The actor handles it in a later turn, so all state access stays on the actor's goroutine.wrapis meant to be pure: it only turns the result into a message. That's the difference from anOnCompleteclosure that reads or writes actor state from another goroutine. A two-step exchange becomes two turns: the first firesAskThenand records the request as pending, the second handles the wrapped reply. A longer pipeline is a chain of these, and if our own caller is waiting on the end result,DetachAskPromiseholds its promise in the pending state until the last step completes it.A couple of details took some care. The delivery back to
selfuses a detached context with the transaction stripped, never the Ask context, which has already expired on the timeout path. Transactions needed a split. A durable callee enqueues synchronously, so its enqueue still joins the caller's transaction; on a single-writer store a separate connection couldn't write while the turn holds the writer anyway. A channel-mailbox callee, though, would otherwise inherit the caller's transaction through the turn context it builds from the Ask. WithAskThen, the caller's turn has already committed or rolled back by then, soAskThenmarks its Ask context and the channel Ask path drops the transaction (and the marker) from the envelope's caller context. Delivery toselfis at most once and best effort: it is dropped (at debug level) ifselfhas stopped, and logged at warning level if a liveself, durable in particular, fails to take it, so a behavior that records a pending request has to give that entry its own expiry. Finally, a durable caller's turn can roll back afterAskThenreturns, in which case the reply shows up with no pending entry; the doc spells out that behaviors should treat that as stale.Await inside a turn
Future.Awaitnow checks, on the slow path only, whether it was called with an active turn context. The policy is package-wide viaSetAwaitInTurnPolicy:AwaitInTurnWarn(default) logs once per call site at Info with the actor ID andfile:line, then waits as before.AwaitInTurnErrorreturnsErrAwaitInTurnwithout waiting. An already-complete future still returns its value.AwaitInTurnAllowturns the check off.The default changes no behavior. It just gives us the list of sites to migrate. Framework-internal waits on helper goroutines (
ThenApply,OnComplete,MapRef.Ask) go through an unguarded internal wait, so they aren't misreported as in-turn awaits.Two
protofsmsites will show up in warn mode and need handling before we can flip the default to Error.StateMachine.Receiveawaits its own FSM driver's future (a self-owned wait, not another actor, so it needs an exemption), andRoutedOutboxEvent.Dispatchdoes a real in-turnrouter.Ask(...).Await, which is a migration target.Testing
The
AskThentests cover OK, callee error, timeout, non-positive timeout, a stoppedself(the helper must return, not park), the transaction split for channel vs. plain Ask, and the A/B shape that deadlocks withAwaiton both sides. The policy tests cover warn-once-per-site (including the reported call site through theFutureinterface), Allow, Error on incomplete vs. complete futures, non-turn and ended-turn contexts, and the internal waits under the Error policy. Policy-mutating tests run serially. Everything passes under-race, as do theprotofsm,timeout,round,vtxo,oorandserverconnsuites.