Repository navigation
actor: fail an Await that would close a wait cycle - #1373
Conversation
801bdda to
40d0a65
Compare
|
/gateway review |
There was a problem hiding this comment.
Gateway review — 2 findings
🔴 0 Blocker · 🟠 2 Major · 🟡 0 Minor · 🔵 0 Nit
Summary
This PR adds a runtime check for wait cycles between actors. Ask futures now record which actor will reply. When a serial turn calls Future.Await and the wait is not already satisfied, it registers an edge from its actor to that target. Registration and the cycle walk happen under one lock, and an Await that would close a cycle fails with ErrWaitCycle. Tying each edge to the turn that registered it is a good choice for the case where a spawned goroutine outlives its turn. The tests cover that case well.
The weak spot is the claim that "an edge exists only while a turn is really blocked". Two things break it:
- A completed reply doesn't clear the edge in time. The edge stays until the woken waiter runs its deferred
release(). Meanwhile the replying actor can start its next turn and Await the original caller. That reports a cycle that doesn't exist and fails a validAwait. - A spawned goroutine can register first. The edge then stands for the turn even though it records the goroutine's target, not the turn's.
Both produce false ErrWaitCycle errors logged at Error level, in code that worked before this change. The ring tests don't cover either sequence.
Bot commands
/gateway re-review— re-run after pushing changes (maintainers)/gateway dismiss <id>— silence a finding (maintainers)/gateway explain <id>— elaborate on a finding (anyone)
|
🤖 gateway audit metadata for this PR — auto-generated, please don't edit. |
2cb7042 to
ae56534
Compare
In this commit, we record, on each future returned by an Ask, the ID of the actor whose reply it represents. The actor and durable actor refs set it directly, and MapRef carries over the target of the future it wraps because it creates its own promise. Router delegates to a concrete ref, so it inherits the target without any change. Futures that are not the reply to an Ask, such as those a state machine completes from its own driver goroutine, keep an empty target. This is groundwork for tracking which actor a blocked turn is waiting on, so that a cycle of such waits can be reported instead of hanging.
In this commit, we add a wait-for graph and make Future.Await fail with ErrWaitCycle instead of parking when waiting would close a cycle of actor turns that each block on another actor's reply, including an actor awaiting its own reply. Once sends from a turn never park, the remaining way for actors to deadlock is a turn blocked in Await on a reply that can only be produced by an actor that is itself blocked. The await-in-turn policy only warns by default, so such a cycle still hung silently until an Ask deadline, if there was one. An edge now exists only while a serial actor's turn is blocked in Await on a future that an Ask created, so a cycle in the graph is a real cycle at that instant. Registering the edge and walking the graph share one lock hold, so of two actors that close a cycle together exactly one sees it, and its failing turn lets the other's Ask be processed. Turns of multi-worker durable actors register nothing, since a parked worker does not park the actor, and futures that are not an Ask reply have no target and never create an edge. Waits outside the framework are invisible to the detector.
In this commit, we add SetWaitCycleHook, which installs a function called each time an Await is refused for closing a wait cycle, with the cycle path and the call site of the Await. A cycle is a code defect, and the actor that receives ErrWaitCycle can swallow it or turn it into an ordinary failure that a test tolerates. A deterministic simulation harness can instead install a hook that fails the run on any call, so the defect is caught wherever it surfaces. The hook is stored atomically and a nil hook clears it.
In this commit, we document ErrWaitCycle, WaitCycle and SetWaitCycleHook in the package guide, along with an invariant that states what the detector covers and what it cannot see. Readers need the scope in one place: edges exist only while a serial actor's turn is blocked in Await on an Ask reply, sends add none, and waits outside the framework are invisible.
ae56534 to
bda5d4e
Compare
In this PR, we add a runtime backstop for the last way actors can wedge each other inside the framework: a cycle of turns, each parked in
Future.Awaiton the next actor's reply (including an actor awaiting a reply from itself). Rather than letting that sit silently until a deadline fires (or forever),Awaitnow detects the cycle at the moment it would close and fails withErrWaitCycle, naming the actors involved. This is the third step of the plan in this comment, stacked on #1372 (AskThenand the await-in-turn policy) and #1371 (sends from a turn never park).With #1371, a turn's sends no longer park, so
Awaitis the only remaining actor-to-actor wait the framework creates. While the await-in-turn policy stays at its Warn default, these cycles can still happen. Once call sites move toAskThen, the detector should never fire, and at that point it serves as an assertion for whatever we missed.How it works
Ask futures now carry the ID of the actor whose turn produces the reply. It's set in
actorRefImpl.AskanddurableActorRefImpl.Ask;Routerreturns the chosen actor's future unchanged, andMapRefandThenApplyinherit the inner future's target. Futures not created by an Ask (subscription futures,protofsm'sAskEvent) have no target and never create an edge.On the slow path of
Await(future not yet complete, after the await-in-turn policy check), a serial turn registerswaiting[self] = targetand walks the chain under one mutex. If the walk comes back toself, it refuses the edge, logs at Error with the path and call site, calls the optional hook, and returnsErrWaitCycleinstead of parking. Otherwise it waits as before and drops the edge on every exit. Because registration and the walk happen under the same lock, when two actors close a cycle concurrently, exactly one of them fails. That one's turn returns, which lets the other's Ask get served. The graph is acyclic by construction, so the walk bound is only a safety net.An edge also stops counting once the reply it waits on has been delivered, since the waiter can be slow to wake and remove it, and in that window the replying actor can legitimately await the original caller. Derived futures (
ThenApply,MapRef) check the root actor future's completion, not their own. An actor can have several live edges, one per wait in flight, each removed only by the wait that registered it, and a cycle through any of them counts. Go has no goroutine identity, so anAwaitfrom a goroutine a behavior spawned with the live turn context looks like the actor's own wait and can be reported as a cycle; the await-in-turn policy already flags those goroutines.Edges remember the turn that registered them, and an edge whose turn has ended counts as absent. That matters for goroutines a behavior spawns with its turn context: once the turn is over they aren't the actor anymore, so a late edge from one of them can't produce a false cycle on an innocent actor, and can't shadow the actor's real next wait. Turns of a durable actor with more than one worker don't register at all, since one parked worker doesn't park the actor.
SetWaitCycleHook(func(WaitCycle))exposes detections to a deterministic simulation harness: an oracle can fail a seed on any call.What it doesn't see
The detector only knows about waits that go through
Future.Awaiton an Ask future. These are listed in the package docs:DetachAskPromise) keeps the coordinator as its target.protofsm's untargetedAskEventfuture) is invisible.Testing
The tests cover:
Router,MapRefandThenApply.ThenApplyandMapRefwhere the derived future is still pending. They fail if liveness ignores the delivered reply.Everything passes under
-race, as do theprotofsm,timeout,round,vtxo,oorandserverconnsuites, and no existing test tripped the detector.