Skip to content

actor: fail an Await that would close a wait cycle - #1373

Merged
Roasbeef merged 4 commits into
actor-askthenfrom
actor-wait-cycle
Oct 5, 2026
Merged

Roasbeef merged 4 commits into
actor-askthenfrom
actor-wait-cycle

Conversation

@Roasbeef

@Roasbeef Roasbeef commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

In this PR, we add a runtime backstop for the last way actors can wedge each other inside the framework: a cycle of turns, each parked in Future.Await on the next actor's reply (including an actor awaiting a reply from itself). Rather than letting that sit silently until a deadline fires (or forever), Await now detects the cycle at the moment it would close and fails with ErrWaitCycle, naming the actors involved. This is the third step of the plan in this comment, stacked on #1372 (AskThen and the await-in-turn policy) and #1371 (sends from a turn never park).

With #1371, a turn's sends no longer park, so Await is the only remaining actor-to-actor wait the framework creates. While the await-in-turn policy stays at its Warn default, these cycles can still happen. Once call sites move to AskThen, the detector should never fire, and at that point it serves as an assertion for whatever we missed.

How it works

Ask futures now carry the ID of the actor whose turn produces the reply. It's set in actorRefImpl.Ask and durableActorRefImpl.Ask; Router returns the chosen actor's future unchanged, and MapRef and ThenApply inherit the inner future's target. Futures not created by an Ask (subscription futures, protofsm's AskEvent) have no target and never create an edge.

On the slow path of Await (future not yet complete, after the await-in-turn policy check), a serial turn registers waiting[self] = target and walks the chain under one mutex. If the walk comes back to self, it refuses the edge, logs at Error with the path and call site, calls the optional hook, and returns ErrWaitCycle instead of parking. Otherwise it waits as before and drops the edge on every exit. Because registration and the walk happen under the same lock, when two actors close a cycle concurrently, exactly one of them fails. That one's turn returns, which lets the other's Ask get served. The graph is acyclic by construction, so the walk bound is only a safety net.

An edge also stops counting once the reply it waits on has been delivered, since the waiter can be slow to wake and remove it, and in that window the replying actor can legitimately await the original caller. Derived futures (ThenApply, MapRef) check the root actor future's completion, not their own. An actor can have several live edges, one per wait in flight, each removed only by the wait that registered it, and a cycle through any of them counts. Go has no goroutine identity, so an Await from a goroutine a behavior spawned with the live turn context looks like the actor's own wait and can be reported as a cycle; the await-in-turn policy already flags those goroutines.

Edges remember the turn that registered them, and an edge whose turn has ended counts as absent. That matters for goroutines a behavior spawns with its turn context: once the turn is over they aren't the actor anymore, so a late edge from one of them can't produce a false cycle on an innocent actor, and can't shadow the actor's real next wait. Turns of a durable actor with more than one worker don't register at all, since one parked worker doesn't park the actor.

SetWaitCycleHook(func(WaitCycle)) exposes detections to a deterministic simulation harness: an oracle can fail a seed on any call.

What it doesn't see

The detector only knows about waits that go through Future.Await on an Ask future. These are listed in the package docs:

  • I/O, mutexes and raw channels are invisible to it.
  • A detached Ask promise (DetachAskPromise) keeps the coordinator as its target.
  • A wait that passes through a state machine's driver goroutine (protofsm's untargeted AskEvent future) is invisible.
  • Pooled durable workers don't register.

Testing

The tests cover:

  • self-ask, and two- and three-actor cycles. Each asserts the error well inside the Await deadline, so a timeout can't pass for a detection, and asserts that the queued work still makes progress afterwards. The path assertions don't depend on which actor registers first.
  • no false positives for chains, single waits and concurrent unrelated waits.
  • non-serial turns and untargeted futures.
  • edge cleanup on cancellation.
  • the hook.
  • target identity through Router, MapRef and ThenApply.
  • three deterministic tests for the stale-edge cases. All three fail when inactive edges are counted as live.
  • reply-then-reverse-Ask with an undeleted edge, including through ThenApply and MapRef where the derived future is still pending. They fail if liveness ignores the delivered reply.
  • several live edges per actor: both are recorded, a helper finishing leaves the actor's own edge, and a cycle through the second edge is detected.

Everything passes under -race, as do the protofsm, timeout, round, vtxo, oor and serverconn suites, and no existing test tripped the detector.

@Roasbeef
Roasbeef force-pushed the actor-wait-cycle branch 2 times, most recently from 801bdda to 40d0a65 Compare October 2, 2026 23:07
@Roasbeef
Roasbeef added this pull request to stack #1378 October 2, 2026 23:24
@Roasbeef

Roasbeef commented Oct 2, 2026

Copy link
Copy Markdown
Member Author

/gateway review

@lightninglabs-gateway lightninglabs-gateway Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gateway review — 2 findings

🔴 0 Blocker · 🟠 2 Major · 🟡 0 Minor · 🔵 0 Nit

Summary

This PR adds a runtime check for wait cycles between actors. Ask futures now record which actor will reply. When a serial turn calls Future.Await and the wait is not already satisfied, it registers an edge from its actor to that target. Registration and the cycle walk happen under one lock, and an Await that would close a cycle fails with ErrWaitCycle. Tying each edge to the turn that registered it is a good choice for the case where a spawned goroutine outlives its turn. The tests cover that case well.

The weak spot is the claim that "an edge exists only while a turn is really blocked". Two things break it:

  1. A completed reply doesn't clear the edge in time. The edge stays until the woken waiter runs its deferred release(). Meanwhile the replying actor can start its next turn and Await the original caller. That reports a cycle that doesn't exist and fails a valid Await.
  2. A spawned goroutine can register first. The edge then stands for the turn even though it records the goroutine's target, not the turn's.

Both produce false ErrWaitCycle errors logged at Error level, in code that worked before this change. The ring tests don't cover either sequence.

Bot commands
  • /gateway re-review — re-run after pushing changes (maintainers)
  • /gateway dismiss <id> — silence a finding (maintainers)
  • /gateway explain <id> — elaborate on a finding (anyone)

Comment thread baselib/actor/wait_graph.go Resolved
Comment thread baselib/actor/wait_graph.go Resolved
@lightninglabs-gateway

lightninglabs-gateway Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

🤖 gateway audit metadata for this PR — auto-generated, please don't edit.

In this commit, we record, on each future returned by an Ask, the ID of
the actor whose reply it represents. The actor and durable actor refs
set it directly, and MapRef carries over the target of the future it
wraps because it creates its own promise. Router delegates to a
concrete ref, so it inherits the target without any change.

Futures that are not the reply to an Ask, such as those a state machine
completes from its own driver goroutine, keep an empty target. This is
groundwork for tracking which actor a blocked turn is waiting on, so
that a cycle of such waits can be reported instead of hanging.
In this commit, we add a wait-for graph and make Future.Await fail with
ErrWaitCycle instead of parking when waiting would close a cycle of
actor turns that each block on another actor's reply, including an
actor awaiting its own reply.

Once sends from a turn never park, the remaining way for actors to
deadlock is a turn blocked in Await on a reply that can only be
produced by an actor that is itself blocked. The await-in-turn policy
only warns by default, so such a cycle still hung silently until an
Ask deadline, if there was one. An edge now exists only while a serial
actor's turn is blocked in Await on a future that an Ask created, so a
cycle in the graph is a real cycle at that instant. Registering the
edge and walking the graph share one lock hold, so of two actors that
close a cycle together exactly one sees it, and its failing turn lets
the other's Ask be processed.

Turns of multi-worker durable actors register nothing, since a parked
worker does not park the actor, and futures that are not an Ask reply
have no target and never create an edge. Waits outside the framework
are invisible to the detector.
In this commit, we add SetWaitCycleHook, which installs a function
called each time an Await is refused for closing a wait cycle, with the
cycle path and the call site of the Await.

A cycle is a code defect, and the actor that receives ErrWaitCycle can
swallow it or turn it into an ordinary failure that a test tolerates.
A deterministic simulation harness can instead install a hook that
fails the run on any call, so the defect is caught wherever it surfaces.
The hook is stored atomically and a nil hook clears it.
In this commit, we document ErrWaitCycle, WaitCycle and
SetWaitCycleHook in the package guide, along with an invariant that
states what the detector covers and what it cannot see.

Readers need the scope in one place: edges exist only while a serial
actor's turn is blocked in Await on an Ask reply, sends add none, and
waits outside the framework are invisible.
@Roasbeef
Roasbeef merged commit 7732899 into main Oct 5, 2026
21 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants