Repository navigation
multi: add non-blocking TryTell and bound timeout actor delivery - #1082
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 45d73d54de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // does not stretch a fast ticker by an order of magnitude: a 100ms ticker | ||
| // retries in 100ms, while anything slower than the standard base keeps it. | ||
| func tickRetryDelay(interval time.Duration, attempt int) time.Duration { | ||
| return backoffDelay(min(interval, retryBaseDelay), attempt) |
There was a problem hiding this comment.
Cap recurring retry delays at the tick interval
When a sub-second recurring callback remains unavailable for multiple attempts, this uses the interval only as the exponential backoff's base, so a 100 ms tick retries after 100 ms, 200 ms, 400 ms, and eventually 30 seconds. Once the consumer recovers, its callback can therefore remain delayed far beyond its configured cadence, contrary to the timeout actor contract that recurring backoff is capped at the tick interval; clamp the final delay to interval, not merely the base.
AGENTS.md reference: timeout/AGENTS.md:L49-L55
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch on the mismatch, but the fix belongs on the doc rather than the code.
The behaviour you describe is intended: only the starting point of the backoff moves for a fast ticker, and the 30s ceiling applies to recurring entries exactly as it does to one-shots. TestTickRetryDelayStartsFromInterval already pinned tickRetryDelay(100ms, 1000) == retryMaxDelay, so the code and its test agreed with each other all along.
Clamping the final delay to interval is the one thing I don't want here. Every retry attempt against a durable callback is a bounded database write, so a 100ms ticker retrying at 100ms forever would hammer the store it is already struggling with, and the backoff exists precisely to stop that. A consumer that stays unreachable long enough to reach the ceiling has bigger problems than its cadence.
What misled you was real though, and it was mine: the AGENTS.md sentence read "a backoff doubling from 1s to 30s, capped at the tick interval for recurring entries", where "capped at" was meant to attach to the 1s base and just as easily reads as attaching to the 30s ceiling. The test name TestTickRetryDelayCapsAtInterval said the same wrong thing.
Both are fixed as fixups: the invariant now spells out that only the starting point moves, says the ceiling still applies, and gives the durable-write reason; the test is renamed and gained an explicit assertion that the delay exceeds the interval at the cap.
In this commit, we add TryTell to the TellOnlyRef and ActorRef interfaces. The mailbox layer has always had a non-blocking TrySend, but the ref layer hid it behind Tell, so a caller that must not park its own receive goroutine had to wrap Tell in a context timeout. That burns the whole deadline against a peer that is already wedged, which is exactly the shape of deadlock we want to make impossible: an actor blocked on a full peer mailbox while that peer waits on a reply from it. TryTell attempts an immediate enqueue and hands the message straight back with ErrMailboxFull when the target has no room, so the caller can drop, stash, or reschedule it instead of waiting. The context is consulted only for an immediate cancellation check, and every other failure keeps the errors Tell already returns, including the dead letter routing for a terminated target. We put the method on the interface rather than behind an optional capability check so the compiler names every ref that has to think about it. That costs a mechanical method on each test double and on downstream implementers, but the alternative quietly falls back to a blocking send in precisely the places we are trying to fix.
In this commit, we stop the timeout actor from parking its receive goroutine on a requester's mailbox. The actor is a process-wide singleton shared by every subsystem, so a single backlogged requester could freeze every timer in the daemon: the fire handler Told the callback with no deadline, and a full mailbox held the loop until that one requester drained. It is the same shape of block that wedged two actors for hours in production. Delivery now goes through TryTell. A requester with no room keeps its entry, which is the only copy of the reminder, and the actor re-arms the same fire on a backoff that doubles from one second up to thirty. Memory stays at one entry per outstanding reminder however many attempts it takes, and since the entry stays in the map a Cancel or a reschedule still owns it, so a cancelled timeout can no longer fire late. A callback that is gone for good earns no retry chain at all, because no backoff repairs a terminated actor. Recurring ticks hold the undelivered tick on the entry so a retry still reports the instant its timer fired. Retried delivery is by definition later than it used to be, and can land out of order against other callbacks, so it is worth saying why every caller in this repo tolerates that. The round actor ignores timeouts for rounds it no longer tracks and re-checks phase state before acting on one, while the OOR and credit timers are accelerants whose work is re-derived at boot if their reminder never shows up.
In this commit, we make the clock goroutine's self-tell non-blocking. Every timer fire hands an internal message to the actor's own mailbox from the goroutine the clock ran the callback on, and it used to do that with an unbounded Tell on a background context. When the actor was behind on its queue, each fire left a goroutine parked there until the queue drained, so the actor leaked one goroutine per fire in exactly the situation it was already struggling with. The fire now goes in with TryTell and, when the mailbox has no room, re-arms itself on a short delay instead of waiting. Each entry has at most one fire outstanding, so deferred fires coalesce into one pending timer rather than a growing pile of parked senders, and the fire is deferred rather than dropped. A stopped actor ends the chain, since no retry repairs a closed mailbox. The retry handle is deliberately not recorded on the entry, because we are off the receive goroutine there and touching the entry maps would race. A Cancel that lands during the deferral is still honoured: the generation check drops the fire when it eventually arrives.
45d73d5 to
3b77c01
Compare
|
@claude review this |
|
Claude finished @Roasbeef's task in 3m 27s —— View job Review:
|
d983712 to
810eb1d
Compare
|
Thanks — this reviewed the head from just before I pushed, so part of finding #1 was already in flight. Disposition below. Finding #1:
|
In this commit, we write down what the previous two changes mean for anyone reaching for these packages. The actor guides now say when to use TryTell over Tell, namely any send made from inside a receive goroutine, and why wrapping Tell in a timeout is the weaker answer. The timeout guide gains the two invariants a caller has to know: the actor never blocks on a callback, and a callback that is behind gets its reminder re-armed on a backoff rather than delivered on time. That makes late and reordered delivery part of the contract instead of a surprise.
In this commit, we fix the retry predicate, which had the test backwards. It retried only a full mailbox and treated everything else as permanent, and that is wrong for the two callback shapes that matter most in the daemon. A durable callback (the OOR session retry timers, the credit poll timers) enqueues through a database write bounded by the durable mailbox's own deadline, so a slow database surfaces as a wrapped DeadlineExceeded and never as a full mailbox. A callback behind a Router reports that no actor is registered while its target is being replaced. Both of those were silently discarding the reminder for good, which is worse than the blocking send we replaced: that one at least delivered once the database recovered. A credit op chains each poll off the previous expiry, so one dropped expiry stops that op polling until the daemon restarts. Only a terminated actor and a closed mailbox are end states now, and everything else earns another attempt on the same backoff. Guessing wrongly in this direction costs a retry, while guessing wrongly in the other direction loses a timer that nothing re-derives. While here we cap the recurring retry base at the tick interval, so one transient failure no longer stretches a 100ms ticker to a full second, and we swap the shift-based backoff for a doubling loop that cannot overflow whatever base it starts from.
In this commit, we let the daemon inject a logger into the timeout actor, because the diagnostics we just added were invisible where they matter. The actor resolved its logger from the receive context alone, but an actor's lifecycle context descends from context.Background() rather than from the actor system's logger-carrying context, so build.LoggerFromContext always handed back btclog.Disabled. Every line about a callback we could not reach, or a reminder we gave up on, went nowhere in a running daemon. That is the same trap as a subsystem whose logger was never wired: the code looks observable and is not. The actor now takes a Config with an optional logger and falls back to the context logger only when nobody supplied one, matching how the credit behaviors do it. All three daemon call sites pass the TMOT sublogger, which also turns that previously dead Subsystem constant into something registered with the log manager and level-tunable like every other subsystem. A test asserts a retry actually writes through the injected logger while the receive context carries none, so this cannot quietly rot back to a disabled logger.
In this commit, we stop two test doubles from lying about the contract the ref interface just grew. Both were given a TryTell that delegates to a Tell which parks the caller until the test releases it, so a caller that asked for a non-blocking send would have been blocked anyway. The doubles that model a wedged subscriber are exactly the ones a future backpressure test would reach for, so leaving them wrong is how the contract gets quietly broken. The blocking subscriber now reports a full mailbox, which is the honest translation of "cannot take this without making you wait". The deferring one panics instead, because a Tell that completes long after the caller's deadline has no non-blocking equivalent at all, and a test that routes into it needs a different double rather than a plausible-looking answer.
In this commit, we document the two ways a durable target departs from the contract the interface describes, and cover them with tests. A durable queue has no capacity, so a durable ref never answers ErrMailboxFull. What it answers under load is a wrapped DeadlineExceeded, because the enqueue is a database write the mailbox bounds with its own short deadline. Any caller that keys its retry decision off ErrMailboxFull therefore throws durable messages away exactly when the database is slow, and the timeout actor was doing precisely that until the previous commit. The second departure is quieter and worse. Tell carries the caller's transaction into a durable enqueue, which is what makes the message and the caller's state change land atomically. TryTell cannot: the mailbox runs its bounded write on a background context, so the row commits on its own and can outlive a rolled back transaction. Swapping one for the other inside a commit closure breaks atomicity with nothing to show for it, so the interface now says so, and the durable implementation strips the transaction from the envelope rather than carrying one the write will not honour. Tests now cover the durable implementation, which was the only one with none, plus the router and the two remaining wrapper refs.
In this commit, we write down the property the whole scheduling arrangement rests on, which was implicit and therefore one careless change away from being lost. Every scheduler in the daemon, round, OOR and credit alike, issues a blocking Tell into the timeout actor's mailbox from inside its own receive loop. That is only safe because the timeout actor's Receive never blocks and its mailbox therefore always drains. Put a blocking call anywhere inside that Receive and the incident shape returns, with the scheduler parked on the timeout mailbox and the timeout actor parked on a scheduler. We also record which delivery errors are terminal and why, warn against handing durable callbacks to the shared timeout singleton when a dedicated instance costs nothing, and note that a retried tick reports the instant its timer fired rather than the instant it was delivered. No consumer in this repo uses recurring ticks, but lumos does, and one computing its deadline from FiredAt plus the interval could read a retried tick as already expired. That wants checking before the pin bumps.
In this commit, we give the round FSM's error reporter a logger that can actually reach an operator. The reporter resolved its logger from a stored context, and the context a round FSM runs under is the actor's lifecycle context, which descends from context.Background() rather than from anything carrying a logger. build.LoggerFromContext therefore handed back btclog.Disabled, and every "FSM error" line went nowhere in a running daemon. The code read as observable and was not. Both construction sites already build a prefixed fsmLogger for the FSM itself, so the reporter now takes that same logger and an FSM error lands beside the transitions that led to it. That also drops the stored context, which was only ever there to be searched for a logger. This is the same trap the timeout actor had earlier in this branch, found by going looking for other instances of it.
In this commit, we fix a self-deadlock in the unroll registry. requestPersist enqueues a persistence attempt by Telling the registry's own mailbox, and every one of its callers but one runs on the registry's receive goroutine. That send is unbounded and the mailbox holds 64 messages, so a registry that ever falls behind parks the one goroutine that could drain it, waiting for room that only it could make. Nothing recovers from that short of shutdown, since the send carries a background context and there is no deadline to expire. The two detached-goroutine self-sends, the async persist result and the retry timer, have the milder version of the same problem: they park one goroutine per event for as long as the registry stays behind. All three now go through TryTell and re-arm on a short delay when the mailbox has no room, which is the shape the timeout actor grew earlier in this branch. Deferring beats dropping because the registry is the only thing that re-derives this work, and a terminated registry ends the chain, since no delay repairs a closed mailbox. The late-admission report in watchChildAdmissionResult keeps its blocking Tell. It runs detached, it is bounded by a real caller context rather than a background one, and it already logs a send it could not complete, so the trade there is the other way round.
810eb1d to
485eca3
Compare
|
@claude I dispositioned your review in the comment above, and pushed since. Re-checking the two spots where I went a different way than you suggested would be useful. Short version:
Worth a look at whether you still consider #1 unresolved given the durable-write argument, and whether the doc now reads unambiguously. |
|
Claude finished @Roasbeef's task in 2m 19s —— View job Re-check of the two dispositioned spotsI re-read the pushed head (
1.
|
|
Successfully created backport PR for |
…ranch [v0.1.x-branch] Backport #1082: multi: add non-blocking TryTell and bound timeout actor delivery
In this PR, we close out the wavelength side of the actor deadlock fixed in
lumos#734: two actors
wedged in production for 8.5 hours because a receive goroutine blocked on an
unbounded Tell into a full peer mailbox while that peer awaited an Ask back
with no deadline. lumos bounded its sends with context timeouts as a stopgap,
but both root causes live in this repo: the ref interface hides the
non-blocking send the mailbox already implements, and the timeout actor
(a process-wide singleton in lumosd, carrying every subsystem's timers)
delivers callbacks with unbounded sends from its single receive goroutine.
See each commit message for a detailed description w.r.t the incremental
changes.
TryTell on the refs
The mailbox layer has had a non-blocking TrySend all along; only the
TellOnlyRef/ActorRefsurface hid it. We addTryTell(ctx, msg) error:it never parks the calling goroutine, a full mailbox returns
ErrMailboxFullso the caller can drop, stash, or schedule a retry, and everything else
matches Tell. This is deliberately a hard interface method rather than an
optional capability check: the bug class being fixed is a silent fallback to
a blocking send, and we want the compiler to name every implementer rather
than letting one quietly keep the old behavior. Downstream, that means the
next lumos pin bump breaks its ref mocks until they grow a one-line
delegation, and in exchange lumos can replace the context.WithTimeout
wrappers from #734 with TryTell for a strictly better failure mode: fail in
nanoseconds instead of burning the deadline against a wedged peer.
One implementation deserves a close read: the durable ref's TryTell is a DB
write bounded at 100ms, never returns
ErrMailboxFull(the queue is anunbounded table), and drops the caller's transaction, so swapping Tell for
TryTell inside a commit closure silently breaks the atomic-enqueue
guarantee. The interface doc spells all three departures out, and the
durable path has its own tests pinning them.
The timeout actor never blocks, and never drops
Callback delivery now goes through TryTell from the receive loop. A delivery
that fails for any non-terminal reason (full mailbox, a slow durable write, a
router with no registered targets) requeues the same fire into the actor's
own timer wheel with doubling backoff, 1s base capped at 30s, one entry per
outstanding reminder no matter how many retries it takes. Only a dead target
(actor terminated, mailbox closed) releases the entry. Nothing is ever
silently dropped: the trade is bounded memory, unbounded time, and a Warn
per attempt naming the target. The clock path coalesces timer fires through
a non-blocking self-signal instead of stacking one parked goroutine per fire
against a backed-up mailbox.
This also fixes a pre-existing cancellation race: the old code deleted the
entry before delivering, so a Cancel racing a fire could not actually stop
it. Entries now live until delivered, so Cancel owns the retry chain too.
Two contract notes for consumers. A recurring tick delivered late carries
its original FiredAt, and a slow consumer stretches its own cadence rather
than accumulating ticks; we checked the one production consumer (lumos's
rounds actor) and it discards FiredAt entirely, treating a tick as a
stateless poll, so the pin bump is safe on this axis. And the timeout actor
now carries a config-injected logger registered as the TMOT subsystem;
before this, its delivery-failure logs resolved to a disabled logger and the
entire failure mode was invisible in a running daemon.
Testing
The regression tests are mutation-verified: reverting delivery to a blocking
Tell freezes the receive loop and fails the responsiveness test, and
reverting the self-signal to a blocking Tell leaks one goroutine per timer
fire and fails the leak test. The retry state machine is covered against the
adversarial interleavings (cancel during an armed retry, reschedule
superseding an armed retry, terminal vs transient errors on both the oneshot
and recurring paths), and every TryTell implementation, including the
durable, router, and wrapper refs, has its semantics pinned. Full suites
pass with -race across baselib, timeout, and every downstream package whose
test doubles grew the new method.
Two more instances of the same shapes
An adversarial pass over the finished branch went looking for other live
instances of what we just fixed, and turned up one of each. Both are
pre-existing, and both are small enough to fold in here rather than leave
next to a change whose whole point is that these shapes are bugs.
The unroll registry had the blocking-send shape in its worst form. Its
requestPersistenqueues a persistence attempt by Telling the registry's ownmailbox, and all but one of its callers run on the registry's receive
goroutine. The send is unbounded, the mailbox holds 64, and the context is a
background one, so a registry that ever falls behind parks the single
goroutine that could drain it and waits for room that only it could make.
That is not the slow leak the timeout actor had, it is a wedge that nothing
short of shutdown clears. The two detached-goroutine self-sends alongside it
have the milder version. All three now use TryTell and re-arm, and the
deadlock test is mutation-verified: put the blocking Tell back and it hangs.
The round FSM had the invisible-logger shape. Its error reporter resolved
through
build.LoggerFromContexton the actor's lifecycle context, whichdescends from
context.Background(), so every "FSM error" landed inbtclog.Disabledexactly like the timeout actor's delivery failures did.Both construction sites already build a prefixed
fsmLoggerfor the FSMitself, so the reporter just takes that.
Two findings from the same pass are filed rather than fixed here, because
they are wider than this change: #1086 (the actor framework's own per-actor
diagnostics resolve to a disabled logger) and #1087 (the residual blocking
Tells on the serverconn ingress and unroll late-admission paths).