Skip to content

multi: add non-blocking TryTell and bound timeout actor delivery - #1082

Merged
Roasbeef merged 11 commits into
mainfrom
actor-send-backpressure
Aug 4, 2026
Merged

Roasbeef merged 11 commits into
mainfrom
actor-send-backpressure

Conversation

@Roasbeef

@Roasbeef Roasbeef commented Aug 2, 2026 •

Copy link
Copy Markdown
Member

In this PR, we close out the wavelength side of the actor deadlock fixed in
lumos#734: two actors
wedged in production for 8.5 hours because a receive goroutine blocked on an
unbounded Tell into a full peer mailbox while that peer awaited an Ask back
with no deadline. lumos bounded its sends with context timeouts as a stopgap,
but both root causes live in this repo: the ref interface hides the
non-blocking send the mailbox already implements, and the timeout actor
(a process-wide singleton in lumosd, carrying every subsystem's timers)
delivers callbacks with unbounded sends from its single receive goroutine.
See each commit message for a detailed description w.r.t the incremental
changes.

TryTell on the refs

The mailbox layer has had a non-blocking TrySend all along; only the
TellOnlyRef/ActorRef surface hid it. We add TryTell(ctx, msg) error:
it never parks the calling goroutine, a full mailbox returns ErrMailboxFull
so the caller can drop, stash, or schedule a retry, and everything else
matches Tell. This is deliberately a hard interface method rather than an
optional capability check: the bug class being fixed is a silent fallback to
a blocking send, and we want the compiler to name every implementer rather
than letting one quietly keep the old behavior. Downstream, that means the
next lumos pin bump breaks its ref mocks until they grow a one-line
delegation, and in exchange lumos can replace the context.WithTimeout
wrappers from #734 with TryTell for a strictly better failure mode: fail in
nanoseconds instead of burning the deadline against a wedged peer.

One implementation deserves a close read: the durable ref's TryTell is a DB
write bounded at 100ms, never returns ErrMailboxFull (the queue is an
unbounded table), and drops the caller's transaction, so swapping Tell for
TryTell inside a commit closure silently breaks the atomic-enqueue
guarantee. The interface doc spells all three departures out, and the
durable path has its own tests pinning them.

The timeout actor never blocks, and never drops

Callback delivery now goes through TryTell from the receive loop. A delivery
that fails for any non-terminal reason (full mailbox, a slow durable write, a
router with no registered targets) requeues the same fire into the actor's
own timer wheel with doubling backoff, 1s base capped at 30s, one entry per
outstanding reminder no matter how many retries it takes. Only a dead target
(actor terminated, mailbox closed) releases the entry. Nothing is ever
silently dropped: the trade is bounded memory, unbounded time, and a Warn
per attempt naming the target. The clock path coalesces timer fires through
a non-blocking self-signal instead of stacking one parked goroutine per fire
against a backed-up mailbox.

This also fixes a pre-existing cancellation race: the old code deleted the
entry before delivering, so a Cancel racing a fire could not actually stop
it. Entries now live until delivered, so Cancel owns the retry chain too.

Two contract notes for consumers. A recurring tick delivered late carries
its original FiredAt, and a slow consumer stretches its own cadence rather
than accumulating ticks; we checked the one production consumer (lumos's
rounds actor) and it discards FiredAt entirely, treating a tick as a
stateless poll, so the pin bump is safe on this axis. And the timeout actor
now carries a config-injected logger registered as the TMOT subsystem;
before this, its delivery-failure logs resolved to a disabled logger and the
entire failure mode was invisible in a running daemon.

Testing

The regression tests are mutation-verified: reverting delivery to a blocking
Tell freezes the receive loop and fails the responsiveness test, and
reverting the self-signal to a blocking Tell leaks one goroutine per timer
fire and fails the leak test. The retry state machine is covered against the
adversarial interleavings (cancel during an armed retry, reschedule
superseding an armed retry, terminal vs transient errors on both the oneshot
and recurring paths), and every TryTell implementation, including the
durable, router, and wrapper refs, has its semantics pinned. Full suites
pass with -race across baselib, timeout, and every downstream package whose
test doubles grew the new method.

Two more instances of the same shapes

An adversarial pass over the finished branch went looking for other live
instances of what we just fixed, and turned up one of each. Both are
pre-existing, and both are small enough to fold in here rather than leave
next to a change whose whole point is that these shapes are bugs.

The unroll registry had the blocking-send shape in its worst form. Its
requestPersist enqueues a persistence attempt by Telling the registry's own
mailbox, and all but one of its callers run on the registry's receive
goroutine. The send is unbounded, the mailbox holds 64, and the context is a
background one, so a registry that ever falls behind parks the single
goroutine that could drain it and waits for room that only it could make.
That is not the slow leak the timeout actor had, it is a wedge that nothing
short of shutdown clears. The two detached-goroutine self-sends alongside it
have the milder version. All three now use TryTell and re-arm, and the
deadlock test is mutation-verified: put the blocking Tell back and it hangs.

The round FSM had the invisible-logger shape. Its error reporter resolved
through build.LoggerFromContext on the actor's lifecycle context, which
descends from context.Background(), so every "FSM error" landed in
btclog.Disabled exactly like the timeout actor's delivery failures did.
Both construction sites already build a prefixed fsmLogger for the FSM
itself, so the reporter just takes that.

Two findings from the same pass are filed rather than fixed here, because
they are wider than this change: #1086 (the actor framework's own per-actor
diagnostics resolve to a disabled logger) and #1087 (the residual blocking
Tells on the serverconn ingress and unroll late-admission paths).

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 45d73d54de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread timeout/actor.go
// does not stretch a fast ticker by an order of magnitude: a 100ms ticker
// retries in 100ms, while anything slower than the standard base keeps it.
func tickRetryDelay(interval time.Duration, attempt int) time.Duration {
return backoffDelay(min(interval, retryBaseDelay), attempt)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Cap recurring retry delays at the tick interval

When a sub-second recurring callback remains unavailable for multiple attempts, this uses the interval only as the exponential backoff's base, so a 100 ms tick retries after 100 ms, 200 ms, 400 ms, and eventually 30 seconds. Once the consumer recovers, its callback can therefore remain delayed far beyond its configured cadence, contrary to the timeout actor contract that recurring backoff is capped at the tick interval; clamp the final delay to interval, not merely the base.

AGENTS.md reference: timeout/AGENTS.md:L49-L55

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the mismatch, but the fix belongs on the doc rather than the code.

The behaviour you describe is intended: only the starting point of the backoff moves for a fast ticker, and the 30s ceiling applies to recurring entries exactly as it does to one-shots. TestTickRetryDelayStartsFromInterval already pinned tickRetryDelay(100ms, 1000) == retryMaxDelay, so the code and its test agreed with each other all along.

Clamping the final delay to interval is the one thing I don't want here. Every retry attempt against a durable callback is a bounded database write, so a 100ms ticker retrying at 100ms forever would hammer the store it is already struggling with, and the backoff exists precisely to stop that. A consumer that stays unreachable long enough to reach the ceiling has bigger problems than its cadence.

What misled you was real though, and it was mine: the AGENTS.md sentence read "a backoff doubling from 1s to 30s, capped at the tick interval for recurring entries", where "capped at" was meant to attach to the 1s base and just as easily reads as attaching to the 30s ceiling. The test name TestTickRetryDelayCapsAtInterval said the same wrong thing.

Both are fixed as fixups: the invariant now spells out that only the starting point moves, says the ceiling still applies, and gives the durable-write reason; the test is renamed and gained an explicit assertion that the delay exceeds the interval at the cap.

In this commit, we add TryTell to the TellOnlyRef and ActorRef
interfaces. The mailbox layer has always had a non-blocking TrySend, but
the ref layer hid it behind Tell, so a caller that must not park its own
receive goroutine had to wrap Tell in a context timeout. That burns the
whole deadline against a peer that is already wedged, which is exactly
the shape of deadlock we want to make impossible: an actor blocked on a
full peer mailbox while that peer waits on a reply from it.

TryTell attempts an immediate enqueue and hands the message straight
back with ErrMailboxFull when the target has no room, so the caller can
drop, stash, or reschedule it instead of waiting. The context is
consulted only for an immediate cancellation check, and every other
failure keeps the errors Tell already returns, including the dead letter
routing for a terminated target.

We put the method on the interface rather than behind an optional
capability check so the compiler names every ref that has to think about
it. That costs a mechanical method on each test double and on downstream
implementers, but the alternative quietly falls back to a blocking send
in precisely the places we are trying to fix.
In this commit, we stop the timeout actor from parking its receive
goroutine on a requester's mailbox. The actor is a process-wide
singleton shared by every subsystem, so a single backlogged requester
could freeze every timer in the daemon: the fire handler Told the
callback with no deadline, and a full mailbox held the loop until that
one requester drained. It is the same shape of block that wedged two
actors for hours in production.

Delivery now goes through TryTell. A requester with no room keeps its
entry, which is the only copy of the reminder, and the actor re-arms the
same fire on a backoff that doubles from one second up to thirty. Memory
stays at one entry per outstanding reminder however many attempts it
takes, and since the entry stays in the map a Cancel or a reschedule
still owns it, so a cancelled timeout can no longer fire late. A
callback that is gone for good earns no retry chain at all, because no
backoff repairs a terminated actor. Recurring ticks hold the undelivered
tick on the entry so a retry still reports the instant its timer fired.

Retried delivery is by definition later than it used to be, and can land
out of order against other callbacks, so it is worth saying why every
caller in this repo tolerates that. The round actor ignores timeouts for
rounds it no longer tracks and re-checks phase state before acting on
one, while the OOR and credit timers are accelerants whose work is
re-derived at boot if their reminder never shows up.
In this commit, we make the clock goroutine's self-tell non-blocking.
Every timer fire hands an internal message to the actor's own mailbox
from the goroutine the clock ran the callback on, and it used to do that
with an unbounded Tell on a background context. When the actor was
behind on its queue, each fire left a goroutine parked there until the
queue drained, so the actor leaked one goroutine per fire in exactly the
situation it was already struggling with.

The fire now goes in with TryTell and, when the mailbox has no room,
re-arms itself on a short delay instead of waiting. Each entry has at
most one fire outstanding, so deferred fires coalesce into one pending
timer rather than a growing pile of parked senders, and the fire is
deferred rather than dropped. A stopped actor ends the chain, since no
retry repairs a closed mailbox.

The retry handle is deliberately not recorded on the entry, because we
are off the receive goroutine there and touching the entry maps would
race. A Cancel that lands during the deferral is still honoured: the
generation check drops the fire when it eventually arrives.
@Roasbeef
Roasbeef force-pushed the actor-send-backpressure branch from 45d73d5 to 3b77c01 Compare August 4, 2026 01:54
@Roasbeef

Roasbeef commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

@claude review this

@claude

claude Bot commented Aug 4, 2026 •

Copy link
Copy Markdown

Claude finished @Roasbeef's task in 3m 27s —— View job


Review: multi: add non-blocking TryTell and bound timeout actor delivery

I read the core diff (the TryTell interface + all ref implementations, the timeout actor retry state machine, the unroll registry self-tell fix, and the round FSM logger change). Overall this is a careful, well-motivated change: the interface contract in interface.go is unusually precise about the three durable departures, the invariants are documented in the AGENTS files, and the regression tests are mutation-verified. I found one substantive issue (which corroborates the Codex comment) and a few smaller notes.


1. tickRetryDelay does not cap at the interval — confirmed (agrees with Codex P2)

timeout/actor.go:79-81:

func tickRetryDelay(interval time.Duration, attempt int) time.Duration {
	return backoffDelay(min(interval, retryBaseDelay), attempt)
}

The interval is used only as the base of the doubling; backoffDelay then doubles up to retryMaxDelay (30s). So a 100ms recurring tick whose consumer stalls retries at 100ms → 200ms → 400ms → … → 30s. Once the consumer recovers it can still wait most of a 30s window for the next attempt — an order of magnitude past its cadence.

What makes this more than a style nit is that four separate places all describe the opposite behavior:

  • timeout/AGENTS.md:53-54: "backoff doubling from 1s to 30s, capped at the tick interval for recurring entries."
  • The function's own doc (actor.go:76-78): "a 100ms ticker retries in 100ms" — true only for the first attempt.
  • The regression test name TestTickRetryDelayCapsAtInterval and its doc: "a fast ticker retries on its own cadence rather than jumping to the one second base."
  • …yet that same test's assertions (actor_backpressure_test.go:184-187) pin 2*fast, 4*fast, and retryMaxDelay — i.e. they encode the un-capped behavior the prose says doesn't happen.

So the implementation and the contract genuinely disagree, and the test was written to the implementation rather than the documented intent. The fix Codex suggests is right — clamp the final delay, not just the base:

func tickRetryDelay(interval time.Duration, attempt int) time.Duration {
	return min(interval, backoffDelay(min(interval, retryBaseDelay), attempt))
}

This reproduces the documented contract exactly: sub-second intervals stay flat at interval; intervals ≥1s double from 1s and cap at interval (and at 30s for intervals >30s). Note the fix requires updating the TestTickRetryDelayCapsAtInterval assertions (they currently assert the wrong thing despite the test's name) and re-checking the integration assertion near actor_backpressure_test.go:620.

Fix this →


2. Notes (no change requested — flagging for awareness)

  • Durable-callback retry amplification via the shared singleton. With a durable callback behind the shared timeout actor, a slow store surfaces DeadlineExceeded (non-terminal → retried), and each retry re-enqueues under a fresh message ID (at-least-once). On SQLite the interface doc notes a durable TryTell "burns its whole deadline and fails" whenever the caller holds the writer — so a shared timeout actor delivering to a durable SQLite callback could retry-and-duplicate up to the 30s cap indefinitely. This is correctly documented (interface.go:189-194, AGENTS.md:84-89 "prefer a dedicated instance") and the in-repo consumers (oor, credit) already register their own instances, so this is a documented footgun rather than a live bug. Worth keeping an eye on downstream (lumos) when the pin bumps.

  • self atomic.Value and mixed concrete types. Actor.self is an atomic.Value storing a TellOnlyRef[Msg]. atomic.Value.Store panics if called with two different concrete types. Start is documented as call-once and each actor instance is wired with a single ref impl, so this is fine in practice — just a latent footgun if Start were ever called with a different impl type (e.g. swapping a real ref for syncSelfRef mid-life).

  • Router.TryTell turn consumption. The doc correctly warns that the strategy consumes its turn even on a full mailbox, so a retried TryTell may land on a different actor. Fine for homogeneous router targets; the retry loop in the timeout actor tolerates it.

3. Things I checked that look correct

  • The cancellation-race fix (entries survive until delivered, so Cancel owns the retry chain via the gen check) — handleTimerFired/retryOneshot/cancelExisting are consistent, and the retry timer handle is recorded on-goroutine while the off-goroutine signalSelf retry deliberately isn't.
  • signalSelf non-blocking self-tell with selfSignalRetry re-arm, coalescing to one outstanding fire per entry.
  • terminalDelivery restricting terminal to ErrActorTerminated/ErrMailboxClosed, everything else retried — matches the documented rationale.
  • The round FSM logger fix (newLoggerErrorReporter taking the prefixed fsmLogger instead of resolving from a background-descended context) and the TMOT logger injection across all three timeout-actor construction sites in waved (main / credit / oor).
  • The unroll tellSelf conversion of requestPersist + the two detached self-sends to TryTell + re-arm.

I did not run the test suites (this is a review pass). The one actionable item is finding #1; everything else is informational.

· branch actor-send-backpressure

@Roasbeef
Roasbeef force-pushed the actor-send-backpressure branch 2 times, most recently from d983712 to 810eb1d Compare August 4, 2026 02:00
@Roasbeef

Roasbeef commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

Thanks — this reviewed the head from just before I pushed, so part of finding #1 was already in flight. Disposition below.

Finding #1: tickRetryDelay — doc fixed, code held

You and Codex are both right that the implementation and the prose disagreed. I fixed the prose, not the code, and it is worth saying why since you proposed the opposite.

The suggested clamp is min(interval, backoffDelay(...)). For any sub-second interval that collapses to a constant: a 100ms ticker retries at 100ms, forever, with no backoff at all. That is the one behaviour I want to avoid here, and your own note in section 2 is the reason. Every attempt against a durable callback is a bounded database write, so a wedged 100ms tick against a slow store would re-offer ten writes a second for as long as it stayed wedged, and each failed attempt that still commits duplicates a row. Clamping the delay makes the amplification you flagged strictly worse, at its worst precisely when the store is least able to take it.

The backoff exists to stop that. A consumer unreachable long enough to walk 100ms to 30s has been failing for about a minute straight; slowing down is the correct response, and the price is bounded recovery latency on a poll-shaped consumer that re-derives its own work. So the code stands and the contract moves to describe it.

All four of the places you found saying the opposite are now fixed:

  • timeout/AGENTS.md spells out that only the starting point moves, that the 30s ceiling applies to recurring entries too, and gives the durable-write reason.
  • tickRetryDelay's own doc says the same, and no longer implies "retries in 100ms" holds past the first attempt.
  • The test is renamed TestTickRetryDelayStartsFromInterval, and its doc says what it pins.
  • It gained an explicit require.Greater(tickRetryDelay(fast, 1000), fast), so the un-capped behaviour is now asserted deliberately rather than incidentally.

The integration assertion near line 620 only exercises attempts 1 and 2, where the base is the interval, so it holds unchanged and needed no edit.

Section 2 notes

  • Durable retry amplification. Agreed, and it is the load-bearing reason for the above. Documented in interface.go and in the AGENTS.md "prefer a dedicated instance" guidance; in-repo consumers already register their own instances. Tracking for the lumos pin bump.
  • self atomic.Value mixed concrete types. Real, and Start's doc was actively misleading: it said subsequent calls "overwrite the self-ref", when a differing concrete type panics. Reworded to say exactly that.
  • Router.TryTell turn consumption. Agreed, no change.

Two findings from an earlier adversarial pass over the same branch are filed rather than fixed here, since both are wider than this change: #1086 and #1087.

In this commit, we write down what the previous two changes mean for
anyone reaching for these packages. The actor guides now say when to use
TryTell over Tell, namely any send made from inside a receive goroutine,
and why wrapping Tell in a timeout is the weaker answer.

The timeout guide gains the two invariants a caller has to know: the
actor never blocks on a callback, and a callback that is behind gets its
reminder re-armed on a backoff rather than delivered on time. That makes
late and reordered delivery part of the contract instead of a surprise.
In this commit, we fix the retry predicate, which had the test
backwards. It retried only a full mailbox and treated everything else as
permanent, and that is wrong for the two callback shapes that matter
most in the daemon. A durable callback (the OOR session retry timers,
the credit poll timers) enqueues through a database write bounded by the
durable mailbox's own deadline, so a slow database surfaces as a wrapped
DeadlineExceeded and never as a full mailbox. A callback behind a Router
reports that no actor is registered while its target is being replaced.

Both of those were silently discarding the reminder for good, which is
worse than the blocking send we replaced: that one at least delivered
once the database recovered. A credit op chains each poll off the
previous expiry, so one dropped expiry stops that op polling until the
daemon restarts. Only a terminated actor and a closed mailbox are end
states now, and everything else earns another attempt on the same
backoff. Guessing wrongly in this direction costs a retry, while
guessing wrongly in the other direction loses a timer that nothing
re-derives.

While here we cap the recurring retry base at the tick interval, so one
transient failure no longer stretches a 100ms ticker to a full second,
and we swap the shift-based backoff for a doubling loop that cannot
overflow whatever base it starts from.
In this commit, we let the daemon inject a logger into the timeout
actor, because the diagnostics we just added were invisible where they
matter. The actor resolved its logger from the receive context alone,
but an actor's lifecycle context descends from context.Background()
rather than from the actor system's logger-carrying context, so
build.LoggerFromContext always handed back btclog.Disabled. Every line
about a callback we could not reach, or a reminder we gave up on, went
nowhere in a running daemon. That is the same trap as a subsystem whose
logger was never wired: the code looks observable and is not.

The actor now takes a Config with an optional logger and falls back to
the context logger only when nobody supplied one, matching how the
credit behaviors do it. All three daemon call sites pass the TMOT
sublogger, which also turns that previously dead Subsystem constant into
something registered with the log manager and level-tunable like every
other subsystem.

A test asserts a retry actually writes through the injected logger while
the receive context carries none, so this cannot quietly rot back to a
disabled logger.
In this commit, we stop two test doubles from lying about the contract
the ref interface just grew. Both were given a TryTell that delegates to
a Tell which parks the caller until the test releases it, so a caller
that asked for a non-blocking send would have been blocked anyway. The
doubles that model a wedged subscriber are exactly the ones a future
backpressure test would reach for, so leaving them wrong is how the
contract gets quietly broken.

The blocking subscriber now reports a full mailbox, which is the honest
translation of "cannot take this without making you wait". The deferring
one panics instead, because a Tell that completes long after the
caller's deadline has no non-blocking equivalent at all, and a test that
routes into it needs a different double rather than a plausible-looking
answer.
In this commit, we document the two ways a durable target departs from
the contract the interface describes, and cover them with tests. A
durable queue has no capacity, so a durable ref never answers
ErrMailboxFull. What it answers under load is a wrapped
DeadlineExceeded, because the enqueue is a database write the mailbox
bounds with its own short deadline. Any caller that keys its retry
decision off ErrMailboxFull therefore throws durable messages away
exactly when the database is slow, and the timeout actor was doing
precisely that until the previous commit.

The second departure is quieter and worse. Tell carries the caller's
transaction into a durable enqueue, which is what makes the message and
the caller's state change land atomically. TryTell cannot: the mailbox
runs its bounded write on a background context, so the row commits on
its own and can outlive a rolled back transaction. Swapping one for the
other inside a commit closure breaks atomicity with nothing to show for
it, so the interface now says so, and the durable implementation strips
the transaction from the envelope rather than carrying one the write
will not honour.

Tests now cover the durable implementation, which was the only one with
none, plus the router and the two remaining wrapper refs.
In this commit, we write down the property the whole scheduling
arrangement rests on, which was implicit and therefore one careless
change away from being lost. Every scheduler in the daemon, round, OOR
and credit alike, issues a blocking Tell into the timeout actor's
mailbox from inside its own receive loop. That is only safe because the
timeout actor's Receive never blocks and its mailbox therefore always
drains. Put a blocking call anywhere inside that Receive and the
incident shape returns, with the scheduler parked on the timeout mailbox
and the timeout actor parked on a scheduler.

We also record which delivery errors are terminal and why, warn against
handing durable callbacks to the shared timeout singleton when a
dedicated instance costs nothing, and note that a retried tick reports
the instant its timer fired rather than the instant it was delivered. No
consumer in this repo uses recurring ticks, but lumos does, and one
computing its deadline from FiredAt plus the interval could read a
retried tick as already expired. That wants checking before the pin
bumps.
In this commit, we give the round FSM's error reporter a logger that can
actually reach an operator. The reporter resolved its logger from a
stored context, and the context a round FSM runs under is the actor's
lifecycle context, which descends from context.Background() rather than
from anything carrying a logger. build.LoggerFromContext therefore
handed back btclog.Disabled, and every "FSM error" line went nowhere in
a running daemon. The code read as observable and was not.

Both construction sites already build a prefixed fsmLogger for the FSM
itself, so the reporter now takes that same logger and an FSM error
lands beside the transitions that led to it. That also drops the stored
context, which was only ever there to be searched for a logger.

This is the same trap the timeout actor had earlier in this branch,
found by going looking for other instances of it.
In this commit, we fix a self-deadlock in the unroll registry.
requestPersist enqueues a persistence attempt by Telling the registry's
own mailbox, and every one of its callers but one runs on the registry's
receive goroutine. That send is unbounded and the mailbox holds 64
messages, so a registry that ever falls behind parks the one goroutine
that could drain it, waiting for room that only it could make. Nothing
recovers from that short of shutdown, since the send carries a
background context and there is no deadline to expire.

The two detached-goroutine self-sends, the async persist result and the
retry timer, have the milder version of the same problem: they park one
goroutine per event for as long as the registry stays behind.

All three now go through TryTell and re-arm on a short delay when the
mailbox has no room, which is the shape the timeout actor grew earlier
in this branch. Deferring beats dropping because the registry is the
only thing that re-derives this work, and a terminated registry ends the
chain, since no delay repairs a closed mailbox.

The late-admission report in watchChildAdmissionResult keeps its
blocking Tell. It runs detached, it is bounded by a real caller context
rather than a background one, and it already logs a send it could not
complete, so the trade there is the other way round.
@Roasbeef
Roasbeef force-pushed the actor-send-backpressure branch from 810eb1d to 485eca3 Compare August 4, 2026 02:04
@Roasbeef

Roasbeef commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

@claude I dispositioned your review in the comment above, and pushed since. Re-checking the two spots where I went a different way than you suggested would be useful.

Short version:

  1. tickRetryDelay: I fixed the contract, not the code. I did not take the min(interval, backoffDelay(...)) clamp. For any sub-second interval it collapses to a constant, so a 100ms ticker retries at 100ms forever with no backoff at all, and that makes the durable retry amplification from your own section-2 note strictly worse: ten bounded database writes a second against a store that is already slow, each failed-but-committed attempt duplicating a row. All four places you found stating the opposite behaviour are now fixed, and the test asserts the un-capped delay deliberately instead of incidentally.

  2. Your two smaller notes turned out to be real doc bugs, both corrected: tickRetryDelay's own comment implied "retries in 100ms" held past the first attempt, and Start's doc said a second call "overwrites the self-ref" when a differing concrete type panics through the atomic.Value.

  3. Chasing your AGENTS.md:53-54 citation found a defect neither of us was looking for. timeout/ and baselib/actor/ had byte-identical AGENTS.md and CLAUDE.md at the merge base, and this PR's docs commits edited only AGENTS.md. The CI sync check covers the root pair only, so nothing caught it, and the stale per-package CLAUDE.md is the file an agent actually loads. Repaired across the whole series so each of the 11 commits is self-consistent rather than just the tip.

Worth a look at whether you still consider #1 unresolved given the durable-write argument, and whether the doc now reads unambiguously.

@claude

claude Bot commented Aug 4, 2026 •

Copy link
Copy Markdown

Claude finished @Roasbeef's task in 2m 19s —— View job


Re-check of the two dispositioned spots

I re-read the pushed head (485eca3). Both disagreements are resolved in your favor, and the surrounding docs/tests are now self-consistent. Detail below.

  • Re-read tickRetryDelay + its contract
  • Verify the durable-write argument holds against a clamp
  • Check the doc now reads unambiguously across all four sites
  • Verify the two smaller doc fixes (tickRetryDelay comment, Start doc)
  • Sanity-check the AGENTS.md/CLAUDE.md sync repair

1. tickRetryDelay — I withdraw the finding; holding the code was the right call

You're right, and I'll be explicit about where my original suggestion was wrong. My proposed min(interval, backoffDelay(min(interval, retryBaseDelay), attempt)) collapses to a constant for any sub-second interval: with interval = 100ms, the base is min(100ms, 1s) = 100ms, and the outer min(100ms, …) re-clamps every doubled value back down to 100ms. So the clamp doesn't cap the backoff — it deletes it. A wedged 100ms durable ticker would then re-offer a bounded DB write ~10×/sec for as long as the store stayed slow, and since a failed-but-committed durable enqueue duplicates a row (the at-least-once amplification from my own section-2 note), the clamp makes that note strictly worse, precisely when the store can least absorb it. The current code (actor.go:86-88, base min(interval, retryBaseDelay), doubling up to the 30s ceiling) is the behavior that actually protects the store. The cost is bounded recovery latency on a poll-shaped consumer that re-derives its own work — the correct thing to trade.

The contract now matches the code at all four sites I flagged, and reads unambiguously:

  • timeout/AGENTS.md:56-61 — "Only the starting point moves: the 30s ceiling applies to recurring entries too … deliberate, since every attempt against a durable callback is a bounded database write."
  • tickRetryDelay doc (actor.go:79-85) — spells out that only the starting point moves and no longer implies "retries in 100ms" survives past attempt 1.
  • Test renamed TestTickRetryDelayStartsFromInterval (actor_backpressure_test.go:182), doc rewritten to match.
  • Assertions now pin the un-capped behavior deliberately: require.Equal(retryMaxDelay, tickRetryDelay(fast, 1000)) plus the explicit require.Greater(tickRetryDelay(fast, 1000), fast) at line 196 — the intent is asserted, not incidental. The slow-ticker (:200-203) and zero-interval (:206) cases hold.

The integration assertion near line 620 only exercises attempts 1–2, where the base is the interval, so it needs no edit — confirmed.

2. The two smaller doc fixes — both correct now

  • tickRetryDelay comment — no longer implies the 100ms figure holds past the first attempt. ✅
  • Start doc (timeout/actor.go:216-223) — now states it plainly: a second call "carrying the same concrete type replaces the first, while one carrying a different concrete type panics," with the atomic.Value named as the mechanism. That's exactly the footgun; the old "overwrites the self-ref" wording is gone. ✅

3. The AGENTS.md/CLAUDE.md sync repair — verified

diff on both pairs at the head is byte-identical (timeout/, baselib/actor/). Good catch that this slipped the root-only CI sync check while the stale per-package CLAUDE.md is the file an agent actually loads — the timeout CLAUDE.md now carries the full load-bearing invariants block, matching AGENTS.md.


Bottom line: #1 is resolved — I no longer consider it open, and the durable-write argument is the deciding factor, not a tie-break. The docs read unambiguously. No new findings on the re-check. The two deferred items (#1086, #1087) remain the right calls to file rather than fold in.

· branch actor-send-backpressure

@Roasbeef
Roasbeef merged commit b825f4a into main Aug 4, 2026
37 of 41 checks passed
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-v0.1.x-branch Backport this merged PR to v0.1.x-branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant