Repository navigation
multi: add attempt-preserving postpone semantics to the durable actor runtime - #1122
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 17b03e7d86
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
📚 Doc drift advisoryScoped audit of the packages this PR touches — diff --git a/db/actordelivery/sqlc/AGENTS.md b/db/actordelivery/sqlc/AGENTS.md
index 35036803..f22694cb 100644
--- a/db/actordelivery/sqlc/AGENTS.md
+++ b/db/actordelivery/sqlc/AGENTS.md
@@ -10,8 +10,14 @@ by `sqlc` from `db/actordelivery/queries/mailbox.sql`; regenerate with
## Key Types
- `Queries` / `Querier` — generated query struct and interface (enqueue,
- lease, peek, ack/nack, extend, expire, outbox claim/complete/fail,
- dedup, FSM checkpoints, dead letters).
+ lease, peek, ack/nack, postpone, extend, expire, outbox
+ claim/complete/fail, dedup, FSM checkpoints, dead letters).
+- `PostponeMailboxMessage` / `PostponeMailboxMessageByID` — the
+ attempt-preserving release pair backing `actor.Postpone(delay)`. The
+ fenced variant validates `lease_token` and decrements attempts (clamped
+ at zero) to compensate the increment the leased claim applied; the by-id
+ variant is unfenced and leaves attempts untouched, because the leaseless
+ peek never incremented them.
- `MailboxMessage`, `OutboxMessage`, `AskResult`, `FsmCheckpoint`,
`DeadLetter`, `ProcessedMessage` — row models for the actor-delivery
tables.
diff --git a/db/actordelivery/sqlc/CLAUDE.md b/db/actordelivery/sqlc/CLAUDE.md
index 35036803..f22694cb 100644
--- a/db/actordelivery/sqlc/CLAUDE.md
+++ b/db/actordelivery/sqlc/CLAUDE.md
@@ -10,8 +10,14 @@ by `sqlc` from `db/actordelivery/queries/mailbox.sql`; regenerate with
## Key Types
- `Queries` / `Querier` — generated query struct and interface (enqueue,
- lease, peek, ack/nack, extend, expire, outbox claim/complete/fail,
- dedup, FSM checkpoints, dead letters).
+ lease, peek, ack/nack, postpone, extend, expire, outbox
+ claim/complete/fail, dedup, FSM checkpoints, dead letters).
+- `PostponeMailboxMessage` / `PostponeMailboxMessageByID` — the
+ attempt-preserving release pair backing `actor.Postpone(delay)`. The
+ fenced variant validates `lease_token` and decrements attempts (clamped
+ at zero) to compensate the increment the leased claim applied; the by-id
+ variant is unfenced and leaves attempts untouched, because the leaseless
+ peek never incremented them.
- `MailboxMessage`, `OutboxMessage`, `AskResult`, `FsmCheckpoint`,
`DeadLetter`, `ProcessedMessage` — row models for the actor-delivery
tables.How to apply: save the diff above and Advisory only — this check never fails the build. Run: https://github.com/lightninglabs/wavelength/actions/runs/31230542413 |
|
An adversarial audit pass ran against the first 6 commits; the 7 commits on top address what it found. Dispositions: Fixed:
|
|
@claude review this |
|
Claude finished @Roasbeef's task in 3m 4s —— View job PR Review: attempt-preserving postpone semanticsI read through the core runtime (
On the Codex P1 finding (zero-row transactional postpone) — I agree it's real, but calibrate the fixThe finding is correct that there is a lease-fence asymmetry on the classic tx path ( rows, ppErr := postponeMessage(ctx, store, delivery.ID, delivery.LeaseToken, delay)
if ppErr == nil && rows == 0 {
// debug log only
}
return ppErr // nil → processInTransaction commits the txCompare the three terminal branches in the same function:
So postpone inherited the nack path's shape, not the ack path's. When a competing owner (another process/replica) reclaims the lease during a long Two things worth adding to the disposition, though: 1. Severity is narrow, and the PR already scopes most of it away. The classic tx path is single-worker-only ( 2. The suggested one-line fix ( The correct shape is: roll the tx back without triggering the destructive nack. That needs Things that look correct and well-handled
Minor / non-blocking
SummarySolid PR. The only substantive item is the Codex finding, and I'd land it slightly differently than proposed: it's a real fence asymmetry worth closing, but it's P2 (needs cross-process contention and a write-then-postpone behavior, neither of which any adopter here does), and the fix must roll back without invoking the unfenced dead-letter nack — returning |
|
Review items addressed in 8d90e52 and 56aa61b: P2 (lost lease mid-turn): fixed to your design constraint. A new errLostLeaseMidTurn sentinel flows from both the postpone and nack zero-row branches of handleResultInTx; processInTransaction treats it as roll-back-and-return with no nack, no dead-letter, and no MarkProcessed, so the legitimate owner's turn governs the row and the stale consumer's writes vanish with the rollback. The zero-row check is fenced-only (LeaseToken != ""): an unfenced by-ID release matching nothing is a benign already-gone duplicate, not a lost lease. The regression test steals the lease mid-turn with MaxAttempts 1 and was verified against both halves: removing the zero-row check commits the stale writes, and removing the sentinel guard reproduces exactly the hazard you predicted (the unfenced dead-letter arm deletes the row its legitimate owner holds). The nack half of the fix predates this PR and is noted as part of #1124's family. Doc drift: fixed. The sqlc guide pair now lists both postpone queries with the fenced/by-ID attempts distinction. Tracking issue: filed as #1125 (keyed-lane FIFO inversion at the max-attempts boundary, with the lease-liveness disjunct as the suggested fix). CI re-running on the new head. |
|
@Roasbeef, remember to re-request review from reviewers when ready |
1 similar comment
|
@Roasbeef, remember to re-request review from reviewers when ready |
56aa61b to
21ad7af
Compare
|
/gateway review Review target: #1122 at head 21ad7af, rebased onto main e5a89c8. Purpose: verify that actor.Postpone(delay) lets valid Tell work wait without consuming attempts, while never committing stale transactional writes, losing durable work, violating lease ownership, or leaving an unbounded retry storm. Since the prior review, the branch:
Please independently trace:
For each finding, give exact preconditions, a concrete reachable trace, consequence, likelihood, existing guards/recovery, whether this PR introduced or worsened it, the smallest failing test, and fix regression risk. Classify it as blocker, follow-up, or non-actionable. Do not re-raise prior findings without new evidence. #1125 deliberately tracks the keyed-lane plus worker-pool final-attempt caveat because no current adopter combines those conditions. Validation is listed in the PR body. All substantive GitHub CI jobs are green. End with exactly one verdict: BLOCKED, READY WITH FOLLOW-UPS, or READY. |
|
❌ gateway review failed: code-review skill failed across all auth modes: You've hit your weekly limit · resets 6am (UTC) See workflow logs for details. |
|
/gateway review |
There was a problem hiding this comment.
Gateway review — 3 findings
🔴 0 Blocker · 🟠 1 Major · 🟡 2 Minor · 🔵 0 Nit
Summary
The PR adds an attempt-preserving Postpone release to the durable actor runtime and adopts it in the OOR registry for over-cap admissions and deferred self-transfer hints. The core mechanics are right: the fenced/unfenced split (PostponeMailboxMessage decrements to compensate the lease-time bump, PostponeMailboxMessageByID leaves attempts alone because the peek never bumped it) leaves the retry budget byte-identical either way, detection runs before TellRetryPolicy on all three execution paths, shouldMarkProcessed is correctly suppressed so a postponed row is not dedup-skipped, and Ask/DurableAsk are structurally excluded because their branches precede the Tell branch in both handleResult and handleResultInTx. The errLostLeaseMidTurn sentinel is the right shape — rolling back without entering Delivery.Nack's unfenced dead-letter arm is exactly the distinction that matters at the final attempt, and the regression test steals the lease with MaxAttempts = 1 so the assertion is load-bearing rather than decorative.
The over-cap horizon is the strongest part of the change: deriving the bound from the durable row's created_at via DeliveryEnqueuedAt rather than a map keyed on an operator-chosen session id is correct for an attacker-controlled stream, since the bookkeeping would otherwise be the resource the cap defends. Treating "no timestamp" as "no horizon information" rather than "infinitely old" is the right failure-open choice.
Three concerns. The blocking one is documentary, not behavioral: the new invariant text asserts a tx-path dead-letter bug that the code on main already guards against, and that claim is replicated across four files and used to justify a tracking issue. The other two are narrow gaps in the PR's own "bound your own horizon" and "the budget is exactly what it was" invariants.
Bot commands
/gateway re-review— re-run after pushing changes (maintainers)/gateway dismiss <id>— silence a finding (maintainers)/gateway explain <id>— elaborate on a finding (anyone)
|
🤖 gateway audit metadata for this PR — auto-generated, please don't edit. |
In this commit, we add the two release queries behind the new postpone semantics. PostponeMailboxMessage is the fenced variant: the leased claim pre-incremented attempts, so the decrement here restores the retry budget to exactly what it was before the delivery, with a CASE clamp so a corrupt row can never wrap negative. PostponeMailboxMessageByID is the leaseless counterpart and leaves attempts untouched, because the peek never incremented them.
In this commit, we regenerate the actor delivery query layer via make sqlc to pick up the two postpone queries. No handwritten changes.
In this commit, we give behaviors a way to say "not now" without walking a message toward the dead-letter table. A nack burns one of the message's finite delivery attempts, so a consumer that is merely waiting on an external condition (a capacity slot, a peer draining, an operator catching up) dead-letters an innocent message one redelivery at a time. Returning actor.Postpone(delay) instead releases the message for redelivery after the delay with its retry budget fully intact. The consume path detects the typed PostponeError before the Tell retry policy on both execution paths (the tx fold and the non-tx tail), never marks a postponed message processed (so the redelivery is not dedup-skipped), and logs at debug level since a postpone is control flow, not a failure. Only Tell deliveries honor it: an Ask has a caller parked on the promise, so a postponed Ask would strand that caller; an Ask behavior returning the error gets ordinary error treatment instead. DeliveryStore gains the fenced/unfenced pair (PostponeMessage / PostponeMessageByID) mirroring the ack/nack shape: the fenced variant decrements attempts to compensate the lease-time bump, the by-ID variant leaves them untouched because the leaseless peek never bumped them. The flip side is deliberate and documented: a postponed message never dead-letters by attempts, so behaviors must bound their own postpone horizon.
In this commit, we stop burning a routed hint's finite delivery attempts for a condition it did nothing to cause. When an incoming admission arrives past MaxConcurrentIncomingSessions, the registry used to fail the turn, which nacks the inbound durable message and walks it one redelivery closer to the dead-letter table. Being over the cap is a not-now condition: it clears the moment a resident session terminates and is reaped, so the hint that happened to arrive while the daemon was full should wait, not die. The over-cap rejection on the routed-message path now returns actor.Postpone(incomingCapBackoff) wrapped alongside the existing errIncomingAdmissionCapped sentinel, so the consume path releases the delivery on a five second backoff with its attempt budget fully intact, while boot restore's skip check and the RPC surface keep matching the sentinel. We apply this only where the admission is driven by a Tell: handleResolveIncoming (the hint the event router pushes) and handleDriveEvent's lazy restore of an already-admitted session. An Ask-driven admission has a caller parked on the promise and gets ordinary error treatment for a postpone anyway, and the incoming cap never applies to StartTransferRequest, which is outgoing. Boot restore keeps its own treatment: it already skips an over-cap row rather than aborting the boot, and it consults ensureChild directly, which still returns the bare sentinel.
In this commit, we retire the custom TellRetryPolicy that kept a deferred self-transfer hint alive, because postpone now expresses what that policy was reaching for and expresses it correctly. The policy answered "retry after 30 seconds, always" for errSelfTransferDeferred so the hint would never dead-letter while its outgoing session ran, and delegated every other error to the default policy. The problem is that a nack-with-retry increments attempts on every release, and the claim and peek queries both filter on attempts < max_attempts. So after ten deferrals (five minutes on the flat backoff) the row fell out of the eligible set entirely: it never dead-lettered, which is what the policy promised, but it also stopped redelivering, which silently retired the crash-safety net the durable copy exists to be. Returning actor.Postpone(selfHintRedeliveryBackoff) from the defer branch gives the same 30 second flat wait with attempts untouched, so the hint stays claim-eligible for as long as the outgoing session takes. With the defer branch postponing, the policy override has nothing left to special-case, so the registry goes back to the default Tell retry policy and real errors keep their exponential backoff and dead-letter boundary.
In this commit, we write down the postpone contract so the next behavior that wants to say "not now" reaches for it instead of inventing another never-dead-letter retry policy. The new Postpone Semantics section in the durable actor architecture doc sets out why a nack is the wrong tool for a transient condition, tabulates the one dimension where the two paths differ (the attempt budget), and explains the fenced and leaseless store mechanics: the leased variant decrements to compensate the claim-time bump, the by-id variant leaves attempts alone because the peek never bumped them. Two consequences get their own treatment because they are easy to miss. Postpone is Tell-only, since an Ask has a caller parked on the promise that a delay would strand. And a postponed message never climbs toward max_attempts, which means nothing dead-letters it automatically: the framework has removed the only mechanism that would eventually give up, so a behavior that postpones has to bound its own horizon or accept that the message waits forever. We also spell out that a postponed head still blocks its correlation-key lane, since the row is very much still in the mailbox, and close with the OOR over-cap admission and the deferred self-transfer hint as the first two adopters. The per-package guides pick up the same material at the altitude each one works at: baselib/actor gains the Postpone key type and four invariants, db/actordelivery gains the two queries and the two store methods plus why their attempt handling differs, and oor gains the over-cap postpone invariant and the story of the retry policy it replaced.
In this commit, we give a postponing behavior the reference it needs to bound its own horizon. Postpone deliberately removes the attempt-based give-up mechanism, so the contract asks every adopter to decide for itself when waiting has stopped making sense. That obligation is hard to discharge without a trustworthy notion of how long the message has already been waiting. The mailbox row already knows. Its created_at is set once at enqueue and no release path rewrites it: a nack moves available_at and bumps attempts, a postpone moves available_at and restores attempts, and neither touches the creation time. Delivery gains an EnqueuedAt field copied from the leased row, and the consume path stamps it onto the processing context via DeliveryEnqueuedAt. The stamp happens in processDelivery, above the fork into the three execution paths, so the tx fold, the non-tx tail, and the Read/Commit handle all agree. Deriving the age from the row rather than from behavior-side state is the point, not an implementation detail. A behavior that tracked waits in a map would key that map on something the sender chose, and for an attacker-controlled message stream the bookkeeping meant to protect the actor becomes the resource the attacker grows. The row's own age costs nothing to consult and cannot be inflated by fabricating messages. The accessor returns a bool alongside the timestamp so absence stays distinguishable from a zero time. A store that reports no creation time, or a behavior invoked directly in a test, means "no horizon information", which is not the same as "infinitely old", and a behavior that conflated the two would give up on every message immediately.
In this commit, we close a coverage gap and a small silence in the transaction path's postpone handling. handleResultInTx is a separate implementation from the non-tx tail, and until now only the latter had a test, so a regression in the tx branch would have passed the suite untouched. The new test drives a classic behavior on a tx-aware store through repeated postpones under a retry policy that fails the test if it is ever consulted for a postpone, and asserts the message survives past the point a nack would have dead-lettered it. We also stop discarding the row count from the tx-path postpone. A zero row count means the release did not happen, which on the leased path means the lease expired or was claimed by another consumer, so the attempt this delivery took stays uncompensated and the message redelivers on the lease-expiry path rather than the requested delay. The nack path treats a zero-row release the same way, so we keep the behavior identical and only add a debug line: a postpone that quietly did not happen is otherwise invisible, because the postpone path logs at debug and never warns.
In this commit, we make the over-cap adopter live up to the rule the postpone contract sets for every adopter: bound your own horizon. As written, a capped hint postponed forever, which is exactly the failure mode the contract warns about, and it mattered here more than most because the hint stream is operator-controlled. Every redelivery of a capped hint re-runs validateIncomingAdmission, a wallet-ownership query against the database, and resolveSelfTransfer, a registry-row read, before the cap check rejects it again. Both run before the cap is consulted, so the work happens on every cycle no matter how long the daemon has been full. An operator streaming fabricated session ids could therefore build a churn queue against the single-worker registry that no amount of waiting drains, with each entry renewing itself every five seconds and none of them ever reaching a horizon. postponeOverCap now postpones only while the delivery is younger than overCapPostponeHorizon, ten minutes, measured against the durable row's enqueue time. Past the horizon it returns the plain capped sentinel, so the ordinary nack path takes over and the hint dead-letters into a table where it is visible and requeue-able rather than churning invisibly. Ten minutes outlasts any realistic transient burst (the cap defaults to 1024 resident sessions and a slot frees on every terminal reap) while converting a hostile backlog on a human timescale. The age comes from actor.DeliveryEnqueuedAt rather than from a per-session map, because the session ids in that stream are operator-chosen and the map would be the very unbounded resource this is defending. A delivery that reports no enqueue time postpones as before, since absence means we have no horizon information rather than that the message is infinitely old. We also correct the backoff constant's comment. Nothing signals the mailbox wake channel when a postponed row becomes eligible, so it is rediscovered by the idle poll backoff and the effective redelivery gap is roughly five to thirty five seconds, not the five the old wording implied.
In this commit, we close an asymmetry in how the registry treats the concurrency cap. handleDriveEvent postpones when lookupOrRestore cannot make a session resident, but handleResumeSession returned the bare sentinel from the identical call, so the two Tell-driven paths gave the same condition opposite treatment. A resume is not a lesser delivery. It arrives as a Tell from the timeout retry callback and carries a real timer expiry that the session needs in order to re-drive its outbox, so nacking it for a transient cap spends its attempts and eventually dead-letters work that was never wrong. Applying the same wrap makes every Tell-driven path that consults lookupOrRestore agree, and the horizon added alongside it keeps the wait bounded here too.
In this commit, we document a boundary case in the claim query's correlation-key anti-join that only postpone can turn into a real ordering bug, and we deliberately leave the SQL alone. The anti-join passes over a predecessor that has exhausted its retry budget, so a dead row cannot wedge its lane forever. A predecessor leased on its final attempt already satisfies attempts == max_attempts after the claim pre-increment, which means it is invisible to the anti-join while it is still being processed and a competing worker may claim its same-key successor. With only ack and nack available that is harmless, because the predecessor can then only complete or dead-letter. A postpone breaks it: the release decrements attempts back below the cap, so the predecessor becomes eligible again and reprocesses after the successor already ran, inverting per-key FIFO. Reaching that state needs all three of keyed correlation lanes, more than one worker, and a postponing behavior, and no adopter combines them today. Complicating the hot claim query for a configuration nothing runs would be the wrong trade, so we record the exclusion where the next person will meet it, next to the predicate itself, along with the intended repair: a lease-liveness disjunct so a currently-leased predecessor blocks its successors regardless of its attempts. The sqlc stubs are regenerated only to carry the comment through.
In this commit, we make the in-memory checkpoint store's PostponeMessage validate the lease token, which its own NackMessage already does and the real SQL query has always done. Without the check the mock accepted a postpone from a stale consumer against a row another consumer now owns, decremented that row's attempts, and reported success where the store would have returned zero rows. Nothing depends on the gap today, but a mock that is more permissive than the thing it stands in for is a test that cannot fail for the right reason, and the fence is precisely what the postpone path relies on to keep the attempt budget honest.
In this commit, we correct the documented retry boundary. We also scope the attempt-neutral, horizon, and FIFO guarantees to where they hold. Both non-transaction and classic transaction paths check ShouldDeadLetter before retry release. A TellRetryPolicy can stop retries early or choose the delay. It cannot extend MaxAttempts. The old OOR self-hint policy therefore dead-lettered at exhaustion on its Read/Commit path. There is no separate go-dark retry-policy asymmetry. A successful postpone preserves the attempt budget. A classic transaction turn that loses its lease is the exception. Rolling the turn back also discards the compensating decrement. The earlier lease claim's increment remains committed. The lost turn spends one attempt. No current classic behavior returns a postpone, so we document the restriction instead of adding cross-transaction compensation to this PR. The self-transfer adopter deliberately has no time horizon. The hint names the wallet's own change output. Discarding it while the outgoing session is pending is worse than retaining it. Terminal reap redrives it immediately. The durable 30 second copy covers crashes and missed redrives. The docs now say that operators should watch old non-terminal sessions and inspect the registry mailbox when one is stuck. The correlation-key section also scopes per-key FIFO for postponing consumers. It holds when the actor is single-worker or the lane's messages never reach their final attempt. We explain the anti-join boundary behind that restriction. The lease-liveness disjunct remains the prerequisite for lifting it. Finally, we retain the row-derived DeliveryEnqueuedAt guidance. We also retain the OOR over-cap adopter's ten minute horizon. Those bounded paths still fail into the normal visible dead-letter flow after their wait expires.
In this commit, we stop the transaction path from committing a behavior's writes under a row it no longer owns. The postpone and nack branches of handleResultInTx both discarded the row count from their fenced release, so a release that matched nothing (the lease expired mid-turn and another consumer claimed the message) looked exactly like success. The closure returned nil, the transaction committed, and one consumer's state changes landed for a message a different consumer was already reprocessing. Both branches now interpret the count through releasedRowsInTx, which distinguishes the two ways a zero can arise. An unfenced by-id release matching nothing is benign, it just means the row is already gone. A fenced release matching nothing means the lease-token comparison failed, so the message has changed hands, and that returns errLostLeaseMidTurn to roll the transaction back. processInTransaction then treats that sentinel as roll-back-and-return: no nack, no dead-letter, no processed mark. Skipping the nack is the load-bearing half rather than a tidiness choice. Delivery.Nack dead-letters once the attempt budget is spent, and that arm is unfenced, running MoveToDeadLetter and DeleteMessage by id with no lease check, so routing a stale consumer through it would let it delete a message the legitimate owner is in the middle of processing. Rolling back and walking away leaves the row and its outcome entirely to that owner. The nack half of this predates the postpone work: that branch has ignored its row count since it was written, and it shares the fence asymmetry described in issue #1124. We fix both together because they are the same mistake in adjacent branches and a partial fix would invite the next reader to assume the other one was deliberate. The regression test steals the lease from inside the behavior and asserts the turn's writes never commit, the row is neither dead-lettered nor deleted, it keeps the new owner's token, and it is not marked processed. The mock's ExecTx grows real rollback semantics for a behavior-visible write log, since without that a test cannot tell a committed turn from a rolled-back one. The non-transaction path is untouched: Delivery.Postpone already surfaces ErrLeaseExpired there.
In this commit, we add the two postpone queries to the generated query layer's own package guide, which still described the mailbox surface as enqueue, lease, peek, ack/nack, extend, and expire. Someone reading that guide to find out what the layer exposes would have concluded postpone did not exist there. The entry spells out why the fenced and by-id variants treat attempts differently, since that asymmetry is the whole point of the pair and is easy to read as an inconsistency: the fenced query decrements to compensate the increment the leased claim applied, and the by-id query leaves attempts alone because the leaseless peek never applied one.
21ad7af to
9ec18f5
Compare
|
/gateway re-review Re-review target: 9ec18f5. All three findings were addressed in folded docs commit 337e754:
No runtime code changed from the prior reviewed head. The three threads are replied to and resolved. All CI checks are green. Please verify only whether the corrections fully address F1-F3 or expose a concrete material defect. Apply the stopping rule once those findings are closed. |
There was a problem hiding this comment.
Gateway review
No findings.
Status of prior findings
- F1 addressed: Fixed in
baselib/actor/AGENTS.md/CLAUDE.md(both now: "Both the non-tx path ... and the classic tx path (handleResultInTx) enforceShouldDeadLetterbefore retry release"),docs/durable_actor_architecture.md(transaction-path bullet now states the retry branch checksShouldDeadLetterand falls through toMarkProcessed+MoveToDeadLetter), and the OOR adopter paragraph (now "the tenth deferral dead-lettered the hint regardless of what the policy answered"). No "go dark" / never-reaches-dead_lettersclaim survives anywhere in the touched docs, the conclusion the passage supports is unchanged and still correct, and the stale premise in #1124 was corrected so nobody acts on it. - F2 addressed: Fixed in
oor/AGENTS.md/CLAUDE.mdanddocs/durable_actor_architecture.md— the exemption is now explicit rather than silent: the hint names the wallet's own change output so discarding it is worse than retaining it, terminal reap is the immediate redrive, the durable 30s copy covers a crash or missed redrive, and the operator signal is named (watch old non-terminal outgoing sessions, inspect the registry mailbox, the hint will not appear indead_letters). That was one of the two options the finding offered, and it is the right one here. - F3 addressed: Fixed in
baselib/actor/AGENTS.md/CLAUDE.md("Fenced postpone is attempt-neutral only while the consumer still owns the lease") anddocs/durable_actor_architecture.md(the fenced-path exception paragraph). The rollback-discards-the-decrement-while-the-claim-increment-stays-committed mechanism, the one-attempt cost per lost turn, and the exhaustion-without-a-dead-letter consequence are all stated, along with the scope that closes it in practice: no current classic-path behavior returns a postpone. Deferring cross-transaction compensation for a latent path rather than adding it speculatively is the correct trade.
Bot commands
/gateway re-review— re-run after pushing changes (maintainers)/gateway dismiss <id>— silence a finding (maintainers)/gateway explain <id>— elaborate on a finding (anyone)
Problem
A durable actor currently has one way to say that it cannot process a Tell: return an error. The runtime nacks the message and spends one delivery attempt.
That is correct for a failed message. It is wrong for a valid message waiting on a transient condition, such as an OOR concurrency slot or a self-transfer's outgoing session. Repeated nacks eventually exhaust
MaxAttemptseven though the message itself is valid.A custom
TellRetryPolicycannot extend that ceiling. Both the non-transaction path and the classic transaction path enforceShouldDeadLetterbefore retry release. A policy can stop retries early or choose the delay, but it cannot express an attempt-preserving wait.Postpone contract
This PR adds
actor.Postpone(delay)for Tell handlers. A postpone:TellRetryPolicy; andAsk deliveries do not honor postpone. An Ask has a caller waiting on its promise, so its error is returned normally and the caller decides whether to retry.
The store exposes two release operations:
PostponeMessagevalidates the lease token and decrementsattemptsto compensate for the leased claim's pre-increment;PostponeMessageByIDleavesattemptsunchanged because the single-worker leaseless peek did not increment it.A successful release leaves the retry budget exactly as it was before delivery. SQLC output is generated from
db/actordelivery/queries/mailbox.sql.Bounded waiting
Postpone deliberately removes attempt-based give-up. Each adopter must explicitly decide whether waiting is bounded and document its recovery and observability path.
The runtime exposes the durable row's immutable enqueue time through
DeliveryEnqueuedAt. The OOR over-cap adopter postpones for five seconds while the delivery is younger than ten minutes. After that horizon it returns the ordinary cap error, allowing the normal retry and dead-letter path to make stuck work visible.The timestamp comes from the durable row rather than a map keyed by sender-controlled data, so the safeguard does not create a new unbounded resource.
OOR adopters
MaxConcurrentIncomingSessionspostpone on all Tell-driven paths: routed hints, lazy drive-event restore, and resume delivery.The self-transfer wait deliberately has no time horizon. The hint names the wallet's own change output, so discarding it while the outgoing session is pending is worse than retaining it. Terminal reap redrives it immediately. The durable postponed copy covers a crash or missed redrive. If the outgoing session never becomes terminal, the hint remains in the registry mailbox and does not enter
dead_letters; operators should watch old non-terminal outgoing sessions and inspect that mailbox when one is stuck.Lease safety
The classic transaction path now treats a zero-row fenced postpone or nack as a lost lease. It rolls the actor transaction back and returns without nacking, dead-lettering, or marking the message processed.
That distinction matters at the final attempt. Routing a stale consumer through the ordinary unfenced dead-letter arm could otherwise delete a row held by the legitimate owner.
The rollback also discards a postpone's compensating attempt decrement, while the earlier lease claim's increment remains committed. A lost-lease turn therefore spends one attempt. No current classic-path behavior returns a postpone. This PR documents the restriction instead of adding cross-transaction compensation for a latent path.
Known scope
A postponed message still blocks later messages in the same correlation-key lane. The documented final-attempt FIFO caveat requires keyed lanes, multiple workers, and a postponing behavior; no current adopter combines those three. Follow-up #1125 tracks the lease-liveness change needed before supporting that combination.
Validation
Local validation after rebasing onto current
main:make sqlc-checkmake tidy-module-checkmake doc-checkmake commitmsg-lint range=origin/main..HEADmake fmt-changedmake lint-changed-localmake unit timeout=15mbaselib/actor,db/actordelivery,oor, andserverconnsuitesstdlog traceThe focused OOR tests prove that capped and self-transfer messages retain their attempts while postponed. Capped messages return to ordinary failure after their configured horizon. Self-transfer messages redrive when the outgoing session becomes terminal. The lost-lease regression proves that stale transactional writes roll back without invoking the destructive dead-letter path.