Skip to content

multi: add attempt-preserving postpone semantics to the durable actor runtime - #1122

Merged
bhandras merged 15 commits into
mainfrom
actor-postpone-semantics
Aug 31, 2026
Merged

bhandras merged 15 commits into
mainfrom
actor-postpone-semantics

Conversation

@Roasbeef

@Roasbeef Roasbeef commented Aug 8, 2026 •

Copy link
Copy Markdown
Member

Problem

A durable actor currently has one way to say that it cannot process a Tell: return an error. The runtime nacks the message and spends one delivery attempt.

That is correct for a failed message. It is wrong for a valid message waiting on a transient condition, such as an OOR concurrency slot or a self-transfer's outgoing session. Repeated nacks eventually exhaust MaxAttempts even though the message itself is valid.

A custom TellRetryPolicy cannot extend that ceiling. Both the non-transaction path and the classic transaction path enforce ShouldDeadLetter before retry release. A policy can stop retries early or choose the delay, but it cannot express an attempt-preserving wait.

Postpone contract

This PR adds actor.Postpone(delay) for Tell handlers. A postpone:

  • releases the durable message after the requested delay;
  • preserves its delivery-attempt budget while the consumer retains ownership;
  • does not mark it processed;
  • bypasses TellRetryPolicy; and
  • logs at debug because it is expected control flow.

Ask deliveries do not honor postpone. An Ask has a caller waiting on its promise, so its error is returned normally and the caller decides whether to retry.

The store exposes two release operations:

  • PostponeMessage validates the lease token and decrements attempts to compensate for the leased claim's pre-increment;
  • PostponeMessageByID leaves attempts unchanged because the single-worker leaseless peek did not increment it.

A successful release leaves the retry budget exactly as it was before delivery. SQLC output is generated from db/actordelivery/queries/mailbox.sql.

Bounded waiting

Postpone deliberately removes attempt-based give-up. Each adopter must explicitly decide whether waiting is bounded and document its recovery and observability path.

The runtime exposes the durable row's immutable enqueue time through DeliveryEnqueuedAt. The OOR over-cap adopter postpones for five seconds while the delivery is younger than ten minutes. After that horizon it returns the ordinary cap error, allowing the normal retry and dead-letter path to make stuck work visible.

The timestamp comes from the durable row rather than a map keyed by sender-controlled data, so the safeguard does not create a new unbounded resource.

OOR adopters

  • Incoming admissions that hit MaxConcurrentIncomingSessions postpone on all Tell-driven paths: routed hints, lazy drive-event restore, and resume delivery.
  • Boot restore keeps skipping capped rows instead of aborting startup.
  • The Ask-driven outgoing start path is unchanged.
  • A self-transfer hint waiting for its outgoing session postpones for 30 seconds. This replaces the custom retry policy that still dead-lettered the hint after ten Read/Commit deliveries.
  • Other OOR failures use the default Tell retry policy.

The self-transfer wait deliberately has no time horizon. The hint names the wallet's own change output, so discarding it while the outgoing session is pending is worse than retaining it. Terminal reap redrives it immediately. The durable postponed copy covers a crash or missed redrive. If the outgoing session never becomes terminal, the hint remains in the registry mailbox and does not enter dead_letters; operators should watch old non-terminal outgoing sessions and inspect that mailbox when one is stuck.

Lease safety

The classic transaction path now treats a zero-row fenced postpone or nack as a lost lease. It rolls the actor transaction back and returns without nacking, dead-lettering, or marking the message processed.

That distinction matters at the final attempt. Routing a stale consumer through the ordinary unfenced dead-letter arm could otherwise delete a row held by the legitimate owner.

The rollback also discards a postpone's compensating attempt decrement, while the earlier lease claim's increment remains committed. A lost-lease turn therefore spends one attempt. No current classic-path behavior returns a postpone. This PR documents the restriction instead of adding cross-transaction compensation for a latent path.

Known scope

A postponed message still blocks later messages in the same correlation-key lane. The documented final-attempt FIFO caveat requires keyed lanes, multiple workers, and a postponing behavior; no current adopter combines those three. Follow-up #1125 tracks the lease-liveness change needed before supporting that combination.

Validation

Local validation after rebasing onto current main:

  • make sqlc-check
  • make tidy-module-check
  • make doc-check
  • make commitmsg-lint range=origin/main..HEAD
  • make fmt-changed
  • make lint-changed-local
  • make unit timeout=15m
  • focused baselib/actor, db/actordelivery, oor, and serverconn suites
  • actor and OOR suites with stdlog trace

The focused OOR tests prove that capped and self-transfer messages retain their attempts while postponed. Capped messages return to ordinary failure after their configured horizon. Self-transfer messages redrive when the outgoing session becomes terminal. The lost-lease regression proves that stale transactional writes roll back without invoking the destructive dead-letter path.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 17b03e7d86

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread baselib/actor/durable_actor.go Resolved
@litbot-9000

Copy link
Copy Markdown
Collaborator

📚 Doc drift advisory

Scoped audit of the packages this PR touches — baselib/actor, db/actordelivery, db/actordelivery/sqlc, oor, serverconn, unroll — found one stale doc: db/actordelivery/sqlc/CLAUDE.md still describes the generated Querier surface without the new PostponeMailboxMessage / PostponeMailboxMessageByID queries (the other five are already current in this PR — serverconn and unroll only gained test-only store stubs, which are not doc-visible).

diff --git a/db/actordelivery/sqlc/AGENTS.md b/db/actordelivery/sqlc/AGENTS.md
index 35036803..f22694cb 100644
--- a/db/actordelivery/sqlc/AGENTS.md
+++ b/db/actordelivery/sqlc/AGENTS.md
@@ -10,8 +10,14 @@ by `sqlc` from `db/actordelivery/queries/mailbox.sql`; regenerate with
 ## Key Types
 
 - `Queries` / `Querier` — generated query struct and interface (enqueue,
-  lease, peek, ack/nack, extend, expire, outbox claim/complete/fail,
-  dedup, FSM checkpoints, dead letters).
+  lease, peek, ack/nack, postpone, extend, expire, outbox
+  claim/complete/fail, dedup, FSM checkpoints, dead letters).
+- `PostponeMailboxMessage` / `PostponeMailboxMessageByID` — the
+  attempt-preserving release pair backing `actor.Postpone(delay)`. The
+  fenced variant validates `lease_token` and decrements attempts (clamped
+  at zero) to compensate the increment the leased claim applied; the by-id
+  variant is unfenced and leaves attempts untouched, because the leaseless
+  peek never incremented them.
 - `MailboxMessage`, `OutboxMessage`, `AskResult`, `FsmCheckpoint`,
   `DeadLetter`, `ProcessedMessage` — row models for the actor-delivery
   tables.
diff --git a/db/actordelivery/sqlc/CLAUDE.md b/db/actordelivery/sqlc/CLAUDE.md
index 35036803..f22694cb 100644
--- a/db/actordelivery/sqlc/CLAUDE.md
+++ b/db/actordelivery/sqlc/CLAUDE.md
@@ -10,8 +10,14 @@ by `sqlc` from `db/actordelivery/queries/mailbox.sql`; regenerate with
 ## Key Types
 
 - `Queries` / `Querier` — generated query struct and interface (enqueue,
-  lease, peek, ack/nack, extend, expire, outbox claim/complete/fail,
-  dedup, FSM checkpoints, dead letters).
+  lease, peek, ack/nack, postpone, extend, expire, outbox
+  claim/complete/fail, dedup, FSM checkpoints, dead letters).
+- `PostponeMailboxMessage` / `PostponeMailboxMessageByID` — the
+  attempt-preserving release pair backing `actor.Postpone(delay)`. The
+  fenced variant validates `lease_token` and decrements attempts (clamped
+  at zero) to compensate the increment the leased claim applied; the by-id
+  variant is unfenced and leaves attempts untouched, because the leaseless
+  peek never incremented them.
 - `MailboxMessage`, `OutboxMessage`, `AskResult`, `FsmCheckpoint`,
   `DeadLetter`, `ProcessedMessage` — row models for the actor-delivery
   tables.

How to apply: save the diff above and git apply it, or run the
doc-gardening skill locally (/doc-gardening db/actordelivery/sqlc) and let
it regenerate the same edit. Remember CLAUDE.md and AGENTS.md must stay
byte-identical.


Advisory only — this check never fails the build. Run: https://github.com/lightninglabs/wavelength/actions/runs/31230542413

@Roasbeef

Roasbeef commented Aug 8, 2026

Copy link
Copy Markdown
Member Author

An adversarial audit pass ran against the first 6 commits; the 7 commits on top address what it found. Dispositions:

Fixed:

  1. Over-cap postpone had no horizon (major). The first adopter violated the PR's own bound-your-horizon rule: every capped hint retried forever at 5s, and each redelivery ran the admission validation queries before the cap check, so a hostile operator streaming fabricated-session hints could build a permanent churn queue on the single-worker registry. The horizon is now derived from the durable row's own enqueue time, exposed to behaviors via the new actor.DeliveryEnqueuedAt(ctx) helper (stateless, so fabricated distinct session ids cannot grow any map): postpone while the delivery is younger than 10 minutes, then fall back to the plain error so the nack path dead-letters the hint where the actor: wire up dead-letter handling with requeue, monitor, and operator surface #1119 tooling can see and requeue it.

  2. Keyed-lane FIFO blind spot at the max-attempts boundary (latent major). A predecessor leased at its final attempt is invisible to the claim anti-join, so a same-key successor can be claimed concurrently, and a postpone brings the predecessor back to reprocess after it. No adopter combines keyed lanes + a worker pool + postpone today, so the fix is scoping, not SQL: the docs now state the exclusion and the anti-join carries a caveat comment naming the lease-liveness disjunct as the future fix.

  3. handleResumeSession asymmetry: now wraps the capped sentinel with the same postpone treatment as the other Tell-driven handlers.

  4. Docs described the wrong failure mode for the retired self-hint policy. The go-dark-forever shape (row stranded past max_attempts, never dead-lettered) is the CLASSIC TX path's behavior; the registry's leaseless non-tx path dead-letters at exhaustion via Delivery.Nack's ShouldDeadLetter arm. Docs corrected and scoped. The pre-existing tx-path strand bug is now filed as actor: tx-path retry policy can strand a message past max_attempts with no dead letter #1124.

  5. Comment fixes (postpone delay is a floor given the idle poll backoff), a debug log on the tx-path lost-lease postpone, a lease-token fence on the serverconn test mock, and a new test covering the classic tx-path postpone branch (verified load-bearing: stubbing the branch fails it).

@Roasbeef

Roasbeef commented Aug 8, 2026

Copy link
Copy Markdown
Member Author

@claude review this

@claude

claude Bot commented Aug 8, 2026 •

Copy link
Copy Markdown

Claude finished @Roasbeef's task in 3m 4s —— View job


PR Review: attempt-preserving postpone semantics

I read through the core runtime (postpone.go, durable_actor.go, delivery.go, delivery_store.go), the SQL layer (mailbox.sql), and the OOR adopter (registry.go). This is a carefully-built feature with unusually thorough docs and invariants. My review focuses on correctness; the design itself is sound.

  • Read core postpone implementation
  • Read DB layer
  • Read OOR adopter + horizon logic
  • Assess the Codex P1 finding
  • Post review

On the Codex P1 finding (zero-row transactional postpone) — I agree it's real, but calibrate the fix

The finding is correct that there is a lease-fence asymmetry on the classic tx path (handleResultInTx, durable_actor.go:1186-1212):

rows, ppErr := postponeMessage(ctx, store, delivery.ID, delivery.LeaseToken, delay)
if ppErr == nil && rows == 0 {
        // debug log only
}
return ppErr   // nil → processInTransaction commits the tx

Compare the three terminal branches in the same function:

  • Success (txDelivery.Ack → delivery.go:217): returns ErrLeaseExpired on zero rows → tx rolls back. Fence enforced.
  • Nack (nackMessage, line ~1232): ignores the row count → returns nil → tx commits. Fence not enforced.
  • Postpone (this PR): ignores the row count → returns nil → tx commits. Fence not enforced.

So postpone inherited the nack path's shape, not the ack path's. When a competing owner (another process/replica) reclaims the lease during a long Receive, this stale consumer's transaction — including any behavior writes made before it returned the postpone — commits anyway. That is a genuine hole.

Two things worth adding to the disposition, though:

1. Severity is narrow, and the PR already scopes most of it away. The classic tx path is single-worker-only (ErrConcurrentClassicBehavior), so a zero-row postpone requires cross-process/replica lease contention — a competing owner can't exist within one daemon. It also only causes damage if the behavior writes state and then postpones; the actual adopter (OOR ensureChild) postpones at the admission choke point before any write, so an empty tx commits harmlessly. This is a defense-in-depth / future-adopter hazard, not a live bug for anything in this PR. Given that, I'd rank it P2, not P1 — but it is still worth closing since the whole point of the feature is that hypothetical future adopters lean on it.

2. The suggested one-line fix (return ErrLeaseExpired) has a wrinkle — don't ship it verbatim. Returning an error routes into processInTransaction's failure handler (durable_actor.go:754-774), which calls delivery.Nack(ctx, err, 10s). On the max-attempts boundary, Nack takes the ShouldDeadLetter branch, and both MoveToDeadLetter(ctx, id, reason) and DeleteMessage(ctx, id) are unfenced (by-ID) — so a stale consumer that lost its lease could dead-letter and delete a row the legitimate owner is actively processing. Off the boundary it's harmless (the fenced nackMessage no-ops on token mismatch), but the boundary case turns a lost-lease postpone into data loss for the rightful consumer.

The correct shape is: roll the tx back without triggering the destructive nack. That needs processInTransaction to distinguish "lease lost — another owner has it, do nothing" from "genuine failure — nack for retry", e.g. a dedicated sentinel returned from handleResultInTx on zero rows that processInTransaction treats as roll-back-and-return. Same treatment would fix the pre-existing nack-path asymmetry too.

Fix this →


Things that look correct and well-handled

  • Fenced/unfenced attempt accounting is right. PostponeMailboxMessage decrements (compensating the lease-time bump) with the CASE WHEN attempts > 0 clamp; PostponeMailboxMessageByID leaves attempts alone (the peek never bumped). Net budget is byte-identical either way, and the clamp can't wrap a corrupt row negative. This is the subtle part and it's implemented correctly (mailbox.sql:186-215).
  • Detection before retry policy, on both paths (handleResultInTx:1176 and handleResult:1336), with postponeDelay matching anywhere in the wrap chain via errors.As. The double-%w in the OOR adopter keeps errIncomingAdmissionCapped matchable for boot-restore/RPC while the consume path sees the postpone — nicely done.
  • shouldMarkProcessed = false for postpone in finishNonTx:854-866 — a postponed message must not be dedup-skipped on redelivery. Correct, and easy to miss.
  • Tell-only enforcement is sound: Ask/DurableAsk never reach the postpone branch, so a parked caller is never stranded.
  • The horizon is the strongest part. Deriving the bound from DeliveryEnqueuedAt (the durable created_at, which no release rewrites) rather than a behavior-side map keyed on an operator-chosen session id is exactly right for an attacker-controlled stream — a per-message map would be the unbounded resource the cap defends. postponeOverCap (registry.go:498-519) treating "no timestamp" as "no horizon info" (postpone as before) rather than "infinitely old" is the correct failure-open choice for tests/stores that don't report one.

Minor / non-blocking

  • The keyed-lane + pool + postpone FIFO-inversion is documented as a scoped-out latent issue (anti-join m2.attempts < m2.max_attempts misses a predecessor leased on its final attempt). I agree scoping is the right call since no adopter combines all three today, and the CAVEAT comment naming the lease-liveness disjunct as the fix is the correct breadcrumb. Worth a tracking issue if there isn't one.
  • Doc drift: the advisory bot already flagged that db/actordelivery/sqlc/CLAUDE.md / AGENTS.md don't list the two new PostponeMailboxMessage* queries. Advisory-only, but trivial to apply while here.

Summary

Solid PR. The only substantive item is the Codex finding, and I'd land it slightly differently than proposed: it's a real fence asymmetry worth closing, but it's P2 (needs cross-process contention and a write-then-postpone behavior, neither of which any adopter here does), and the fix must roll back without invoking the unfenced dead-letter nack — returning ErrLeaseExpired as-is trades one narrow hazard for a worse one at the max-attempts boundary.
• actor-postpone-semantics

@Roasbeef

Roasbeef commented Aug 8, 2026

Copy link
Copy Markdown
Member Author

Review items addressed in 8d90e52 and 56aa61b:

P2 (lost lease mid-turn): fixed to your design constraint. A new errLostLeaseMidTurn sentinel flows from both the postpone and nack zero-row branches of handleResultInTx; processInTransaction treats it as roll-back-and-return with no nack, no dead-letter, and no MarkProcessed, so the legitimate owner's turn governs the row and the stale consumer's writes vanish with the rollback. The zero-row check is fenced-only (LeaseToken != ""): an unfenced by-ID release matching nothing is a benign already-gone duplicate, not a lost lease. The regression test steals the lease mid-turn with MaxAttempts 1 and was verified against both halves: removing the zero-row check commits the stale writes, and removing the sentinel guard reproduces exactly the hazard you predicted (the unfenced dead-letter arm deletes the row its legitimate owner holds). The nack half of the fix predates this PR and is noted as part of #1124's family.

Doc drift: fixed. The sqlc guide pair now lists both postpone queries with the fenced/by-ID attempts distinction.

Tracking issue: filed as #1125 (keyed-lane FIFO inversion at the max-attempts boundary, with the lease-liveness disjunct as the suggested fix).

CI re-running on the new head.

@litbot-9000

Copy link
Copy Markdown
Collaborator

@Roasbeef, remember to re-request review from reviewers when ready

1 similar comment
@litbot-9000

Copy link
Copy Markdown
Collaborator

@Roasbeef, remember to re-request review from reviewers when ready

@bhandras
bhandras force-pushed the actor-postpone-semantics branch from 56aa61b to 21ad7af Compare August 29, 2026 07:36
@bhandras

Copy link
Copy Markdown
Member

/gateway review

Review target: #1122 at head 21ad7af, rebased onto main e5a89c8.

Purpose: verify that actor.Postpone(delay) lets valid Tell work wait without consuming attempts, while never committing stale transactional writes, losing durable work, violating lease ownership, or leaving an unbounded retry storm.

Since the prior review, the branch:

  • added a dedicated lost-lease sentinel so zero-row fenced postpone and nack roll back without entering the unfenced dead-letter path;
  • added a regression test that steals a final-attempt lease and proves stale writes roll back without deleting the legitimate owner's row;
  • bounded OOR cap postponement by the durable enqueue time, then falls back to normal retry/dead-letter after ten minutes;
  • covered resume-session and self-transfer Tell paths;
  • updated the SQLC docs;
  • rebased all 15 commits onto current main without dropping current actor retry-policy behavior.

Please independently trace:

  1. leased versus leaseless attempt accounting;
  2. Tell versus Ask handling;
  3. mark-processed and dead-letter behavior;
  4. lost-lease transaction rollback;
  5. OOR cap horizon and self-transfer recovery;
  6. current single-worker/keyed-lane constraints.

For each finding, give exact preconditions, a concrete reachable trace, consequence, likelihood, existing guards/recovery, whether this PR introduced or worsened it, the smallest failing test, and fix regression risk. Classify it as blocker, follow-up, or non-actionable. Do not re-raise prior findings without new evidence. #1125 deliberately tracks the keyed-lane plus worker-pool final-attempt caveat because no current adopter combines those conditions.

Validation is listed in the PR body. All substantive GitHub CI jobs are green. End with exactly one verdict: BLOCKED, READY WITH FOLLOW-UPS, or READY.

@lightninglabs-gateway

lightninglabs-gateway Bot commented Aug 29, 2026 •

Copy link
Copy Markdown

❌ gateway review failed: code-review skill failed across all auth modes: You've hit your weekly limit · resets 6am (UTC)

See workflow logs for details.

@bhandras

Copy link
Copy Markdown
Member

/gateway review

@lightninglabs-gateway lightninglabs-gateway Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gateway review — 3 findings

🔴 0 Blocker · 🟠 1 Major · 🟡 2 Minor · 🔵 0 Nit

Summary

The PR adds an attempt-preserving Postpone release to the durable actor runtime and adopts it in the OOR registry for over-cap admissions and deferred self-transfer hints. The core mechanics are right: the fenced/unfenced split (PostponeMailboxMessage decrements to compensate the lease-time bump, PostponeMailboxMessageByID leaves attempts alone because the peek never bumped it) leaves the retry budget byte-identical either way, detection runs before TellRetryPolicy on all three execution paths, shouldMarkProcessed is correctly suppressed so a postponed row is not dedup-skipped, and Ask/DurableAsk are structurally excluded because their branches precede the Tell branch in both handleResult and handleResultInTx. The errLostLeaseMidTurn sentinel is the right shape — rolling back without entering Delivery.Nack's unfenced dead-letter arm is exactly the distinction that matters at the final attempt, and the regression test steals the lease with MaxAttempts = 1 so the assertion is load-bearing rather than decorative.

The over-cap horizon is the strongest part of the change: deriving the bound from the durable row's created_at via DeliveryEnqueuedAt rather than a map keyed on an operator-chosen session id is correct for an attacker-controlled stream, since the bookkeeping would otherwise be the resource the cap defends. Treating "no timestamp" as "no horizon information" rather than "infinitely old" is the right failure-open choice.

Three concerns. The blocking one is documentary, not behavioral: the new invariant text asserts a tx-path dead-letter bug that the code on main already guards against, and that claim is replicated across four files and used to justify a tracking issue. The other two are narrow gaps in the PR's own "bound your own horizon" and "the budget is exactly what it was" invariants.

Bot commands
  • /gateway re-review — re-run after pushing changes (maintainers)
  • /gateway dismiss <id> — silence a finding (maintainers)
  • /gateway explain <id> — elaborate on a finding (anyone)

Comment thread baselib/actor/AGENTS.md Resolved
Comment thread oor/registry.go Resolved
Comment thread baselib/actor/durable_actor.go Resolved
@lightninglabs-gateway

lightninglabs-gateway Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

🤖 gateway audit metadata for this PR — auto-generated, please don't edit.

In this commit, we add the two release queries behind the new postpone
semantics. PostponeMailboxMessage is the fenced variant: the leased
claim pre-incremented attempts, so the decrement here restores the
retry budget to exactly what it was before the delivery, with a CASE
clamp so a corrupt row can never wrap negative.
PostponeMailboxMessageByID is the leaseless counterpart and leaves
attempts untouched, because the peek never incremented them.
In this commit, we regenerate the actor delivery query layer via make
sqlc to pick up the two postpone queries. No handwritten changes.
In this commit, we give behaviors a way to say "not now" without
walking a message toward the dead-letter table. A nack burns one of the
message's finite delivery attempts, so a consumer that is merely
waiting on an external condition (a capacity slot, a peer draining, an
operator catching up) dead-letters an innocent message one redelivery
at a time. Returning actor.Postpone(delay) instead releases the message
for redelivery after the delay with its retry budget fully intact.

The consume path detects the typed PostponeError before the Tell retry
policy on both execution paths (the tx fold and the non-tx tail), never
marks a postponed message processed (so the redelivery is not
dedup-skipped), and logs at debug level since a postpone is control
flow, not a failure. Only Tell deliveries honor it: an Ask has a caller
parked on the promise, so a postponed Ask would strand that caller; an
Ask behavior returning the error gets ordinary error treatment instead.

DeliveryStore gains the fenced/unfenced pair (PostponeMessage /
PostponeMessageByID) mirroring the ack/nack shape: the fenced variant
decrements attempts to compensate the lease-time bump, the by-ID
variant leaves them untouched because the leaseless peek never bumped
them. The flip side is deliberate and documented: a postponed message
never dead-letters by attempts, so behaviors must bound their own
postpone horizon.
In this commit, we stop burning a routed hint's finite delivery attempts
for a condition it did nothing to cause. When an incoming admission
arrives past MaxConcurrentIncomingSessions, the registry used to fail
the turn, which nacks the inbound durable message and walks it one
redelivery closer to the dead-letter table. Being over the cap is a
not-now condition: it clears the moment a resident session terminates
and is reaped, so the hint that happened to arrive while the daemon was
full should wait, not die.

The over-cap rejection on the routed-message path now returns
actor.Postpone(incomingCapBackoff) wrapped alongside the existing
errIncomingAdmissionCapped sentinel, so the consume path releases the
delivery on a five second backoff with its attempt budget fully intact,
while boot restore's skip check and the RPC surface keep matching the
sentinel. We apply this only where the admission is driven by a Tell:
handleResolveIncoming (the hint the event router pushes) and
handleDriveEvent's lazy restore of an already-admitted session. An
Ask-driven admission has a caller parked on the promise and gets
ordinary error treatment for a postpone anyway, and the incoming cap
never applies to StartTransferRequest, which is outgoing.

Boot restore keeps its own treatment: it already skips an over-cap row
rather than aborting the boot, and it consults ensureChild directly,
which still returns the bare sentinel.
In this commit, we retire the custom TellRetryPolicy that kept a
deferred self-transfer hint alive, because postpone now expresses what
that policy was reaching for and expresses it correctly. The policy
answered "retry after 30 seconds, always" for errSelfTransferDeferred so
the hint would never dead-letter while its outgoing session ran, and
delegated every other error to the default policy.

The problem is that a nack-with-retry increments attempts on every
release, and the claim and peek queries both filter on attempts <
max_attempts. So after ten deferrals (five minutes on the flat backoff)
the row fell out of the eligible set entirely: it never dead-lettered,
which is what the policy promised, but it also stopped redelivering,
which silently retired the crash-safety net the durable copy exists to
be. Returning actor.Postpone(selfHintRedeliveryBackoff) from the defer
branch gives the same 30 second flat wait with attempts untouched, so
the hint stays claim-eligible for as long as the outgoing session takes.

With the defer branch postponing, the policy override has nothing left
to special-case, so the registry goes back to the default Tell retry
policy and real errors keep their exponential backoff and dead-letter
boundary.
In this commit, we write down the postpone contract so the next behavior
that wants to say "not now" reaches for it instead of inventing another
never-dead-letter retry policy. The new Postpone Semantics section in
the durable actor architecture doc sets out why a nack is the wrong tool
for a transient condition, tabulates the one dimension where the two
paths differ (the attempt budget), and explains the fenced and leaseless
store mechanics: the leased variant decrements to compensate the
claim-time bump, the by-id variant leaves attempts alone because the
peek never bumped them.

Two consequences get their own treatment because they are easy to miss.
Postpone is Tell-only, since an Ask has a caller parked on the promise
that a delay would strand. And a postponed message never climbs toward
max_attempts, which means nothing dead-letters it automatically: the
framework has removed the only mechanism that would eventually give up,
so a behavior that postpones has to bound its own horizon or accept that
the message waits forever. We also spell out that a postponed head still
blocks its correlation-key lane, since the row is very much still in the
mailbox, and close with the OOR over-cap admission and the deferred
self-transfer hint as the first two adopters.

The per-package guides pick up the same material at the altitude each
one works at: baselib/actor gains the Postpone key type and four
invariants, db/actordelivery gains the two queries and the two store
methods plus why their attempt handling differs, and oor gains the
over-cap postpone invariant and the story of the retry policy it
replaced.
In this commit, we give a postponing behavior the reference it needs to
bound its own horizon. Postpone deliberately removes the attempt-based
give-up mechanism, so the contract asks every adopter to decide for
itself when waiting has stopped making sense. That obligation is hard to
discharge without a trustworthy notion of how long the message has
already been waiting.

The mailbox row already knows. Its created_at is set once at enqueue and
no release path rewrites it: a nack moves available_at and bumps
attempts, a postpone moves available_at and restores attempts, and
neither touches the creation time. Delivery gains an EnqueuedAt field
copied from the leased row, and the consume path stamps it onto the
processing context via DeliveryEnqueuedAt. The stamp happens in
processDelivery, above the fork into the three execution paths, so the
tx fold, the non-tx tail, and the Read/Commit handle all agree.

Deriving the age from the row rather than from behavior-side state is
the point, not an implementation detail. A behavior that tracked waits
in a map would key that map on something the sender chose, and for an
attacker-controlled message stream the bookkeeping meant to protect the
actor becomes the resource the attacker grows. The row's own age costs
nothing to consult and cannot be inflated by fabricating messages.

The accessor returns a bool alongside the timestamp so absence stays
distinguishable from a zero time. A store that reports no creation time,
or a behavior invoked directly in a test, means "no horizon
information", which is not the same as "infinitely old", and a behavior
that conflated the two would give up on every message immediately.
In this commit, we close a coverage gap and a small silence in the
transaction path's postpone handling. handleResultInTx is a separate
implementation from the non-tx tail, and until now only the latter had a
test, so a regression in the tx branch would have passed the suite
untouched. The new test drives a classic behavior on a tx-aware store
through repeated postpones under a retry policy that fails the test if
it is ever consulted for a postpone, and asserts the message survives
past the point a nack would have dead-lettered it.

We also stop discarding the row count from the tx-path postpone. A zero
row count means the release did not happen, which on the leased path
means the lease expired or was claimed by another consumer, so the
attempt this delivery took stays uncompensated and the message
redelivers on the lease-expiry path rather than the requested delay. The
nack path treats a zero-row release the same way, so we keep the
behavior identical and only add a debug line: a postpone that quietly
did not happen is otherwise invisible, because the postpone path logs at
debug and never warns.
In this commit, we make the over-cap adopter live up to the rule the
postpone contract sets for every adopter: bound your own horizon. As
written, a capped hint postponed forever, which is exactly the failure
mode the contract warns about, and it mattered here more than most
because the hint stream is operator-controlled.

Every redelivery of a capped hint re-runs validateIncomingAdmission, a
wallet-ownership query against the database, and resolveSelfTransfer, a
registry-row read, before the cap check rejects it again. Both run
before the cap is consulted, so the work happens on every cycle no
matter how long the daemon has been full. An operator streaming
fabricated session ids could therefore build a churn queue against the
single-worker registry that no amount of waiting drains, with each entry
renewing itself every five seconds and none of them ever reaching a
horizon.

postponeOverCap now postpones only while the delivery is younger than
overCapPostponeHorizon, ten minutes, measured against the durable row's
enqueue time. Past the horizon it returns the plain capped sentinel, so
the ordinary nack path takes over and the hint dead-letters into a table
where it is visible and requeue-able rather than churning invisibly. Ten
minutes outlasts any realistic transient burst (the cap defaults to 1024
resident sessions and a slot frees on every terminal reap) while
converting a hostile backlog on a human timescale.

The age comes from actor.DeliveryEnqueuedAt rather than from a
per-session map, because the session ids in that stream are
operator-chosen and the map would be the very unbounded resource this is
defending. A delivery that reports no enqueue time postpones as before,
since absence means we have no horizon information rather than that the
message is infinitely old.

We also correct the backoff constant's comment. Nothing signals the
mailbox wake channel when a postponed row becomes eligible, so it is
rediscovered by the idle poll backoff and the effective redelivery gap
is roughly five to thirty five seconds, not the five the old wording
implied.
In this commit, we close an asymmetry in how the registry treats the
concurrency cap. handleDriveEvent postpones when lookupOrRestore cannot
make a session resident, but handleResumeSession returned the bare
sentinel from the identical call, so the two Tell-driven paths gave the
same condition opposite treatment.

A resume is not a lesser delivery. It arrives as a Tell from the timeout
retry callback and carries a real timer expiry that the session needs in
order to re-drive its outbox, so nacking it for a transient cap spends
its attempts and eventually dead-letters work that was never wrong.
Applying the same wrap makes every Tell-driven path that consults
lookupOrRestore agree, and the horizon added alongside it keeps the wait
bounded here too.
In this commit, we document a boundary case in the claim query's
correlation-key anti-join that only postpone can turn into a real
ordering bug, and we deliberately leave the SQL alone.

The anti-join passes over a predecessor that has exhausted its retry
budget, so a dead row cannot wedge its lane forever. A predecessor
leased on its final attempt already satisfies attempts == max_attempts
after the claim pre-increment, which means it is invisible to the
anti-join while it is still being processed and a competing worker may
claim its same-key successor. With only ack and nack available that is
harmless, because the predecessor can then only complete or dead-letter.
A postpone breaks it: the release decrements attempts back below the
cap, so the predecessor becomes eligible again and reprocesses after the
successor already ran, inverting per-key FIFO.

Reaching that state needs all three of keyed correlation lanes, more
than one worker, and a postponing behavior, and no adopter combines them
today. Complicating the hot claim query for a configuration nothing runs
would be the wrong trade, so we record the exclusion where the next
person will meet it, next to the predicate itself, along with the
intended repair: a lease-liveness disjunct so a currently-leased
predecessor blocks its successors regardless of its attempts. The sqlc
stubs are regenerated only to carry the comment through.
In this commit, we make the in-memory checkpoint store's PostponeMessage
validate the lease token, which its own NackMessage already does and the
real SQL query has always done. Without the check the mock accepted a
postpone from a stale consumer against a row another consumer now owns,
decremented that row's attempts, and reported success where the store
would have returned zero rows.

Nothing depends on the gap today, but a mock that is more permissive
than the thing it stands in for is a test that cannot fail for the right
reason, and the fence is precisely what the postpone path relies on to
keep the attempt budget honest.
In this commit, we correct the documented retry boundary. We also scope
the attempt-neutral, horizon, and FIFO guarantees to where they hold.

Both non-transaction and classic transaction paths check
ShouldDeadLetter before retry release. A TellRetryPolicy can stop
retries early or choose the delay. It cannot extend MaxAttempts. The old
OOR self-hint policy therefore dead-lettered at exhaustion on its
Read/Commit path. There is no separate go-dark retry-policy asymmetry.

A successful postpone preserves the attempt budget. A classic
transaction turn that loses its lease is the exception. Rolling the turn
back also discards the compensating decrement. The earlier lease claim's
increment remains committed. The lost turn spends one attempt. No
current classic behavior returns a postpone, so we document the
restriction instead of adding cross-transaction compensation to this PR.

The self-transfer adopter deliberately has no time horizon. The hint
names the wallet's own change output. Discarding it while the outgoing
session is pending is worse than retaining it. Terminal reap redrives it
immediately. The durable 30 second copy covers crashes and missed
redrives. The docs now say that operators should watch old non-terminal
sessions and inspect the registry mailbox when one is stuck.

The correlation-key section also scopes per-key FIFO for postponing
consumers. It holds when the actor is single-worker or the lane's
messages never reach their final attempt. We explain the anti-join
boundary behind that restriction. The lease-liveness disjunct remains
the prerequisite for lifting it.

Finally, we retain the row-derived DeliveryEnqueuedAt guidance. We also
retain the OOR over-cap adopter's ten minute horizon. Those bounded
paths still fail into the normal visible dead-letter flow after their
wait expires.
In this commit, we stop the transaction path from committing a
behavior's writes under a row it no longer owns. The postpone and nack
branches of handleResultInTx both discarded the row count from their
fenced release, so a release that matched nothing (the lease expired
mid-turn and another consumer claimed the message) looked exactly like
success. The closure returned nil, the transaction committed, and one
consumer's state changes landed for a message a different consumer was
already reprocessing.

Both branches now interpret the count through releasedRowsInTx, which
distinguishes the two ways a zero can arise. An unfenced by-id release
matching nothing is benign, it just means the row is already gone. A
fenced release matching nothing means the lease-token comparison failed,
so the message has changed hands, and that returns errLostLeaseMidTurn
to roll the transaction back.

processInTransaction then treats that sentinel as roll-back-and-return:
no nack, no dead-letter, no processed mark. Skipping the nack is the
load-bearing half rather than a tidiness choice. Delivery.Nack
dead-letters once the attempt budget is spent, and that arm is unfenced,
running MoveToDeadLetter and DeleteMessage by id with no lease check, so
routing a stale consumer through it would let it delete a message the
legitimate owner is in the middle of processing. Rolling back and
walking away leaves the row and its outcome entirely to that owner.

The nack half of this predates the postpone work: that branch has
ignored its row count since it was written, and it shares the fence
asymmetry described in issue #1124. We fix both together because they
are the same mistake in adjacent branches and a partial fix would invite
the next reader to assume the other one was deliberate.

The regression test steals the lease from inside the behavior and
asserts the turn's writes never commit, the row is neither dead-lettered
nor deleted, it keeps the new owner's token, and it is not marked
processed. The mock's ExecTx grows real rollback semantics for a
behavior-visible write log, since without that a test cannot tell a
committed turn from a rolled-back one. The non-transaction path is
untouched: Delivery.Postpone already surfaces ErrLeaseExpired there.
In this commit, we add the two postpone queries to the generated query
layer's own package guide, which still described the mailbox surface as
enqueue, lease, peek, ack/nack, extend, and expire. Someone reading that
guide to find out what the layer exposes would have concluded postpone
did not exist there.

The entry spells out why the fenced and by-id variants treat attempts
differently, since that asymmetry is the whole point of the pair and is
easy to read as an inconsistency: the fenced query decrements to
compensate the increment the leased claim applied, and the by-id query
leaves attempts alone because the leaseless peek never applied one.
@bhandras

Copy link
Copy Markdown
Member

/gateway re-review

Re-review target: 9ec18f5.

All three findings were addressed in folded docs commit 337e754:

  • F1: corrected the false tx-path go-dark claim across both actor doc pairs, the architecture guide, the PR body, and closed issue actor: tx-path retry policy can strand a message past max_attempts with no dead letter #1124;
  • F2: documented the deliberate no-horizon self-transfer exemption, terminal-reap recovery, durable fallback, and operator signal;
  • F3: scoped attempt neutrality to successful owned-lease postpones and documented that lost-lease rollback spends the separately committed claim attempt.

No runtime code changed from the prior reviewed head. The three threads are replied to and resolved. All CI checks are green. Please verify only whether the corrections fully address F1-F3 or expose a concrete material defect. Apply the stopping rule once those findings are closed.

@lightninglabs-gateway lightninglabs-gateway Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gateway review

No findings.


Status of prior findings

  • F1 addressed: Fixed in baselib/actor/AGENTS.md / CLAUDE.md (both now: "Both the non-tx path ... and the classic tx path (handleResultInTx) enforce ShouldDeadLetter before retry release"), docs/durable_actor_architecture.md (transaction-path bullet now states the retry branch checks ShouldDeadLetter and falls through to MarkProcessed + MoveToDeadLetter), and the OOR adopter paragraph (now "the tenth deferral dead-lettered the hint regardless of what the policy answered"). No "go dark" / never-reaches-dead_letters claim survives anywhere in the touched docs, the conclusion the passage supports is unchanged and still correct, and the stale premise in #1124 was corrected so nobody acts on it.
  • F2 addressed: Fixed in oor/AGENTS.md / CLAUDE.md and docs/durable_actor_architecture.md — the exemption is now explicit rather than silent: the hint names the wallet's own change output so discarding it is worse than retaining it, terminal reap is the immediate redrive, the durable 30s copy covers a crash or missed redrive, and the operator signal is named (watch old non-terminal outgoing sessions, inspect the registry mailbox, the hint will not appear in dead_letters). That was one of the two options the finding offered, and it is the right one here.
  • F3 addressed: Fixed in baselib/actor/AGENTS.md / CLAUDE.md ("Fenced postpone is attempt-neutral only while the consumer still owns the lease") and docs/durable_actor_architecture.md (the fenced-path exception paragraph). The rollback-discards-the-decrement-while-the-claim-increment-stays-committed mechanism, the one-attempt cost per lost turn, and the exhaustion-without-a-dead-letter consequence are all stated, along with the scope that closes it in practice: no current classic-path behavior returns a postpone. Deferring cross-transaction compensation for a latent path rather than adding it speculatively is the correct trade.
Bot commands
  • /gateway re-review — re-run after pushing changes (maintainers)
  • /gateway dismiss <id> — silence a finding (maintainers)
  • /gateway explain <id> — elaborate on a finding (anyone)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants