Skip to content

Isolate approval expiry follow-ups; skip orphaned suspended children - #2066

Merged
ppXD merged 1 commit into
mainfrom
fix/isolate-expiry-followups-and-orphaned-children
Sep 30, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/isolate-expiry-followups-and-orphaned-children

Conversation

@ppXD

@ppXD ppXD commented Sep 30, 2026

Copy link
Copy Markdown
Owner

Summary

  • Approval expiry sweep: each expired row's wake and card mirror now catch and log their own failure against the ledger id, as the decision sweep's do. Under the recurring job's transactional command the post-commit drain already kept a failure to its own row, but logged it without naming the row, and a wake that threw skipped that row's mirror. With no transaction open (ad-hoc, or were the command ever made non-transactional) one card's failed save ended the follow-ups of every later row, and no tick selects an Expired row again. MarkTimedOutAsync already saves through SaveMirrorOrForgetAsync, so a failed card save leaves nothing tracked. A shutdown's cancellation still stops the sweep.
  • ExpireStaleToolApprovalsCommand is still transactional (allow-listed as atomic by design in RecurringJobTransactionInventoryTests, unlike ExpireStaleDecisionsCommand); left as is, that being a separate decision. A fault in the ledger CAS therefore still rolls the whole tick back and every row stays for the next one; a failed mirror leaves its row Expired and its card Open, where a click is refused.
  • Stranded-Suspended sweep: a sub-workflow child whose parent has finished is no longer revived, the guard RedispatchStuckPendingAsync already has. It is keyed on SourceType (a rerun's ParentRunId is finished lineage) and reuses TerminalRunStatuses. The exclusion is in the query so a child left Suspended cannot take a place in the batch every tick, and the sweep logs at Information how many it left alone. Those children stay Suspended; nothing here cancels them.

Test plan

  • Unit ToolApprovalExpiryServiceTests, 4 new: a failed mirror on the second of three cards leaves the first and third woken and mirrored and the count returned; a failed wake still mirrors its card; an approval with no card is woken; a shutdown's cancellation stops the sweep. Mutation: removing the per-step catches turned 2 red; dropping the cancellation filter turned the shutdown case red.
  • Integration ToolApprovalExpiryServiceTests, 2 new cases of one theory (service called with no transaction, and through the mediator command), real Postgres with a fault injected into one card's UPDATE message: rows Expired, all calls woken, neighbouring cards mirrored, the failed card Open, a warning naming its ledger id, nothing tracked. 5/5 with the 3 existing. Mutations: removing the per-step catches turned both red (inline the sweep throws; through the command no warning names the ledger id); replacing SaveMirrorOrForgetAsync with a plain save in MarkTimedOutAsync turned both red.
  • Integration StuckRunReconcilerFlowTests, 7 new: a Suspended child with no pending wait is not re-dispatched under a Cancelled, Failure or Success parent, is under a Running or Suspended parent, and a rerun with a finished ParentRunId still is; the Information line carries the count. 41/41. Mutations: dropping the guard turned the 3 finished-parent cases red; keying it on ParentRunId alone turned the rerun case red; dropping the log turned the log case red.
  • Regression: McpToolApprovalFlowTests 19, DecisionReaperFlowTests 13, SubworkflowCancelFlowTests 8, AgentRunRecoveryFlowTests 23, MessageRespondFlowTests 18, the other reconciler flow classes, both transaction inventories, and the full unit project (11624 passed).

The approval sweep wakes each expired row's blocked call and mirrors its
card, neither step guarded. Under the job's transactional command the
post-commit drain already kept a failure to its own row, but logged it
without the ledger id, and a wake that threw skipped the same row's
mirror. With no transaction open, one card's failed save ended the
follow-ups of every later row: their calls stayed unwoken and their
cards Open, and no later tick selects an Expired row again. Each step
now catches and logs its own failure against the ledger id, as the
decision sweep's do, and a shutdown's cancellation still stops the
sweep. The command stays transactional.

The stranded-Suspended sweep revived a sub-workflow child whose parent
had finished (its cancel failed in the parent's teardown, then its waits
were closed), so it ran for a parent no one was waiting on. It now
leaves such children alone, keyed on source type because ParentRunId is
also a rerun's lineage, and logs how many it left. The exclusion is in
the query so a child left Suspended cannot take a place in the batch
every tick and crowd out the runs that are wanted.
@ppXD
ppXD merged commit 99529d1 into main Sep 30, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant