Isolate approval expiry follow-ups; skip orphaned suspended children - #2066
Merged
Merged
Conversation
The approval sweep wakes each expired row's blocked call and mirrors its card, neither step guarded. Under the job's transactional command the post-commit drain already kept a failure to its own row, but logged it without the ledger id, and a wake that threw skipped the same row's mirror. With no transaction open, one card's failed save ended the follow-ups of every later row: their calls stayed unwoken and their cards Open, and no later tick selects an Expired row again. Each step now catches and logs its own failure against the ledger id, as the decision sweep's do, and a shutdown's cancellation still stops the sweep. The command stays transactional. The stranded-Suspended sweep revived a sub-workflow child whose parent had finished (its cancel failed in the parent's teardown, then its waits were closed), so it ran for a parent no one was waiting on. It now leaves such children alone, keyed on source type because ParentRunId is also a rerun's lineage, and logs how many it left. The exclusion is in the query so a child left Suspended cannot take a place in the batch every tick and crowd out the runs that are wanted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MarkTimedOutAsyncalready saves throughSaveMirrorOrForgetAsync, so a failed card save leaves nothing tracked. A shutdown's cancellation still stops the sweep.ExpireStaleToolApprovalsCommandis still transactional (allow-listed as atomic by design inRecurringJobTransactionInventoryTests, unlikeExpireStaleDecisionsCommand); left as is, that being a separate decision. A fault in the ledger CAS therefore still rolls the whole tick back and every row stays for the next one; a failed mirror leaves its row Expired and its card Open, where a click is refused.RedispatchStuckPendingAsyncalready has. It is keyed onSourceType(a rerun'sParentRunIdis finished lineage) and reusesTerminalRunStatuses. The exclusion is in the query so a child left Suspended cannot take a place in the batch every tick, and the sweep logs at Information how many it left alone. Those children stay Suspended; nothing here cancels them.Test plan
ToolApprovalExpiryServiceTests, 4 new: a failed mirror on the second of three cards leaves the first and third woken and mirrored and the count returned; a failed wake still mirrors its card; an approval with no card is woken; a shutdown's cancellation stops the sweep. Mutation: removing the per-step catches turned 2 red; dropping the cancellation filter turned the shutdown case red.ToolApprovalExpiryServiceTests, 2 new cases of one theory (service called with no transaction, and through the mediator command), real Postgres with a fault injected into one card'sUPDATE message: rows Expired, all calls woken, neighbouring cards mirrored, the failed card Open, a warning naming its ledger id, nothing tracked. 5/5 with the 3 existing. Mutations: removing the per-step catches turned both red (inline the sweep throws; through the command no warning names the ledger id); replacingSaveMirrorOrForgetAsyncwith a plain save inMarkTimedOutAsyncturned both red.StuckRunReconcilerFlowTests, 7 new: a Suspended child with no pending wait is not re-dispatched under a Cancelled, Failure or Success parent, is under a Running or Suspended parent, and a rerun with a finishedParentRunIdstill is; the Information line carries the count. 41/41. Mutations: dropping the guard turned the 3 finished-parent cases red; keying it onParentRunIdalone turned the rerun case red; dropping the log turned the log case red.McpToolApprovalFlowTests19,DecisionReaperFlowTests13,SubworkflowCancelFlowTests8,AgentRunRecoveryFlowTests23,MessageRespondFlowTests18, the other reconciler flow classes, both transaction inventories, and the full unit project (11624 passed).