Repository navigation
fix(runtime): re-dispatch stale Cancelling instances - #2
Open
JohnyKid123 wants to merge 1 commit into
Open
JohnyKid123 wants to merge 1 commit into
JohnyKid123 wants to merge 1 commit into
Conversation
A crash after the Suspended Cancelling mark and before enqueue can leave the row Cancelling. Scan running, non-executing Cancelling rows older than CancellingRecoveryThreshold, claim them by refreshing UpdatedAt through TryTransitionSubStatusAsync, and re-dispatch cancellation. Interrupted recovery is unchanged. Co-authored-by: Alex Venkov <JohnyKid123@users.noreply.github.com>
|
@greptileai please review this recovery-only follow-up to elsa-workflows#9115. Phase A mark scope is unchanged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Re-dispatch cancellation for workflow instances left in
Cancellingafter a crash or a lost enqueue. This is the Phase B follow-up to elsa-workflows#9115 and does not change Phase A's mark scope.Scope
Select one primary concern:
Description
Problem
Phase A (elsa-workflows#9115, for elsa-workflows#5187) marks a suspended instance
Cancellingbefore dispatch. A crash after that mark and before the cancellation request is enqueued leaves the rowRunning/Cancelling. Nothing on main looks at that row again.InterruptedRecoveryScanneronly restartsInterruptedinstances.Solution
This PR is stacked on
fix/workflow-cancellation-5187-phase-aso the review diff is recovery only. It targets the 3.11 milestone, same as elsa-workflows#9115. Phase A stays as it is.CancellingRecoveryScannerselects running, non-executing instances inCancellingwhoseUpdatedAtis older thanRuntimeOptions.CancellingRecoveryThreshold(default 5 minutes, same order of magnitude asInactivityThreshold).TimeSpan.Zerodisables the scan. Pages useCancellingRecoveryBatchSize(default 100).TryTransitionSubStatusAsync: expectedCancellingplus the storedUpdatedAt, nextCancelling, new stamp from the clock. A concurrentCancelledcommit or restore makes the compare-and-swap fail, and that row is not re-dispatched.Cancelling→Cancellingis now a permitted transition so the claim can refresh the stamp. EnteringCancellingis stillSuspendedonly. Pending and Interrupted stay unmarked.IWorkflowCancellationDispatcherwith the instance id only. Cancellation is idempotent. The request omitsPreviousSubStatusandCancellingAt, so an observed failure of this re-dispatch leavesCancellingin place for the next scan. The new stamp also makes an older restore compare-and-swap miss the row.RecoverCancellingWorkflowsStartupTask) and every 5 minutes (RecoverCancellingWorkflowsTask), both single-node, registered on the runtime feature and the shell feature.InterruptedRecoveryScannerandRestartInterruptedWorkflowsTaskare unchanged.Depends on elsa-workflows#9115. Related to elsa-workflows#5187.
Verification
net10.0:dotnet test core/test/unit/Elsa.Workflows.Runtime.UnitTests/Elsa.Workflows.Runtime.UnitTests.csproj --filter FullyQualifiedName~CancellingRecoveryScannerTests— 6 passeddotnet test core/test/unit/Elsa.Workflows.Management.UnitTests/Elsa.Workflows.Management.UnitTests.csproj --filter FullyQualifiedName~MemoryWorkflowInstanceStoreTests— 20 passedSteps:
CancellingwithUpdatedAtolder than the threshold. The scanner claims it (stamp moves, sub-status staysCancelling) and dispatches cancellation without restore metadata.Cancelled, or restore it before the claim. The compare-and-swap fails and cancellation is not re-dispatched.Cancellingrow newer than the threshold, an executingCancellingrow, and Suspended, Pending, and Interrupted rows are left alone.Expected outcome: only stale, idle
Cancellingrows are claimed and re-dispatched. Phase A still marks only Suspended instances.Screenshots / Recordings (if applicable)
No UI changes.
Checklist
AI assistance
AI assistance was used for implementation and tests. The recovery claim is a
Cancellingstamp refresh on the existingTryTransitionSubStatusAsyncprimitive from elsa-workflows#9115.