Skip to content

fix(runtime): re-dispatch stale Cancelling instances - #2

Open
JohnyKid123 wants to merge 1 commit into
fix/workflow-cancellation-5187-phase-afrom
fix/workflow-cancellation-cancelling-recovery
Open

JohnyKid123 wants to merge 1 commit into
fix/workflow-cancellation-5187-phase-afrom
fix/workflow-cancellation-cancelling-recovery

Conversation

@JohnyKid123

Copy link
Copy Markdown
Owner

Purpose

Re-dispatch cancellation for workflow instances left in Cancelling after a crash or a lost enqueue. This is the Phase B follow-up to elsa-workflows#9115 and does not change Phase A's mark scope.


Scope

Select one primary concern:

  • Bug fix (behavior change)
  • Refactor (no behavior change)
  • Documentation update
  • Formatting / code cleanup
  • Dependency / build update
  • New feature

Description

Problem

Phase A (elsa-workflows#9115, for elsa-workflows#5187) marks a suspended instance Cancelling before dispatch. A crash after that mark and before the cancellation request is enqueued leaves the row Running / Cancelling. Nothing on main looks at that row again. InterruptedRecoveryScanner only restarts Interrupted instances.

Solution

This PR is stacked on fix/workflow-cancellation-5187-phase-a so the review diff is recovery only. It targets the 3.11 milestone, same as elsa-workflows#9115. Phase A stays as it is.

  • CancellingRecoveryScanner selects running, non-executing instances in Cancelling whose UpdatedAt is older than RuntimeOptions.CancellingRecoveryThreshold (default 5 minutes, same order of magnitude as InactivityThreshold). TimeSpan.Zero disables the scan. Pages use CancellingRecoveryBatchSize (default 100).
  • The scan reads the matching rows first, then claims each one with TryTransitionSubStatusAsync: expected Cancelling plus the stored UpdatedAt, next Cancelling, new stamp from the clock. A concurrent Cancelled commit or restore makes the compare-and-swap fail, and that row is not re-dispatched. Cancelling → Cancelling is now a permitted transition so the claim can refresh the stamp. Entering Cancelling is still Suspended only. Pending and Interrupted stay unmarked.
  • After a successful claim, the scan calls IWorkflowCancellationDispatcher with the instance id only. Cancellation is idempotent. The request omits PreviousSubStatus and CancellingAt, so an observed failure of this re-dispatch leaves Cancelling in place for the next scan. The new stamp also makes an older restore compare-and-swap miss the row.
  • The scan runs at startup (RecoverCancellingWorkflowsStartupTask) and every 5 minutes (RecoverCancellingWorkflowsTask), both single-node, registered on the runtime feature and the shell feature. InterruptedRecoveryScanner and RestartInterruptedWorkflowsTask are unchanged.
  • No Studio UI.

Depends on elsa-workflows#9115. Related to elsa-workflows#5187.


Verification

net10.0:

  • dotnet test core/test/unit/Elsa.Workflows.Runtime.UnitTests/Elsa.Workflows.Runtime.UnitTests.csproj --filter FullyQualifiedName~CancellingRecoveryScannerTests — 6 passed
  • dotnet test core/test/unit/Elsa.Workflows.Management.UnitTests/Elsa.Workflows.Management.UnitTests.csproj --filter FullyQualifiedName~MemoryWorkflowInstanceStoreTests — 20 passed

Steps:

  1. Leave a running, non-executing instance in Cancelling with UpdatedAt older than the threshold. The scanner claims it (stamp moves, sub-status stays Cancelling) and dispatches cancellation without restore metadata.
  2. Move the stamp, finish the instance as Cancelled, or restore it before the claim. The compare-and-swap fails and cancellation is not re-dispatched.
  3. A Cancelling row newer than the threshold, an executing Cancelling row, and Suspended, Pending, and Interrupted rows are left alone.

Expected outcome: only stale, idle Cancelling rows are claimed and re-dispatched. Phase A still marks only Suspended instances.


Screenshots / Recordings (if applicable)

No UI changes.


Checklist

  • The PR is focused on a single concern
  • Commit messages follow the recommended convention
  • Tests added or updated (if applicable)
  • Documentation updated (if applicable)
  • No unrelated cleanup included
  • All tests pass — the targeted unit tests above. Full CI is the remaining check.

AI assistance

AI assistance was used for implementation and tests. The recovery claim is a Cancelling stamp refresh on the existing TryTransitionSubStatusAsync primitive from elsa-workflows#9115.

Open in Web Open in Cursor 

A crash after the Suspended Cancelling mark and before enqueue can leave
the row Cancelling. Scan running, non-executing Cancelling rows older than
CancellingRecoveryThreshold, claim them by refreshing UpdatedAt through
TryTransitionSubStatusAsync, and re-dispatch cancellation. Interrupted
recovery is unchanged.

Co-authored-by: Alex Venkov <JohnyKid123@users.noreply.github.com>
@cursor

cursor Bot commented Oct 27, 2026

Copy link
Copy Markdown

@greptileai please review this recovery-only follow-up to elsa-workflows#9115. Phase A mark scope is unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants