Skip to content

test(workflow): fix the RetiredStreamCannotOverwriteCurrentStreamCache race - #1924

Draft
nelson-parente wants to merge 1 commit into
dapr:masterfrom
nelson-parente:test/stateful-history-retired-stream-race
Draft

nelson-parente wants to merge 1 commit into
dapr:masterfrom
nelson-parente:test/stateful-history-retired-stream-race

Conversation

@nelson-parente

Copy link
Copy Markdown
Contributor

Description

GrpcProtocolHandlerStatefulHistoryTests.RetiredStreamCannotOverwriteCurrentStreamCache fails intermittently in the Unit Tests .NET 10.0 / test/Dapr.Workflow.Test job of sdk_build.yml. It fails with TaskCanceledException after its 20-second wait. Since 2026-09-29 that job ran in 12 sdk_build runs, and this test failed in 5 of them. The failed runs are master run 36749821550 and PR runs 36598831761, 36600468164, 36750059463 and 37230827195.

The cause is a race in the test, not in GrpcProtocolHandler:

  1. The first stream yields the retired turn and then, with no gate, the failing delta. ReceiveLoopAsync starts each one with Task.Run(..., streamToken).
  2. The failing delta's history fetch throws, so ProcessWorkflowAsync calls teardownStream(), which cancels the stream token.
  3. If the retired turn's task has not started at that moment, it never runs. Task.Run does not start a delegate whose token is already cancelled, and _orchestrationSemaphore.WaitAsync(cancellationToken) throws as well.
  4. Then retiredHandlerStarted is never set, and the current turn's handler awaits it forever. The worker sends no completion, and RunHandlerUntilAsync times out after 20 seconds.

The handler behavior is correct: the sidecar re-dispatches the in-flight items of a stream that disconnects. Only the test assumes that the retired turn always reaches its handler.

This PR gates the failing delta on retiredHandlerStarted.Task, so the stream delivers it only when the retired turn's handler is running. The test then always checks its purpose. A turn that finishes after the handler retires its stream does not overwrite the new stream's history cache. The change is in the test only.

Issue reference

No issue.

Checklist

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation (not applicable)

Local checks (.NET SDK 10.0.401, macOS arm64)

  • To force the race, a local-only probe (not in this PR) capped the thread pool at one worker inside the test (DOTNET_PROCESSOR_COUNT=1 and ThreadPool.SetMaxThreads(1, 1)). With the original gating, the test failed in 3 of 3 runs with the same TaskCanceledException after 20 seconds as CI. With the gate from this PR, it passed in 3 of 3 runs.
  • Without that cap, the race did not occur on this machine. On master, the test passed 20 of 20 single-test runs and 31 of 31 full Dapr.Workflow.Test runs (446 tests, DOTNET_PROCESSOR_COUNT 4 and 2). One of those full runs had an unrelated failure in WorkflowTraceTests.StartActivityTrace_ShouldEmitActivitySourceSpan_WhenListenerIsRegistered.
  • With this change, on net10.0: 20 of 20 single-test runs and 10 of 10 full Dapr.Workflow.Test runs passed.
  • The test project builds for net8.0, net9.0 and net10.0 with no warnings. Only the .NET 10 runtime was available locally, so the net8.0 and net9.0 tests did not run.

🤖 Generated with Claude Code

https://claude.ai/code/session_017JhKUDV9dbs5JDz2uwZqz5

…eCurrentStreamCache

The failing delta tears down the first stream, which cancels its token.
When the retired turn's task has not started at that moment, Task.Run
never runs it. Then retiredHandlerStarted is never set, and the current
turn waits until the 20-second timeout. Deliver the failing delta only
after the retired turn's handler has started.

Signed-off-by: Nelson Parente <nelson_parente@live.com.pt>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017JhKUDV9dbs5JDz2uwZqz5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant