test(workflow): fix the RetiredStreamCannotOverwriteCurrentStreamCache race - #1924
Draft
nelson-parente wants to merge 1 commit into
Draft
nelson-parente wants to merge 1 commit into
nelson-parente wants to merge 1 commit into
Conversation
…eCurrentStreamCache The failing delta tears down the first stream, which cancels its token. When the retired turn's task has not started at that moment, Task.Run never runs it. Then retiredHandlerStarted is never set, and the current turn waits until the 20-second timeout. Deliver the failing delta only after the retired turn's handler has started. Signed-off-by: Nelson Parente <nelson_parente@live.com.pt> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017JhKUDV9dbs5JDz2uwZqz5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
GrpcProtocolHandlerStatefulHistoryTests.RetiredStreamCannotOverwriteCurrentStreamCachefails intermittently in theUnit Tests .NET 10.0 / test/Dapr.Workflow.Testjob ofsdk_build.yml. It fails withTaskCanceledExceptionafter its 20-second wait. Since 2026-09-29 that job ran in 12sdk_buildruns, and this test failed in 5 of them. The failed runs are master run 36749821550 and PR runs 36598831761, 36600468164, 36750059463 and 37230827195.The cause is a race in the test, not in
GrpcProtocolHandler:ReceiveLoopAsyncstarts each one withTask.Run(..., streamToken).ProcessWorkflowAsynccallsteardownStream(), which cancels the stream token.Task.Rundoes not start a delegate whose token is already cancelled, and_orchestrationSemaphore.WaitAsync(cancellationToken)throws as well.retiredHandlerStartedis never set, and the current turn's handler awaits it forever. The worker sends no completion, andRunHandlerUntilAsynctimes out after 20 seconds.The handler behavior is correct: the sidecar re-dispatches the in-flight items of a stream that disconnects. Only the test assumes that the retired turn always reaches its handler.
This PR gates the failing delta on
retiredHandlerStarted.Task, so the stream delivers it only when the retired turn's handler is running. The test then always checks its purpose. A turn that finishes after the handler retires its stream does not overwrite the new stream's history cache. The change is in the test only.Issue reference
No issue.
Checklist
Local checks (.NET SDK 10.0.401, macOS arm64)
DOTNET_PROCESSOR_COUNT=1andThreadPool.SetMaxThreads(1, 1)). With the original gating, the test failed in 3 of 3 runs with the sameTaskCanceledExceptionafter 20 seconds as CI. With the gate from this PR, it passed in 3 of 3 runs.Dapr.Workflow.Testruns (446 tests,DOTNET_PROCESSOR_COUNT4 and 2). One of those full runs had an unrelated failure inWorkflowTraceTests.StartActivityTrace_ShouldEmitActivitySourceSpan_WhenListenerIsRegistered.Dapr.Workflow.Testruns passed.🤖 Generated with Claude Code
https://claude.ai/code/session_017JhKUDV9dbs5JDz2uwZqz5