Keep a heartbeat loop alive when its error reporter throws - #2017
Merged
ppXD merged 1 commit intoSep 23, 2026
Merged
Conversation
HeartbeatLoop.RunAsync handed a failed ping to the caller's onPingError outside the ping's guard. A reporter that threw therefore faulted the loop, and every caller awaits the loop in the finally around the work it protects: AgentRunExecutor's launch and re-attach heartbeats, and the three supervisor grading call sites. The reporter's exception then replaced the protected result, and in AgentRunExecutor it also skipped the credential revoke and workspace cleanup that follow the await. A reporter that threw an OperationCanceledException was caught by the loop's own cancellation exit instead, ending the heartbeat early while the work ran on. Guard the report so nothing it throws escapes. The loop now completes only by its own cancellation, never faulted, however the caller's reporter is wired.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
HeartbeatLoop.RunAsynchanded a failed ping to the caller'sonPingErroroutside the ping's guard, so a reporter that threw faulted the loop. Every caller awaits the loop in thefinallyaround the work it protects (AgentRunExecutor.cs:702,:896;SupervisorTurnService.Rehydrate.cs:657,:885,:1566), so the reporter's exception replaced the protected result, the same shape Keep a grading-heartbeat fault from replacing the grade #2012 closed for a failed pulse. InAgentRunExecutorit also skipped the credential revoke and workspace cleanup that follow the await. A reporter that threwOperationCanceledExceptionwas caught by the loop's own cancellation exit instead, and ended the heartbeat early while the work ran on.ReportQuietly, which swallows whatever the reporter throws. Cadence, the cancellation exit and the parameter list are unchanged. Production reporters log through SerilogWriteTosinks, which do not surface sink failures to the caller (seeProgram.BuildLogger), so this is hardening: the loop's contract no longer depends on how a caller's reporter is wired.catch (OperationCanceledException) { }aroundawait heartbeatatSupervisorTurnService.Rehydrate.cs:658,:886and:1567is dead code, and already was before this change:RunAsync's owncatch (OperationCanceledException)(HeartbeatLoop.cs:53) swallows every cancellation, so the task never ends canceled. Left in place here.AgentRunExecutor.cs:702and:896await the loop bare, so there is no such catch there.Test plan
HeartbeatLoopTests.A_reporter_that_throws_neither_faults_nor_ends_the_loop, a theory overInvalidOperationExceptionandOperationCanceledException. A failing ping plus a throwing reporter still beats once per interval on the fake clock, and cancelling completes the loop with no exception (Record.ExceptionAsyncis null). Red on main for both cases: the loop never re-arms after the first report.OperationCanceledExceptionthrough the guard reds only that case. The otherHeartbeatLoopTestsand allSupervisorGradingHeartbeatTestsstay green (12/12 with the fix).dotnet build backend/CodeSpace.sln: 0 errorsCategory!=RealBucket): 10941 passed, 0 failed, 1 skipped (the existingLocalRwxAtomicCreateTestsskip)