Skip to content

Keep a heartbeat loop alive when its error reporter throws - #2017

Merged
ppXD merged 1 commit into
mainfrom
fix/keep-a-heartbeat-loop-alive-when-its-error-reporter-throws
Sep 23, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/keep-a-heartbeat-loop-alive-when-its-error-reporter-throws

Conversation

@ppXD

@ppXD ppXD commented Sep 23, 2026

Copy link
Copy Markdown
Owner

Summary

  • HeartbeatLoop.RunAsync handed a failed ping to the caller's onPingError outside the ping's guard, so a reporter that threw faulted the loop. Every caller awaits the loop in the finally around the work it protects (AgentRunExecutor.cs:702, :896; SupervisorTurnService.Rehydrate.cs:657, :885, :1566), so the reporter's exception replaced the protected result, the same shape Keep a grading-heartbeat fault from replacing the grade #2012 closed for a failed pulse. In AgentRunExecutor it also skipped the credential revoke and workspace cleanup that follow the await. A reporter that threw OperationCanceledException was caught by the loop's own cancellation exit instead, and ended the heartbeat early while the work ran on.
  • The report now goes through ReportQuietly, which swallows whatever the reporter throws. Cadence, the cancellation exit and the parameter list are unchanged. Production reporters log through Serilog WriteTo sinks, which do not surface sink failures to the caller (see Program.BuildLogger), so this is hardening: the loop's contract no longer depends on how a caller's reporter is wired.
  • Call sites re-checked: the loop now completes only when its token is cancelled, and always successfully. The catch (OperationCanceledException) { } around await heartbeat at SupervisorTurnService.Rehydrate.cs:658, :886 and :1567 is dead code, and already was before this change: RunAsync's own catch (OperationCanceledException) (HeartbeatLoop.cs:53) swallows every cancellation, so the task never ends canceled. Left in place here. AgentRunExecutor.cs:702 and :896 await the loop bare, so there is no such catch there.

Test plan

  • Unit: HeartbeatLoopTests.A_reporter_that_throws_neither_faults_nor_ends_the_loop, a theory over InvalidOperationException and OperationCanceledException. A failing ping plus a throwing reporter still beats once per interval on the fake clock, and cancelling completes the loop with no exception (Record.ExceptionAsync is null). Red on main for both cases: the loop never re-arms after the first report.
  • Mutation: bypassing the guard reds both cases; letting a reporter's OperationCanceledException through the guard reds only that case. The other HeartbeatLoopTests and all SupervisorGradingHeartbeatTests stay green (12/12 with the fix).
  • dotnet build backend/CodeSpace.sln: 0 errors
  • Full unit suite (Category!=RealBucket): 10941 passed, 0 failed, 1 skipped (the existing LocalRwxAtomicCreateTests skip)

HeartbeatLoop.RunAsync handed a failed ping to the caller's onPingError
outside the ping's guard. A reporter that threw therefore faulted the
loop, and every caller awaits the loop in the finally around the work
it protects: AgentRunExecutor's launch and re-attach heartbeats, and
the three supervisor grading call sites. The reporter's exception then
replaced the protected result, and in AgentRunExecutor it also skipped
the credential revoke and workspace cleanup that follow the await. A
reporter that threw an OperationCanceledException was caught by the
loop's own cancellation exit instead, ending the heartbeat early while
the work ran on.

Guard the report so nothing it throws escapes. The loop now completes
only by its own cancellation, never faulted, however the caller's
reporter is wired.
@ppXD
ppXD merged commit ae45683 into main Sep 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant