Keep a grading-heartbeat fault from replacing the grade - #2012
Merged
ppXD merged 1 commit intoSep 23, 2026
Merged
Conversation
The grading heartbeat runs beside the grade it keeps alive, and every call site awaits it in the finally around that grade. The loop caught only cancellation, so a pulse that failed ended it faulted and the await replaced the grade with that error — the opposite of the fold's promise never to strand the terminal row. The live trigger was the pulse's own record logger: it shared the turn service's scoped DbContext, so a pulse that fell due while a grade query was in flight died on EF's "a second operation was started on this context" guard. An integration test that holds a real query open on the scope's context reproduces exactly that. The loop is now HeartbeatLoop.RunAsync, so a failed pulse is a warning naming the run and node and the next pulse still fires, and each pulse writes through a record logger from a DI scope of its own, so it never touches the grade's context and a failed insert never lingers in a change tracker for the next pulse to retry. The heartbeat tests pinned how many beats landed but not when: nudging the fake clock in tenths until something arrived let any sleep between a tenth of the interval and twenty of them pass, and the old tests went green on a loop sleeping half its interval. Each beat is now bracketed — nothing a step short of the interval, exactly one the step that completes it — and the 90s cadence SupervisorLane calls pinned is pinned. The honest-redo line now says the prior attempt pushed no branch "for this repository": a multi-repo attempt whose primary push failed while a sibling's succeeded still owes the line, and without the qualifier it was false. The broker's accept loop says why a wait lost to a close strands nothing: what is left is an unrooted cycle, collected with its lease. Two stale comments are fixed.
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SupervisorTurnService.Rehydrate.cs(resolve fold, per-unit fold, stop-target grade) await that loop in thefinallyaround the grade, so the pulse's error replaced the grade. The live trigger: the pulse'sIRunRecordLoggershared the turn service's scopedCodeSpaceDbContext, so a pulse landing mid-query died on EF's "a second operation was started on this context" guard. The loop is nowHeartbeatLoop.RunAsync(a failed pulse is a structured warning naming{SupervisorRunId}/{NodeId}, and the next pulse still fires), and each pulse resolves its record logger from its own DI scope viaIServiceScopeFactory, the one new constructor parameter. The call-sitefinallyshape is unchanged; itscatch (OperationCanceledException)was already unreachable and stays as a guard.HeartbeatLoopTestspass a loop that sleeps half its interval. Each beat is now bracketed (a step short of the interval: nothing; the step that completes it: exactly one), andSupervisorLane.AcceptanceGradeHeartbeatInterval, documented as pinned but never pinned, is pinned at 90s inside the reconciler's liveness window.HonestNoContinuityHintnow ends "(your prior attempt pushed no branch of its own for this repository)", which stays true for a multi-repo attempt whose primary push failed but a sibling's succeeded (AgentCodeNode.RepinWorkspaceToPriorAttempt); nothing in the frontend reads the text.LoopbackModelCredentialBroker.AcceptAsyncnow says why an accept wait lost to a close strands nothing: the stranded loop and its lease are an unrooted cycle the GC collects (checked on .NET 10; no code change). Two stale comments are fixed (SupervisorDependencyStagingTests,ArtifactRetentionPolicy.For).Test plan
SupervisorGradingHeartbeatIsolationFlowTests: a real query holds the grade scope's context while a pulse falls due. Red on main (the pulse'sInvalidOperationExceptionsurfaced from the grade'sfinally); green now (the pulse row lands on its own context and the grade survives)SupervisorGradingHeartbeatTests(6) andHeartbeatLoopTests(4); stable over 5 integration and 10 unit repeatsA_failed_pulse_is_reported_and_never_replaces_the_gradered; per-pulse scope removed → the integration test red (no pulse lands within 30s) plus 4 unit tests; one scope per loop → 2 unit tests red; grading loop sleepsinterval / 2→ cadence test red;AcceptanceGradeHeartbeatIntervalhalved → pin red;HeartbeatLoopsleepsinterval / 2→ 3 tests red (main'sHeartbeatLoopTestsstay 4/4 green)dotnet build: 0 errors, no new warnings