Skip to content

Keep a grading-heartbeat fault from replacing the grade - #2012

Merged
ppXD merged 1 commit into
mainfrom
fix/keep-a-grading-heartbeat-fault-from-replacing-the-grade
Sep 23, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/keep-a-grading-heartbeat-fault-from-replacing-the-grade

Conversation

@ppXD

@ppXD ppXD commented Sep 23, 2026

Copy link
Copy Markdown
Owner

Summary

  • A grading-heartbeat pulse that failed ended its loop faulted, and all three grade call sites in SupervisorTurnService.Rehydrate.cs (resolve fold, per-unit fold, stop-target grade) await that loop in the finally around the grade, so the pulse's error replaced the grade. The live trigger: the pulse's IRunRecordLogger shared the turn service's scoped CodeSpaceDbContext, so a pulse landing mid-query died on EF's "a second operation was started on this context" guard. The loop is now HeartbeatLoop.RunAsync (a failed pulse is a structured warning naming {SupervisorRunId}/{NodeId}, and the next pulse still fires), and each pulse resolves its record logger from its own DI scope via IServiceScopeFactory, the one new constructor parameter. The call-site finally shape is unchanged; its catch (OperationCanceledException) was already unreachable and stays as a guard.
  • The heartbeat tests pinned how many beats landed, not when: main's HeartbeatLoopTests pass a loop that sleeps half its interval. Each beat is now bracketed (a step short of the interval: nothing; the step that completes it: exactly one), and SupervisorLane.AcceptanceGradeHeartbeatInterval, documented as pinned but never pinned, is pinned at 90s inside the reconciler's liveness window.
  • HonestNoContinuityHint now ends "(your prior attempt pushed no branch of its own for this repository)", which stays true for a multi-repo attempt whose primary push failed but a sibling's succeeded (AgentCodeNode.RepinWorkspaceToPriorAttempt); nothing in the frontend reads the text. LoopbackModelCredentialBroker.AcceptAsync now says why an accept wait lost to a close strands nothing: the stranded loop and its lease are an unrooted cycle the GC collects (checked on .NET 10; no code change). Two stale comments are fixed (SupervisorDependencyStagingTests, ArtifactRetentionPolicy.For).

Test plan

  • Integration SupervisorGradingHeartbeatIsolationFlowTests: a real query holds the grade scope's context while a pulse falls due. Red on main (the pulse's InvalidOperationException surfaced from the grade's finally); green now (the pulse row lands on its own context and the grade survives)
  • Unit SupervisorGradingHeartbeatTests (6) and HeartbeatLoopTests (4); stable over 5 integration and 10 unit repeats
  • Mutations: swallow-and-continue removed → A_failed_pulse_is_reported_and_never_replaces_the_grade red; per-pulse scope removed → the integration test red (no pulse lands within 30s) plus 4 unit tests; one scope per loop → 2 unit tests red; grading loop sleeps interval / 2 → cadence test red; AcceptanceGradeHeartbeatInterval halved → pin red; HeartbeatLoop sleeps interval / 2 → 3 tests red (main's HeartbeatLoopTests stay 4/4 green)
  • Full unit suite: 10949 passed, 1 skipped (pre-existing)
  • Integration, every touched class plus the retry world-state and agent-node classes that read the hint: 233 run, 232 passed, 1 real-model skip
  • dotnet build: 0 errors, no new warnings

The grading heartbeat runs beside the grade it keeps alive, and every
call site awaits it in the finally around that grade. The loop caught
only cancellation, so a pulse that failed ended it faulted and the await
replaced the grade with that error — the opposite of the fold's promise
never to strand the terminal row. The live trigger was the pulse's own
record logger: it shared the turn service's scoped DbContext, so a pulse
that fell due while a grade query was in flight died on EF's "a second
operation was started on this context" guard. An integration test that
holds a real query open on the scope's context reproduces exactly that.

The loop is now HeartbeatLoop.RunAsync, so a failed pulse is a warning
naming the run and node and the next pulse still fires, and each pulse
writes through a record logger from a DI scope of its own, so it never
touches the grade's context and a failed insert never lingers in a
change tracker for the next pulse to retry.

The heartbeat tests pinned how many beats landed but not when: nudging
the fake clock in tenths until something arrived let any sleep between
a tenth of the interval and twenty of them pass, and the old tests went
green on a loop sleeping half its interval. Each beat is now bracketed —
nothing a step short of the interval, exactly one the step that
completes it — and the 90s cadence SupervisorLane calls pinned is pinned.

The honest-redo line now says the prior attempt pushed no branch "for
this repository": a multi-repo attempt whose primary push failed while
a sibling's succeeded still owes the line, and without the qualifier it
was false. The broker's accept loop says why a wait lost to a close
strands nothing: what is left is an unrooted cycle, collected with its
lease. Two stale comments are fixed.
@ppXD
ppXD merged commit 7da1cea into main Sep 23, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant