Fence stale supervisor writes and close stopped work on Continue

[ppXD]

Summary

• A supervisor turn a Continue overtook writes nothing under the revived run. The stopped walk (on another host, so the stop never tripped it) can still be deciding, claiming or staging its next turn when the Continue lands. The engine carries the generation it claimed to every node it runs (RunGenerationFence.Claim in WorkflowEngine.RunNodeOnceAsync). Every supervisor write takes the engine park's SELECT … FOR SHARE on the run at that generation inside its own transaction (RunGenerationFence.EnterAsync / CommitUnderClaimAsync):

  • the decision's claim, begin and terminal record (SupervisorDecisionLog);
  • the spawn wave (RealSupervisorActionExecutor.StageAgentsAndParkAsync);
  • an ask_human question: its card, its wait and the ask's decision record now commit as one transaction, opened by the turn service (SupervisorTurnService.ExecuteAndRecordTogetherAsync). A question anyone can see therefore always has its token on the tape that ISupervisorAskAnswerService, the plan confirmation and the human-touch reader answer from.

  The terminal record takes the fence before it reads the decision's status. A walk whose decision the revived walk finished first stands down, instead of throwing an illegal Succeeded → Succeeded into the engine's generic catch as the supervisor step failing. An overtaken turn throws RunSupersededException, which the engine lets through as the walk's end, never as a node failure. A decision it claimed before the revive stays in flight, and the revived walk finishes it once, as it finishes a crashed walk's.

  • Only a turn already past its fence when the revive lands finishes that write. The Continue's bump waits for it, then closes it like the rest of the stopped attempt: an ask whose question and record committed is re-opened, not posted twice. External I/O such a turn had already started (a merge push, a publish) can still complete. Its terminal record is then refused rather than recorded, and the revived walk re-executes the decision, exactly as after a walk that crashed past its side effect.
  • Rejected: a terminal Superseded decision status. It needs a migration and a partial unique index, because the revived walk's deterministic re-decision collides on the (run, key) index, and the Duplicate path would replay the superseded row as an empty turn that is never counted. Also rejected for the ask: skipping the node's human re-entry guard while an ask is in flight, so a replay records the token late. That leaves a window where a posted question has no token on the tape; the single transaction has none.

• Budget admission runs inside the fenced wave transaction (AdmitWaveAsync, after EnterAsync; the ledger joins the transaction). An overtaken turn reserves nothing, and a refusal rolls back with the wave. A slot that already holds a row (a closed slot restaged, or a finished slot kept) keeps the reservation it was first admitted under. Before, a re-reservation computed a new deadline, which ReserveAsync reads as a different intent, so a capped run's restage was refused as over budget. Rejected: tolerating a recomputed deadline, or re-reserving a Released row, in the ledger; both break its pinned identical-intent contract.

• Continue ends what the stopped attempt left, in the revive's transaction after the bump (CloseEndedAttemptAsync). It discards every Pending wait, cancels Queued agents, cancels staged Pending/Enqueued child runs, and CASes Running agents to Cancelled with the row half of the cancel (IRunningAgentCancellation.CancelRunningRowAsync).

  • Their kill, credential revoke, spend settlement and decision expiry run after the commit (FinishRunningCancelAsync, one post-commit action per agent). They run outside any transaction, as the decision lock order from Close an ended agent run's decisions and cancel started sub-workflows #2029 requires. They also run on CancellationToken.None, as does the revived run's dispatch, so a client that disconnects after the commit cannot cut them short.
  • Before, the revived supervisor folded a still-running agent onto its tape as "Running" for good. The rehydrate fold now also leaves a wave unfolded while any of its agents is Queued or Running.
  • A Continue after a Failure, which has no teardown, now gets its kills from the revive.
  • The split is a sibling interface rather than new members on IAgentRunService.

• A closed wave restages only its closed slots. A finished agent keeps its answered wait and runs no second time (TurnWave; DropClosedWaveAsync deletes Discarded rows only).

• The stop's teardown can no longer be deadlocked out of its kill-wave, or leave a wait open. A Continue's revive holds every Pending wait of the run until it commits, and it and the teardown locked those rows in opposite orders.

  • Both wait closes skip rows another transaction holds (FOR UPDATE SKIP LOCKED), then pass over them again, up to three times 200 ms apart (CloseWaitsSkippingHeldOnesAsync). A holder that rolls back instead of closing them therefore releases them to a later pass, rather than leaving them Pending for the kill to answer. What stays held is logged.
  • Each step runs on its own (TearDownStepAsync), so a failed step never costs the steps after it.
  • Rejected: id-ordered locking on both sides, which would keep the teardown queued behind the Continue and cannot be tested deterministically.

• Accepted residuals:

  • A capped wave now holds the team-wide budget advisory lock for its whole staging transaction: a latency cost for concurrent capped admissions of the team; no deadlock was found.
  • A decision begun before the bump (a plan, a merge, a publish, a stop's acceptance grading, a spawn's pre-staging integration work) still runs to completion concurrently with the revived walk's re-run; only its writes after the bump are refused.
  • The pre-claim writes in ChooseDecisionAsync, the rehydrate fold's outcome write and ReopenDiscardedAskAsync are not fenced.

• Follow-ups:

  • The engine's failure ceremony (WorkflowEngine.CancelPendingWaitsAndChildrenAsync) is still unfenced on the generation, so a slow ceremony can discard waits the revived walk already parked.
  • A stale turn's inline review agent (AgentReviewRunner), created after the revive, runs under the live revived run, and nothing kills it.

Test plan

• New and extended integration tests over real Postgres. Each drives either the real engine walk on another host's cancellation registry, or the real turn service under that walk's claim, with the hold point named in each test. A fixture-root decoration of the decision log (ScriptedDecisionLog, armed per run by SupervisorDecisionScript.HoldDecisionLog) lets a test hold the engine's own walk inside the supervisor node.

  • SupervisorStopContinueFlowTests (the wave cases run both capped and uncapped):
    • A_turn_a_continue_overtook_while_it_decided_…
    • …claims_nothing_after_the_revived_turn_decided_otherwise
    • A_stop_teardown_landing_after_a_continue_finds_nothing_…
    • A_stopped_wave_whose_decision_was_still_in_flight_…
    • A_stopped_wave_staged_afresh_keeps_the_agent_that_had_already_finished
    • A_turn_that_claimed_its_decision_before_the_stop_begins_nothing_after_the_continue
    • A_turn_already_assembling_its_wave_when_the_continue_lands_reserves_nothing
    • A_turn_whose_wave_committed_before_the_stop_records_no_terminal_after_the_continue
    • A_stop_landing_while_a_wave_is_staged_waits_for_the_whole_wave_…
    • A_walk_whose_decision_the_revived_walk_finished_first_stands_down_without_failing_the_step: through the engine, with the revived walk recording the plan before the overtaken walk's executor returns
    • A_continue_whose_request_went_away_after_its_commit_still_ends_the_running_agents_and_dispatches_the_run
  • SupervisorAskHumanStopContinueFlowTests, each answering through ISupervisorAskAnswerService, never the wait's token:
    • An_ask_a_continue_overtook_before_its_question_posts_no_card_parks_no_wait_and_records_nothing
    • A_continue_landing_while_an_ask_posts_its_card_waits_for_it_and_the_revived_run_answers_from_that_card
    • A_gate_question_asked_with_no_conversation_as_a_continue_lands_stays_answerable_through_the_ask_api
  • ContinueParkedRunFlowTests:
    • A_continue_holding_the_stopped_attempts_waits_never_stalls_the_stop_teardown_short_of_its_kill_wave
    • A_teardown_step_that_fails_never_costs_the_kill_wave (a 40P01 injected into the close)
    • A_wait_the_teardown_had_to_skip_is_closed_once_the_transaction_holding_it_lets_go (a second connection holds the wait, then lets go)
    • The existing child test also asserts that the revive cancelled the staged child.
  • OperatorCancelInProgressWalkFlowTests: the park lock-hold test now asserts the stopped walk's outcome and releases its hold in a finally. It uses the kit's helpers, whose lock-wait check fails at once when the writer finishes without waiting.

• Unit: RunGenerationFenceTests (claim read back for its run only, nested restore, async flow, lock refused outside a transaction, no-op outside a walk) and five fold-guard cases in SupervisorAgentResultsFoldTests.

• Mutations, each red on this head:

  Mutation	Red test(s)
  M1a claim unfenced	both overtaken-decision tests (4 cases)
  M1b begin unfenced	…begins_nothing_after_the_continue
  M1c terminal unfenced	…records_no_terminal_after_the_continue, A_walk_whose_decision_the_revived_walk_finished_first…
  N1 terminal legality checked before its fence	A_walk_whose_decision_the_revived_walk_finished_first… (a node.failed lands in the revived run)
  M2a admission ahead of the fence	…reserves_nothing(capped)
  M2b closed slot reserves again	the three capped restage tests
  N2 ask record outside its question's transaction	all three ask tests
  M4a revive leaves Running agents running; M4b post-commit finish skipped	A_stop_teardown_landing_after_a_continue_finds_nothing_…, …request_went_away…
  M4c fold keeps a live status	the fold unit cases
  N3a post-commit kill honours the request token; N3b dispatch honours the request token	…request_went_away…
  M5 revive leaves staged children	the continue child test
  M6a teardown closes wait on locks	…never_stalls_the_stop_teardown…, …skip_is_closed_once…
  M6b teardown as one best-effort block	…never_costs_the_kill_wave
  N4 skipped waits get no second pass	…skip_is_closed_once…
  M7 closed wave drops answered slots	…keeps_the_agent_that_had_already_finished
  M8a spawn fence takes no lock	A_stop_landing_while_a_wave_is_staged…
  M8b commit fence takes no lock	the two ask lock-wait tests
  M9a–c claim read for any run, not restored, lock outside a transaction	RunGenerationFenceTests
  E1/E2 engine drops its claim, or records the stand-down as a failure	the overtaken-walk tests

  M1d (no stand-down read before the re-park) survives by design: the fenced terminal record already refuses what the re-park does next.

• SupervisorStopContinueFlowTests 19/19, SupervisorAskHumanStopContinueFlowTests 3/3, ContinueParkedRunFlowTests 10/10, OperatorCancelInProgressWalkFlowTests 15/15

• Every non-real-model Workflows.Supervisor* integration class 694/694; budget ledger and settlement classes 94/94; cancel, continue, decision, sub-workflow and reconciler neighbours 634/634

• Unit suite: 11630 passed, 1 skipped

• dotnet build CodeSpace.sln: 0 errors