Repository navigation
Description
Reusing a managed run can temporarily expose the previous launch's checkpoint view as though it belonged to the new launch. A client waiting for a new capture by comparing checkpoint.count can then wait forever after both captures succeed.
Observed on Savio in allocation 40501331 while reviewing issue 72: test_reuse_in_place_continues_the_run_and_refuses_a_live_owner timed out at _request(home) (tests/test_codex_state_runner.py:1108). The final status showed count 2, idle/published, and a new initial generation, rather than a failed capture. Three isolated runs of the unchanged-main test passed; the original failure is intermittent.
Source inspection explains the race:
Status.load()retains the prior launch'scheckpointdictionary during reuse.Checkpointer.__init__()starts a new counter at zero.- Placement/replication can be observed before the new checkpointer initializes its status.
- The test's
_wait_idle()can accept the old idle count of 2, so_request()recordsbefore=2. - The new scheduler can consume the requested and initial captures at counts 1 and 2. Its captures succeed, but
count > beforenever becomes true.
This is separate from issue 70's final-seal reader ordering. The issue 72 runtime changes do not alter this lifecycle.
Suggested deterministic regression: hold the second launch's Checkpointer.start() behind a private gate after placement, submit the request while old status is visible, then release it. This controlled-delay probe has been designed but not yet executed.
Acceptance: expose checkpoint status with a current-launch identity/readiness marker, or clear its transient view at reuse startup; retain acknowledged generation/history facts. Update the test helper to require current-launch readiness. Prove the request completes under that delayed schedule. Do not inflate the timeout or discard acknowledged generation data.
Private review log/XML: ~/.sucoder/reviews/issue72/40501331-full.{log,xml}. Investigated by Sue.
Backlog reconciliation — 2026-10-30
PR 80 merged into main as c09d361. Status.load now clears the launch-transient checkpoint view while retaining acknowledged/pending generations, provenance and transition history. The deterministic gated-startup regression submits a request before the reused checkpointer starts and verifies completion and subsequent requests; the history-preservation regression covers retained facts. Both fail at the stale-status assertion on baseline and pass with the correction. The PR records 99 passing managed-runner tests and green Python 3.9/3.11/docs CI. No timeout or preservation rule was relaxed. Issue 70's final-seal correction is separate (PR 84); issue 76's intermittent checkpoint investigation remains open.
Activity