Skip to content

Reused managed runs can expose stale checkpoint counts from the prior launch #73

Closed

Description

@ligon

Reusing a managed run can temporarily expose the previous launch's checkpoint view as though it belonged to the new launch. A client waiting for a new capture by comparing checkpoint.count can then wait forever after both captures succeed.

Observed on Savio in allocation 40501331 while reviewing issue 72: test_reuse_in_place_continues_the_run_and_refuses_a_live_owner timed out at _request(home) (tests/test_codex_state_runner.py:1108). The final status showed count 2, idle/published, and a new initial generation, rather than a failed capture. Three isolated runs of the unchanged-main test passed; the original failure is intermittent.

Source inspection explains the race:

  • Status.load() retains the prior launch's checkpoint dictionary during reuse.
  • Checkpointer.__init__() starts a new counter at zero.
  • Placement/replication can be observed before the new checkpointer initializes its status.
  • The test's _wait_idle() can accept the old idle count of 2, so _request() records before=2.
  • The new scheduler can consume the requested and initial captures at counts 1 and 2. Its captures succeed, but count > before never becomes true.

This is separate from issue 70's final-seal reader ordering. The issue 72 runtime changes do not alter this lifecycle.

Suggested deterministic regression: hold the second launch's Checkpointer.start() behind a private gate after placement, submit the request while old status is visible, then release it. This controlled-delay probe has been designed but not yet executed.

Acceptance: expose checkpoint status with a current-launch identity/readiness marker, or clear its transient view at reuse startup; retain acknowledged generation/history facts. Update the test helper to require current-launch readiness. Prove the request completes under that delayed schedule. Do not inflate the timeout or discard acknowledged generation data.

Private review log/XML: ~/.sucoder/reviews/issue72/40501331-full.{log,xml}. Investigated by Sue.

Backlog reconciliation — 2026-10-30

PR 80 merged into main as c09d361. Status.load now clears the launch-transient checkpoint view while retaining acknowledged/pending generations, provenance and transition history. The deterministic gated-startup regression submits a request before the reused checkpointer starts and verifies completion and subsequent requests; the history-preservation regression covers retained facts. Both fail at the stale-status assertion on baseline and pass with the correction. The PR records 99 passing managed-runner tests and green Python 3.9/3.11/docs CI. No timeout or preservation rule was relaxed. Issue 70's final-seal correction is separate (PR 84); issue 76's intermittent checkpoint investigation remains open.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions