Investigate intermittent failed checkpoint after a new thread replica catches up #76

The full Savio suite intermittently reported checkpoint.last_outcome = failed where test_a_thread_discovered_after_the_backup_waits_for_its_replica expects published after the second thread's replica catches up.

Observed once at f1c7430 in dedicated Slurm job 40509511, n0033.savio4, one CPU/8 GiB, Python 3.11.15 and pytest 7.4.4. Full result: 2300 passed and two failed; the other failure is the already recorded issue 70 real sealed-generation case.

The assertion is at tests/test_codex_state_runner.py:1672. The fixture uses Managed(timeout=2, SUCODER_CODEX_STATE_REPLICATION_INTERVAL=8). It first proves a newly discovered thread is not durable, waits for the replica to reach the captured length, then requests a new capture. That second request returned failed. The assertion omitted the checkpoint dictionary, so the capture failure reason was not retained in the traceback. The receiving job's local scratch was released normally.

Three isolated repeats on unchanged main debae3f and three on f1c7430 all passed in a new dedicated job 40509995. These runs used a short basetemp and a private failure hook that would record only checkpoint state/outcome/reason if the failure recurred. No production source or existing test was changed.

Cause is unclassified. This is not evidence that the requirement to wait for a durable replica is wrong, nor proof of a new dispatch-profile regression. The project-profile changes do not touch replication/checkpoint code. Relative to main, the dispatch-core changes add installation_id to the bind-file list and a separate descriptor-mount inspection helper; their relevance to this failure is unproven.

Suggested next investigation:

1. Include the checkpoint dictionary in the final assertion or capture its reason through an external failure hook, retaining private runner diagnostics before Slurm cleanup.
2. Reproduce under the full-suite schedule or a controlled capture/replication delay, distinguishing a capture timeout from a replication-proof defect.
3. Fix the ordering/runtime issue or correct a demonstrated fixture assumption; do not simply enlarge the timeout or accept a failed checkpoint as published.
4. Preserve the assertions that the initial undiscovered rollout is non-durable and that successful publication includes the full immutable rollout prefix.

Private review artifacts: ~/.sucoder/reviews/issue72/profile-f1c7430/40509511-full.{xml,log} and ~/.sucoder/reviews/issue72/profile-d739279/40509995-{main,feature}-{1,2,3}.{xml,log}. Raw tracebacks are intentionally not attached because other tests can include inherited credentials.

Reported by Sue (GPT-6) during independent dispatch-profile review. Keep this investigation separate from PR75.

Investigation update — 2026-10-30

PR 85 retains the checkpoint dictionary on the second assertion and adds deterministic backup/publication deadline cases. Three focused repetitions and a separate full suite passed (2,480 tests); the two added deadline cases and the existing shared-shutdown-budget case passed at the final test pin. Source/import receipts and private logs are recorded in the PR ledger.

The historical failure remains unclassified: the original XML contains no checkpoint reason and this bounded repeat did not reproduce it. Caught-up replicas do not guarantee publication within the same two-second budget, but injected deadline failures do not establish the original cause. No timeout, replica proof, runtime policy or existing publication/prefix requirement was relaxed. Keep this issue open. Next action: capture the already computed checkpoint reason on an actual recurrence under the supported test schedule, then add a regression/correction for that demonstrated cause. The diagnostics PR is ready for review.