From be58dd07f1fb3527749f5b63d294ba100b7693ee Mon Sep 17 00:00:00 2001 From: Ethan Ligon Date: Fri, 30 Oct 2026 09:55:38 -0700 Subject: [PATCH 1/4] ledger: map replica discovery and checkpoint capture invariants --- docs/replica-checkpoint-ledger.org | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 docs/replica-checkpoint-ledger.org diff --git a/docs/replica-checkpoint-ledger.org b/docs/replica-checkpoint-ledger.org new file mode 100644 index 0000000..1b36ff8 --- /dev/null +++ b/docs/replica-checkpoint-ledger.org @@ -0,0 +1,28 @@ +#+TITLE: Replica catch-up checkpoint investigation (issue 76) +#+DATE: 2026-10-30 + +* 1. Task +Classify the failed second capture in test_a_thread_discovered_after_the_backup_waits_for_its_replica without weakening publication or replication proof. Base: GitHub main 52cc04f40f3858a5cf7d8063f009a4f10db6a07c. The historical failure reason was not retained. + +* 2. Existing machinery +- =sucoder/codex_state_runner.py:2188=, Replicator.pass_once :: Discovers and follows rollouts, then publishes a replica view; tested by the replication, directory-sync and discovery regressions. +- =sucoder/codex_state_runner.py:2744=, Checkpointer.capture :: Reports distinct failed/not_durable/published outcomes with reason; uses one deadline for backup, inspection, replica waiting and publication. +- =sucoder/codex_state_runner.py:2957=, Checkpointer.bundle :: Copies stores and inspects required bytes; keeps a local manifest for a non-durable capture. +- =sucoder/codex_state_runner.py:3059=, Checkpointer.wait_for_replicas :: Requires max(projection watermark, captured rollout length), hashes the immutable local prefix, and reports lagging threads. +- =sucoder/codex_state_runner.py:3092=, Checkpointer.publish :: Enforces the remaining capture deadline before durable copy/rename; covered by publication-failure and deadline tests. +- =tests/test_codex_state_runner.py:1727= :: The reported test uses a two-second capture budget and eight-second replication interval. First capture must be not_durable; second must publish after discovery. +- =tests/test_codex_state_runner.py:254=, _request :: Waits for a completed current-launch capture, irrespective of outcome. Managed starts an isolated fake child through the real runner. InProcess supplies a separate disposable direct harness. + +Search tier: rg, direct source/caller inspection. GitNexus query, upstream impact on the reported test, analyze --skip-agents-md and detect-changes were attempted; all fail loading libssl.so.3. No shared install repair is in scope. The test has no production callers; pytest collects it. Initial diagnostic edit has low blast radius, confined to assertion output. + +* 3. Definitions and conventions +=not_durable= means copied stores exist but replica proof is incomplete. =failed= may arise from backup/inspection, hashing, filesystem errors or publication deadline; replica catch-up alone is not a publication guarantee. =published= requires verified copies and durable directory acknowledgement. A single checkpoint timeout covers the complete capture, not only replica waiting. + +* 4. Invariants +Retain the first non-durable outcome and local manifest, required captured rollout length even with projection offset zero, and full prefix/hash equality on successful publication. Preserve capture/shutdown budgets, ownership, placement and other-holder refusal. Do not infer the historical cause from a newly injected failure. Run nested managed/native tests in a separate test allocation, not beneath the coordinator's placement monitor. Retain only bounded checkpoint diagnostics in public reports; keep full logs private. + +* 5. Reuse decision +Extend the existing assertion with its checkpoint dictionary so failures expose the already computed reason. Reuse runner and fixture machinery for baseline/full-suite and controlled-delay diagnosis. No production hook, new configuration or runtime correction is justified before evidence. Refine this ledger when the failure is classified. + +* 6. Open questions +Did the original capture exhaust its two-second budget during backup/publication, or fail an independent proof/storage check? Can a bounded full-suite repeat retain the actual reason? Is the wall-clock fixture's discovery ordering reliable under load? These are investigative questions, not user-approval gates. From 8cd3f31f53b648ef7ce9d4cb9f50916db201f0b6 Mon Sep 17 00:00:00 2001 From: Ethan Ligon Date: Fri, 30 Oct 2026 09:55:54 -0700 Subject: [PATCH 2/4] test: retain the replica catch-up checkpoint failure reason --- tests/test_codex_state_runner.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/test_codex_state_runner.py b/tests/test_codex_state_runner.py index ee642aa..9089d9e 100644 --- a/tests/test_codex_state_runner.py +++ b/tests/test_codex_state_runner.py @@ -1753,7 +1753,8 @@ def discovered(st): return entry is not None and entry["replicated_bytes"] == entry["local_bytes"] == size2 _wait(home, discovered, "the second thread's replica", timeout=30) status = _request(home) - assert status["checkpoint"]["last_outcome"] == "published" + checkpoint = status["checkpoint"] + assert checkpoint["last_outcome"] == "published", checkpoint thread = _manifest(status)["threads"][tid2] assert thread["replica_bytes"] == size2 and thread["next_rollout_byte_offset"] == 0 assert (m.durable_run / "rollouts" / f"{tid2}.jsonl").read_bytes() == local2.read_bytes() From 4f7a573e5aaa4f107b3a96fe1dab5b7161db271b Mon Sep 17 00:00:00 2001 From: Ethan Ligon Date: Fri, 30 Oct 2026 10:01:26 -0700 Subject: [PATCH 3/4] test: distinguish replica catch-up from capture deadline proof --- tests/test_codex_state_runner.py | 34 ++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/tests/test_codex_state_runner.py b/tests/test_codex_state_runner.py index 9089d9e..4d3dd36 100644 --- a/tests/test_codex_state_runner.py +++ b/tests/test_codex_state_runner.py @@ -1763,6 +1763,40 @@ def discovered(st): m.close() +@pytest.mark.parametrize("boundary,reason", [ + ("backup_store", "capture overdue before state_5.sqlite"), + ("publish", "publication overdue before state_5.sqlite"), +]) +def test_replica_catchup_does_not_bypass_capture_deadline(runner, tmp_path, monkeypatch, boundary, reason): + """A caught-up replica is one proof, not a fresh capture budget (ledger §§3–4).""" + from types import SimpleNamespace + + now = [0.0] + clock = SimpleNamespace(**vars(time)) + clock.monotonic = lambda: now[0] + monkeypatch.setattr(runner, "time", clock) + h = InProcess(runner, tmp_path, timeout=2) + h.replicator.pass_once() + assert h.replica.read_bytes() == h.local_rollout.read_bytes() + assert h.capture() == ("published", None) + previous = h.cp.last_generation + + target = runner if boundary == "backup_store" else h.cp + original = getattr(target, boundary) + + def delayed(*args, **kwargs): + now[0] += h.cp.timeout + 0.01 + return original(*args, **kwargs) + + monkeypatch.setattr(target, boundary, delayed) + assert h.capture() == ("failed", reason) + checkpoint = h.status_json()["checkpoint"] + assert checkpoint["last_outcome"] == "failed" and checkpoint["reason"] == reason + assert h.cp.last_generation == previous + assert sorted(os.listdir(h.generations)) == [previous["name"]] + assert h.replica.read_bytes() == h.local_rollout.read_bytes() + + def test_holders_of_reads_the_links_and_stats_nothing(runner, tmp_path, monkeypatch): """Finding 9: a stat of a descriptor follows the link onto its filesystem; holders are found by the link text alone, deleted stores included.""" From fc8c21fa64a62aaba172104d0ba3085121f28aa8 Mon Sep 17 00:00:00 2001 From: Ethan Ligon Date: Fri, 30 Oct 2026 10:06:02 -0700 Subject: [PATCH 4/4] ledger: record bounded checkpoint diagnosis and passing validation --- docs/replica-checkpoint-ledger.org | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/docs/replica-checkpoint-ledger.org b/docs/replica-checkpoint-ledger.org index 1b36ff8..472554d 100644 --- a/docs/replica-checkpoint-ledger.org +++ b/docs/replica-checkpoint-ledger.org @@ -22,7 +22,20 @@ Search tier: rg, direct source/caller inspection. GitNexus query, upstream impac Retain the first non-durable outcome and local manifest, required captured rollout length even with projection offset zero, and full prefix/hash equality on successful publication. Preserve capture/shutdown budgets, ownership, placement and other-holder refusal. Do not infer the historical cause from a newly injected failure. Run nested managed/native tests in a separate test allocation, not beneath the coordinator's placement monitor. Retain only bounded checkpoint diagnostics in public reports; keep full logs private. * 5. Reuse decision -Extend the existing assertion with its checkpoint dictionary so failures expose the already computed reason. Reuse runner and fixture machinery for baseline/full-suite and controlled-delay diagnosis. No production hook, new configuration or runtime correction is justified before evidence. Refine this ledger when the failure is classified. +Extend the existing assertion with its checkpoint dictionary so failures expose the already computed reason. Reuse runner and fixture machinery for baseline/full-suite and controlled-delay diagnosis. The added parametric test reuses InProcess, real stores and real replicas; a private runner-module clock crosses the deadline exactly at backup/publication. It requires honest failure, the specific reason, and retention of the previous acknowledged generation. No production hook, new configuration or runtime correction is justified before evidence. -* 6. Open questions -Did the original capture exhaust its two-second budget during backup/publication, or fail an independent proof/storage check? Can a bounded full-suite repeat retain the actual reason? Is the wall-clock fixture's discovery ordering reliable under load? These are investigative questions, not user-approval gates. +* 6. Open questions and validation +The original cause remains unclassified. Its retained XML failure is only 252 characters and has no capture reason. Three focused repetitions and the full suite on the diagnostic assertion commit did not reproduce it. A scheduling/filesystem delay consuming the two-second capture budget is plausible, not established. Keep issue 76 open; retain the next actual failed checkpoint reason before proposing a runtime correction. No timeout increase or fixture relaxation is supported by this evidence. + +Dedicated Slurm job =40654245= on =n0034.savio4= used one CPU/8 GiB under =co_carleton/savio4_htc/carleton_htc4_normal=. Three focused repetitions passed in 18.53, 17.27 and 17.38 seconds. The full suite on =8cd3f31f53b648ef7ce9d4cb9f50916db201f0b6= had *2480 passed*, zero failures/errors/skips, in 325.60 seconds; the reported test passed in 17.286 seconds. This full run predates only the two added deadline cases, not any production change. + +Dedicated job =40654291= exercised the final code/test pin =4f7a573e5aaa4f107b3a96fe1dab5b7161db271b=: the two controlled deadline cases plus the existing shared-shutdown-deadline regression had *3 passed, 101 deselected* in 2.82 seconds. The injected failures classify the controlled experiment only; they do not recreate historical scheduling or prove the original failure's cause. Both jobs completed with scheduler exit 0:0 and released normally. Source-head and import-path receipts identify each receiving-node checkout. + +Durable private evidence: =/global/scratch/fsa/fc_jevons/ligon/burst/sucoder-issue76-2f7275wc/=, with =test.sh=, =deadline.sh=, =test-head=, =deadline-test-head=, import receipts, focused logs/XML, =full.log/xml= and =deadline.log/xml=. Test commands run from the receiving SuCoder root: +#+begin_src sh +.venv/bin/python -m pytest -q -o showlocals=false --tb=short tests/test_codex_state_runner.py::test_a_thread_discovered_after_the_backup_waits_for_its_replica +.venv/bin/python -m pytest -q -o showlocals=false --tb=short +.venv/bin/python -m pytest -q -o showlocals=false --tb=short tests/test_codex_state_runner.py -k 'replica_catchup_does_not_bypass_capture_deadline or final_shares_one_shutdown_deadline' +#+end_src + +Independent source review found no blocking issue and confirmed clock isolation, existing timeout/refusal semantics and preservation of the original replica/prefix assertions. Reviewer did not run tests. Ledger verification: OK anchored on sections 2–5; no duplicate runtime mechanism or definition contradiction. GitNexus detect-changes remains unavailable (libssl.so.3); manual comparison limits this branch to the diagnostic assertion, two test cases and this ledger. The coordinator and shared installation were unchanged. Priority-account usage at session start was 75,938.47 SUs of 1,500,000; both test jobs used co_carleton rather than that billed account.