Preserve SHA-only source commits in dispatch snapshots

docs/dispatch-backlog-20261030.org

@@ -0,0 +1,165 @@
+#+TITLE: Dispatch backlog: reconciliation and SHA-only source fix
+#+DATE: 2026-10-30
+
+* Scope and coverage
+
+Repository: [[https://github.com/ligon/sucoder][ligon/sucoder]], default and merge target =main=.
+Inspected implementation base: =d2556f0dc84f40b239a8e014934d0af821087a6e=,
+including merged PRs 74, 75, 81 and 78. No development/release branch is targeted.
+This implementation batch follows the user's request to address dispatch issues
+using backlog-workflow. It selects issue 82; the other issues retain independent
+acceptance criteria and are reconciled below, not silently treated as fixed.
+
+Inventory retrieved 2026-10-30, approximately 13:10–13:20 UTC: 14 open issues,
+one open PR (80). REST returned 15 issue/PR entries with per_page=100: one
+complete page. All issues have no labels or assignees; no priority labels apply.
+Connector issue search returned issue records even for a PR-only query, so
+explicit-repository gh REST/list commands supplied completeness and PR checks.
+Bodies and all comments were read for dispatch issues 70, 72, 76, 77, 79 and 82;
+code, relevant merged PRs and acceptance records were compared. No sampling of
+those six. Seven unrelated issues (14, 54, 57, 58, 59, 65, 66) are metadata-only.
+Related issue 73 is tracked through PR 80 and the earlier
+[[file:backlog-20261029.org][backlog report]], rather than duplicated.
+Timeline cross-reference graphs were not exhaustively traversed; explicit
+links, merged dispatch PR search and the complete open-PR list were checked.
+
+The user subsequently authorized merging PR 80. Its exact head =58586ba3=
+had green CI run [[https://github.com/ligon/sucoder/actions/runs/40990436269][40990436269]],
+no review submissions or review threads, and a mergeable state. Source/tests and
+commit messages were inspected. Merge succeeded as =c09d36161b93162fc75d4534dfa1a2cbdf2b9ac9=.
+Issue 73 therefore becomes a resolved candidate; issue state was not changed.
+This update supersedes the initial open-PR inventory, not its retrieval record.
+
+* Dispositions and ordered work
+
+Counts for the six examined dispatch issues: two resolved candidates, one
+partially addressed, one needs diagnosis, one ready, one implemented pending PR.
+No shared root cause or blocking cycle is inferred from their common dispatch context.
+
+| Issue | Disposition | Impact / urgency | Effort / confidence | Next action and completion criterion |
+|-------+-------------+------------------+---------------------+--------------------------------------|
+| [[https://github.com/ligon/sucoder/issues/82][82]] | Implemented in this batch; pending merge | Medium: valid sources cannot launch; ref workaround exists | Small, high confidence | Review exact-SHA staging fix; loose/packed unreferenced commits must stage without source mutation, failures before Slurm. |
+| [[https://github.com/ligon/sucoder/issues/70][70]] | Ready | High: clean exit may prevent sealed recovery | Substantial lifecycle change, strong ordering evidence | Drain owned captures before class selection within the existing deadline; test other holders, unclean exit, stalls, real restore and reuse. |
+| [[https://github.com/ligon/sucoder/issues/76][76]] | Needs diagnosis | Medium: intermittent capture failure | Uncertain, hours to days | Retain sanitized checkpoint reason under controlled/full-suite scheduling; distinguish timeout from replica-proof defect before changing behavior. |
+| [[https://github.com/ligon/sucoder/issues/77][77]] | Partially addressed | Medium: pool creation and turnover convenience remain | Multi-part, high confidence on completed slice | Narrow to the residual below; verify explicit creation, named-node and maintenance-release behavior. |
+| [[https://github.com/ligon/sucoder/issues/72][72]] | Resolved candidate | Original launch gap covered | Small reconciliation, high confidence | Closure draft below; broader runner reliability remains separately tracked. |
+| [[https://github.com/ligon/sucoder/issues/79][79]] | Resolved candidate for bounded correction | Original refusal was high impact | Small reconciliation, bounded native evidence | Closure draft below retains historical timing uncertainty and separate publication failures. |
+
+Issue 70 has the highest residual correctness impact. Issue 82 was selected
+first because it is independently reproducible and has a narrow fix and complete
+verification available without changing checkpoint or placement policy. Then
+address 70, diagnose 76, and implement the remaining 77 capability. Neither
+70 nor 76 blocks source packaging. PR 80 fixes readiness, not final sealing.
+
+* Evidence and proposed tracker reconciliation
+
+These are drafts, not applied issue-body changes or comments. No issue was
+closed, relabeled, assigned or split in this batch.
+
+** Issue 72: proposed closure rationale
+
+PR 74 merged the supported compute-side dispatch CLI, private receiving roots,
+managed runner, credential isolation, descriptor/mount checks, documented
+invocation and honest result reporting. Its installed-wheel concurrent pair
+used jobs 40504297 (Codex, n0028) and 40504303 (Claude, n0033), with independent
+identities, a shared concurrency barrier and unchanged source metadata.
+Codex observed local SQLite and sealed; Claude preserved stopped text state.
+Fake-worker cross-node tests covered dirty snapshots, TERM and missing results.
+PR 81 additionally exercised two concurrent native Codex workers on distinct
+nodes through sealed preservation and verified publication. Deterministic tests
+cover stale bindings and shared-filesystem refusal. See
+[[https://github.com/ligon/sucoder/pull/74][PR 74]], [[file:dispatch.org][dispatch guidance]],
+and [[file:dispatch-exit-proof-validation.org][native pair and guard regressions]].
+The supported entry-point gap is covered on main. Issue 70's separate final-seal
+race remains open; no claim of universal workload reliability is made.
+
+** Issue 79: proposed closure rationale and distinct follow-ups
+
+PR 81 (=641d8fdb=, merged into main) fixes exit-phase inspection handling using
+fresh identity-bound terminal proof. Deterministic regressions cover delayed
+I/R/D exit, identity mismatch and live unreadability; no live-process exemption
+is introduced. Profile-matched jobs 40621405 and 40621415 on different nodes
+overlapped, completed source churn and sealed/published successfully, recovering
+four actual metadata errors with terminal proof. Historical I-state duration
+was not measured; native observed recoveries were below 50 ms. Longer-than-budget
+or persistent live unreadability still refuses. These limits are explicit in
+[[file:dispatch-exit-proof-validation.org][the acceptance record]].
+PR 78's longer pool campaign adds sibling/replacement acceptance, not a substitute
+for the separate-allocation pilot. Later issue comments explicitly identify
+DVC cache misplacement, evidence size/mode violations and non-fast-forward result
+publication as different failures. Do not fold these into process-exit diagnosis.
+Suggested distinct follow-up: document/preflight evidence privacy and size, and
+define clean-base branch publication/recovery. No new follow-up issue was filed;
+retain the original comment links and do not reclassify failed tasks as successes.
+
+** Issue 77: proposed residual title and body
+
+Title: Create named-node allocation pools and complete maintenance turnover.
+
+Body draft: PR 78 merged attachment to owned single-node allocations, private
+worker scopes, immutable source/image reuse, aggregate CPU/memory reservations,
+exact step receipts and cleanup, status reconciliation and submission cutoffs.
+The [[file:dispatch-pool-acceptance.org][bounded native acceptance]] ran two workers
+plus a later replacement in job 40631087: survivor 660 cycles, interrupted worker
+61, replacement 180. Shared image/source identity and private runtime/database
+scopes were verified; one step's termination did not stop the sibling. Stopped
+native-state hashes and evidence were independently checked. Intentional TERM
+preserved a crash generation, not an automatically restorable sealed state.
+The old October 26 all-worker failure checkpoint remains historical evidence;
+it is superseded for this bounded attach/replacement acceptance.
+
+Remaining: supported allocation creation with explicit requested-node selection;
+owner-controlled, bounded idle holds and maintenance turnover that stops new work,
+publishes commits/evidence/state/handoffs, then releases while storage is available;
+and explicit acceptance of approved immutable input-copy reuse when added beyond
+the currently measured source/image assets. Preserve original uid/cluster/node,
+expiry/stale-root, resource, state-isolation and ordinary-dispatch requirements.
+The source-only three-worker campaign does not demonstrate eight-country capacity,
+real-data reuse, automatic recovery or a long maintenance hold.
+
+** Issues 70 and 76: bounded next checks
+
+At the inspected base, =codex_state_runner.run= selects class and calls
+=holders_of(db_dir)= before =Checkpointer.final= drains in-flight capture.
+The owned-reader ordering defect remains; issue 73's readiness reset does not
+change it. Start with an event-held reader regression and retain the same shutdown
+budget and conservative treatment of other holders. A broader fix belongs in
+its own PR with real sealed restore/reuse acceptance.
+
+For 76, the final assertion in
+=test_a_thread_discovered_after_the_backup_waits_for_its_replica= still omits the
+checkpoint reason. The original reason is unavailable and six isolated controls
+passed. Record only checkpoint outcome/reason and retained fixture diagnostics;
+do not expose inherited environment through full pytest locals. Preserve both
+the first non-durable result and full immutable-prefix verification. There is
+no evidence that raising the timeout or accepting failure would be correct.
+
+* Issue 82 implementation and verification
+
+Reuse decision and invariants: [[file:dispatch-ledger.org][dispatch ledger]], sections 2–5.
+Ordinary transport clone only copies reachable source refs. After cloning,
+=submit= now fetches the already-resolved SHA from the explicit local source
+into the private repository using the existing sanitized, file-only Git helper,
+with tags/submodule recursion/FETCH_HEAD writes disabled and the existing timeout.
+The existing private branch update then anchors the commit. Pool local-copy
+staging already includes these objects and retains its independent checks.
+
+On unchanged =d2556f0= production code, both new loose/packed regressions failed
+at the reported missing-commit check. With the correction both pass, verify the
+selected tree and private branch, run connectivity checks, and compare every
+source file's bytes before/after. Error/timeout tests prove one staging attempt,
+no scheduler call, and no submission receipt. No native worker or production
+research repository is needed for this pre-scheduler failure.
+
+From the isolated worktree root:
+#+begin_src sh
+pytest -q -o showlocals=false --tb=short tests/test_dispatch*.py
+#+end_src
+Result: 387 passed in 66.78 seconds. The default =python= lacked pytest; the
+available pytest executable ran the suite successfully against this worktree.
+No new full-suite or native-allocation acceptance is claimed for this narrow fix.
+Self-review: OK anchored on ledger sections 2–5; reused Git sanitization and
+timeouts, preserved source immutability, and kept pool/worker lifecycle unchanged.
+No independent reviewer was used. GitNexus impact and detect-changes were attempted
+but failed to load =libssl.so.3=; manual callers and diff review substitute for
+unavailable graph results. Production change is confined to =dispatch.submit=.

docs/dispatch-ledger.org

@@ -1,5 +1,5 @@
 #+TITLE: Prior-art ledger - headless worker dispatch
-#+DATE: 2026-10-25
+#+DATE: 2026-10-30
 #+AUTHOR: Sue
 
 * 1. Task and evidence
@@ -12,8 +12,15 @@ Search uses git, ripgrep, AST inspection, installed Claude help, and existing te
 GitNexus upstream impact for =cli.main= was attempted before editing; its existing native dependency cannot load =libssl.so.3= on this host.
 Do not claim a successful graph analysis; review source callers and execution routes and run the command again before committing.
 
+Current batch: issue 82, based on main =d2556f0=. Ordinary transport cloning
+omits locally available commits that have no advertised source ref. Extend
+private snapshot preparation to fetch the already-resolved exact commit from
+the same local source; never anchor a ref in the source.
+
 * 2. Existing machinery and tests
 
+- =sucoder/dispatch.py:submit= :: Resolves the immutable commit before dry-run returns; ordinary snapshots use transport clone, pool snapshots copy local objects. Direct callers are =dispatch_cli.submit= and =dispatch_pool.submit=. Existing source-isolation test: =test_submit_pins_private_source_and_allowlisted_secrets=.
+- =sucoder/dispatch.py:_git= :: Reuse sanitized Git environment, disabled lazy fetching, explicit file-only transport and bounded subprocess timeout. New SHA-only source regression must verify the selected tree, private branch, source bytes and no scheduler call on staging failure.
 - =sucoder/local_tier.py:local_root_shell=, =work_path=, =work_path_shell=, =cache_exports_sh= :: Reuse receiving-allocation paths and cache environment. Covered by =tests/test_local_tier.py=.
 - =sucoder/local_tier.py:POST_COMMIT_HOOK= :: Reuse nonforced commit publication, with an absolute private publication target pinned independently of mutable origin configuration. Never install this against the source repository.
 - =sucoder/local_tier.py:build_prepare_script= :: Existing interactive preparation follows the shared branch, reconciles and restores WIP. A headless worker needs a separate pinned path; do not silently change interactive semantics.
@@ -52,6 +59,7 @@ Preserve current interactive launch, synchronization and recovery behavior.
 
 * 5. Reuse decisions
 
+- Extend :: Ordinary =submit= snapshot staging with an exact-SHA local fetch before its existing commit check/private ref update. Pool local-copy already includes unreferenced objects and keeps its ownership/connectivity checks.
 - Reuse :: Local path/cache helpers, nonforced commit hook, managed Codex runner and its standard-library descriptor helpers.
 - Extend :: Shared local-tier preparation with an exact-commit worker path, keeping the interactive path intact.
 - New :: Dispatch manifest, private job/result directory and bounded Slurm submission/status/wait interfaces, because interactive session records and tmux panes do not represent headless jobs.
@@ -61,6 +69,11 @@ Preserve current interactive launch, synchronization and recovery behavior.
 
 * 6. Validation and decisions
 
+Issue 82 has no unresolved design choice. GitNexus impact was attempted again
+on 2026-10-30 and failed to load =libssl.so.3=; direct caller/search inspection
+shows a moderate staging risk shared by CLI and pool callers, with the new
+fetch restricted to ordinary dispatch. No scheduler or worker lifecycle changes.
+
 The first adapters are Codex and Claude, selected by the user.
 Unit tests must cover stale inherited bindings, source pinning, private results, refusal of shared state placement, bounded submission/wait failures and exact exit propagation.
 Use fake executables for deterministic tests and separate Slurm allocations for real-agent acceptance, including concurrent workers on different nodes.

sucoder/dispatch.py

@@ -290,6 +290,11 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None,
     project_files = stage_profile(project, directory) if project is not None else []
     # Private self-contained source snapshot. No source ref/config/index changes.
     _git(repo, *(["-c", "pack.threads=1"] if _execution is not None else []), "clone", "--bare", "--local" if _execution is not None else "--no-local", "--no-hardlinks", str(repo), str(directory / "results.git"), timeout=timeout, file_protocol=True)
+    if _execution is None:
+        # Transport clone copies advertised refs, not every locally held SHA.
+        # Reuse the exact pin and file-only transport (dispatch ledger, §§2–4).
+        _git(directory / "results.git", "fetch", "--no-tags", "--no-write-fetch-head",
+             "--no-recurse-submodules", str(repo), commit, timeout=timeout, file_protocol=True)
     _git(directory / "results.git", "cat-file", "-e", commit + "^{commit}")
     if _execution is not None:
         report["source_copy"] = verify_copy(str(common), str(directory / "results.git"))

tests/test_dispatch.py

@@ -165,6 +165,64 @@ def test_submit_pins_private_source_and_allowlisted_secrets(source, monkeypatch)
     assert (git(source["repo"], "show-ref"), git(source["repo"], "status", "--porcelain"), (source["repo"] / ".git/index").read_bytes(), (source["repo"] / ".git/config").read_bytes()) == before
 
 
+@pytest.mark.parametrize("packed", [False, True])
+def test_submit_snapshots_commit_available_only_by_sha(source, monkeypatch, tmp_path, packed):
+    repo = source["repo"]
+    donor = tmp_path / "donor"
+    git(repo, "clone", "-q", str(repo), str(donor))
+    git(donor, "config", "user.name", "Test")
+    git(donor, "config", "user.email", "test@example.com")
+    (donor / "selected-only").write_text("selected tree\n")
+    git(donor, "add", ".")
+    git(donor, "commit", "-qm", "selected")
+    selected = git(donor, "rev-parse", "HEAD")
+    git(repo, "-c", "fetch.unpackLimit=" + ("1" if packed else "100"),
+        "fetch", "--no-write-fetch-head", "--no-tags", str(donor), selected)
+    assert git(repo, "cat-file", "-t", selected) == "commit"
+    assert not git(repo, "for-each-ref", "--contains=" + selected)
+    assert bool(list((repo / ".git/objects/pack").glob("*.pack"))) == packed
+    before = {str(p.relative_to(repo)): p.read_bytes()
+              for p in repo.rglob("*") if p.is_file()}
+    calls = scheduler(monkeypatch)
+    source["ref"] = selected
+    dry = dispatch.submit(**source, dry_run=True)
+    assert dry["commit"] == selected
+    assert not Path(dry["directory"]).exists() and not calls
+    report = dispatch.submit(**source)
+    results = Path(report["directory"]) / "results.git"
+    assert len(calls) == 1
+    assert git(results, "rev-parse", "dispatch/" + report["dispatch_id"]) == selected
+    assert git(results, "show", selected + ":selected-only") == "selected tree"
+    git(results, "fsck", "--connectivity-only")
+    assert not (results / "objects/info/alternates").exists()
+    assert before == {str(p.relative_to(repo)): p.read_bytes()
+                      for p in repo.rglob("*") if p.is_file()}
+
+
+@pytest.mark.parametrize("timed_out", [False, True])
+def test_source_fetch_failure_never_reaches_scheduler(source, monkeypatch, timed_out):
+    calls = scheduler(monkeypatch)
+    real_run = dispatch._run
+    fetches = []
+
+    def run(argv, **kwargs):
+        if argv[0] == "git" and "fetch" in argv:
+            fetches.append(argv)
+            assert kwargs["timeout"] == 17
+            assert kwargs["env"]["GIT_ALLOW_PROTOCOL"] == "file"
+            assert kwargs["env"]["GIT_NO_LAZY_FETCH"] == "1"
+            if timed_out:
+                raise subprocess.TimeoutExpired(argv, 17)
+            return SimpleNamespace(returncode=1, stdout="", stderr="selected commit unavailable")
+        return real_run(argv, **kwargs)
+
+    monkeypatch.setattr(dispatch, "_run", run)
+    with pytest.raises(subprocess.TimeoutExpired if timed_out else ValueError):
+        dispatch.submit(**source, timeout=17)
+    assert len(fetches) == 1 and not calls
+    assert not list(dispatch.DEFAULT_DIRECTORY_ROOT.glob("*/submission.json"))
+
+
 def test_submission_timeout_is_unknown_and_never_retries(source, monkeypatch):
     calls = scheduler(monkeypatch, error=subprocess.TimeoutExpired("sbatch", 1))
     report = dispatch.submit(**source, timeout=1)