diff --git a/docs/dispatch-backlog-20261030.org b/docs/dispatch-backlog-20261030.org new file mode 100644 index 0000000..46cc415 --- /dev/null +++ b/docs/dispatch-backlog-20261030.org @@ -0,0 +1,165 @@ +#+TITLE: Dispatch backlog: reconciliation and SHA-only source fix +#+DATE: 2026-10-30 + +* Scope and coverage + +Repository: [[https://github.com/ligon/sucoder][ligon/sucoder]], default and merge target =main=. +Inspected implementation base: =d2556f0dc84f40b239a8e014934d0af821087a6e=, +including merged PRs 74, 75, 81 and 78. No development/release branch is targeted. +This implementation batch follows the user's request to address dispatch issues +using backlog-workflow. It selects issue 82; the other issues retain independent +acceptance criteria and are reconciled below, not silently treated as fixed. + +Inventory retrieved 2026-10-30, approximately 13:10–13:20 UTC: 14 open issues, +one open PR (80). REST returned 15 issue/PR entries with per_page=100: one +complete page. All issues have no labels or assignees; no priority labels apply. +Connector issue search returned issue records even for a PR-only query, so +explicit-repository gh REST/list commands supplied completeness and PR checks. +Bodies and all comments were read for dispatch issues 70, 72, 76, 77, 79 and 82; +code, relevant merged PRs and acceptance records were compared. No sampling of +those six. Seven unrelated issues (14, 54, 57, 58, 59, 65, 66) are metadata-only. +Related issue 73 is tracked through PR 80 and the earlier +[[file:backlog-20261029.org][backlog report]], rather than duplicated. +Timeline cross-reference graphs were not exhaustively traversed; explicit +links, merged dispatch PR search and the complete open-PR list were checked. + +The user subsequently authorized merging PR 80. Its exact head =58586ba3= +had green CI run [[https://github.com/ligon/sucoder/actions/runs/40990436269][40990436269]], +no review submissions or review threads, and a mergeable state. Source/tests and +commit messages were inspected. Merge succeeded as =c09d36161b93162fc75d4534dfa1a2cbdf2b9ac9=. +Issue 73 therefore becomes a resolved candidate; issue state was not changed. +This update supersedes the initial open-PR inventory, not its retrieval record. + +* Dispositions and ordered work + +Counts for the six examined dispatch issues: two resolved candidates, one +partially addressed, one needs diagnosis, one ready, one implemented pending PR. +No shared root cause or blocking cycle is inferred from their common dispatch context. + +| Issue | Disposition | Impact / urgency | Effort / confidence | Next action and completion criterion | +|-------+-------------+------------------+---------------------+--------------------------------------| +| [[https://github.com/ligon/sucoder/issues/82][82]] | Implemented in this batch; pending merge | Medium: valid sources cannot launch; ref workaround exists | Small, high confidence | Review exact-SHA staging fix; loose/packed unreferenced commits must stage without source mutation, failures before Slurm. | +| [[https://github.com/ligon/sucoder/issues/70][70]] | Ready | High: clean exit may prevent sealed recovery | Substantial lifecycle change, strong ordering evidence | Drain owned captures before class selection within the existing deadline; test other holders, unclean exit, stalls, real restore and reuse. | +| [[https://github.com/ligon/sucoder/issues/76][76]] | Needs diagnosis | Medium: intermittent capture failure | Uncertain, hours to days | Retain sanitized checkpoint reason under controlled/full-suite scheduling; distinguish timeout from replica-proof defect before changing behavior. | +| [[https://github.com/ligon/sucoder/issues/77][77]] | Partially addressed | Medium: pool creation and turnover convenience remain | Multi-part, high confidence on completed slice | Narrow to the residual below; verify explicit creation, named-node and maintenance-release behavior. | +| [[https://github.com/ligon/sucoder/issues/72][72]] | Resolved candidate | Original launch gap covered | Small reconciliation, high confidence | Closure draft below; broader runner reliability remains separately tracked. | +| [[https://github.com/ligon/sucoder/issues/79][79]] | Resolved candidate for bounded correction | Original refusal was high impact | Small reconciliation, bounded native evidence | Closure draft below retains historical timing uncertainty and separate publication failures. | + +Issue 70 has the highest residual correctness impact. Issue 82 was selected +first because it is independently reproducible and has a narrow fix and complete +verification available without changing checkpoint or placement policy. Then +address 70, diagnose 76, and implement the remaining 77 capability. Neither +70 nor 76 blocks source packaging. PR 80 fixes readiness, not final sealing. + +* Evidence and proposed tracker reconciliation + +These are drafts, not applied issue-body changes or comments. No issue was +closed, relabeled, assigned or split in this batch. + +** Issue 72: proposed closure rationale + +PR 74 merged the supported compute-side dispatch CLI, private receiving roots, +managed runner, credential isolation, descriptor/mount checks, documented +invocation and honest result reporting. Its installed-wheel concurrent pair +used jobs 40504297 (Codex, n0028) and 40504303 (Claude, n0033), with independent +identities, a shared concurrency barrier and unchanged source metadata. +Codex observed local SQLite and sealed; Claude preserved stopped text state. +Fake-worker cross-node tests covered dirty snapshots, TERM and missing results. +PR 81 additionally exercised two concurrent native Codex workers on distinct +nodes through sealed preservation and verified publication. Deterministic tests +cover stale bindings and shared-filesystem refusal. See +[[https://github.com/ligon/sucoder/pull/74][PR 74]], [[file:dispatch.org][dispatch guidance]], +and [[file:dispatch-exit-proof-validation.org][native pair and guard regressions]]. +The supported entry-point gap is covered on main. Issue 70's separate final-seal +race remains open; no claim of universal workload reliability is made. + +** Issue 79: proposed closure rationale and distinct follow-ups + +PR 81 (=641d8fdb=, merged into main) fixes exit-phase inspection handling using +fresh identity-bound terminal proof. Deterministic regressions cover delayed +I/R/D exit, identity mismatch and live unreadability; no live-process exemption +is introduced. Profile-matched jobs 40621405 and 40621415 on different nodes +overlapped, completed source churn and sealed/published successfully, recovering +four actual metadata errors with terminal proof. Historical I-state duration +was not measured; native observed recoveries were below 50 ms. Longer-than-budget +or persistent live unreadability still refuses. These limits are explicit in +[[file:dispatch-exit-proof-validation.org][the acceptance record]]. +PR 78's longer pool campaign adds sibling/replacement acceptance, not a substitute +for the separate-allocation pilot. Later issue comments explicitly identify +DVC cache misplacement, evidence size/mode violations and non-fast-forward result +publication as different failures. Do not fold these into process-exit diagnosis. +Suggested distinct follow-up: document/preflight evidence privacy and size, and +define clean-base branch publication/recovery. No new follow-up issue was filed; +retain the original comment links and do not reclassify failed tasks as successes. + +** Issue 77: proposed residual title and body + +Title: Create named-node allocation pools and complete maintenance turnover. + +Body draft: PR 78 merged attachment to owned single-node allocations, private +worker scopes, immutable source/image reuse, aggregate CPU/memory reservations, +exact step receipts and cleanup, status reconciliation and submission cutoffs. +The [[file:dispatch-pool-acceptance.org][bounded native acceptance]] ran two workers +plus a later replacement in job 40631087: survivor 660 cycles, interrupted worker +61, replacement 180. Shared image/source identity and private runtime/database +scopes were verified; one step's termination did not stop the sibling. Stopped +native-state hashes and evidence were independently checked. Intentional TERM +preserved a crash generation, not an automatically restorable sealed state. +The old October 26 all-worker failure checkpoint remains historical evidence; +it is superseded for this bounded attach/replacement acceptance. + +Remaining: supported allocation creation with explicit requested-node selection; +owner-controlled, bounded idle holds and maintenance turnover that stops new work, +publishes commits/evidence/state/handoffs, then releases while storage is available; +and explicit acceptance of approved immutable input-copy reuse when added beyond +the currently measured source/image assets. Preserve original uid/cluster/node, +expiry/stale-root, resource, state-isolation and ordinary-dispatch requirements. +The source-only three-worker campaign does not demonstrate eight-country capacity, +real-data reuse, automatic recovery or a long maintenance hold. + +** Issues 70 and 76: bounded next checks + +At the inspected base, =codex_state_runner.run= selects class and calls +=holders_of(db_dir)= before =Checkpointer.final= drains in-flight capture. +The owned-reader ordering defect remains; issue 73's readiness reset does not +change it. Start with an event-held reader regression and retain the same shutdown +budget and conservative treatment of other holders. A broader fix belongs in +its own PR with real sealed restore/reuse acceptance. + +For 76, the final assertion in +=test_a_thread_discovered_after_the_backup_waits_for_its_replica= still omits the +checkpoint reason. The original reason is unavailable and six isolated controls +passed. Record only checkpoint outcome/reason and retained fixture diagnostics; +do not expose inherited environment through full pytest locals. Preserve both +the first non-durable result and full immutable-prefix verification. There is +no evidence that raising the timeout or accepting failure would be correct. + +* Issue 82 implementation and verification + +Reuse decision and invariants: [[file:dispatch-ledger.org][dispatch ledger]], sections 2–5. +Ordinary transport clone only copies reachable source refs. After cloning, +=submit= now fetches the already-resolved SHA from the explicit local source +into the private repository using the existing sanitized, file-only Git helper, +with tags/submodule recursion/FETCH_HEAD writes disabled and the existing timeout. +The existing private branch update then anchors the commit. Pool local-copy +staging already includes these objects and retains its independent checks. + +On unchanged =d2556f0= production code, both new loose/packed regressions failed +at the reported missing-commit check. With the correction both pass, verify the +selected tree and private branch, run connectivity checks, and compare every +source file's bytes before/after. Error/timeout tests prove one staging attempt, +no scheduler call, and no submission receipt. No native worker or production +research repository is needed for this pre-scheduler failure. + +From the isolated worktree root: +#+begin_src sh +pytest -q -o showlocals=false --tb=short tests/test_dispatch*.py +#+end_src +Result: 387 passed in 66.78 seconds. The default =python= lacked pytest; the +available pytest executable ran the suite successfully against this worktree. +No new full-suite or native-allocation acceptance is claimed for this narrow fix. +Self-review: OK anchored on ledger sections 2–5; reused Git sanitization and +timeouts, preserved source immutability, and kept pool/worker lifecycle unchanged. +No independent reviewer was used. GitNexus impact and detect-changes were attempted +but failed to load =libssl.so.3=; manual callers and diff review substitute for +unavailable graph results. Production change is confined to =dispatch.submit=. diff --git a/docs/dispatch-ledger.org b/docs/dispatch-ledger.org index 9e66fe8..8e2fd31 100644 --- a/docs/dispatch-ledger.org +++ b/docs/dispatch-ledger.org @@ -1,5 +1,5 @@ #+TITLE: Prior-art ledger - headless worker dispatch -#+DATE: 2026-10-25 +#+DATE: 2026-10-30 #+AUTHOR: Sue * 1. Task and evidence @@ -12,8 +12,15 @@ Search uses git, ripgrep, AST inspection, installed Claude help, and existing te GitNexus upstream impact for =cli.main= was attempted before editing; its existing native dependency cannot load =libssl.so.3= on this host. Do not claim a successful graph analysis; review source callers and execution routes and run the command again before committing. +Current batch: issue 82, based on main =d2556f0=. Ordinary transport cloning +omits locally available commits that have no advertised source ref. Extend +private snapshot preparation to fetch the already-resolved exact commit from +the same local source; never anchor a ref in the source. + * 2. Existing machinery and tests +- =sucoder/dispatch.py:submit= :: Resolves the immutable commit before dry-run returns; ordinary snapshots use transport clone, pool snapshots copy local objects. Direct callers are =dispatch_cli.submit= and =dispatch_pool.submit=. Existing source-isolation test: =test_submit_pins_private_source_and_allowlisted_secrets=. +- =sucoder/dispatch.py:_git= :: Reuse sanitized Git environment, disabled lazy fetching, explicit file-only transport and bounded subprocess timeout. New SHA-only source regression must verify the selected tree, private branch, source bytes and no scheduler call on staging failure. - =sucoder/local_tier.py:local_root_shell=, =work_path=, =work_path_shell=, =cache_exports_sh= :: Reuse receiving-allocation paths and cache environment. Covered by =tests/test_local_tier.py=. - =sucoder/local_tier.py:POST_COMMIT_HOOK= :: Reuse nonforced commit publication, with an absolute private publication target pinned independently of mutable origin configuration. Never install this against the source repository. - =sucoder/local_tier.py:build_prepare_script= :: Existing interactive preparation follows the shared branch, reconciles and restores WIP. A headless worker needs a separate pinned path; do not silently change interactive semantics. @@ -52,6 +59,7 @@ Preserve current interactive launch, synchronization and recovery behavior. * 5. Reuse decisions +- Extend :: Ordinary =submit= snapshot staging with an exact-SHA local fetch before its existing commit check/private ref update. Pool local-copy already includes unreferenced objects and keeps its ownership/connectivity checks. - Reuse :: Local path/cache helpers, nonforced commit hook, managed Codex runner and its standard-library descriptor helpers. - Extend :: Shared local-tier preparation with an exact-commit worker path, keeping the interactive path intact. - New :: Dispatch manifest, private job/result directory and bounded Slurm submission/status/wait interfaces, because interactive session records and tmux panes do not represent headless jobs. @@ -61,6 +69,11 @@ Preserve current interactive launch, synchronization and recovery behavior. * 6. Validation and decisions +Issue 82 has no unresolved design choice. GitNexus impact was attempted again +on 2026-10-30 and failed to load =libssl.so.3=; direct caller/search inspection +shows a moderate staging risk shared by CLI and pool callers, with the new +fetch restricted to ordinary dispatch. No scheduler or worker lifecycle changes. + The first adapters are Codex and Claude, selected by the user. Unit tests must cover stale inherited bindings, source pinning, private results, refusal of shared state placement, bounded submission/wait failures and exact exit propagation. Use fake executables for deterministic tests and separate Slurm allocations for real-agent acceptance, including concurrent workers on different nodes. diff --git a/sucoder/dispatch.py b/sucoder/dispatch.py index a567428..12840fc 100644 --- a/sucoder/dispatch.py +++ b/sucoder/dispatch.py @@ -290,6 +290,11 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, project_files = stage_profile(project, directory) if project is not None else [] # Private self-contained source snapshot. No source ref/config/index changes. _git(repo, *(["-c", "pack.threads=1"] if _execution is not None else []), "clone", "--bare", "--local" if _execution is not None else "--no-local", "--no-hardlinks", str(repo), str(directory / "results.git"), timeout=timeout, file_protocol=True) + if _execution is None: + # Transport clone copies advertised refs, not every locally held SHA. + # Reuse the exact pin and file-only transport (dispatch ledger, §§2–4). + _git(directory / "results.git", "fetch", "--no-tags", "--no-write-fetch-head", + "--no-recurse-submodules", str(repo), commit, timeout=timeout, file_protocol=True) _git(directory / "results.git", "cat-file", "-e", commit + "^{commit}") if _execution is not None: report["source_copy"] = verify_copy(str(common), str(directory / "results.git")) diff --git a/tests/test_dispatch.py b/tests/test_dispatch.py index 8acd555..3cad61c 100644 --- a/tests/test_dispatch.py +++ b/tests/test_dispatch.py @@ -165,6 +165,64 @@ def test_submit_pins_private_source_and_allowlisted_secrets(source, monkeypatch) assert (git(source["repo"], "show-ref"), git(source["repo"], "status", "--porcelain"), (source["repo"] / ".git/index").read_bytes(), (source["repo"] / ".git/config").read_bytes()) == before +@pytest.mark.parametrize("packed", [False, True]) +def test_submit_snapshots_commit_available_only_by_sha(source, monkeypatch, tmp_path, packed): + repo = source["repo"] + donor = tmp_path / "donor" + git(repo, "clone", "-q", str(repo), str(donor)) + git(donor, "config", "user.name", "Test") + git(donor, "config", "user.email", "test@example.com") + (donor / "selected-only").write_text("selected tree\n") + git(donor, "add", ".") + git(donor, "commit", "-qm", "selected") + selected = git(donor, "rev-parse", "HEAD") + git(repo, "-c", "fetch.unpackLimit=" + ("1" if packed else "100"), + "fetch", "--no-write-fetch-head", "--no-tags", str(donor), selected) + assert git(repo, "cat-file", "-t", selected) == "commit" + assert not git(repo, "for-each-ref", "--contains=" + selected) + assert bool(list((repo / ".git/objects/pack").glob("*.pack"))) == packed + before = {str(p.relative_to(repo)): p.read_bytes() + for p in repo.rglob("*") if p.is_file()} + calls = scheduler(monkeypatch) + source["ref"] = selected + dry = dispatch.submit(**source, dry_run=True) + assert dry["commit"] == selected + assert not Path(dry["directory"]).exists() and not calls + report = dispatch.submit(**source) + results = Path(report["directory"]) / "results.git" + assert len(calls) == 1 + assert git(results, "rev-parse", "dispatch/" + report["dispatch_id"]) == selected + assert git(results, "show", selected + ":selected-only") == "selected tree" + git(results, "fsck", "--connectivity-only") + assert not (results / "objects/info/alternates").exists() + assert before == {str(p.relative_to(repo)): p.read_bytes() + for p in repo.rglob("*") if p.is_file()} + + +@pytest.mark.parametrize("timed_out", [False, True]) +def test_source_fetch_failure_never_reaches_scheduler(source, monkeypatch, timed_out): + calls = scheduler(monkeypatch) + real_run = dispatch._run + fetches = [] + + def run(argv, **kwargs): + if argv[0] == "git" and "fetch" in argv: + fetches.append(argv) + assert kwargs["timeout"] == 17 + assert kwargs["env"]["GIT_ALLOW_PROTOCOL"] == "file" + assert kwargs["env"]["GIT_NO_LAZY_FETCH"] == "1" + if timed_out: + raise subprocess.TimeoutExpired(argv, 17) + return SimpleNamespace(returncode=1, stdout="", stderr="selected commit unavailable") + return real_run(argv, **kwargs) + + monkeypatch.setattr(dispatch, "_run", run) + with pytest.raises(subprocess.TimeoutExpired if timed_out else ValueError): + dispatch.submit(**source, timeout=17) + assert len(fetches) == 1 and not calls + assert not list(dispatch.DEFAULT_DIRECTORY_ROOT.glob("*/submission.json")) + + def test_submission_timeout_is_unknown_and_never_retries(source, monkeypatch): calls = scheduler(monkeypatch, error=subprocess.TimeoutExpired("sbatch", 1)) report = dispatch.submit(**source, timeout=1)