From 45286aed52e279eb67af8fe411b7e55a79c224af Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 22:13:03 -0700 Subject: [PATCH 01/18] Record allocation-pool dispatch reuse plan for issue 77 Co-Authored-By: GPT-6 --- docs/dispatch-pool-ledger.org | 44 +++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 docs/dispatch-pool-ledger.org diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org new file mode 100644 index 0000000..58132f2 --- /dev/null +++ b/docs/dispatch-pool-ledger.org @@ -0,0 +1,44 @@ +#+title: Existing-allocation worker dispatch -- issue 77 + +* Scope and existing machinery +Based on aa79d455447f9b297e425a1ec874fbb5c284faad, including accepted +34070a0. Extend dispatch submission, private publication, managed runner, +project profiles and receiving-node placement checks. Do not launch raw Codex. +No scientific LSMS changes, pool creation daemon, shared installation or runtime +mount ownership change is included. + +* Ownership and invariants +JobOwnership remains the default for independent jobs. A validated pool binding +selects StepOwnership(job_id, step_id, proc_root, uid, supervisor_pid), matching +numeric step components in every observed Slurm hierarchy and PID incarnation. +All existing cleanup call sites use that owner; no pool action cancels a job. +Each worker uses /local/job/workers/ for managed state, SQLite, +checkout, data, caches and temporary files. Shared immutable image bytes may be +read from an explicitly pinned allocation asset cache; mounts remain private. + +* Reuse plan +Reuse dispatch.submit staging and durable results.git publication. An internal +execution binding changes only scheduling to srun in an already owned allocation. +Pool attach validates scheduler uid, node, capacity and end time. Atomic mkdir +locking serializes conservative CPU/memory reservations; unknown submissions +retain budget until exact step terminal proof. No age-based lock stealing. +Reuse project profiles without new generic scientific or credential policy. + +* Verification plan +Fake /proc and scheduler tests cover step 4 versus 40, sibling/batch/extern, +UID/incarnation changes, inconsistent hierarchies, resource overcommit, +expiry, ambiguous submission, asset identity and cleanup. Existing job dispatch +tests must retain behavior. Native/scheduler acceptance belongs to root release. + +* Impact map and tooling limitation +Before edits, gitnexus impact JobOwnership --direction upstream failed because +its native LadybugDB module cannot load libssl.so.3, the same limitation recorded +in dispatch-ledger.org. No shared tool repair. Manual review covers every +stop_allocation call in dispatch_worker.run, receiving_root/environment_for, +local_tier prepare/snapshot builders, dispatch receipt/status matching and CLI. +Step cleanup has high impact: accidentally selecting JobOwnership would signal +coordinator and sibling workers. Explicit binding and negative tests are required. + +* Outcome +Implementation and deterministic test results will be recorded additively. +No actual worker or scheduler acceptance has run under this change. From 0340f0b1fe3c319db9ed2e26ff9061d9afcdabba Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 22:29:33 -0700 Subject: [PATCH 02/18] Add step-owned workers in existing allocation pools for issue 77 Preserve independent job ownership; reserve aggregate budgets conservatively and publish exact receiving step identities. Reuse managed runners, private results repositories and immutable allocation-local source/image assets. Co-Authored-By: GPT-6 --- README.org | 4 + docs/dispatch-pool-ledger.org | 27 +++ docs/dispatch-pools.org | 82 ++++++++ sucoder/dispatch.py | 44 +++- sucoder/dispatch_cli.py | 4 + sucoder/dispatch_pool.py | 347 ++++++++++++++++++++++++++++++++ sucoder/dispatch_pool_assets.py | 77 +++++++ sucoder/dispatch_pool_cli.py | 86 ++++++++ sucoder/dispatch_worker.py | 87 +++++++- sucoder/local_tier.py | 14 +- tests/test_dispatch_pool.py | 343 +++++++++++++++++++++++++++++++ 11 files changed, 1103 insertions(+), 12 deletions(-) create mode 100644 docs/dispatch-pools.org create mode 100644 sucoder/dispatch_pool.py create mode 100644 sucoder/dispatch_pool_assets.py create mode 100644 sucoder/dispatch_pool_cli.py create mode 100644 tests/test_dispatch_pool.py diff --git a/README.org b/README.org index 89e2c30..b949178 100644 --- a/README.org +++ b/README.org @@ -268,6 +268,10 @@ Workers use a pinned source commit, a private branch and result repository, and their receiving node's local storage for writable agent state and caches. The source checkout stays untouched; the coordinator reviews returned work before integration. +=sucoder dispatch pool= adds explicitly budgeted, step-owned workers inside an +already held allocation. See [[file:docs/dispatch-pools.org][existing-allocation pools]] for isolation, +immutable assets, conservative reservations and stop semantics. + See [[file:docs/dispatch.org][headless dispatch]] for the complete submission, status, wait and result commands and the supported state guarantees. diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index 58132f2..715c243 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -42,3 +42,30 @@ coordinator and sibling workers. Explicit binding and negative tests are require * Outcome Implementation and deterministic test results will be recorded additively. No actual worker or scheduler acceptance has run under this change. + +** Source implementation qualification +Reuse now includes immutable base cloning followed by an exact-SHA fetch from +the already staged private results.git. This supports a newly authored reviewer +candidate without changing cache refs. Both source and worker-child actual +mounts are checked before managed state/project writes. +Slurm step state is queried through scontrol, not job-format %T on squeue -s. +Blank accounting step UID is accepted only with exact parent UID/node/StartTime, +receiving step proof, and exact step node/start-time bounds. A terminal parent +alone is never a worker release proof. +The requested duration and global cutoff both bound the publication deadline. +New Git transfers pin pack.threads=1 only on the pool path. + +** Deterministic verification before scheduler release +Fake-only suite v8: 282 passed, 480 pre-existing Typer deprecation warnings. +Final targeted suite v9: 52 passed, including independently authored ownership +and actual-Slurm-output fixtures. No real Slurm or native model was started. +Logs/JUnit are under /local/job40498884/control/pool77-tests-v*. Earlier failed +preparations remain: missing pytest in the old private interpreter; test-fixture +import path; and a synthetic step date that was future in the local timezone. +The latter was corrected in test data only, preserving start-time refusal. +Final full exact-commit confirmation will be recorded separately. +GitNexus detect-changes was attempted before commit and failed with the same +libssl.so.3 native-module error; no tool installation was changed. Manual +call-site review and independent source review cover the high-impact ownership +boundary. Actual supervisor/sibling/daemon survival and native local-SQLite +placement remain root-owned release gates. diff --git a/docs/dispatch-pools.org b/docs/dispatch-pools.org new file mode 100644 index 0000000..0a39d2a --- /dev/null +++ b/docs/dispatch-pools.org @@ -0,0 +1,82 @@ +#+title: Workers in an existing Slurm allocation + +* Scope +=dispatch pool= attaches to an already held, owned, running single-node allocation. +It does not create or release allocations, install tools, or start raw Codex. +The ordinary =dispatch submit= whole-job path is unchanged. +Each pool worker is one numeric Slurm step with its own receiving-node root: +=/local/job/workers/=. +Managed SQLite, HOME, checkout, data, caches and temporary files live below it. +Do not use a project bootstrap that hardcodes the allocation-wide runtime path. + +* Attach and submit +Use a new private directory on shared storage for each allocation pool. +CPU and memory budgets below are the TOTAL worker budget, after subtracting +coordinator, keeper and verifier capacity. The pool does not change the parent +batch step's affinity or prove that its mask is disjoint from the workers. +The scheduler must support exclusive exact steps inside the held allocation. +Node names must match Slurm's NodeList and receiving SLURMD_NODENAME exactly. +Cutoffs are finite Unix timestamps, interpreted against the scheduler EndTime. + +#+begin_src sh +sucoder dispatch pool attach /shared/private/pool-A \ + --job-id 123 --node node.example --cpus 4 --mem 32G \ + --submit-before 1793000000 --worker-deadline 1793003600 --expires 1793005400 \ + --source /local/job123/assets/source.git \ + --image /local/job123/assets/venv.sqfs --image-sha256 FULL_SHA256 +sucoder dispatch --root /shared/private/workers pool submit \ + /shared/private/pool-A /shared/full/source-repository \ + --ref FULL_COMMIT_SHA --task-file /shared/private/task.org \ + --agent codex --executable /shared/tools/codex \ + --agent-home /shared/private/cold-auth-home \ + --project-profile /shared/private/profile.json \ + --cpus 1 --mem 8G --time 01:00:00 +#+end_src + +An explicit private, self-contained =results.git= is staged before launch. +Receiving preparation clones the immutable allocation cache, then fetches the +exact requested SHA from that worker's private results repository. A new author +commit can therefore be reviewed without mutating the allocation cache. +No moving branch or remote provider hydration is used. +Cold credentials retain the existing per-worker private seeding contract and +are never part of the allocation asset cache. + +* Asset and bootstrap contract +The controller records allocation-local paths but cannot inspect another node's +=/local=. The receiver verifies the image SHA256, read-only mode and actual local +mount; the bare source must be full, self-contained and non-promisor. +A private allocation-local receipt plus atomic mkdir lock verifies image bytes +and source connectivity once per pool. Subsequent workers verify the recorded +inode/size/mtime/mode and source refs. This is cooperative same-uid immutability, +not protection against a malicious process with the same Unix identity. +No mount daemon is shared: a project may mount the immutable image privately. + +The supervisor supplies these protected bootstrap inputs: +- =SUCODER_LOCAL_ROOT=: unique worker-local root. +- =SUCODER_POOL_IMAGE=: verified allocation-local image file. +- =SUCODER_POOL_IMAGE_SHA256=: pinned image digest. +- =SUCODER_POOL_SOURCE_REPOSITORY=: immutable allocation-local bare source cache. + +* Observation, stop and conservative reservations +=dispatch pool status POOL= reconciles exact receiving step receipts and queries +that step, never the parent job or a sibling with a similar numeric prefix. +=dispatch pool stop-worker POOL DISPATCH_ID= requests TERM for =job.step= only. +The supervisor traps TERM, enumerates every owned step member (including detached +daemons), rechecks PID incarnation before signals, and publishes stopped state. +All error/normal-exit cleanup uses the same step owner. No pool command cancels +the allocation. Ordinary dispatch status/result/logs also accept pooled workers. + +CPU/memory reservations are serialized by atomic directory creation, not NFS +flock. Locks are not stolen by age. A launch without a receiving receipt remains +submission_unknown; it is never retried automatically. A late receiving receipt +can be reconciled without launching anything. Budget is released only on exact +step-terminal scheduler evidence, not native exit zero or a task completion file. +Unknown scheduler state and pre-submission failures retain their reservation; +manual reconciliation is required rather than assuming no step exists. +The detached srun frontend remains on the submitting host; loss of that host is +not claimed to provide portable frontend recovery or automatic resubmission. + +The worker begins shutdown 120 seconds before the earlier requested-duration or +pool worker deadline. Slurm's time request is an additional hard bound. +Publication receipts distinguish complete, incomplete and unverified outcomes; +no CLI exit alone proves scientific or infrastructure acceptance. diff --git a/sucoder/dispatch.py b/sucoder/dispatch.py index 9237d59..3d589f2 100644 --- a/sucoder/dispatch.py +++ b/sucoder/dispatch.py @@ -161,7 +161,7 @@ def _resources(values): def submit(repo, ref, task_file, agent, executable, resources, *, model=None, directory_root=None, local_disk_root="/local", source_agent_home=None, source_claude_user_config=None, required_evidence=None, snapshot_minutes=10, - dry_run=False, timeout=30, project_profile=None): + dry_run=False, timeout=30, project_profile=None, _execution=None): """Pin an offline commit, stage a private worker and submit exactly once. Dry runs validate and report the planned command without writing artifacts @@ -214,7 +214,7 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, commit = _git(repo, "rev-parse", "--verify", ref + "^{commit}") if not re.fullmatch(r"[0-9a-f]{40,64}", commit): raise ValueError("source did not resolve to a commit") - dispatch_id = uuid.uuid4().hex + dispatch_id = _execution["dispatch_id"] if _execution is not None else uuid.uuid4().hex directory = root / dispatch_id cluster = os.environ.get("SLURM_CLUSTER_NAME") or None manifest = dict(schema_version=SCHEMA_VERSION, dispatcher_version=__version__, adapter_version=1, @@ -228,6 +228,11 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, source_agent_home=None, source_claude_user_config=None, resources=resources, scope=scope_id(cluster or "unknown", str(os.getuid()), str(repo), dispatch_id), checkpoint_minutes=10, checkpoint_timeout=30, snapshot_minutes=float(snapshot_minutes)) + if _execution is not None: + from .dispatch_pool import validate_execution + validate_execution(_execution, resources) + manifest["execution"] = _execution + manifest["cluster"] = cluster = _execution["cluster"] source_home = source_agent_home if source_agent_home is not None else Path.home() / (".codex" if agent == "codex" else ".claude") source_home_path = _shared_path(Path(source_home), "source agent home") if source_agent_home is not None and not source_home_path.is_dir(): @@ -250,6 +255,14 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, argv.insert(3, "--qos=" + resources["qos"]) if resources["exclude"] is not None: argv.insert(3, "--exclude=" + resources["exclude"]) + if _execution is not None: + argv = ["srun", "--jobid=" + str(_execution["job_id"]), "--nodelist=" + _execution["node"], + "--exclusive", "--exact", "--nodes=1", "--ntasks=1", "--cpu-bind=cores", + "--cpus-per-task=" + str(resources["cpus"]), "--mem=" + resources["mem"], + "--time=" + resources["time"], "--export=NONE", "--chdir=/tmp", + "--comment=sucoder-dispatch:" + dispatch_id, + "--output=" + str(directory / "slurm.log"), "--error=" + str(directory / "slurm.stderr.log"), + "/bin/bash", str(directory / "batch.sh")] report = dict(schema_version=SCHEMA_VERSION, dispatch_id=dispatch_id, uid=os.getuid(), cluster=cluster, directory=str(directory), commit=commit, command=argv, state="dry_run" if dry_run else "preparing") if project is not None: @@ -265,7 +278,7 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, (directory / "evidence").mkdir(mode=0o700) project_files = stage_profile(project, directory) if project is not None else [] # Private self-contained source snapshot. No source ref/config/index changes. - _git(repo, "clone", "--bare", "--no-local", "--no-hardlinks", str(repo), str(directory / "results.git"), timeout=timeout, file_protocol=True) + _git(repo, *(["-c", "pack.threads=1"] if _execution is not None else []), "clone", "--bare", "--no-local", "--no-hardlinks", str(repo), str(directory / "results.git"), timeout=timeout, file_protocol=True) _git(directory / "results.git", "cat-file", "-e", commit + "^{commit}") # Make the private initial reference explicit; this never touches source. _git(directory / "results.git", "update-ref", "refs/heads/" + manifest["branch"], commit) @@ -276,12 +289,15 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, _private_write(directory / "MESSAGES_TO_AGENT.txt", "Coordinator steering messages appear here.\n") _private_write(directory / "QUESTIONS_FROM_AGENT.txt", "Record questions here; continue independent work.\n") _write_json(directory / "environment.json", {key: os.environ[key] for key in ENVIRONMENT_ALLOWLIST if key in os.environ}) - _private_write(directory / "prepare.sh", build_worker_prepare_script(results_path=str(directory / "results.git"), dispatch_id=dispatch_id, commit=commit, local_disk_root=local_disk_root), True) - _private_write(directory / "snapshot.sh", build_worker_snapshot_script(results_path=str(directory / "results.git"), dispatch_id=dispatch_id, local_disk_root=local_disk_root), True) + _private_write(directory / "prepare.sh", build_worker_prepare_script(results_path=str(directory / "results.git"), dispatch_id=dispatch_id, commit=commit, local_disk_root=local_disk_root, isolated=_execution is not None), True) + _private_write(directory / "snapshot.sh", build_worker_snapshot_script(results_path=str(directory / "results.git"), dispatch_id=dispatch_id, local_disk_root=local_disk_root, isolated=_execution is not None), True) _private_write(directory / "runner.py", runner_source()) _private_write(directory / "worker.py", Path(__file__).with_name("dispatch_worker.py").read_text()) _private_write(directory / "dispatch_protocol.py", Path(__file__).with_name("dispatch_protocol.py").read_text()) _private_write(directory / "dispatch_artifacts.py", Path(__file__).with_name("dispatch_artifacts.py").read_text()) + if _execution is not None: + _private_write(directory / "dispatch_pool_assets.py", Path(__file__).with_name("dispatch_pool_assets.py").read_text()) + project_files.append("dispatch_pool_assets.py") if project is not None: _private_write(directory / "dispatch_project.py", Path(__file__).with_name("dispatch_project.py").read_text()) project_files.append("dispatch_project.py") @@ -291,6 +307,9 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, report["state"] = "submission_unknown" report["submission_started_at"] = time.time() _write_json(directory / "submission.json", report) + if _execution is not None: + from .dispatch_pool import launch_step + return launch_step(directory, manifest, argv, report, timeout) try: response = _run(argv, timeout=timeout) except (subprocess.TimeoutExpired, OSError) as exc: @@ -360,6 +379,14 @@ def _receipt(path, manifest): source.get("schema_version") != SCHEMA_VERSION or any(source.get(key) != receipt.get(key) for key in ("dispatch_id", "uid", "job_id", "cluster"))): return None, "unreadable" + if manifest.get("execution") is not None: + execution = manifest["execution"] + if (job != execution["job_id"] or not re.fullmatch(r"[0-9]+", str(receipt.get("step_id", ""))) + or any(source.get(key) != receipt.get(key) for key in ("step_id",)) + or source.get("pool_id") != execution["pool_id"] or source.get("node") != execution["node"]): + return None, "unreadable" + receipt = dict(receipt, step_source=source, allocation_start=execution["allocation_start"], + node=execution["node"]) return receipt, state @@ -367,6 +394,10 @@ def _matches_worker(value, manifest, receipt): if (value is None or receipt is None or not receipt.get("cluster") or type(receipt.get("job_id")) is not int or receipt["job_id"] <= 0): return False + if manifest.get("execution") is not None: + if (value.get("step_id") != receipt.get("step_id") + or value.get("pool_id") != manifest["execution"]["pool_id"]): + return False return (value.get("schema_version") == SCHEMA_VERSION and value.get("dispatch_id") == manifest["dispatch_id"] and value.get("uid") == manifest["uid"] and @@ -390,6 +421,9 @@ def _scheduler(receipt, timeout): timeout = _finite_seconds(timeout, "scheduler timeout") if receipt is None or not isinstance(receipt.get("job_id"), int): return {"available": False, "state": "not_submitted"} + if receipt.get("step_id") is not None: + from .dispatch_pool import observe_step + return observe_step(receipt, timeout) job = str(receipt["job_id"]) if not receipt.get("cluster"): return {"available": False, "state": "unknown_source", "reason": "receipt names no cluster; no current-cluster job lookup attempted"} diff --git a/sucoder/dispatch_cli.py b/sucoder/dispatch_cli.py index f06d59b..1d79784 100644 --- a/sucoder/dispatch_cli.py +++ b/sucoder/dispatch_cli.py @@ -28,6 +28,10 @@ def _error(exc): raise typer.Exit(2) +from .dispatch_pool_cli import app as pool_app +app.add_typer(pool_app, name="pool") + + @app.callback() def main(ctx: typer.Context, root: Path = typer.Option( control.DEFAULT_DIRECTORY_ROOT, "--root", help="Private dispatch records on shared storage.")): diff --git a/sucoder/dispatch_pool.py b/sucoder/dispatch_pool.py new file mode 100644 index 0000000..b39d7e7 --- /dev/null +++ b/sucoder/dispatch_pool.py @@ -0,0 +1,347 @@ +"""Explicit existing-allocation pools; no allocation creation or job-wide stop.""" +from contextlib import contextmanager +from datetime import datetime +import json +import math +import os +from pathlib import Path +import re +import subprocess +import time +import uuid + +from . import dispatch as control +from .dispatch_protocol import atomic_json + +TERMINAL = frozenset(('COMPLETED', 'CANCELLED', 'FAILED', 'TIMEOUT', 'NODE_FAIL', + 'OUT_OF_MEMORY', 'PREEMPTED', 'BOOT_FAIL', 'DEADLINE', 'REVOKED')) + + +def memory_mib(value): + match = re.fullmatch(r'([1-9][0-9]*)([KMGT]?)', str(value), re.I) + if not match: + raise ValueError('memory must be an integer Slurm memory quantity') + return math.ceil(int(match[1]) * {'': 1, 'K': 1 / 1024, 'M': 1, 'G': 1024, 'T': 1024 ** 2}[match[2].upper()]) + + +def _job(job_id, timeout=10): + answer = control._run(['scontrol', 'show', 'job', '--oneliner', str(job_id)], timeout=timeout) + if answer.returncode: + raise ValueError('cannot verify allocation with scontrol') + fields = dict(re.findall(r'(\w+)=([^\s]+)', answer.stdout)) + owner = re.search(r'\(([0-9]+)\)$', fields.get('UserId', '')) + node = fields.get('NodeList', '') + if (fields.get('JobId') != str(job_id) or owner is None or int(owner[1]) != os.getuid() + or fields.get('JobState') != 'RUNNING' or not re.fullmatch(r'[A-Za-z0-9_.-]+', node) + or fields.get('NumNodes') != '1'): + raise ValueError('pool requires this uid\'s running single-node allocation') + tres = dict(part.split('=', 1) for part in fields.get('AllocTRES', '').split(',') if '=' in part) + try: + fields['verified_cpus'] = int(tres['cpu']) + fields['verified_mem_mib'] = memory_mib(tres['mem']) + fields['verified_end'] = datetime.fromisoformat(fields['EndTime']).timestamp() + fields['verified_start'] = datetime.fromisoformat(fields['StartTime']).timestamp() + except (ValueError, KeyError): + raise ValueError('allocation CPU/memory/end-time evidence is incomplete') + return fields + + +def attach(directory, job_id, *, node, cpus, mem, submit_before, worker_deadline, + expires, source, image, image_sha256, timeout=10): + directory = control._shared_path(Path(directory), 'pool directory') + if directory.exists(): + raise ValueError('pool directory must be new') + if not re.fullmatch(r'[1-9][0-9]*', str(job_id)): + raise ValueError('pool job ID must be numeric') + if type(cpus) is not int or cpus < 1: + raise ValueError('pool CPUs must be positive') + if any(type(value) not in (int, float) or not math.isfinite(value) + for value in (submit_before, worker_deadline, expires)): + raise ValueError('pool cutoffs must be finite Unix timestamps') + facts = _job(job_id, timeout) + if facts['NodeList'] != node or cpus > facts['verified_cpus'] or memory_mib(mem) > facts['verified_mem_mib']: + raise ValueError('pool node/budget differs from allocation evidence') + if not time.time() < submit_before < worker_deadline <= expires <= facts['verified_end']: + raise ValueError('pool cutoffs must fit allocation end and reserve publication time') + if not re.fullmatch(r'[0-9a-f]{64}', image_sha256): + raise ValueError('pool image requires full SHA256') + allocation = '/local/job' + str(job_id) + '/' + for path in (str(source), str(image)): + if not path.startswith(allocation) or os.path.normpath(path) != path: + raise ValueError('asset paths must be receiving-allocation-local absolute paths') + cluster = os.environ.get('SLURM_CLUSTER_NAME') + if not cluster or not re.fullmatch(r'[A-Za-z0-9_.-]+', cluster): + raise ValueError('explicit current Slurm cluster identity is required') + profile = dict(schema_version=1, pool_id=uuid.uuid4().hex, uid=os.getuid(), cluster=cluster, + job_id=int(job_id), node=node, cpus=cpus, mem_mib=memory_mib(mem), + submit_before=submit_before, worker_deadline=worker_deadline, expires=expires, + assets=dict(source=str(source), image=str(image), image_sha256=image_sha256), + allocation_evidence=facts, created_at=time.time()) + directory.mkdir(mode=0o700, parents=True) + os.chmod(directory, 0o700) # shared SGID parents must not make a private control directory 2700 + control._write_json(directory / 'pool.json', profile) + control._write_json(directory / 'reservations.json', {}) + return dict(profile, directory=str(directory)) + + +def _load(directory): + path = Path(directory) + if path.is_symlink() or path.resolve() != path or not path.is_dir(): + raise ValueError('pool directory must be an absolute nonredirected directory') + info = path.stat() + if info.st_uid != os.getuid() or info.st_mode & 0o077: + raise ValueError('pool directory must be private and owned') + profile, state = control._read_json(path / 'pool.json') + if (state != 'present' or profile.get('uid') != os.getuid() or profile.get('schema_version') != 1 + or not re.fullmatch(r'[0-9a-f]{32}', str(profile.get('pool_id', '')))): + raise ValueError('invalid pool identity') + return path, profile + + +@contextmanager +def _locked(path): + lock = path / '.reservation-lock' + try: + lock.mkdir(mode=0o700) + except FileExistsError: + raise ValueError('pool reservation lock held; no automatic stale lock stealing') + try: + yield + finally: + lock.rmdir() + + +def validate_execution(value, resources): + if (value.get('scope') != 'step' or not re.fullmatch(r'[0-9a-f]{32}', str(value.get('dispatch_id', ''))) + or not re.fullmatch(r'[0-9a-f]{32}', str(value.get('pool_id', ''))) + or type(value.get('job_id')) is not int or value['job_id'] < 1 + or not re.fullmatch(r'[A-Za-z0-9_.-]+', str(value.get('node', ''))) + or not re.fullmatch(r'[A-Za-z0-9_.-]+', str(value.get('cluster', ''))) + or type(value.get('deadline')) not in (int, float) or not math.isfinite(value['deadline']) + or value['deadline'] <= time.time() + 120): + raise ValueError('invalid or expired pool execution binding') + if resources.get('exclude'): + raise ValueError('pool workers use the bound node, not exclusion routing') + + +def submit(directory, repo, ref, task_file, agent, executable, *, cpus=1, mem='8G', + time_limit='00:30:00', timeout=30, **options): + path, profile = _load(directory) + if os.environ.get('SLURM_CLUSTER_NAME') != profile['cluster']: + raise ValueError('pool submission requires its original current Slurm cluster') + if options.get('dry_run'): + raise ValueError('pool submission requires an actual reservation; use normal submit --dry-run for staging validation') + facts = _job(profile['job_id'], timeout) + if (facts['NodeList'] != profile['node'] or facts['verified_cpus'] < profile['cpus'] + or facts['verified_mem_mib'] < profile['mem_mib'] or facts['verified_end'] < profile['expires'] + or facts['StartTime'] != profile['allocation_evidence']['StartTime']): + raise ValueError('allocation identity/budget changed since attach') + resources = control._resources(dict(account=facts['Account'], partition=facts['Partition'], + qos=facts.get('QOS'), cpus=cpus, mem=mem, time=time_limit)) + # Slurm time strings are normalized independently of the scheduler. Worker + # deadline is an additional hard cap, not a fresh full duration per retry. + days, clock = time_limit.split('-', 1) if '-' in time_limit else ('0', time_limit) + parts = [int(p) for p in clock.split(':')] + seconds = (parts[0] * 60 if len(parts) == 1 else parts[0] * 60 + parts[1] + if len(parts) == 2 else parts[0] * 3600 + parts[1] * 60 + parts[2]) + int(days) * 86400 + if '-' in time_limit and len(parts) < 3: + seconds = int(days) * 86400 + parts[0] * 3600 + (parts[1] * 60 if len(parts) == 2 else 0) + dispatch_id = uuid.uuid4().hex + execution = dict(scope='step', pool_id=profile['pool_id'], dispatch_id=dispatch_id, + job_id=profile['job_id'], node=profile['node'], cluster=profile['cluster'], + allocation_start=facts['StartTime'], + deadline=min(profile['worker_deadline'], time.time() + seconds), assets=profile['assets']) + result_root = control._shared_path(Path(options.get('directory_root') or control.DEFAULT_DIRECTORY_ROOT), 'result directory') + with _locked(path): + if time.time() >= profile['submit_before'] or time.time() + seconds > profile['worker_deadline']: + raise ValueError('pool submission cutoff/worker deadline would be exceeded') + reservations = json.loads((path / 'reservations.json').read_text()) + live = [row for row in reservations.values() if row['state'] != 'released'] + if (sum(row['cpus'] for row in live) + cpus > profile['cpus'] + or sum(row['mem_mib'] for row in live) + memory_mib(mem) > profile['mem_mib']): + raise ValueError('pool aggregate CPU/memory budget exhausted') + reservations[dispatch_id] = dict(state='reserved', cpus=cpus, mem_mib=memory_mib(mem), + reserved_at=time.time(), execution=execution, + directory=str(result_root / dispatch_id)) + atomic_json(str(path / 'reservations.json'), reservations) + try: + report = control.submit(repo, ref, task_file, agent, executable, resources, + timeout=timeout, _execution=execution, **options) + except Exception: + # There may already be a staged or live srun after an interrupted call. + # Retain budget; only exact scheduler terminal proof releases it. + raise + with _locked(path): + reservations = json.loads((path / 'reservations.json').read_text()) + reservations[dispatch_id].update(directory=report['directory'], submission=report) + atomic_json(str(path / 'reservations.json'), reservations) + return report + + +def launch_step(directory, manifest, argv, report, timeout): + started = time.monotonic() + try: + with (directory / 'srun-frontend.log').open('ab') as stream: + child = subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=stream, stderr=stream, + start_new_session=True, close_fds=True) + except OSError: + report.update(state='submission_failed', reason='srun unavailable; reservation retained for explicit reconciliation') + else: + report.update(frontend_pid=child.pid, frontend_host=os.uname().nodename, + frontend_scope='detached submitter process; not worker signal ownership') + while time.monotonic() - started < timeout: + if adopt_step(directory, manifest, report): + break + if child.poll() is not None: + report.update(reason='srun exited before trustworthy receiving-step receipt; reservation retained', frontend_exit=child.returncode) + break + time.sleep(0.1) + else: + report.update(reason='receiving-step receipt deadline expired; live/unknown submission retained, no retry') + atomic_json(str(directory / 'submission.json'), report) + return report + + +def adopt_step(directory, manifest, report): + """Reconcile a late receiving receipt without resubmitting or inferring exit.""" + source, state = control._read_json(directory / 'step-source.json') + execution = manifest['execution'] + if (state != 'present' or source.get('dispatch_id') != manifest['dispatch_id'] + or source.get('uid') != manifest['uid'] or source.get('job_id') != execution['job_id'] + or source.get('pool_id') != execution['pool_id'] or source.get('node') != execution['node'] + or source.get('cluster') != execution['cluster'] + or not re.fullmatch(r'[0-9]+', str(source.get('step_id', '')))): + return False + report.update(state='submitted', job_id=execution['job_id'], cluster=execution['cluster'], step_id=source['step_id']) + existing, existing_state = control._read_json(directory / 'scheduler-source.json') + if existing_state == 'missing': + control._write_json(directory / 'scheduler-source.json', source) + elif existing != source: + raise ValueError('receiving step conflicts with previously recorded scheduler identity') + atomic_json(str(directory / 'submission.json'), report) + return True + + +def _worker_receipt(directory, profile): + path, manifest = control._directory(directory) + execution = manifest.get('execution', {}) + if (execution.get('pool_id') != profile['pool_id'] or execution.get('job_id') != profile['job_id'] + or execution.get('node') != profile['node'] or manifest['cluster'] != profile['cluster']): + raise ValueError('worker does not belong to this pool') + receipt, state = control._receipt(path, manifest) + if receipt is not None and receipt.get('state') == 'submission_unknown': + if adopt_step(path, manifest, receipt): + receipt, state = control._receipt(path, manifest) + if state != 'present' or receipt is None or not re.fullmatch(r'[0-9]+', str(receipt.get('step_id', ''))): + raise ValueError('worker has no trustworthy numeric step receipt; do not signal parent job') + return path, manifest, receipt + + +def status(directory, *, timeout=10): + path, profile = _load(directory) + with _locked(path): + reservations = json.loads((path / 'reservations.json').read_text()) + for row in reservations.values(): + if row['state'] == 'released' or 'directory' not in row: + continue + try: + _, _, receipt = _worker_receipt(row['directory'], profile) + observation = control._scheduler(receipt, timeout) + except (ValueError, OSError): + continue + row['scheduler'] = observation + if observation.get('available') and observation.get('state', '').split()[0].rstrip('+') in TERMINAL: + row.update(state='released', release_proof=observation, released_at=time.time()) + atomic_json(str(path / 'reservations.json'), reservations) + return dict(pool=profile, reservations=reservations, + used_cpus=sum(row['cpus'] for row in reservations.values() if row['state'] != 'released'), + used_mem_mib=sum(row['mem_mib'] for row in reservations.values() if row['state'] != 'released')) + + +def stop_worker(directory, dispatch_id, *, timeout=10): + path, profile = _load(directory) + reservations = json.loads((path / 'reservations.json').read_text()) + row = reservations.get(dispatch_id) + if row is None or 'directory' not in row: + raise ValueError('unknown worker; no allocation signal attempted') + _, _, receipt = _worker_receipt(row['directory'], profile) + observation = control._scheduler(receipt, timeout) + if not observation.get('available') or observation.get('state') not in ('RUNNING', 'COMPLETING', 'PENDING'): + raise ValueError('cannot verify active owned step; no signal attempted') + target = str(receipt['job_id']) + '.' + receipt['step_id'] + answer = control._run(['scancel', '--clusters=' + profile['cluster'], '--signal=TERM', target], timeout=timeout) + if answer.returncode: + raise ValueError('step stop request failed; reservation remains held') + return dict(state='stop_requested', dispatch_id=dispatch_id, target=target, + reservation='held until exact scheduler step terminal proof') + + +def observe_step(receipt, timeout): + """Slurm step format differs from job format; never use job %T on squeue -s. + + Accounting can omit step UID. Accept it only with an exact parent row + binding uid, node and original allocation StartTime plus receiving proof. + """ + timeout = control._finite_seconds(timeout, 'scheduler timeout') + job, step = str(receipt['job_id']), str(receipt['step_id']) + source = receipt.get('step_source', {}) + node, start = receipt.get('node'), receipt.get('allocation_start') + target = job + '.' + step + if (not re.fullmatch(r'[0-9]+', step) or not receipt.get('cluster') or not node or not start + or source.get('step_id') != step or source.get('uid') != receipt.get('uid') + or source.get('job_id') != receipt['job_id'] or source.get('node') != node + or type(source.get('started_at')) not in (int, float)): + return dict(available=False, state='unknown_source', reason='step/parent incarnation proof incomplete') + deadline = time.monotonic() + timeout + cluster = '--clusters=' + receipt['cluster'] + def remaining(): + return max(.001, deadline - time.monotonic()) + try: + query = control._run(['scontrol', cluster, '--oneliner', 'show', 'step', target], timeout=remaining()) + if not query.returncode: + fields = dict(re.findall(r'(\w+)=([^\s]+)', query.stdout)) + owner = re.search(r'(?:^|\()([0-9]+)\)?$', fields.get('UserId', '')) + try: + live_start = datetime.fromisoformat(fields.get('StartTime', '')).timestamp() + parent_start = datetime.fromisoformat(start).timestamp() + live_incarnation = parent_start <= live_start <= source['started_at'] + 2 + except ValueError: + live_incarnation = False + if (fields.get('StepId') == target and owner and int(owner[1]) == receipt['uid'] + and fields.get('NodeList') == node and live_incarnation + and fields.get('State') in ('RUNNING', 'COMPLETING', 'PENDING')): + # Require the currently live parent incarnation too; a stale + # private receipt cannot authorize a recycled job/step number. + parent = control._run(['scontrol', cluster, '--oneliner', 'show', 'job', job], timeout=remaining()) + facts = dict(re.findall(r'(\w+)=([^\s]+)', parent.stdout)) + parent_uid = re.search(r'\(([0-9]+)\)$', facts.get('UserId', '')) + if (not parent.returncode and facts.get('JobId') == job and parent_uid + and int(parent_uid[1]) == receipt['uid'] and facts.get('NodeList') == node + and facts.get('StartTime') == start): + return dict(available=True, state=fields['State'], node=node, source='scontrol-step', target=target) + if time.monotonic() >= deadline: + return dict(available=False, state='unavailable') + query = control._run(['sacct', cluster, '--jobs=' + job, '--noheader', '--parsable2', + '--format=JobIDRaw,UID,State,ExitCode,NodeList,Start'], timeout=remaining()) + if query.returncode: + return dict(available=False, state='unavailable') + rows = [line.strip().split('|') for line in query.stdout.splitlines()] + parents = [row for row in rows if len(row) >= 6 and row[0] == job + and row[1] == str(receipt['uid']) and row[4] == node and row[5] == start] + if len(parents) != 1: + return dict(available=False, state='unknown_source', reason='accounting parent identity not proven') + exact = [row for row in rows if len(row) >= 6 and row[0] == target] + if len(exact) == 1: + row = exact[0] + try: + step_start = datetime.fromisoformat(row[5]).timestamp() + parent_start = datetime.fromisoformat(start).timestamp() + except ValueError: + return dict(available=False, state='unknown_source', reason='step start time unavailable') + if (row[1] in ('', str(receipt['uid'])) and row[4] == node + and parent_start <= step_start <= source['started_at'] + 2): + return dict(available=True, state=row[2], exit_code=row[3], node=node, + source='sacct-step-with-parent', target=target, allocation_start=start, step_start=row[5]) + return dict(available=True, state='unknown', reason='exact step incarnation absent; no result inferred') + except (OSError, subprocess.TimeoutExpired): + return dict(available=False, state='unavailable') diff --git a/sucoder/dispatch_pool_assets.py b/sucoder/dispatch_pool_assets.py new file mode 100644 index 0000000..0a737a6 --- /dev/null +++ b/sucoder/dispatch_pool_assets.py @@ -0,0 +1,77 @@ +"""Receiving-side immutable allocation asset reuse (Python 3.6 stdlib).""" +import hashlib +import json +import os +import stat +import subprocess +import time + + +def _identity(path, directory=False): + info = os.lstat(path) + if (os.path.realpath(path) != path or info.st_uid != os.getuid() + or (not stat.S_ISDIR(info.st_mode) if directory else not stat.S_ISREG(info.st_mode))): + raise ValueError("pool asset is not a nonredirected owned file/directory") + if not directory and info.st_mode & 0o222: + raise ValueError("pool image must have no write permission bits") + return [info.st_dev, info.st_ino, info.st_size, info.st_mtime_ns, info.st_mode] + + +def verify(execution, commit, root, environment): + assets = execution["assets"] + allocation_root = root.split("/workers/", 1)[0] + for name in ("source", "image"): + path = assets[name] + if not path.startswith(allocation_root + "/") or os.path.realpath(path) != path: + raise ValueError("pool assets must be on receiving allocation-local disk") + source, image = assets["source"], assets["image"] + env = dict(environment, GIT_NO_LAZY_FETCH="1", GIT_ALLOW_PROTOCOL="", GIT_OPTIONAL_LOCKS="0") + def git(*args): + return subprocess.check_output(["git", "-C", source] + list(args), env=env, + stderr=subprocess.DEVNULL, timeout=60).decode().strip() + if git("rev-parse", "--is-bare-repository") != "true": + raise ValueError("pool source must be a full private bare repository") + if os.path.exists(os.path.join(source, "shallow")) or os.path.lexists(os.path.join(source, "objects/info/alternates")): + raise ValueError("pool source must be self-contained, not shallow or alternate-backed") + config = git("config", "--local", "--list").lower() + if "partialclone" in config or ".promisor=true" in config: + raise ValueError("promisor pool sources are unsupported") + # Cache is a frozen base-object source, not a moving publication target. + # prepare.sh fetches the exact requested SHA from this worker's private, + # self-contained results.git; freshly authored candidates need not already + # exist in the allocation cache. Never mutate or hydrate the shared cache. + refs = hashlib.sha256(git("show-ref").encode()).hexdigest() + identity = dict(source=source, image=image, image_sha256=assets["image_sha256"], + source_identity=_identity(source, True), image_identity=_identity(image), refs_sha256=refs) + receipt = os.path.join(os.path.dirname(image), ".sucoder-pool-" + execution["pool_id"] + ".json") + lock = receipt + ".lock" + until = time.monotonic() + 90 + while True: + try: + os.mkdir(lock, 0o700) + break + except FileExistsError: + if time.monotonic() >= until: + raise ValueError("pool asset verification lock remains held; no stale-lock stealing") + time.sleep(0.1) + try: + if os.path.exists(receipt): + with open(receipt) as stream: + if json.load(stream) != identity: + raise ValueError("verified immutable pool assets changed") + else: + digest = hashlib.sha256() + with open(image, "rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + if digest.hexdigest() != assets["image_sha256"]: + raise ValueError("pool image SHA256 mismatch") + git("fsck", "--connectivity-only", "--no-reflogs") + if identity["image_identity"] != _identity(image): + raise ValueError("pool image changed during verification") + fd = os.open(receipt, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + with os.fdopen(fd, "w") as stream: + json.dump(identity, stream, sort_keys=True) + return dict(identity, verification_receipt=receipt, commit=commit) + finally: + os.rmdir(lock) diff --git a/sucoder/dispatch_pool_cli.py b/sucoder/dispatch_pool_cli.py new file mode 100644 index 0000000..e5e4ddb --- /dev/null +++ b/sucoder/dispatch_pool_cli.py @@ -0,0 +1,86 @@ +"""Existing-allocation pool commands; no scheduler allocation creation.""" +from pathlib import Path +from typing import List, Optional +import json +import subprocess +import typer +from . import dispatch_pool as control + +app = typer.Typer(help='Attach, reserve and submit isolated steps in an existing allocation.', no_args_is_help=True) + + +def _emit(value): + typer.echo(json.dumps(value, sort_keys=True)) + + +def _error(exc): + _emit(dict(schema_version=1, state='error', reason=str(exc))) + raise typer.Exit(2) + + +@app.command() +def attach(directory: Path, job_id: int = typer.Option(..., '--job-id'), + node: str = typer.Option(..., '--node'), cpus: int = typer.Option(..., '--cpus'), + mem: str = typer.Option(..., '--mem'), submit_before: float = typer.Option(..., '--submit-before'), + worker_deadline: float = typer.Option(..., '--worker-deadline'), expires: float = typer.Option(..., '--expires'), + source: str = typer.Option(..., '--source'), image: str = typer.Option(..., '--image'), + image_sha256: str = typer.Option(..., '--image-sha256')): + """Bind a new private pool record to this uid's running single-node job. + + Cutoffs are Unix seconds. --cpus/--mem are the total worker budget, with + coordinator/keeper/verifier resources already subtracted by the caller. + """ + try: + value = control.attach(directory, job_id, node=node, cpus=cpus, mem=mem, + submit_before=submit_before, worker_deadline=worker_deadline, + expires=expires, source=source, image=image, image_sha256=image_sha256) + except (ValueError, OSError, subprocess.SubprocessError) as exc: + _error(exc) + _emit(value) + + +@app.command() +def submit(ctx: typer.Context, pool: Path, repository: Path, + ref: str = typer.Option(..., '--ref'), task_file: Path = typer.Option(..., '--task-file'), + agent: str = typer.Option(..., '--agent'), executable: Path = typer.Option(..., '--executable'), + agent_home: Optional[Path] = typer.Option(None, '--agent-home'), + project_profile: Optional[Path] = typer.Option(None, '--project-profile'), + model: Optional[str] = typer.Option(None, '--model'), cpus: int = typer.Option(1, '--cpus'), + mem: str = typer.Option('8G', '--mem'), time: str = typer.Option('00:30:00', '--time'), + require_evidence: List[str] = typer.Option(None, '--require-evidence'), + snapshot_minutes: float = typer.Option(10, '--snapshot-minutes', min=0.1), + timeout: float = typer.Option(30, '--timeout')): + """Reserve budget and start exactly one existing-allocation step. + + A missing receiving receipt is ambiguous and is never retried automatically. + Use ordinary dispatch status/result/logs for a returned dispatch ID. + """ + try: + value = control.submit(pool, repository, ref, task_file, agent, executable, + cpus=cpus, mem=mem, time_limit=time, timeout=timeout, + directory_root=ctx.obj['root'], source_agent_home=agent_home, + project_profile=project_profile, model=model, + required_evidence=require_evidence or [], snapshot_minutes=snapshot_minutes) + except (ValueError, OSError, subprocess.SubprocessError) as exc: + _error(exc) + _emit(value) + if value['state'] != 'submitted': + raise typer.Exit(3) + + +@app.command() +def status(pool: Path, timeout: float = typer.Option(10, '--timeout')): + """Reconcile exact step terminal proof; unknown steps retain their budget.""" + try: + _emit(control.status(pool, timeout=timeout)) + except (ValueError, OSError, subprocess.SubprocessError) as exc: + _error(exc) + + +@app.command() +def stop_worker(pool: Path, dispatch_id: str, timeout: float = typer.Option(10, '--timeout')): + """Send TERM only to this pool worker's proven numeric step, never its job.""" + try: + _emit(control.stop_worker(pool, dispatch_id, timeout=timeout)) + except (ValueError, OSError, subprocess.SubprocessError) as exc: + _error(exc) diff --git a/sucoder/dispatch_worker.py b/sucoder/dispatch_worker.py index a29ae60..a2e6828 100644 --- a/sucoder/dispatch_worker.py +++ b/sucoder/dispatch_worker.py @@ -125,6 +125,34 @@ def describe(self): return {"job_id": self.job_id, "prefixes": sorted(set(self.prefixes.values()))} +class StepOwnership(JobOwnership): + """A numeric Slurm step, never its parent allocation, owns this worker.""" + def __init__(self, job_id, step_id, proc_root="/proc", uid=None, supervisor_pid=None): + if not re.fullmatch(r"[0-9]+", str(step_id)): + raise Refused("pool workers require a numeric Slurm step, not batch/extern") + super().__init__(job_id, proc_root, uid, supervisor_pid) + self.step_id = str(step_id) + prefixes = {} + for hierarchy, controllers, path in self._cgroups("self"): + key = (hierarchy, controllers) + if key not in self.prefixes: + continue + tail = path[len(self.prefixes[key]):].split("/") + steps = [part for part in tail if part.startswith("step_")] + if steps != ["step_" + self.step_id]: + raise Refused("receiving SLURM_STEP_ID differs from actual supervisor cgroup") + parts = path.split("/") + prefixes[key] = "/".join(parts[:parts.index(steps[0]) + 1]) + if not prefixes or len(prefixes) != len(self.prefixes): + raise Refused("unable to prove numeric step ownership") + self.prefixes = prefixes + + def describe(self): + result = super().describe() + result.update(scope="step", step_id=self.step_id) + return result + + def stop_allocation(ownership, timeout): """Stop only proven members, including daemons outside the leader group.""" deadline = time.monotonic() + timeout @@ -222,6 +250,8 @@ def load_manifest(path): required = {os.path.basename(manifest[key]) for key in ("task_file", "prepare_script", "snapshot_script", "runner_path", "worker_path", "environment_file")} required.update((os.path.basename(path), "dispatch_protocol.py", "dispatch_artifacts.py")) + if manifest.get("execution") is not None: + required.add("dispatch_pool_assets.py") if manifest.get("project") is not None: project = manifest["project"] if (not isinstance(project, dict) or project.get("profile") != "project.json" @@ -243,6 +273,15 @@ def load_manifest(path): if manifest.get("project") is not None: if digests["project.json"] != manifest["project"]["sha256"]: raise Refused("project profile differs from immutable submission") + if manifest.get("execution") is not None: + execution = manifest["execution"] + if (not isinstance(execution, dict) or execution.get("scope") != "step" + or not re.fullmatch(r"[0-9a-f]{32}", str(execution.get("pool_id", ""))) + or not re.fullmatch(r"[1-9][0-9]*", str(execution.get("job_id", ""))) + or not isinstance(execution.get("node"), str) + or type(execution.get("deadline")) not in (int, float) + or not math.isfinite(execution["deadline"])): + raise Refused("invalid pool execution binding") return manifest @@ -280,6 +319,25 @@ def receiving_root(manifest, environment, mountinfo_path="/proc/self/mountinfo") os.makedirs(root, mode=0o700, exist_ok=True) if os.stat(root).st_uid != os.getuid() or not os.path.isdir(root): raise Refused("receiving allocation root is not owned by this uid") + execution = manifest.get("execution") + if execution is not None: + if (str(execution["job_id"]) != job_id + or environment.get("SLURMD_NODENAME") != execution["node"] + or not re.fullmatch(r"[0-9]+", environment.get("SLURM_STEP_ID", ""))): + raise Refused("receiving pool job/node/numeric step differs from binding") + root = os.path.join(root, "workers", manifest["dispatch_id"]) + if os.path.realpath(root) != root: + raise Refused("pool worker root must not redirect") + workers = os.path.dirname(root) + os.makedirs(workers, mode=0o700, exist_ok=True) + if os.path.islink(workers) or os.stat(workers).st_uid != os.getuid() or os.stat(workers).st_mode & 0o077: + raise Refused("pool workers parent must be private and owned") + os.mkdir(root, mode=0o700) + os.chmod(root, 0o700) + child_mount = mount_for(root, mountinfo_path) + if child_mount["filesystem"] not in LOCAL_FILESYSTEMS: + raise Refused("pool worker child root is not on supported local disk") + mount = child_mount return root, job_id, mount @@ -856,9 +914,32 @@ def final_publication(): if manifest.get("cluster") is not None and cluster != manifest["cluster"]: raise Refused("receiving Slurm cluster differs from immutable submission source") record.update(local_root=root, job_id=job_id, cluster=cluster, mount=mount) - ownership = (ownership_factory or JobOwnership)(job_id) + if manifest.get("execution") is not None: + step_id = receiving_environment["SLURM_STEP_ID"] + ownership = (ownership_factory or StepOwnership)(job_id, step_id) + record.update(step_id=step_id, pool_id=manifest["execution"]["pool_id"]) + atomic_json(os.path.join(directory, "step-source.json"), + dict(schema_version=SCHEMA_VERSION, dispatch_id=manifest["dispatch_id"], + uid=os.getuid(), job_id=int(job_id), cluster=cluster, + step_id=step_id, pool_id=record["pool_id"], node=receiving_environment["SLURMD_NODENAME"], + ownership=ownership.describe(), supervisor_pid=os.getpid(), started_at=time.time())) + else: + ownership = (ownership_factory or JobOwnership)(job_id) record["allocation_ownership"] = ownership.describe() environment = environment_for(manifest, receiving_environment, root) + if manifest.get("execution") is not None: + module_name = (__package__ + ".dispatch_pool_assets") if __package__ else "dispatch_pool_assets" + spec = importlib.util.spec_from_file_location(module_name, os.path.join(directory, "dispatch_pool_assets.py")) + asset_helper = importlib.util.module_from_spec(spec) + spec.loader.exec_module(asset_helper) + for asset_path in (manifest["execution"]["assets"]["source"], manifest["execution"]["assets"]["image"]): + if mount_for(asset_path, mountinfo_path)["filesystem"] not in LOCAL_FILESYSTEMS: + raise Refused("pool immutable asset is not on supported receiving-local disk") + assets = asset_helper.verify(manifest["execution"], manifest["commit"], root, environment) + record["pool_assets"] = assets + environment["SUCODER_POOL_SOURCE_REPOSITORY"] = assets["source"] + environment["SUCODER_POOL_IMAGE"] = assets["image"] + environment["SUCODER_POOL_IMAGE_SHA256"] = assets["image_sha256"] home = seed_home(manifest, environment) if manifest["agent"] == "codex": record["canonical_home"] = environment["CODEX_HOME"] @@ -899,6 +980,8 @@ def final_publication(): "names that interpreter. Do not replace the runtime or mutate shared caches.\n" % directory).encode("utf-8") runner = load_runner(manifest["runner_path"]) command = command_for(manifest, environment) + if manifest.get("execution") is not None and time.time() >= manifest["execution"]["deadline"] - 120: + raise Refused("pool worker reached its publication cutoff before native launch") if stop_signal[0] is not None: raise Refused("cancelled before native version inspection") record["versions"]["native"] = native_version(manifest, environment) @@ -918,6 +1001,8 @@ def final_publication(): observed, databases = set(), set() deadline = time.monotonic() + manifest.get("snapshot_minutes", manifest["checkpoint_minutes"]) * 60 while child.poll() is None: + if manifest.get("execution") is not None and time.time() >= manifest["execution"]["deadline"] - 120: + stop_signal[0] = signal.SIGTERM evidence = inspect_placement(child, runner, root, environment["HOME"], expected_environment=environment, ownership=ownership, manifest=manifest) observed.update(evidence["observed"]) diff --git a/sucoder/local_tier.py b/sucoder/local_tier.py index 1ad1f89..741c195 100644 --- a/sucoder/local_tier.py +++ b/sucoder/local_tier.py @@ -395,7 +395,7 @@ def build_post_commit_hook(remote: str = "origin") -> str: def build_worker_prepare_script(*, results_path: str, dispatch_id: str, commit: str, - local_disk_root: str = DEFAULT_LOCAL_DISK) -> str: + local_disk_root: str = DEFAULT_LOCAL_DISK, isolated: bool = False) -> str: """Prepare an independent pinned worker; never reconcile a shared checkout.""" import re if not re.fullmatch(r"[0-9a-f]{32}", dispatch_id): @@ -405,12 +405,14 @@ def build_worker_prepare_script(*, results_path: str, dispatch_id: str, commit: return "\n".join([ "#!/bin/bash", "set -eu", "umask 077", 'case "${SLURM_JOB_ID:-}" in ""|*[!0-9]*) exit 1;; esac', - cache_exports_sh(local_disk_root), + (': "${SUCODER_LOCAL_ROOT:?}" "${SUCODER_POOL_SOURCE_REPOSITORY:?}"' if isolated else cache_exports_sh(local_disk_root)), 'mkdir -p "$SUCODER_LOCAL_ROOT/mirrors" "$TMPDIR" "$UV_CACHE_DIR" "$PIP_CACHE_DIR" "$npm_config_cache"', - "WORK=" + work_path_shell(local_disk_root, "dispatch_" + dispatch_id), + ('WORK="$SUCODER_LOCAL_ROOT/mirrors/dispatch_' + dispatch_id + '"' if isolated else "WORK=" + work_path_shell(local_disk_root, "dispatch_" + dispatch_id)), 'if [ -e "$WORK" ]; then echo "dispatch clone already exists" >&2; exit 1; fi', - "git -c protocol.file.allow=always clone --quiet --no-local --no-hardlinks --no-checkout " + shlex.quote(results_path) + ' "$WORK"', + "git " + ("-c pack.threads=1 " if isolated else "") + "-c protocol.file.allow=always clone --quiet --no-local --no-hardlinks --no-checkout " + ('"$SUCODER_POOL_SOURCE_REPOSITORY"' if isolated else shlex.quote(results_path)) + ' "$WORK"', 'cd "$WORK"', + "git remote set-url origin " + shlex.quote(results_path), + *(["git -c pack.threads=1 -c protocol.file.allow=always fetch --quiet --no-tags --no-recurse-submodules origin " + commit] if isolated else []), "git checkout --quiet -b " + shlex.quote("dispatch/" + dispatch_id) + " " + commit, "git config user.name 'Sue the Coder'", "git config user.email coder@sucoder.dev", "cat > .git/hooks/post-commit <<'SUCODER_HOOK'", build_post_commit_hook(results_path).rstrip(), @@ -419,7 +421,7 @@ def build_worker_prepare_script(*, results_path: str, dispatch_id: str, commit: def build_worker_snapshot_script(*, results_path: str, dispatch_id: str, - local_disk_root: str = DEFAULT_LOCAL_DISK) -> str: + local_disk_root: str = DEFAULT_LOCAL_DISK, isolated: bool = False) -> str: """Capture dirty files with a temporary index into a unique, nonforced ref.""" if not results_path.startswith("/") or "\x00" in results_path: raise ValueError("worker publication target must be an absolute path") @@ -429,7 +431,7 @@ def build_worker_snapshot_script(*, results_path: str, dispatch_id: str, return "\n".join([ "#!/bin/bash", "set -eu", "umask 077", 'case "${SLURM_JOB_ID:-}" in ""|*[!0-9]*) exit 1;; esac', - "WORK=" + work_path_shell(local_disk_root, "dispatch_" + dispatch_id), + ('WORK="$SUCODER_LOCAL_ROOT/mirrors/dispatch_' + dispatch_id + '"' if isolated else "WORK=" + work_path_shell(local_disk_root, "dispatch_" + dispatch_id)), # Tasks may reconfigure Git. Automatic publication must not follow a # new linked worktree, symlink or shared Git metadata redirection. 'if [ "$(readlink -f "$WORK")" != "$WORK" ] || [ ! -O "$WORK" ] || [ ! -d "$WORK/.git" ] || [ -L "$WORK/.git" ] || [ ! -O "$WORK/.git" ]; then', diff --git a/tests/test_dispatch_pool.py b/tests/test_dispatch_pool.py new file mode 100644 index 0000000..1f45e1e --- /dev/null +++ b/tests/test_dispatch_pool.py @@ -0,0 +1,343 @@ +"""Existing-allocation pool tests: disposable Git, fake Slurm, no native model.""" +import hashlib +import json +import os +from pathlib import Path +import subprocess +import time +from types import SimpleNamespace + +import pytest + +from sucoder import dispatch, dispatch_pool as pool, dispatch_pool_assets as assets +from sucoder import dispatch_worker as worker +from sucoder.local_tier import build_worker_prepare_script, build_worker_snapshot_script +from tests.test_dispatch import source # existing disposable source fixture, never a real project + + +def facts(): + return dict(JobId='234', NodeList='n1', Account='a', Partition='p', QOS='q', StartTime='2020-01-01T01:00:00', + verified_cpus=8, verified_mem_mib=48 * 1024, verified_end=time.time() + 8000) + + +@pytest.fixture +def attached(tmp_path, monkeypatch): + monkeypatch.setenv('SLURM_CLUSTER_NAME', 'cluster') + monkeypatch.setattr(dispatch, '_shared_path', lambda path, label: path.resolve()) + monkeypatch.setattr(pool, '_job', lambda *args, **kwargs: facts()) + path = tmp_path / 'pool' + pool.attach(path, 234, node='n1', cpus=4, mem='32G', submit_before=time.time() + 2000, + worker_deadline=time.time() + 4000, expires=time.time() + 5000, + source='/local/job234/assets/source.git', image='/local/job234/assets/image.sqfs', + image_sha256='a' * 64) + return path + + +def request(source): + return {key: source[key] for key in ('repo', 'ref', 'task_file', 'agent', 'executable')} + + +def test_attach_rejects_wrong_uid_or_multi_node_scheduler(monkeypatch): + monkeypatch.setattr(dispatch, '_run', lambda *a, **kw: SimpleNamespace(returncode=0, stdout=( + 'JobId=234 UserId=x(%d) JobState=RUNNING NodeList=n1 NumNodes=2 AllocTRES=cpu=8,mem=48G EndTime=2099-01-01T00:00:00' % os.getuid()))) + with pytest.raises(ValueError, match='single-node'): + pool._job(234) + + +def test_attach_binds_finite_budget_and_private_metadata(attached): + profile = json.loads((attached / 'pool.json').read_text()) + assert profile['cpus'] == 4 and profile['mem_mib'] == 32768 + assert profile['uid'] == os.getuid() and profile['node'] == 'n1' + assert attached.stat().st_mode & 0o077 == 0 + + +@pytest.mark.parametrize('cpus,mem', [(5, '1G'), (1, '33G')]) +def test_over_budget_refuses_before_submission(attached, source, monkeypatch, cpus, mem): + monkeypatch.setattr(dispatch, 'submit', lambda *a, **k: pytest.fail('overcommit submitted')) + with pytest.raises(ValueError, match='budget exhausted'): + pool.submit(attached, **request(source), cpus=cpus, mem=mem) + + +def test_unknown_reserves_aggregate_budget_and_never_retries(attached, source, monkeypatch): + seen = [] + def submit(*a, **k): + seen.append(k['_execution']) + return dict(directory=str(source['repo'] / 'unknown'), state='submission_unknown') + monkeypatch.setattr(dispatch, 'submit', submit) + for _ in range(4): + pool.submit(attached, **request(source), cpus=1, mem='8G') + with pytest.raises(ValueError, match='budget exhausted'): + pool.submit(attached, **request(source)) + assert len(seen) == 4 and len({row['dispatch_id'] for row in seen}) == 4 + assert all(row['scope'] == 'step' and row['job_id'] == 234 for row in seen) + + +def test_submit_cutoff_and_worker_duration_are_enforced(attached, source, monkeypatch): + monkeypatch.setattr(dispatch, 'submit', lambda *a, **k: pytest.fail('expired submission')) + with pytest.raises(ValueError, match='deadline'): + pool.submit(attached, **request(source), time_limit='02:00:00') + profile = json.loads((attached / 'pool.json').read_text()) + profile['submit_before'] = time.time() - 1 + (attached / 'pool.json').write_text(json.dumps(profile)) + with pytest.raises(ValueError, match='cutoff'): + pool.submit(attached, **request(source)) + + +def test_lock_is_atomic_and_not_stolen(attached, source): + (attached / '.reservation-lock').mkdir() + with pytest.raises(ValueError, match='no automatic stale'): + pool.submit(attached, **request(source)) + + +def test_staged_pool_command_has_exact_step_and_private_publication(attached, source, monkeypatch): + seen = {} + def launch(directory, manifest, argv, report, timeout): + seen.update(manifest=manifest, argv=argv) + return report + monkeypatch.setattr(pool, 'launch_step', launch) + report = pool.submit(attached, **request(source)) + argv = seen['argv'] + assert argv[0] == 'srun' and '--jobid=234' in argv and '--exclusive' in argv and '--exact' in argv + assert '--cpu-bind=cores' in argv and '--export=NONE' in argv + assert not any(arg.startswith('--overlap') or arg == 'sbatch' for arg in argv) + directory = Path(report['directory']) + assert (directory / 'results.git/HEAD').exists() + assert 'SUCODER_LOCAL_ROOT/mirrors/' in (directory / 'prepare.sh').read_text() + assert 'SUCODER_POOL_SOURCE_REPOSITORY' in (directory / 'prepare.sh').read_text() + digests = json.loads((directory / 'digests.json').read_text()) + assert digests['dispatch_pool_assets.py'] == hashlib.sha256((directory / 'dispatch_pool_assets.py').read_bytes()).hexdigest() + + +def test_launch_timeout_keeps_live_srun_and_unknown_reservation(tmp_path, monkeypatch): + child = SimpleNamespace(pid=987, poll=lambda: None) + monkeypatch.setattr(pool.subprocess, 'Popen', lambda *a, **k: child) + directory = tmp_path + manifest = dict(execution=dict(pool_id='a' * 32, job_id=234, node='n1', cluster='cluster'), dispatch_id='b' * 32, uid=os.getuid()) + report = dict(state='submission_unknown') + answer = pool.launch_step(directory, manifest, ['fake-srun'], report, .001) + assert answer['state'] == 'submission_unknown' and answer['frontend_pid'] == 987 + assert 'no retry' in answer['reason'] + + +def test_step_receipt_is_bound_to_pool_and_scheduler_source(attached, source, monkeypatch): + def launch(directory, manifest, argv, report, timeout): + execution = manifest['execution'] + identity = dict(schema_version=1, uid=os.getuid(), dispatch_id=manifest['dispatch_id'], + job_id=234, cluster='cluster', pool_id=execution['pool_id'], node='n1', step_id='4') + dispatch._write_json(directory / 'step-source.json', identity) + dispatch._write_json(directory / 'scheduler-source.json', identity) + report.update(state='submitted', job_id=234, cluster='cluster', step_id='4') + from sucoder.dispatch_protocol import atomic_json + atomic_json(str(directory / 'submission.json'), report) + return report + monkeypatch.setattr(pool, 'launch_step', launch) + report = pool.submit(attached, **request(source)) + path = Path(report['directory']) + manifest = json.loads((path / 'manifest.json').read_text()) + receipt, state = dispatch._receipt(path, manifest) + assert state == 'present' and receipt['step_id'] == '4' + wrong = json.loads((path / 'scheduler-source.json').read_text()) + wrong['step_id'] = '40' + (path / 'scheduler-source.json').write_text(json.dumps(wrong)) + assert dispatch._receipt(path, manifest) == (None, 'unreadable') + + +def test_exact_terminal_step_proof_required_to_release(attached, monkeypatch): + rows = {'a' * 32: dict(state='reserved', cpus=1, mem_mib=8192, directory='/fake')} + (attached / 'reservations.json').write_text(json.dumps(rows)) + monkeypatch.setattr(pool, '_worker_receipt', lambda *a: (None, None, {})) + monkeypatch.setattr(dispatch, '_scheduler', lambda *a: dict(available=False, state='unavailable')) + assert pool.status(attached)['used_cpus'] == 1 + monkeypatch.setattr(dispatch, '_scheduler', lambda *a: dict(available=True, state='RUNNING')) + assert pool.status(attached)['used_cpus'] == 1 + monkeypatch.setattr(dispatch, '_scheduler', lambda *a: dict(available=True, state='COMPLETED', source='sacct')) + result = pool.status(attached) + assert result['used_cpus'] == 0 + assert result['reservations']['a' * 32]['release_proof']['source'] == 'sacct' + + +def test_stop_only_targets_exact_verified_step(attached, monkeypatch): + rows = {'a' * 32: dict(state='reserved', cpus=1, mem_mib=8192, directory='/fake')} + (attached / 'reservations.json').write_text(json.dumps(rows)) + monkeypatch.setattr(pool, '_worker_receipt', lambda *a: (None, None, dict(job_id=234, step_id='4'))) + monkeypatch.setattr(dispatch, '_scheduler', lambda *a: dict(available=True, state='RUNNING')) + seen = [] + monkeypatch.setattr(dispatch, '_run', lambda argv, **k: (seen.append(argv) or SimpleNamespace(returncode=0))) + answer = pool.stop_worker(attached, 'a' * 32) + assert seen == [['scancel', '--clusters=cluster', '--signal=TERM', '234.4']] + assert answer['reservation'].startswith('held') + assert json.loads((attached / 'reservations.json').read_text())['a' * 32]['state'] == 'reserved' + + +def step_receipt(step='0'): + source = dict(step_id=step, uid=os.getuid(), job_id=234, node='n1', started_at=time.time()) + return dict(job_id=234, uid=os.getuid(), cluster='cluster', step_id=step, node='n1', + allocation_start='2020-01-01T01:00:00', step_source=source) + + +def test_step_scheduler_never_accepts_parent_or_sibling(monkeypatch): + calls = [] + def run(argv, **kwargs): + calls.append(argv) + if argv[0] == 'scontrol': + return SimpleNamespace(returncode=1, stdout='') + return SimpleNamespace(returncode=0, stdout='234|%s|RUNNING|0:0|n1|2020-01-01T01:00:00\n234.40||COMPLETED|0:0|n1|2020-01-01T01:01:00\n' % os.getuid()) + monkeypatch.setattr(dispatch, '_run', run) + answer = dispatch._scheduler(step_receipt('4'), 1) + assert answer['state'] == 'unknown' + assert calls[0][0] == 'scontrol' and calls[0][-1] == '234.4' + + +@pytest.mark.parametrize('parent_uid', [None, 'wrong', str(os.getuid())]) +def test_blank_accounting_step_uid_requires_bound_parent(monkeypatch, parent_uid): + def run(argv, **kwargs): + if argv[0] == 'scontrol': + return SimpleNamespace(returncode=1, stdout='') + parent = '' if parent_uid is None else '234|%s|RUNNING|0:0|n1|2020-01-01T01:00:00\n' % parent_uid + return SimpleNamespace(returncode=0, stdout=parent + '234.0||COMPLETED|0:0|n1|2020-01-01T01:01:00\n') + monkeypatch.setattr(dispatch, '_run', run) + answer = dispatch._scheduler(step_receipt(), 1) + assert (answer.get('state') == 'COMPLETED') == (parent_uid == str(os.getuid())) + + +def test_live_step_uses_native_step_state_and_parent_incarnation(monkeypatch): + def run(argv, **kwargs): + if 'step' in argv: + return SimpleNamespace(returncode=0, stdout='StepId=234.0 UserId=%s State=RUNNING NodeList=n1 StartTime=2020-01-01T01:01:00' % os.getuid()) + return SimpleNamespace(returncode=0, stdout='JobId=234 UserId=user(%s) NodeList=n1 StartTime=2020-01-01T01:00:00' % os.getuid()) + monkeypatch.setattr(dispatch, '_run', run) + assert dispatch._scheduler(step_receipt(), 1)['state'] == 'RUNNING' + + +def test_isolated_prepare_does_not_reset_root_or_share_origin(tmp_path): + script = build_worker_prepare_script(results_path='/shared/private/results.git', dispatch_id='a' * 32, commit='b' * 40, isolated=True) + assert 'export SUCODER_LOCAL_ROOT=' not in script + assert 'git remote set-url origin /shared/private/results.git' in script + assert '--no-hardlinks' in script + assert 'SUCODER_LOCAL_ROOT/mirrors/' in build_worker_snapshot_script(results_path='/shared/private/results.git', dispatch_id='a' * 32, isolated=True) + + +def test_pool_root_uses_actual_job_node_and_private_child(tmp_path): + disk = tmp_path / 'disk' + disk.mkdir() + mounts = tmp_path / 'mounts' + mounts.write_text('1 0 0:1 / / rw - ext4 /dev/test rw\n') + manifest = dict(local_disk_root=str(disk), dispatch_id='a' * 32, execution=dict(job_id=234, node='n1')) + env = dict(SLURM_JOB_ID='234', SLURM_STEP_ID='4', SLURMD_NODENAME='n1') + root, job, _ = worker.receiving_root(manifest, env, str(mounts)) + assert root == str(disk / 'job234/workers' / ('a' * 32)) and job == '234' + assert Path(root).stat().st_mode & 0o077 == 0 + with pytest.raises(worker.Refused, match='job/node'): + worker.receiving_root(manifest, dict(env, SLURM_STEP_ID='batch'), str(mounts)) + + +def test_cached_base_can_prepare_new_private_candidate_without_cache_mutation(tmp_path): + source = tmp_path / 'source' + source.mkdir() + def git(path, *args): + return subprocess.check_output(['git', '-C', str(path)] + list(args), text=True).strip() + git(source, 'init', '-q') + git(source, 'config', 'user.name', 'Test') + git(source, 'config', 'user.email', 'test@example.invalid') + (source / 'tracked').write_text('base') + git(source, 'add', '.') + git(source, 'commit', '-qm', 'base') + cache = tmp_path / 'cache.git' + subprocess.check_call(['git', 'clone', '-q', '--bare', '--no-hardlinks', str(source), str(cache)]) + cache_refs = git(cache, 'show-ref') + (source / 'tracked').write_text('private candidate') + git(source, 'commit', '-qam', 'candidate') + commit = git(source, 'rev-parse', 'HEAD') + results = tmp_path / 'results.git' + subprocess.check_call(['git', 'clone', '-q', '--bare', '--no-hardlinks', str(source), str(results)]) + root = tmp_path / 'job234/workers' / ('a' * 32) + root.mkdir(parents=True) + script = build_worker_prepare_script(results_path=str(results), dispatch_id='a' * 32, commit=commit, isolated=True) + env = dict(os.environ, SLURM_JOB_ID='234', SUCODER_LOCAL_ROOT=str(root), SUCODER_POOL_SOURCE_REPOSITORY=str(cache), + TMPDIR=str(root / 'tmp'), UV_CACHE_DIR=str(root / 'cache/uv'), PIP_CACHE_DIR=str(root / 'cache/pip'), + npm_config_cache=str(root / 'cache/npm')) + subprocess.check_call(['/bin/bash', '-c', script], env=env) + work = root / 'mirrors' / ('dispatch_' + 'a' * 32) + assert git(work, 'rev-parse', 'HEAD') == commit + assert git(work, 'remote', 'get-url', 'origin') == str(results) + assert not (work / '.git/objects/info/alternates').exists() + assert (work / 'tracked').read_text() == 'private candidate' + assert git(cache, 'show-ref') == cache_refs + assert subprocess.run(['git', '-C', str(cache), 'cat-file', '-e', commit], stderr=subprocess.DEVNULL).returncode != 0 + + +def test_receiving_nested_shared_mount_refuses_before_state(tmp_path): + disk = tmp_path / 'disk' + disk.mkdir() + target = disk / 'job234/workers' / ('a' * 32) + mounts = tmp_path / 'mounts' + mounts.write_text('1 0 0:1 / / rw - ext4 /dev/test rw\n2 1 0:2 / %s rw - nfs host:/shared rw\n' % target) + manifest = dict(local_disk_root=str(disk), dispatch_id='a' * 32, execution=dict(job_id=234, node='n1')) + with pytest.raises(worker.Refused, match='child root'): + worker.receiving_root(manifest, dict(SLURM_JOB_ID='234', SLURM_STEP_ID='4', SLURMD_NODENAME='n1'), str(mounts)) + assert not (target / 'state').exists() + + +def test_pool_image_verified_once_and_mutation_refuses(tmp_path): + allocation = tmp_path / 'job234' + cache = allocation / 'assets/source.git' + cache.mkdir(parents=True) + subprocess.check_call(['git', 'init', '-q', '--bare', str(cache)]) + # A source ref is necessary to bind a nonempty immutable source namespace. + source = tmp_path / 'source' + source.mkdir() + subprocess.check_call(['git', '-C', str(source), 'init', '-q']) + subprocess.check_call(['git', '-C', str(source), '-c', 'user.name=Test', '-c', 'user.email=x@y', 'commit', '--allow-empty', '-qm', 'base']) + subprocess.check_call(['git', '-C', str(cache), 'fetch', '-q', str(source), 'HEAD:refs/heads/base']) + image = allocation / 'assets/image.sqfs' + image.write_bytes(b'owned synthetic image') + image.chmod(0o400) + root = allocation / 'workers' / ('a' * 32) + root.mkdir(parents=True) + execution = dict(pool_id='b' * 32, assets=dict(source=str(cache), image=str(image), image_sha256=hashlib.sha256(image.read_bytes()).hexdigest())) + first = assets.verify(execution, 'c' * 40, str(root), os.environ) + assert Path(first['verification_receipt']).exists() + assert assets.verify(execution, 'd' * 40, str(root), os.environ)['image_sha256'] == first['image_sha256'] + image.chmod(0o600) + with pytest.raises(ValueError, match='write permission'): + assets.verify(execution, 'c' * 40, str(root), os.environ) + + +def test_late_receipt_reconciles_without_new_srun(tmp_path, monkeypatch): + execution = dict(pool_id='a' * 32, job_id=234, node='n1', cluster='cluster') + manifest = dict(execution=execution, dispatch_id='b' * 32, uid=os.getuid()) + report = dict(state='submission_unknown') + source = dict(dispatch_id='b' * 32, uid=os.getuid(), job_id=234, node='n1', cluster='cluster', pool_id='a' * 32, step_id='7') + (tmp_path / 'step-source.json').write_text(json.dumps(source)) + monkeypatch.setattr(pool.subprocess, 'Popen', lambda *a, **k: pytest.fail('reconciliation launched a duplicate')) + assert pool.adopt_step(tmp_path, manifest, report) + assert report['step_id'] == '7' and report['state'] == 'submitted' + assert json.loads((tmp_path / 'scheduler-source.json').read_text()) == source + + +def test_worker_duration_gets_early_publication_deadline(attached, source, monkeypatch): + before = time.time() + seen = [] + def submit(*args, **kwargs): + seen.append(kwargs['_execution']) + return dict(directory=str(source['repo']), state='submission_unknown') + monkeypatch.setattr(dispatch, 'submit', submit) + pool.submit(attached, **request(source), time_limit='00:05:00') + assert before + 300 <= seen[0]['deadline'] <= time.time() + 300 + assert seen[0]['deadline'] < json.loads((attached / 'pool.json').read_text())['worker_deadline'] + + +def test_pool_cli_preserves_evidence_and_snapshot_contract(monkeypatch, tmp_path): + from typer.testing import CliRunner + from sucoder.dispatch_cli import app + seen = {} + def submit(*args, **kwargs): + seen.update(kwargs) + return dict(state='submitted') + monkeypatch.setattr(pool, 'submit', submit) + result = CliRunner().invoke(app, ['--root', str(tmp_path), 'pool', 'submit', '/shared/pool', '/shared/repo', + '--ref', 'b' * 40, '--task-file', '/shared/task', '--agent', 'codex', '--executable', '/shared/codex', + '--require-evidence', 'owner-status.json', '--require-evidence', 'handoff.org', '--snapshot-minutes', '2']) + assert result.exit_code == 0, result.output + assert seen['required_evidence'] == ['owner-status.json', 'handoff.org'] + assert seen['snapshot_minutes'] == 2 and seen['directory_root'] == tmp_path From b3fb113effdd2d9f56c2ffa628464068a0eda874 Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 22:31:11 -0700 Subject: [PATCH 03/18] Record finite pool dispatch verification for issue 77 Co-Authored-By: GPT-6 --- docs/dispatch-pool-ledger.org | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index 715c243..8648e5b 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -69,3 +69,20 @@ libssl.so.3 native-module error; no tool installation was changed. Manual call-site review and independent source review cover the high-impact ownership boundary. Actual supervisor/sibling/daemon survival and native local-SQLite placement remain root-owned release gates. + +** Exact committed confirmation +At production/test commit 0340f0b1fe3c319db9ed2e26ff9061d9afcdabba the complete +finite fake-only selection passed: 290 passed, 0 failed/errors/skips, +480 inherited Typer deprecation warnings, 59.44 seconds. This includes both +independent fake-proc and actual-Slurm-output fixture modules plus existing +dispatch, project, worker, CLI and local-tier tests. Full log and JUnit: +/local/job40498884/control/pool77-tests-final.log and pool77-tests-final.xml. + +Root's actual step-survival-v2 test exercised two numeric steps on job 40513023, +with detached own-child cleanup and sibling synthetic SQLite/WAL descriptors +and keeper progress preserved. Candidate dispatch_worker.py is byte-identical +to that proven helper: b09ef8620c4e4ab80c867044dafc9dc0c84e10d67c0eb68eea49767068d6f762. +The requested ownership/cleanup/mount AST closure is identical too, recorded in +/local/job40498884/control/pool77-actual-cleanup-closure.json. This is narrow +cleanup acceptance, not complete dispatch.run/project/native placement proof. +Root retains those native canary release and interpretation responsibilities. From 16fe2d1ec9f0fcf1e9684af194b7b9ec7e38807d Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 22:40:09 -0700 Subject: [PATCH 04/18] Record bounded pool clone transport amendment for issue 77 Co-Authored-By: GPT-6 --- docs/dispatch-pool-ledger.org | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index 8648e5b..f85e9fe 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -86,3 +86,14 @@ The requested ownership/cleanup/mount AST closure is identical too, recorded in /local/job40498884/control/pool77-actual-cleanup-closure.json. This is narrow cleanup acceptance, not complete dispatch.run/project/native placement proof. Root retains those native canary release and interpretation responsibilities. + +** Additive pool-only clone transport correction +Root observed pool source staging spend several minutes in upload-pack/pack-objects +before any worker was launched. This is a measured transport cost, not a filesystem +failure diagnosis. The original transaction and its eventual result stay intact. +Plan: use Git's local-copy path with --no-hardlinks only for explicitly owned, +full, non-alternate/non-shallow pool sources, preserving the ordinary job path. +Receiving immutable-cache cloning uses the same independent-copy transport. +Verify copied object inode separation, self-contained closure and unchanged source +bytes/refs with disposable repositories; do not accept alternates or symlinked +objects. Existing exact-SHA private-results fetch remains unchanged. From cd40f6c3bfcf21b8f76d9a628752f7a5a9e7a863 Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 22:46:15 -0700 Subject: [PATCH 05/18] Copy owned pool Git objects without upload-pack recompression Use --local --no-hardlinks only for validated full pool sources; refuse dependent, redirected or nonregular object metadata and verify copied objects are independent before native launch. Preserve ordinary job transport. Co-Authored-By: GPT-6 --- docs/dispatch-pools.org | 13 ++++++ sucoder/dispatch.py | 10 ++++- sucoder/dispatch_pool_assets.py | 49 +++++++++++++++++++++ sucoder/local_tier.py | 4 +- tests/test_dispatch_pool.py | 78 +++++++++++++++++++++++++++++++++ 5 files changed, 152 insertions(+), 2 deletions(-) diff --git a/docs/dispatch-pools.org b/docs/dispatch-pools.org index 0a39d2a..82415ae 100644 --- a/docs/dispatch-pools.org +++ b/docs/dispatch-pools.org @@ -80,3 +80,16 @@ The worker begins shutdown 120 seconds before the earlier requested-duration or pool worker deadline. Slurm's time request is an additional hard bound. Publication receipts distinguish complete, incomplete and unverified outcomes; no CLI exit alone proves scientific or infrastructure acceptance. + +* Pool-only source transport +Pool staging and receiving-cache preparation use =git clone --local --no-hardlinks= +for explicitly owned, full repositories. This copies object files without the +upload-pack recompression used by =--no-local=. It does not hardlink or use +alternates. Object metadata must contain owned regular files/directories, +without symlinks, shallow boundaries, alternate stores or promisor objects. +Each copy checks that destination objects have link count one and differ from +matching source inodes, then validates Git connectivity. Receiving proof is +retained as =source-copy-receiving.json= in the private dispatch directory. +The ordinary independent-job transport retains =--no-local=. +These checks refuse an incomplete/racing copy before native launch; they do +not make concurrent source garbage collection an atomic repository snapshot. diff --git a/sucoder/dispatch.py b/sucoder/dispatch.py index 3d589f2..990fb7a 100644 --- a/sucoder/dispatch.py +++ b/sucoder/dispatch.py @@ -211,6 +211,11 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, # Git metadata can itself be redirected into node-local storage by a worktree. _shared_path(gitdir, "source Git metadata") _shared_path(common, "source common Git metadata") + if _execution is not None: + from .dispatch_pool_assets import validate_local_source, verify_copy + if any(path.stat().st_uid != os.getuid() for path in (repo, gitdir, common)): + raise ValueError("pool local-copy source and Git metadata must be owned by this uid") + validate_local_source(str(common)) commit = _git(repo, "rev-parse", "--verify", ref + "^{commit}") if not re.fullmatch(r"[0-9a-f]{40,64}", commit): raise ValueError("source did not resolve to a commit") @@ -278,8 +283,11 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, (directory / "evidence").mkdir(mode=0o700) project_files = stage_profile(project, directory) if project is not None else [] # Private self-contained source snapshot. No source ref/config/index changes. - _git(repo, *(["-c", "pack.threads=1"] if _execution is not None else []), "clone", "--bare", "--no-local", "--no-hardlinks", str(repo), str(directory / "results.git"), timeout=timeout, file_protocol=True) + _git(repo, *(["-c", "pack.threads=1"] if _execution is not None else []), "clone", "--bare", "--local" if _execution is not None else "--no-local", "--no-hardlinks", str(repo), str(directory / "results.git"), timeout=timeout, file_protocol=True) _git(directory / "results.git", "cat-file", "-e", commit + "^{commit}") + if _execution is not None: + report["source_copy"] = verify_copy(str(common), str(directory / "results.git")) + _git(directory / "results.git", "fsck", "--connectivity-only", "--no-reflogs", timeout=timeout) # Make the private initial reference explicit; this never touches source. _git(directory / "results.git", "update-ref", "refs/heads/" + manifest["branch"], commit) _write_json(directory / "manifest.json", manifest) diff --git a/sucoder/dispatch_pool_assets.py b/sucoder/dispatch_pool_assets.py index 0a737a6..284981a 100644 --- a/sucoder/dispatch_pool_assets.py +++ b/sucoder/dispatch_pool_assets.py @@ -4,9 +4,51 @@ import os import stat import subprocess +import sys import time +def validate_local_source(gitdir): + """Local-copy transport is only for owned, full, independent Git objects.""" + if os.path.realpath(gitdir) != gitdir or os.lstat(gitdir).st_uid != os.getuid(): + raise ValueError("pool local-copy Git metadata must be nonredirected and owned") + for name in ("shallow", "objects/info/alternates", "objects/info/http-alternates"): + if os.path.lexists(os.path.join(gitdir, name)): + raise ValueError("pool local-copy sources must be full and have no alternates") + objects = os.path.join(gitdir, "objects") + if os.path.islink(objects) or not os.path.isdir(objects): + raise ValueError("pool local-copy object directory must not be redirected") + for parent, directories, files in os.walk(objects): + for name in directories + files: + path = os.path.join(parent, name) + info = os.lstat(path) + if name.endswith(".promisor"): + raise ValueError("pool local-copy source must not contain promisor objects") + if (info.st_uid != os.getuid() or + not (stat.S_ISDIR(info.st_mode) if name in directories else stat.S_ISREG(info.st_mode))): + raise ValueError("pool local-copy objects must be owned regular files/directories, without symlinks") + + +def verify_copy(source_gitdir, destination_gitdir): + """Prove local transport made separate files, not alternates or hardlinks.""" + validate_local_source(source_gitdir) + validate_local_source(destination_gitdir) + objects = os.path.join(destination_gitdir, "objects") + count = 0 + for parent, directories, files in os.walk(objects): + for name in files: + destination = os.path.join(parent, name) + if os.stat(destination).st_nlink != 1: + raise ValueError("pool private source copy retained a hardlinked object") + original = os.path.join(source_gitdir, os.path.relpath(destination, destination_gitdir)) + if os.path.exists(original): + left, right = os.stat(original), os.stat(destination) + if (left.st_dev, left.st_ino) == (right.st_dev, right.st_ino): + raise ValueError("pool private source copy retained a hardlinked object") + count += 1 + return {"independent_object_files": count, "hardlinks": False, "alternates": False, "shallow": False} + + def _identity(path, directory=False): info = os.lstat(path) if (os.path.realpath(path) != path or info.st_uid != os.getuid() @@ -25,6 +67,7 @@ def verify(execution, commit, root, environment): if not path.startswith(allocation_root + "/") or os.path.realpath(path) != path: raise ValueError("pool assets must be on receiving allocation-local disk") source, image = assets["source"], assets["image"] + validate_local_source(source) env = dict(environment, GIT_NO_LAZY_FETCH="1", GIT_ALLOW_PROTOCOL="", GIT_OPTIONAL_LOCKS="0") def git(*args): return subprocess.check_output(["git", "-C", source] + list(args), env=env, @@ -75,3 +118,9 @@ def git(*args): return dict(identity, verification_receipt=receipt, commit=commit) finally: os.rmdir(lock) + + +if __name__ == "__main__": + if len(sys.argv) != 4 or sys.argv[1] != "--verify-copy": + sys.exit("internal pool helper: expected --verify-copy SOURCE_GIT DESTINATION_GIT") + print(json.dumps(verify_copy(sys.argv[2], sys.argv[3]), sort_keys=True)) diff --git a/sucoder/local_tier.py b/sucoder/local_tier.py index 741c195..8950679 100644 --- a/sucoder/local_tier.py +++ b/sucoder/local_tier.py @@ -409,10 +409,12 @@ def build_worker_prepare_script(*, results_path: str, dispatch_id: str, commit: 'mkdir -p "$SUCODER_LOCAL_ROOT/mirrors" "$TMPDIR" "$UV_CACHE_DIR" "$PIP_CACHE_DIR" "$npm_config_cache"', ('WORK="$SUCODER_LOCAL_ROOT/mirrors/dispatch_' + dispatch_id + '"' if isolated else "WORK=" + work_path_shell(local_disk_root, "dispatch_" + dispatch_id)), 'if [ -e "$WORK" ]; then echo "dispatch clone already exists" >&2; exit 1; fi', - "git " + ("-c pack.threads=1 " if isolated else "") + "-c protocol.file.allow=always clone --quiet --no-local --no-hardlinks --no-checkout " + ('"$SUCODER_POOL_SOURCE_REPOSITORY"' if isolated else shlex.quote(results_path)) + ' "$WORK"', + "git " + ("-c pack.threads=1 " if isolated else "") + "-c protocol.file.allow=always clone --quiet " + ("--local" if isolated else "--no-local") + " --no-hardlinks --no-checkout " + ('"$SUCODER_POOL_SOURCE_REPOSITORY"' if isolated else shlex.quote(results_path)) + ' "$WORK"', + *(["python3 " + shlex.quote(results_path.rsplit("/", 1)[0] + "/dispatch_pool_assets.py") + ' --verify-copy "$SUCODER_POOL_SOURCE_REPOSITORY" "$WORK/.git" >' + shlex.quote(results_path.rsplit("/", 1)[0] + "/source-copy-receiving.json")] if isolated else []), 'cd "$WORK"', "git remote set-url origin " + shlex.quote(results_path), *(["git -c pack.threads=1 -c protocol.file.allow=always fetch --quiet --no-tags --no-recurse-submodules origin " + commit] if isolated else []), + *(['git fsck --connectivity-only --no-reflogs >/dev/null'] if isolated else []), "git checkout --quiet -b " + shlex.quote("dispatch/" + dispatch_id) + " " + commit, "git config user.name 'Sue the Coder'", "git config user.email coder@sucoder.dev", "cat > .git/hooks/post-commit <<'SUCODER_HOOK'", build_post_commit_hook(results_path).rstrip(), diff --git a/tests/test_dispatch_pool.py b/tests/test_dispatch_pool.py index 1f45e1e..c93dbf1 100644 --- a/tests/test_dispatch_pool.py +++ b/tests/test_dispatch_pool.py @@ -252,6 +252,7 @@ def git(path, *args): subprocess.check_call(['git', 'clone', '-q', '--bare', '--no-hardlinks', str(source), str(results)]) root = tmp_path / 'job234/workers' / ('a' * 32) root.mkdir(parents=True) + (tmp_path / 'dispatch_pool_assets.py').write_bytes(Path(assets.__file__).read_bytes()) script = build_worker_prepare_script(results_path=str(results), dispatch_id='a' * 32, commit=commit, isolated=True) env = dict(os.environ, SLURM_JOB_ID='234', SUCODER_LOCAL_ROOT=str(root), SUCODER_POOL_SOURCE_REPOSITORY=str(cache), TMPDIR=str(root / 'tmp'), UV_CACHE_DIR=str(root / 'cache/uv'), PIP_CACHE_DIR=str(root / 'cache/pip'), @@ -263,6 +264,8 @@ def git(path, *args): assert not (work / '.git/objects/info/alternates').exists() assert (work / 'tracked').read_text() == 'private candidate' assert git(cache, 'show-ref') == cache_refs + receipt = json.loads((tmp_path / 'source-copy-receiving.json').read_text()) + assert receipt['independent_object_files'] > 0 and receipt['hardlinks'] is False assert subprocess.run(['git', '-C', str(cache), 'cat-file', '-e', commit], stderr=subprocess.DEVNULL).returncode != 0 @@ -341,3 +344,78 @@ def submit(*args, **kwargs): assert result.exit_code == 0, result.output assert seen['required_evidence'] == ['owner-status.json', 'handoff.org'] assert seen['snapshot_minutes'] == 2 and seen['directory_root'] == tmp_path + + +def test_pool_staging_uses_independent_local_object_copy(attached, source, monkeypatch): + real = dispatch._git + calls = [] + def git(path, *args, **kwargs): + calls.append(args) + return real(path, *args, **kwargs) + monkeypatch.setattr(dispatch, '_git', git) + monkeypatch.setattr(pool, 'launch_step', lambda directory, manifest, argv, report, timeout: report) + original = source['repo'] / '.git' + before = {str(p.relative_to(original)): p.read_bytes() for p in original.rglob('*') if p.is_file()} + report = pool.submit(attached, **request(source)) + clone = next(args for args in calls if 'clone' in args) + assert '--local' in clone and '--no-hardlinks' in clone and '--no-local' not in clone + destination = Path(report['directory']) / 'results.git' + assert report['source_copy']['independent_object_files'] > 0 + for p in (destination / 'objects').rglob('*'): + other = original / p.relative_to(destination) + if p.is_file() and other.exists(): + assert (p.stat().st_dev, p.stat().st_ino) != (other.stat().st_dev, other.stat().st_ino) + after = {str(p.relative_to(original)): p.read_bytes() for p in original.rglob('*') if p.is_file()} + assert before == after + assert not (destination / 'objects/info/alternates').exists() + + +@pytest.mark.parametrize('kind', ['alternates', 'http-alternates', 'shallow', 'object-symlink']) +def test_pool_local_copy_refuses_dependent_or_redirected_sources(attached, source, monkeypatch, kind): + metadata = source['repo'] / '.git' + if kind in ('alternates', 'http-alternates'): + (metadata / 'objects/info' / kind).write_text('/unread/foreign/objects\n') + elif kind == 'shallow': + commit = subprocess.check_output(['git', '-C', str(source['repo']), 'rev-parse', 'HEAD'], text=True).strip() + (metadata / 'shallow').write_text(commit + '\n') + else: + (metadata / 'objects/link').symlink_to(source['repo'] / 'tracked') + monkeypatch.setattr(pool, 'launch_step', lambda *a: pytest.fail('unsafe source reached launch')) + with pytest.raises(ValueError, match='alternates|symlinks'): + pool.submit(attached, **request(source)) + assert not dispatch.DEFAULT_DIRECTORY_ROOT.exists() + + +def test_hardlinked_copy_is_rejected(source, tmp_path): + destination = tmp_path / 'unsafe.git' + subprocess.check_call(['git', 'clone', '--bare', '--local', '-q', str(source['repo']), str(destination)]) + with pytest.raises(ValueError, match='hardlinked'): + assets.verify_copy(str(source['repo'] / '.git'), str(destination)) + + +def test_independent_job_keeps_no_local_transport(source, monkeypatch): + from tests.test_dispatch import scheduler + scheduler(monkeypatch) + real = dispatch._git + calls = [] + def git(path, *args, **kwargs): + calls.append(args) + return real(path, *args, **kwargs) + monkeypatch.setattr(dispatch, '_git', git) + dispatch.submit(**source) + clone = next(args for args in calls if 'clone' in args) + assert '--no-local' in clone and '--local' not in clone + + +def test_nonregular_object_refuses_without_opening_fifo(source): + metadata = source['repo'] / '.git' + os.mkfifo(metadata / 'objects/owned-fifo') + with pytest.raises(ValueError, match='regular files'): + assets.validate_local_source(str(metadata)) + + +def test_promisor_marker_refuses_even_without_config(source): + metadata = source['repo'] / '.git' + (metadata / 'objects/pack/empty.promisor').write_bytes(b'') + with pytest.raises(ValueError, match='promisor'): + assets.validate_local_source(str(metadata)) From a7b25736bd62788ba522b5af2038047b6350f8b2 Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 22:47:09 -0700 Subject: [PATCH 06/18] Record pool local-copy verification for issue 77 Co-Authored-By: GPT-6 --- docs/dispatch-pool-ledger.org | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index f85e9fe..7c37caf 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -97,3 +97,17 @@ Receiving immutable-cache cloning uses the same independent-copy transport. Verify copied object inode separation, self-contained closure and unchanged source bytes/refs with disposable repositories; do not accept alternates or symlinked objects. Existing exact-SHA private-results fetch remains unchanged. + +** Clone-transport amendment verification +Production/test amendment cd40f6c3bfcf21b8f76d9a628752f7a5a9e7a863 passed the +finite pool/control/local-tier selection (153 passed), then the final pool-only +selection (34 passed, including FIFO/promisor refusal controls). Tests exercised +actual disposable-Git local copying, inode/link-count separation, unchanged source +bytes/refs, a newer private SHA absent the immutable cache, hardlink rejection, +and the unchanged independent-job --no-local transport. No model, scheduler or +real-project staging was performed by the author. The receiving worker module +remains byte-identical to root's proven ownership/cleanup helper. +GitNexus detect-changes again failed with the recorded libssl.so.3 limitation; +static checks and Python 3.6 helper compilation passed. Real-repository latency +and a new canary transaction remain root-owned release measurements. Preserve +original Tanzania staging timeout and its explicit prelaunch reconciliation. From 1f8f5d64099158feab60d91ddeb5f692980091b8 Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 23:00:57 -0700 Subject: [PATCH 07/18] Record receiving Python binding correction for pool dispatch Co-Authored-By: GPT-6 --- docs/dispatch-pool-ledger.org | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index 7c37caf..3d3e17f 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -111,3 +111,15 @@ GitNexus detect-changes again failed with the recorded libssl.so.3 limitation; static checks and Python 3.6 helper compilation passed. Real-repository latency and a new canary transaction remain root-owned release measurements. Preserve original Tanzania staging timeout and its explicit prelaunch reconciliation. + +** Additive receiving-Python batch binding +Root reports the first local-copy canary completed project bootstrap, then +refused before managed-child Popen with PermissionError naming an empty path. +The code uses sys.executable to launch the managed runner; an unexported Bash +PATH can leave CPython's executable unresolved when argv[0] is only python3. +Root owns the exact empty-environment Slurm reproduction and original failures. +Plan: change only pooled batch generation to resolve python3 on the receiving +node with command -v, require an absolute executable path, and exec that path. +No controller-local Python path, worker cleanup change or ordinary-job change. +Test the generated batch with a disposable JSON-only worker in an empty local +environment; no native model/managed runner or Slurm launch in author tests. From 23dcf397fb51bbe8e2484dff724b413d0545b4c8 Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 23:03:00 -0700 Subject: [PATCH 08/18] Bind receiving Python absolutely for pooled batch workers Resolve python3 on the receiving node before exec so an unexported default Bash PATH cannot leave CPython sys.executable empty. Preserve the ordinary job bootstrap and worker ownership bytes. Co-Authored-By: GPT-6 --- sucoder/dispatch.py | 12 +++++++++++- tests/test_dispatch_pool.py | 17 +++++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/sucoder/dispatch.py b/sucoder/dispatch.py index 990fb7a..96dccd7 100644 --- a/sucoder/dispatch.py +++ b/sucoder/dispatch.py @@ -309,7 +309,17 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, if project is not None: _private_write(directory / "dispatch_project.py", Path(__file__).with_name("dispatch_project.py").read_text()) project_files.append("dispatch_project.py") - _private_write(directory / "batch.sh", "#!/bin/bash\nset -eu\numask 077\nexec python3 " + shlex.quote(str(directory / "worker.py")) + " " + shlex.quote(str(directory / "manifest.json")) + "\n", True) + interpreter = "exec python3 " + if _execution is not None: + # Resolve on the receiver, not the controller. With --export=NONE, + # Bash can have an unexported default PATH; CPython started as bare + # 'python3' may then report sys.executable == ''. The managed runner + # must receive an actual interpreter path for its own subprocess. + interpreter = ('SUCODER_RECEIVING_PYTHON=$(command -v python3)\n' + 'case "$SUCODER_RECEIVING_PYTHON" in /*) ;; *) exit 70;; esac\n' + 'test -x "$SUCODER_RECEIVING_PYTHON"\n' + 'exec "$SUCODER_RECEIVING_PYTHON" ') + _private_write(directory / "batch.sh", "#!/bin/bash\nset -eu\numask 077\n" + interpreter + shlex.quote(str(directory / "worker.py")) + " " + shlex.quote(str(directory / "manifest.json")) + "\n", True) _write_json(directory / "digests.json", {name: hashlib.sha256((directory / name).read_bytes()).hexdigest() for name in ["task.txt", "prepare.sh", "snapshot.sh", "runner.py", "worker.py", "dispatch_protocol.py", "dispatch_artifacts.py", "environment.json", "manifest.json"] + project_files}) # Persist the token before contacting Slurm. A crash here is conservatively unknown. report["state"] = "submission_unknown" diff --git a/tests/test_dispatch_pool.py b/tests/test_dispatch_pool.py index c93dbf1..bdba389 100644 --- a/tests/test_dispatch_pool.py +++ b/tests/test_dispatch_pool.py @@ -419,3 +419,20 @@ def test_promisor_marker_refuses_even_without_config(source): (metadata / 'objects/pack/empty.promisor').write_bytes(b'') with pytest.raises(ValueError, match='promisor'): assets.validate_local_source(str(metadata)) + + +def test_pool_batch_resolves_absolute_receiving_python_without_exported_path(attached, source, monkeypatch): + monkeypatch.setattr(pool, 'launch_step', lambda directory, manifest, argv, report, timeout: report) + report = pool.submit(attached, **request(source)) + directory = Path(report['directory']) + # Only this disposable staged worker is replaced: no managed runner, + # provider, model, or scheduler runs. Exercise the actual generated batch. + (directory / 'worker.py').write_text('import json, sys\nprint(json.dumps({"executable": sys.executable, "manifest": sys.argv[1]}))\n') + completed = subprocess.run(['/bin/bash', str(directory / 'batch.sh')], env={'LC_ALL': 'C'}, + cwd='/tmp', text=True, capture_output=True, timeout=10) + assert completed.returncode == 0, completed.stderr + observed = json.loads(completed.stdout) + assert os.path.isabs(observed['executable']) + assert observed['manifest'] == str(directory / 'manifest.json') + assert os.access(observed['executable'], os.X_OK) + assert 'SUCODER_RECEIVING_PYTHON' in (directory / 'batch.sh').read_text() From 5ca1cc79b240fb5ba66778149012e41d3141377d Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 23:04:09 -0700 Subject: [PATCH 09/18] Record receiving Python bootstrap counterfactual and verification Co-Authored-By: GPT-6 --- docs/dispatch-pool-ledger.org | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index 3d3e17f..52b0c82 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -123,3 +123,17 @@ node with command -v, require an absolute executable path, and exec that path. No controller-local Python path, worker cleanup change or ordinary-job change. Test the generated batch with a disposable JSON-only worker in an empty local environment; no native model/managed runner or Slurm launch in author tests. + +** Receiving-Python correction verification +Production/test pin 23dcf397fb51bbe8e2484dff724b413d0545b4c8 passes 123 finite +pool/control cases. The added actual generated-batch control first failed on the +old template because sys.executable was the empty string, then passed with an +absolute receiving interpreter under the same empty environment. Only a +JSON-emitting disposable worker ran; no native model/managed runner/Slurm actor. +Root separately confirmed the same bare-versus-absolute Python distinction in an +owned held-allocation step and retained step-python-executable-proof.json. +The ordinary-job template and dispatch_worker.py remain unchanged. The latter's +SHA256 is still b09ef8620c4e4ab80c867044dafc9dc0c84e10d67c0eb68eea49767068d6f762. +Original canary failures remain preserved; new productive/native canaries are +root-owned releases, not implied by this local test. GitNexus remains unavailable +with the recorded native libssl.so.3 load error; static/compile checks passed. From 52a999c0e3d19e6cab6bb1f8beb0489a0537fc60 Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 23:18:50 -0700 Subject: [PATCH 10/18] Record bounded incarnation-bound terminal proof plan Co-Authored-By: GPT-6 --- docs/dispatch-pool-ledger.org | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index 52b0c82..4823036 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -137,3 +137,17 @@ SHA256 is still b09ef8620c4e4ab80c867044dafc9dc0c84e10d67c0eb68eea49767068d6f762 Original canary failures remain preserved; new productive/native canaries are root-owned releases, not implied by this local test. GitNexus remains unavailable with the recorded native libssl.so.3 load error; static/compile checks passed. + +** Bounded terminal-proof continuation after metadata refusal +Root retained four native-owner failures at descriptor directory/mount guards +(errno13/22); current survivors must remain untouched. EINVAL/EACCES alone are +not exit proof, and vanished PIDs after cleanup do not explain original cause. +Extend the34070 terminal-proof precedent with a <=50ms total inspection-pass +polling budget after metadata errors, max11 attempts at5ms intervals. Anchor +proof to owned real UID and original starttime. Only that incarnation's fresh +Z/X or a fully missing proc directory authorizes skipping it; changed UID/starttime, +live or unreadable/unknown existing processes never pass. Do not stat descriptor +targets. Keep failure category/errno/proc filename and incarnation/proof diagnostics. +Use fake clock/proc controls before production commit; enrich old synthetic +fixtures with actual UID/starttime fields where needed without weakening their +assertions. Root owns actual lifecycle probes and new managed transactions. From a76c753e9358599ce410b8795a5681c06af37fc8 Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 23:27:03 -0700 Subject: [PATCH 11/18] Bound terminal-proof polling to owned process incarnations After descriptor metadata errors, wait at most50ms for fresh same-UID/starttime Z/X or fully gone proof. Preserve live/unknown/PID-reuse refusal and retain error-boundary/incarnation diagnostics; never stat descriptor targets. Co-Authored-By: GPT-6 --- sucoder/dispatch_worker.py | 94 +++++++++++++++++++++-- tests/test_dispatch_exit_races.py | 9 ++- tests/test_dispatch_terminal_grace.py | 104 ++++++++++++++++++++++++++ 3 files changed, 200 insertions(+), 7 deletions(-) create mode 100644 tests/test_dispatch_terminal_grace.py diff --git a/sucoder/dispatch_worker.py b/sucoder/dispatch_worker.py index a2e6828..89ee973 100644 --- a/sucoder/dispatch_worker.py +++ b/sucoder/dispatch_worker.py @@ -645,18 +645,98 @@ def process_has_exited(pid, proc_root): return False +def process_identity(pid, proc_root): + """Read only kernel identity metadata; an incomplete snapshot is unknown.""" + base = os.path.join(proc_root, str(pid)) + try: + with open(os.path.join(base, "stat")) as stream: + first = stream.read().rsplit(")", 1)[1].split() + with open(os.path.join(base, "status")) as stream: + match = re.search(r"^Uid:\s+([0-9]+)\s+", stream.read(), re.M) + with open(os.path.join(base, "stat")) as stream: + last = stream.read().rsplit(")", 1)[1].split() + if match is None or first[19] != last[19] or first[0] != last[0]: + return None + return {"uid": int(match.group(1)), "starttime": int(last[19]), "state": last[0]} + except (OSError, IndexError, ValueError): + return None + + +def terminal_proof_after_error(pid, proc_root, expected, deadline=None): + """Wait briefly for terminal proof, never accept an unreadable live PID. + + UID/incarnation changes refuse even if a reused PID is now a zombie. All + failures in one inspection share the caller's <=50ms monotonic deadline. + """ + began = time.monotonic() + deadline = min(began + .05, deadline) if deadline is not None else began + .05 + proof = {"terminal": False, "reason": "unknown_identity", "expected": expected, + "attempts": 0, "last_identity": None} + if not isinstance(expected, dict) or any(type(expected.get(key)) is not int for key in ("uid", "starttime")): + proof["elapsed_seconds"] = max(0, time.monotonic() - began) + return proof + for attempt in range(11): + proof["attempts"] = attempt + 1 + try: + os.stat(os.path.join(proc_root, str(pid))) + except OSError as exc: + if exc.errno in (errno.ENOENT, errno.ESRCH): + proof.update(terminal=True, reason="proc_directory_gone") + break + current = process_identity(pid, proc_root) + proof["last_identity"] = current + if current is not None: + if any(current[key] != expected[key] for key in ("uid", "starttime")): + proof["reason"] = "identity_changed" + break + if current["state"] in ("Z", "X"): + proof.update(terminal=True, reason="same_incarnation_" + current["state"]) + break + proof["reason"] = "still_live" + else: + proof["reason"] = "unknown_identity" + remaining = deadline - time.monotonic() + if remaining <= 0 or attempt == 10: + break + time.sleep(min(.005, remaining)) + proof["elapsed_seconds"] = max(0, time.monotonic() - began) + return proof + + def inspect_placement(child, runner, root, home, proc_root="/proc", expected_environment=None, ownership=None, manifest=None): """Reuse the managed runner's process/FD machinery, checking readability.""" members = ownership.members() if ownership is not None else [] pids = [record["pid"] for record in members] if ownership is not None else [child.pid] + runner.descendants(child.pid) + identities = {record["pid"]: {"uid": getattr(ownership, "uid", os.getuid()), "starttime": record.get("starttime")} + for record in members if type(record.get("starttime")) is int} + races, grace = [], [None] + def terminal_after_error(pid, exc, boundary): + if grace[0] is None: + grace[0] = time.monotonic() + .05 + proof = terminal_proof_after_error(pid, proc_root, identities.get(pid), deadline=grace[0]) + filename = getattr(exc, "filename", None) + context = {"pid": pid, "boundary": boundary, "errno": exc.errno, + "filename": filename if isinstance(filename, str) and beneath_text(filename, os.path.join(proc_root, str(pid))) else None, + "proof": proof} + if proof["terminal"]: + races.append(context) + return True + failure = Refused("cannot inspect owned child %s (pid=%s errno=%s identity=%s proof=%s)" % + (boundary, pid, exc.errno, identities.get(pid), proof["reason"])) + failure.inspection_failure = context + raise failure readable = [] for pid in pids: + if pid not in identities: + captured = process_identity(pid, proc_root) + if captured is not None: + identities[pid] = {key: captured[key] for key in ("uid", "starttime")} path = os.path.join(proc_root, str(pid), "fd") try: entries = os.listdir(path) except OSError as exc: - if exc.errno in (errno.ENOENT, errno.ESRCH) or process_has_exited(pid, proc_root): + if exc.errno in (errno.ENOENT, errno.ESRCH) or terminal_after_error(pid, exc, "file descriptors"): continue # process teardown can remove fd/ before its proc dir raise Refused("cannot inspect owned child file descriptors (pid=%s errno=%s)" % (pid, exc.errno)) # Descriptor links are proc metadata: never stat a magic FD link, @@ -668,7 +748,7 @@ def inspect_placement(child, runner, root, home, proc_root="/proc", expected_env os.readlink(descriptor) except OSError as exc: if exc.errno not in (errno.ENOENT, errno.ESRCH): - if process_has_exited(pid, proc_root): + if terminal_after_error(pid, exc, "descriptor links"): exited = True break raise Refused("cannot inspect owned child descriptor links (pid=%s errno=%s)" % (pid, exc.errno)) @@ -680,7 +760,7 @@ def inspect_placement(child, runner, root, home, proc_root="/proc", expected_env with open(os.path.join(proc_root, str(child.pid), "environ"), "rb") as stream: effective = dict(entry.split(b"=", 1) for entry in stream.read().split(b"\x00") if b"=" in entry) except OSError as exc: - if exc.errno not in (errno.ENOENT, errno.ESRCH) and not process_has_exited(child.pid, proc_root): + if exc.errno not in (errno.ENOENT, errno.ESRCH) and not terminal_after_error(child.pid, exc, "effective environment"): raise Refused("cannot inspect owned child effective environment (pid=%s errno=%s)" % (child.pid, exc.errno)) else: for key in ("HOME", "CODEX_HOME", "CLAUDE_CONFIG_DIR", "SUCODER_LOCAL_ROOT"): @@ -693,7 +773,7 @@ def inspect_placement(child, runner, root, home, proc_root="/proc", expected_env try: descriptors.extend(runner.inspect_file_mounts([pid], proc_root=proc_root)) except OSError as exc: - if process_has_exited(pid, proc_root): + if terminal_after_error(pid, exc, "descriptor mounts"): continue raise Refused("cannot inspect owned child descriptor mounts (pid=%s errno=%s)" % (pid, exc.errno)) for descriptor in descriptors: @@ -727,7 +807,7 @@ def inspect_placement(child, runner, root, home, proc_root="/proc", expected_env return {"observed": sorted(set(observed)), "databases": sorted(set(databases)), "misplaced": sorted(set(misplaced)), "inspected_pids": readable, "environment_verified": environment_verified, "members": members, - "durable_checkpoint_copies": sorted(set(checkpoint_copies))} + "durable_checkpoint_copies": sorted(set(checkpoint_copies)), "inspection_races": races} def stop_owned(child, timeout): @@ -1010,6 +1090,8 @@ def final_publication(): record["placement"].update(observed=sorted(observed), sqlite="observed_local" if databases else "not_observed", databases=sorted(databases), inspected_pids=evidence["inspected_pids"]) record["placement"]["durable_checkpoint_copies"] = evidence["durable_checkpoint_copies"] + if evidence.get("inspection_races"): + record["placement"].setdefault("inspection_races", []).extend(evidence["inspection_races"]) record["placement"]["environment_verified"] = (record["placement"].get("environment_verified", False) or evidence["environment_verified"]) if evidence["misplaced"]: @@ -1068,6 +1150,8 @@ def final_publication(): else: record["state"] = "failed" except Exception as exc: + if getattr(exc, "inspection_failure", None) is not None: + record["inspection_failure"] = exc.inspection_failure if record.get("project", {}).get("state") == "preparing": record["project"].update(state="failed", reason=str(exc)) # Project setup can start mount daemons before any native child exists. diff --git a/tests/test_dispatch_exit_races.py b/tests/test_dispatch_exit_races.py index b551e45..7450b8a 100644 --- a/tests/test_dispatch_exit_races.py +++ b/tests/test_dispatch_exit_races.py @@ -19,14 +19,19 @@ def test_permission_race_requires_fresh_exit_proof(tmp_path, monkeypatch, bounda process = tmp_path / "123" (process / "fd").mkdir(parents=True) status = process / "stat" - status.write_text("123 (native child) R 1 2\n") + def write_state(value): + fields = [value] + ["0"] * 19 + fields[19] = "42" + status.write_text("123 (native child) " + " ".join(fields) + "\n") + (process / "status").write_text("Uid:\t%s\t%s\t%s\t%s\n" % ((os.getuid(),) * 4)) + write_state("R") original_listdir, original_readlink, original_open = os.listdir, os.readlink, builtins.open def deny_after_transition(): if state == "unknown": status.unlink() else: - status.write_text("123 (native child) " + state + " 1 2\n") + write_state(state) raise OSError(metadata_errno, "process changed during inspection") def listdir(path): diff --git a/tests/test_dispatch_terminal_grace.py b/tests/test_dispatch_terminal_grace.py new file mode 100644 index 0000000..a535426 --- /dev/null +++ b/tests/test_dispatch_terminal_grace.py @@ -0,0 +1,104 @@ +"""Finite fake-kernel terminal transitions; no scheduler/native/store access.""" +import errno +import os +from types import SimpleNamespace + +import pytest +from sucoder import dispatch_worker as worker + + +def process(root, state='R', started=42, uid=None): + directory = root / '123' + directory.mkdir(exist_ok=True) + fields = [state] + ['0'] * 19 + fields[19] = str(started) + (directory / 'stat').write_text('123 (fake child) ' + ' '.join(fields) + '\n') + uid = os.getuid() if uid is None else uid + (directory / 'status').write_text('Uid:\t%s\t%s\t%s\t%s\n' % ((uid,) * 4)) + return directory + + +def clock(monkeypatch, callback=lambda now: None): + value = [0.0] + def sleep(seconds): + assert 0 <= seconds <= .005 + value[0] += seconds + callback(value[0]) + monkeypatch.setattr(worker.time, 'monotonic', lambda: value[0]) + monkeypatch.setattr(worker.time, 'sleep', sleep) + return value + + +@pytest.mark.parametrize('boundary', ['directory', 'mounts']) +def test_delayed_terminal_transition_preserves_strict_inspection(tmp_path, monkeypatch, boundary): + directory = process(tmp_path) + (directory / 'fd').mkdir() + value = clock(monkeypatch, lambda now: process(tmp_path, state='Z') if now >= .01 else None) + original = worker.os.listdir + def listdir(path): + if str(path) == str(directory / 'fd') and boundary == 'directory': + raise OSError(errno.EACCES, 'kernel transition', str(path)) + return original(path) + monkeypatch.setattr(worker.os, 'listdir', listdir) + def mounts(*args, **kwargs): + raise OSError(errno.EINVAL, 'kernel transition', str(directory / 'mountinfo')) + owner = SimpleNamespace(uid=os.getuid(), members=lambda: [{'pid': 123, 'starttime': 42}]) + result = worker.inspect_placement(SimpleNamespace(pid=123), SimpleNamespace(inspect_file_mounts=mounts), + str(tmp_path), str(tmp_path), proc_root=str(tmp_path), ownership=owner) + proof = result['inspection_races'][0]['proof'] + assert proof['terminal'] and proof['reason'] == 'same_incarnation_Z' + assert proof['attempts'] >= 3 and .01 <= value[0] <= .05 + assert result['databases'] == result['misplaced'] == [] + + +@pytest.mark.parametrize('state', ['R', 'S', 'D']) +def test_persistent_live_identity_refuses_at_finite_bound(tmp_path, monkeypatch, state): + process(tmp_path, state=state) + value = clock(monkeypatch) + result = worker.terminal_proof_after_error(123, str(tmp_path), {'uid': os.getuid(), 'starttime': 42}) + assert not result['terminal'] and result['reason'] == 'still_live' + assert result['attempts'] <= 11 and value[0] <= .05 + + +@pytest.mark.parametrize('changed', ['uid', 'starttime']) +def test_reused_or_changed_identity_never_passes_even_as_zombie(tmp_path, monkeypatch, changed): + process(tmp_path) + def change(now): + if now >= .005: + process(tmp_path, state='Z', started=43 if changed == 'starttime' else 42, + uid=os.getuid() + 1 if changed == 'uid' else os.getuid()) + clock(monkeypatch, change) + result = worker.terminal_proof_after_error(123, str(tmp_path), {'uid': os.getuid(), 'starttime': 42}) + assert not result['terminal'] and result['reason'] == 'identity_changed' + + +@pytest.mark.parametrize('missing', ['stat', 'status', 'malformed']) +def test_existing_unknown_process_is_not_terminal(tmp_path, monkeypatch, missing): + directory = process(tmp_path) + if missing == 'malformed': + (directory / 'stat').write_text('malformed') + else: + (directory / missing).unlink() + value = clock(monkeypatch) + result = worker.terminal_proof_after_error(123, str(tmp_path), {'uid': os.getuid(), 'starttime': 42}) + assert not result['terminal'] and value[0] <= .05 + + +def test_fully_absent_process_is_explicit_terminal_proof(tmp_path, monkeypatch): + clock(monkeypatch) + result = worker.terminal_proof_after_error(123, str(tmp_path), {'uid': os.getuid(), 'starttime': 42}) + assert result['terminal'] and result['reason'] == 'proc_directory_gone' + + +def test_shared_pass_deadline_prevents_new_full_grace(tmp_path, monkeypatch): + process(tmp_path) + value = clock(monkeypatch) + result = worker.terminal_proof_after_error(123, str(tmp_path), {'uid': os.getuid(), 'starttime': 42}, deadline=.01) + assert not result['terminal'] and value[0] <= .01 + + +def test_unknown_original_identity_cannot_use_later_absence(tmp_path, monkeypatch): + clock(monkeypatch) + result = worker.terminal_proof_after_error(123, str(tmp_path), None) + assert not result['terminal'] and result['reason'] == 'unknown_identity' + assert result['attempts'] == 0 From 105d08fb308a68168ac701c8a7f80c5af6bd48b6 Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Sun, 25 Oct 2026 23:29:50 -0700 Subject: [PATCH 12/18] Record bounded terminal-proof counterfactual and independent gate Co-Authored-By: GPT-6 --- docs/dispatch-pool-ledger.org | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index 4823036..ea605e7 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -151,3 +151,22 @@ targets. Keep failure category/errno/proc filename and incarnation/proof diagnos Use fake clock/proc controls before production commit; enrich old synthetic fixtures with actual UID/starttime fields where needed without weakening their assertions. Root owns actual lifecycle probes and new managed transactions. + +** Incarnation-bound terminal continuation verification +Exact production/test pin a76c753e9358599ce410b8795a5681c06af37fc8 passes 206 +finite fake/core/pool cases in20.75s, including52 separately authored immutable +fake-clock negative/positive controls. Both delayed fd-directory and mount cases +fail meaningfully against the unchanged5ca baseline worker, before its one-shot +proof can observe the later terminal transition. Baseline carrier differs only +in importing that frozen module; original logs and carrier remain preserved. +The34070 assertions are unchanged; their fake UID/stat fixture now supplies the +original-incarnation metadata needed for stronger proof. Unknown original +identity cannot use later absence, and reused/changed-UID zombies refuse. + +Independent source review at a76c found no remaining blocker. Worker SHA256: +c53c405017a7b7a2f7022031061d4834f56a95347713394ad4dcd38be04a2702. +GitNexus still fails on libssl.so.3; static and receiving-Python3.6 compilation +passed. These synthetic controls establish the mechanism, not the cause of old +actor failures. Root retains the real transition probe, partial-state recovery +and any new productive managed transaction release. No author native/Slurm +launch, state alteration, cancellation or retry occurred. From b1ecf3d5878723cf58cfd01fe52cd3ca57e8d372 Mon Sep 17 00:00:00 2001 From: Sue the Coder Date: Wed, 28 Oct 2026 12:25:51 -0700 Subject: [PATCH 13/18] Retry a complete placement inspection after recoverable metadata errors Reinspect once after a still-live metadata refusal, preserving all placement checks and refusing known UID/starttime changes before later absence can erase the observation. Retain the initial failure and outcome. Record independent controls and real nondumpable-child checks; native lifecycle acceptance remains pending. Co-Authored-By: GPT-6 Codex --- docs/dispatch-pool-ledger.org | 49 ++++++ sucoder/dispatch_worker.py | 58 ++++++- tests/test_dispatch_reinspection.py | 251 ++++++++++++++++++++++++++++ 3 files changed, 356 insertions(+), 2 deletions(-) create mode 100644 tests/test_dispatch_reinspection.py diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index ea605e7..0b0b1a9 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -170,3 +170,52 @@ passed. These synthetic controls establish the mechanism, not the cause of old actor failures. Root retains the real transition probe, partial-state recovery and any new productive managed transaction release. No author native/Slurm launch, state alteration, cancellation or retry occurred. + +** One complete reinspection after recoverable live metadata failure -- 2026-10-28 + +Prior art is the105d08f terminal-proof mechanism above, process_identity and +inspect_placement, plus the immutable managed runner FD/mount reader. +A real owned-child counterfactual changed dumpability from0 to1 after20ms: +FD access recovered while the same UID/starttime remained S; the original +50ms identity-only proof still refused. A second complete placement inspection +passed. A persistent nondumpable child remained refused. This demonstrates a +mechanism limitation, not the cause of five native country-worker failures. + +Reuse the original full inspection unchanged as an inner pass. Allow exactly +ONE new full pass only after an inspection_failure with still_live proof and +fresh identical UID/starttime. Reinspect every process/environment/descriptor +and placement rule. After successful retry require the original failed PID to +be fully inspected at the same live incarnation, or fresh terminal/gone proof. +Changed/unknown identities, persistent unreadability, misplaced stores and all +other refusals must still stop. Never exempt a live unreadable process. +Preserve initial failure and retry outcome in result/status/final failure. +The two inner passes retain their existing <=50ms proof budgets; no recursive +retry, age-based skip, target stat, deployment or worker restart is implied. + +Impact analysis was attempted with installed GitNexus CLI in the new isolated +clone, but native LadybugDB cannot load libssl.so.3. The same missing-library +limitation is recorded in the preceding accepted ledger. Static call analysis +finds one production caller in dispatch_worker's run supervision loop, +plus direct placement/exit/grace tests. The change affects the placement gate +for both agent modes; this is security-sensitive. Frozen105d08f source and all +running workers remain untouched. Independent review and real probes precede +any new frozen release/native acceptance transaction. + +Independent source review found a reused-PID race in the first wrapper draft: +a known changed UID/starttime fell through to a later gone-directory proof. +Refuse a known changed identity immediately; disappearance cannot erase that +observation. Separate before/after-success disappearing-reuse controls are +required. Initial18cases passed after a fake-clock rounding assertion fix; +first full targeted cohort256passed. Preserve those earlier results as draft +mechanism checks, not acceptance of the review correction or deployment. + +Final corrected worker86f9f524 passes260 targeted tests in29.16seconds, +including22 newcases (two actual Linux nondumpable-child controls) and four +independent disappearing-reuse regressions. Those four fail meaningfully on +the exact prior13ea draft, pass on corrected86f; eleven additional pure-policy +controls retain their outcomes. The actual original inspect_placement AST is +unchanged; only new inspect_placement_rechecked and the run caller/diagnostic +persistence differ. Python3.6 grammar parses. GitNexus detect_changes was +attempted and fails the same missing libssl3; manual AST scope check passes. +These checks are not native lifecycle acceptance, deployment, scientific data +verification or an explanation of the five original country failures. diff --git a/sucoder/dispatch_worker.py b/sucoder/dispatch_worker.py index 89ee973..f080fd3 100644 --- a/sucoder/dispatch_worker.py +++ b/sucoder/dispatch_worker.py @@ -810,6 +810,58 @@ def terminal_after_error(pid, exc, boundary): "durable_checkpoint_copies": sorted(set(checkpoint_copies)), "inspection_races": races} +def inspect_placement_rechecked(child, runner, root, home, proc_root="/proc", expected_environment=None, + ownership=None, manifest=None): + """Retry one complete inspection after a recoverable same-incarnation error. + + Readability can recover while a process remains alive. A second complete + pass must actually inspect it; elapsed time or a live identity never + authorizes skipping unreadable metadata. The original terminal proof and + every placement check remain in force on both passes. + """ + arguments = dict(proc_root=proc_root, expected_environment=expected_environment, + ownership=ownership, manifest=manifest) + try: + return inspect_placement(child, runner, root, home, **arguments) + except Refused as initial: + context = getattr(initial, "inspection_failure", None) + if not isinstance(context, dict) or context.get("proof", {}).get("reason") != "still_live": + raise + pid, expected = context["pid"], context["proof"]["expected"] + + def verified_identity(): + current = process_identity(pid, proc_root) + if current is not None: + if any(current[key] != expected[key] for key in ("uid", "starttime")): + return None + return current + proof = terminal_proof_after_error(pid, proc_root, expected) + if proof["terminal"]: + return {"state": "terminal", "proof": proof} + return None + + before = verified_identity() + if before is None: + raise + try: + result = inspect_placement(child, runner, root, home, **arguments) + except Refused as final: + previous = getattr(final, "inspection_failure", None) + final.inspection_failure = dict(context, reinspection={ + "outcome": "refused", "failure": previous, "reason": str(final)}) + raise + after = verified_identity() + if after is None or (after["state"] not in ("Z", "X", "terminal") + and pid not in result["inspected_pids"]): + initial.inspection_failure = dict(context, reinspection={ + "outcome": "identity_unknown_changed_or_not_inspected", "after": after}) + raise initial + result["inspection_retries"] = [{"initial_failure": context, + "before": before, "after": after, + "outcome": "full_inspection_completed"}] + return result + + def stop_owned(child, timeout): # Popen(start_new_session=True) makes the child the owner of this group. # Signal the group even if its leader has exited but left descendants. @@ -1083,8 +1135,8 @@ def final_publication(): while child.poll() is None: if manifest.get("execution") is not None and time.time() >= manifest["execution"]["deadline"] - 120: stop_signal[0] = signal.SIGTERM - evidence = inspect_placement(child, runner, root, environment["HOME"], expected_environment=environment, - ownership=ownership, manifest=manifest) + evidence = inspect_placement_rechecked(child, runner, root, environment["HOME"], expected_environment=environment, + ownership=ownership, manifest=manifest) observed.update(evidence["observed"]) databases.update(evidence["databases"]) record["placement"].update(observed=sorted(observed), sqlite="observed_local" if databases else "not_observed", @@ -1092,6 +1144,8 @@ def final_publication(): record["placement"]["durable_checkpoint_copies"] = evidence["durable_checkpoint_copies"] if evidence.get("inspection_races"): record["placement"].setdefault("inspection_races", []).extend(evidence["inspection_races"]) + if evidence.get("inspection_retries"): + record["placement"].setdefault("inspection_retries", []).extend(evidence["inspection_retries"]) record["placement"]["environment_verified"] = (record["placement"].get("environment_verified", False) or evidence["environment_verified"]) if evidence["misplaced"]: diff --git a/tests/test_dispatch_reinspection.py b/tests/test_dispatch_reinspection.py new file mode 100644 index 0000000..6381bb7 --- /dev/null +++ b/tests/test_dispatch_reinspection.py @@ -0,0 +1,251 @@ +"""Recovered metadata must be fully reread; live/unknown refusal stays strict.""" +import builtins +import errno +import os +from pathlib import Path +import subprocess +import sys +from types import SimpleNamespace + +import pytest + +from sucoder import dispatch_worker as worker + + +def identity(root, uid=None, starttime=42): + base = root / '123' + base.mkdir(exist_ok=True) + fields = ['S'] + ['0'] * 19 + fields[19] = str(starttime) + (base / 'stat').write_text('123 (owned child) ' + ' '.join(fields)) + uid = os.getuid() if uid is None else uid + (base / 'status').write_text('Uid:\t%s\n' % uid) + return base + + +@pytest.fixture +def kernel(tmp_path, monkeypatch): + base = identity(tmp_path) + (base / 'fd').mkdir() + (base / 'environ').write_bytes(('HOME=' + str(tmp_path)).encode()) + now = [0.0] + monkeypatch.setattr(worker.time, 'monotonic', lambda: now[0]) + def sleep(seconds): + assert 0 <= seconds <= .005 + now[0] += seconds + monkeypatch.setattr(worker.time, 'sleep', sleep) + owner = SimpleNamespace(uid=os.getuid(), members=lambda: [{'pid': 123, 'starttime': 42}]) + child = SimpleNamespace(pid=123) + real_listdir, real_readlink, real_open = os.listdir, os.readlink, builtins.open + counts = dict(directory=0, link=0, environment=0, mounts=0) + + def bind(boundary, persistent=False, filesystem='ext4'): + def denied(name): + counts[name] += 1 + if name == boundary and (persistent or now[0] < .01): + raise OSError(errno.EINVAL if name == 'mounts' else errno.EACCES, + 'invented proc transition', str(base / name)) + def listdir(path): + if str(path) == str(base / 'fd'): + denied('directory') + return ['7'] + return real_listdir(path) + def readlink(path): + if str(path) == str(base / 'fd/7'): + denied('link') + return str(tmp_path / '.codex/fixture.sqlite') + return real_readlink(path) + def open_file(path, *args, **kwargs): + if str(path) == str(base / 'environ'): + denied('environment') + return real_open(path, *args, **kwargs) + def mounts(pids, **kwargs): + assert pids == [123] + denied('mounts') + return [{'pid': 123, 'path': str(tmp_path / '.codex/fixture.sqlite'), + 'filesystem': filesystem}] + monkeypatch.setattr(worker.os, 'listdir', listdir) + monkeypatch.setattr(worker.os, 'readlink', readlink) + monkeypatch.setattr(builtins, 'open', open_file) + runner = SimpleNamespace(inspect_file_mounts=mounts) + return lambda: worker.inspect_placement_rechecked( + child, runner, str(tmp_path), str(tmp_path), proc_root=str(tmp_path), + expected_environment={'HOME': str(tmp_path)}, ownership=owner) + return bind, now, counts, tmp_path + + +@pytest.mark.parametrize('boundary', ['directory', 'link', 'environment', 'mounts']) +def test_recovered_live_metadata_requires_a_complete_second_pass(kernel, boundary): + bind, now, counts, root = kernel + result = bind(boundary)() + assert result['inspected_pids'] == [123] + assert result['databases'] == [str(root / '.codex/fixture.sqlite')] + assert result['environment_verified'] and not result['misplaced'] + assert counts[boundary] == 2 + assert all(counts[name] >= 1 for name in counts) + proof = result['inspection_retries'][0] + assert proof['initial_failure']['proof']['reason'] == 'still_live' + assert proof['before']['starttime'] == proof['after']['starttime'] == 42 + assert proof['after']['state'] == 'S' + assert now[0] == pytest.approx(.05, abs=1e-12) + + +@pytest.mark.parametrize('boundary', ['directory', 'link', 'environment', 'mounts']) +def test_persistent_live_unreadability_still_refuses_after_one_retry(kernel, boundary): + bind, now, counts, _ = kernel + with pytest.raises(worker.Refused) as caught: + bind(boundary, persistent=True)() + assert counts[boundary] == 2 + assert now[0] == pytest.approx(.1, abs=1e-12) + diagnostic = caught.value.inspection_failure + assert diagnostic['proof']['reason'] == 'still_live' + assert diagnostic['reinspection']['outcome'] == 'refused' + assert diagnostic['reinspection']['failure']['proof']['reason'] == 'still_live' + + +@pytest.mark.parametrize('when', ['before', 'after']) +@pytest.mark.parametrize('change', ['uid', 'starttime', 'unknown']) +def test_changed_or_unknown_incarnation_never_passes_reinspection(kernel, monkeypatch, when, change): + bind, _, _, root = kernel + invoke = bind('directory') + original = worker.inspect_placement + def mutate(): + if change == 'unknown': + (root / '123/status').unlink() + else: + identity(root, uid=os.getuid() + 1 if change == 'uid' else None, + starttime=43 if change == 'starttime' else 42) + def inspected(*args, **kwargs): + try: + result = original(*args, **kwargs) + except worker.Refused: + if when == 'before': + mutate() + raise + if when == 'after': + mutate() + return result + monkeypatch.setattr(worker, 'inspect_placement', inspected) + with pytest.raises(worker.Refused): + invoke() + + +def test_full_retry_retains_misplaced_store_diagnosis(kernel): + bind, _, _, root = kernel + result = bind('directory', filesystem='nfs')() + assert result['misplaced'] == [str(root / '.codex/fixture.sqlite')] + assert result['databases'] == [] + + +def test_nonmetadata_refusal_is_not_retried(monkeypatch): + calls = [] + def refuse(*args, **kwargs): + calls.append(True) + raise worker.Refused('effective environment changed') + monkeypatch.setattr(worker, 'inspect_placement', refuse) + with pytest.raises(worker.Refused, match='effective environment changed'): + worker.inspect_placement_rechecked(None, None, '', '') + assert calls == [True] + + +@pytest.mark.skipif(sys.platform != 'linux', reason='Linux proc/dumpability boundary') +@pytest.mark.parametrize('recover', [False, True]) +def test_real_owned_child_readability_recovery_and_persistent_refusal(tmp_path, monkeypatch, recover): + code = '''import ctypes,sys +lib=ctypes.CDLL(None,use_errno=True) +print("START",flush=True) +assert sys.stdin.readline().strip()=="GO" +assert lib.prctl(4,0,0,0,0)==0 +print("HIDDEN",flush=True) +command=sys.stdin.readline().strip() +if command=="RESTORE": + assert lib.prctl(4,1,0,0,0)==0 + print("READABLE",flush=True) + command=sys.stdin.readline().strip() +assert command=="EXIT" +''' + child = subprocess.Popen([sys.executable, '-I', '-S', '-c', code], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, text=True) + try: + assert child.stdout.readline().strip() == 'START' + initial = worker.process_identity(child.pid, '/proc') + child.stdin.write('GO\n'); child.stdin.flush() + assert child.stdout.readline().strip() == 'HIDDEN' + original = os.listdir + denied = [] + def listdir(path): + try: + return original(path) + except OSError as exc: + if str(path) == '/proc/%s/fd' % child.pid: + assert exc.errno == errno.EACCES + denied.append(True) + if recover and len(denied) == 1: + child.stdin.write('RESTORE\n'); child.stdin.flush() + raise + monkeypatch.setattr(worker.os, 'listdir', listdir) + def mounts(pids, proc_root='/proc'): + for pid in pids: + with open('%s/%s/mountinfo' % (proc_root, pid), 'rb') as stream: + stream.read() + return [] + owner = SimpleNamespace(uid=os.getuid(), members=lambda: [ + {'pid': child.pid, 'starttime': initial['starttime']}]) + args = (child, SimpleNamespace(inspect_file_mounts=mounts), str(tmp_path), str(tmp_path)) + if recover: + result = worker.inspect_placement_rechecked(*args, ownership=owner) + assert result['inspected_pids'] == [child.pid] + assert result['inspection_retries'][0]['after']['starttime'] == initial['starttime'] + assert result['inspection_retries'][0]['after']['state'] not in ('Z', 'X') + assert denied == [True] + else: + with pytest.raises(worker.Refused): + worker.inspect_placement_rechecked(*args, ownership=owner) + assert len(denied) == 2 + finally: + child.stdin.write('EXIT\n'); child.stdin.flush() + child.wait(timeout=5) + assert child.returncode == 0, child.stderr.read() + + +@pytest.mark.parametrize('when', ['before', 'after']) +@pytest.mark.parametrize('change', ['uid', 'starttime']) +def test_observed_reused_identity_cannot_be_erased_by_later_absence(monkeypatch, when, change): + """A known replacement disappearing later cannot prove the original PID safe. + + The full second pass may finish before the post-check notices reuse. + Do not consult a later directory-gone proof after observing that change. + """ + expected = {'uid': os.getuid(), 'starttime': 42} + original = dict(expected, state='S') + replacement = dict(original) + replacement[change] += 1 + identities = iter([replacement, None] if when == 'before' + else [original, replacement]) + inspections, terminal_checks = [], [] + + def inspect(*args, **kwargs): + inspections.append(True) + if len(inspections) == 1: + error = worker.Refused('first metadata refusal') + error.inspection_failure = { + 'pid': 123, 'boundary': 'file descriptors', 'errno': errno.EACCES, + 'proof': {'reason': 'still_live', 'terminal': False, + 'expected': dict(expected)}, + } + raise error + return {'inspected_pids': [] if when == 'before' else [123]} + + def later_gone(*args, **kwargs): + terminal_checks.append(True) + return {'terminal': True, 'reason': 'proc_directory_gone', + 'expected': dict(expected)} + + monkeypatch.setattr(worker, 'inspect_placement', inspect) + monkeypatch.setattr(worker, 'process_identity', lambda *args: next(identities)) + monkeypatch.setattr(worker, 'terminal_proof_after_error', later_gone) + with pytest.raises(worker.Refused): + worker.inspect_placement_rechecked(None, None, '', '') + assert len(inspections) == (1 if when == 'before' else 2) + assert terminal_checks == [] From 6183affc35eda065466905bf1a3b54de46160921 Mon Sep 17 00:00:00 2001 From: Ethan Ligon Date: Thu, 29 Oct 2026 21:15:42 -0700 Subject: [PATCH 14/18] Keep incomplete pool accounting states reserved during reconciliation --- docs/dispatch-pool-ledger.org | 24 ++++++++++++++++++++++++ sucoder/dispatch_pool.py | 3 ++- tests/test_dispatch_pool.py | 27 +++++++++++++++++++++++++++ 3 files changed, 53 insertions(+), 1 deletion(-) diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index 0b0b1a9..d0f2fbe 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -219,3 +219,27 @@ persistence differ. Python3.6 grammar parses. GitNexus detect_changes was attempted and fails the same missing libssl3; manual AST scope check passes. These checks are not native lifecycle acceptance, deployment, scientific data verification or an explanation of the five original country failures. + +** Pool-status reconciliation review -- 2026-10-29 + +Reviewed integration head 5166d258fdaba5465534791dffad6239954bc9ba, which includes +main's bounded exit proof. Its 363 dispatch tests pass locally. A new accounting +control reproduced an IndexError when an exact step row had an empty or +whitespace-only State. This stopped reconciliation before a completed sibling +could release its reservation. Require a nonempty state before testing terminal +membership; incomplete observations retain their budget and evidence. + +Both new controls fail on the integration head and pass with the correction. +The pool/reinspection selection passes 61 tests. Manual AST comparison finds +only dispatch_pool.status changed in production; its caller is the pool-status +CLI. Receiving worker/asset helpers still parse as Python 3.6. GitNexus analyze +(with --skip-agents-md), query, upstream impact and compare detect-changes were +attempted but cannot load libssl.so.3. No shared tool installation was changed. + +PR 78 remains draft. Its recorded sequential source-only native canaries do not +establish long-running concurrent native-state preservation. Remaining release +gates are current-head long-running concurrent workers, interrupted-state +preservation, and scientific/source/API checks. The retained native failures +remain evidence of an unresolved acceptance boundary; this accounting fix +neither explains them nor changes live-process inspection policy. Allocation +creation/requested-node convenience remains outside this attach-only PR. diff --git a/sucoder/dispatch_pool.py b/sucoder/dispatch_pool.py index b39d7e7..0f23fb3 100644 --- a/sucoder/dispatch_pool.py +++ b/sucoder/dispatch_pool.py @@ -250,7 +250,8 @@ def status(directory, *, timeout=10): except (ValueError, OSError): continue row['scheduler'] = observation - if observation.get('available') and observation.get('state', '').split()[0].rstrip('+') in TERMINAL: + states = observation.get('state', '').split() + if observation.get('available') and states and states[0].rstrip('+') in TERMINAL: row.update(state='released', release_proof=observation, released_at=time.time()) atomic_json(str(path / 'reservations.json'), reservations) return dict(pool=profile, reservations=reservations, diff --git a/tests/test_dispatch_pool.py b/tests/test_dispatch_pool.py index bdba389..6b17b2d 100644 --- a/tests/test_dispatch_pool.py +++ b/tests/test_dispatch_pool.py @@ -175,6 +175,33 @@ def step_receipt(step='0'): allocation_start='2020-01-01T01:00:00', step_source=source) +@pytest.mark.parametrize('missing_state', ['', ' ']) +def test_incomplete_accounting_state_retains_budget_and_reconciles_siblings(attached, monkeypatch, missing_state): + rows = {name: dict(state='reserved', cpus=1, mem_mib=8192, directory=name) + for name in ('incomplete', 'completed')} + (attached / 'reservations.json').write_text(json.dumps(rows)) + monkeypatch.setattr(pool, '_worker_receipt', lambda directory, profile: + (None, None, step_receipt('0' if directory == 'incomplete' else '1'))) + + def run(argv, **kwargs): + if argv[0] == 'scontrol': + return SimpleNamespace(returncode=1, stdout='') + return SimpleNamespace(returncode=0, stdout=( + '234|%s|RUNNING|0:0|n1|2020-01-01T01:00:00\n' + '234.0||%s|0:0|n1|2020-01-01T01:01:00\n' + '234.1||COMPLETED|0:0|n1|2020-01-01T01:02:00\n' + ) % (os.getuid(), missing_state)) + + monkeypatch.setattr(dispatch, '_run', run) + result = pool.status(attached) + assert result['used_cpus'] == 1 and result['used_mem_mib'] == 8192 + assert result['reservations']['incomplete']['state'] == 'reserved' + assert 'release_proof' not in result['reservations']['incomplete'] + assert result['reservations']['completed']['state'] == 'released' + assert result['reservations']['completed']['release_proof']['target'] == '234.1' + assert json.loads((attached / 'reservations.json').read_text()) == result['reservations'] + + def test_step_scheduler_never_accepts_parent_or_sibling(monkeypatch): calls = [] def run(argv, **kwargs): From 606c40e3808335f77f5a614cca63f72216cf13ce Mon Sep 17 00:00:00 2001 From: Ethan Ligon Date: Thu, 29 Oct 2026 21:24:22 -0700 Subject: [PATCH 15/18] Handle processes disappearing during test liveness reads --- tests/test_observations.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/tests/test_observations.py b/tests/test_observations.py index a82634b..25d3136 100644 --- a/tests/test_observations.py +++ b/tests/test_observations.py @@ -368,10 +368,18 @@ def process_running(pid): # Zombies have exited and cannot perform work; the init process owns # reaping after a parent is terminated. return Path(f"/proc/{pid}/stat").read_text().split()[2] != "Z" - except FileNotFoundError: + except (FileNotFoundError, ProcessLookupError): return False +@pytest.mark.parametrize('error', [FileNotFoundError, ProcessLookupError]) +def test_process_running_handles_disappearance_during_proc_read(monkeypatch, error): + def disappeared(path): + raise error() + monkeypatch.setattr(Path, 'read_text', disappeared) + assert not process_running(123) + + def test_capture_cleans_owned_descendants(tmp_path): pidfile = tmp_path / "pid" code = ("import subprocess,sys,time; from pathlib import Path; " From 78f43ad1bceb1eb256a740d8f813a1fbd94184a0 Mon Sep 17 00:00:00 2001 From: Ethan Ligon Date: Thu, 29 Oct 2026 22:23:08 -0700 Subject: [PATCH 16/18] Allow bounded workspace publication budgets for slow shared storage --- README.org | 4 ++++ config.example.yaml | 4 ++++ docs/dispatch-pool-ledger.org | 31 +++++++++++++++++++++++++++++++ docs/dispatch-pools.org | 8 ++++++-- docs/dispatch.org | 6 ++++++ sucoder/dispatch.py | 22 ++++++++++++++-------- sucoder/dispatch_cli.py | 2 ++ sucoder/dispatch_pool.py | 10 +++++++--- sucoder/dispatch_pool_cli.py | 4 +++- sucoder/dispatch_worker.py | 14 ++++++++++---- tests/test_dispatch.py | 19 +++++++++++++++++++ tests/test_dispatch_pool.py | 14 ++++++++++++++ tests/test_dispatch_worker.py | 27 +++++++++++++++++++++++++++ 13 files changed, 147 insertions(+), 18 deletions(-) diff --git a/README.org b/README.org index 3638bb7..7298072 100644 --- a/README.org +++ b/README.org @@ -271,6 +271,10 @@ before integration. =sucoder dispatch pool= adds explicitly budgeted, step-owned workers inside an already held allocation. See [[file:docs/dispatch-pools.org][existing-allocation pools]] for isolation, immutable assets, conservative reservations and stop semantics. +Both submit commands accept =--publication-timeout= (seconds, default 30, +maximum 300) for each Git/WIP publication. Larger values extend the early +shutdown window to at least that budget plus 60 seconds; managed-state +checkpoint and process-cleanup timeouts remain separate. See [[file:docs/dispatch.org][headless dispatch]] for the complete submission, status, wait and result commands and the supported state guarantees. diff --git a/config.example.yaml b/config.example.yaml index 55319cc..8ee9f82 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -10,6 +10,10 @@ system_prompt: /home//.sucoder/system_prompt.org # /shared/private/project.json` or `dispatch verify-stage --project-profile ...`. # Schema, LSMS image/reader examples and deterministic burst adaptation: # docs/dispatch-projects.org (examples/lsms/make_profile.py builds a profile). +# Workspace publication is a dispatch CLI setting, not a YAML key: +# `dispatch submit --publication-timeout 120` (also `dispatch pool submit`). +# Seconds per Git/WIP publication: default 30, positive, maximum 300. +# Early shutdown reserves max(120, ceil(publication_timeout) + 60) seconds. # Agent account only (optional): reach live cluster sessions through the # human's warm SSH for `sessions`, `message` and `peek`, via diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index d0f2fbe..a14a196 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -243,3 +243,34 @@ preservation, and scientific/source/API checks. The retained native failures remain evidence of an unresolved acceptance boundary; this accounting fix neither explains them nor changes live-process inspection policy. Allocation creation/requested-node convenience remains outside this attach-only PR. + +** Measured publication budget -- 2026-10-29 + +At 606c40e, fresh allocation 40629482/n0043 ran source-only LSMS workers through +the pool CLI. Step 0 completed 58 synthetic scientific/API tests and 660 source +churn cycles, exited native zero and sealed state, but final Git/WIP publication +exceeded its 30-second allowance (supervisor 74). A bounded trace in that stopped +worker's private checkout measured one Git push at 31.92 seconds, including +17.97 seconds in receive-side connectivity traversal. Its original failed result +is retained; the later diagnostic publication does not repair that result. +Step 1 sealed state too but its workload stopped at a separate 30-second Git +commit deadline before scientific tests; that is a harness limit, not native +acceptance. Both original transactions remain under +=/global/scratch/fsa/fc_jevons/ligon/burst/sucoder-pr78-acceptance-dyb0orpw=. + +Add =--publication-timeout= to both submit CLIs: positive seconds, default 30, +maximum 300, frozen in the manifest and validated on the receiver. Periodic +and final Git/WIP publication use it; native checkpoints and cleanup do not. +Early shutdown reserves =max(120, ceil(publication_timeout) + 60)= seconds, +including the ordinary-job scheduler notice and pool deadline checks. Requests +too short for that window refuse. Legacy manifests retain checkpoint_timeout +as their publication budget. This is a finite budget, not a guarantee that all +shared-storage operations complete. The follow-on harness separately allows +120 seconds for its documentation commit and still performs no retry. + +382 dispatch tests pass, including invalid-budget refusal, immutable budget, +extended scheduler/step window, and both periodic/final worker propagation. +Python 3.6 worker grammar passes. GitNexus impact/detect-changes again failed +on libssl.so.3; manual review covers submit/resource validation, both CLI callers, +receiver validation and the worker's two publication calls and deadline checks. +No original worker, result or native state was edited for this correction. diff --git a/docs/dispatch-pools.org b/docs/dispatch-pools.org index 82415ae..8e039a5 100644 --- a/docs/dispatch-pools.org +++ b/docs/dispatch-pools.org @@ -76,8 +76,12 @@ manual reconciliation is required rather than assuming no step exists. The detached srun frontend remains on the submitting host; loss of that host is not claimed to provide portable frontend recovery or automatic resubmission. -The worker begins shutdown 120 seconds before the earlier requested-duration or -pool worker deadline. Slurm's time request is an additional hard bound. +The worker begins shutdown at least 120 seconds before the earlier requested-duration or +pool worker deadline. =--publication-timeout= bounds each periodic/final Git/WIP +publication (default 30 seconds, positive, maximum 300). The early shutdown +window is =max(120, ceil(publication_timeout) + 60)= seconds, and a shorter +requested duration is refused. This does not increase managed-state or process +cleanup timeouts. Slurm's time request is an additional hard bound. Publication receipts distinguish complete, incomplete and unverified outcomes; no CLI exit alone proves scientific or infrastructure acceptance. diff --git a/docs/dispatch.org b/docs/dispatch.org index 661ae6b..84149bc 100644 --- a/docs/dispatch.org +++ b/docs/dispatch.org @@ -108,6 +108,12 @@ Use repeatable =--require-evidence junit.xml= options to make missing proof file Git/WIP snapshots run every =--snapshot-minutes= (default ten minutes) and on orderly exit or TERM cleanup. Snapshot refs are append-only, and the result/status records the latest observed publication and time. +=--publication-timeout= sets the budget for each Git/WIP publication (default +30 seconds, positive, maximum 300). The scheduler's early TERM notice reserves +=max(120, ceil(publication_timeout) + 60)= seconds; requests shorter than this +window are refused. Managed-state checkpoint and process-cleanup budgets are +unchanged. Use a larger publication budget when shared-storage Git operations +take longer than 30 seconds; failure still reports incomplete publication. SIGKILL cannot trigger a final snapshot; only the already published refs and evidence are durable. Codex's own final checkpoint remains inside the existing runner lifecycle and can precede the supervisor's final Git snapshot. No result claims that an unpublished interval was preserved. diff --git a/sucoder/dispatch.py b/sucoder/dispatch.py index 96dccd7..a567428 100644 --- a/sucoder/dispatch.py +++ b/sucoder/dispatch.py @@ -123,7 +123,7 @@ def _read_json(path: Path): return None, "unreadable" -def _resources(values): +def _resources(values, shutdown_seconds=120): required = {"account", "partition", "qos", "cpus", "mem", "time"} if not required.issubset(values) or set(values) - required - {"exclude"}: raise ValueError("resources require account, partition, qos, cpus, mem and time; optional exclude") @@ -153,26 +153,31 @@ def _resources(values): seconds = fields[0] * 60 + fields[1] else: seconds = fields[0] * 3600 + fields[1] * 60 + fields[2] - if seconds < 120: - raise ValueError("dispatch time must allow the 120-second shutdown signal window") + if seconds < shutdown_seconds: + raise ValueError("dispatch time must allow the %s-second shutdown signal window" % shutdown_seconds) return values def submit(repo, ref, task_file, agent, executable, resources, *, model=None, directory_root=None, local_disk_root="/local", source_agent_home=None, source_claude_user_config=None, required_evidence=None, snapshot_minutes=10, - dry_run=False, timeout=30, project_profile=None, _execution=None): + dry_run=False, timeout=30, project_profile=None, _execution=None, + publication_timeout=30): """Pin an offline commit, stage a private worker and submit exactly once. Dry runs validate and report the planned command without writing artifacts or contacting Slurm. They do not reserve their reported random dispatch ID. """ timeout = _finite_seconds(timeout, "timeout") + publication_timeout = _finite_seconds(publication_timeout, "publication timeout") + if publication_timeout > 300: + raise ValueError("publication timeout must not exceed 300 seconds") + shutdown_seconds = max(120, math.ceil(publication_timeout) + 60) if type(snapshot_minutes) not in (int, float) or not math.isfinite(snapshot_minutes) or snapshot_minutes <= 0: raise ValueError("snapshot_minutes must be finite and positive") if agent not in ("codex", "claude"): raise ValueError("headless agent must be codex or claude") - resources = _resources(resources) + resources = _resources(resources, shutdown_seconds) project = read_profile(project_profile, shared_path=_shared_path) if project_profile is not None else None if project is not None and any(value > resources["cpus"] for value in project["profile"]["thread_limits"].values()): raise ValueError("project thread limit exceeds requested CPU budget") @@ -232,10 +237,11 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, worker_path=str(directory / "worker.py"), environment_file=str(directory / "environment.json"), source_agent_home=None, source_claude_user_config=None, resources=resources, scope=scope_id(cluster or "unknown", str(os.getuid()), str(repo), dispatch_id), - checkpoint_minutes=10, checkpoint_timeout=30, snapshot_minutes=float(snapshot_minutes)) + checkpoint_minutes=10, checkpoint_timeout=30, snapshot_minutes=float(snapshot_minutes), + publication_timeout=publication_timeout) if _execution is not None: from .dispatch_pool import validate_execution - validate_execution(_execution, resources) + validate_execution(_execution, resources, shutdown_seconds) manifest["execution"] = _execution manifest["cluster"] = cluster = _execution["cluster"] source_home = source_agent_home if source_agent_home is not None else Path.home() / (".codex" if agent == "codex" else ".claude") @@ -249,7 +255,7 @@ def submit(repo, ref, task_file, agent, executable, resources, *, model=None, raise ValueError("Claude user config must be an existing file") if agent == "claude" and claude_config.exists(): manifest["source_claude_user_config"] = str(_shared_path(claude_config, "Claude user config")) - argv = ["sbatch", "--parsable", "--no-requeue", "--signal=B:TERM@120", "--export=NONE", "--chdir=/tmp", "--nodes=1", "--ntasks=1", + argv = ["sbatch", "--parsable", "--no-requeue", "--signal=B:TERM@" + str(shutdown_seconds), "--export=NONE", "--chdir=/tmp", "--nodes=1", "--ntasks=1", "--account=" + resources["account"], "--partition=" + resources["partition"], "--cpus-per-task=" + str(resources["cpus"]), "--mem=" + resources["mem"], "--time=" + resources["time"], diff --git a/sucoder/dispatch_cli.py b/sucoder/dispatch_cli.py index 1d79784..48f68ec 100644 --- a/sucoder/dispatch_cli.py +++ b/sucoder/dispatch_cli.py @@ -64,6 +64,7 @@ def submit( claude_user_config: Optional[Path] = typer.Option(None, "--claude-user-config"), require_evidence: List[str] = typer.Option(None, "--require-evidence", help="Required evidence filename, relative to the private evidence directory; repeatable."), snapshot_minutes: float = typer.Option(10, "--snapshot-minutes", min=0.1, help="Periodic Git/WIP publication cadence; final publication also runs on orderly exit."), + publication_timeout: float = typer.Option(30, "--publication-timeout", help="Seconds per workspace publication (positive, at most 300); extends the early shutdown window."), project_profile: Optional[Path] = typer.Option(None, "--project-profile", help="Explicit project JSON profile; bootstrap and configuration are frozen before submission."), local_disk_root: str = typer.Option("/local", "--local-disk-root"), timeout: float = typer.Option(30, "--timeout", min=0.1, help="Submission subprocess deadline in seconds."), @@ -80,6 +81,7 @@ def submit( source_agent_home=agent_home, source_claude_user_config=claude_user_config, required_evidence=require_evidence or [], snapshot_minutes=snapshot_minutes, + publication_timeout=publication_timeout, project_profile=project_profile, timeout=timeout, dry_run=dry_run, ) diff --git a/sucoder/dispatch_pool.py b/sucoder/dispatch_pool.py index 0f23fb3..042ae91 100644 --- a/sucoder/dispatch_pool.py +++ b/sucoder/dispatch_pool.py @@ -111,14 +111,14 @@ def _locked(path): lock.rmdir() -def validate_execution(value, resources): +def validate_execution(value, resources, shutdown_seconds=120): if (value.get('scope') != 'step' or not re.fullmatch(r'[0-9a-f]{32}', str(value.get('dispatch_id', ''))) or not re.fullmatch(r'[0-9a-f]{32}', str(value.get('pool_id', ''))) or type(value.get('job_id')) is not int or value['job_id'] < 1 or not re.fullmatch(r'[A-Za-z0-9_.-]+', str(value.get('node', ''))) or not re.fullmatch(r'[A-Za-z0-9_.-]+', str(value.get('cluster', ''))) or type(value.get('deadline')) not in (int, float) or not math.isfinite(value['deadline']) - or value['deadline'] <= time.time() + 120): + or value['deadline'] <= time.time() + shutdown_seconds): raise ValueError('invalid or expired pool execution binding') if resources.get('exclude'): raise ValueError('pool workers use the bound node, not exclusion routing') @@ -127,6 +127,10 @@ def validate_execution(value, resources): def submit(directory, repo, ref, task_file, agent, executable, *, cpus=1, mem='8G', time_limit='00:30:00', timeout=30, **options): path, profile = _load(directory) + publication_timeout = control._finite_seconds(options.get('publication_timeout', 30), 'publication timeout') + if publication_timeout > 300: + raise ValueError('publication timeout must not exceed 300 seconds') + shutdown_seconds = max(120, math.ceil(publication_timeout) + 60) if os.environ.get('SLURM_CLUSTER_NAME') != profile['cluster']: raise ValueError('pool submission requires its original current Slurm cluster') if options.get('dry_run'): @@ -137,7 +141,7 @@ def submit(directory, repo, ref, task_file, agent, executable, *, cpus=1, mem='8 or facts['StartTime'] != profile['allocation_evidence']['StartTime']): raise ValueError('allocation identity/budget changed since attach') resources = control._resources(dict(account=facts['Account'], partition=facts['Partition'], - qos=facts.get('QOS'), cpus=cpus, mem=mem, time=time_limit)) + qos=facts.get('QOS'), cpus=cpus, mem=mem, time=time_limit), shutdown_seconds) # Slurm time strings are normalized independently of the scheduler. Worker # deadline is an additional hard cap, not a fresh full duration per retry. days, clock = time_limit.split('-', 1) if '-' in time_limit else ('0', time_limit) diff --git a/sucoder/dispatch_pool_cli.py b/sucoder/dispatch_pool_cli.py index e5e4ddb..267cf42 100644 --- a/sucoder/dispatch_pool_cli.py +++ b/sucoder/dispatch_pool_cli.py @@ -49,6 +49,7 @@ def submit(ctx: typer.Context, pool: Path, repository: Path, mem: str = typer.Option('8G', '--mem'), time: str = typer.Option('00:30:00', '--time'), require_evidence: List[str] = typer.Option(None, '--require-evidence'), snapshot_minutes: float = typer.Option(10, '--snapshot-minutes', min=0.1), + publication_timeout: float = typer.Option(30, '--publication-timeout', help='Seconds per workspace publication (positive, at most 300).'), timeout: float = typer.Option(30, '--timeout')): """Reserve budget and start exactly one existing-allocation step. @@ -60,7 +61,8 @@ def submit(ctx: typer.Context, pool: Path, repository: Path, cpus=cpus, mem=mem, time_limit=time, timeout=timeout, directory_root=ctx.obj['root'], source_agent_home=agent_home, project_profile=project_profile, model=model, - required_evidence=require_evidence or [], snapshot_minutes=snapshot_minutes) + required_evidence=require_evidence or [], snapshot_minutes=snapshot_minutes, + publication_timeout=publication_timeout) except (ValueError, OSError, subprocess.SubprocessError) as exc: _error(exc) _emit(value) diff --git a/sucoder/dispatch_worker.py b/sucoder/dispatch_worker.py index 1dedf93..ce798bf 100644 --- a/sucoder/dispatch_worker.py +++ b/sucoder/dispatch_worker.py @@ -242,6 +242,10 @@ def load_manifest(path): if type(manifest.get(key)) is not int or manifest[key] <= 0: raise Refused("invalid " + key) cadence = manifest.get("snapshot_minutes", manifest["checkpoint_minutes"]) + publication_timeout = manifest.get("publication_timeout", manifest["checkpoint_timeout"]) + if (type(publication_timeout) not in (int, float) or not math.isfinite(publication_timeout) + or not 0 < publication_timeout <= 300): + raise Refused("invalid publication timeout") if type(cadence) not in (int, float) or not math.isfinite(cadence) or cadence <= 0: raise Refused("invalid workspace snapshot cadence") digest_path = os.path.join(directory, "digests.json") @@ -1017,6 +1021,8 @@ def native_state_files(home): def run(manifest_path, inherited=None, mountinfo_path="/proc/self/mountinfo", ownership_factory=None): os.umask(0o077) manifest = load_manifest(manifest_path) + publication_timeout = manifest.get("publication_timeout", manifest["checkpoint_timeout"]) + shutdown_seconds = max(120, math.ceil(publication_timeout) + 60) directory = manifest["directory"] record = {"schema_version": SCHEMA_VERSION, "dispatch_id": manifest["dispatch_id"], "uid": os.getuid(), "agent": manifest["agent"], "commit": manifest["commit"], "branch": manifest["branch"], @@ -1055,7 +1061,7 @@ def final_publication(): publish("final_publication", force=True) try: snapshot = json.loads(run_script(manifest["snapshot_script"], environment, - manifest["checkpoint_timeout"], heartbeat=publish)) + publication_timeout, heartbeat=publish)) record["workspace_snapshot"] = snapshot head = subprocess.check_output(["git", "-C", work, "rev-parse", "HEAD"], env=environment, stderr=subprocess.DEVNULL, @@ -1143,7 +1149,7 @@ def final_publication(): "names that interpreter. Do not replace the runtime or mutate shared caches.\n" % directory).encode("utf-8") runner = load_runner(manifest["runner_path"]) command = command_for(manifest, environment) - if manifest.get("execution") is not None and time.time() >= manifest["execution"]["deadline"] - 120: + if manifest.get("execution") is not None and time.time() >= manifest["execution"]["deadline"] - shutdown_seconds: raise Refused("pool worker reached its publication cutoff before native launch") if stop_signal[0] is not None: raise Refused("cancelled before native version inspection") @@ -1164,7 +1170,7 @@ def final_publication(): observed, databases = set(), set() deadline = time.monotonic() + manifest.get("snapshot_minutes", manifest["checkpoint_minutes"]) * 60 while child.poll() is None: - if manifest.get("execution") is not None and time.time() >= manifest["execution"]["deadline"] - 120: + if manifest.get("execution") is not None and time.time() >= manifest["execution"]["deadline"] - shutdown_seconds: stop_signal[0] = signal.SIGTERM evidence = inspect_placement_rechecked(child, runner, root, environment["HOME"], expected_environment=environment, ownership=ownership, manifest=manifest) @@ -1196,7 +1202,7 @@ def final_publication(): if time.monotonic() >= deadline: try: record["workspace_snapshot"] = json.loads(run_script(manifest["snapshot_script"], environment, - manifest["checkpoint_timeout"], stop_signal, + publication_timeout, stop_signal, heartbeat=publish)) except Exception as exc: record["workspace_snapshot"] = {"state": "failed", "reason": str(exc)} diff --git a/tests/test_dispatch.py b/tests/test_dispatch.py index b02dede..8acd555 100644 --- a/tests/test_dispatch.py +++ b/tests/test_dispatch.py @@ -71,6 +71,25 @@ def test_dry_run_has_no_mutation_or_scheduler(source, monkeypatch): assert "--comment=sucoder-dispatch:" + report["dispatch_id"] in report["command"] +@pytest.mark.parametrize("value", [0, -1, True, float("inf"), float("nan"), 301]) +def test_publication_budget_refuses_before_staging(source, value): + with pytest.raises(ValueError, match="publication timeout"): + dispatch.submit(**source, publication_timeout=value) + assert not dispatch.DEFAULT_DIRECTORY_ROOT.exists() + + +def test_publication_budget_is_frozen_and_extends_scheduler_notice(source, monkeypatch): + scheduler(monkeypatch) + report = dispatch.submit(**source, publication_timeout=120.5) + manifest = json.loads((Path(report['directory']) / 'manifest.json').read_text()) + assert manifest['publication_timeout'] == 120.5 + assert manifest['checkpoint_timeout'] == 30 + assert '--signal=B:TERM@181' in report['command'] + source['resources']['time'] = '00:02:00' + with pytest.raises(ValueError, match='181-second'): + dispatch.submit(**source, publication_timeout=120.5) + + @pytest.mark.parametrize("timeout", [float("nan"), float("inf"), -float("inf"), True, "5", None, 0, -1]) def test_submit_rejects_invalid_timeout_before_source_or_scheduler(source, monkeypatch, timeout): def forbidden(*args, **kwargs): diff --git a/tests/test_dispatch_pool.py b/tests/test_dispatch_pool.py index 6b17b2d..b168b72 100644 --- a/tests/test_dispatch_pool.py +++ b/tests/test_dispatch_pool.py @@ -37,6 +37,20 @@ def request(source): return {key: source[key] for key in ('repo', 'ref', 'task_file', 'agent', 'executable')} +def test_publication_budget_refuses_short_pool_duration_before_reservation(attached, source): + with pytest.raises(ValueError, match='180-second'): + pool.submit(attached, **request(source), publication_timeout=120, time_limit='00:02:00') + assert json.loads((attached / 'reservations.json').read_text()) == {} + + +def test_execution_requires_the_extended_publication_window(): + execution = dict(scope='step', dispatch_id='a' * 32, pool_id='b' * 32, + job_id=234, node='n1', cluster='cluster', deadline=time.time() + 150) + pool.validate_execution(execution, {}) + with pytest.raises(ValueError, match='expired'): + pool.validate_execution(execution, {}, shutdown_seconds=180) + + def test_attach_rejects_wrong_uid_or_multi_node_scheduler(monkeypatch): monkeypatch.setattr(dispatch, '_run', lambda *a, **kw: SimpleNamespace(returncode=0, stdout=( 'JobId=234 UserId=x(%d) JobState=RUNNING NodeList=n1 NumNodes=2 AllocTRES=cpu=8,mem=48G EndTime=2099-01-01T00:00:00' % os.getuid()))) diff --git a/tests/test_dispatch_worker.py b/tests/test_dispatch_worker.py index 5d69263..52ca476 100644 --- a/tests/test_dispatch_worker.py +++ b/tests/test_dispatch_worker.py @@ -685,6 +685,33 @@ def test_durable_copy_exception_is_only_for_the_managed_supervisor(staged): assert report["misplaced"] == [path] +@pytest.mark.parametrize('budget', [None, 120]) +def test_workspace_budget_reaches_periodic_and_final_publication(staged, monkeypatch, budget): + staged.manifest['snapshot_minutes'] = .001 + if budget is not None: + staged.manifest['publication_timeout'] = budget + save(staged) + original = worker.run_script + calls = [] + def observed(path, environment, timeout, *args, **kwargs): + if path == staged.manifest['snapshot_script']: + calls.append((timeout, bool(args))) + return original(path, environment, timeout, *args, **kwargs) + monkeypatch.setattr(worker, 'run_script', observed) + assert fake_run(staged, {'SLURM_JOB_ID': '234', 'PATH': os.environ['PATH']}) == 0 + assert any(periodic for _, periodic in calls) and any(not periodic for _, periodic in calls) + assert all(value == (budget if budget is not None else 2) for value, _ in calls) + assert staged.manifest['checkpoint_timeout'] == 2 + + +@pytest.mark.parametrize('budget', [0, -1, True, float('nan'), float('inf'), 301]) +def test_receiving_manifest_refuses_invalid_publication_budget(staged, budget): + staged.manifest['publication_timeout'] = budget + save(staged) + with pytest.raises(worker.Refused, match='publication timeout'): + worker.load_manifest(str(staged.control / 'manifest.json')) + + def test_heartbeat_tracks_phases_without_fsync_every_descriptor_poll(staged, monkeypatch): writes = [] original = worker.atomic_json From 23f74ae3a00ffaed2de223db7d7f48fd1f248b10 Mon Sep 17 00:00:00 2001 From: Ethan Ligon Date: Thu, 29 Oct 2026 23:00:22 -0700 Subject: [PATCH 17/18] Preserve launched pool receipts when summary bookkeeping is busy --- docs/dispatch-pool-ledger.org | 17 +++++++++++++++++ sucoder/dispatch_pool.py | 21 ++++++++++++++++----- tests/test_dispatch_pool.py | 21 +++++++++++++++++++++ 3 files changed, 54 insertions(+), 5 deletions(-) diff --git a/docs/dispatch-pool-ledger.org b/docs/dispatch-pool-ledger.org index a14a196..a1d3c1f 100644 --- a/docs/dispatch-pool-ledger.org +++ b/docs/dispatch-pool-ledger.org @@ -274,3 +274,20 @@ Python 3.6 worker grammar passes. GitNexus impact/detect-changes again failed on libssl.so.3; manual review covers submit/resource validation, both CLI callers, receiver validation and the worker's two publication calls and deadline checks. No original worker, result or native state was edited for this correction. + +** Late reservation-summary contention -- 2026-10-29 + +The follow-on concurrent trial on allocation 40631087 exposed a distinct +controller race: both workers launched with durable submission receipts, but +one CLI reported a reservation-lock error while writing its duplicate summary. +Its original reservation already bound the dispatch directory; pool status +reconciled that receipt without resubmission. Preserve the successful submission +report if only that final summary lock is busy, with an explicit deferred-update +annotation. Initial reservation contention still refuses, no lock is stolen, +and no budget is released. Other metadata errors still propagate. + +The 40 pool tests pass, including the new late-lock control and the existing +prelaunch lock refusal. Production scope is the lock exception type and the +post-submission summary update; receiving helpers are byte-identical to 78f43ad, +the native-trial pin. GitNexus impact/detect-changes remain unavailable; manual +callers are pool submit/status and their existing ValueError-handling CLIs. diff --git a/sucoder/dispatch_pool.py b/sucoder/dispatch_pool.py index 042ae91..5e25aa4 100644 --- a/sucoder/dispatch_pool.py +++ b/sucoder/dispatch_pool.py @@ -17,6 +17,10 @@ 'OUT_OF_MEMORY', 'PREEMPTED', 'BOOT_FAIL', 'DEADLINE', 'REVOKED')) +class ReservationLockHeld(ValueError): + """A cooperating operation currently owns the pool reservation lock.""" + + def memory_mib(value): match = re.fullmatch(r'([1-9][0-9]*)([KMGT]?)', str(value), re.I) if not match: @@ -104,7 +108,7 @@ def _locked(path): try: lock.mkdir(mode=0o700) except FileExistsError: - raise ValueError('pool reservation lock held; no automatic stale lock stealing') + raise ReservationLockHeld('pool reservation lock held; no automatic stale lock stealing') try: yield finally: @@ -175,10 +179,17 @@ def submit(directory, repo, ref, task_file, agent, executable, *, cpus=1, mem='8 # There may already be a staged or live srun after an interrupted call. # Retain budget; only exact scheduler terminal proof releases it. raise - with _locked(path): - reservations = json.loads((path / 'reservations.json').read_text()) - reservations[dispatch_id].update(directory=report['directory'], submission=report) - atomic_json(str(path / 'reservations.json'), reservations) + try: + with _locked(path): + reservations = json.loads((path / 'reservations.json').read_text()) + reservations[dispatch_id].update(directory=report['directory'], submission=report) + atomic_json(str(path / 'reservations.json'), reservations) + except ReservationLockHeld: + # The reservation already binds its directory, and control.submit has + # persisted the authoritative submission receipt there. A duplicate + # summary is optional: never disguise a launched step as a failed + # submission, steal the other operation's lock, or launch again. + report = dict(report, reservation_update='deferred; reconcile with pool status') return report diff --git a/tests/test_dispatch_pool.py b/tests/test_dispatch_pool.py index b168b72..7d156a1 100644 --- a/tests/test_dispatch_pool.py +++ b/tests/test_dispatch_pool.py @@ -103,6 +103,27 @@ def test_lock_is_atomic_and_not_stolen(attached, source): pool.submit(attached, **request(source)) +def test_late_summary_lock_conflict_preserves_submission_and_budget(attached, source, monkeypatch): + launches = [] + def launched(directory, manifest, argv, report, timeout): + launches.append(manifest['dispatch_id']) + report.update(state='submitted', job_id=234, step_id='4') + (directory / 'submission.json').write_text(json.dumps(report)) + (attached / '.reservation-lock').mkdir() + return report + monkeypatch.setattr(pool, 'launch_step', launched) + report = pool.submit(attached, **request(source)) + assert report['state'] == 'submitted' and report['step_id'] == '4' + assert report['reservation_update'].startswith('deferred') + rows = json.loads((attached / 'reservations.json').read_text()) + assert list(rows) == launches == [report['dispatch_id']] + assert rows[report['dispatch_id']]['state'] == 'reserved' + assert rows[report['dispatch_id']]['directory'] == report['directory'] + assert (attached / '.reservation-lock').is_dir() + receipt = json.loads((Path(report['directory']) / 'submission.json').read_text()) + assert receipt['state'] == 'submitted' and receipt['step_id'] == '4' + + def test_staged_pool_command_has_exact_step_and_private_publication(attached, source, monkeypatch): seen = {} def launch(directory, manifest, argv, report, timeout): From 839cbe745c519ef6f882ec7f250d342dab918f75 Mon Sep 17 00:00:00 2001 From: Ethan Ligon Date: Thu, 29 Oct 2026 23:26:49 -0700 Subject: [PATCH 18/18] Record concurrent pool interruption and replacement acceptance --- docs/dispatch-pool-acceptance.org | 118 ++++++++++++++++++++++++++++++ docs/dispatch-pools.org | 2 + 2 files changed, 120 insertions(+) create mode 100644 docs/dispatch-pool-acceptance.org diff --git a/docs/dispatch-pool-acceptance.org b/docs/dispatch-pool-acceptance.org new file mode 100644 index 0000000..a7be43e --- /dev/null +++ b/docs/dispatch-pool-acceptance.org @@ -0,0 +1,118 @@ +#+title: PR 78: bounded pool acceptance on Savio +#+date: 2026-10-29 + +* Result and scope + +The attach-only pool implementation passed a concurrent native trial with an +intentional worker interruption and a later replacement in the same allocation. +Native trial pin: =78f43ad1bceb1eb256a740d8f813a1fbd94184a0=. +The subsequent controller-only correction =23f74ae3a00ffaed2de223db7d7f48fd1f248b10= +preserves an already launched submission when its optional reservation-summary +update encounters a busy lock. Receiving helpers, workspace scripts, manifest +generation and CLI arguments are byte-identical between those pins. Its 40 pool +tests and Python 3.9/3.11/docs CI pass; the preceding full dispatch cohort passed +382 tests. GitNexus remains unavailable because its native library cannot load +libssl.so.3; manual caller and source comparisons are recorded in the ledger. + +This accepts bounded pool lifecycle, synthetic scientific/API execution and +durable interrupted partial state. It does not establish credentialed full-country +or full-corpus scientific correctness, eight-worker capacity, automatic recovery, +or arbitrary-duration reliability. Allocation creation/requested-node convenience +remains outside the PR, and issue 77 remains open for broader follow-up. + +* Native experiment + +Fresh allocation =40631087= on =n0041.savio4= held four CPUs/32 GiB for at most +one hour. The explicit pool reserved two CPUs/16 GiB; each worker received one +CPU/8 GiB. Immutable source and the approved image were staged once in that +allocation. Each worker had a private checkout, HOME, native scope, runtime, +configuration/data/cache directories and actual local SQLite/WAL/SHM descriptors. + +Source pin: =d65e8b13683409ab0005f1a5590208cd7d94ae62=. +Image SHA256: =12c7eab8ae37da2fd0adee71d4f741ea81c4c1937fc53644ced43d282a21f86e=. +Each native worker ran the same 58 synthetic tests from +=tests/test_grain_collapse_cluster_features.py= and +=tests/test_gh871_most_complete_row.py=, with zero failures, errors or skips. +No survey data or reader credentials were supplied. Each created one private +documentation commit; verification proved that scientific source was unchanged. + +| Role | Dispatch | Step | Native/supervisor | Workload cycles | Generation | +|------+----------+------+-------------------+-----------------+------------| +| Survivor | 1afff5010f5041ff9f7ad0891412b1e7 | 0 | 0 / 0 | 660 completed | sealed | +| Interrupted | d78622ebe2874d1ea73d11a50f7d3289 | 1 | 143 / 143 | 61 retained | crash | +| Replacement | d16201f5e0dc479da13cd772256cd469 | 2 | 0 / 0 | 180 completed | sealed | + +Each cycle read five pinned source files through Git, checked tracked status, +and paused one second. The survivor's complete program took 736 seconds. +Once both initial workers had completed at least 61 cycles, the supported +=pool stop-worker= command signalled only =40631087.1=. The survivor continued +from cycle 61 to 660. Only exact terminal step evidence released the interrupted +reservation; the replacement then ran concurrently with the survivor. Concurrent +CPU affinities were disjoint, and the allocation keeper and coordinator remained +alive. Three metadata races in the survivor recovered through terminal proof; +no full-reinspection retry was observed in this trial. + +* Durable verification + +All three workers completed workspace and evidence publication. Separate +post-exit verification steps required exact scheduler terminal evidence and +absence of live members of the original worker cgroup before hashing native +payloads. For each worker, six database files matched the generation's full +SHA256/size records. Each recorded rollout prefix matched its authenticated +hash, with zero trailing bytes. No SQLite API or rollout deserialization was used. +Evidence inventory, file hashes, JUnit counts, source hashes, the single-document +commit, runtime identities and frozen receiving helpers were checked separately. + +All workers reused the same image/source inode identities and asset-verification +receipt, while runtime roots and native identities were distinct. All pool +reservations were released from exact step/accounting proof. Both test allocations +were released after verification; coordinator allocation =40618485= remained running. + +The deliberately interrupted native exited 143. The existing runner therefore +produced a =crash= generation, not a =sealed= one. Its retained partial state is +hash-verified, not a completed task or an unattended continuation authorization. +The runner's existing refusal of unattended crash-generation restoration remains +in force. Semantic resume of this partial conversation was not attempted. + +* Original failures and corrections + +Earlier allocation =40629482= is retained as a failed trial. One worker completed +the same tests and 660 cycles and sealed native state, but its final workspace +publication exceeded 30 seconds. A diagnostic run in the stopped private checkout +measured one Git push at 31.92 seconds. The other worker stopped at the harness's +30-second documentation-commit deadline. Neither original result was relabelled, +retried or resumed. A follow-on request was refused before reservation/launch +because that allocation no longer had enough lifetime; its empty reservation +record is preserved. + +The new =--publication-timeout= gives each Git/WIP publication a bounded budget +(default 30, maximum 300 seconds) and expands the shutdown window accordingly. +The accepted trial used 120 seconds. The harness separately allowed 120 seconds +for its documentation commit. Managed-state/cleanup timeouts and inspection +policy were not weakened. A late reservation-summary lock conflict in the accepted +trial was reconciled through the already durable receipt, without duplicate +submission; the subsequent controller correction makes that successful launch +explicit in the CLI result. + +Earlier country-worker inspection failures recorded in the PR and ledger remain +historical failures. This acceptance does not retroactively establish their +scientific output, native preservation, or exit-interval duration. + +* Evidence + +Durable root: +=/global/scratch/fsa/fc_jevons/ligon/burst/sucoder-pr78-acceptance-dyb0orpw/=. + +The accepted trial is in =publication-v3/=, including =acceptance-summary.json=, +the three =*-acceptance.json= files, =frozen-source-check.json=, =before-stop.json=, +=stop-receipt.json=, =pool-status.json=, =observations.jsonl=, frozen workload/profile, +all submission intents and receipts, and private =records//= artifacts. +The original failures, stopped-checkout trace and refused preparation remain in +the parent and =publication-v2/= directories. These are ordinary evidence paths; +native stores are retained privately within their original dispatch scopes. + +The earlier local full-suite run had 2443 passes, one skip and five failures. +All five reproduced in a separate unchanged =main= checkout at =10c3c7f=: +the in-process drain-state test and four observation tests. This local suite was +not green. CI on the implemented publication and bookkeeping changes passed +Python 3.9, Python 3.11 and docs (runs 41008166831 and 41011577741). diff --git a/docs/dispatch-pools.org b/docs/dispatch-pools.org index 8e039a5..267159e 100644 --- a/docs/dispatch-pools.org +++ b/docs/dispatch-pools.org @@ -1,6 +1,8 @@ #+title: Workers in an existing Slurm allocation * Scope +The bounded native acceptance and retained limitations are recorded in +[[file:dispatch-pool-acceptance.org][PR 78 pool acceptance]]. =dispatch pool= attaches to an already held, owned, running single-node allocation. It does not create or release allocations, install tools, or start raw Codex. The ordinary =dispatch submit= whole-job path is unchanged.