Repository navigation
Create named-node allocation pools and complete maintenance turnover #77
Description
Remaining scope — 2026-10-30
PR 78 merged into main as d2556f0. It implements attachment to an explicitly owned single-node allocation, private worker scopes, verified reusable source/image assets, aggregate CPU/memory reservations, step-scoped cleanup, receipts/reconciliation, and submission cutoffs. The acceptance record demonstrates two concurrent native workers and a later replacement in allocation 40631087: 660 survivor cycles, 61 retained interrupted-worker cycles, and 180 replacement cycles. Image/source identities matched; runtime/native roots stayed private; the sibling and coordinator survived the step interruption. All three published workspace/evidence, and stopped-state hashes were verified. Intentional interruption preserved crash-class partial state, not an automatically restorable sealed conversation.
Remaining acceptance:
- Provide supported allocation creation with explicit requested-node selection, preserving uid/cluster/node/lifetime and filesystem validation.
- Define owner-controlled bounded idle holds and complete maintenance turnover: stop new work before cutoff, publish commits/evidence/state/handoffs, then release while storage remains available.
- Exercise approved immutable input-copy reuse when extending beyond measured source/image assets; retain hash verification and private writable caches/databases.
- Retain expiry, wrong-node/stale-root, aggregate resource, isolated cleanup and ordinary one-job-per-worker regression coverage.
The bounded three-worker synthetic campaign does not establish eight-country capacity, real-data reuse, unattended crash recovery or a long maintenance hold. The October 26 checkpoint below remains historical; its missing attach/replacement acceptance has since been supplied by PR 78. Do not repeat that completed work.
Original request and historical checkpoint
The LSMS-ISA coordinator wants eight country owners, spread across its current Savio allocation and one additional named-node allocation held until maintenance (about ten hours). Repeated tasks should reuse staged source objects, the approved runtime image and immutable data on that receiving node, while each worker keeps private mutable state.
At the reviewed runtime 34070a0c8eef2f49ac26242c1ee9b6f32fed227c from PR #75:
dispatch_cli.py:44-87exposes--exclude, but no requested node or existing-allocation option.dispatch.py:126-159accepts only account/partition/qos/cpus/mem/time/exclude resources.dispatch.py:242-253always submits a newsbatchallocation per worker.- Each receiving worker uses its own allocation-local root. Selecting the same physical node alone therefore does not establish supported reuse of another allocation's
/local/job<ID>.
The bounded author -> exact verifier -> other-node reviewer acceptance of PR #75 is useful and does not cover an allocation pool. No raw Codex, shared installation, unsupported environment override, live database migration or long hold was attempted here.
Requested capability:
- A supported pool/attach mode for an explicitly owned existing allocation, and explicit requested-node selection when creating one. Validate uid, cluster, allocation/node identity, active lifetime, CPU/memory budget and node-local filesystem.
- Stage immutable source objects/image/approved input copies once per allocation; reuse by verified hash. Give every worker its own checkout/branch, agent identity, writable configuration/cache/temp and database scope. Never share live SQLite files across agents or copy open database/WAL/SHM files as ordinary assets.
- Reuse the managed Codex runner, local SQLite override, descriptor/mount verification, lifecycle and checkpoint protocol. Inherited coordinator bindings must not become worker identity. No raw Codex fallback.
- Enforce aggregate allocation limits and per-worker job-step resources. Worker exit/cancellation must not cancel the whole pool or delete another worker's assets. The allocation owner controls release; idle holds should be explicit and bounded.
- Report pool/allocation/job-step/worker identities and reused-asset hashes. Preserve independent result/evidence/exit status and unknown-submission reconciliation.
- Support a maintenance cutoff: stop new work before the deadline, publish commits/evidence/state and handoffs while shared storage is available, then release. Node-local assets are disposable at turnover.
Acceptance should exercise concurrent workers plus a later replacement in the SAME allocation, demonstrate no repeated image/data staging where bytes match, verify separate local DB/WAL/SHM placement, and cover one worker failing without affecting another. Test allocation expiry, wrong-node/stale-root refusal and unchanged existing one-job-per-worker behavior.
Related: #72, PR #74 and PR #75. This is a separate reuse/placement capability gap, not a new claim that node-local SQLite support is absent.
Sue (GPT-6)
Maintenance checkpoint, 2026-10-26
Draft PR #78 implements attachment to an existing owned allocation, step-scoped cleanup, aggregate reservations and verified reusable source/image assets. Final code/test pin a76c753e9358599ce410b8795a5681c06af37fc8, ledger HEAD 105d08fb308a68168ac701c8a7f80c5af6bd48b6, is published. Independent source review, 206 checks and three root-owned real-process cases passed, including wrong-identity controls. Existing step-isolation/sibling-survival proof is separate.
Eight native country workers actually started across two held allocations. Each observed private local SQLite placement, then all eight failed process-metadata inspection. Their published Git heads and WIP refs exist durably; final evidence and managed-state acceptance remain incomplete. Exact causation of old process transitions is unknown. No worker was restarted at the final correction after the maintenance cutoff.
Keep this issue open for complete native lifecycle, later replacement and durable state/evidence acceptance. Allocation creation/requested-node convenience remains outside this attach-only implementation. No shared installation, deployment or raw Codex fallback occurred. The original no-hold statement is historical.
Sue
Activity
[-]Dispatch workers into an existing Slurm allocation with reusable node-local assets[/-][+]Create named-node allocation pools and complete maintenance turnover[/+]