Skip to content

Dispatch isolated workers within held Slurm allocations - #78

Merged
ligon merged 19 commits into
mainfrom
fix/77-allocation-pool
Oct 30, 2026
Merged

ligon merged 19 commits into
mainfrom
fix/77-allocation-pool

Conversation

@ligon

@ligon ligon commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Workers can now reuse an explicitly owned, running single-node Slurm allocation while keeping private checkouts, native state, runtimes, caches and numeric-step cleanup boundaries.

This attach-only implementation provides:

  • dispatch pool attach/submit/status/stop-worker, allocation identity/lifetime checks and aggregate CPU/memory reservations.
  • Verified reusable source/image assets with independent Git object copies and receiving-node filesystem checks.
  • Step-scoped cleanup, durable submission receipts and conservative reconciliation. Unknown submissions retain budget; a busy optional summary update cannot disguise an already launched worker as a failed submission.
  • One complete placement reinspection after a same-incarnation live metadata refusal, retaining all placement and terminal-proof safeguards.
  • --publication-timeout on both submit commands: default 30 seconds, maximum 300, with a corresponding early shutdown window. Native-state and cleanup budgets remain separate.

The ordinary one-job-per-worker path retains allocation-wide ownership and its existing defaults.

Validation:

  • Final CI passes Python 3.9, Python 3.11 and docs at 839cbe7.
  • 382 dispatch tests passed after the publication-budget change; all 40 pool tests passed after the late-lock correction. Receiving helpers parse as Python 3.6.
  • Fresh allocation 40631087/n0041 exercised two concurrent native workers, intentional step-only interruption, and a later replacement in that same allocation. Each passed 58 synthetic scientific/API tests with zero failures/errors/skips.
  • The survivor completed 660 source-churn cycles (736 seconds), native/supervisor 0/0, with sealed state. The replacement completed 180 cycles, 0/0, sealed. The deliberately stopped worker retained 61 cycles, 143/143, with crash-class partial state under the existing runner policy.
  • All three published workspace and evidence completely. Post-exit checks verified six database hashes and one rollout prefix per worker, with zero trailing bytes, after exact step-terminal evidence and writer quiescence. Shared immutable asset identities matched; runtime/native roots were private and concurrent affinities disjoint. The survivor and coordinator survived the interruption; all reservations and test allocations were released.
  • Native trial pin was 78f43ad; the later controller-only lock correction leaves receiving helpers and workspace scripts byte-identical.
  • The local full suite had 2,443 passes, one skip and five failures; all five reproduced on unchanged main at 10c3c7f. GitNexus analysis was attempted but its native library still cannot load libssl.so.3; manual caller/AST checks are recorded.

Acceptance record and evidence locations preserve the original failed trials, the measured 30-second publication-budget failure, the prelaunch lifetime refusal, and the successful follow-on. No historical failure was relabelled.

Limits: this establishes bounded pool lifecycle and synthetic scientific/API behavior, not credentialed full-country/corpus correctness, eight-worker capacity or arbitrary-duration reliability. Intentional interruption preserves crash-class partial state; unattended crash restoration remains refused and semantic resume was not attempted. Allocation creation/requested-node convenience and broader operational rollout remain follow-up work. No shared installation or deployment occurred.

Related: issue 77, which remains open for the broader capability and acceptance scope.

Sue the Coder and others added 12 commits October 25, 2026 22:13
Co-Authored-By: GPT-6 <noreply@openai.com>
Preserve independent job ownership; reserve aggregate budgets conservatively and publish exact receiving step identities. Reuse managed runners, private results repositories and immutable allocation-local source/image assets.

Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Use --local --no-hardlinks only for validated full pool sources; refuse dependent, redirected or nonregular object metadata and verify copied objects are independent before native launch. Preserve ordinary job transport.

Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Resolve python3 on the receiving node before exec so an unexported default Bash PATH cannot leave CPython sys.executable empty. Preserve the ordinary job bootstrap and worker ownership bytes.

Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
After descriptor metadata errors, wait at most50ms for fresh same-UID/starttime Z/X or fully gone proof. Preserve live/unknown/PID-reuse refusal and retain error-boundary/incarnation diagnostics; never stat descriptor targets.

Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
@ligon
ligon changed the base branch from feat/lsms-dispatch-profile to main October 26, 2026 14:23
Sue the Coder and others added 7 commits October 28, 2026 12:25
Reinspect once after a still-live metadata refusal, preserving all placement checks and refusing known UID/starttime changes before later absence can erase the observation. Retain the initial failure and outcome. Record independent controls and real nondumpable-child checks; native lifecycle acceptance remains pending.

Co-Authored-By: GPT-6 Codex <noreply@openai.com>
Retain main identity mismatch detection, shared per-pass exit grace, and bounded race history alongside pool deadlines and complete reinspection. Adapt retry identity checks to reject partial mismatch records without losing evidence or raising KeyError; cover before and after retry.

Validation: 363 dispatch tests passed; after removing duplicate race accumulation, 107 worker and proof checks passed. Independent review reran 95 focused cases. Python 3.6 grammar passes. Native acceptance limits of the draft remain unchanged.

Co-Authored-By: GPT-6 <noreply@openai.com>
@ligon
ligon marked this pull request as ready for review October 30, 2026 06:46
@ligon
ligon merged commit d2556f0 into main Oct 30, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant