Repository navigation
Dispatch isolated workers within held Slurm allocations - #78
Merged
Merged
Conversation
Co-Authored-By: GPT-6 <noreply@openai.com>
Preserve independent job ownership; reserve aggregate budgets conservatively and publish exact receiving step identities. Reuse managed runners, private results repositories and immutable allocation-local source/image assets. Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Use --local --no-hardlinks only for validated full pool sources; refuse dependent, redirected or nonregular object metadata and verify copied objects are independent before native launch. Preserve ordinary job transport. Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Resolve python3 on the receiving node before exec so an unexported default Bash PATH cannot leave CPython sys.executable empty. Preserve the ordinary job bootstrap and worker ownership bytes. Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
After descriptor metadata errors, wait at most50ms for fresh same-UID/starttime Z/X or fully gone proof. Preserve live/unknown/PID-reuse refusal and retain error-boundary/incarnation diagnostics; never stat descriptor targets. Co-Authored-By: GPT-6 <noreply@openai.com>
Co-Authored-By: GPT-6 <noreply@openai.com>
Reinspect once after a still-live metadata refusal, preserving all placement checks and refusing known UID/starttime changes before later absence can erase the observation. Retain the initial failure and outcome. Record independent controls and real nondumpable-child checks; native lifecycle acceptance remains pending. Co-Authored-By: GPT-6 Codex <noreply@openai.com>
Retain main identity mismatch detection, shared per-pass exit grace, and bounded race history alongside pool deadlines and complete reinspection. Adapt retry identity checks to reject partial mismatch records without losing evidence or raising KeyError; cover before and after retry. Validation: 363 dispatch tests passed; after removing duplicate race accumulation, 107 worker and proof checks passed. Independent review reran 95 focused cases. Python 3.6 grammar passes. Native acceptance limits of the draft remain unchanged. Co-Authored-By: GPT-6 <noreply@openai.com>
ligon
marked this pull request as ready for review
October 30, 2026 06:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Workers can now reuse an explicitly owned, running single-node Slurm allocation while keeping private checkouts, native state, runtimes, caches and numeric-step cleanup boundaries.
This attach-only implementation provides:
dispatch pool attach/submit/status/stop-worker, allocation identity/lifetime checks and aggregate CPU/memory reservations.--publication-timeouton both submit commands: default 30 seconds, maximum 300, with a corresponding early shutdown window. Native-state and cleanup budgets remain separate.The ordinary one-job-per-worker path retains allocation-wide ownership and its existing defaults.
Validation:
839cbe7.78f43ad; the later controller-only lock correction leaves receiving helpers and workspace scripts byte-identical.10c3c7f. GitNexus analysis was attempted but its native library still cannot load libssl.so.3; manual caller/AST checks are recorded.Acceptance record and evidence locations preserve the original failed trials, the measured 30-second publication-budget failure, the prelaunch lifetime refusal, and the successful follow-on. No historical failure was relabelled.
Limits: this establishes bounded pool lifecycle and synthetic scientific/API behavior, not credentialed full-country/corpus correctness, eight-worker capacity or arbitrary-duration reliability. Intentional interruption preserves crash-class partial state; unattended crash restoration remains refused and semantic resume was not attempted. Allocation creation/requested-node convenience and broader operational rollout remain follow-up work. No shared installation or deployment occurred.
Related: issue 77, which remains open for the broader capability and acceptance scope.