Onboard Tillicum: cluster doc, Slurm port, and first-login recon (#51/#70) - #91
Open
jonfroehlich wants to merge 7 commits into
Open
Onboard Tillicum: cluster doc, Slurm port, and first-login recon (#51/#70)#91jonfroehlich wants to merge 7 commits into
jonfroehlich wants to merge 7 commits into
Conversation
The makelab group was provisioned on Tillicum 2026-07-29. Tillicum is not klone with different hostnames — no partitions (QoS only), a hard 8 CPU : 1 GPU ratio, a 24 h ceiling on the default QoS, and usage billing at $0.90/GPU-hour. Our run_yolo_train.slurm is non-submittable as written; the diff table records exactly what changes and why. Motivation is #51/#70: klone's ckpt partition consumed 496.5 GPU-hours on the YOLO baseline and, in one night, produced 170 job restarts and zero completed epochs across five of six arms. Tillicum does not preempt. Flags the two decisions that need measurement rather than argument: whether our CUDA 11.8 conda env transfers to H200/Rocky 9 (vs. an NGC container), and whether 16 dataloader CPUs makes the I/O-bound tiles epoch more than 2x faster — which is the only thing that justifies its 2x billing rate, and which also settles whether tiles-vs-pano in #51 is an architecture finding or a storage artifact. Sections that are inference rather than documented fact are marked UNVERIFIED; no job has been run on Tillicum yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Tillicum migration plan depends on how much data has to cross from klone's /gscratch to /gpfs, and the answer changes the approach. Measured on klone: tiles 210 GB / 718,415 files, pano 76 GB / 192,938 files. 286 GB fits the 1 TB allocation comfortably, so storage quota is not a blocker. The ~911,000 files are: this is the many-small-file pathology in pure form, and it is measured rather than assumed -- du -sh over the tiles tree exceeded a 2-minute timeout three times before completing on a 550 s budget, and even ls -lU | head -201 on tiles/images/train timed out. That is metadata throughput, not bandwidth. Consequences now in the doc: transfer tar archives rather than the tree (a per-file rsync would pay ~911k round trips), run the tar inside a Slurm job rather than on a login node, stage pano first so the smoke test is unblocked while tiles moves, and time the untar -- it is our first real measurement of whether Tillicum's flash storage fixes the bottleneck that the tiles training arm is also probing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two pieces that need no Tillicum account, so the only remaining gate is Jon's Duo login. run_yolo_train_tillicum.slurm is a port of the klone script, kept as a separate file rather than a branch inside it: nearly every scheduler directive differs, and the klone version is the preserved record of the #51 runs and should not churn. QoS instead of partitions, 24 h instead of 72, --gres=gpu:N, 8 CPUs per GPU, no --requeue, and an explicit --mail-type=NONE so the site default cannot decide for us the way klone's did. It also separates the GPU ALLOCATION from the ultralytics DEVICE, which is the non-obvious part. Tillicum fixes CPUs at 8 per GPU, so getting 16 dataloader workers for the I/O-bound tiles arm means allocating 2 GPUs. But letting ultralytics use both puts it in DDP, changing effective batch and LR dynamics and breaking comparability with the klone single-GPU runs. So we allocate 2 and pin DEVICE=0, paying for an idle GPU on purpose: it isolates dataloader throughput as the only changed variable, which is exactly the question -- does the 2x bill buy more than a 2x speedup. tillicum_recon.sh settles, in one read-only pass, every item docs/tillicum.md currently marks UNVERIFIED: the undocumented home path (wsl-ssh.ps1 is guessing /gpfs/projects/makelab), whether long QoS is already granted, whether hyakusage and apptainer are on PATH, and whether Tillicum forces job mail the way klone's lua job_submit plugin does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t inherit Ultralytics defaults save_period=-1, keeping only last.pt and best.pt. That permanently forecloses any analysis needing a checkpoint from a specific epoch, and it cannot be recovered after the fact -- results.csv has per-epoch val metrics but no weights. The analysis we would lose: RampNet's published model trained ~1 epoch / ~9.4k steps at constant LR (#84) while this baseline gets 60 epochs with a schedule. If RampNet wins anyway that is a stronger result, but only if we can also report a COMPUTE-MATCHED YOLO point rather than just the converged one. Cost is ~150 MB per checkpoint against a 1 TB allocation. Also documents a limitation found while wiring it: resume=True reuses the args saved in the checkpoint, so an arm resumed from a klone checkpoint keeps that run's save_period=-1 and this setting does not apply to it. Per-epoch weights therefore exist only for runs started under this script. Less costly than it sounds -- the compute-matched point we most want is roughly RampNet's one-epoch budget, which is exactly the ep1 best.pt that four of the six klone arms are still holding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d budget All six questions to UW-IT came back (Sumaiya Sathar, 2026-07-30). Three answers change the plan, so they are recorded in the doc rather than left in a mailbox. Budget is set to $1,500/month with enforcement ACTIVE, and enforcement does not cancel running jobs -- it only blocks new submissions. That removes the only reason we had to prefer warn-only, so the runaway-spend guard is free. There is no way to get CPUs without the matching GPU allocation, and it is a rate-model constraint rather than a scheduling one, so the DEVICE=0 / 2-GPU design in the launcher is now confirmed necessary rather than provisional. The transfer plan changes outright: many-small-file datasets belong on /gpfs/scrubbed rather than the 1 TB project quota, and UW-IT recommends SquashFS. So pack_yolo_dataset.slurm replaces the tar approach -- tar fixed the transfer but not the destination, since untarring recreates all ~911k files on /gpfs and pays the metadata cost permanently. A SquashFS image is mounted read-only and the destination only ever holds one file. The subtle part, called out in both the script and the doc because getting it wrong is invisible: Ultralytics validates labels/<split>.cache against a hash of absolute paths. The .cache files must be inside the image (they exist on klone already, and a read-only mount cannot create them), and mounting at a different path silently invalidates them and forces a full ~911k-file rescan. Training still works -- it is just permanently slow. Also confirmed: 1 TB is correct for us (100 GB applies to demo accounts without a worktag), there is no project end date, storage grows in 1 TB increments on request, and the 100 free GPU hours do not expire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The SquashFS plan worked and still did not get the data across. klone job 37940649 packed pano into a single 76 GB image in 2h38m, solving the destination problem exactly as designed. But there is no automated klone -> Tillicum route: no shared filesystem, and neither end can authenticate to the other non-interactively, since both are Duo/keyboard-interactive with no publickey and BatchMode fails in both directions. Globus CLI is on neither side. The transfer, not the destination, was the blocker. So this regenerates the dataset on Tillicum from Hugging Face instead. That is safe rather than merely convenient: prepare_yolo_dataset.py thins background tiles with an md5 of the file stem, specifically so the choice is stable across processes and runs, so the same source and the same flags must reproduce the same split. VERIFIED, and further than counts. Job 198910 (4h40m) reproduced all four counts in the #51 record. Because klone's tree is still live, we could also compare identity rather than cardinality: the md5 of the sorted filename list is identical on both clusters for all eight image and label directories. The two datasets hold the same files under the same train/val split, not merely the same number of them. 767,840 boxes over 192,938 panos, 0 read errors. Anything trained on Tillicum is comparable to the klone arms on the data axis. The launcher is committed as-run, byte-identical to the copy that produced the dataset, per the provenance rule in the yolo_baseline README. That includes a cosmetic quirk on the last line, where backticks inside a double-quoted echo execute hyakusage rather than printing its name. Harmless, and it happens to log the job's cost, but it is an accident and not a design -- left alone because rewriting it would break the byte-identical claim. Two caveats recorded with the numbers. It cost $4.20, because Tillicum rejects CPU-only jobs and a prep job must hold an H200 it never uses -- structural, and the argument for keeping prep on klone whenever the data is reachable there. And it landed on /gpfs/scrubbed, not the 1 TB project quota, so it is subject to the inactivity purge and must not be assumed to survive a gap between arms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both ran; neither was checked in. tillicum_smoke.slurm is committed byte-identical to the copy that ran as job 198638 (md5 687cdc31, 5042 bytes). WHAT THE ENV SCRIPT PROVED. It reproduces the klone #51 toolchain exactly -- verified by running the interpreter, not by reading a lockfile: 3.11.15 / torch 2.13.0+cu126 / ultralytics 8.4.105, character-for-character what klone's training logs report. It has since driven a full 4h40m production job without incident, so the toolchain axis of comparability is closed the same way the data axis now is. The non-obvious part, recorded in the doc: conda is not a preference here. Tillicum's system python is 3.9.25 and the cu126 index tops out at torch 2.8.0 for 3.9, so a naive pip install silently yields 2.8.0 + 8.4.113 -- a baseline nobody can publish, arrived at with no error message. The script's version check is a gate rather than a report: it refuses to install a substitute and exits non-zero, which is what catches ultralytics pulling its own torch over the pinned one during its dependency resolution. Note this does NOT verify environment.yml, which is a different stack (CUDA 11.8, conda-forge, linux-64) that nothing on the YOLO path needs. That entry stays UNVERIFIED rather than being quietly upgraded. A DISCREPANCY THE SMOKE SCRIPT'S HEADER CANNOT SEE. It asserts the debug QoS "COSTS NOTHING," and sacctmgr agrees: UsageFactor is exactly 0.000000, with priority 50 against normal's 25, so probes are both free and faster to start. But hyakusage disagrees -- it reports the 2-minute smoke job as 0.03 GPU-hours and $0.03, which is raw wall-clock x $0.90 with the zero multiplier apparently not applied, contradicting its own header line about billable hours. We do not know which one ITBill follows. Left as a documented open question rather than a silent edit, since the script is the as-run record and its sacctmgr citation is accurate as far as it goes. The exposure is bounded at $0.90 per job either way, because debug caps at 1 h and 1 GPU, so this is worth asking UW-IT but not worth blocking on. The full QoS table is now in the doc so the claim can be rechecked without cluster access. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Onboards the lab onto Tillicum, UW-IT's usage-billed H200 cluster (makelab provisioned
2026-07-29), and prepares the YOLO baseline to move there. Additive only — no existing file
changes, nothing in flight is touched.
Why
klone's
ckptscavenger partition has stopped converting compute into results for #51.Measured 2026-07-30:
sacct, all arms)best.ptThe binding constraint is not the nominal 8.24 h checkpoint slice but the effective
contiguous run the partition hands out, which has collapsed to minutes. Any arm whose epoch
exceeds that can never complete one. Tillicum does not preempt — that is the entire
reason to pay for it.
This matters beyond convenience. The #71 protocol already concedes that every reportable
checkpoint is undertrained, so any benchmark number is a lower bound on supervised
YOLO. "We ran out of patience on a scavenger partition" is not a defensible reason for a
baseline to underperform, and "you undertrained YOLO" is the first objection any reviewer
raises against the paper's central comparison.
What's here
docs/tillicum.md— the first general cluster doc in the repo (we have task runbooksfor klone, but nothing describing a cluster itself). QoS table, cost model, storage terms,
software story, and a directive-by-directive Slurm migration diff.
The headline for anyone submitting: our
run_yolo_train.slurmis not submittable onTillicum as written. No partitions exist (QoS only), the default QoS caps at 24 h against
our
--time=72:00:00, and--cpus-per-task=12with one GPU violates a hard 8 CPU : 1 GPUratio and is rejected outright.
scripts/model_comparison/run_yolo_train_tillicum.slurm— a port kept as a separatefile rather than a branch inside the klone script, since nearly every directive differs and
the klone version is the preserved record of the #51 runs and should not churn.
The non-obvious part is that it separates the GPU allocation from the ultralytics
device. Tillicum fixes CPUs at 8 per GPU, so 16 dataloader workers for the I/O-bound
tiles arm means allocating 2 GPUs — but letting ultralytics use both puts it in DDP, which
changes effective batch and LR dynamics and breaks comparability with the klone single-GPU
runs. So we allocate 2 and pin
DEVICE=0, paying for an idle GPU on purpose. That isolatesdataloader throughput as the only changed variable, which is exactly the question: does the
2× bill buy more than a 2× speedup?
It also sets
save_period=5. Ultralytics defaults to keeping onlylast.pt/best.pt,which permanently forecloses a compute-matched comparison — RampNet's published model
trained ~1 epoch / ~9.4k steps at constant LR (#84) while this baseline gets 60 epochs with
a schedule, and reporting both the converged and the matched point is much stronger than
the converged one alone.
results.csvhas per-epoch metrics but no weights, so it cannot bereconstructed later.
scripts/tillicum_recon.sh— read-only, ~10 s, submits nothing, costs nothing. Settlesin one pass every item the doc currently marks UNVERIFIED.
Measured, not assumed
The data that has to cross: 286 GB in ~911,000 files (tiles 210 GB / 718,415; pano
76 GB / 192,938).
du -shover the tiles tree beat three separate timeoutsbefore completing on a 550 s budget, and even
ls -lU | head -201ontiles/images/traintimed out. That is metadata throughput, not bandwidth.
Consequences recorded in the doc: transfer archives rather than the tree (a per-file rsync
pays ~911k round trips), run the
tarinside a Slurm job rather than on a login node, stagepanofirst so the smoke test is unblocked whiletilesmoves, and time the untar —it is our first direct measurement of whether Tillicum's flash fixes the bottleneck the
tiles training arm is also probing.
Status: nothing here has run
No job has been submitted on Tillicum and nobody has logged in. Both scripts are
syntax-checked only. Sections written from the public docs rather than from experience are
marked UNVERIFIED inline, including two things to fix on first login: the HostName (the
get-started page gives
tillicum.hyak.uw.eduin one place andtillicum.hyak.eduin itsown ssh example) and the per-user home path, which is undocumented.
Companion change outside this repo:
dotfilesgains atillicumtarget forwsl-ssh.ps1plus its operating rules, which are auto-imported into every Claude session.
Related
🤖 Generated with Claude Code (claude-opus-5[1m])