Skip to content

Record where the YOLO baseline weights actually live (#51) - #92

Closed
jonfroehlich wants to merge 1 commit into
mainfrom
docs/yolo-weights-provenance
Closed

Record where the YOLO baseline weights actually live (#51)#92
jonfroehlich wants to merge 1 commit into
mainfrom
docs/yolo-weights-provenance

Conversation

@jonfroehlich

Copy link
Copy Markdown
Member

One-section fix to scripts/model_comparison/yolo_baseline/README.md.

The problem

That record is meticulous about provenance — toolchain versions, resolved hyperparameters
(including optimizer=auto silently resolving to MuSGD(lr=0.01) and overriding the passed
lr0), the Slurm job↔config map, dataset prep decisions, and two documented self-corrections.

Its "Where the weights live" section was still a TODO, for the one artifact in the whole
record that cannot be regenerated.

The fix

They have been durable since 2026-07-29 and nothing pointed at them:

/gscratch/makelab/jonf/rampnet_yolo_baseline_51 — klone lab storage, outside the
~21-day /gscratch/scrubbed purge that this record exists to survive. 1.8 GB; each arm
directory holds best.pt, last.pt, args.yaml, results.csv. A MANIFEST.md there
already carries the snapshot time, the job↔config map, per-arm epoch/mAP state, and
sha256 for all 10 .pt files verified against source at copy time — it simply was not
referenced from the record.

Two caveats included rather than left implicit

  • It is a snapshot, not a live mirror, taken 2026-07-29T20:51Z while all five arms were
    still training. They have advanced since — y11l_pano 13 → 15 epochs, y26_pano 18 → 20
    as of 2026-07-30 — so it must be re-taken before anything is quoted as final, and
    /gscratch/scrubbed/jfroehli/yolo_runs/<arm>/results.csv stays the live source until then.
  • y26_tiles_l40s is missing entirely. That arm postdates the snapshot (Slurm 37889646,
    forked to a dedicated gpu-l40s node on 2026-07-29 to escape the ckpt preemption ceiling)
    and needs adding on the next pass.

Staging to Hugging Face is still worth doing for external reproducibility, and the section
says so — it is just no longer the only thing standing between these weights and the purge.

Related: #51 · #91 (Tillicum onboarding, which re-snapshots as part of the migration)

🤖 Generated with Claude Code (claude-opus-5[1m])

The record's "Where the weights live" section still carried a TODO to stage them
somewhere durable. They have been durable since 2026-07-29: /gscratch/makelab/
jonf/rampnet_yolo_baseline_51 on klone lab storage, outside the ~21-day
/gscratch/scrubbed purge this record exists to survive, with a MANIFEST.md
carrying sha256 for all 10 .pt files verified against source at copy time.

A record this careful about provenance should not point at a TODO for the one
artifact that cannot be regenerated.

Two caveats included rather than left implicit: it is a snapshot taken while the
arms were still training (y11l_pano has since gone 13 -> 15 epochs, y26_pano
18 -> 20), so it is not a live mirror and must be re-taken before anything is
called final; and the y26_tiles_l40s fork postdates it and is missing entirely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jonfroehlich

Copy link
Copy Markdown
Member Author

Closing as redundant — PR #87 already fixes this TODO, and more thoroughly than this branch does. I flagged it as an open gap after reading main, without checking that an open PR already closed it. #87 additionally documents the sha256 verification result (10 ok / 0 mismatched), the /gscratch/scrubbed purge rationale, and why last.pt is kept alongside best.pt (optimizer/EMA state for resume — exactly what the y26_tiles_l40s fork needed). Both would have edited the same lines and conflicted.

Two small deltas from this branch that #87 does not currently carry, noted here so they are not lost:

  1. The snapshot has measurably drifted. Refresh the YOLO baseline record: durable weights, ckpt ceiling, fork (#51) #87 says it was staged "while the runs were still going"; the concrete drift as of 2026-07-30 is y11l_pano 13 → 15 epochs and y26_pano 18 → 20. Worth an explicit "re-snapshot before quoting as final," with /gscratch/scrubbed/jfroehli/yolo_runs/<arm>/results.csv named as the live source until then.
  2. y26_tiles_l40s's own weights are not in the durable snapshot. Refresh the YOLO baseline record: durable weights, ckpt ceiling, fork (#51) #87 mentions the fork as a consumer of last.pt, but the fork postdates the 2026-07-29T20:51Z copy, so its weights are not covered by it.

Happy to add both to #87 as a commit if wanted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant