Spot GPUs: checkpoint snapshots, restore points, reclaim notices and resume (#10)

[pgodlews]

Stream training checkpoints out and resume elsewhere, for spot and other interruptible GPUs. The investigation, measurements and design are in #10 (two comments).

Closes #10

LichtFeld patch (scripts/lichtfeld-patches/)

Applied by setup_lichtfeld.sh, which stops the build if it no longer applies. It adds options and changes no default, so TRAINER does not move. Intended for upstream later.

• --save-steps a,b,...: project snapshot iterations, the same parser and step scaling as --eval-steps. Until now save_steps could only be set through --config, which replaces the whole optimisation parameter set.
• SIGUSR1 in plain headless mode requests a snapshot without stopping training. The request runs on its own worker thread, so it never delays SIGTERM, and it is only wired during train().

Tested on an RTX 3090, sm_86 image built from this branch:

• --save-steps works without --config.
• A SIGUSR1 snapshot was prepared 0.15–0.3 s after the signal, early in training and at steady state.
• SIGUSR1 followed by SIGTERM still exits in about 10 s.
• The exports are written.
• PSNR was 19.958 with snapshots and 19.952 without, against 19.952–19.959 before the patch.

Queue

• train.checkpoint_every: a snapshot every N training steps (0 off, at least 500). It becomes --save-steps in LichtFeld's unscaled units. It is not a cache key term: the keys are identical to main's for existing configs, with or without it.
• Restore points (checkpoints.py, scripts/licht_restore_point.py):
  • After each committed snapshot, LichtFeld's clean_project_file keeps only the newest checkpoint (1.56 GB to 467 MB measured) and it must verify.
  • It is packed with restore.json (config, keys, TRAINER, version terms, cache root, step, sha256) as job<id>-restore.tar.
  • The tar goes to CHECKPOINT_UPLOAD_URL, replacing the previous one. S3, R2 and GCS replace an object only on a complete upload. QUEUE_RESTORE_POINTS=1 keeps them locally instead.
  • A failure is recorded and training goes on.
• Reclaim notices (preempt.py, QUEUE_PREEMPT_WATCH=aws|gcp):
  • On a reclaim notice, SIGUSR1 goes to each trainer that has reported itself up, and the queue is paused.
  • A rebalance recommendation snapshots only.
  • POST /api/snapshot does the same by hand.
• Resume: RESUME_URL downloads into QUEUE_ROOT/resume/, and POST /api/resume/import imports with or without a handoff bundle. It is refused unless the trainer, version terms, cache root and keys match. The job's train stage runs --resume … --export=….
• Telemetry: checkpoints, resumed and transfers.resume, with no URLs or paths. Documented in docs/job-telemetry.md, and a "Spot GPUs" section in docs/cloud.md.

Tests

queue/test_checkpoint.py: 26 CPU-only tests, added to CI and AGENTS.md. They use a fake trainer and a stub lichtfeld.io module. The log parser was also replayed against real patched-trainer logs: every snapshot was found at its real step.

Not yet run: licht_restore_point.py against the real lichtfeld.io module in an image (the lab box went down). It should be exercised on the next rc.