trainer hygiene: duplicated AMP/loss branches, two seeding routines, patience not checkpointed, weights stored twice

[jayhesselberth]

Tier P3 · effort S · from the 2026-09-13 review (training F8, F14, F15)

• training.py:943-957 and :1092-1104 are identical bodies per AMP flag; torch.autocast("cuda", enabled=self.use_mixed_precision) collapses them. validate re-implements the CE-target logic (:1086-1104) without label smoothing or _weighted_ce_global, so under DDP with weighted CE the reported val loss is not the trained loss. Labels are moved to the device and back (:1083, :1137). :524-527 has identical if/else arms. compute_class_weights and compute_class_weights_from_labels (:254-336) are the same function twice (one caller of the first, :2088, already passes an array).
• Two seeding routines: model_loading.py:19-52 (writes seed.txt) and training.py:1691-1722 (writes training_seed.txt); train_model omits random.seed.
• patience_counter is not in the checkpoint (training.py:1234), so a resumed run restarts early stopping. _resume_from_checkpoint loads into self.model, the compiled module on GPU, so a CPU-written checkpoint resuming on GPU is untested and would fail on key names.
• Every checkpoint carries best_model_state_dict beside model_state_dict (training.py:1477-1485), so model_best.pt holds the weights twice plus the optimizer under save_optim_every=1.
• profiling.py:325-327 batch_size = steps[0].total_ms and (...) is a truthiness accident (train_loader.batch_size is meant). commands/benchmark.py:28-99 is a third loader+model builder; :72 iterates a row view per chunk for max_label though the dataset has _multiclass.

Acceptance criteria

• One _compute_batch_loss(batch, aux_heads: bool) used by train_epoch and validate; one class-weight function; one seeding routine (torch-free core, see the prepare fail-loudly issue) writing one file.
• patience_counter and the RNG states are checkpointed and restored; a resume test asserts early stopping continues where it left off; a CPU→GPU resume test (skipped without CUDA) loads through clean_state_dict.
• model_best.pt holds one copy of the weights; the epoch checkpoint carries best_* only if a documented consumer needs it (grep and state).
• profiling.py reads the loader's batch size; benchmark uses the shared loader factory from the TrainConfig issue.
• tests/test_training.py::test_validate_reads_at_most_two_tensors_per_batch tightens to one.

Non-goals

• Anything that changes the trained loss value in the single-process path (bit-equality test stays).

[jayhesselberth]

The "patience_counter and the RNG states are checkpointed and restored" acceptance bullet moves to #330, which owns interrupt-safe resume (rolling per-epoch model_resume.pt, complete trainer state, recipe guard). The remaining items here — loss/AMP dedup, one seeding routine, model_best.pt weight dedup, profiling — are unchanged. Note #330 needs best_model_state_dict in the rolling checkpoint (it recreates a model_best.pt Snakemake deleted), which answers the "grep and state" question for that file; model_best.pt itself can still be slimmed.