numerics: fp16 probabilities under AMP feed AUROC and checkpoint selection; eval and predict use different precision policies

[jayhesselberth]

Tier P1 · effort S · from the 2026-09-13 review (training F9, inference F9, F10) — verify on the shipped model first

Under --mixed-precision the final Linear emits fp16 and torch.sigmoid is
not on autocast's fp32 list, so preds_arr in Trainer.validate
(src/leech/training.py:1093-1098 → :1134-1136) and in train_epoch
(:981-984) is float16: sigmoid rounds to exactly 1.0 above x≈11 and carries
~3 significant digits near 0.5. roc_auc_score then ranks a quantised score,
and val_auc is the checkpoint criterion. evaluation.py:203-213 has the same
pattern under its own autocast, and evaluation.py:126-136 compiles the model
unconditionally, on CPU too. The predict path scores in fp32.

The training reviewer rated this latent because leech's own train.smk does
not pass --mixed-precision. The production charging recipe does
(escapepod-models/workflow/rules/charging.smk, LEECH_TRAIN_PRODUCTION,
mixed_precision: True), so the shipped charging_tcn_rna004 was selected and
evaluated on fp16-quantised probabilities. The operating point sits at FPR
0.0034, the region where ties among confident scores matter most.

Acceptance criteria

• First, measure: load the shipped model's per-chunk test_scores.npz (escapepod-models charging_leech_eval output) and report the number of distinct probability values, the count of exact 0.0/1.0, and whether every value is representable in float16. Put the numbers in the PR. If quantised, note it for escapepod-models so the published AUROC can be re-derived from fp32 scores.
• torch.sigmoid(logits.float()) (or ranking on the logits, which AUROC permits) in train_epoch, validate and evaluate_model; probabilities written to disk are float32.
• One precision policy shared by eval test and predict (the same autocast decision and the same torch.compile gate, reusing PR fix(predict): make torch.compile take effect, and unblock the GPU pipeline #253's threshold and --no-compile); never compile on CPU.
• evaluate_model accumulates per-batch arrays and concatenates once instead of list.extend(ndarray) of boxed scalars (evaluation.py:207-209, 314-316); labels are not moved to the device and back (:301).
• A test trains one epoch with mixed_precision=True on the fixture and asserts the returned probabilities have float32 dtype and more distinct values than the fp16 grid allows.

Non-goals

• Changing the loss computation's precision (BCE-with-logits is already on the fp32 list).

[jayhesselberth]

Fixed in #297 (logits cast to float32 before sigmoid/AUROC in Trainer.train_epoch/validate and evaluate_model, predict/eval torch.compile policy unified, per-batch arrays concatenated once), shipped in v0.12.0. The PR didn't reference this issue number, so it wasn't auto-closed.