[glennhickey]

Prototype for scoring bar by divergence, with models trained automatically per pair of genomes, plus a pangenome last-train speed-up. Draft.

Stacked on #2027 (lastz scoring by divergence, which trains a model per pair of genomes in the blast step), so the base is lastz-score-models. It should be retargeted to master once #2027 merges. Trained bar models are on by default (<bar trainedModels="1">), as is #2027's lastz training. So the shipped config now trains a model per pair of genomes and scores both lastz and bar with it. On evolver mammals that takes the shipped config from 0.816 / 0.739 to 0.836 / 0.742 (pooled leaf pairs). abPOA's trained rows cost up to 0.9 points of ancestor recall against lastz training alone. trainedModels="0" keeps bar's own scoring.

What it adds

Trained bar models (<bar trainedModels>, on by default). The blast step's last-train run on each pair of genomes it aligns also produces that pair's model for bar.

• Same training as lastz: it's the same run Optional lastz scoring by divergence: a table, or per-pair last-train models (prototype) #2027 uses for lastz. With this switch on, the blast step trains even if lastz doesn't use the models.
• abPOA: each row is scored with the model for its genome and the nearest ingroup, which is always a pair blast trained. The model is on HOXD70's scale like the shipped matrix, with the trained gap as the first piece and a long-gap piece built the way the pangenome code builds it.
• pecan: each pair of sequences is scored with the model for its two genomes: a five-state pair HMM made from the trained probabilities.
• Untrained pairs: pairs blast never trains, such as pecan's outgroup–outgroup pairs, take the model trained nearest their distance in log terms, on the reconstruction tree.
• Carrying the models: in a single cactus run they go straight from blast to the consolidated job. cactus-blast leaves them next to its PAF as <paf>.models.json, where cactus-align picks them up. cactus-align --trainedModels gives another path; cactus-prepare's WDL and Toil workflows use it. Pangenome alignments come from minigraph, so cactus-align doesn't look for them there.

What the models plug into:

• Per-row abPOA models: <poa partialOrderAlignmentRowModels="N"> with <rowModelK> children.
• Per-pair pecan models: <pecan pecanPairModels="N"> with <pairModelK> children.
• Selection: each model can name a pair of genomes (genomes="A B"), and that pair uses it. Otherwise a model applies to the distances its maxDistance covers. Tables can be written by hand too.
• abPOA per-read scoring: the row models rely on a new abPOA field, abpoa_para_t.read_abpt, which lets each read be scored with its own parameters.
• pecan per-pair scoring: the pair models rely on a per-pair callback added to cPecan's multiple aligner.

Deterministic cPecan multiple aligner. The same sequences and models now always give the same alignment, independent of the random generator's state and of memory addresses.

• Effect in cactus: an end's alignment no longer depends on how many random draws the ends aligned before it used.
• Evolver mammals: two runs were identical on every genome, and accuracy was unchanged to four decimals.

Pangenome last-train (lastdb -c, -C2). These are LAST's own genome-to-genome recipe; without them each lastal pass spends most of its time aligning tandem repeats. Per pass, CHM13 against GRCh38 chr17 went from 2,831 to 207 CPU-s on the first pass and from about 350 to 60 after it, with the same model.

Evolver mammals

Pooled leaf-pair precision / recall. Rows marked * are from the same harness on this branch before its restack onto the latest master (which added indel placement; the shipped baseline moved by under 0.001):

lastz	bar	P / R
HOXD70	abPOA, shipped	0.816 / 0.739
HOXD70	abPOA, trained per pair	0.824 / 0.736
HOXD70	abPOA, hand-built per-row table*	0.827 / 0.734
HOXD70	pecan, built-in model*	0.838 / 0.752
HOXD70	pecan, trained per pair	0.841 / 0.754
HOXD70	pecan, hand-built per-pair table*	0.841 / 0.755
trained per pair (#2027)	abPOA, trained per pair	0.836 / 0.742
trained per pair	pecan, trained per pair	0.855 / 0.760
trained per pair	pecan, hand-built table*	0.855 / 0.760

• Automatic training matches hand-built tables: the per-pair models do as well as tables built from an evolver last-train ladder, with nothing to build.
• abPOA's trained rows gain precision and lose a little recall, ancestor recall included.
• pecan's per-pair models gain about 0.3–0.4 points of precision and 0.2 of recall over its built-in model. The hand-built table's gain held on every one of the 10 leaf pairs.
• Combined: with lastz also trained, pecan reaches 0.855 / 0.760.
• Both run modes work: cactus-prepare's separate steps and a single cactus run both applied every ancestor's models, and agreed to within 0.0014. With the new defaults, the shipped config reproduces the explicitly switched-on run byte for byte.
• Tests: the bar and cPecan unit tests pass.
• cactus-prepare's WDL and Toil workflows: CI showed the WDL passing <paf>.models.json to cactus-align as a second alignment file, and the Toil workflow did the same. Both now pass it with --trainedModels.
  • Local runs: both workflows applied every ancestor's models. Both passed the evolver tests' accuracy check at about 0.874 / 0.654, against a baseline of 0.895 / 0.707 with tolerances of 0.05 / 0.13. A single cactus run of the config shipped before this work scores 0.853 / 0.655 on the same check.
  • WDL: it ran under cromwell 49 with this branch's binaries instead of the docker image.

Submodules

• abPOA: pinned to main-per-read-params (Optional per-read scoring parameters (abpoa_para_t.read_abpt) abPOA#13).
• cPecan: pinned to pecan-params (Per-pair state machines in the multiple aligner, and make it deterministic cPecan#10).

Those should merge first, and these pins then move to the merge commits.

Not in this PR

• Real-data validation. The 577-way pilot would test whether trained bar scoring is worth it beyond evolver's simulated sequence.

🤖 Generated with Claude Code