[glennhickey]

Fixes the crash that stopped the oct9 CHM13 HPRC build: cactus_consolidated(chr2) died 9 times out of 9 with abPOA's Error in cg_backtrack. chr2's last-train model was the only one of the 25 to learn a gap extension of 2, and the ×30 scaling for the long-gap piece turned it into GapOpen 1290; GapExtend 60; GapOpen2 3870; GapExtend2 2. abPOA's 16-bit gate did not account for the 512·max(gap extension) margin of its "minus infinity" floor, which came out at +1824, above real scores. Windows that did not abort could be misaligned silently. The learned extension sat just over a rounding edge (1.515 against 1.5), and other models are close to it (1.43), so this can happen again.

This bumps abPOA to a version that picks its score width from bounds on the scores (ComparativeGenomicsToolkit/abPOA#14). It also makes trained models smaller without changing them.

What changes:

• abPOA is pinned at 12f99c4, the tip of Pick the DP score width from bounds on the scores; check it, retry at 32 bits abPOA#14.
  • It chooses 16 or 32 bits from bounds on the scores, checks the alignments it cannot prove safe, and redoes a failed 16-bit alignment at 32 bits.
  • It stops with a message naming the scores only if a 32-bit alignment fails its checks.
  • Its output is byte-identical wherever the old version was right, and the DP is as fast as before.
  • That PR should be merged first, with a merge commit, so the pin stays reachable.
• Dividing by the common factor: apply_long_gap() now divides the scaled model by the greatest common divisor of all its scores (reduce_scores()).
  • Every alignment and every tie stays the same, with smaller numbers.
  • chr2's model becomes 645, 30, 1935, 1 with its matrix halved. A learned extension of 1, as every other model so far has, is left alone.
  • A learned extension of 2, 3, 5 or 6 reaches abPOA with a gap extension of 30, 4 with 60, and 7 with 210.
  • This is logged at info, since nothing about the model changes.
• Rejecting unusable models: parse_train_file() rejects a model with a gap extension below 1, a negative gap open, or any match score of 0 or less.
  • Inside last_train() the alignment then falls back to the defaults, or to a borrowed model, as when training fails.
  • A --scoresFile model with these problems is now an error.
• Warning: abpoa_exact_depth() mirrors abPOA's rule for when a 32-bit alignment is proven exact. If a model only allows a window-long sequence (partialOrderAlignmentWindow, 10 kb) to be proven exact on graphs less than 50 windows deep, a warning names the model.
  • A trained model with extension 1 allows about 1.5 M rows, chr2's reduced model 3.2 M and the defaults 5 M, so the warning fires only once scores are about three times today's trained ones.
  • It matches abPOA's own C function on 20,010 random and edge-case score sets.
  • abPOA still aligns deeper graphs; it checks them instead.
• Log line: Overriding abPOA scores with trained values: … keeps its prefix, which logs are grepped for. It now ends, for example, with (chr2: learned gap extension 2; all scores multiplied by 30, then divided by their common factor 2). cactus_align passes the chromosome name in.
• pafTest.py now runs in CI:
  • It imports from cactus.paf.paf, so pytest can collect it.
  • It is in testModules, so make test runs it (GitLab test-job).
  • test_get_leaf_event_pairs, broken since 2022, is updated to get_event_pairs().
• Smaller fixes: a stale comment in apply_scores_to_config() now gives the actual factor defaults (3 for the open, 30 for the extension), and there is a ReleaseNotes.md entry.

Results:

• cactus_consolidated on the captured yeast chrI and chrIV inputs (single-threaded; the old abPOA is 36110ac):

scores	chrI	chrIV
defaults (-O 400,1200 -E 30,1)	identical	identical
trained, gap extension 30 (-O 1410,4230 -E 30,1)	identical	identical
chr2 model halved (-O 645,1935 -E 30,1)	identical	identical
chr2 model as oct9 had it (-O 1290,3870 -E 60,2)	old: Error in cg_backtrack; new: same as halved	same

"Identical" means byte-identical c2h, fasta and ref output. Peak memory is the same.

• abPOA on its own (from the abPOA PR):
  • the MSA and consensus are identical to 36110ac on 7,026 window × score-set comparisons: yeast BAR windows, HPRC haplotypes at chr10 sites, HPRC-like chr9 windows and short stress windows, with the defaults, a trained model and the halved chr2 model;
  • with the chr2 model itself, on 2,327 of these windows, 36110ac aborted on 1,030 and gave a wrong MSA on 860, while the new abPOA gives the halved model's result on all 2,327;
  • on a window that runs entirely at 32 bits it takes 0.997 of 36110ac's cycles, in cactus's AVX2 build.

Tests:

• test_trained_scores_beyond_16_bits in bar/tests/poaBarTest.c: 200 random windows aligned with the chr2 model must give the same MSA as with every score halved. cactus_barTests passes 22/22. Linked against 36110ac, it aborts in this test with Error in cg_backtrack.
• LastScoringTest in src/cactus/paf/pafTest.py checks:
  • that an extension of 1 is unchanged;
  • that an extension of 2 becomes 645, 30, 1935, 1;
  • that the reduction is exact for extensions 1–8;
  • that the config gets the reduced scores;
  • the log levels and the override line;
  • the depth bound, including the convex case;
  • the rejection of unusable models.
• make paf/pafTest.py_runtest passes 12/12.

🤖 Generated with Claude Code

https://claude.ai/code/session_013YWGkqE3ND862JF8QRZERV