[glennhickey]

This moves taffy/submodules/abPOA from 017a3ac (upstream v1.5.1, January 2024) to 12f99c4 on the CGT fork, the tip of ComparativeGenomicsToolkit/abPOA#14. That is upstream v1.5.7, plus the fork's packed DP rows and a fix to how abPOA chooses between 16- and 32-bit scores. cactus is moving to the same abPOA (ComparativeGenomicsToolkit/cactus#2040). The submodule URL now points at the CGT fork, as cactus's does, since that is where the commit lives.

Depends on ComparativeGenomicsToolkit/abPOA#14, which should be merged into the fork's main with a merge commit (not a squash), so that 12f99c4 stays reachable.

What the update brings

Upstream's band-end fix (abPOA #66, v1.5.2, April 2024). At 017a3ac, abPOA reads DP cells past the end of the band that were never written. taffy norm hits this on its own test data:

• valgrind reports uninitialised reads inside abPOA (simd_abpoa_align_sequence_to_subgraph, abpoa_add_graph_edge and others), all reached through align_interstitial_gaps_abpoa:

  • evolverMammals, norm -b: 15.5 million errors from 244 contexts;
  • yeast chrIV from hal2maf, norm -a: 162,000 errors from 188 contexts.

  With this PR both report 0 errors.

• The output depends on the SIMD build, and can change from run to run. On the 447-way input with -n 1000, today's AVX2 binary gave two different outputs in three runs, and today's SSE2 build (CACTUS_LEGACY_ARCH) a third.

• A saved abPOA input from January 2024 failed with "Error in cg_backtrack" before get_column_iterator SEGV #66. It has 8 sequences, 36 kb in all, and was run with the same abPOA settings taffy uses. Placed as the gap between two blocks and merged with -n 10000:

build	columns	aligned base pairs	peak memory
today's taffy, AVX2	20,114	26,573	1.77 GB
today's taffy, SSE2	18,504	31,876	1.52 GB
this PR, AVX2 and SSE2	11,998	54,508	0.84 GB

The score width fix (CGT fork). abPOA used to pick 16-bit scores with a test that ignored part of the margin below its "minus infinity". A trained scoring model in cactus (gap extension 60) then made it exit with "Error in cg_backtrack", and some windows were misaligned without any error. It now picks the width from bounds on the scores, checks any result it cannot prove, and redoes a 16-bit alignment that fails a check in 32 bits. taffy's scores are fixed (gap extension 30, match at most 100), so this changes no taffy output: it is identical to the fork's commit before the fix (36110ac). taffy takes it so that it builds the same abPOA as cactus.

Other changes since 017a3ac: upstream v1.5.2 to v1.5.7 (an overflow fix (#79), changes to how the graph's edges are ordered, put_gap_at_end, a SIMD dispatch build, incremental MSA (v1.5.6), and klib names prefixed so they no longer clash with htslib's kstring functions), and the fork's packed DP rows (peak memory 5 to 14% lower on the -n 1000 inputs below).

Changes in taffy

• taffy/impl/merge_adjacent_alignments.c:
  • Reset the shared abPOA handle before each abpoa_msa call. Since v1.5.6, abpoa_msa starts afresh only when the handle holds no sequences; otherwise it adds the new sequences to the previous call's graph. Without the reset, taffy norm on the evolverMammals test grows past 9 GB, and 10 norm tests fail. With the old abPOA the reset changes no output.
  • Set put_gap_at_end. v1.5.4 turned "keep the last gap at the end" (abPOA Update README.md #36), which used to be always on, into this option, off by default. Setting it keeps taffy's gap placement as it was.
• Makefile: abPOA now names its library after the SIMD level it is built for (libabpoa_avx2.a with avx2=1, which include.mk exports, and libabpoa_sse2.a with CACTUS_LEGACY_ARCH), so -labpoa failed to link. Link whichever library was built as lib/libabpoa.a, as cactus does.
• .gitmodules: abPOA's URL is the CGT fork.

Output

Builds were made as cactus's downloadMafTools does (htslib 1.23, HAL), in AVX2 unless noted. "Same" means byte-identical output.

input	this PR vs today's taffy
yeast chrI, norm -a	same
yeast chrI, norm -a -n 1000 -m 1000	same
yeast chrIV, norm -a	same
yeast chrIV, norm -a -n 1000 -m 1000	same
447-way chr22 test file, norm	same
447-way, norm -n 1000 -m 1000	same as today's SSE2 build; today's AVX2 build is not deterministic here
evolverMammals, add-gap-bases then norm	differs: gap placement only, same size
evolverMammals, norm -b	differs: gap placement only, same 689,346 columns and 17,704,820 aligned base pairs
evolverMammals, norm -b -n 1000 -m 1000	differs: same 1,014,797 columns, 1 aligned base pair fewer (of 17.8 million)
the cg_backtrack window, norm -b -n 10000	differs: the table above

The evolverMammals changes come from upstream v1.5.4's reordering of graph edges, which changes which of several equal-scoring alignments abPOA returns (for example, where a run of Ns is placed against a gap).

Testing

• make test passes: 46 C tests and the test_tai.py checks. It also passes in the SSE2 build.
• With abPOA at 36110ac instead, the fork commit before the score width fix, taffy norm output is the same on all 10 inputs above, in AVX2 and SSE2 (measured before put_gap_at_end was set, with the same taffy code in both builds).
• Speed: old and new ran at the same time, each pinned to its own core, under perf stat, 10 pairs per input. A build paired with itself stays within 0.2% of itself.
  • Against 36110ac, the score width fix costs 0.3 to 0.9% more instructions. Median cycles change by −0.7% to +1.6%; single pairs vary by up to ±5%.
  • Against today's taffy, cycles rise 6 to 8% with -n 1000, where abPOA does most of the work, and 0.5% at default settings. Bisecting puts most of this in the upstream commits up to the Fixes to taffy norm to make it not make duplicate rows. #79 overflow fix (v1.5.2, v1.5.3 and Fixes to taffy norm to make it not make duplicate rows. #79), which include get_column_iterator SEGV #66.

Once this is merged, cactus's build-tools/downloadMafTools needs its taffy commit bumped.

🤖 Generated with Claude Code

https://claude.ai/code/session_013YWGkqE3ND862JF8QRZERV