Skip to content

Optimize depth pipelines and add upstream Seqair backend - #114

Draft
sstadick wants to merge 1 commit into
masterfrom
perf/seqair-depth
Draft

sstadick wants to merge 1 commit into
masterfrom
perf/seqair-depth

Conversation

@sstadick

@sstadick sstadick commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Scope

Bring the useful application-level results from the Goose/Rust experiments into Perbase, using published upstream Seqair 0.10.1, not the experimental fork. HTSlib remains the default. Build with --features seqair and select --seqair on either depth command to opt in.

  • Both backends format output in counting workers and send bounded 64 KiB blocks in region order. Readers and scratch allocations are reused. A small -C budget constrains queues rather than reducing counting to one worker.
  • Base-depth reuses same-name groups, uses singleton/pair shortcuts, and preserves stable tie handling for larger groups. Seqair also bypasses mate grouping when read spans prove a name cannot overlap itself, caches filtering decisions, and omits retained auxiliary tags.
  • Only-depth reuses one input record and its difference array, avoids cloning names for existing groups, and replaces interval-query index construction with stable sorting and one-pass union. Existing clipping, deletion, reference-skip, mate, coordinate, and merging behavior is preserved.
  • Seqair only-depth requests core fields and, for mate fixing, names. It uses the raw-record API rather than a pileup store.
  • Safe small-counter formatting and the mate fast paths incorporate Rob's suggestions. No custom allocator, unchecked vector pushes, or new unsafe blocks.

Compatibility and limitations

Seqair's generic depth cap does not match HTSlib's whole-read admission at boundaries; the adapter preserves Perbase's existing rule instead. The comparison tests include low and zero caps, ties, filters, and all nine mate strategies.

The optional backend requires Rust 1.98.1 or newer and an external FASTA for CRAM, including embedded-reference CRAM. BAM long-CIGAR records expose an upstream RecordStore limitation: it counts the two placeholder CIGAR operations rather than expanding CG. Base-depth now fails with an actionable instruction to rerun without --seqair, rather than returning wrong counts. HTSlib and the Seqair only-depth raw reader handle the tested long-CIGAR fixture correctly.

Validation

  • cargo +1.98.1 test --locked: 132 unit tests and one doctest passed.
  • cargo +1.98.1 test --locked --features seqair: 137 unit tests and one doctest passed.
  • cargo fmt --check and git diff --check passed. Clippy completed; existing warnings in examples/older tests and dependency future-compatibility warnings remain.
  • Ran the built release executable through the Goose CLI differential suite with both backends: independent CIGAR/depth oracles, BAM/CRAM, all mate strategies, filters, depth caps, missing sequence, BAI/CSI, BED/VCF regions, empty and quoted references, repeated regions, chunk/worker changes, BGZF output, and output failures. For Seqair's embedded/no-reference CRAM cases, supplied the fixture FASTA to both implementations; the unsupported reference-free mode is not claimed. Skipped only the Goose benchmark-runner's self-test, which depends on that runner's local package cache; it is not a Perbase functionality test.
  • Added a CLI depth test with explicit expected coverage across skips, deletions, mate overlap, filtering, and BED output. Existing depth tests run unchanged against the replacement counter.
  • Reproduced incorrect base-depth results with a 65,536-operation BAM before the long-CIGAR guard, then verified nonzero exit and the recovery message. Only-depth still produces the same depth as HTSlib.
  • CI now tests both default and Seqair-feature builds; both passed.

Performance

Fresh measurements of this PR's build versus a clean build of released Perbase 1.4.1. These are not the experimental fork's R7 numbers.

Chr20 BAM on the same computer, -t 32 -c 1000000, default queue budgets, uncompressed TSV written to a regular file (no fsync). One warmup followed by two measured runs for each variant/workload, run sequentially in rotating order without concurrent builds or tests. All 27 outputs matched the corresponding workload's SHA-256. These are preliminary two-run measurements, not confidence intervals.

Workflow 1.4.1 median seconds PR + HTSlib PR + upstream Seqair
base-depth -m 49.425 19.875 7.340
only-depth 10.065 4.175 4.140
only-depth -m 10.175 4.150 4.120

The application changes improve both backends. Seqair gives another approximately 2.71× speedup over the optimized HTSlib base-depth path here; only-depth is essentially tied between the two new backends.

Workflow 1.4.1 peak RSS, GiB PR + HTSlib PR + upstream Seqair
base-depth -m 10.24 2.51 5.59
only-depth 1.53 0.31 0.46
only-depth -m 1.52 0.38 0.51

RSS is the maximum over the two timed runs, not an average. Seqair's base-depth speed comes with more memory than the PR's HTSlib path because it retains each active region's reads.

Samples and build provenance

Seconds in measured order:

  • Base + mates: release [49.47, 49.38]; PR HTSlib [19.55, 20.20]; PR Seqair [7.33, 7.35].
  • Depth: release [9.78, 10.35]; PR HTSlib [4.22, 4.13]; PR Seqair [4.18, 4.10].
  • Depth + mates: release [10.19, 10.16]; PR HTSlib [4.16, 4.14]; PR Seqair [4.11, 4.13].

Both source revisions built with Rust 1.98.1, their release profile, RUSTFLAGS='-C target-cpu=native', CC=clang, and CFLAGS='-O3 -march=native'. Candidate built with --features seqair; both PR columns use that same executable, changing only --seqair. Baseline built from clean 1.4.1 with no features. Both commands used --locked and their respective Cargo.lock files.

  • Input SHA-256: a3527928f449355c693275c95fdf2712efaa6279ed57e461652b758d78ff6804.
  • Release executable: 136131baf86623e9d55110318e691878ade0ce98704f6612a57b9ed076044060.
  • PR executable: b10b1722946bea9a8b952bf416579b94ba46c390b86a5b863fd3965f850c1eab.
  • Base + mates TSV: b4e1e41f38b69f5ae99c2a9b5fa652773dde07df41f4467572a6a2bc8de93ed7.
  • Depth TSV: 825dd615bcba2d4e841e46e775a7b4a9ab75d146ddfb15f8b10a1956e5d20012.
  • Depth + mates TSV: 24556d4290684b96765e16bb8b0750f6cd8c1766438475d910224ce07fd61060.

Deferred Seqair work

No Seqair fork changes or upstream submissions are included. Follow-up candidates:

  1. Correctly expand BAM CG long CIGARs in RecordStore, not only raw records; investigate operation-count width for other formats too.
  2. Allow preparation for sequential pileup without automatic primary-mate linking and random-access reach/window indexes when callers do not need them.
  3. Evaluate compact counting entries and active-read state through a public opt-in API, retaining the general API's metadata and correctness guarantees.
  4. Evaluate direct common-match contributions and skipping unused CIGAR summary work separately. Upstream already supports optional auxiliary retention, so use that API rather than reimplement it.

The experiment narrative contains the historical checkpoints. These library follow-ups are deliberately separate from this application PR.

AI disclosure

Prepared with Amp, including code and tests. This is a draft for Seth's full human review before merge, as required by CONTRIBUTING.md. No merge or release is requested by this PR.

Reuse ordered output buffers and read-name groups, incorporate safe mate
and formatting fast paths, and stream only-depth from reusable records.
Use published Seqair 0.10.1 without the experimental fork or unchecked
vector operations. Preserve depth admission and reject unsupported BAM
long-CIGAR pileups explicitly.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1ca76-7f80-72d6-ac92-b1b4e5bacdcf
Co-authored-by: Amp <amp@ampcode.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant