Skip to content

Native single-sample correctness: multi-sample VariantCaller + aux-field parsing fix#7

Open
nh13 wants to merge 2 commits into
IPNP-BIPN:apple-silicon-native-v2-prfrom
nh13:nh_dv-6-caller-aux
Open

Native single-sample correctness: multi-sample VariantCaller + aux-field parsing fix#7
nh13 wants to merge 2 commits into
IPNP-BIPN:apple-silicon-native-v2-prfrom
nh13:nh_dv-6-caller-aux

Conversation

@nh13

@nh13 nh13 commented Jun 24, 2026

Copy link
Copy Markdown

Stacked on #5 — review after it merges. Two general correctness fixes to the native single-sample make_examples path, found while validating methylation but valuable on their own (they affect all single-sample WGS/WES runs). Apple-only; both realign the port to upstream — nothing ports upstream. Default behavior change is limited to a small, concordance-validated accuracy improvement (item 2).

1. --parse_sam_aux_fields was a no-op

It was set on MakeExamplesOptions but never on the SamReaderOptions used to open the BAM, and sam_reader.cc's ParseAuxFields does nothing unless aux_field_handling == PARSE_ALL_AUX_FIELDS. So no aux fields (MM/ML base modifications, HP, …) were ever parsed into the Read proto — read.base_modifications() was always empty, and the BAM HP tags the phasing fallback reads were never populated. Now wired (gated on the flag; default off = byte-identical).

2. Single-sample path used the wrong VariantCaller

The single-sample path constructed vcf_candidate_importer::VariantCaller (variant_calling.cc). Upstream always routes normal calling through the multi-sample VerySensitiveCaller (variant_calling_multisample.cc) for every sample count — vcf_candidate_importer is reserved for the special --variant_caller=vcf_candidate_importer import mode. Switched the single-sample path to multi_sample::VariantCaller (matching the trio path and upstream), calling CallsFromAlleleCounts with a single-element {sample_name -> AlleleCounter*} map and role sample.

This aligns the single-sample candidate set with upstream's and is a net accuracy improvement.

Validation

  • 14/14 ctest green.
  • Full chr20 HG002 WGS (b37) vs GIAB v4.2.1: F1 0.9974 → 0.9975, recall 0.9955 → 0.9957, precision 0.9993 unchanged (57 FP). The broader (upstream-matching) candidate set recovers a few true positives the importer caller missed, with no precision cost.

Why this PR exists

These two fixes are prerequisites for native methylation-aware phasing (PR #6, stacked on this one): without them, read.base_modifications() is empty and the single-sample path never emits methylated reference sites, so the 5mC vote can never engage. But the caller realignment changes all single-sample runs (slightly better accuracy, more upstream-faithful), so it is split out here for review on its own merits rather than buried in an opt-in feature PR.

@BenjaminDEMAILLE

Copy link
Copy Markdown

Heads-up @nh13: I had to squash apple-silicon-native-v2-pr from 273 commits down to 1, because Google's CLA bot can't auto-scan PRs with more than 250 commits and this branch is the head of the upstream PR google#1085. The squash is content-identical (same tree hash 810ff855), only the history was flattened.

Side effect: this rewrote the base of your 4 open PRs (#3, #4, #6, #7), so they now show an inflated diff against r1.10. A rebase onto the new base fixes it cleanly (no real conflicts, the content is unchanged). The old base tip was 0cd41f859de445f5d8a60524b86b54bb418a043b, so:

git fetch origin
git rebase --onto origin/apple-silicon-native-v2-pr 0cd41f859de445f5d8a60524b86b54bb418a043b <your-branch>
git push --force-with-lease

Sorry for the churn, and thanks again for all the fixes. Your authorship is preserved via a Co-Authored-By trailer on the squashed commit.

nh13 added 2 commits June 24, 2026 11:15
--parse_sam_aux_fields was set on MakeExamplesOptions but never on the
SamReaderOptions used to open the BAM, and sam_reader.cc's ParseAuxFields is a
no-op unless aux_field_handling == PARSE_ALL_AUX_FIELDS. So the flag did
nothing: no aux fields (MM/ML base modifications, HP, ...) were ever parsed
into the Read proto. Set aux_field_handling = PARSE_ALL_AUX_FIELDS on sam_opts
when the flag is on; default off keeps the byte-identical baseline (WGS chr20
unchanged at 182 examples; 14/14 ctest green).

Found while validating methylation-aware phasing on real PacBio HiFi 5mC data:
without this, read.base_modifications() is always empty so the 5mC path can
never engage. (It also gates BAM HP-tag parsing used by the phasing fallback.)
The single-sample path used vcf_candidate_importer::VariantCaller
(variant_calling.cc), which does not emit methylated reference sites
(alt='.'); only multi_sample::VariantCaller (variant_calling_multisample.cc)
does. Upstream always routes through the multi-sample VerySensitiveCaller for
every sample count — vcf_candidate_importer is reserved for the special
--variant_caller=vcf_candidate_importer import mode. So methylation-aware
phasing could never get methylated_ref_sites in the single-sample path.

Switch the single-sample path to multi_sample::VariantCaller (matching the
trio path and upstream), calling CallsFromAlleleCounts with a single-element
{sample_name -> AlleleCounter*} map and role 'sample'. This is the change that
makes methylation-aware phasing actually engage.

Validated on the chr20 HG002 WGS fixture (full chr20, b37, vs GIAB v4.2.1):
F1 0.9974 -> 0.9975, recall 0.9955 -> 0.9957, precision 0.9993 unchanged
(57 FP) — the broader candidate set recovers a few true positives the
narrower importer caller missed, with no precision cost, i.e. it is also more
upstream-faithful. On real PacBio HiFi 5mC data the methylated-ref-site path
now engages (6 methylated ref sites/region; PerformMethylationAwarePhasing
runs). 14/14 ctest green. Apple-only; nothing ports upstream.
@nh13 nh13 force-pushed the nh_dv-6-caller-aux branch from 24bdb8c to adc7229 Compare June 24, 2026 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants