[theontho]

Fixes the bugs on our side that #167's research found, and adds corrected and current chip templates.

Fixes

Bug	Before	After (whole-genome hg38 CRAM, --parallel)
FTDNA_V1_Affy templates are NCBI36. The target skipped them and the converter matched by rsID.	49.5% no-calls	0.1% no-calls; 99.93% agree with a real 23andMe chip on 81,918 shared rsIDs
The hg38 target lifted chrM through UCSC hg19's Yoruba chrM, but templates and hg38 are both rCRS.	23andMe V5 MT 70.5% no-calls	0.0%; 99.98% agree with the real chip
102 MT markers in the reflib 23andMe V4/V5/V35 templates sit at Yoruba positions.	always no-call	corrected via corrections.tsv
Vendor chromosome XY (MyHeritage PAR) wasn't mapped.	no-call	mapped to X
23andMe writes haploid calls as one letter (MT, Y, and a man's non-PAR X); we doubled them.	AA	A. X exact-string match with the real chip is 99.3%.
Heterozygous calls were in REF/ALT order; chips use alphabetical order.	TC	CT
CRLF head + LF body produced mixed line endings.	mixed	body follows the head
The help said --formats all writes every vendor file.		It writes only the combined kit; docs fixed.

The genome-wide target now has a version line (##wgsextract_microarray_target_version=2) and is rebuilt automatically when stale. It grew from 1.80M to 2.39M markers (FTDNA_V1 lifted, plus the new chips).

Trade-off: upstream's reflib ships microarray/All_SNPs_hg38_ref.tab.gz at the same path with no version line. A library that has it gets it rebuilt by our builder. A target passed with --ref-vcf-tab is never touched.

New templates (IDs + positions only, no genotypes)

• 23andme_v5_2025 (643,161 markers), ancestry_v2_2024 (677,436) and nucleus (665,107), from 2024–25 exports. A vendor's exports for different people have identical marker lists.
• scripts/microarray_template_from_raw.py is generalized across vendors. The new scripts/microarray_template_corrections.py regenerates corrections.tsv.

Validation

• pytest: 1,922 passed. New tests cover:

  • target versioning and rebuild
  • chrM identity on hg38
  • NCBI36 lift and conversion by position
  • XY mapping
  • allele sort
  • haploid single letters for male and female kits, including the PAR
  • head-driven line endings
  • corrections
  • the new format keys

• ruff and mypy are clean.

• A real-data chrM+chrY run, then a whole-genome hg38 run (18.5 min). Every vendor file has 0.0–0.3% no-calls overall and uniform line endings.

• Agreement with the real chips (same alleles) by chromosome:

  Chip	chr1	X	Y	MT
  23andMe 2025	99.67%	99.30%	98.57%	99.98%
  Ancestry 2024	99.84%	99.87%	99.15%	93.0%
  MyHeritage	99.52%	98.57%	96.66%	–
  Nucleus	99.80%	99.95%	100%	–

  Ancestry MT disagreements are mostly strand-complemented chip calls, which are on Ancestry's side.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

• Source mode: Auto
• Repository: theontho/wgsextract-cli
• Branch: fix/microarray-167-kit-fixes
• File: .coderabbit.yaml
• Commit: 9f8560e
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Advanced
• Run ID: 60d08fb8-fe6e-4fce-a1e3-3a9e224d30e7

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

🟡 Changes recommended

Remaining defects make CSV vendor templates undiscoverable, prevent reuse of current targets in read-only libraries, and produce incorrect alleles for reverse-strand hg18-to-hg19 mappings.

3 open issues

• Use vendor-specific suffixes for generated template paths · New
• Check current targets before creating build locks · New
• Preserve strand and complement genotypes during coordinate conversion · New

What changed in this PR

Improves microarray kit generation following #167’s research, correcting coordinate handling and vendor formatting while adding support for recent chip exports.

Changes:

• Corrects mitochondrial and NCBI36 marker handling, adds target versioning, and rebuilds stale library targets.
• Adds current 23andMe, AncestryDNA, and Nucleus formats, with updated haploid calls, allele ordering, and line endings.
• Expands regression tests, template-generation tools, and usage documentation.

File	Description
tests/​test_sample_ploidy.py	Updates the reference-library fixture.
tests/​test_microarray_vendor_formats.py	Tests haploid output, corrections, line endings, and NCBI36 conversion.
tests/​test_microarray_targets.py	Tests lifted markers, mitochondrial identity, and target versioning.
tests/​test_microarray_simulation.py	Updates expected vendor-specific haploid calls.
tests/​test_microarray_pr43_regressions.py	Tests new format aliases and alphabetical allele ordering.
tests/​test_lineage_microarray_cleanup.py	Updates the reference-library fixture.
src/​wgsextract_cli/​core/​microarray_utils.py	Implements template corrections, NCBI36 lookup, and vendor formatting changes.
src/​wgsextract_cli/​core/​microarray_targets.py	Adds versioned targets, new formats, and coordinate corrections.
src/​wgsextract_cli/​core/​messages.py	Clarifies --formats all and lists new formats.
src/​wgsextract_cli/​commands/​microarray.py	Connects new formats, target rebuilding, and NCBI36 conversion.
src/​wgsextract_cli/​commands/​_microarray_combined.py	Sorts heterozygous alleles alphabetically.
src/​wgsextract_cli/​assets/​microarray/​raw_file_templates/​README.md	Documents template sources, corrections, and regeneration.
src/​wgsextract_cli/​assets/​microarray/​raw_file_templates/​head/​Ancestry_V2_2024.txt	Adds the AncestryDNA export header.
src/​wgsextract_cli/​assets/​microarray/​raw_file_templates/​head/​23andMe_V5_2025.txt	Adds the 23andMe export header.
src/​wgsextract_cli/​assets/​microarray/​raw_file_templates/​corrections.tsv	Supplies mitochondrial marker-position corrections.
site_src/​workflows.md	Updates format examples and output behavior documentation.
scripts/​microarray_template_from_raw.py	Generalizes template extraction across vendor exports.
scripts/​microarray_template_corrections.py	Adds mitochondrial correction-table regeneration.

_{Review effort: Balanced}