Veritas

Veritas is a command-line tool for benchmarking pathogen genomic data generated by strategies guided by reference mapping. It supports comparing variants from VCF or FASTA (consensus) files against truth datasets and provides comprehensive performance metrics.

Table of Contents

• Disclaimer
• Features
• Requirements
• Installation
• Commands
  • List Available Datasets
  • Download Dataset
  • Validate Variants
  • Convert FASTA to VCF
  • Format Reference for RTG Tools
• Command Options
• Output Files
• Metrics Calculation
  • Variant Classification
  • Formulas
  • BED Region Metrics
• Example Workflows
• Development Information
  • Project Structure
  • Formatting with black
  • Running Tests
• Planned Functionalities
• Contributing
• Citation
• Authors

Disclaimer

While Veritas allows benchmarking using a FASTA (consensus) as input query file, we strongly recommend using VCF files when possible, for two important reasons:

1. Preserves original variant calls: The VCF file contains the exact variant calls produced by your original bioinformatics workflow.
2. Avoids re-calling artifacts: When a FASTA file is provided as query, Veritas re-aligns it using GSAlign to generate a VCF. This VCF may differ from the original variant calls used to create the consensus sequence.

Gold-standard benchmark datasets are provided in the veritas_data repository, but Veritas is designed to be dataset-agnostic and can work with any user-provided files.

Features

• Dataset Management

  • List available benchmark datasets from veritas_data repository
  • Download datasets (supports both real SRA data and artificial Zenodo datasets)
  • Automatic metadata handling

• Variant Validation

  • Compare query VCF against truth VCF using RTG vcfeval
  • Support for FASTA consensus validation via GSAlign alignment
  • Automatic sample name matching between query and truth datasets. It is a requirement for the rtg vcfeval tool to work.

• Comprehensive Metrics

  • Calculate precision, recall, and F1-score for SNVs and INDELs
  • Separate variant categories (TP/FP/FN) for different genomic regions (primers, masked regions, low coverage)
  • Export results in TSV format

• Annotation Support

  • Annotate variants with BED regions (primers, mask, low coverage)
  • Track variants in specific genomic contexts
  • Detailed variant classification (TP/FP/FN)

• Utility Commands

  • Convert FASTA to VCF using GSAlign
  • Format reference genomes for RTG Tools

Requirements

• Python >= 3.8
• RTG Tools (for vcfeval and reference formatting)
• GSAlign (for FASTA-to-VCF conversion)
• BCFtools (for VCF manipulation)
• parallel-fastq-dump (for SRA downloads, optional)
• seqkit (for FASTQ processing, optional)
• wget (for Zenodo downloads, optional)

Installation

# Clone the repository
git clone https://github.com/WallauBioinfo/veritas.git
cd veritas
micromamba env create -f env.yaml # or conda another environment manager
micromamba activate veritas
veritas --help

Commands

1. List Available Datasets

View all benchmark datasets available in the veritas_data repository with detailed metadata.

veritas list-datasets

Output example:

┌─────────────────────────────────┬────────────────┬──────────────┬────────────────────────┬─────────┬──────────────┬──────────────┬─────────────┐
│ dataset                         │ dataset_version│ pathogen     │ sequencing_technology  │ layout  │ dataset_type │ strategy     │ read_length │
├─────────────────────────────────┼────────────────┼──────────────┼────────────────────────┼─────────┼──────────────┼──────────────┼─────────────┤
│ veritas/sars-cov-2/SEARCH-8113  │ v1.0.0         │ SARS-CoV-2   │ Illumina               │ paired  │ real         │ amplicon     │ 150         │
├─────────────────────────────────┼────────────────┼──────────────┼────────────────────────┼─────────┼──────────────┼──────────────┼─────────────┤
│ veritas/sars-cov-2/MSv3-Coinfec │ v1.0.0         │ SARS-CoV-2   │ Illumina               │ paired  │ artificial   │ amplicon     │ 250         │
└─────────────────────────────────┴────────────────┴──────────────┴────────────────────────┴─────────┴──────────────┴──────────────┴─────────────┘

Dataset Types:

• real: Real sequencing data downloaded from SRA (requires parallel-fastq-dump and seqkit)
• artificial: Simulated data downloaded from Zenodo (requires wget)

2. Download Dataset

Download a specific benchmark dataset including all necessary files for validation.

veritas get-dataset \
  --dataset-name veritas/sars-cov-2/SEARCH-8113 \
  --output-dir datasets

What gets downloaded:

• Reference genome (reference.fa)
• Truth VCF (validated variant calls)
• Metadata (metadata.yaml) with dataset information
• FASTQ files:
  • Real datasets: Downloaded from SRA and paired-end filtered
  • Artificial datasets: Downloaded from Zenodo
• Optional files: primers.bed, mask.bed, low_cov.bed (if available)

Options:

Option	Short	Description	Default
--dataset-name	-d	Dataset name (e.g., veritas/sars-cov-2/SEARCH-8113)	Required
--output-dir	-o	Directory to save dataset	veritas_datasets

Example:

# Download to default directory
veritas get-dataset -d veritas/sars-cov-2/SEARCH-8113

# Download to custom directory
veritas get-dataset \
  -d veritas/sars-cov-2/SEARCH-8113 \
  -o my_datasets

Once downloaded, the reference fasta file and fastq files can be used as input in your workflow. The primers.bed or primers.fasta can be used if your workflow has features to trim primer regions based on bed or fasta files. If a dataset has a mask.bed file, it can be used to mask regions of the reference genome that are not informative for variant calling or that can represents sequencing artifacts.

Your workflow will produce a consensus fasta file - or in a better scenario, a VCF file - that can be used as input in the validate command.

3. Validate Variants

The validate command compares query variants against a truth VCF and calculates performance metrics.

Option A: Using Dataset Directory (Recommended)

veritas validate \
  --dataset-dir datasets/veritas/sars-cov-2/SEARCH-8113 \
  --query-vcf query.vcf.gz \
  --output-dir validation_results

The --dataset-dir option automatically discovers:

• Truth VCF (*.vcf.gz)
• Reference FASTA (reference.fa)
• RTG reference SDF (rtg_sdf/)
• Primers BED (primers.bed)
• Mask BED (mask.bed)
• Low coverage BED (low_cov.bed)

Option B: Validate with FASTA and Dataset Directory

veritas validate \
  --dataset-dir datasets/veritas/sars-cov-2/SEARCH-8113 \
  --query-fasta consensus.fa \
  --output-dir validation_results

Option C: Manual File Specification

You can still specify files individually if needed:

veritas validate \
  --truth-vcf truth.vcf.gz \
  --query-vcf query.vcf.gz \
  --rtg-reference reference_sdf \
  --output-dir validation_results \
  --primerd-bed primers.bed \
  --low-cov-truth-bed low_cov.bed

When using --query-fasta, veritas will:

1. Run GSAlign to align the consensus against the reference
2. Convert the alignment to VCF format with the same sample name as the truth VCF
3. Validate the VCF against the truth dataset

4. Convert FASTA to VCF

The convert command allows you to convert a FASTA consensus sequence to VCF format using GSAlign, without running validation. It is useful when you want to create an artificial dataset based on a consensus genome that differs from a reference.

veritas convert \
  --reference reference.fa \
  --query consensus.fa \
  --sample MySample \
  --output consensus.vcf.gz

Options:

Option	Short	Description	Required
--reference	-r	Reference FASTA file	Yes
--query	-q	Query FASTA file	Yes
--sample	-s	Sample name for VCF header	No (uses FASTA header or "SAMPLE")
--output	-o	Output VCF file path	No (uses query filename)

Example:

# Convert with automatic sample name from FASTA header
veritas convert -r NC_045512.2.fa -q consensus.fa

# Convert with custom sample name and output path
veritas convert \
  -r NC_045512.2.fa \
  -q my_sample.fa \
  -s MySample \
  -o results/my_sample.vcf.gz

FASTA-to-VCF Conversion Pipeline

When a FASTA consensus is provided (via convert or validate --query-fasta), Veritas performs the following steps:

1. Pairwise alignment: GSAlign aligns the consensus FASTA against the reference genome and produces a raw VCF.
2. Header normalisation: Veritas patches the raw GSAlign VCF to add a standard FILTER=<PASS> definition and FORMAT=<GT> field, which are required by RTG Tools.
3. Sample column injection: The sample name (taken from the truth VCF or provided via --sample) is written into the VCF column header and each record.
4. Compression & indexing: The processed VCF is bgzipped and tabix-indexed, producing the final .vcf.gz file used in downstream validation.

Treatment of N and gap characters

Important

Understanding how ambiguous bases and gaps are handled is essential when using a FASTA consensus as input instead of a VCF.

Character in consensus	How GSAlign treats it	Effect on metrics
Point substitution (e.g., A→G)	Called as an SNV	Counted in TP/FP/FN for the SNV category
Insertion (extra bases relative to reference)	Called as an INDEL	Counted in TP/FP/FN for the INDEL category
Deletion (- in alignment)	Called as an INDEL	Counted in TP/FP/FN for the INDEL category
Ambiguous base N	Not called as a variant	Positions masked as N are treated as missing data — they generate no FP, but any truth variant at that position becomes an FN

Because N positions are silently skipped rather than called as mismatches, a consensus with large masked regions can appear to have a low FP rate while quietly accumulating FNs. For this reason, we strongly recommend using the original VCF from your workflow whenever possible (see Disclaimer).

5. Format Reference for RTG Tools

The rtg-format command formats a reference FASTA file as an RTG SDF directory, required by RTG vcfeval. It is useful when you want to use a reference genome that is not provided in the veritas_data repository.

veritas rtg-format \
  --reference reference.fa \
  --output reference_sdf

Options:

Option	Short	Description	Required
--reference	-r	Reference FASTA file	Yes
--output	-o	Output SDF directory name	Yes

Example:

# Format reference for a dataset
veritas rtg-format \
  -r datasets/sars-cov-2/SEARCH-8113/reference.fa \
  -o datasets/sars-cov-2/SEARCH-8113/rtg_sdf

# Or use rtg directly
rtg format -o reference_sdf reference.fa

Command Options

validate

Option	Short	Description	Required
--dataset-dir	-d	Dataset directory (auto-discovers files)	No (but recommended)
--truth-vcf	-t	Truth VCF file (gzipped)	Yes (unless using --dataset-dir)
--query-vcf	-q	Query VCF file (gzipped)	Either this or --query-fasta
--query-fasta	-f	Query FASTA consensus file	Either this or --query-vcf
--reference	-r	Reference FASTA	Conditional (unless using --dataset-dir)
--rtg-reference		RTG SDF reference directory	Yes (unless using --dataset-dir)
--output-dir	-o	Output directory	No (default: veritas_output)
--primerd-bed	-p	BED file with primer regions	No (auto-discovered from --dataset-dir)
--mask-bed	-m	BED file with masked regions	No (auto-discovered from --dataset-dir)
--low-cov-truth-bed	-lt	BED file with low coverage in truth	No (auto-discovered from --dataset-dir)
--low-cov-query-bed	-lq	BED file with low coverage in query	No

Output Files

After running validate, you'll find the following files in the output directory:

output_dir/
├── gsalign/                    # GSAlign outputs (if using FASTA)
│   ├── alignment.fa           # Aligned sequences
│   └── query.vcf.gz          # Processed query VCF
│       └── query.vcf.gz.tbi  # Index
├── rtg/                       # RTG vcfeval working directory
│   ├── tp.vcf.gz             # True positives
│   ├── fp.vcf.gz             # False positives
│   ├── fn.vcf.gz             # False negatives
│   └── summary.txt           # RTG summary
├── truth_query.vcf.gz        # Final annotated VCF with all variants
├── truth_query.vcf.gz.tbi    # Index
└── metrics.tsv               # Performance metrics

Metrics File Format

The metrics.tsv file contains precision, recall, and F1-score for SNVs and INDELs:

Category    Region              TP  FP  FN  Precision   Recall      F1-Score
SNV         ALL                 95  5   2   0.950       0.979       0.964
INDEL       ALL                 18  2   3   0.900       0.857       0.878
SNV         PRIMER BED          10  1   0 
INDEL       PRIMER BED          2   0   1
SNV         LOW_COV_TRUTH BED   3   2   1
...

Region Types:

• ALL: All variants (no filtering)
• PRIMER BED: Variants within primer regions (if --primerd-bed provided)
• MASK BED: Variants within masked regions (if --mask-bed provided)
• LOW_COV_TRUTH BED: Variants in low coverage regions of truth (if --low-cov-truth-bed provided)
• LOW_COV_QUERY BED: Variants in low coverage regions of query (if --low-cov-query-bed provided)

Metrics Calculation

Variant Classification

Each variant in the merged output VCF is tagged using RTG vcfeval results:

Tag	Meaning
TP	True Positive — variant present in both truth and query
FP	False Positive — variant present only in query
FN	False Negative — variant present only in truth

Variants are independently classified as SNV or INDEL:

• SNV (Single Nucleotide Variant): len(REF) == len(ALT) — simple substitutions
• INDEL (Insertion or Deletion): len(REF) != len(ALT) — any length-changing variant

SNV and INDEL metrics are always reported separately, so a pipeline that calls substitutions well but misses indels (or vice versa) is visible in the output.

Formulas

For each variant category (SNV / INDEL) and region, Veritas computes:

$$\text{Precision} = \frac{TP}{TP + FP}$$

$$\text{Recall} = \frac{TP}{TP + FN}$$

$$\text{F1-Score} = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

All three metrics return 0 whenever the denominator is zero (e.g., no variants in that category).

BED Region Metrics

When BED files are provided, Veritas reports metrics both globally (ALL) and broken down by genomic context (primer regions, masked regions, low-coverage regions). A single variant can appear in multiple BED regions simultaneously. This allows you to:

• Identify whether FPs are concentrated in amplicon primer regions (a common artefact)
• Distinguish FNs in low-coverage areas from genuine pipeline failures
• Assess whether masked regions are inflating performance by excluding difficult sites

N bases and gaps (FASTA input only)

See FASTA-to-VCF Conversion Pipeline for a full table of how N, substitutions, insertions, and deletions are treated during conversion from a consensus FASTA to VCF before metric calculation.

Example Workflows

Workflow 1: Complete Benchmark with Dataset Download

# 1. List available datasets
veritas list-datasets

# 2. Download a benchmark dataset
veritas get-dataset \
  -d veritas/sars-cov-2/SEARCH-8113 \
  -o datasets

# 3. Run your workflow
<your_workflow> \
  --reference datasets/veritas/sars-cov-2/SEARCH-8113/reference.fa \
  --r1 datasets/veritas/sars-cov-2/SEARCH-8113/SEARCH-8113_R1.fastq.gz \
  --r2 datasets/veritas/sars-cov-2/SEARCH-8113/SEARCH-8113_R2.fastq.gz \
  --primers datasets/veritas/sars-cov-2/SEARCH-8113/primers.bed \
  --output output_directory/results.vcf.gz

# 4. Validate your VCF
veritas validate \
  -d datasets/veritas/sars-cov-2/SEARCH-8113 \
  -q output_directory/results.vcf.gz \
  -o validation_directory

# 5. Review results
cat validation_directory/metrics.tsv

Workflow 2: Validate FASTA Consensus

# Validate a FASTA consensus with a downloaded dataset
veritas validate \
  -d datasets/veritas/sars-cov-2/SEARCH-8113 \
  -f my_consensus.fa \
  -o validation_directory

Workflow 3: Custom Dataset (User-Provided Files)

# 1. Format your reference
veritas rtg-format -r my_reference.fa -o my_reference_sdf

# 2. Run validation with all custom files
veritas validate \
  -t my_truth.vcf.gz \
  -q my_query.vcf.gz \
  -r my_reference.fa \
  --rtg-reference my_reference_sdf \
  -p primers.bed \
  -m mask.bed \
  -lt low_cov_truth.bed \
  -o my_results

Development Information

Project Structure

veritas/
├── veritas/
│   ├── commands/
│   │   └── main.py          # CLI commands
│   ├── __init__.py
│   ├── datasets.py          # Dataset management
│   ├── logger.py            # Logging configuration
│   ├── tools.py             # Tools commands
│   └── validate.py          # Validation logic
├── tests/
│   ├── conftest.py          # Test fixtures
│   ├── test_validate.py     # Unit tests
│   └── fixtures/            # Test data
├── setup.py
└── README.md

Formatting with black

Before commiting, please format the code with black.

black .

Running Tests

Veritas includes a comprehensive test suite using pytest.

Quick Start

# Run all tests
pytest tests/ -v

# Run specific test file
pytest tests/test_validate.py -v

# Run specific test class
pytest tests/test_metrics.py::TestCalcMetrics -v

Test Structure

tests/
├── conftest.py           # Pytest fixtures (temp dirs, sample files, etc.)
├── test_validate.py      # Tests for validation functions (metrics, BED parsing)
├── test_datasets.py      # Tests for dataset management (table formatting)
├── test_commands.py      # Tests for CLI commands (help, argument validation)
├── test_metrics.py       # Tests for metrics calculation and TSV export
├── fixtures/             # Test data files
└── README.md            # Test documentation

Notes

• Some tests create temporary files automatically cleaned up after execution
• Tests requiring external tools (RTG, GSAlign) are skipped if tools not available
• All fixtures are defined in conftest.py
• See tests/README.md for detailed testing documentation
• This suite of tests were generated using Gemini 3 Pro

Planned Functionalities

• Download datasets based on metadata, for example: all datasets for SARS-CoV-2, or all datasets generated using Illumina amplicon sequencing.
• Support the analysis and validation of structural variants (SVs), which is particularly important in the context of Mpox data.
• Generate genomic plots that highlight variants and their classifications (true positives, false positives, and false negatives).

Contributing

Contributions are welcome! Please:

1. Create a feature branch
2. Add tests for new functionality
3. Ensure all tests pass
4. Submit a pull request

Citation

If you use Veritas in your research, please cite:

• RTG vcfeval: https://github.com/rtg-wasm/vcfeval
• GSAlign: https://github.com/GenomicsSoftwares/GSAlign
• bcfTools: https://github.com/samtools/bcftools
• seqkit: https://github.com/shenwei356/seqkit
• parallel-fastq-dump: https://github.com/ncbi/parallel-fastq-dump
• veritas: https://github.com/WallauBioinfo/veritas

Authors

Filipe Dezordi

• GitHub: @dezordi