diff --git a/README.md b/README.md index 3fde4c68..49f6d86d 100644 --- a/README.md +++ b/README.md @@ -293,7 +293,12 @@ not exclude samples from this analysis. with a library-size offset. The effect is log2 fold change; positive values indicate greater abundance in the second condition. `offset` selects `tss` (default) or `rle`; `dispersion` selects maximum-likelihood theta or a - cross-taxon `glmGamPoi` trend. + cross-taxon `glmGamPoi` trend. Both options Wald-test the group coefficient + with the dispersion treated as known, so every `ml` result — and a + `glmGamPoi` result whose smaller group has at most 12 samples — carries a + calibration caveat in its diagnostics, shown with the result and written + into its CSV export. The caveat changes no p-value; the null-simulation + evidence and full replication protocol live in `bench/null_calibration`. - `zinb`: a zero-inflated negative-binomial model with separate logistic zero and count components, a count-component library-size offset, a joint 2-df likelihood-ratio test, and BH correction. diff --git a/bench/null_calibration/.gitignore b/bench/null_calibration/.gitignore new file mode 100644 index 00000000..088565ad --- /dev/null +++ b/bench/null_calibration/.gitignore @@ -0,0 +1,4 @@ +# Raw per-taxon p-values are large, append-only working state. The seeds in +# calibration.R regenerate them dataset-for-dataset; committed evidence is +# the summarised CSVs and REPORT.md under results/. +results/raw/ diff --git a/bench/null_calibration/README.md b/bench/null_calibration/README.md new file mode 100644 index 00000000..f9598d2a --- /dev/null +++ b/bench/null_calibration/README.md @@ -0,0 +1,124 @@ +# Null-calibration bench for `nb_glm` (issue #193) + +Issue #193 measured that the default `nb_glm` path — `MASS::glm.nb` per taxon +with a fixed-scale Wald test, `analysis.differential.dispersion: ml` — rejects +true nulls too often in small studies (nominal 0.05 rejecting at 0.132 with 4 +samples per group and 0.069 with 12). The mechanism is textbook: the Wald test +treats the per-taxon maximum-likelihood theta as known, ignoring its sampling +uncertainty, which is large when groups are small. + +This bench replicates that finding at the scale #193 asked for, measures the +candidate remedies on identical datasets, and records what was actually run. +It is a bench, not a unit test: slow, stochastic, never run by CI. + +## What is compared + +Every simulated dataset is analysed by every enabled test (paired design — no +test gets an easier dataset): + +| test | what it is | status | +|---|---|---| +| `wald_ml` | the production default, through the real `_FIT_R` block extracted from `src/analysis/differential.jl` (`da_use_gp = FALSE`) | production | +| `wald_trend` | the production `dispersion: glmGamPoi` option on this branch, through the same block (`da_use_gp = TRUE`): dispersion trend plugged into a fixed-theta Wald fit. **Not** a quasi-likelihood test | production option | +| `lrt` | `MASS::glm.nb` likelihood-ratio test, full vs intercept-only, each fit estimating its own theta; `2 × ΔlogLik` against χ²(1) | bench-only candidate | +| `glmgampoi_ql` | glmGamPoi's own quasi-likelihood F-test (`glm_gp` + `test_de`, numeric contrast weights) | reference (the calibrated path in #193) | +| `edger_ql` | edgeR's quasi-likelihood F-test, fed the same offsets | outside reference only, not a dependency | + +A test whose package is unavailable is skipped loudly (stdout + provenance +file), never silently replaced. + +## Protocol (mapped to #193's checklist) + +- **Group sizes** `n ∈ {3, 4, 6, 8, 12, 20}` per group, equal groups. +- **Summaries**: rejection rates at α = 0.01 / 0.05 / 0.10 with Wilson 95% + intervals, Kolmogorov–Smirnov D and p of the null p-values against + Uniform(0, 1), and per-test fit-failure shares (`summarise.R`). +- **Data shapes** (`--shapes`): + - `original` — the #193 generator: taxon means exp(U(log 2, log 500)), + dispersion `0.05 + 2/√μ`, library sizes lognormal(0, 0.3); + - `constant` — same means, dispersion 0.5 for every taxon; + - `sparse` — amplicon-like: means exp(N(log 3, 1.6)), so most taxa are rare + and a few dominate; many zeros; + - `unequal_lib` — `original` with group B library sizes ×2.5. +- **Offsets** (`--offsets`): `tss` and `rle`, implemented to mirror + `tss_factors` / `rle_factors` in `differential.jl`. An RLE refusal (no taxon + present in every sample) is counted, never replaced by TSS. +- **Power** (`--truth=power --log2fc=L`): 25% of taxa truly change by ±L (log2), + the rest are null. Reports raw and BH power on the changed taxa, the BH + false discovery rate at 0.05 (averaged per dataset, BH applied within each + dataset over its tested family, as production does), and null-taxon + calibration within the same cells. +- **Taxa filter**: the #193 script's filter is kept for fidelity — a taxon + needs reads in ≥2 samples and non-constant counts. Taxon parameters are + redrawn for every dataset, as in #193. Datasets with <5 testable taxa are + skipped and counted. +- **Seeds**: `seed0 + cell_index × 1000003 + sim`, one per dataset, so cells + can run in any order or in parallel processes and reproduce exactly. + Default `seed0 = 20261020` (the #193 seed). + +### Reading a cell + +A test is **flagged anti-conservative** in a cell when the lower bound of the +Wilson 95% interval for its rejection rate at α = 0.05 exceeds 0.05, and +**flagged conservative** when the upper bound is below 0.05. The KS statistic +is supporting evidence, not the criterion — with ~10⁴ p-values it detects +departures too small to matter at any single α. + +## Running it + +Native R with the pinned library (the supported route; `Rscript -e +'renv::restore()'` first): + +```sh +# smoke: 10 sims, one cell, a few seconds +Rscript bench/null_calibration/calibration.R --mode=quick + +# one direct-replication cell +Rscript bench/null_calibration/calibration.R \ + --shapes=original --offsets=tss --n-per=4 --sims=200 --taxa=40 \ + --tests=wald_ml,lrt + +# the full #193 grid — 1,000 sims per cell; expect hours, use several +# processes by splitting --n-per / --shapes across terminals +Rscript bench/null_calibration/calibration.R --mode=full + +# aggregate whatever raw cells exist +Rscript bench/null_calibration/summarise.R +``` + +Outputs append to `results/raw/pvalues_.csv` (gitignored), one row per +taxon × test, so an interrupted run keeps finished simulations; each cell also +writes `provenance_.txt` with arguments, package versions and +`sessionInfo()`. `summarise.R` writes `results/calibration_summary.csv` (and +`power_summary.csv` when power cells exist). + +### No native R? The WebR fallback + +`webr_run.mjs` runs the same `calibration.R` / `summarise.R` under WebR (R in +WebAssembly) for environments with Node but no R. `glm.nb` is pure R (MASS +7.3-65's `negbin.R` makes no `.C`/`.Call`/`.Fortran` call), so sourcing that +one file provides genuine MASS code; `calibration.R` detects this mode, +strips the block's `MASS::` prefixes as a single mechanical substitution, and +records the fact in every provenance file. glmGamPoi and edgeR are compiled +Bioconductor packages and cannot run this way — their tests skip loudly. +Setup and usage are in the header of `webr_run.mjs`. Prefer native Rscript +whenever it exists. + +## What has actually been run + +`REPORT.md` records each committed run: cells, sizes, environment and results, +with the corresponding summary CSVs in `results/`. Anything not in `REPORT.md` +has not been run — in particular, a committed bench is not a result. + +## Limitations (read before quoting rates) + +- Taxa are simulated independently. Real amplicon tables are compositional and + correlated; independence makes BH behaviour here cleaner than in practice. +- Groups are equal-sized. Unequal groups change both theta estimation and + offset behaviour; #193's grid did not cross that factor either. +- The ≥2-sample presence filter mirrors #193, not production's default + `min_prevalence = 0` (under which untestable taxa instead surface as failed + rows outside the BH family). Failure shares are reported so the tested + family is never a hidden denominator. +- `original`-shape library sizes are mild (lognormal σ = 0.3). Harsher depth + variation is covered only through `unequal_lib`'s group shift. diff --git a/bench/null_calibration/REPORT.md b/bench/null_calibration/REPORT.md new file mode 100644 index 00000000..68c1f34a --- /dev/null +++ b/bench/null_calibration/REPORT.md @@ -0,0 +1,112 @@ +# Null-calibration bench report (issue #193) + +This report records the one reduced run that has actually been executed. +Anything not listed here has not been run. The committed summary CSVs in +`results/` (`calibration_summary.csv`, `power_summary.csv`) are the source of +truth for every number below; per-cell run records are summarised in +`results/provenance_summary.csv` (the raw p-value files under `results/raw/` +are gitignored and reproducible from the recorded seeds and arguments). + +## Run identity + +- **Date:** 2026-10-22 (UTC, per-cell provenance records). +- **Executor:** WebR 0.6.0 (R 4.6.0, wasm32) through `webr_run.mjs`, because the + sandbox used for this run had Node but no native R. `MASS::glm.nb` was + provided by sourcing MASS 7.3-65's pure-R `negbin.R` (the exact MASS version + pinned in `renv.lock`); in that mode `calibration.R` strips the `MASS::` + prefixes from the extracted production block as a single mechanical + substitution and records the fact in every provenance file. Native + `Rscript` runs use the extracted block byte-identically. The sandbox R + version (4.6.0) differs from the lockfile's R 4.5.0. +- **Production path under test:** the real `_FIT_R` block extracted from + `src/analysis/differential.jl` (`wald_ml`, `dispersion: ml`), paired on + identical datasets with a bench-only likelihood-ratio test (`lrt`: full vs + intercept-only `glm.nb`, each fit estimating its own theta, `2 × ΔlogLik` + against χ²(1)). +- **Not run in this sandbox:** `wald_trend`, `glmgampoi_ql`, `edger_ql` + (glmGamPoi/edgeR are compiled Bioconductor packages and were absent — the + provenance records say so). Their code paths in `calibration.R` are + unexecuted here and need a native R/Bioconductor run. +- **Cells:** direct replication (original/TSS null) at 3/4/6/8/12/20 samples + per group, 200 sims × 40 taxa; shape/offset variations (constant, sparse, + unequal library sizes, original/RLE) at 4 and 12 per group, 150 × 40; power + cells (original/TSS, 25% of taxa at |log2FC| = 1) at 4 and 12 per group, + 150 × 40. This is a *reduced* run: the full 1,000-sim grid in `README.md` + has not been run. + +## Results + +A cell is flagged anti-conservative when the lower bound of the Wilson 95% +interval for its α = 0.05 rejection rate exceeds 0.05. **All 32 summarised +α = 0.05 test-cells were flagged**, including 20 samples per group; the +closest cell was LRT at 20/group on original/TSS (lower bound 0.0534). + +### Direct replication (original shape, TSS offset, null) + +Rejection rates at α = 0.01 / 0.05 / 0.10. Issue #193 reported Wald at +4/group: 0.060 / 0.132 / 0.198 and 12/group: 0.019 / 0.069 / 0.124 — the +replication lands essentially on those values. + +| n/group | sims | `wald_ml` (production) | `lrt` (bench-only) | Wald fit failures | LRT fit failures | +|---:|---:|---|---|---:|---:| +| 3 | 200 | 0.092 / 0.179 / 0.251 | 0.046 / 0.131 / 0.203 | 6.41% | 8.43% | +| 4 | 200 | 0.063 / 0.138 / 0.204 | 0.030 / 0.103 / 0.169 | 2.68% | 3.61% | +| 6 | 200 | 0.045 / 0.111 / 0.171 | 0.023 / 0.087 / 0.149 | 0.79% | 1.20% | +| 8 | 200 | 0.029 / 0.089 / 0.145 | 0.017 / 0.070 / 0.129 | 0.20% | 0.33% | +| 12 | 200 | 0.025 / 0.076 / 0.130 | 0.017 / 0.065 / 0.119 | 0.01% | 0.04% | +| 20 | 200 | 0.019 / 0.065 / 0.116 | 0.014 / 0.058 / 0.110 | 0.00% | 0.00% | + +### Other shapes and the RLE offset (α = 0.05, Wilson 95% interval) + +| shape / offset | n/group | sims | `wald_ml` | `lrt` | Wald failures | LRT failures | +|---|---:|---:|---|---|---:|---:| +| `constant` / `tss` | 4 | 150 | 0.135 (0.126–0.144) | 0.104 (0.096–0.112) | 4.03% | 5.35% | +| `constant` / `tss` | 12 | 150 | 0.073 (0.066–0.080) | 0.065 (0.059–0.072) | 0.18% | 0.23% | +| `sparse` / `tss` | 4 | 150 | 0.107 (0.098–0.116) | 0.082 (0.074–0.090) | 15.12% | 18.36% | +| `sparse` / `tss` | 12 | 150 | 0.068 (0.062–0.075) | 0.060 (0.054–0.067) | 4.52% | 5.07% | +| `unequal_lib` / `tss` | 4 | 150 | 0.149 (0.140–0.158) | 0.114 (0.106–0.122) | 1.68% | 3.17% | +| `unequal_lib` / `tss` | 12 | 150 | 0.074 (0.068–0.081) | 0.062 (0.056–0.068) | 0.00% | 0.08% | +| `original` / `rle` | 4 | 150 | 0.138 (0.130–0.147) | 0.100 (0.093–0.108) | 2.57% | 3.37% | +| `original` / `rle` | 12 | 150 | 0.080 (0.074–0.087) | 0.068 (0.061–0.074) | 0.02% | 0.02% | + +The inflation is not specific to the original mean-varying dispersion shape, +and the RLE offset does not rescue calibration. + +### Power and BH behaviour (25% of taxa at |log2FC| = 1) + +BH is applied within each simulated dataset over its tested family, as +production does. “Null rate” is the α = 0.05 rejection rate among the truly +null taxa in the same cells. + +| n/group | test | sims | null rate @0.05 | raw power | BH power | BH FDR | mean discoveries | +|---:|---|---:|---:|---:|---:|---:|---:| +| 4 | `lrt` | 150 | 0.114 | 0.414 | 0.147 | 0.184 | 1.89 | +| 4 | `wald_ml` | 150 | 0.150 | 0.480 | 0.290 | 0.379 | 4.70 | +| 12 | `lrt` | 150 | 0.083 | 0.668 | 0.422 | 0.101 | 4.82 | +| 12 | `wald_ml` | 150 | 0.096 | 0.688 | 0.509 | 0.129 | 5.99 | + +## Interpretation + +- #193 replicates. The production fixed-scale Wald test on per-taxon ML theta + rejects true nulls well above nominal at every measured group size, + including 20/group (0.065 at α = 0.05). +- The LRT is consistently closer to nominal but remains anti-conservative in + every cell, and fails more often than Wald at tiny n (it needs two fits). + It is a mitigation, not a calibration fix. +- Wald's extra BH “power” is bought with uncontrolled FDR (0.38 at 4/group + against a nominal 0.05). Nothing measured in this sandbox “stays + calibrated”; the calibrated comparator from #193 (glmGamPoi’s + quasi-likelihood F-test) still needs the native run. +- Because even 20/group remained flagged, the production calibration caveat + (see `_calibration_caveat` in `src/analysis/differential.jl`) uses “caution” + severity through the largest measured size and only drops to “info” beyond + the measured range. That caveat is fork-side only; any change to upstream + defaults remains an upstream decision (see `UPSTREAM_DRAFT.md`). + +## Checklist mapping (#193) + +Done in this reduced run: direct replication at larger sim counts and more +group sizes; additional shapes; RLE offset at 4/12 per group; an additional +test (LRT) on identical datasets; power and BH-FDR at |log2FC| = 1. +Not done: the full 1,000-sim grid; `wald_trend`; glmGamPoi QL F; edgeR QL; +native Rscript/Bioconductor execution; power at other effect/group sizes. diff --git a/bench/null_calibration/UPSTREAM_DRAFT.md b/bench/null_calibration/UPSTREAM_DRAFT.md new file mode 100644 index 00000000..5de5b181 --- /dev/null +++ b/bench/null_calibration/UPSTREAM_DRAFT.md @@ -0,0 +1,51 @@ +# Draft upstream report: null calibration of the `glm.nb` Wald test + +*Status: draft prepared downstream for review before anything is filed +upstream. All numbers are from `REPORT.md` in this directory, which defines +exactly what ran, where, and what did not run.* + +## Summary + +In null simulations that drive the production negative-binomial fitting code +(`MASS::glm.nb` per taxon, fixed-scale Wald test, per-taxon ML theta treated +as known), true nulls are rejected more often than nominal at every measured +group size. At α = 0.05 the rejection rate was 0.138 with 4 samples per group +and 0.076 with 12 (original/TSS shape, 200 sims × 40 taxa), and still 0.065 +at 20 per group. A likelihood-ratio test on the same fits/datasets was closer +to nominal (0.103 / 0.065 / 0.058) but still anti-conservative in every cell +(Wilson 95% lower bound > 0.05 in all 32 summarised test-cells). Under a +power design (25% of taxa at |log2FC| = 1), Benjamini–Hochberg at 0.05 +delivered FDR 0.38 (Wald, 4/group) and 0.13 (12/group). + +The mechanism is the standard one: the Wald test plugs in the per-taxon ML +theta and ignores its sampling uncertainty. The bench that produces these +numbers extracts the production fitting block directly from +`src/analysis/differential.jl`, so the `wald_ml` column is the shipped code +path, not a reimplementation. + +## What was and was not measured + +Measured (reduced WebR run, genuine MASS 7.3-65 `glm.nb`; see `REPORT.md` for +provenance limits): Wald vs LRT, paired on identical datasets; group sizes +3–20; constant/sparse/unequal-library shapes; RLE offsets; one power design. + +Not yet measured: the glmGamPoi quasi-likelihood F-test as an in-bench +comparator on this branch (the calibrated path reported in the downstream +issue), the dispersion-trend Wald option, edgeR, and the full 1,000-sim grid. +The bench (`bench/null_calibration/`, native `Rscript` route) is ready to run +those where R + Bioconductor are available. + +## Options for the upstream maintainer's consideration + +1. **Result-notes caveat for small samples** (adopted downstream): record a + calibration caveat with `ml`-dispersion results, scaled by group size. + Changes no statistics or defaults. +2. **LRT instead of Wald for the `glm.nb` path:** measurably closer to + nominal in this bench, but still anti-conservative, slower, and fails + more often at tiny n. The bench quantifies the trade-off; it is not a + calibration fix. +3. **Prefer a quasi-likelihood path below a group-size threshold:** the + downstream measurements that motivated this bench found the QL F-test at + nominal rates where Wald was inflated. Any such default change is + upstream's call; this draft only supplies the replication bench and + numbers to decide with. diff --git a/bench/null_calibration/calibration.R b/bench/null_calibration/calibration.R new file mode 100644 index 00000000..a424dc4c --- /dev/null +++ b/bench/null_calibration/calibration.R @@ -0,0 +1,395 @@ +#!/usr/bin/env Rscript +# SPDX-License-Identifier: AGPL-3.0-only +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# +# Null-calibration and power bench for the nb_glm tests in +# src/analysis/differential.jl (fork issue #193). This is a bench, not a unit +# test: it is slow and stochastic, and it never runs in CI by default. +# +# What it measures, on the SAME simulated datasets (paired comparison): +# wald_ml - the production default: the real `_FIT_R` block extracted +# from differential.jl, da_use_gp = FALSE (MASS::glm.nb per +# taxon, Wald test at fixed scale 1). +# wald_trend - the production glmGamPoi option ON THIS BRANCH: the same +# `_FIT_R` block with da_use_gp = TRUE (glm_gp dispersion +# trend plugged into a fixed-theta Wald fit). This is NOT +# glmGamPoi's quasi-likelihood test. Needs glmGamPoi. +# lrt - MASS::glm.nb likelihood-ratio test, full (~ group) against +# intercept-only, each fit estimating its own theta; +# 2 * (logLik(full) - logLik(null)) against chisq(1). +# Bench-only candidate fix; production does not use it. +# glmgampoi_ql - glmGamPoi's own quasi-likelihood F-test (glm_gp + test_de), +# as on the upstream PR 62 branch. Needs glmGamPoi. +# edger_ql - edgeR's quasi-likelihood F-test as an outside reference +# only. Needs edgeR. Not a dependency of the app. +# +# Tests whose package is unavailable are skipped LOUDLY (a line on stdout and +# in the provenance file), never silently replaced by another test. +# +# Usage (from the repo root, with the pinned R library restored): +# Rscript bench/null_calibration/calibration.R --mode=quick +# Rscript bench/null_calibration/calibration.R --mode=full # hours; see README +# Rscript bench/null_calibration/calibration.R --truth=power --log2fc=1 \ +# --n-per=4,12 --sims=200 +# Any --key=value below overrides the mode defaults. Under WebR (no Rscript +# argv), settings may instead come from the NB_CALIB_ARGS environment +# variable as one space-separated string; see webr_run.mjs. +# +# Output: one appended CSV per cell under --out (default +# bench/null_calibration/results/raw), so an interrupted long run keeps the +# simulations it finished. summarise.R aggregates them. + +## ---------------------------------------------------------------- settings + +parse_args <- function(argv) { + out <- list() + for (a in argv) { + if (!grepl("^--[a-z0-9-]+=", a)) stop("unrecognised argument '", a, "' (want --key=value)") + kv <- strsplit(sub("^--", "", a), "=", fixed = TRUE)[[1]] + out[[kv[1]]] <- paste(kv[-1], collapse = "=") + } + out +} + +argv <- commandArgs(trailingOnly = TRUE) +if (length(argv) == 0 && nzchar(Sys.getenv("NB_CALIB_ARGS"))) { + argv <- strsplit(trimws(Sys.getenv("NB_CALIB_ARGS")), "\\s+")[[1]] +} +args <- parse_args(argv) + +mode <- if (!is.null(args$mode)) args$mode else "quick" +defaults <- switch(mode, + quick = list(`n-per` = "4", sims = "10", taxa = "20", shapes = "original", + offsets = "tss", tests = "wald_ml,lrt", truth = "null", + log2fc = "1", seed = "20261020"), + full = list(`n-per` = "3,4,6,8,12,20", sims = "1000", taxa = "40", + shapes = "original,constant,sparse,unequal_lib", + offsets = "tss,rle", + tests = "wald_ml,wald_trend,lrt,glmgampoi_ql,edger_ql", + truth = "null", log2fc = "1", seed = "20261020"), + stop("unknown --mode '", mode, "' (want quick or full, plus overrides)") +) +for (k in names(defaults)) if (is.null(args[[k]])) args[[k]] <- defaults[[k]] +if (is.null(args$out)) args$out <- file.path("bench", "null_calibration", "results", "raw") +if (is.null(args[["differential-jl"]])) { + args[["differential-jl"]] <- file.path("src", "analysis", "differential.jl") +} + +csv_ints <- function(s) as.integer(strsplit(s, ",", fixed = TRUE)[[1]]) +csv_strs <- function(s) trimws(strsplit(s, ",", fixed = TRUE)[[1]]) + +n_pers <- csv_ints(args[["n-per"]]) +n_sims <- as.integer(args$sims) +n_taxa <- as.integer(args$taxa) +shapes <- csv_strs(args$shapes) +offsets <- csv_strs(args$offsets) +want <- csv_strs(args$tests) +truth <- args$truth +lfc <- as.numeric(args$log2fc) +seed0 <- as.integer(args$seed) +out_dir <- args$out +stopifnot(truth %in% c("null", "power")) +dir.create(out_dir, recursive = TRUE, showWarnings = FALSE) + +## ------------------------------------------------------------ MASS access +# Normally MASS is installed and the extracted production block runs +# byte-identical. In a minimal R with no MASS but with MASS's pure-R +# negbin.R already sourced (glm.nb is pure R; MASS 7.3-65 negbin.R makes no +# .C/.Call/.Fortran call), the block's `MASS::` prefixes are stripped as a +# single mechanical substitution and that fact is recorded in the +# provenance. The functions executed are still MASS's own source. + +have_mass_ns <- requireNamespace("MASS", quietly = TRUE) +mass_mode <- if (have_mass_ns) { + paste0("installed MASS ", as.character(packageVersion("MASS"))) +} else if (exists("glm.nb", envir = globalenv())) { + "sourced negbin.R (MASS:: stripped from the extracted block; see header)" +} else { + stop("MASS is not available and glm.nb is not sourced: restore renv.lock first") +} +glm_nb <- if (have_mass_ns) MASS::glm.nb else get("glm.nb", envir = globalenv()) + +have_gp <- requireNamespace("glmGamPoi", quietly = TRUE) +have_edger <- requireNamespace("edgeR", quietly = TRUE) + +## ------------------------------------------------------- production block + +extract_fit_r <- function(path) { + lines <- readLines(path, warn = FALSE) + start <- grep('const _FIT_R = raw"""', lines, fixed = TRUE) + if (length(start) != 1) stop("could not find exactly one _FIT_R block in ", path) + rest <- lines[(start + 1):length(lines)] + end <- which(rest == '"""')[1] + if (is.na(end)) stop("_FIT_R block in ", path, " is not terminated") + block <- paste(rest[seq_len(end - 1)], collapse = "\n") + if (!grepl("da_result", block, fixed = TRUE) || !grepl("glm.nb", block, fixed = TRUE)) { + stop("extracted _FIT_R does not look like the production fit loop") + } + block +} + +fit_code <- extract_fit_r(args[["differential-jl"]]) +if (!have_mass_ns) fit_code <- gsub("MASS::", "", fit_code, fixed = TRUE) + +# Every da_* global fit_nb sets or the block creates on the current branch +# (mirrors the cleanup list in differential.jl's fit_nb). +DA_VARS <- c("da_counts", "da_group", "da_levels", "da_offset", "da_theta_upper", + "da_theta_lower", "da_use_gp", "da_g", "da_term", "da_n", "da_result", + "da_phi", "da_gp_failed", "da_gp", "da_j", "da_y", "da_warns", + "da_fit", "da_co", "da_row", "da_theta", "da_absent") + +run_block <- function(counts, offset, use_gp, n_per) { + g <- globalenv() + assign("da_counts", counts, g) + assign("da_group", rep(c("A", "B"), each = n_per), g) + assign("da_levels", c("A", "B"), g) + assign("da_offset", offset, g) + assign("da_theta_upper", 1e7, g) + assign("da_theta_lower", 1e-8, g) + assign("da_use_gp", use_gp, g) + on.exit(rm(list = intersect(ls(g), DA_VARS), envir = g)) + suppressWarnings(eval(parse(text = fit_code), envir = g)) + res <- get("da_result", envir = g) + res +} + +## ---------------------------------------------------------------- offsets +# Mirror src/analysis/differential.jl exactly: factors centred to geometric +# mean 1, offset = log(factor). RLE refuses (returns NULL) when no taxon is +# present in every sample, as rle_factors does; the refusal is recorded, +# never replaced by TSS. + +geomean <- function(x) exp(mean(log(x))) + +make_offset <- function(counts, method) { + if (method == "tss") { + lib <- rowSums(counts) + if (any(lib <= 0)) return(NULL) + return(log(lib / geomean(lib))) + } + if (method == "rle") { + shared <- which(colSums(counts > 0) == nrow(counts)) + if (length(shared) == 0) return(NULL) + sub <- counts[, shared, drop = FALSE] + gm <- apply(sub, 2, geomean) + raw <- apply(sub, 1, function(row) median(row / gm)) + return(log(raw / geomean(raw))) + } + stop("unknown offset method '", method, "'") +} + +## ------------------------------------------------------------- simulation +# One dataset: samples x taxa integer counts plus the true log2 fold change +# per taxon (all zero under the null). Taxon parameters are redrawn per +# dataset, as in the #193 script. The data-generating library sizes are +# realisations, not observed library sizes: offsets are computed from the +# simulated counts afterwards, as production computes them from data. + +simulate <- function(shape, n_per, n_taxa, truth, lfc) { + n <- 2 * n_per + mu <- switch(shape, + original = exp(runif(n_taxa, log(2), log(500))), + constant = exp(runif(n_taxa, log(2), log(500))), + sparse = exp(rnorm(n_taxa, mean = log(3), sd = 1.6)), + unequal_lib = exp(runif(n_taxa, log(2), log(500))), + stop("unknown shape '", shape, "'") + ) + disp <- if (shape == "constant") rep(0.5, n_taxa) else 0.05 + 2 / sqrt(mu) + lib <- exp(rnorm(n, 0, 0.3)) + if (shape == "unequal_lib") lib[(n_per + 1):n] <- lib[(n_per + 1):n] * 2.5 + true_lfc <- rep(0, n_taxa) + if (truth == "power") { + n_de <- max(1L, round(0.25 * n_taxa)) + de <- seq_len(n_de) + true_lfc[de] <- rep(c(lfc, -lfc), length.out = n_de) + } + y <- vapply(seq_len(n_taxa), function(j) { + m <- mu[j] * lib + if (true_lfc[j] != 0) m[(n_per + 1):n] <- m[(n_per + 1):n] * 2^true_lfc[j] + as.integer(rnbinom(n, mu = m, size = 1 / disp[j])) + }, integer(n)) + storage.mode(y) <- "integer" + # The #193 script's filter, kept so the replication is faithful: a taxon + # needs reads in at least 2 samples and non-constant counts to be tested. + keep <- colSums(y > 0) >= 2 & apply(y, 2, function(v) length(unique(v)) > 1) + list(counts = y[, keep, drop = FALSE], true_lfc = true_lfc[keep], + zero_frac = mean(y == 0)) +} + +## ------------------------------------------------------------ bench tests + +# Production's per-taxon screens, applied to the bench-only tests so every +# test faces the same family: constant counts, and complete separation (no +# reads in any sample of one group). Returns "" when fittable. +screen <- function(yv, g) { + if (length(unique(yv)) < 2L) return("constant counts") + if (all(yv[g == "A"] == 0) || all(yv[g == "B"] == 0)) return("complete separation") + "" +} + +fit_lrt <- function(counts, offset, n_per) { + g <- factor(rep(c("A", "B"), each = n_per), levels = c("A", "B")) + out <- data.frame(status = "failed", pvalue = NA_real_) + out <- out[rep(1, ncol(counts)), , drop = FALSE] + rownames(out) <- NULL + one <- function(formula, yv) { + warns <- character(0) + fit <- tryCatch( + withCallingHandlers(glm_nb(formula, control = glm.control(maxit = 100)), + warning = function(w) { warns <<- c(warns, conditionMessage(w)); invokeRestart("muffleWarning") }), + error = function(e) e) + list(fit = fit, warns = warns) + } + bad_fit <- function(r) { + inherits(r$fit, "error") || !isTRUE(r$fit[["converged"]]) || + !is.null(r$fit[["th.warn"]]) || + any(grepl("iteration limit|alternation limit|did not converge|NaNs produced", r$warns)) + } + off <- offset + for (j in seq_len(ncol(counts))) { + yv <- counts[, j] + if (nzchar(screen(yv, g))) next + # glm.nb looks its variables up in the formula's environment; yv, g and + # off all live in this frame (one()'s enclosing environment), which is + # where these formulas are created. + full <- one(yv ~ g + offset(off), yv) + if (bad_fit(full)) next + null <- one(yv ~ 1 + offset(off), yv) + if (bad_fit(null)) next + lr <- 2 * (as.numeric(logLik(full$fit)) - as.numeric(logLik(null$fit))) + if (!is.finite(lr) || lr < 0) next + p <- pchisq(lr, df = 1, lower.tail = FALSE) + if (!is.finite(p)) next + out$status[j] <- "ok" + out$pvalue[j] <- p + } + out +} + +fit_glmgampoi_ql <- function(counts, offset, n_per) { + g <- factor(rep(c("A", "B"), each = n_per), levels = c("A", "B")) + out <- data.frame(status = rep("failed", ncol(counts)), pvalue = NA_real_) + screens <- vapply(seq_len(ncol(counts)), + function(j) nzchar(screen(counts[, j], g)), logical(1)) + res <- tryCatch(suppressWarnings({ + gp <- glmGamPoi::glm_gp(t(counts), design = model.matrix(~ g), + size_factors = exp(offset), + overdispersion_shrinkage = TRUE) + # Numeric contrast weights on the design columns ((Intercept), gB): + # test_de parses a TEXT contrast as R code, so bench and production + # alike must never hand it a label. + glmGamPoi::test_de(gp, contrast = c(0, 1), sort_by = "none") + }), error = function(e) e) + if (inherits(res, "error")) return(out) + pcol <- intersect(c("pval", "pvalue", "p_value"), names(res))[1] + if (is.na(pcol) || length(res[[pcol]]) != ncol(counts)) return(out) + p <- as.numeric(res[[pcol]]) + ok <- !screens & is.finite(p) & p >= 0 & p <= 1 + out$status[ok] <- "ok" + out$pvalue[ok] <- p[ok] + out +} + +fit_edger_ql <- function(counts, offset, n_per) { + g <- factor(rep(c("A", "B"), each = n_per), levels = c("A", "B")) + out <- data.frame(status = rep("failed", ncol(counts)), pvalue = NA_real_) + screens <- vapply(seq_len(ncol(counts)), + function(j) nzchar(screen(counts[, j], g)), logical(1)) + res <- tryCatch(suppressWarnings({ + y <- edgeR::DGEList(counts = t(counts), group = g) + # Feed edgeR the bench offset rather than its own normalisation, so all + # tests see the same library-size handling: effective library size + # proportional to exp(offset). + y$samples$lib.size <- exp(offset) * geomean(rowSums(counts)) + y$samples$norm.factors <- 1 + design <- model.matrix(~ g) + y <- edgeR::estimateDisp(y, design) + fit <- edgeR::glmQLFit(y, design) + qlf <- edgeR::glmQLFTest(fit, coef = 2) + edgeR::topTags(qlf, n = Inf, sort.by = "none")$table + }), error = function(e) e) + if (inherits(res, "error")) return(out) + pcol <- intersect(c("pvalue", "p.value", "pvalue"), names(res))[1] + if (is.na(pcol) || length(res[[pcol]]) != ncol(counts)) return(out) + p <- as.numeric(res[[pcol]]) + ok <- !screens & is.finite(p) & p >= 0 & p <= 1 + out$status[ok] <- "ok" + out$pvalue[ok] <- p[ok] + out +} + +## ------------------------------------------------------------------- main + +enabled <- want +for (t in want) { + if (t %in% c("wald_trend", "glmgampoi_ql") && !have_gp) { + cat("SKIPPED test", t, ": glmGamPoi is not available in this R library\n") + enabled <- setdiff(enabled, t) + } + if (t == "edger_ql" && !have_edger) { + cat("SKIPPED test edger_ql: edgeR is not available in this R library\n") + enabled <- setdiff(enabled, t) + } +} +if (length(enabled) == 0) stop("no requested test can run in this R library") + +cell_index <- 0L +for (shape in shapes) for (off_m in offsets) for (n_per in n_pers) { + cell_index <- cell_index + 1L + # The taxa count is part of the cell identity: reruns append to a cell's + # CSV, so two settings must never share a file by accident. + cell <- sprintf("%s_%s_%s_n%d_t%d%s", truth, shape, off_m, n_per, n_taxa, + if (truth == "power") sprintf("_lfc%s", lfc) else "") + f <- file.path(out_dir, paste0("pvalues_", cell, ".csv")) + prov <- file.path(out_dir, paste0("provenance_", cell, ".txt")) + writeLines(c( + paste("cell:", cell), paste("date:", format(Sys.time(), tz = "UTC", usetz = TRUE)), + paste("args:", paste(argv, collapse = " ")), + paste("MASS:", mass_mode), + paste("glmGamPoi:", if (have_gp) as.character(packageVersion("glmGamPoi")) else "absent"), + paste("edgeR:", if (have_edger) as.character(packageVersion("edgeR")) else "absent"), + paste("tests run:", paste(enabled, collapse = ",")), + capture.output(sessionInfo()) + ), prov) + cat(sprintf("cell %s: %d sims x %d taxa, tests %s\n", + cell, n_sims, n_taxa, paste(enabled, collapse = "+"))) + header_done <- file.exists(f) + skipped_sims <- 0L + refused_offsets <- 0L + for (s in seq_len(n_sims)) { + # One seed per (cell, sim): cells can run in any order, in parallel + # processes, and reproduce dataset-for-dataset. + set.seed(seed0 + cell_index * 1000003L + s) + sim <- simulate(shape, n_per, n_taxa, truth, lfc) + counts <- sim$counts + if (ncol(counts) < 5) { skipped_sims <- skipped_sims + 1L; next } + offset <- make_offset(counts, off_m) + if (is.null(offset)) { refused_offsets <- refused_offsets + 1L; next } + fits <- list() + if ("wald_ml" %in% enabled) { + r <- run_block(counts, offset, FALSE, n_per) + fits$wald_ml <- data.frame(status = r$status, pvalue = r$pvalue) + } + if ("wald_trend" %in% enabled) { + r <- run_block(counts, offset, TRUE, n_per) + fits$wald_trend <- data.frame(status = r$status, pvalue = r$pvalue) + } + if ("lrt" %in% enabled) fits$lrt <- fit_lrt(counts, offset, n_per) + if ("glmgampoi_ql" %in% enabled) fits$glmgampoi_ql <- fit_glmgampoi_ql(counts, offset, n_per) + if ("edger_ql" %in% enabled) fits$edger_ql <- fit_edger_ql(counts, offset, n_per) + rows <- do.call(rbind, lapply(names(fits), function(t) { + data.frame(truth = truth, shape = shape, offset = off_m, n_per = n_per, + taxa = n_taxa, sim = s, taxon = seq_len(ncol(counts)), + true_log2fc = sim$true_lfc, test = t, + status = fits[[t]]$status, pvalue = fits[[t]]$pvalue, + stringsAsFactors = FALSE) + })) + write.table(rows, f, sep = ",", row.names = FALSE, + col.names = !header_done, append = header_done, quote = FALSE) + header_done <- TRUE + if (s %% 25 == 0) { cat(sprintf(" %s sim %d/%d\n", cell, s, n_sims)); flush.console() } + } + cat(sprintf("cell %s done: %d sims skipped (<5 testable taxa), %d offsets refused\n", + cell, skipped_sims, refused_offsets)) +} +cat("calibration run complete\n") diff --git a/bench/null_calibration/results/calibration_summary.csv b/bench/null_calibration/results/calibration_summary.csv new file mode 100644 index 00000000..a73478c8 --- /dev/null +++ b/bench/null_calibration/results/calibration_summary.csv @@ -0,0 +1,33 @@ +"truth","shape","offset","n_per_group","n_taxa_setting","test","n_sims","n_attempted","n_pvalues","n_failed","n_boundary","fail_rate","rate_01","rate_01_lo","rate_01_hi","rate_05","rate_05_lo","rate_05_hi","rate_10","rate_10_lo","rate_10_hi","ks_D","ks_p","mean_p_null" +"null","constant","tss",4,40,"lrt",150,6000,5679,321,0,0.0535,0.0343370311674591,0.0299070144118022,0.0393966009845745,0.103539355520338,0.0958817540765541,0.111732951941575,0.167635147032928,0.158145307258128,0.177574327478367,0.0843830805075691,1.50521819708956e-35,0.444054181891207 +"null","constant","tss",4,40,"wald_ml",150,6000,5758,242,147,0.0403333333333333,0.067210837096214,0.0610277913962478,0.0739709697428515,0.134769017019799,0.126192001748904,0.14383303624288,0.195206668982286,0.185173587708648,0.205646165822299,0.104078202508359,1.3341509762296e-54,0.433539982517614 +"null","constant","tss",12,40,"lrt",150,6000,5986,14,0,0.00233333333333333,0.0143668559973271,0.0116487013673261,0.0177079118301315,0.0649849649181423,0.0590152557624449,0.0715126495578783,0.115603073838958,0.107748388407623,0.123950809035566,0.0376376568717833,8.62249043869314e-08,0.485492376920367 +"null","constant","tss",12,40,"wald_ml",150,6000,5989,11,6,0.00183333333333333,0.0213725162798464,0.0180049117092347,0.0253537288632171,0.0728001335782267,0.066490416983351,0.0796575271661756,0.127233260978461,0.119031959024314,0.135912455794454,0.0445719596399606,9.25695055432286e-11,0.481410606985188 +"null","original","rle",4,40,"lrt",150,5997,5795,202,0,0.0336835084208771,0.0291630716134599,0.0251329702386332,0.0338169874810617,0.100431406384815,0.0929553445016146,0.108436858943485,0.167385677308024,0.15799492693621,0.177217110265444,0.0999852967297406,9.5741484666021e-51,0.437673327317577 +"null","original","rle",4,40,"wald_ml",150,5997,5843,154,79,0.0256795064198766,0.0641793599178504,0.0581774162795938,0.0707539843971647,0.138113982543214,0.129504908831245,0.147198584878162,0.199897313024131,0.189841646759709,0.210347322840407,0.121629858990801,1.65933724634618e-75,0.426544757195022 +"null","original","rle",12,40,"lrt",150,6000,5999,1,0,0.000166666666666667,0.0170028338056343,0.0140269075883063,0.0205969385096592,0.0675112518753126,0.0614348261387838,0.0741412113767197,0.126187697949658,0.118023356072251,0.134830474773197,0.0374388440525218,9.94059353574812e-08,0.476768173372017 +"null","original","rle",12,40,"wald_ml",150,6000,5999,1,0,0.000166666666666667,0.024837472912152,0.021192855359116,0.029090241552791,0.0801800300050008,0.0735734789459446,0.0873239003112516,0.138523087181197,0.130012534390866,0.14749628709781,0.0425443113010688,7.40678934295594e-10,0.472785456890374 +"null","original","tss",3,40,"lrt",200,7988,7315,673,0,0.0842513770655984,0.045933014354067,0.0413694275154052,0.0509732557662729,0.131237183868763,0.12369250293294,0.139168971747875,0.203417634996582,0.19434974613276,0.21279685973194,0.135182796664952,1.55071172931089e-116,0.415587740716814 +"null","original","tss",3,40,"wald_ml",200,7988,7476,512,273,0.0640961442163245,0.0918940609951846,0.0855537288027265,0.0986535791203389,0.179240235420011,0.170711249407119,0.188098689760206,0.250668806848582,0.240974270777598,0.260619443404176,0.161656443920632,4.03457010651865e-170,0.401366722446981 +"null","original","tss",4,40,"lrt",200,7997,7708,289,0,0.0361385519569839,0.0303580695381422,0.026755637175536,0.0344283824052878,0.102880124545926,0.0962945813035816,0.109861298236999,0.169434353918007,0.161224866817917,0.177973166853889,0.0933013909161196,1.0455020069296e-58,0.443086204545911 +"null","original","tss",4,40,"wald_ml",200,7997,7783,214,106,0.0267600350131299,0.0628292432224078,0.0576509835270075,0.0684388391582219,0.13825003212129,0.130760030253546,0.146096956012513,0.203520493383014,0.194723061730016,0.212610447703097,0.113643078400003,9.87146482984532e-88,0.431436851581758 +"null","original","tss",6,40,"lrt",200,8000,7904,96,0,0.012,0.0232793522267206,0.0201794197046358,0.0268424459659169,0.086918016194332,0.0809063372182911,0.09333102780675,0.148658906882591,0.14098681509377,0.156672346521709,0.0645951152356356,4.52120174669666e-29,0.461303882402388 +"null","original","tss",6,40,"wald_ml",200,8000,7937,63,28,0.007875,0.0447272269119315,0.0403957707152817,0.0494991683188881,0.111251102431649,0.104520582486221,0.118357744451632,0.170719415396245,0.162601431411705,0.179155984712773,0.0787526584971869,3.50452954348045e-43,0.453506610955192 +"null","original","tss",8,40,"lrt",200,8000,7974,26,0,0.00325,0.0171808377225984,0.0145524202088425,0.0202742256019111,0.0701028342111863,0.0647034005936096,0.0759162726113939,0.128668171557562,0.121497414397098,0.136196533199172,0.0453296175635426,1.17327512113714e-14,0.473696316973658 +"null","original","tss",8,40,"wald_ml",200,8000,7984,16,6,0.002,0.0293086172344689,0.0258291673085302,0.0332407906085308,0.0886773547094188,0.0826378982356264,0.0951124322059991,0.145415831663327,0.137853814201029,0.153318897577363,0.0552258902999433,1.41431816022389e-21,0.467708032138111 +"null","original","tss",12,40,"lrt",200,8000,7997,3,0,0.000375,0.0171314242841065,0.0145105151972706,0.0202160145372216,0.0645241965737151,0.059345808818977,0.0701207559357476,0.119294735525822,0.112372755059871,0.126582293438552,0.0329734074224069,5.6092770626337e-08,0.480625097706312 +"null","original","tss",12,40,"wald_ml",200,8000,7999,1,1,0.000125,0.0251281410176272,0.0219194190982992,0.028792751186341,0.0757594699337417,0.0701620857850788,0.0817641350549703,0.130016252031504,0.122823186861121,0.137564510376576,0.0381036842306565,1.63501153735572e-10,0.476689665536424 +"null","original","tss",20,40,"lrt",200,8000,8000,0,0,0,0.014125,0.0117624417845792,0.0169539514627402,0.058375,0.0534462764601491,0.0637276410450682,0.11025,0.103572963173447,0.117301159323374,0.020738749813529,0.00205314046936761,0.487703822259244 +"null","original","tss",20,40,"wald_ml",200,8000,8000,0,0,0,0.018625,0.0158851798715362,0.0218268938087898,0.0645,0.0593234953117845,0.0700945427828028,0.116,0.109166456166853,0.123202146883254,0.026145410693358,3.55641366754023e-05,0.48530796517549 +"null","sparse","tss",4,40,"lrt",150,5365,4380,985,0,0.183597390493942,0.0237442922374429,0.0196354002458669,0.0286878479396827,0.0817351598173516,0.0739836169914801,0.0902197342433841,0.144063926940639,0.133976279706903,0.154775371262067,0.0691867976694867,1.2302251751396e-18,0.455338729993458 +"null","sparse","tss",4,40,"wald_ml",150,5365,4554,811,236,0.15116495806151,0.0408432147562582,0.0354710856191992,0.0469893207412142,0.106719367588933,0.0980810744949493,0.116020593506725,0.170838823012736,0.160186222147154,0.182046273852014,0.0823244156861178,3.1118285502842e-27,0.44737008911558 +"null","sparse","tss",12,40,"lrt",150,5859,5562,297,0,0.0506912442396313,0.015282272563826,0.0123767061844007,0.0188569284972326,0.0604099244875944,0.054446958150994,0.0669796875450717,0.110931319669184,0.102945034249156,0.119454663618446,0.0326604118895657,1.40505153549206e-05,0.480162876414684 +"null","sparse","tss",12,40,"wald_ml",150,5859,5594,265,33,0.0452295613585936,0.0196639256346085,0.0163415019144068,0.0236456003405133,0.0682874508401859,0.0619693876083744,0.075198030309613,0.119413657490168,0.111176063604179,0.128173597988946,0.0345742276076426,3.11048126202616e-06,0.476462685961144 +"null","unequal_lib","tss",4,40,"lrt",150,5999,5809,190,0,0.0316719453242207,0.0395937338612498,0.0348758076860468,0.0449201857375891,0.113616801514891,0.105710109456138,0.122034181902751,0.17868824238251,0.169050117338897,0.188751049879355,0.0933208396676188,2.28941310680147e-44,0.440256995667013 +"null","unequal_lib","tss",4,40,"wald_ml",150,5999,5898,101,56,0.0168361393565594,0.0764665988470668,0.0699568766474143,0.0835276696829271,0.14852492370295,0.139678040812683,0.157829350917821,0.211766700576467,0.20152919358122,0.222379424818132,0.115545521434757,8.05155216532249e-69,0.429486140222383 +"null","unequal_lib","tss",12,40,"lrt",150,6000,5995,5,0,0.000833333333333333,0.0128440366972477,0.0102895388904349,0.0160224515004104,0.0618849040867389,0.0560617457461293,0.0682691711384984,0.116263552960801,0.108394130866014,0.124624439205597,0.023602226833183,0.00251353912720786,0.48526062352396 +"null","unequal_lib","tss",12,40,"wald_ml",150,6000,6000,0,1,0,0.0196666666666667,0.0164483267012244,0.0234996733360393,0.0741666666666667,0.0678051993175382,0.0810730588695571,0.13,0.121726673024468,0.138746803756492,0.0305808238880169,2.67468380464776e-05,0.481384434000172 +"power","original","tss",4,40,"lrt",150,5997,5768,229,0,0.0381857595464399,0.0325109522711552,0.0276325349035269,0.038216785218464,0.114134194143417,0.105010338234011,0.123940999484436,0.183306433018215,0.172073169738517,0.195100216790487,0.105618632297901,1.89778614361325e-42,0.434784684139481 +"power","original","tss",4,40,"wald_ml",150,5997,5841,156,88,0.0260130065032516,0.0721602549510585,0.0648766537321604,0.0801914513597557,0.150466651491008,0.140199738589643,0.161344329177655,0.218757113589802,0.206780906784307,0.23122475633943,0.127157614018003,4.02317867444891e-62,0.422783825089475 +"power","original","tss",12,40,"lrt",150,6000,5997,3,0,5e-04,0.0237777777777778,0.0197159062402261,0.0286520171896841,0.0831111111111111,0.0753968197212776,0.0915365560857739,0.150888888888889,0.140728778500017,0.161644533543095,0.0677456389208428,2.30351108889158e-18,0.456624861778725 +"power","original","tss",12,40,"wald_ml",150,6000,5998,2,1,0.000333333333333333,0.0328888888888889,0.0280635063862089,0.0385110969967455,0.096,0.0877341616927958,0.104955005267343,0.16,0.1495793559677,0.171000636028456,0.0737015111182361,1.17376844197725e-21,0.452433541509862 diff --git a/bench/null_calibration/results/power_summary.csv b/bench/null_calibration/results/power_summary.csv new file mode 100644 index 00000000..b9685f56 --- /dev/null +++ b/bench/null_calibration/results/power_summary.csv @@ -0,0 +1,5 @@ +"shape","offset","n_per_group","test","n_sims","power_raw_05","power_bh_05","bh_fdr_05","mean_discoveries" +"original","tss",4,"lrt",150,0.413696715583508,0.147449336128581,0.183793650793651,1.89333333333333 +"original","tss",4,"wald_ml",150,0.479972375690608,0.290055248618785,0.379349206349206,4.7 +"original","tss",12,"lrt",150,0.668002672010688,0.421509686038744,0.100953102453102,4.82 +"original","tss",12,"wald_ml",150,0.688251001335114,0.509345794392523,0.128748011248011,5.99333333333333 diff --git a/bench/null_calibration/results/provenance_summary.csv b/bench/null_calibration/results/provenance_summary.csv new file mode 100644 index 00000000..5cc78144 --- /dev/null +++ b/bench/null_calibration/results/provenance_summary.csv @@ -0,0 +1,17 @@ +cell,date_utc,args,mass_mode,glmGamPoi,edgeR,tests_run,r_version,platform,webr +null_constant_tss_n12_t40,2026-10-22 22:42:23 UTC,"--shapes=constant --offsets=tss --n-per=12 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_constant_tss_n4_t40,2026-10-22 22:43:30 UTC,"--shapes=constant --offsets=tss --n-per=4 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_original_rle_n12_t40,2026-10-22 22:54:06 UTC,"--shapes=original --offsets=rle --n-per=12 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_original_rle_n4_t40,2026-10-22 22:55:33 UTC,"--shapes=original --offsets=rle --n-per=4 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_original_tss_n12_t40,2026-10-22 22:32:15 UTC,"--shapes=original --offsets=tss --n-per=12 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_original_tss_n20_t40,2026-10-22 22:37:21 UTC,"--shapes=original --offsets=tss --n-per=20 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_original_tss_n3_t40,2026-10-22 22:27:17 UTC,"--shapes=original --offsets=tss --n-per=3 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_original_tss_n4_t40,2026-10-22 22:32:41 UTC,"--shapes=original --offsets=tss --n-per=4 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_original_tss_n6_t40,2026-10-22 22:38:09 UTC,"--shapes=original --offsets=tss --n-per=6 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_original_tss_n8_t40,2026-10-22 22:27:17 UTC,"--shapes=original --offsets=tss --n-per=8 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_sparse_tss_n12_t40,2026-10-22 22:46:13 UTC,"--shapes=sparse --offsets=tss --n-per=12 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_sparse_tss_n4_t40,2026-10-22 22:47:35 UTC,"--shapes=sparse --offsets=tss --n-per=4 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_unequal_lib_tss_n12_t40,2026-10-22 22:50:11 UTC,"--shapes=unequal_lib --offsets=tss --n-per=12 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +null_unequal_lib_tss_n4_t40,2026-10-22 22:51:15 UTC,"--shapes=unequal_lib --offsets=tss --n-per=4 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +power_original_tss_n12_t40_lfc1,2026-10-22 22:57:57 UTC,"--shapes=original --offsets=tss --n-per=12 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=power --log2fc=1 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +power_original_tss_n4_t40_lfc1,2026-10-22 22:59:41 UTC,"--shapes=original --offsets=tss --n-per=4 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=power --log2fc=1 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 diff --git a/bench/null_calibration/summarise.R b/bench/null_calibration/summarise.R new file mode 100644 index 00000000..21231182 --- /dev/null +++ b/bench/null_calibration/summarise.R @@ -0,0 +1,131 @@ +#!/usr/bin/env Rscript +# SPDX-License-Identifier: AGPL-3.0-only +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# +# Aggregate calibration.R's per-cell p-value CSVs into the #193 summary +# tables. Base R only, so it runs anywhere calibration.R's output lands. +# +# Rscript bench/null_calibration/summarise.R \ +# --results=bench/null_calibration/results/raw \ +# --out=bench/null_calibration/results +# +# Writes calibration_summary.csv (one row per cell x test: rejection rates +# at alpha = 0.01/0.05/0.10 with Wilson 95% intervals, Kolmogorov-Smirnov D +# and p against Uniform(0,1), fit-failure share) and, when power cells are +# present, power_summary.csv (raw and BH power on truly-changed taxa, and +# the Benjamini-Hochberg false discovery rate at 0.05 averaged per dataset). +# +# A "test" here is one taxon in one simulated dataset that produced a +# p-value (production's BH family: status ok or boundary). Failures are +# counted separately and never enter a rate's denominator silently: every +# row reports both n_pvalues and n_failed. + +argv <- commandArgs(trailingOnly = TRUE) +if (length(argv) == 0 && nzchar(Sys.getenv("NB_CALIB_ARGS"))) { + argv <- strsplit(trimws(Sys.getenv("NB_CALIB_ARGS")), "\\s+")[[1]] +} +args <- list() +for (a in argv) { + if (!grepl("^--[a-z0-9-]+=", a)) stop("unrecognised argument '", a, "' (want --key=value)") + kv <- strsplit(sub("^--", "", a), "=", fixed = TRUE)[[1]] + args[[kv[1]]] <- paste(kv[-1], collapse = "=") +} +results_dir <- if (!is.null(args$results)) args$results else + file.path("bench", "null_calibration", "results", "raw") +out_dir <- if (!is.null(args$out)) args$out else + file.path("bench", "null_calibration", "results") +dir.create(out_dir, recursive = TRUE, showWarnings = FALSE) + +files <- list.files(results_dir, pattern = "^pvalues_.*[.]csv$", full.names = TRUE) +if (length(files) == 0) stop("no pvalues_*.csv under ", results_dir) +dat <- do.call(rbind, lapply(files, function(f) { + d <- read.csv(f, stringsAsFactors = FALSE) + d$file <- basename(f) + d +})) +if (!"taxa" %in% names(dat)) dat$taxa <- NA_integer_ +dat$has_p <- !is.na(dat$pvalue) & dat$status %in% c("ok", "boundary") + +# Wilson score interval for k/n at 95% (the same interval prop.test uses +# without continuity correction, written out so the bench needs nothing +# beyond base R distributions). +wilson <- function(k, n) { + if (n == 0) return(c(NA_real_, NA_real_, NA_real_)) + z <- qnorm(0.975) + p <- k / n + den <- 1 + z^2 / n + mid <- (p + z^2 / (2 * n)) / den + half <- z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2)) / den + c(p, max(0, mid - half), min(1, mid + half)) +} + +key_of <- function(d) paste(d$truth, d$shape, d$offset, d$n_per, d$taxa, d$test, sep = "|") +groups <- split(dat, key_of(dat)) + +summ_rows <- lapply(groups, function(d) { + p <- d$pvalue[d$has_p] + # Under a power cell only the truly-null taxa calibrate the test. + p_null <- d$pvalue[d$has_p & d$true_log2fc == 0] + ks <- if (length(p_null) >= 10) { + k <- suppressWarnings(ks.test(p_null, "punif")) + c(unname(k$statistic), k$p.value) + } else c(NA_real_, NA_real_) + w <- function(alpha) wilson(sum(p_null < alpha), length(p_null)) + w01 <- w(0.01); w05 <- w(0.05); w10 <- w(0.10) + data.frame( + truth = d$truth[1], shape = d$shape[1], offset = d$offset[1], + n_per_group = d$n_per[1], n_taxa_setting = d$taxa[1], test = d$test[1], + n_sims = length(unique(d$sim)), n_attempted = nrow(d), + n_pvalues = length(p), n_failed = sum(d$status == "failed"), + n_boundary = sum(d$status == "boundary"), + fail_rate = sum(d$status == "failed") / nrow(d), + rate_01 = w01[1], rate_01_lo = w01[2], rate_01_hi = w01[3], + rate_05 = w05[1], rate_05_lo = w05[2], rate_05_hi = w05[3], + rate_10 = w10[1], rate_10_lo = w10[2], rate_10_hi = w10[3], + ks_D = ks[1], ks_p = ks[2], mean_p_null = if (length(p_null)) mean(p_null) else NA_real_, + stringsAsFactors = FALSE + ) +}) +summary_df <- do.call(rbind, summ_rows) +summary_df <- summary_df[order(summary_df$truth, summary_df$shape, summary_df$offset, + summary_df$n_per_group, summary_df$test), ] +write.csv(summary_df, file.path(out_dir, "calibration_summary.csv"), row.names = FALSE) +cat(sprintf("wrote %s (%d rows)\n", file.path(out_dir, "calibration_summary.csv"), nrow(summary_df))) + +pow <- dat[dat$truth == "power", , drop = FALSE] +if (nrow(pow) > 0) { + pow$padj <- NA_real_ + # BH within each simulated dataset and test, over that dataset's tested + # family - the same family production adjusts over. + fams <- split(seq_len(nrow(pow)), + paste(pow$file, pow$sim, pow$test, sep = "|")) + for (idx in fams) { + fam <- idx[pow$has_p[idx]] + if (length(fam) > 0) pow$padj[fam] <- p.adjust(pow$pvalue[fam], method = "BH") + } + pg <- split(pow, key_of(pow)) + prows <- lapply(pg, function(d) { + de <- d[d$true_log2fc != 0 & d$has_p, , drop = FALSE] + by_sim <- split(d, d$sim) + fdp <- vapply(by_sim, function(s) { + disc <- s$has_p & !is.na(s$padj) & s$padj < 0.05 + if (!any(disc)) 0 else mean(s$true_log2fc[disc] == 0) + }, numeric(1)) + data.frame( + shape = d$shape[1], offset = d$offset[1], n_per_group = d$n_per[1], + test = d$test[1], n_sims = length(by_sim), + power_raw_05 = if (nrow(de)) mean(de$pvalue < 0.05) else NA_real_, + power_bh_05 = if (nrow(de)) mean(!is.na(de$padj) & de$padj < 0.05) else NA_real_, + bh_fdr_05 = mean(fdp), + mean_discoveries = mean(vapply(by_sim, function(s) + sum(s$has_p & !is.na(s$padj) & s$padj < 0.05), numeric(1))), + stringsAsFactors = FALSE + ) + }) + power_df <- do.call(rbind, prows) + power_df <- power_df[order(power_df$shape, power_df$offset, + power_df$n_per_group, power_df$test), ] + write.csv(power_df, file.path(out_dir, "power_summary.csv"), row.names = FALSE) + cat(sprintf("wrote %s (%d rows)\n", file.path(out_dir, "power_summary.csv"), nrow(power_df))) +} +cat("summarise complete\n") diff --git a/bench/null_calibration/webr_run.mjs b/bench/null_calibration/webr_run.mjs new file mode 100644 index 00000000..50eedd29 --- /dev/null +++ b/bench/null_calibration/webr_run.mjs @@ -0,0 +1,104 @@ +#!/usr/bin/env node +// SPDX-License-Identifier: AGPL-3.0-only +// SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +// +// Sandbox-only driver: run calibration.R / summarise.R under WebR (R in +// WebAssembly) when no native R is available. The R code executed is the +// bench's own, unmodified; MASS comes from sourcing MASS's pure-R negbin.R +// (glm.nb makes no compiled calls), which calibration.R detects and records +// in its provenance output. Prefer native Rscript with the renv.lock pin +// whenever it exists - it is faster and carries glmGamPoi/edgeR. +// +// One-time setup (outside the repo, so no repo lockfile changes): +// mkdir ~/webr-probe && cd ~/webr-probe && npm init -y && npm i webr@0.6.0 +// git clone https://github.com/cran/MASS.git mass-src +// git -C mass-src checkout b6a2274 # MASS 7.3-65, the renv.lock pin +// +// Run, from the repo root: +// WEBR_PATH=~/webr-probe/node_modules/webr \ +// node bench/null_calibration/webr_run.mjs \ +// --mass-r ~/webr-probe/mass-src/R/negbin.R \ +// --script calibration -- --n-per=4 --sims=200 --tests=wald_ml,lrt +// Files land in bench/null_calibration/results/raw (override with +// --host-out). For summarise, existing raw CSVs in --host-out are mounted +// in first, and the summary CSVs are copied back to --host-summary-out +// (default bench/null_calibration/results). + +import { existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath, pathToFileURL } from 'node:url'; + +const here = dirname(fileURLToPath(import.meta.url)); +const argv = process.argv.slice(2); +const sep = argv.indexOf('--'); +const driverArgs = sep === -1 ? argv : argv.slice(0, sep); +const rArgs = sep === -1 ? [] : argv.slice(sep + 1); + +function opt(name, fallback) { + const hit = driverArgs.find((a) => a.startsWith(`${name}=`)); + if (hit) return hit.slice(name.length + 1); + const i = driverArgs.indexOf(name); + return i === -1 ? fallback : driverArgs[i + 1]; +} + +const script = opt('--script', 'calibration'); +const massR = opt('--mass-r', ''); +const hostOut = resolve(opt('--host-out', join(here, 'results', 'raw'))); +const hostSummaryOut = resolve(opt('--host-summary-out', join(here, 'results'))); +const webrPath = process.env.WEBR_PATH || 'webr'; +// WEBR_PATH may name the package directory or its ESM entry directly. +const webrEntry = webrPath === 'webr' ? 'webr' + : pathToFileURL(resolve(webrPath.endsWith('.mjs') ? webrPath : join(webrPath, 'dist', 'webr.mjs'))).href; +const { WebR } = await import(webrEntry); + +// No stdout/stderr callbacks here: under Node the WebR worker cannot clone +// function options, and the defaults already stream R output to this +// process's stdout/stderr line by line. +const webR = new WebR({ interactive: false }); +await webR.init(); + +const put = async (vfsPath, hostPath) => { + const dir = vfsPath.slice(0, vfsPath.lastIndexOf('/')); + if (dir) await webR.evalRVoid(`dir.create("${dir}", recursive = TRUE, showWarnings = FALSE)`); + await webR.FS.writeFile(vfsPath, readFileSync(hostPath)); +}; + +await put('/work/calibration.R', join(here, 'calibration.R')); +await put('/work/summarise.R', join(here, 'summarise.R')); +await put('/work/differential.jl', join(here, '..', '..', 'src', 'analysis', 'differential.jl')); +if (massR) await put('/work/negbin.R', resolve(massR.replace(/^~/, process.env.HOME || ''))); + +// Mount any existing raw results so summarise (and resumed calibration +// cells appending to the same CSV) see them. +if (existsSync(hostOut)) { + for (const f of readdirSync(hostOut)) { + if (/\.(csv|txt)$/.test(f)) await put(`/work/results/raw/${f}`, join(hostOut, f)); + } +} + +const benchArgs = [...rArgs.filter((a) => !a.startsWith('--differential-jl=') && !a.startsWith('--out=') && !a.startsWith('--results=')), + '--differential-jl=/work/differential.jl', + // calibration.R writes raw cells to --out; summarise.R writes summaries to + // --out. They are different directories. + script === 'summarise' ? '--out=/work/results' : '--out=/work/results/raw', + '--results=/work/results/raw']; +const quote = (s) => `'${s.replace(/'/g, "'\\''")}'`; +await webR.evalRVoid(`Sys.setenv(NB_CALIB_ARGS = ${quote(benchArgs.join(' '))})`); +if (massR) await webR.evalRVoid('source("/work/negbin.R")'); +await webR.evalRVoid('setwd("/work")'); +await webR.evalRVoid(`source("/work/${script}.R")`); + +mkdirSync(hostOut, { recursive: true }); +mkdirSync(hostSummaryOut, { recursive: true }); +for (const [vfsDir, hostDir] of [['/work/results/raw', hostOut], ['/work/results', hostSummaryOut]]) { + const listing = await (await webR.evalR( + `list.files("${vfsDir}", pattern = "\\\\.(csv|txt)$")`)).toJs(); + for (const name of listing.values) { + if (vfsDir.endsWith('/raw') === false && name.startsWith('pvalues_')) continue; + const bytes = await webR.FS.readFile(`${vfsDir}/${name}`); + writeFileSync(join(hostDir, name), bytes); + console.log(`[webr_run] ${name} -> ${join(hostDir, name)}`); + } +} +await webR.close(); +process.exit(0); diff --git a/config/defaults/pipeline.yml b/config/defaults/pipeline.yml index 0be643e4..d9930ae9 100644 --- a/config/defaults/pipeline.yml +++ b/config/defaults/pipeline.yml @@ -365,6 +365,12 @@ analysis: # steadies low-count taxa -- and fits each taxon through # MASS::negative.binomial(theta = 1/dispersion), so the Wald test is # unchanged. Requires the glmGamPoi R package; refused for non-nb_glm methods. + # Both fixed-theta Wald tests treat the dispersion they use as known rather + # than propagating its estimation uncertainty: every ml result therefore + # carries a calibration caveat in its diagnostics, strongest when a group + # is small, as does a glmGamPoi result whose smaller group has at most 12 + # samples. The caveat records the limitation; it changes no p-value. + # Evidence and the replication protocol: bench/null_calibration. dispersion: ml # Fraction of samples, 0 to 1, in which a taxon must have reads to be # tested. Taxa below it are listed as filtered, without a p-value. Applied diff --git a/frontend/src/api/types.ts b/frontend/src/api/types.ts index 8c4edf97..48df92c7 100644 --- a/frontend/src/api/types.ts +++ b/frontend/src/api/types.ts @@ -474,6 +474,18 @@ export interface DifferentialResult { } diagnostics: { n_taxa: number; n_tested: number; n_failed: number; n_boundary: number; n_filtered: number + /** + * nb_glm only: interpretation caveat for the fixed-theta Wald test. + * Present for every `ml` result, scaled by the smaller group, and for + * `glmGamPoi` results whose smaller group has at most 12 samples. It + * changes no statistic. + */ + calibration_caveat?: { + severity: 'info' | 'caution' | 'strong' + dispersion: 'ml' | 'glmGamPoi' + min_group_n: number + message: string + } /** Safety note for two-part models. */ model_warning?: string /** clr_welch only. */ diff --git a/frontend/src/components/DifferentialPanel.tsx b/frontend/src/components/DifferentialPanel.tsx index 0f61f6e9..545aa875 100644 --- a/frontend/src/components/DifferentialPanel.tsx +++ b/frontend/src/components/DifferentialPanel.tsx @@ -237,6 +237,15 @@ export function DifferentialPanel({ study, runs, option, aggregate }: { return ` Reference frame taxon: ${s.reference_taxon ?? 'none'} over ${s.n_identifiable_taxa} identifiable taxa${sep}.` })()}

+ {result.diagnostics.calibration_caveat && ( +

+ Calibration caveat: {result.diagnostics.calibration_caveat.message} +

+ )} {result.diagnostics.zero_replacement && result.diagnostics.zero_replacement.zeros_replaced > 0 && (() => { const z = result.diagnostics.zero_replacement! diff --git a/frontend/src/components/PipelineStages.tsx b/frontend/src/components/PipelineStages.tsx index 86ad5d24..9d84a40e 100644 --- a/frontend/src/components/PipelineStages.tsx +++ b/frontend/src/components/PipelineStages.tsx @@ -152,7 +152,7 @@ export const CONFIG_DESCRIPTIONS: Record = { 'analysis.beta.depth': 'Reads per sample for rarefy or srs; 0 uses the smallest non-empty sample.', 'analysis.differential.method': 'Differential model: nb_glm (negative-binomial GLM); clr_welch (Welch test on zero-replaced CLR values); zinb or hurdle_nb (two-part count models with a joint zero/count-process test); ancom_bc (bias-corrected log-linear model); aldex2 (Monte-Carlo CLR); songbird (CLR-centred multinomial regression); multinomial or dirichlet_multinomial (joint compositional models with log ratios against a reference taxon). Compositional effects are relative log-ratios, not absolute abundance changes.', 'analysis.differential.offset': 'Library-size offset in the count component of nb_glm, zinb and hurdle_nb: tss (total reads per sample) or rle (median-of-ratios over taxa present in every sample; refused when there are none).', - 'analysis.differential.dispersion': 'nb_glm only: ml estimates theta separately with MASS::glm.nb; glmGamPoi uses a cross-taxon dispersion trend in fixed-scale negative-binomial Wald fits.', + 'analysis.differential.dispersion': 'nb_glm only: ml estimates theta separately with MASS::glm.nb; glmGamPoi uses a cross-taxon dispersion trend in fixed-scale negative-binomial Wald fits. Both Wald tests treat the dispersion as known, so ml results always carry a calibration caveat (strongest in small groups; see bench/null_calibration).', 'analysis.differential.min_prevalence': 'Fraction of samples, 0 to 1, in which a taxon must have reads to be tested; taxa below it are listed as filtered. Compositional models filter before constructing their subcomposition.', 'analysis.differential.replacement_method': 'clr_welch only: bayesian_multiplicative uses cmultRepl GBM posterior estimates (Martín-Fernández et al. 2015); multiplicative_replacement uses CZM (Martín-Fernández et al. 2003). Both preserve observed-part ratios, but every replacement is biased. For sparse data consider NB on raw counts, prevalence filtering, occupancy or zero-inflated models instead.', 'analysis.differential.replacement_delta': "clr_welch only: cmultRepl frac, strictly between 0 and 1. CZM starts at delta × 0.5/sample total; either method caps high imputations at delta × the taxon's smallest observed proportion. Below 0.01 or at least 0.9 warns. Do not tune delta for significance; every replacement is biased.", diff --git a/frontend/src/components/differentialTable.test.ts b/frontend/src/components/differentialTable.test.ts index 523e893b..07e3f1dd 100644 --- a/frontend/src/components/differentialTable.test.ts +++ b/frontend/src/components/differentialTable.test.ts @@ -193,3 +193,34 @@ describe('advanced compositional methods (ancom_bc, aldex2, songbird)', () => { expect(configLine(sb)).toBe('method songbird, min_prevalence 0.2; estimate is a CLR-centred multinomial differential, not an absolute fold change') }) }) + +describe('calibration caveat', () => { + const caveated: DifferentialResult = { + ...result, + config: { method: 'nb_glm', offset: 'tss', dispersion: 'ml', min_prevalence: 0 }, + diagnostics: { + ...result.diagnostics, + calibration_caveat: { + severity: 'strong', dispersion: 'ml', min_group_n: 2, + message: 'Wald p-values treat the per-taxon theta as known.', + }, + }, + } + + test('the settings line records the dispersion the result carries', () => { + expect(configLine(caveated)).toBe('method nb_glm, offset tss, dispersion ml, min_prevalence 0') + }) + + test('the caveat travels into the CSV header, after the settings line', () => { + const lines = differentialCsv(caveated).trimEnd().split('\r\n') + expect(lines[2]).toBe(`# ${configLine(caveated)}`) + expect(lines[3]).toBe('# Calibration caveat (strong): Wald p-values treat the per-taxon theta as known.') + expect(lines[4]).toBe('taxon,status,log2_fold_change,estimate,standard_error,statistic,pvalue,padj,dispersion_theta,zero_process_log_odds,prevalence,note') + expect(lines).toHaveLength(7) + }) + + test('a result without a caveat keeps the three-line header', () => { + const lines = differentialCsv(result).trimEnd().split('\r\n') + expect(lines[3]).toBe('taxon,status,log2_fold_change,estimate,standard_error,statistic,pvalue,padj,dispersion_theta,zero_process_log_odds,prevalence,note') + }) +}) diff --git a/frontend/src/components/differentialTable.ts b/frontend/src/components/differentialTable.ts index f173314f..70dfae03 100644 --- a/frontend/src/components/differentialTable.ts +++ b/frontend/src/components/differentialTable.ts @@ -22,10 +22,14 @@ export function csvField(value: string | number | null): string { */ export function differentialCsv(result: DifferentialResult): string { const { reference, contrast } = result.groups + const caveat = result.diagnostics.calibration_caveat const header = [ `# Differential abundance: ${contrast} vs ${reference} (reference), rank ${result.rank}, table ${result.table}`, `# ${result.method}`, `# ${configLine(result)}`, + // The caveat travels with the numbers it qualifies: a downloaded table + // must not look better-calibrated than the result it came from. + ...(caveat ? [`# Calibration caveat (${caveat.severity}): ${caveat.message}`] : []), ].map(l => l.replace(/[\r\n]+/g, ' ')) const rows = result.rows.map(r => COLUMNS.map(c => csvField(r[c] ?? null)).join(',')) return [...header, COLUMNS.join(','), ...rows].join('\r\n') + '\r\n' diff --git a/src/analysis/differential.jl b/src/analysis/differential.jl index be010b84..aaec5ba7 100644 --- a/src/analysis/differential.jl +++ b/src/analysis/differential.jl @@ -1130,6 +1130,66 @@ _row(taxon, prevalence, note) = Dict{String,Any}( "pvalue" => nothing, "padj" => nothing, "dispersion_theta" => nothing, "prevalence" => prevalence) +""" + _calibration_caveat(dispersion, n_ref, n_con) -> Union{Dict{String,Any},Nothing} + +A result-level calibration caveat for the negative-binomial Wald tests, or +`nothing` when none is recorded. Every `"ml"` result carries one, scaled by +the smaller group size: `MASS::glm.nb` estimates theta taxon by taxon and the +fixed-scale Wald test then treats that estimate as known, ignoring its +sampling uncertainty (issue #193). The `"glmGamPoi"` dispersion trend shares +information across taxa, so its caveat is recorded only through 12 samples in +the smaller group; even there, this implementation plugs the trend into the +same fixed-theta Wald test rather than propagating trend uncertainty or using +glmGamPoi's quasi-likelihood F-test. The caveat changes no statistic: it is +provenance and interpretation guidance recorded with the result. +""" +function _calibration_caveat(dispersion::AbstractString, n_ref::Integer, n_con::Integer) + min_group_n = min(n_ref, n_con) + # "caution" runs through 20, the largest group size the bench measured: + # the ml Wald test was still anti-conservative there. Beyond the measured + # range the caveat drops to "info" but, for "ml", never disappears. + severity = min_group_n <= 4 ? "strong" : min_group_n <= 20 ? "caution" : "info" + if dispersion == "ml" + message = if severity == "strong" + "The smallest group has $min_group_n sample(s). With dispersion \"ml\", " * + "theta is estimated separately for each taxon and the fixed-scale Wald test " * + "then treats that estimate as known. Null simulations for issue #193 rejected " * + "substantially too often at 4 samples per group; treat small p-values and " * + "Benjamini-Hochberg adjusted p-values at this size as exploratory rather than " * + "confirmatory. See bench/null_calibration." + elseif severity == "caution" + "The smallest group has $min_group_n samples. With dispersion \"ml\", theta " * + "is estimated separately for each taxon and the fixed-scale Wald test then " * + "treats that estimate as known. Null simulations in bench/null_calibration " * + "(issue #193) found above-nominal rejection at every measured group size " * + "through 20 per group, so interpret borderline p-values and " * + "Benjamini-Hochberg adjusted p-values cautiously." + else + "With dispersion \"ml\", theta is estimated separately for each taxon and " * + "the fixed-scale Wald test treats that estimate as known; theta uncertainty is " * + "not propagated. Null simulations in bench/null_calibration (issue #193) " * + "found above-nominal rejection through 20 samples per group, the largest " * + "size measured; beyond that range the approximation is expected to improve, " * + "but it has not been measured there." + end + return Dict{String,Any}("severity" => severity, "dispersion" => "ml", + "min_group_n" => min_group_n, "message" => message) + elseif dispersion == "glmGamPoi" && min_group_n <= 12 + message = "The smallest group has $min_group_n sample(s). The glmGamPoi " * + "dispersion trend shares information across taxa and steadies theta " * + "relative to a separate maximum-likelihood estimate for every taxon, " * + "but this implementation still plugs the trend into a fixed-theta, " * + "fixed-scale Wald test. It does not propagate trend uncertainty and " * + "does not implement glmGamPoi's quasi-likelihood F-test, so interpret " * + "borderline p-values cautiously at this sample size. " * + "See bench/null_calibration and issue #193." + return Dict{String,Any}("severity" => severity, "dispersion" => "glmGamPoi", + "min_group_n" => min_group_n, "message" => message) + end + nothing +end + # Two-group ANCOM-BC (Lin & Peddada 2020, Nat. Commun. 11:3514): # (1) Estimate within-group sample-specific sampling-fraction offsets on # log(counts + 1) by inverse-variance-weighted least squares, centred within @@ -1993,7 +2053,12 @@ enter each negative-binomial model as `log(factor)`; the estimate is a natural-log fold change. `config.dispersion` selects per-taxon maximum-likelihood theta (`"ml"`) or fixed theta from `glmGamPoi::glm_gp`'s cross-taxon dispersion trend (`"glmGamPoi"`), fitted in one call on the taxa passing the prevalence filter. -The result's `config` includes the selected dispersion name. +The result's `config` includes the selected dispersion name. Its `diagnostics` +also records a `calibration_caveat` for every `"ml"` result (scaled by the +smaller group) and for a `"glmGamPoi"` result whose smaller group has at most +12 samples: both fixed-theta Wald paths treat the dispersion used by the test +as known rather than propagating its estimation uncertainty. The caveat alters +no estimate, p-value or adjusted p-value. `clr_welch`, in this order: (1) `min_prevalence` chooses the taxa, and taxa with no reads in any sample are always left out; (2) the zeros of those taxa only @@ -2090,6 +2155,8 @@ function differential_abundance(counts::AbstractMatrix{<:Real}, config_echo = Dict{String,Any}("method" => config.method, "offset" => config.offset, "min_prevalence" => config.min_prevalence, "dispersion" => config.dispersion) + caveat = _calibration_caveat(config.dispersion, n_ref, n_con) + isnothing(caveat) || (diagnostics["calibration_caveat"] = caveat) elseif config.method == "clr_welch" (n_ref >= 2 && n_con >= 2) || throw(ArgumentError( "Welch's t-test needs at least 2 samples in each group to estimate its variance; " * diff --git a/src/server/server.jl b/src/server/server.jl index 44f916f4..ebf5abb6 100644 --- a/src/server/server.jl +++ b/src/server/server.jl @@ -116,6 +116,7 @@ module Server include(joinpath(@__DIR__, "routes", "doi.jl")) include(joinpath(@__DIR__, "routes", "figures.jl")) include(joinpath(@__DIR__, "routes", "phylogeny.jl")) + include(joinpath(@__DIR__, "routes", "cladistics.jl")) include(joinpath(@__DIR__, "routes", "clade_cumulus.jl")) include(joinpath(@__DIR__, "routes", "cladistics.jl")) diff --git a/test/unit/test_differential.jl b/test/unit/test_differential.jl index 234437dc..b8543eff 100644 --- a/test/unit/test_differential.jl +++ b/test/unit/test_differential.jl @@ -220,6 +220,23 @@ end @test_throws ArgumentError replace_zeros([0 1; 2 3]; method="pseudocount") end + @testset "Calibration caveat scales with the smaller group" begin + # Pure thresholds, no R needed: every "ml" result is caveated, the + # glmGamPoi trend only through 12 in the smaller group. + caveat = Differential._calibration_caveat + @test caveat("ml", 4, 4)["severity"] == "strong" + @test caveat("ml", 30, 3)["severity"] == "strong" + @test caveat("ml", 5, 5)["severity"] == "caution" + @test caveat("ml", 12, 12)["severity"] == "caution" + @test caveat("ml", 13, 20)["severity"] == "caution" + @test caveat("ml", 13, 20)["min_group_n"] == 13 + @test caveat("ml", 21, 25)["severity"] == "info" + @test caveat("glmGamPoi", 4, 9)["severity"] == "strong" + @test caveat("glmGamPoi", 12, 12)["severity"] == "caution" + @test isnothing(caveat("glmGamPoi", 13, 13)) + @test isnothing(caveat("glmGamPoi", 20, 13)) + end + @testset "Input validation" begin taxa = ["a", "b", "c"] groups = ["A", "A", "B"] @@ -258,6 +275,9 @@ end "TSS size-factor offset; Wald test of the group coefficient; " * "Benjamini-Hochberg adjustment over the fitted taxa" @test res["diagnostics"]["n_tested"] == 3 + caveat = res["diagnostics"]["calibration_caveat"] + @test caveat["severity"] == "caution" && caveat["dispersion"] == "ml" + @test caveat["min_group_n"] == 12 && occursin("theta", caveat["message"]) @test all(r -> r["status"] == "ok", res["rows"]) @test isapprox(byt["t_up"]["estimate"], log(4); atol=0.5) @test isapprox(byt["t_up"]["log2_fold_change"], 2; atol=0.8) @@ -271,6 +291,13 @@ end reference="B", contrast="A") sb = Dict(r["taxon"] => r for r in swapped["rows"]) @test sb["t_up"]["estimate"] ≈ -byt["t_up"]["estimate"] rtol=1e-6 + # The caveat strengthens as the smaller group shrinks; the p-values + # themselves are untouched by it (checked against MASS below). + keep = [1:4; 13:16] + small = differential_abundance(effect[keep, :], effect_samples[keep], effect_taxa, + effect_groups[keep]; reference="A", contrast="B") + @test small["diagnostics"]["calibration_caveat"]["severity"] == "strong" + @test small["diagnostics"]["calibration_caveat"]["min_group_n"] == 4 @testset "matches a direct MASS::glm.nb call" begin off = log.(tss_factors(effect, effect_samples)) @@ -308,6 +335,10 @@ end "with dispersion trend from glmGamPoi::glm_gp, log link) with a " * "TSS size-factor offset; Wald test of the group coefficient; " * "Benjamini-Hochberg adjustment over the fitted taxa" + caveat = res["diagnostics"]["calibration_caveat"] + @test caveat["severity"] == "caution" && caveat["dispersion"] == "glmGamPoi" + @test caveat["min_group_n"] == 12 && + occursin("quasi-likelihood F-test", caveat["message"]) byt = Dict(r["taxon"] => r for r in res["rows"]) off = log.(tss_factors(effect, effect_samples)) g = effect_groups @@ -876,6 +907,7 @@ end @test isempty(res["size_factors"]) @test res["config"]["method"] == "clr_welch" && !haskey(res["config"], "offset") @test res["diagnostics"]["zero_replacement"]["zeros_replaced"] == 0 + @test !haskey(res["diagnostics"], "calibration_caveat") @test occursin("Welch", res["method"]) && occursin("cmultRepl", res["method"]) # Known answer, independent of R: the estimate is mean(B) - mean(A) # of each taxon's CLR.