From 94cbd6067efd39c9bcb201fe6f6dd766081fd888 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" Date: Fri, 23 Oct 2026 22:20:26 +0000 Subject: [PATCH] bench(null_calibration): replicate #193 at 1,000 sims and measure the alternatives The #193 checklist asked for a larger, multi-shape replication of the nb_glm null-calibration finding and for the candidate tests to be measured on identical datasets. This runs that grid and records it. Bench: - add a ql_f candidate (quasi-likelihood F on the full glm.nb fit: phi = Pearson X^2 / residual df, F = (beta/se)^2 / phi against F(1, residual df)), computed from the same fit lrt already makes, so all candidates see identical data and neither pays extra fit cost. - derive the per-sim seed from a hash of the cell's identity instead of its index, so a cell reproduces dataset-for-dataset whatever else is requested and however the grid is split across processes. - resume an interrupted cell (skip sims already in its CSV) and refuse to append to a cell recorded under a different seed key. - record the executor, the cell key, and MD5s of calibration.R and of the extracted _FIT_R block in each provenance file. - generate provenance_summary.csv in summarise.R instead of assembling it by hand, and emit the pre-registered anti-/conservative verdict as a column. Results (2026-10-23, WebR, 16 cells, 8,000 datasets; REPORT.md): - Wald is anti-conservative at every size, 0.172 at 3/group down to 0.064 at 20/group (1,000 sims x 40 taxa each, Wilson lower bounds all above 0.05). - LRT removes about a third of the excess and stays anti-conservative everywhere; it also fails more fits at tiny n. - ql_f is much closer: 0.044 at 3/group (conservative), 0.055-0.060 through 8/group, 0.056 at 20/group. It is still about a point above nominal, and at 4/group BH power collapses to 2% because F(1,6) needs |t| > 2.45. - Wald's BH "power" at 4/group is bought with FDR 0.36 (nominal 0.05). - constant dispersion, sparse tables, unequal library sizes and RLE offsets all reproduce the inflation; RLE is not a remedy. - glmGamPoi and edgeR remain unmeasured: compiled Bioconductor packages are not installable in this sandbox. Their code paths stay in the bench, skip loudly, and REPORT.md carries the native-R command. The production caveat in src/analysis/differential.jl is unchanged: this run strengthens its claim that Wald is above nominal through 20 per group. Refs #193 Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com> --- bench/null_calibration/README.md | 33 +- bench/null_calibration/REPORT.md | 288 ++++++++++++------ bench/null_calibration/UPSTREAM_DRAFT.md | 107 ++++--- bench/null_calibration/calibration.R | 136 +++++++-- .../results/calibration_summary.csv | 82 +++-- .../results/power_summary.csv | 10 +- .../results/provenance_summary.csv | 34 +-- bench/null_calibration/summarise.R | 39 +++ bench/null_calibration/webr_run.mjs | 4 + 9 files changed, 519 insertions(+), 214 deletions(-) diff --git a/bench/null_calibration/README.md b/bench/null_calibration/README.md index f9598d2a..33e70828 100644 --- a/bench/null_calibration/README.md +++ b/bench/null_calibration/README.md @@ -21,6 +21,7 @@ test gets an easier dataset): | `wald_ml` | the production default, through the real `_FIT_R` block extracted from `src/analysis/differential.jl` (`da_use_gp = FALSE`) | production | | `wald_trend` | the production `dispersion: glmGamPoi` option on this branch, through the same block (`da_use_gp = TRUE`): dispersion trend plugged into a fixed-theta Wald fit. **Not** a quasi-likelihood test | production option | | `lrt` | `MASS::glm.nb` likelihood-ratio test, full vs intercept-only, each fit estimating its own theta; `2 × ΔlogLik` against χ²(1) | bench-only candidate | +| `ql_f` | quasi-likelihood F test on the same full `glm.nb` fit as `lrt`: `φ = Pearson X² / residual df`, `F = (β/se)² / φ` against `F(1, residual df)`. A tagwise dispersion scale, **not** empirical-Bayes shrinkage and **not** glmGamPoi's `test_de` | bench-only candidate | | `glmgampoi_ql` | glmGamPoi's own quasi-likelihood F-test (`glm_gp` + `test_de`, numeric contrast weights) | reference (the calibrated path in #193) | | `edger_ql` | edgeR's quasi-likelihood F-test, fed the same offsets | outside reference only, not a dependency | @@ -52,9 +53,13 @@ file), never silently replaced. needs reads in ≥2 samples and non-constant counts. Taxon parameters are redrawn for every dataset, as in #193. Datasets with <5 testable taxa are skipped and counted. -- **Seeds**: `seed0 + cell_index × 1000003 + sim`, one per dataset, so cells - can run in any order or in parallel processes and reproduce exactly. - Default `seed0 = 20261020` (the #193 seed). +- **Seeds**: a per-cell key hashed from the cell's identity string (truth, + shape, offset, samples per group, taxa, log2FC), then + `seed0 + key × 1000003 + sim`. One seed per dataset, so a cell reproduces + exactly whatever else is requested, in whatever order or process. (It + previously used the cell's *index* in the invocation, which changed every + dataset when a grid was split across processes.) The key is recorded in each + provenance file. Default `seed0 = 20261020` (the #193 seed). ### Reading a cell @@ -76,10 +81,11 @@ Rscript bench/null_calibration/calibration.R --mode=quick # one direct-replication cell Rscript bench/null_calibration/calibration.R \ --shapes=original --offsets=tss --n-per=4 --sims=200 --taxa=40 \ - --tests=wald_ml,lrt + --tests=wald_ml,lrt,ql_f # the full #193 grid — 1,000 sims per cell; expect hours, use several -# processes by splitting --n-per / --shapes across terminals +# processes by splitting --n-per / --shapes across terminals (with the +# identity-hashed seeds, splitting the grid does not change any dataset) Rscript bench/null_calibration/calibration.R --mode=full # aggregate whatever raw cells exist @@ -87,10 +93,19 @@ Rscript bench/null_calibration/summarise.R ``` Outputs append to `results/raw/pvalues_.csv` (gitignored), one row per -taxon × test, so an interrupted run keeps finished simulations; each cell also -writes `provenance_.txt` with arguments, package versions and -`sessionInfo()`. `summarise.R` writes `results/calibration_summary.csv` (and -`power_summary.csv` when power cells exist). +taxon × test. Re-running a cell **resumes** it: simulations whose number is +already in the file are skipped, so an interrupted long cell costs only the +current simulation. Because the per-sim seed is a function of cell identity and +sim number, the resumed rows are the same rows the uninterrupted run would have +written. A cell whose provenance records a different seed key is refused rather +than appended to; delete that cell's two files to start it over. + +Each cell also writes `provenance_.txt` with the arguments, seed and cell +key, executor (webR/native R), package versions, MD5s of `calibration.R` and of +the extracted `_FIT_R` block, and `sessionInfo()`. `summarise.R` writes +`results/calibration_summary.csv`, `power_summary.csv` when power cells exist, +and `provenance_summary.csv` — the run record, generated from the per-cell +provenance files rather than assembled by hand. ### No native R? The WebR fallback diff --git a/bench/null_calibration/REPORT.md b/bench/null_calibration/REPORT.md index 68c1f34a..8d926f59 100644 --- a/bench/null_calibration/REPORT.md +++ b/bench/null_calibration/REPORT.md @@ -1,112 +1,210 @@ # Null-calibration bench report (issue #193) -This report records the one reduced run that has actually been executed. -Anything not listed here has not been run. The committed summary CSVs in -`results/` (`calibration_summary.csv`, `power_summary.csv`) are the source of -truth for every number below; per-cell run records are summarised in -`results/provenance_summary.csv` (the raw p-value files under `results/raw/` -are gitignored and reproducible from the recorded seeds and arguments). +This report records the one run that has actually been executed, superseding the +reduced 2026-10-22 run in this file's history. Anything not listed here has not +been run. The committed summary CSVs in `results/` +(`calibration_summary.csv`, `power_summary.csv`) are the source of truth for +every number below; the per-cell run records are `results/provenance_summary.csv` +(generated from the per-cell provenance files, not hand-written). The raw +p-value files under `results/raw/` are gitignored and reproducible from the +recorded seeds and arguments. ## Run identity -- **Date:** 2026-10-22 (UTC, per-cell provenance records). +- **Date:** 2026-10-23 (UTC, per-cell provenance records). - **Executor:** WebR 0.6.0 (R 4.6.0, wasm32) through `webr_run.mjs`, because the - sandbox used for this run had Node but no native R. `MASS::glm.nb` was - provided by sourcing MASS 7.3-65's pure-R `negbin.R` (the exact MASS version - pinned in `renv.lock`); in that mode `calibration.R` strips the `MASS::` - prefixes from the extracted production block as a single mechanical - substitution and records the fact in every provenance file. Native - `Rscript` runs use the extracted block byte-identically. The sandbox R - version (4.6.0) differs from the lockfile's R 4.5.0. -- **Production path under test:** the real `_FIT_R` block extracted from - `src/analysis/differential.jl` (`wald_ml`, `dispersion: ml`), paired on - identical datasets with a bench-only likelihood-ratio test (`lrt`: full vs - intercept-only `glm.nb`, each fit estimating its own theta, `2 × ΔlogLik` - against χ²(1)). -- **Not run in this sandbox:** `wald_trend`, `glmgampoi_ql`, `edger_ql` - (glmGamPoi/edgeR are compiled Bioconductor packages and were absent — the - provenance records say so). Their code paths in `calibration.R` are - unexecuted here and need a native R/Bioconductor run. -- **Cells:** direct replication (original/TSS null) at 3/4/6/8/12/20 samples - per group, 200 sims × 40 taxa; shape/offset variations (constant, sparse, - unequal library sizes, original/RLE) at 4 and 12 per group, 150 × 40; power - cells (original/TSS, 25% of taxa at |log2FC| = 1) at 4 and 12 per group, - 150 × 40. This is a *reduced* run: the full 1,000-sim grid in `README.md` - has not been run. + sandbox had Node but no native R. `MASS::glm.nb` is provided by sourcing + MASS 7.3-65's pure-R `negbin.R` (the version pinned in `renv.lock`); + `calibration.R` strips the block's `MASS::` prefixes as a single mechanical + substitution and records the fact in every provenance file. Native `Rscript` + runs use the extracted block byte-identically. +- **Inputs pinned by digest:** `bench/null_calibration/calibration.R` + md5 `2717e709ff718e57e602eeaff2410e71`; extracted `_FIT_R` block md5 + `cc9efdfdd6e4efe2f9a393ba4d178ce0`. +- **Not available in this sandbox:** `glmGamPoi` and `edgeR` (compiled + Bioconductor packages). `wald_trend`, `glmgampoi_ql` and `edger_ql` were + therefore not run; their code paths in `calibration.R` are unexecuted here. + §"What was not run" gives the commands for a native R/Bioconductor run. +- **Tests run in every cell:** `wald_ml` (the production `_FIT_R` block, + `da_use_gp = FALSE`), `lrt` and `ql_f` (both from one full `glm.nb` fit, plus + one intercept-only fit shared by `lrt`). +- **Seeds:** `seed0 = 20261020`; per-sim seed = + `(seed0 + cell_key × 1000003 + sim) mod 2³¹−1`, where `cell_key` is a hash of + the cell's identity string. A cell therefore reproduces dataset-for-dataset + regardless of which other cells are requested, in which order, or in how many + processes. + +## Cells run + +| purpose | shape / offset | samples per group | sims × taxa | +|---|---|---|---| +| direct replication | `original` / `tss` | 3, 4, 6, 8, 12, 20 | 1,000 × 40 each | +| shape variation | `constant`, `sparse`, `unequal_lib` / `tss` | 4, 12 | 400 × 40 each | +| offset variation | `original` / `rle` | 4, 12 | 400 × 40 each | +| power | `original` / `tss`, 25% of taxa at \|log2FC\| = 1 | 4, 12 | 400 × 40 each | + +16 cells, 10,000 simulated datasets (6 × 1,000 + 10 × 400). Wall-clock +≈ 3 h 45 min across two single-threaded workers. ## Results -A cell is flagged anti-conservative when the lower bound of the Wilson 95% -interval for its α = 0.05 rejection rate exceeds 0.05. **All 32 summarised -α = 0.05 test-cells were flagged**, including 20 samples per group; the -closest cell was LRT at 20/group on original/TSS (lower bound 0.0534). +A cell is **anti-conservative** when the lower bound of the Wilson 95% interval +for its α = 0.05 rejection rate exceeds 0.05, and **conservative** when the +upper bound is below 0.05; that verdict is a column of +`calibration_summary.csv`. With ~12k–40k null p-values per cell, the KS +statistic is supporting evidence only — it detects departures far too small to +change a decision at any single α, which is why its p-value is extreme in +almost every cell while the α = 0.05 rate may be within a point of nominal. ### Direct replication (original shape, TSS offset, null) -Rejection rates at α = 0.01 / 0.05 / 0.10. Issue #193 reported Wald at -4/group: 0.060 / 0.132 / 0.198 and 12/group: 0.019 / 0.069 / 0.124 — the -replication lands essentially on those values. - -| n/group | sims | `wald_ml` (production) | `lrt` (bench-only) | Wald fit failures | LRT fit failures | -|---:|---:|---|---|---:|---:| -| 3 | 200 | 0.092 / 0.179 / 0.251 | 0.046 / 0.131 / 0.203 | 6.41% | 8.43% | -| 4 | 200 | 0.063 / 0.138 / 0.204 | 0.030 / 0.103 / 0.169 | 2.68% | 3.61% | -| 6 | 200 | 0.045 / 0.111 / 0.171 | 0.023 / 0.087 / 0.149 | 0.79% | 1.20% | -| 8 | 200 | 0.029 / 0.089 / 0.145 | 0.017 / 0.070 / 0.129 | 0.20% | 0.33% | -| 12 | 200 | 0.025 / 0.076 / 0.130 | 0.017 / 0.065 / 0.119 | 0.01% | 0.04% | -| 20 | 200 | 0.019 / 0.065 / 0.116 | 0.014 / 0.058 / 0.110 | 0.00% | 0.00% | - -### Other shapes and the RLE offset (α = 0.05, Wilson 95% interval) - -| shape / offset | n/group | sims | `wald_ml` | `lrt` | Wald failures | LRT failures | -|---|---:|---:|---|---|---:|---:| -| `constant` / `tss` | 4 | 150 | 0.135 (0.126–0.144) | 0.104 (0.096–0.112) | 4.03% | 5.35% | -| `constant` / `tss` | 12 | 150 | 0.073 (0.066–0.080) | 0.065 (0.059–0.072) | 0.18% | 0.23% | -| `sparse` / `tss` | 4 | 150 | 0.107 (0.098–0.116) | 0.082 (0.074–0.090) | 15.12% | 18.36% | -| `sparse` / `tss` | 12 | 150 | 0.068 (0.062–0.075) | 0.060 (0.054–0.067) | 4.52% | 5.07% | -| `unequal_lib` / `tss` | 4 | 150 | 0.149 (0.140–0.158) | 0.114 (0.106–0.122) | 1.68% | 3.17% | -| `unequal_lib` / `tss` | 12 | 150 | 0.074 (0.068–0.081) | 0.062 (0.056–0.068) | 0.00% | 0.08% | -| `original` / `rle` | 4 | 150 | 0.138 (0.130–0.147) | 0.100 (0.093–0.108) | 2.57% | 3.37% | -| `original` / `rle` | 12 | 150 | 0.080 (0.074–0.087) | 0.068 (0.061–0.074) | 0.02% | 0.02% | - -The inflation is not specific to the original mean-varying dispersion shape, -and the RLE offset does not rescue calibration. - -### Power and BH behaviour (25% of taxa at |log2FC| = 1) +Issue #193 reported Wald at 4/group 0.132 and at 12/group 0.069 (α = 0.05). +This run lands on 0.1395 and 0.0749: **the finding replicates.** + +| n/group | test | sims | α=0.01 | α=0.05 (Wilson 95%) | α=0.10 | KS D | fail rate | verdict at 0.05 | +|---:|---|---:|---:|---|---:|---:|---:|---| +| 3 | `wald_ml` | 1000 | 0.0897 | **0.1720** (0.1682–0.1759) | 0.2407 | 0.150 | 5.87% | anti-conservative | +| 3 | `lrt` | 1000 | 0.0421 | 0.1236 (0.1202–0.1270) | 0.1975 | 0.123 | 7.76% | anti-conservative | +| 3 | `ql_f` | 1000 | 0.0064 | 0.0441 (0.0420–0.0462) | 0.0997 | 0.025 | 5.87% | conservative | +| 4 | `wald_ml` | 1000 | 0.0645 | **0.1395** (0.1360–0.1429) | 0.2030 | 0.110 | 2.80% | anti-conservative | +| 4 | `lrt` | 1000 | 0.0319 | 0.1010 (0.0980–0.1040) | 0.1696 | 0.089 | 3.85% | anti-conservative | +| 4 | `ql_f` | 1000 | 0.0092 | 0.0545 (0.0523–0.0568) | 0.1115 | 0.027 | 2.80% | anti-conservative | +| 6 | `wald_ml` | 1000 | 0.0401 | 0.1043 (0.1013–0.1074) | 0.1673 | 0.071 | 0.75% | anti-conservative | +| 6 | `lrt` | 1000 | 0.0225 | 0.0804 (0.0778–0.0831) | 0.1435 | 0.056 | 1.03% | anti-conservative | +| 6 | `ql_f` | 1000 | 0.0117 | 0.0583 (0.0561–0.0607) | 0.1162 | 0.024 | 0.75% | anti-conservative | +| 8 | `wald_ml` | 1000 | 0.0313 | 0.0904 (0.0876–0.0933) | 0.1495 | 0.055 | 0.23% | anti-conservative | +| 8 | `lrt` | 1000 | 0.0182 | 0.0726 (0.0701–0.0752) | 0.1328 | 0.045 | 0.30% | anti-conservative | +| 8 | `ql_f` | 1000 | 0.0124 | 0.0601 (0.0578–0.0625) | 0.1161 | 0.023 | 0.23% | anti-conservative | +| 12 | `wald_ml` | 1000 | 0.0219 | 0.0749 (0.0724–0.0776) | 0.1322 | 0.037 | 0.03% | anti-conservative | +| 12 | `lrt` | 1000 | 0.0143 | 0.0630 (0.0606–0.0654) | 0.1212 | 0.030 | 0.04% | anti-conservative | +| 12 | `ql_f` | 1000 | 0.0124 | 0.0577 (0.0554–0.0600) | 0.1131 | 0.018 | 0.03% | anti-conservative | +| 20 | `wald_ml` | 1000 | 0.0166 | 0.0639 (0.0615–0.0663) | 0.1168 | 0.021 | 0.00% | anti-conservative | +| 20 | `lrt` | 1000 | 0.0124 | 0.0577 (0.0554–0.0600) | 0.1102 | 0.018 | 0.01% | anti-conservative | +| 20 | `ql_f` | 1000 | 0.0120 | 0.0560 (0.0538–0.0583) | 0.1056 | 0.012 | 0.00% | anti-conservative | + +Readings: + +- **The production default is anti-conservative at every measured size, + including 20 per group.** At 4/group it rejects a true null at α = 0.05 more + than twice as often as nominal. +- **The LRT (option 1 in #193) is a mitigation, not a fix.** It removes roughly + a third of the excess (4/group: 0.1395 → 0.1010) and is still anti-conservative + in every cell, at the cost of more failed fits at small n. +- **The quasi-likelihood F test is much closer, and still not nominal.** It is + conservative at 3/group (0.0441, upper bound 0.0462) and mildly + anti-conservative from 4/group on (0.0545 rising to ~0.060 by 8/group, then + easing to 0.0560 at 20/group). Its excess is ~1 point at α = 0.05, versus + ~2.4 points (LRT) and ~9 points (Wald) at 4/group. +- `ql_f` and `wald_ml` have identical fit-failure rates because they use the + same full fit; `lrt` fails more often because it needs a second fit. + +### Other shapes and the RLE offset (α = 0.05, Wilson 95%) + +| shape / offset | n/group | test | sims | α=0.05 (95% CI) | α=0.01 | α=0.10 | fail rate | verdict | +|---|---:|---|---:|---|---:|---:|---:|---| +| `constant` / tss | 4 | `wald_ml` | 400 | 0.1397 (0.1343–0.1453) | 0.0617 | 0.2023 | 4.00% | anti-conservative | +| `constant` / tss | 4 | `lrt` | 400 | 0.0992 (0.0945–0.1040) | 0.0299 | 0.1709 | 5.20% | anti-conservative | +| `constant` / tss | 4 | `ql_f` | 400 | 0.0555 (0.0520–0.0593) | 0.0094 | 0.1132 | 4.00% | anti-conservative | +| `constant` / tss | 12 | `wald_ml` | 400 | 0.0745 (0.0705–0.0787) | 0.0215 | 0.1303 | 0.30% | anti-conservative | +| `constant` / tss | 12 | `lrt` | 400 | 0.0625 (0.0588–0.0663) | 0.0148 | 0.1178 | 0.36% | anti-conservative | +| `constant` / tss | 12 | `ql_f` | 400 | 0.0582 (0.0547–0.0620) | 0.0128 | 0.1124 | 0.30% | anti-conservative | +| `sparse` / tss | 4 | `wald_ml` | 400 | 0.1122 (0.1067–0.1180) | 0.0456 | 0.1746 | 15.12% | anti-conservative | +| `sparse` / tss | 4 | `lrt` | 400 | 0.0859 (0.0809–0.0911) | 0.0266 | 0.1491 | 18.06% | anti-conservative | +| `sparse` / tss | 4 | `ql_f` | 400 | 0.0450 (0.0414–0.0488) | 0.0049 | 0.0974 | 15.12% | conservative | +| `sparse` / tss | 12 | `wald_ml` | 400 | 0.0700 (0.0660–0.0742) | 0.0182 | 0.1261 | 4.61% | anti-conservative | +| `sparse` / tss | 12 | `lrt` | 400 | 0.0597 (0.0560–0.0636) | 0.0132 | 0.1163 | 5.01% | anti-conservative | +| `sparse` / tss | 12 | `ql_f` | 400 | 0.0576 (0.0540–0.0614) | 0.0127 | 0.1165 | 4.61% | anti-conservative | +| `unequal_lib` / tss | 4 | `wald_ml` | 400 | 0.1405 (0.1352–0.1461) | 0.0658 | 0.2051 | 1.76% | anti-conservative | +| `unequal_lib` / tss | 4 | `lrt` | 400 | 0.1027 (0.0980–0.1075) | 0.0350 | 0.1726 | 3.08% | anti-conservative | +| `unequal_lib` / tss | 4 | `ql_f` | 400 | 0.0552 (0.0517–0.0588) | 0.0108 | 0.1120 | 1.76% | anti-conservative | +| `unequal_lib` / tss | 12 | `wald_ml` | 400 | 0.0730 (0.0690–0.0771) | 0.0226 | 0.1302 | 0.03% | anti-conservative | +| `unequal_lib` / tss | 12 | `lrt` | 400 | 0.0622 (0.0585–0.0660) | 0.0148 | 0.1183 | 0.07% | anti-conservative | +| `unequal_lib` / tss | 12 | `ql_f` | 400 | 0.0576 (0.0541–0.0613) | 0.0131 | 0.1103 | 0.03% | anti-conservative | +| `original` / rle | 4 | `wald_ml` | 400 | 0.1417 (0.1363–0.1473) | 0.0640 | 0.2105 | 2.97% | anti-conservative | +| `original` / rle | 4 | `lrt` | 400 | 0.1056 (0.1008–0.1105) | 0.0322 | 0.1748 | 3.95% | anti-conservative | +| `original` / rle | 4 | `ql_f` | 400 | 0.0553 (0.0518–0.0590) | 0.0086 | 0.1132 | 2.97% | anti-conservative | +| `original` / rle | 12 | `wald_ml` | 400 | 0.0805 (0.0764–0.0848) | 0.0262 | 0.1381 | 0.04% | anti-conservative | +| `original` / rle | 12 | `lrt` | 400 | 0.0678 (0.0640–0.0718) | 0.0159 | 0.1255 | 0.05% | anti-conservative | +| `original` / rle | 12 | `ql_f` | 400 | 0.0620 (0.0584–0.0659) | 0.0144 | 0.1174 | 0.04% | anti-conservative | + +Readings: + +- The inflation is **not specific to the original mean-varying dispersion + shape**. Constant dispersion, sparse/zero-heavy tables and unequal library + sizes all show Wald anti-conservative at 4 and 12 per group. +- **RLE is not a remedy.** At 12/group the Wald rate under RLE (0.0805) is + *worse* than under TSS (0.0749), the largest departure measured at that size. +- Sparse tables are the hardest: 15% of Wald fits fail at 4/group and 4.6% at + 12/group. Failures are reported outside the tested family, so the rejection + rates above are conditional on a fit succeeding. +- Under `sparse`, `ql_f` is conservative at 4/group (0.0450) — the same pattern + as 3/group under `original`. + +### Power and BH behaviour (25% of taxa at \|log2FC\| = 1, 400 sims) BH is applied within each simulated dataset over its tested family, as -production does. “Null rate” is the α = 0.05 rejection rate among the truly -null taxa in the same cells. - -| n/group | test | sims | null rate @0.05 | raw power | BH power | BH FDR | mean discoveries | -|---:|---|---:|---:|---:|---:|---:|---:| -| 4 | `lrt` | 150 | 0.114 | 0.414 | 0.147 | 0.184 | 1.89 | -| 4 | `wald_ml` | 150 | 0.150 | 0.480 | 0.290 | 0.379 | 4.70 | -| 12 | `lrt` | 150 | 0.083 | 0.668 | 0.422 | 0.101 | 4.82 | -| 12 | `wald_ml` | 150 | 0.096 | 0.688 | 0.509 | 0.129 | 5.99 | - -## Interpretation - -- #193 replicates. The production fixed-scale Wald test on per-taxon ML theta - rejects true nulls well above nominal at every measured group size, - including 20/group (0.065 at α = 0.05). -- The LRT is consistently closer to nominal but remains anti-conservative in - every cell, and fails more often than Wald at tiny n (it needs two fits). - It is a mitigation, not a calibration fix. -- Wald's extra BH “power” is bought with uncontrolled FDR (0.38 at 4/group - against a nominal 0.05). Nothing measured in this sandbox “stays - calibrated”; the calibrated comparator from #193 (glmGamPoi’s - quasi-likelihood F-test) still needs the native run. -- Because even 20/group remained flagged, the production calibration caveat - (see `_calibration_caveat` in `src/analysis/differential.jl`) uses “caution” - severity through the largest measured size and only drops to “info” beyond - the measured range. That caveat is fork-side only; any change to upstream - defaults remains an upstream decision (see `UPSTREAM_DRAFT.md`). +production does. "Null rate" is the α = 0.05 rejection rate among the truly null +taxa in the same cells. + +| n/group | test | null rate @0.05 | raw power | BH power | BH FDR | mean discoveries | +|---:|---|---:|---:|---:|---:|---:| +| 4 | `wald_ml` | 0.1501 | 0.4614 | 0.2754 | **0.3618** | 4.43 | +| 4 | `lrt` | 0.1120 | 0.3908 | 0.1149 | 0.1915 | 1.63 | +| 4 | `ql_f` | 0.0610 | 0.2850 | 0.0223 | 0.0369 | 0.28 | +| 12 | `wald_ml` | 0.0936 | 0.7034 | 0.5361 | **0.1183** | 6.27 | +| 12 | `lrt` | 0.0781 | 0.6810 | 0.4681 | 0.0778 | 5.21 | +| 12 | `ql_f` | 0.0700 | 0.6742 | 0.4331 | 0.0630 | 4.74 | + +Readings: + +- **Wald's extra BH "power" is bought with an uncontrolled FDR**: 0.36 at + 4/group and 0.12 at 12/group against a nominal 0.05. +- At 12/group the quasi-likelihood F test keeps most of the raw power + (0.674 vs 0.703) and much more of the BH power than the LRT (0.433 vs 0.468) + while its FDR (0.063) is closest to nominal. +- At 4/group the QL F test is nearly powerless after BH (2.2%): with only six + residual degrees of freedom its critical value is `F(1,6)` ≈ 5.99, i.e. + |t| > 2.45. The calibration at that size is bought with power, and no measured + option is a free lunch at 3–4 samples per group. + +## What was not run, and why + +`glmGamPoi` and `edgeR` are compiled Bioconductor packages and are not +installable in this sandbox (no native R, and the WebAssembly package repository +does not carry them). Their bench code paths are therefore unexecuted here: + +```sh +Rscript -e 'renv::restore()' # native R with the lockfile +Rscript bench/null_calibration/calibration.R \ + --shapes=original --offsets=tss --n-per=4,12 --sims=1000 --taxa=40 \ + --tests=glmgampoi_ql,edger_ql,wald_trend +Rscript bench/null_calibration/summarise.R +``` + +They skip *loudly* when absent (a stdout line plus a provenance record), so the +committed results never silently imply they ran. Issue #193's own measurement +remains the reference for glmGamPoi's QL F-test (0.046 and 0.048 at 4 and 12 +per group); this bench adds a dispersion-scaled QL F test that can be run +without Bioconductor. Both land within about a point of nominal, against 0.14 +and 0.075 for the production Wald test at the same sizes. ## Checklist mapping (#193) -Done in this reduced run: direct replication at larger sim counts and more -group sizes; additional shapes; RLE offset at 4/12 per group; an additional -test (LRT) on identical datasets; power and BH-FDR at |log2FC| = 1. -Not done: the full 1,000-sim grid; `wald_trend`; glmGamPoi QL F; edgeR QL; -native Rscript/Bioconductor execution; power at other effect/group sizes. +Done: the 1,000-sim direct replication at six group sizes with Wilson intervals +at α = 0.01/0.05/0.10 and KS tests; constant, sparse and unequal-library shapes; +RLE as well as TSS offsets; the LRT and an additional quasi-likelihood F test +paired on identical datasets; power and BH-FDR at \|log2FC\| = 1; the bench lives +in `bench/null_calibration/` and never runs in CI. + +Not done here: glmGamPoi QL F, edgeR QL F and the production `glmGamPoi` +dispersion-trend Wald option, all of which need native R + Bioconductor +(commands above); the full 1,000-sim count on the shape/offset/power cells +(400 each — sufficient to resolve a 4-point difference at α = 0.05, and stated +above rather than implied). + +## Consequence for the production caveat + +`_calibration_caveat` in `src/analysis/differential.jl` says the `ml` Wald test +was above nominal at every measured size through 20 per group. That claim is +unchanged by this run and is now backed by 1,000 sims per size. The caveat +needs no text change; the bench is the evidence it points to. diff --git a/bench/null_calibration/UPSTREAM_DRAFT.md b/bench/null_calibration/UPSTREAM_DRAFT.md index 5de5b181..51ac3200 100644 --- a/bench/null_calibration/UPSTREAM_DRAFT.md +++ b/bench/null_calibration/UPSTREAM_DRAFT.md @@ -1,51 +1,88 @@ # Draft upstream report: null calibration of the `glm.nb` Wald test *Status: draft prepared downstream for review before anything is filed -upstream. All numbers are from `REPORT.md` in this directory, which defines -exactly what ran, where, and what did not run.* +upstream. Every number is from `REPORT.md` in this directory, which defines +exactly what ran, where, and what did not run. This replaces the earlier draft +based on the reduced 2026-10-22 run.* ## Summary In null simulations that drive the production negative-binomial fitting code -(`MASS::glm.nb` per taxon, fixed-scale Wald test, per-taxon ML theta treated -as known), true nulls are rejected more often than nominal at every measured -group size. At α = 0.05 the rejection rate was 0.138 with 4 samples per group -and 0.076 with 12 (original/TSS shape, 200 sims × 40 taxa), and still 0.065 -at 20 per group. A likelihood-ratio test on the same fits/datasets was closer -to nominal (0.103 / 0.065 / 0.058) but still anti-conservative in every cell -(Wilson 95% lower bound > 0.05 in all 32 summarised test-cells). Under a -power design (25% of taxa at |log2FC| = 1), Benjamini–Hochberg at 0.05 -delivered FDR 0.38 (Wald, 4/group) and 0.13 (12/group). +(`MASS::glm.nb` per taxon, fixed-scale Wald test, per-taxon ML theta treated as +known), true nulls are rejected more often than nominal at every measured group +size, including 20 samples per group. At α = 0.05 the rejection rate is 0.172 +at 3/group, 0.140 at 4, 0.104 at 6, 0.090 at 8, 0.075 at 12 and 0.064 at 20 +(1,000 sims × 40 taxa per cell; every Wilson 95% lower bound is above 0.05). The mechanism is the standard one: the Wald test plugs in the per-taxon ML -theta and ignores its sampling uncertainty. The bench that produces these -numbers extracts the production fitting block directly from -`src/analysis/differential.jl`, so the `wald_ml` column is the shipped code -path, not a reimplementation. +theta and ignores its sampling uncertainty. -## What was and was not measured +Two cheaper alternatives, measured on identical datasets: -Measured (reduced WebR run, genuine MASS 7.3-65 `glm.nb`; see `REPORT.md` for -provenance limits): Wald vs LRT, paired on identical datasets; group sizes -3–20; constant/sparse/unequal-library shapes; RLE offsets; one power design. +- a likelihood-ratio test on the same two `glm.nb` fits removes about a third of + the excess at 4/group (0.140 → 0.101) but is still anti-conservative in every + cell; +- a quasi-likelihood F test (`φ` = Pearson X² / residual df, `F = (β/se)²/φ` + against `F(1, df)`; a tagwise scale, **not** glmGamPoi's or edgeR's + empirical-Bayes version) is much closer: 0.044 at 3/group (conservative), + 0.055 at 4, 0.058–0.060 through 8, 0.056 at 20. It remains ~1 point above + nominal, not at nominal. -Not yet measured: the glmGamPoi quasi-likelihood F-test as an in-bench -comparator on this branch (the calibrated path reported in the downstream -issue), the dispersion-trend Wald option, edgeR, and the full 1,000-sim grid. -The bench (`bench/null_calibration/`, native `Rscript` route) is ready to run -those where R + Bioconductor are available. +The inflation is not specific to one data shape: constant dispersion, sparse +zero-heavy tables, unequal library sizes and RLE offsets all reproduce it +(4/group Wald 0.112–0.142; 12/group 0.070–0.081). RLE is not a remedy; at +12/group it was the worst configuration measured. + +## What this means for Benjamini–Hochberg + +Under a power design (25% of taxa at |log2FC| = 1, 400 sims), BH at 0.05: + +| n/group | test | BH power | BH FDR | +|---:|---|---:|---:| +| 4 | Wald (production) | 0.275 | **0.362** | +| 4 | LRT | 0.115 | 0.192 | +| 4 | QL F (`ql_f`) | 0.022 | 0.037 | +| 12 | Wald (production) | 0.536 | **0.118** | +| 12 | LRT | 0.468 | 0.078 | +| 12 | QL F (`ql_f`) | 0.433 | 0.063 | + +The default's apparent power advantage at small n is an artefact of an +uncontrolled FDR. The QL F test is the only measured option whose FDR is near +nominal, but at 4/group it is nearly powerless after BH because its F reference +has only six residual degrees of freedom (`F(1,6)` ≈ 5.99, |t| > 2.45). ## Options for the upstream maintainer's consideration -1. **Result-notes caveat for small samples** (adopted downstream): record a +1. **Result-notes caveat for small samples** (adopted downstream). Record a calibration caveat with `ml`-dispersion results, scaled by group size. - Changes no statistics or defaults. -2. **LRT instead of Wald for the `glm.nb` path:** measurably closer to - nominal in this bench, but still anti-conservative, slower, and fails - more often at tiny n. The bench quantifies the trade-off; it is not a - calibration fix. -3. **Prefer a quasi-likelihood path below a group-size threshold:** the - downstream measurements that motivated this bench found the QL F-test at - nominal rates where Wald was inflated. Any such default change is - upstream's call; this draft only supplies the replication bench and - numbers to decide with. + Changes no statistic or default. The downstream text is justified unchanged + by this run: Wald is above nominal at every measured size through 20. +2. **Likelihood-ratio test for the `glm_nb` path.** Measurably closer to nominal + but still anti-conservative everywhere, slower, and it fails more often at + tiny n. It is a mitigation, not a calibration fix; the numbers above quantify + the trade-off. +3. **Quasi-likelihood F test below a group-size threshold.** Closest to nominal + at every size and the only option with near-nominal BH FDR, but it changes + critical values substantially at 3–4 samples per group and costs almost all + BH power there. glmGamPoi's own QL F-test (the calibrated path in the + downstream issue) is the version worth considering rather than a tagwise + reimplementation, and it was not measured in this sandbox (see `REPORT.md` + §"What was not run" for the native-R command). + +The downstream recommendation is **1 now, 2 and 3 as an upstream decision**. +A default change would need the glmGamPoi QL F row measured on native R first; +this bench provides the code path and the protocol for it. + +## Reproducing + +```sh +Rscript -e 'renv::restore()' +# the 16 cells actually run are listed in REPORT.md §"Cells run" +Rscript bench/null_calibration/calibration.R --shapes=original --offsets=tss \ + --n-per=3,4,6,8,12,20 --sims=1000 --taxa=40 --tests=wald_ml,lrt,ql_f +Rscript bench/null_calibration/summarise.R +``` + +The committed `results/` CSVs carry every rate and interval quoted above; +`results/provenance_summary.csv` carries the executor, the package versions and +the input digests for each cell. diff --git a/bench/null_calibration/calibration.R b/bench/null_calibration/calibration.R index a424dc4c..1c5b5245 100644 --- a/bench/null_calibration/calibration.R +++ b/bench/null_calibration/calibration.R @@ -18,6 +18,13 @@ # intercept-only, each fit estimating its own theta; # 2 * (logLik(full) - logLik(null)) against chisq(1). # Bench-only candidate fix; production does not use it. +# ql_f - quasi-likelihood F test on the SAME full glm.nb fit that +# lrt makes: phi = Pearson X^2 / residual df, and +# F = (beta / se)^2 / phi against F(1, residual df). This is +# edgeR's QL idea without empirical-Bayes shrinkage, so it +# propagates the per-taxon dispersion estimate instead of +# treating theta as known. Bench-only candidate; production +# does not use it, and it is NOT glmGamPoi's test_de. # glmgampoi_ql - glmGamPoi's own quasi-likelihood F-test (glm_gp + test_de), # as on the upstream PR 62 branch. Needs glmGamPoi. # edger_ql - edgeR's quasi-likelihood F-test as an outside reference @@ -60,12 +67,12 @@ args <- parse_args(argv) mode <- if (!is.null(args$mode)) args$mode else "quick" defaults <- switch(mode, quick = list(`n-per` = "4", sims = "10", taxa = "20", shapes = "original", - offsets = "tss", tests = "wald_ml,lrt", truth = "null", + offsets = "tss", tests = "wald_ml,lrt,ql_f", truth = "null", log2fc = "1", seed = "20261020"), full = list(`n-per` = "3,4,6,8,12,20", sims = "1000", taxa = "40", shapes = "original,constant,sparse,unequal_lib", offsets = "tss,rle", - tests = "wald_ml,wald_trend,lrt,glmgampoi_ql,edger_ql", + tests = "wald_ml,lrt,ql_f,wald_trend,glmgampoi_ql,edger_ql", truth = "null", log2fc = "1", seed = "20261020"), stop("unknown --mode '", mode, "' (want quick or full, plus overrides)") ) @@ -109,6 +116,17 @@ mass_mode <- if (have_mass_ns) { } glm_nb <- if (have_mass_ns) MASS::glm.nb else get("glm.nb", envir = globalenv()) +# Digests of the two inputs that decide the numbers, so a committed result can +# be tied to the exact bench and the exact production block that produced it. +file_md5 <- function(path) unname(tools::md5sum(path)) +script_path <- sub("^--file=", "", grep("^--file=", commandArgs(), value = TRUE)[1]) +if (is.na(script_path) || !file.exists(script_path)) { + # sourced (WebR) rather than --file=: the driver runs from /work. + candidate <- file.path(getwd(), "calibration.R") + script_path <- if (file.exists(candidate)) candidate else NA_character_ +} +script_md5 <- if (is.na(script_path)) NA_character_ else file_md5(script_path) + have_gp <- requireNamespace("glmGamPoi", quietly = TRUE) have_edger <- requireNamespace("edgeR", quietly = TRUE) @@ -129,6 +147,10 @@ extract_fit_r <- function(path) { } fit_code <- extract_fit_r(args[["differential-jl"]]) +fit_tmp <- tempfile() +writeLines(fit_code, fit_tmp) +fit_md5 <- file_md5(fit_tmp) +unlink(fit_tmp) if (!have_mass_ns) fit_code <- gsub("MASS::", "", fit_code, fixed = TRUE) # Every da_* global fit_nb sets or the block creates on the current branch @@ -227,12 +249,24 @@ screen <- function(yv, g) { "" } -fit_lrt <- function(counts, offset, n_per) { +# Both model-preserving candidates are computed from ONE full glm.nb fit and, +# for lrt only, one intercept-only fit, so the two tests see identical fits and +# neither gets an easier dataset. ql_f uses the full fit's Pearson dispersion: +# phi = sum(Pearson residuals^2) / residual df +# F = (beta / se)^2 / phi against F(1, residual df) +# which is a quasi-likelihood F test with a tagwise (not empirical-Bayes) +# dispersion estimate. It is deliberately not described as edgeR's or +# glmGamPoi's QL test: both shrink phi across taxa, which this does not. +fit_nb_candidates <- function(counts, offset, n_per, want) { g <- factor(rep(c("A", "B"), each = n_per), levels = c("A", "B")) - out <- data.frame(status = "failed", pvalue = NA_real_) - out <- out[rep(1, ncol(counts)), , drop = FALSE] - rownames(out) <- NULL - one <- function(formula, yv) { + blank <- function() { + out <- data.frame(status = "failed", pvalue = NA_real_) + out <- out[rep(1, ncol(counts)), , drop = FALSE] + rownames(out) <- NULL + out + } + out <- setNames(lapply(want, function(t) blank()), want) + one <- function(formula) { warns <- character(0) fit <- tryCatch( withCallingHandlers(glm_nb(formula, control = glm.control(maxit = 100)), @@ -252,16 +286,41 @@ fit_lrt <- function(counts, offset, n_per) { # glm.nb looks its variables up in the formula's environment; yv, g and # off all live in this frame (one()'s enclosing environment), which is # where these formulas are created. - full <- one(yv ~ g + offset(off), yv) + full <- one(yv ~ g + offset(off)) if (bad_fit(full)) next - null <- one(yv ~ 1 + offset(off), yv) - if (bad_fit(null)) next - lr <- 2 * (as.numeric(logLik(full$fit)) - as.numeric(logLik(null$fit))) - if (!is.finite(lr) || lr < 0) next - p <- pchisq(lr, df = 1, lower.tail = FALSE) - if (!is.finite(p)) next - out$status[j] <- "ok" - out$pvalue[j] <- p + if ("lrt" %in% want) { + null <- one(yv ~ 1 + offset(off)) + if (!bad_fit(null)) { + lr <- 2 * (as.numeric(logLik(full$fit)) - as.numeric(logLik(null$fit))) + if (is.finite(lr) && lr >= 0) { + p <- pchisq(lr, df = 1, lower.tail = FALSE) + if (is.finite(p)) { + out$lrt$status[j] <- "ok" + out$lrt$pvalue[j] <- p + } + } + } + } + if ("ql_f" %in% want) { + co <- summary(full$fit, dispersion = 1)[["coefficients"]] + idx <- match("gB", rownames(co)) + if (is.na(idx)) idx <- grep("^g", rownames(co))[1] + dfr <- df.residual(full$fit) + phi <- if (is.na(dfr) || dfr < 1) NA_real_ else + sum(residuals(full$fit, type = "pearson")^2) / dfr + if (!is.na(idx) && is.finite(phi) && phi > 0) { + est <- co[idx, 1] + se <- co[idx, 2] + if (is.finite(est) && is.finite(se) && se > 0) { + fstat <- (est / se)^2 / phi + p <- pf(fstat, df1 = 1, df2 = dfr, lower.tail = FALSE) + if (is.finite(p)) { + out$ql_f$status[j] <- "ok" + out$ql_f$pvalue[j] <- p + } + } + } + } } out } @@ -333,9 +392,17 @@ for (t in want) { } if (length(enabled) == 0) stop("no requested test can run in this R library") -cell_index <- 0L +# A cell's datasets must not depend on which cells were requested in this +# invocation, or in what order. The per-sim seed is therefore hashed from the +# cell's identity string (previously it used the cell's index, so splitting a +# grid across processes silently changed every dataset). +stable_key <- function(x) { + h <- 0 + for (ch in utf8ToInt(x)) h <- (h * 131 + ch) %% 1000003 + h +} + for (shape in shapes) for (off_m in offsets) for (n_per in n_pers) { - cell_index <- cell_index + 1L # The taxa count is part of the cell identity: reruns append to a cell's # CSV, so two settings must never share a file by accident. cell <- sprintf("%s_%s_%s_n%d_t%d%s", truth, shape, off_m, n_per, n_taxa, @@ -345,6 +412,11 @@ for (shape in shapes) for (off_m in offsets) for (n_per in n_pers) { writeLines(c( paste("cell:", cell), paste("date:", format(Sys.time(), tz = "UTC", usetz = TRUE)), paste("args:", paste(argv, collapse = " ")), + paste("seed0:", seed0), paste("cell key:", stable_key(cell)), + paste("executor:", if (nzchar(Sys.getenv("NB_EXECUTOR"))) + Sys.getenv("NB_EXECUTOR") else R.version.string), + paste("calibration.R md5:", script_md5), + paste("_FIT_R md5:", fit_md5), paste("MASS:", mass_mode), paste("glmGamPoi:", if (have_gp) as.character(packageVersion("glmGamPoi")) else "absent"), paste("edgeR:", if (have_edger) as.character(packageVersion("edgeR")) else "absent"), @@ -354,12 +426,30 @@ for (shape in shapes) for (off_m in offsets) for (n_per in n_pers) { cat(sprintf("cell %s: %d sims x %d taxa, tests %s\n", cell, n_sims, n_taxa, paste(enabled, collapse = "+"))) header_done <- file.exists(f) + done_sims <- integer(0) + if (header_done) { + # Resuming is only safe when the existing rows came from the same seed + # scheme. The cell key is recorded in the provenance file; a mismatch means + # a stale cell from a different convention, not a resumable one. + prev <- tryCatch(read.csv(f, stringsAsFactors = FALSE), error = function(e) NULL) + if (!is.null(prev) && "sim" %in% names(prev)) done_sims <- unique(as.integer(prev$sim)) + if (file.exists(prov)) { + old_key <- grep("^cell key:", readLines(prov, warn = FALSE), value = TRUE) + if (length(old_key) && !identical(trimws(sub("^cell key:[ ]*", "", old_key[1])), + as.character(stable_key(cell)))) { + stop("cell ", cell, " has rows from a different seed scheme (key ", + sub("^cell key:[ ]*", "", old_key[1]), ", now ", stable_key(cell), + "); delete ", f, " and ", prov, " before rerunning") + } + } + } skipped_sims <- 0L refused_offsets <- 0L - for (s in seq_len(n_sims)) { + if (length(done_sims)) cat(sprintf(" resuming: %d sims already present\n", length(done_sims))) + for (s in setdiff(seq_len(n_sims), done_sims)) { # One seed per (cell, sim): cells can run in any order, in parallel # processes, and reproduce dataset-for-dataset. - set.seed(seed0 + cell_index * 1000003L + s) + set.seed(as.integer((seed0 + stable_key(cell) * 1000003 + s) %% 2147483647)) sim <- simulate(shape, n_per, n_taxa, truth, lfc) counts <- sim$counts if (ncol(counts) < 5) { skipped_sims <- skipped_sims + 1L; next } @@ -374,7 +464,11 @@ for (shape in shapes) for (off_m in offsets) for (n_per in n_pers) { r <- run_block(counts, offset, TRUE, n_per) fits$wald_trend <- data.frame(status = r$status, pvalue = r$pvalue) } - if ("lrt" %in% enabled) fits$lrt <- fit_lrt(counts, offset, n_per) + nb_want <- intersect(c("lrt", "ql_f"), enabled) + if (length(nb_want) > 0) { + nb <- fit_nb_candidates(counts, offset, n_per, nb_want) + for (t in names(nb)) fits[[t]] <- nb[[t]] + } if ("glmgampoi_ql" %in% enabled) fits$glmgampoi_ql <- fit_glmgampoi_ql(counts, offset, n_per) if ("edger_ql" %in% enabled) fits$edger_ql <- fit_edger_ql(counts, offset, n_per) rows <- do.call(rbind, lapply(names(fits), function(t) { diff --git a/bench/null_calibration/results/calibration_summary.csv b/bench/null_calibration/results/calibration_summary.csv index a73478c8..214ab96f 100644 --- a/bench/null_calibration/results/calibration_summary.csv +++ b/bench/null_calibration/results/calibration_summary.csv @@ -1,33 +1,49 @@ -"truth","shape","offset","n_per_group","n_taxa_setting","test","n_sims","n_attempted","n_pvalues","n_failed","n_boundary","fail_rate","rate_01","rate_01_lo","rate_01_hi","rate_05","rate_05_lo","rate_05_hi","rate_10","rate_10_lo","rate_10_hi","ks_D","ks_p","mean_p_null" -"null","constant","tss",4,40,"lrt",150,6000,5679,321,0,0.0535,0.0343370311674591,0.0299070144118022,0.0393966009845745,0.103539355520338,0.0958817540765541,0.111732951941575,0.167635147032928,0.158145307258128,0.177574327478367,0.0843830805075691,1.50521819708956e-35,0.444054181891207 -"null","constant","tss",4,40,"wald_ml",150,6000,5758,242,147,0.0403333333333333,0.067210837096214,0.0610277913962478,0.0739709697428515,0.134769017019799,0.126192001748904,0.14383303624288,0.195206668982286,0.185173587708648,0.205646165822299,0.104078202508359,1.3341509762296e-54,0.433539982517614 -"null","constant","tss",12,40,"lrt",150,6000,5986,14,0,0.00233333333333333,0.0143668559973271,0.0116487013673261,0.0177079118301315,0.0649849649181423,0.0590152557624449,0.0715126495578783,0.115603073838958,0.107748388407623,0.123950809035566,0.0376376568717833,8.62249043869314e-08,0.485492376920367 -"null","constant","tss",12,40,"wald_ml",150,6000,5989,11,6,0.00183333333333333,0.0213725162798464,0.0180049117092347,0.0253537288632171,0.0728001335782267,0.066490416983351,0.0796575271661756,0.127233260978461,0.119031959024314,0.135912455794454,0.0445719596399606,9.25695055432286e-11,0.481410606985188 -"null","original","rle",4,40,"lrt",150,5997,5795,202,0,0.0336835084208771,0.0291630716134599,0.0251329702386332,0.0338169874810617,0.100431406384815,0.0929553445016146,0.108436858943485,0.167385677308024,0.15799492693621,0.177217110265444,0.0999852967297406,9.5741484666021e-51,0.437673327317577 -"null","original","rle",4,40,"wald_ml",150,5997,5843,154,79,0.0256795064198766,0.0641793599178504,0.0581774162795938,0.0707539843971647,0.138113982543214,0.129504908831245,0.147198584878162,0.199897313024131,0.189841646759709,0.210347322840407,0.121629858990801,1.65933724634618e-75,0.426544757195022 -"null","original","rle",12,40,"lrt",150,6000,5999,1,0,0.000166666666666667,0.0170028338056343,0.0140269075883063,0.0205969385096592,0.0675112518753126,0.0614348261387838,0.0741412113767197,0.126187697949658,0.118023356072251,0.134830474773197,0.0374388440525218,9.94059353574812e-08,0.476768173372017 -"null","original","rle",12,40,"wald_ml",150,6000,5999,1,0,0.000166666666666667,0.024837472912152,0.021192855359116,0.029090241552791,0.0801800300050008,0.0735734789459446,0.0873239003112516,0.138523087181197,0.130012534390866,0.14749628709781,0.0425443113010688,7.40678934295594e-10,0.472785456890374 -"null","original","tss",3,40,"lrt",200,7988,7315,673,0,0.0842513770655984,0.045933014354067,0.0413694275154052,0.0509732557662729,0.131237183868763,0.12369250293294,0.139168971747875,0.203417634996582,0.19434974613276,0.21279685973194,0.135182796664952,1.55071172931089e-116,0.415587740716814 -"null","original","tss",3,40,"wald_ml",200,7988,7476,512,273,0.0640961442163245,0.0918940609951846,0.0855537288027265,0.0986535791203389,0.179240235420011,0.170711249407119,0.188098689760206,0.250668806848582,0.240974270777598,0.260619443404176,0.161656443920632,4.03457010651865e-170,0.401366722446981 -"null","original","tss",4,40,"lrt",200,7997,7708,289,0,0.0361385519569839,0.0303580695381422,0.026755637175536,0.0344283824052878,0.102880124545926,0.0962945813035816,0.109861298236999,0.169434353918007,0.161224866817917,0.177973166853889,0.0933013909161196,1.0455020069296e-58,0.443086204545911 -"null","original","tss",4,40,"wald_ml",200,7997,7783,214,106,0.0267600350131299,0.0628292432224078,0.0576509835270075,0.0684388391582219,0.13825003212129,0.130760030253546,0.146096956012513,0.203520493383014,0.194723061730016,0.212610447703097,0.113643078400003,9.87146482984532e-88,0.431436851581758 -"null","original","tss",6,40,"lrt",200,8000,7904,96,0,0.012,0.0232793522267206,0.0201794197046358,0.0268424459659169,0.086918016194332,0.0809063372182911,0.09333102780675,0.148658906882591,0.14098681509377,0.156672346521709,0.0645951152356356,4.52120174669666e-29,0.461303882402388 -"null","original","tss",6,40,"wald_ml",200,8000,7937,63,28,0.007875,0.0447272269119315,0.0403957707152817,0.0494991683188881,0.111251102431649,0.104520582486221,0.118357744451632,0.170719415396245,0.162601431411705,0.179155984712773,0.0787526584971869,3.50452954348045e-43,0.453506610955192 -"null","original","tss",8,40,"lrt",200,8000,7974,26,0,0.00325,0.0171808377225984,0.0145524202088425,0.0202742256019111,0.0701028342111863,0.0647034005936096,0.0759162726113939,0.128668171557562,0.121497414397098,0.136196533199172,0.0453296175635426,1.17327512113714e-14,0.473696316973658 -"null","original","tss",8,40,"wald_ml",200,8000,7984,16,6,0.002,0.0293086172344689,0.0258291673085302,0.0332407906085308,0.0886773547094188,0.0826378982356264,0.0951124322059991,0.145415831663327,0.137853814201029,0.153318897577363,0.0552258902999433,1.41431816022389e-21,0.467708032138111 -"null","original","tss",12,40,"lrt",200,8000,7997,3,0,0.000375,0.0171314242841065,0.0145105151972706,0.0202160145372216,0.0645241965737151,0.059345808818977,0.0701207559357476,0.119294735525822,0.112372755059871,0.126582293438552,0.0329734074224069,5.6092770626337e-08,0.480625097706312 -"null","original","tss",12,40,"wald_ml",200,8000,7999,1,1,0.000125,0.0251281410176272,0.0219194190982992,0.028792751186341,0.0757594699337417,0.0701620857850788,0.0817641350549703,0.130016252031504,0.122823186861121,0.137564510376576,0.0381036842306565,1.63501153735572e-10,0.476689665536424 -"null","original","tss",20,40,"lrt",200,8000,8000,0,0,0,0.014125,0.0117624417845792,0.0169539514627402,0.058375,0.0534462764601491,0.0637276410450682,0.11025,0.103572963173447,0.117301159323374,0.020738749813529,0.00205314046936761,0.487703822259244 -"null","original","tss",20,40,"wald_ml",200,8000,8000,0,0,0,0.018625,0.0158851798715362,0.0218268938087898,0.0645,0.0593234953117845,0.0700945427828028,0.116,0.109166456166853,0.123202146883254,0.026145410693358,3.55641366754023e-05,0.48530796517549 -"null","sparse","tss",4,40,"lrt",150,5365,4380,985,0,0.183597390493942,0.0237442922374429,0.0196354002458669,0.0286878479396827,0.0817351598173516,0.0739836169914801,0.0902197342433841,0.144063926940639,0.133976279706903,0.154775371262067,0.0691867976694867,1.2302251751396e-18,0.455338729993458 -"null","sparse","tss",4,40,"wald_ml",150,5365,4554,811,236,0.15116495806151,0.0408432147562582,0.0354710856191992,0.0469893207412142,0.106719367588933,0.0980810744949493,0.116020593506725,0.170838823012736,0.160186222147154,0.182046273852014,0.0823244156861178,3.1118285502842e-27,0.44737008911558 -"null","sparse","tss",12,40,"lrt",150,5859,5562,297,0,0.0506912442396313,0.015282272563826,0.0123767061844007,0.0188569284972326,0.0604099244875944,0.054446958150994,0.0669796875450717,0.110931319669184,0.102945034249156,0.119454663618446,0.0326604118895657,1.40505153549206e-05,0.480162876414684 -"null","sparse","tss",12,40,"wald_ml",150,5859,5594,265,33,0.0452295613585936,0.0196639256346085,0.0163415019144068,0.0236456003405133,0.0682874508401859,0.0619693876083744,0.075198030309613,0.119413657490168,0.111176063604179,0.128173597988946,0.0345742276076426,3.11048126202616e-06,0.476462685961144 -"null","unequal_lib","tss",4,40,"lrt",150,5999,5809,190,0,0.0316719453242207,0.0395937338612498,0.0348758076860468,0.0449201857375891,0.113616801514891,0.105710109456138,0.122034181902751,0.17868824238251,0.169050117338897,0.188751049879355,0.0933208396676188,2.28941310680147e-44,0.440256995667013 -"null","unequal_lib","tss",4,40,"wald_ml",150,5999,5898,101,56,0.0168361393565594,0.0764665988470668,0.0699568766474143,0.0835276696829271,0.14852492370295,0.139678040812683,0.157829350917821,0.211766700576467,0.20152919358122,0.222379424818132,0.115545521434757,8.05155216532249e-69,0.429486140222383 -"null","unequal_lib","tss",12,40,"lrt",150,6000,5995,5,0,0.000833333333333333,0.0128440366972477,0.0102895388904349,0.0160224515004104,0.0618849040867389,0.0560617457461293,0.0682691711384984,0.116263552960801,0.108394130866014,0.124624439205597,0.023602226833183,0.00251353912720786,0.48526062352396 -"null","unequal_lib","tss",12,40,"wald_ml",150,6000,6000,0,1,0,0.0196666666666667,0.0164483267012244,0.0234996733360393,0.0741666666666667,0.0678051993175382,0.0810730588695571,0.13,0.121726673024468,0.138746803756492,0.0305808238880169,2.67468380464776e-05,0.481384434000172 -"power","original","tss",4,40,"lrt",150,5997,5768,229,0,0.0381857595464399,0.0325109522711552,0.0276325349035269,0.038216785218464,0.114134194143417,0.105010338234011,0.123940999484436,0.183306433018215,0.172073169738517,0.195100216790487,0.105618632297901,1.89778614361325e-42,0.434784684139481 -"power","original","tss",4,40,"wald_ml",150,5997,5841,156,88,0.0260130065032516,0.0721602549510585,0.0648766537321604,0.0801914513597557,0.150466651491008,0.140199738589643,0.161344329177655,0.218757113589802,0.206780906784307,0.23122475633943,0.127157614018003,4.02317867444891e-62,0.422783825089475 -"power","original","tss",12,40,"lrt",150,6000,5997,3,0,5e-04,0.0237777777777778,0.0197159062402261,0.0286520171896841,0.0831111111111111,0.0753968197212776,0.0915365560857739,0.150888888888889,0.140728778500017,0.161644533543095,0.0677456389208428,2.30351108889158e-18,0.456624861778725 -"power","original","tss",12,40,"wald_ml",150,6000,5998,2,1,0.000333333333333333,0.0328888888888889,0.0280635063862089,0.0385110969967455,0.096,0.0877341616927958,0.104955005267343,0.16,0.1495793559677,0.171000636028456,0.0737015111182361,1.17376844197725e-21,0.452433541509862 +"truth","shape","offset","n_per_group","n_taxa_setting","test","n_sims","n_attempted","n_pvalues","n_failed","n_boundary","fail_rate","rate_01","rate_01_lo","rate_01_hi","rate_05","rate_05_lo","rate_05_hi","verdict_05","rate_10","rate_10_lo","rate_10_hi","ks_D","ks_p","mean_p_null" +"null","constant","tss",4,40,"lrt",400,16000,15168,832,0,0.052,0.0299314345991561,0.0273364412929117,0.0327644674395983,0.0991561181434599,0.0945008383529729,0.104014382557779,"anti-conservative",0.170886075949367,0.164979343144035,0.176959469814267,0.0918405566293181,1.49990553356728e-111,0.439219952352868 +"null","constant","tss",4,40,"ql_f",400,16000,15360,640,0,0.04,0.00944010416666667,0.00802877994032958,0.0110967401718292,0.0555338541666667,0.0520219321225168,0.0592680381237616,"anti-conservative",0.113151041666667,0.108237818136324,0.118257714777604,0.030709736808881,5.23356154601085e-13,0.479449047676663 +"null","constant","tss",4,40,"wald_ml",400,16000,15360,640,374,0.04,0.06171875,0.0580215939274056,0.0656350751363041,0.139713541666667,0.134320885555989,0.145286364384381,"anti-conservative",0.20234375,0.19606514574096,0.20877120169803,0.111421831621594,4.65928533910885e-166,0.42759202597785 +"null","constant","tss",12,40,"lrt",400,16000,15942,58,0,0.003625,0.0148036632793878,0.0130424744684032,0.0167986261125557,0.0624764772299586,0.058823984717749,0.0663397743710925,"anti-conservative",0.117802032367332,0.112889645435554,0.122898567342698,0.0293055798622485,2.56420045291805e-12,0.484647558687368 +"null","constant","tss",12,40,"ql_f",400,16000,15952,48,0,0.003,0.0127883650952859,0.0111583070862361,0.0146530210022001,0.0582372116349047,0.0547082269029707,0.0619789101324398,"anti-conservative",0.112399699097292,0.107591186106801,0.117394846004728,0.0164903161223888,0.000341376591863202,0.492592723011031 +"null","constant","tss",12,40,"wald_ml",400,16000,15952,48,18,0.003,0.0215020060180542,0.0193636121037609,0.0238708021138194,0.0744734202607824,0.0705009337249709,0.0786508026446,"anti-conservative",0.130328485456369,0.125192939979835,0.135642031936983,0.0367114999620834,4.23825272250876e-19,0.480641458161521 +"null","original","rle",4,40,"lrt",400,15998,15366,632,0,0.0395049381172647,0.0322139789144865,0.0295370271094338,0.0351247627654599,0.105557724847065,0.100797566100429,0.110515053325243,"anti-conservative",0.174801509826891,0.168877899234466,0.180887677282624,0.101509237276185,5.95072319292083e-138,0.43298847856611 +"null","original","rle",4,40,"ql_f",400,15998,15523,475,0,0.0296912114014252,0.00863235199381563,0.00729376522762286,0.0102140749562069,0.0552728209753269,0.0517868559892686,0.058978843743156,"anti-conservative",0.113186883978612,0.108298324291759,0.11826684468678,0.0399482645040839,6.0789412627354e-22,0.475401408335383 +"null","original","rle",4,40,"wald_ml",400,15998,15523,475,210,0.0296912114014252,0.0639695935064098,0.0602270443853,0.0679278971137824,0.141725181988018,0.136327264674104,0.147300379138727,"anti-conservative",0.210461895252206,0.2041213312497,0.216945727118864,0.12385017546402,3.05820688689302e-207,0.42073558822519 +"null","original","rle",12,40,"lrt",400,16000,15992,8,0,5e-04,0.0158829414707354,0.0140582558514039,0.0179401519934329,0.067783891945973,0.0639907816525002,0.0717845987432002,"anti-conservative",0.125500250125063,0.1204555083675,0.130724866804643,0.0393669850721669,5.94476464801926e-22,0.478659927974232 +"null","original","rle",12,40,"ql_f",400,16000,15994,6,0,0.000375,0.0143803926472427,0.0126484851165406,0.0163455176065547,0.0620232587220208,0.0583893659989178,0.0658674885234567,"anti-conservative",0.117356508690759,0.112460265284936,0.122436515539751,0.0263015855697064,4.90644908698071e-10,0.488340752358446 +"null","original","rle",12,40,"wald_ml",400,16000,15994,6,0,0.000375,0.0261973239964987,0.0238334466810007,0.0287887436930969,0.0804676753782669,0.0763520750157164,0.0847847549407622,"anti-conservative",0.138114292859822,0.132854082773477,0.143548297522812,0.0474345530099633,1.10418476013339e-31,0.474625148697665 +"null","original","tss",3,40,"lrt",1000,39942,36841,3101,0,0.0776375744830003,0.0420726907521511,0.0400700092451098,0.0441708596507348,0.123557992454059,0.120236875994278,0.126957604921049,"anti-conservative",0.197497353492033,0.193463741445605,0.201594043647294,0.123237317760909,0,0.419266887820315 +"null","original","tss",3,40,"ql_f",1000,39942,37597,2345,0,0.0587101296880477,0.00643668377796101,0.0056772265814611,0.00729698994917062,0.0440726653722371,0.042044064499513,0.046194425121904,"conservative",0.0996622070909913,0.0966750999468533,0.102731114593109,0.0248069133355436,1.60278630528837e-20,0.486474929567557 +"null","original","tss",3,40,"wald_ml",1000,39942,37597,2345,1250,0.0587101296880477,0.0897146048886879,0.0868677351995007,0.0926453075129321,0.172008404925925,0.168227271231178,0.175856556599904,"anti-conservative",0.24068409713541,0.236389508829338,0.245031671030301,0.150287491071984,0,0.404772218656843 +"null","original","tss",4,40,"lrt",1000,39993,38455,1538,0,0.0384567299277374,0.0319334286828761,0.0302223427015046,0.0337380202609247,0.100975165778182,0.0980035341654789,0.104026510528929,"anti-conservative",0.169574827720713,0.165857258091312,0.173358406342817,0.0886169435333972,9.99082155284299e-263,0.441921230871433 +"null","original","tss",4,40,"ql_f",1000,39993,38874,1119,0,0.0279798964818843,0.00915779184030457,0.0082581705686396,0.0101544118165968,0.0545094407573185,0.0522963938772562,0.0568105240907498,"anti-conservative",0.111539846684159,0.108448811025935,0.114707648624998,0.0265036422210414,3.82533897684242e-24,0.483566471912181 +"null","original","tss",4,40,"wald_ml",1000,39993,38874,1119,511,0.0279798964818843,0.0644904048978752,0.0620914904701154,0.066975383372034,0.139450532489582,0.13604251365803,0.142929801981165,"anti-conservative",0.202963420280908,0.198994641631955,0.206990898371312,0.110279357515923,0,0.430344435878016 +"null","original","tss",6,40,"lrt",1000,40000,39588,412,0,0.0103,0.0225320804284126,0.0211158426703377,0.0240409722931749,0.0804031524704456,0.0777651237770187,0.0831226052124485,"anti-conservative",0.143452561382237,0.140034153375079,0.14694015850779,0.0563901437604384,9.11423388108106e-110,0.464001080332149 +"null","original","tss",6,40,"ql_f",1000,40000,39700,300,0,0.0075,0.0116624685138539,0.0106526217135078,0.0127668113855562,0.0583375314861461,0.056074429401325,0.060686097753933,"anti-conservative",0.116171284634761,0.113056354291349,0.119360488003154,0.0242182105154358,1.19127531576029e-20,0.485908232244797 +"null","original","tss",6,40,"wald_ml",1000,40000,39700,300,104,0.0075,0.0400755667506297,0.0381902951626203,0.042049836314958,0.104307304785894,0.101338793532537,0.107352384811363,"anti-conservative",0.16727959697733,0.163640490826569,0.170983086405883,0.0712365881867924,2.05075831158841e-175,0.456166434320474 +"null","original","tss",8,40,"lrt",1000,40000,39878,122,0,0.00305,0.0182306033401876,0.0169631849725304,0.0195908307272202,0.0725713425949145,0.0700660339928397,0.0751589919077562,"anti-conservative",0.132779978935754,0.129484800163162,0.136145899707083,0.0451291652670526,5.71048927358849e-71,0.471685582866364 +"null","original","tss",8,40,"ql_f",1000,40000,39908,92,0,0.0023,0.0123784704821089,0.0113396477366664,0.0135111590055891,0.0600882028665932,0.0577986596178781,0.0624624279035524,"anti-conservative",0.116142126891851,0.113035573789148,0.119322571559496,0.0229758676555331,1.00558669115875e-18,0.486163395222542 +"null","original","tss",8,40,"wald_ml",1000,40000,39908,92,23,0.0023,0.0313220406935953,0.029657673536544,0.0330766270445015,0.0904079382579934,0.0876337343565309,0.0932609874835979,"anti-conservative",0.149543951087501,0.146078812420887,0.153076551561497,0.0548497059066133,1.03700838224511e-104,0.465810400391627 +"null","original","tss",12,40,"lrt",1000,40000,39985,15,0,0.000375,0.0143303738902088,0.0132112365828849,0.0155428212210844,0.0629736151056646,0.0606343669360676,0.065396827641314,"anti-conservative",0.121245467050144,0.1180824163193,0.124481286577902,0.0299948859648441,1.13294579102053e-31,0.480516955373766 +"null","original","tss",12,40,"ql_f",1000,40000,39988,12,0,3e-04,0.0123787136140842,0.0113408750101777,0.0135102301794657,0.057667300190057,0.0554246930950396,0.0599948847599484,"anti-conservative",0.113108932679804,0.110041699993278,0.116250491831379,0.017803461229884,1.95840776820909e-11,0.488359359340764 +"null","original","tss",12,40,"wald_ml",1000,40000,39988,12,1,3e-04,0.0219065719715915,0.020517131352401,0.023387860135142,0.0749474842452736,0.0724073698243338,0.0775692564082099,"anti-conservative",0.132189656897069,0.128905292683181,0.135544681937532,0.0373011023694641,9.42561169731967e-49,0.476542997844207 +"null","original","tss",20,40,"lrt",1000,40000,39998,2,0,5e-05,0.0124256212810641,0.0113858791490894,0.0135590089468225,0.0576778838941947,0.0554353541639921,0.060005367823131,"anti-conservative",0.110230511525576,0.107198707448857,0.113337176327948,0.0178111785266293,1.90371209670042e-11,0.490543787197504 +"null","original","tss",20,40,"ql_f",1000,40000,40000,0,0,0,0.01205,0.0110266319970337,0.0131670809946792,0.056025,0.0538136741069206,0.0583215932882956,"anti-conservative",0.105575,0.102601366979639,0.108624384115261,0.012304842848764,1.09783498280435e-05,0.494272237075444 +"null","original","tss",20,40,"wald_ml",1000,40000,40000,0,0,0,0.0166,0.0153935245916351,0.0178993145521173,0.063875,0.0615202777593929,0.0663134820080234,"anti-conservative",0.1168,0.11368920969043,0.119984385592743,0.020692014825702,2.66223333913683e-15,0.488151114851894 +"null","sparse","tss",4,40,"lrt",400,14243,11671,2572,0,0.180579934002668,0.026561562848085,0.0237964049650823,0.0296382785825357,0.0858538257218747,0.0809065542324058,0.0910736363254462,"anti-conservative",0.149087481792477,0.142741123212274,0.155664767018342,0.0759962866334588,5.6714811559108e-59,0.450893552419821 +"null","sparse","tss",4,40,"ql_f",400,14243,12090,2153,0,0.151161974303166,0.00488006617038875,0.00378543176159301,0.00628923799785125,0.0449958643507031,0.041443071678938,0.0488377098691246,"conservative",0.0974358974358974,0.0922769831959352,0.10285055066962,0.0322825734699355,2.27514636892002e-11,0.482638683081742 +"null","sparse","tss",4,40,"wald_ml",400,14243,12090,2153,617,0.151161974303166,0.0455748552522746,0.0419993364985109,0.0494390590341297,0.112241521918941,0.106737464678928,0.117991912506551,"anti-conservative",0.174607113316791,0.167943754324407,0.181477186342758,0.0930253425526498,2.66866271154838e-91,0.441522274534205 +"null","sparse","tss",12,40,"lrt",400,15654,14870,784,0,0.0500830458668711,0.0131809011432414,0.0114694703877993,0.0151437928983941,0.0596503026227303,0.055956168404868,0.0635718945869809,"anti-conservative",0.116274377942165,0.111220987261952,0.121525977831434,0.0245889470139347,3.10352545496201e-08,0.484743280371751 +"null","sparse","tss",12,40,"ql_f",400,15654,14932,722,0,0.0461223968314808,0.0127243503884275,0.0110478070310365,0.0146515457559474,0.0575944280739352,0.0539701757134296,0.061446251514555,"anti-conservative",0.116528261451915,0.111480235424825,0.121773543307757,0.0276508972223738,2.42496332938364e-10,0.484091704944378 +"null","sparse","tss",12,40,"wald_ml",400,15654,14932,722,56,0.0461223968314808,0.0182159121350121,0.0161915483231139,0.0204881031267985,0.0699839271363515,0.0660015780110123,0.074187474249189,"anti-conservative",0.126105009375837,0.120876416179525,0.131525932172555,0.0318099175592042,1.50417138537635e-13,0.480734410089792 +"null","unequal_lib","tss",4,40,"lrt",400,15999,15507,492,0,0.0307519219951247,0.0350164441864964,0.0322364589408286,0.0380267477145991,0.102663313342362,0.0979841395474483,0.107539298190215,"anti-conservative",0.172567227703618,0.166701069078759,0.178595572179535,0.0951082760260242,2.91182816568892e-122,0.438477644862895 +"null","unequal_lib","tss",4,40,"ql_f",400,15999,15718,281,0,0.0175635977248578,0.0108156253976333,0.00931392323696445,0.0125563811876984,0.0551596895279298,0.0516982198913055,0.0588385428265047,"anti-conservative",0.111973533528439,0.107138338567342,0.116998348477269,0.0324245561725308,8.86071126290534e-15,0.480567005607464 +"null","unequal_lib","tss",4,40,"wald_ml",400,15999,15718,281,143,0.0175635977248578,0.0658480722738262,0.0620758722102559,0.0698324328221149,0.140539508843364,0.135194018736591,0.146060659374912,"anti-conservative",0.205115154599822,0.198875080406063,0.211499332522577,0.117226942674989,4.8584810698117e-188,0.426578894778762 +"null","unequal_lib","tss",12,40,"lrt",400,16000,15988,12,0,0.00075,0.014823617713285,0.0130635619882632,0.0168167654306363,0.0621716287215412,0.0585328679459516,0.0660207342111061,"anti-conservative",0.118276207155367,0.113361950233272,0.123373854618802,0.0276741008052646,4.63009030332263e-11,0.484182480012251 +"null","unequal_lib","tss",12,40,"ql_f",400,16000,15996,4,0,0.00025,0.0130657664416104,0.0114192458001177,0.0149461066258048,0.0575768942235559,0.0540721352011722,0.0612940991187988,"anti-conservative",0.110277569392348,0.105516669426873,0.115225609045742,0.0157193148610208,0.000737698351289778,0.492049771835005 +"null","unequal_lib","tss",12,40,"wald_ml",400,16000,15996,4,0,0.00025,0.0225681420355089,0.0203785740864267,0.0249869666090118,0.0729557389347337,0.0690272885189459,0.0770892505003733,"anti-conservative",0.130157539384846,0.125031889904336,0.135460782447925,0.0355374064141356,5.67885690806382e-18,0.48025434999029 +"power","original","tss",4,40,"lrt",400,15995,15368,627,0,0.0391997499218506,0.0364565292691375,0.0331894087086501,0.0400319440805047,0.111967440249394,0.106344831262011,0.117848122567282,"anti-conservative",0.181503290613093,0.17457973504827,0.188638672491149,0.103629190599653,3.83504828847747e-108,0.431031506541484 +"power","original","tss",4,40,"ql_f",400,15995,15518,477,0,0.029821819318537,0.0103755788029498,0.00869096588334082,0.0123826497395434,0.0609672440404733,0.0567675185739818,0.0654561087446114,"anti-conservative",0.121505745155205,0.115700391218543,0.127560368739323,0.0412497772864611,1.16215793111554e-17,0.473274424304459 +"power","original","tss",4,40,"wald_ml",400,15995,15518,477,207,0.029821819318537,0.0698850968959012,0.0653980829922073,0.074655376896477,0.150060024009604,0.143693609780552,0.156656902541162,"anti-conservative",0.212913736923341,0.205579137288196,0.220437406513103,0.126782330449227,3.03482904674379e-163,0.419585574140408 +"power","original","tss",12,40,"lrt",400,16000,15990,10,0,0.000625,0.0207621112315517,0.0183594618953389,0.0234716701191393,0.0781289085299758,0.0734597246931148,0.0830682635499914,"anti-conservative",0.14274993746352,0.136603544971756,0.1491251170754,0.0592563248233505,5.29342366452238e-37,0.462622479333866 +"power","original","tss",12,40,"ql_f",400,16000,15993,7,0,0.0004375,0.0171752542938136,0.0149999075335797,0.0196597818987199,0.0700350175087544,0.0656040667920865,0.0747412998807453,"anti-conservative",0.132816408204102,0.12686018942349,0.139007756053949,0.0469478205889672,2.18301052832755e-23,0.470583086605784 +"power","original","tss",12,40,"wald_ml",400,16000,15993,7,0,0.0004375,0.0305152576288144,0.0275842134048629,0.0337469403152682,0.0936301484075371,0.0885460034156546,0.0989745157162791,"anti-conservative",0.154994163748541,0.148628025035243,0.161581229820715,0.065167074521525,1.14557856102766e-44,0.458450919981907 diff --git a/bench/null_calibration/results/power_summary.csv b/bench/null_calibration/results/power_summary.csv index b9685f56..e9a9ba7d 100644 --- a/bench/null_calibration/results/power_summary.csv +++ b/bench/null_calibration/results/power_summary.csv @@ -1,5 +1,7 @@ "shape","offset","n_per_group","test","n_sims","power_raw_05","power_bh_05","bh_fdr_05","mean_discoveries" -"original","tss",4,"lrt",150,0.413696715583508,0.147449336128581,0.183793650793651,1.89333333333333 -"original","tss",4,"wald_ml",150,0.479972375690608,0.290055248618785,0.379349206349206,4.7 -"original","tss",12,"lrt",150,0.668002672010688,0.421509686038744,0.100953102453102,4.82 -"original","tss",12,"wald_ml",150,0.688251001335114,0.509345794392523,0.128748011248011,5.99333333333333 +"original","tss",4,"lrt",400,0.390837696335079,0.114921465968586,0.191532738095238,1.6325 +"original","tss",4,"ql_f",400,0.285010373443983,0.0223029045643154,0.036875,0.275 +"original","tss",4,"wald_ml",400,0.461358921161826,0.275414937759336,0.361786075036075,4.4325 +"original","tss",12,"lrt",400,0.681010758068551,0.468101075806855,0.0777915764790765,5.2075 +"original","tss",12,"ql_f",400,0.674168542135534,0.433108277069267,0.0629935064935065,4.7425 +"original","tss",12,"wald_ml",400,0.703425856464116,0.536134033508377,0.118255217005217,6.27 diff --git a/bench/null_calibration/results/provenance_summary.csv b/bench/null_calibration/results/provenance_summary.csv index 5cc78144..431453c1 100644 --- a/bench/null_calibration/results/provenance_summary.csv +++ b/bench/null_calibration/results/provenance_summary.csv @@ -1,17 +1,17 @@ -cell,date_utc,args,mass_mode,glmGamPoi,edgeR,tests_run,r_version,platform,webr -null_constant_tss_n12_t40,2026-10-22 22:42:23 UTC,"--shapes=constant --offsets=tss --n-per=12 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_constant_tss_n4_t40,2026-10-22 22:43:30 UTC,"--shapes=constant --offsets=tss --n-per=4 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_original_rle_n12_t40,2026-10-22 22:54:06 UTC,"--shapes=original --offsets=rle --n-per=12 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_original_rle_n4_t40,2026-10-22 22:55:33 UTC,"--shapes=original --offsets=rle --n-per=4 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_original_tss_n12_t40,2026-10-22 22:32:15 UTC,"--shapes=original --offsets=tss --n-per=12 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_original_tss_n20_t40,2026-10-22 22:37:21 UTC,"--shapes=original --offsets=tss --n-per=20 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_original_tss_n3_t40,2026-10-22 22:27:17 UTC,"--shapes=original --offsets=tss --n-per=3 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_original_tss_n4_t40,2026-10-22 22:32:41 UTC,"--shapes=original --offsets=tss --n-per=4 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_original_tss_n6_t40,2026-10-22 22:38:09 UTC,"--shapes=original --offsets=tss --n-per=6 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_original_tss_n8_t40,2026-10-22 22:27:17 UTC,"--shapes=original --offsets=tss --n-per=8 --sims=200 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_sparse_tss_n12_t40,2026-10-22 22:46:13 UTC,"--shapes=sparse --offsets=tss --n-per=12 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_sparse_tss_n4_t40,2026-10-22 22:47:35 UTC,"--shapes=sparse --offsets=tss --n-per=4 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_unequal_lib_tss_n12_t40,2026-10-22 22:50:11 UTC,"--shapes=unequal_lib --offsets=tss --n-per=12 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -null_unequal_lib_tss_n4_t40,2026-10-22 22:51:15 UTC,"--shapes=unequal_lib --offsets=tss --n-per=4 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=null --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -power_original_tss_n12_t40_lfc1,2026-10-22 22:57:57 UTC,"--shapes=original --offsets=tss --n-per=12 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=power --log2fc=1 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 -power_original_tss_n4_t40_lfc1,2026-10-22 22:59:41 UTC,"--shapes=original --offsets=tss --n-per=4 --sims=150 --taxa=40 --tests=wald_ml,lrt --truth=power --log2fc=1 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw",sourced negbin.R (MASS:: stripped from the extracted block; see header),absent,absent,"wald_ml,lrt",R version 4.6.0 (2026-04-24),wasm32-unknown-emscripten (32-bit),webr_0.6.0 +"cell","date_utc","args","seed0","cell_key","executor","calibration_md5","fit_r_md5","mass_mode","glmGamPoi","edgeR","tests_run","r_version","platform" +"null_constant_tss_n12_t40","2026-10-23 20:53:03 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=constant --offsets=tss --n-per=12 --sims=400 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","521491","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_constant_tss_n4_t40","2026-10-23 20:38:35 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=constant --offsets=tss --n-per=4 --sims=400 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","673367","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_original_rle_n12_t40","2026-10-23 21:48:56 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=original --offsets=rle --n-per=12 --sims=400 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","736875","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_original_rle_n4_t40","2026-10-23 19:54:06 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=original --offsets=rle --n-per=4 --sims=400 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","423102","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_original_tss_n12_t40","2026-10-23 19:36:36 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=original --offsets=tss --n-per=12 --sims=1000 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","488430","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_original_tss_n20_t40","2026-10-23 20:07:51 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=original --offsets=tss --n-per=20 --sims=1000 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","864269","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_original_tss_n3_t40","2026-10-23 18:31:05 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=original --offsets=tss --n-per=3 --sims=1000 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","441252","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_original_tss_n4_t40","2026-10-23 19:07:16 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=original --offsets=tss --n-per=4 --sims=1000 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","940291","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_original_tss_n6_t40","2026-10-23 18:31:05 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=original --offsets=tss --n-per=6 --sims=1000 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","938366","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_original_tss_n8_t40","2026-10-23 19:05:07 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=original --offsets=tss --n-per=8 --sims=1000 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","936441","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_sparse_tss_n12_t40","2026-10-23 21:06:54 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=sparse --offsets=tss --n-per=12 --sims=400 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","712268","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_sparse_tss_n4_t40","2026-10-23 19:41:15 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=sparse --offsets=tss --n-per=4 --sims=400 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","308410","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_unequal_lib_tss_n12_t40","2026-10-23 21:36:05 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=unequal_lib --offsets=tss --n-per=12 --sims=400 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","386958","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"null_unequal_lib_tss_n4_t40","2026-10-23 21:21:48 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=unequal_lib --offsets=tss --n-per=4 --sims=400 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","221957","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"power_original_tss_n12_t40_lfc1","2026-10-23 22:01:51 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=original --offsets=tss --n-per=12 --sims=400 --truth=power --log2fc=1 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","514172","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" +"power_original_tss_n4_t40_lfc1","2026-10-23 20:07:34 UTC","--taxa=40 --tests=wald_ml,lrt,ql_f --shapes=original --offsets=tss --n-per=4 --sims=400 --truth=power --log2fc=1 --differential-jl=/work/differential.jl --out=/work/results/raw --results=/work/results/raw","20261020","415815","webR 0.6.0","2717e709ff718e57e602eeaff2410e71","cc9efdfdd6e4efe2f9a393ba4d178ce0","sourced negbin.R (MASS:: stripped from the extracted block; see header)","absent","absent","wald_ml,lrt,ql_f","R version 4.6.0 (2026-04-24)","wasm32-unknown-emscripten (32-bit)" diff --git a/bench/null_calibration/summarise.R b/bench/null_calibration/summarise.R index 21231182..7778318e 100644 --- a/bench/null_calibration/summarise.R +++ b/bench/null_calibration/summarise.R @@ -72,6 +72,13 @@ summ_rows <- lapply(groups, function(d) { } else c(NA_real_, NA_real_) w <- function(alpha) wilson(sum(p_null < alpha), length(p_null)) w01 <- w(0.01); w05 <- w(0.05); w10 <- w(0.10) + # The pre-registered reading (README): anti-conservative when the LOWER + # Wilson bound at alpha = 0.05 exceeds 0.05, conservative when the UPPER + # bound is below it. Emitted as a column so a report never re-derives it. + verdict <- if (is.na(w05[2])) NA_character_ + else if (w05[2] > 0.05) "anti-conservative" + else if (w05[3] < 0.05) "conservative" + else "not flagged" data.frame( truth = d$truth[1], shape = d$shape[1], offset = d$offset[1], n_per_group = d$n_per[1], n_taxa_setting = d$taxa[1], test = d$test[1], @@ -81,6 +88,7 @@ summ_rows <- lapply(groups, function(d) { fail_rate = sum(d$status == "failed") / nrow(d), rate_01 = w01[1], rate_01_lo = w01[2], rate_01_hi = w01[3], rate_05 = w05[1], rate_05_lo = w05[2], rate_05_hi = w05[3], + verdict_05 = verdict, rate_10 = w10[1], rate_10_lo = w10[2], rate_10_hi = w10[3], ks_D = ks[1], ks_p = ks[2], mean_p_null = if (length(p_null)) mean(p_null) else NA_real_, stringsAsFactors = FALSE @@ -128,4 +136,35 @@ if (nrow(pow) > 0) { write.csv(power_df, file.path(out_dir, "power_summary.csv"), row.names = FALSE) cat(sprintf("wrote %s (%d rows)\n", file.path(out_dir, "power_summary.csv"), nrow(power_df))) } +# One machine-generated run record per cell, from the provenance files the +# calibration run wrote. Previously this CSV was assembled by hand, which made +# "what was actually run" a hand-maintained claim rather than an artefact. +prov_files <- list.files(results_dir, pattern = "^provenance_.*[.]txt$", full.names = TRUE) +if (length(prov_files) > 0) { + field <- function(lines, key) { + hit <- lines[startsWith(lines, paste0(key, ":"))] + if (length(hit) == 0) return(NA_character_) + trimws(sub(paste0("^", key, ":[ ]*"), "", hit[1])) + } + prow <- lapply(prov_files, function(f) { + lines <- readLines(f, warn = FALSE) + r_line <- grep("^R version ", lines, value = TRUE) + p_line <- grep("^Platform: ", lines, value = TRUE) + data.frame( + cell = field(lines, "cell"), date_utc = field(lines, "date"), + args = field(lines, "args"), seed0 = field(lines, "seed0"), + cell_key = field(lines, "cell key"), executor = field(lines, "executor"), + calibration_md5 = field(lines, "calibration.R md5"), + fit_r_md5 = field(lines, "_FIT_R md5"), + mass_mode = field(lines, "MASS"), glmGamPoi = field(lines, "glmGamPoi"), + edgeR = field(lines, "edgeR"), tests_run = field(lines, "tests run"), + r_version = if (length(r_line)) r_line[1] else NA_character_, + platform = if (length(p_line)) trimws(sub("^Platform: ", "", p_line[1])) else NA_character_, + stringsAsFactors = FALSE) + }) + prov_df <- do.call(rbind, prow) + prov_df <- prov_df[order(prov_df$cell), ] + write.csv(prov_df, file.path(out_dir, "provenance_summary.csv"), row.names = FALSE) + cat(sprintf("wrote %s (%d rows)\n", file.path(out_dir, "provenance_summary.csv"), nrow(prov_df))) +} cat("summarise complete\n") diff --git a/bench/null_calibration/webr_run.mjs b/bench/null_calibration/webr_run.mjs index 50eedd29..d56ad583 100644 --- a/bench/null_calibration/webr_run.mjs +++ b/bench/null_calibration/webr_run.mjs @@ -56,6 +56,10 @@ const { WebR } = await import(webrEntry); // process's stdout/stderr line by line. const webR = new WebR({ interactive: false }); await webR.init(); +// Record the executor in every provenance file the R code writes. Native +// Rscript runs of calibration.R record R.version.string instead. +const executor = `webR ${webR.version || "0.6.0"}`; +await webR.evalRVoid(`Sys.setenv(NB_EXECUTOR = ${JSON.stringify(executor)})`); const put = async (vfsPath, hostPath) => { const dir = vfsPath.slice(0, vfsPath.lastIndexOf('/'));