@@ -305,6 +305,79 @@ not exclude samples from this analysis.
305305 multiplicative (CZM) replacement; `replacement_delta` sets the replacement
306306 fraction and optional `bayesian_alpha` overrides GBM's concentration. The
307307 offset is not used. See [zero-handling notes](docs/statistics/zero-handling.md).
308+ - `ancom_bc` : the bias-corrected log-linear model of Lin & Peddada (2020), which
309+ estimates within-group sample-specific offsets and a between-group sampling
310+ fraction from a 3-component Gaussian mixture fitted by safeguarded SQUAREM.
311+ The effect is a bias-corrected log2 fold change; `offset` and the zero
312+ replacement settings are not used.
313+ - `aldex2` : ALDEx2's Dirichlet Monte-Carlo CLR instances (Fernandes et al. 2014),
314+ with a Welch test per instance and a per-instance sign-consistency check; the
315+ effect is the standardized CLR difference. `mc_samples` sets the number of
316+ Dirichlet draws (whole number in [16, 4096], seeded by the study seed).
317+ - `songbird` : Songbird's baseline-category compositional multinomial regression
318+ (Morton et al. 2019; Zhang et al. 2017), fitted in R by damped Newton-Raphson
319+ with exact Sherman-Morrison information inversion. It reports CLR-centred
320+ differentials, which sum to zero over the fitted taxa. `songbird` and
321+ `multinomial` are two implementations of the same model, in R and in Julia :
322+ ` songbird` reports centred differentials, `multinomial` reports log ratios
323+ against one reference taxon and has the Dirichlet-multinomial sibling.
324+ - `multinomial` : one multinomial logistic model over the whole composition
325+ (Songbird's model, Morton et al. 2019), fitted in Julia. The effect is a log2
326+ fold change in relative abundance against a reference taxon. `offset` is not
327+ used and no zero is replaced : a zero count is part of the likelihood.
328+ ` mn_penalty` sets an L2 (ridge) penalty on the log-odds (0 is maximum
329+ likelihood); `mn_reference_taxon` names the reference (empty picks the most
330+ abundant fitted taxon).
331+ - `dirichlet_multinomial` : the same model with a Dirichlet prior over the
332+ composition, whose concentration absorbs between-sample overdispersion (La
333+ Rosa et al. 2012). The result reports the concentration `c` and
334+ ` theta = 1/(1+c)` ; `theta` near 0 is the multinomial.
335+ `dm_overdispersion_method` picks the estimator : ` mle` (default) profiles the
336+ likelihood alternately with the coefficients, `mom` takes one
337+ method-of-moments estimate.
338+
339+ # ### Why a count model rather than a t-test
340+
341+ Sequencing measures a sample of the library, not an absolute quantity : a sample
342+ of twice the depth has twice every count, and one taxon can only increase at
343+ another's expense. Comparisons of raw or rarefied abundances therefore carry the
344+ compositionality problem (Gloor et al. 2017). The multinomial models fit the whole
345+ composition at once and condition on each sample's total, so each coefficient is
346+ the log ratio of two taxa. That ratio is unchanged when every count in a sample
347+ is multiplied by a constant, which is why no size factor and no pseudocount is
348+ needed. Because a log ratio is the sum of log ratios through any third taxon,
349+ every reported effect is coherent under compositionality and can be
350+ re-expressed against any other taxon.
351+
352+ # ### Choosing the reference taxon
353+
354+ Every reported effect is anchored on the reference taxon, so the choice matters
355+ for interpretability and for stability, but not for the fit : the likelihood and
356+ the log ratio between two non-reference taxa are the same whichever taxon is the
357+ reference. An empty `mn_reference_taxon` takes the most abundant taxon that
358+ passes the prevalence filter. An explicit one must be a taxon at the analysed
359+ rank with reads in at least half the samples, and reads in both groups. Taxa with
360+ no reads in one group are removed from the composition and listed as `failed`
361+ with their note rather than fitted : in a joint model a single diverging
362+ coefficient would otherwise pull the whole fit with it.
363+
364+ # ### Overdispersion
365+
366+ The multinomial treats every read as independent, so its variance is the sampling
367+ variance and nothing else. Real samples vary more than that : the same community,
368+ sequenced twice, differs by more than resampling reads from one library. The
369+ Dirichlet-multinomial adds that between-sample variation as a concentration `c`,
370+ with `theta = 1/(1+c)` the model's intra-sample correlation. When the data are
371+ overdispersed the multinomial's standard errors are too small, and the
372+ Dirichlet-multinomial corrects them; when they are not, `c` runs to its upper
373+ bound, the fit collapses to the multinomial exactly, and the affected rows are
374+ marked `boundary` with a note saying so. The two models are otherwise the same
375+ model on the same scale, so their coefficients are directly comparable.
376+
377+ ` max_iter` , `tolerance` and `max_seconds` bound one multinomial fit; a fit that
378+ runs out of time fails with an explicit error instead of holding the request
379+ open. Runtime and memory at 100, 1000 and 10000 taxa are measured by
380+ ` bench/compositional_scaling/scaling.jl` , which writes `results/scaling.csv`.
308381
309382ZINB and hurdle models can overfit sparse data; do not select ZINB solely
310383because counts are sparse. Compare against NB diagnostics. The methods report
@@ -316,15 +389,19 @@ histories and is not selectable in the standard analysis configuration or UI.
316389It currently has no detection covariates. Do not use it as a substitute for
317390the full workflow tracked in [issue # 205](https://github.com/hyperpolymath/MetaManifold-WebUI/issues/205).
318391See [occupancy model notes](docs/occupancy-models.md).
319- Across the differential methods, `min_prevalence` (default 0) filters taxa by the fraction of
320- samples with nonzero counts. Benjamini–Hochberg adjustment includes only taxa
321- that produced a p-value. Failed or filtered taxa have no substitute p-value;
322- inspect their status and note in the results table. Invalid input or an
323- unavailable required R package produces an explicit error.
324-
325- Implementation and reference tests : ` src/analysis/differential.jl` ,
392+ Across the differential methods, `min_prevalence` (default 0) filters taxa by the
393+ fraction of samples with nonzero counts. Benjamini–Hochberg adjustment is applied
394+ to every fitted taxon and cannot be turned off : it includes only taxa that
395+ produced a p-value. Failed or filtered taxa have no substitute p-value; inspect
396+ their status and note in the results table. Invalid input or an unavailable
397+ required R package produces an explicit error.
398+
399+ Implementation and reference tests : ` src/analysis/compositional.jl` (the two
400+ multinomial models), `src/analysis/differential.jl`,
326401` src/server/routes/analysis.jl` , `bench/compositional_differential/benchmark.jl`,
327- and `test/unit/test_differential.jl`.
402+ and `test/unit/test_differential.jl`. The reference values in the tests come from
403+ an independent implementation; the special functions are cross-checked against
404+ mpmath at 40 digits.
328405
329406# ## Figures and report
330407
@@ -662,13 +739,20 @@ analysis:
662739 normalisation: hellinger # hellinger | none | rarefy | srs
663740 depth: 0 # rarefy/srs reads per sample; 0 = smallest non-empty sample
664741 differential:
665- method: nb_glm # nb_glm | clr_welch | ancom_bc | aldex2 | songbird
742+ method: nb_glm # nb_glm | clr_welch | zinb | hurdle_nb | ancom_bc
743+ # | aldex2 | songbird | multinomial | dirichlet_multinomial
666744 dispersion: ml # nb_glm: ml | glmGamPoi
667745 replacement_method: bayesian_multiplicative
668746 # clr_welch: bayesian_multiplicative | multiplicative_replacement
669747 replacement_delta: 0.65 # clr_welch: strictly between 0 and 1
670748 bayesian_alpha: ~ # clr_welch: null (GBM estimate) or a number > 0
671749 mc_samples: 128 # aldex2: Dirichlet Monte-Carlo instances in [16, 4096]
750+ mn_reference_taxon: ~ # multinomial/DM: reference taxon, ~ picks the most abundant
751+ mn_penalty: 0.0 # multinomial/DM: L2 (ridge) penalty, 0 = maximum likelihood
752+ dm_overdispersion_method: mle # DM: mle (profiled) | mom (method of moments)
753+ max_iter: 500 # multinomial/DM: Newton iteration cap
754+ tolerance: 1.0e-8 # multinomial/DM: convergence tolerance
755+ max_seconds: 300.0 # multinomial/DM: wall-clock budget for one fit
672756 nmds:
673757 max_stress: 0.2 # warn if NMDS stress exceeds this value
674758` ` `
0 commit comments