Skip to content

Commit b1022be

Browse files
fix(core): load the compositional models and unbreak the build tree
`using MetaManifold` fails on main. The merge that landed the compositional differential work left two load-time faults and dropped the documentation for the models that were merged: - `src/core/doi.jl` declared `module DOI` twice (lines 4 and 21). The inner declaration swallowed the module body and left the outer one without an `end`, so the file could not be included at all. - `src/MetaManifold.jl` included `analysis/clade_cumulus.jl` twice and never included `analysis/compositional.jl`. `Differential` does `using ..CompositionalModels`, so the package died with `UndefVarError: CompositionalModels not defined` before any test could run. The duplicate include is dropped and the compositional module is included ahead of differential.jl, with a comment saying why the order matters. - The README had lost the `ancom_bc`, `aldex2`, `songbird`, `multinomial` and `dirichlet_multinomial` bullets, the sections that explain why the joint models need no size factor or zero replacement, and the new `analysis.differential` settings. The config block still advertised five methods and six keys. ROADMAP follows suit. The routes, the frontend and the tests were already repaired by #211 and #213; this is the remaining part that keeps the package from loading. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
1 parent c978994 commit b1022be

5 files changed

Lines changed: 107 additions & 12 deletions

File tree

‎README.md‎

Lines changed: 93 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -305,6 +305,79 @@ not exclude samples from this analysis.
305305
multiplicative (CZM) replacement; `replacement_delta` sets the replacement
306306
fraction and optional `bayesian_alpha` overrides GBM's concentration. The
307307
offset is not used. See [zero-handling notes](docs/statistics/zero-handling.md).
308+
- `ancom_bc`: the bias-corrected log-linear model of Lin & Peddada (2020), which
309+
estimates within-group sample-specific offsets and a between-group sampling
310+
fraction from a 3-component Gaussian mixture fitted by safeguarded SQUAREM.
311+
The effect is a bias-corrected log2 fold change; `offset` and the zero
312+
replacement settings are not used.
313+
- `aldex2`: ALDEx2's Dirichlet Monte-Carlo CLR instances (Fernandes et al. 2014),
314+
with a Welch test per instance and a per-instance sign-consistency check; the
315+
effect is the standardized CLR difference. `mc_samples` sets the number of
316+
Dirichlet draws (whole number in [16, 4096], seeded by the study seed).
317+
- `songbird`: Songbird's baseline-category compositional multinomial regression
318+
(Morton et al. 2019; Zhang et al. 2017), fitted in R by damped Newton-Raphson
319+
with exact Sherman-Morrison information inversion. It reports CLR-centred
320+
differentials, which sum to zero over the fitted taxa. `songbird` and
321+
`multinomial` are two implementations of the same model, in R and in Julia:
322+
`songbird` reports centred differentials, `multinomial` reports log ratios
323+
against one reference taxon and has the Dirichlet-multinomial sibling.
324+
- `multinomial`: one multinomial logistic model over the whole composition
325+
(Songbird's model, Morton et al. 2019), fitted in Julia. The effect is a log2
326+
fold change in relative abundance against a reference taxon. `offset` is not
327+
used and no zero is replaced: a zero count is part of the likelihood.
328+
`mn_penalty` sets an L2 (ridge) penalty on the log-odds (0 is maximum
329+
likelihood); `mn_reference_taxon` names the reference (empty picks the most
330+
abundant fitted taxon).
331+
- `dirichlet_multinomial`: the same model with a Dirichlet prior over the
332+
composition, whose concentration absorbs between-sample overdispersion (La
333+
Rosa et al. 2012). The result reports the concentration `c` and
334+
`theta = 1/(1+c)`; `theta` near 0 is the multinomial.
335+
`dm_overdispersion_method` picks the estimator: `mle` (default) profiles the
336+
likelihood alternately with the coefficients, `mom` takes one
337+
method-of-moments estimate.
338+
339+
#### Why a count model rather than a t-test
340+
341+
Sequencing measures a sample of the library, not an absolute quantity: a sample
342+
of twice the depth has twice every count, and one taxon can only increase at
343+
another's expense. Comparisons of raw or rarefied abundances therefore carry the
344+
compositionality problem (Gloor et al. 2017). The multinomial models fit the whole
345+
composition at once and condition on each sample's total, so each coefficient is
346+
the log ratio of two taxa. That ratio is unchanged when every count in a sample
347+
is multiplied by a constant, which is why no size factor and no pseudocount is
348+
needed. Because a log ratio is the sum of log ratios through any third taxon,
349+
every reported effect is coherent under compositionality and can be
350+
re-expressed against any other taxon.
351+
352+
#### Choosing the reference taxon
353+
354+
Every reported effect is anchored on the reference taxon, so the choice matters
355+
for interpretability and for stability, but not for the fit: the likelihood and
356+
the log ratio between two non-reference taxa are the same whichever taxon is the
357+
reference. An empty `mn_reference_taxon` takes the most abundant taxon that
358+
passes the prevalence filter. An explicit one must be a taxon at the analysed
359+
rank with reads in at least half the samples, and reads in both groups. Taxa with
360+
no reads in one group are removed from the composition and listed as `failed`
361+
with their note rather than fitted: in a joint model a single diverging
362+
coefficient would otherwise pull the whole fit with it.
363+
364+
#### Overdispersion
365+
366+
The multinomial treats every read as independent, so its variance is the sampling
367+
variance and nothing else. Real samples vary more than that: the same community,
368+
sequenced twice, differs by more than resampling reads from one library. The
369+
Dirichlet-multinomial adds that between-sample variation as a concentration `c`,
370+
with `theta = 1/(1+c)` the model's intra-sample correlation. When the data are
371+
overdispersed the multinomial's standard errors are too small, and the
372+
Dirichlet-multinomial corrects them; when they are not, `c` runs to its upper
373+
bound, the fit collapses to the multinomial exactly, and the affected rows are
374+
marked `boundary` with a note saying so. The two models are otherwise the same
375+
model on the same scale, so their coefficients are directly comparable.
376+
377+
`max_iter`, `tolerance` and `max_seconds` bound one multinomial fit; a fit that
378+
runs out of time fails with an explicit error instead of holding the request
379+
open. Runtime and memory at 100, 1000 and 10000 taxa are measured by
380+
`bench/compositional_scaling/scaling.jl`, which writes `results/scaling.csv`.
308381

309382
ZINB and hurdle models can overfit sparse data; do not select ZINB solely
310383
because counts are sparse. Compare against NB diagnostics. The methods report
@@ -316,15 +389,19 @@ histories and is not selectable in the standard analysis configuration or UI.
316389
It currently has no detection covariates. Do not use it as a substitute for
317390
the full workflow tracked in [issue #205](https://github.com/hyperpolymath/MetaManifold-WebUI/issues/205).
318391
See [occupancy model notes](docs/occupancy-models.md).
319-
Across the differential methods, `min_prevalence` (default 0) filters taxa by the fraction of
320-
samples with nonzero counts. Benjamini–Hochberg adjustment includes only taxa
321-
that produced a p-value. Failed or filtered taxa have no substitute p-value;
322-
inspect their status and note in the results table. Invalid input or an
323-
unavailable required R package produces an explicit error.
324-
325-
Implementation and reference tests: `src/analysis/differential.jl`,
392+
Across the differential methods, `min_prevalence` (default 0) filters taxa by the
393+
fraction of samples with nonzero counts. Benjamini–Hochberg adjustment is applied
394+
to every fitted taxon and cannot be turned off: it includes only taxa that
395+
produced a p-value. Failed or filtered taxa have no substitute p-value; inspect
396+
their status and note in the results table. Invalid input or an unavailable
397+
required R package produces an explicit error.
398+
399+
Implementation and reference tests: `src/analysis/compositional.jl` (the two
400+
multinomial models), `src/analysis/differential.jl`,
326401
`src/server/routes/analysis.jl`, `bench/compositional_differential/benchmark.jl`,
327-
and `test/unit/test_differential.jl`.
402+
and `test/unit/test_differential.jl`. The reference values in the tests come from
403+
an independent implementation; the special functions are cross-checked against
404+
mpmath at 40 digits.
328405

329406
### Figures and report
330407

@@ -662,13 +739,20 @@ analysis:
662739
normalisation: hellinger # hellinger | none | rarefy | srs
663740
depth: 0 # rarefy/srs reads per sample; 0 = smallest non-empty sample
664741
differential:
665-
method: nb_glm # nb_glm | clr_welch | ancom_bc | aldex2 | songbird
742+
method: nb_glm # nb_glm | clr_welch | zinb | hurdle_nb | ancom_bc
743+
# | aldex2 | songbird | multinomial | dirichlet_multinomial
666744
dispersion: ml # nb_glm: ml | glmGamPoi
667745
replacement_method: bayesian_multiplicative
668746
# clr_welch: bayesian_multiplicative | multiplicative_replacement
669747
replacement_delta: 0.65 # clr_welch: strictly between 0 and 1
670748
bayesian_alpha: ~ # clr_welch: null (GBM estimate) or a number > 0
671749
mc_samples: 128 # aldex2: Dirichlet Monte-Carlo instances in [16, 4096]
750+
mn_reference_taxon: ~ # multinomial/DM: reference taxon, ~ picks the most abundant
751+
mn_penalty: 0.0 # multinomial/DM: L2 (ridge) penalty, 0 = maximum likelihood
752+
dm_overdispersion_method: mle # DM: mle (profiled) | mom (method of moments)
753+
max_iter: 500 # multinomial/DM: Newton iteration cap
754+
tolerance: 1.0e-8 # multinomial/DM: convergence tolerance
755+
max_seconds: 300.0 # multinomial/DM: wall-clock budget for one fit
672756
nmds:
673757
max_stress: 0.2 # warn if NMDS stress exceeds this value
674758
```

‎ROADMAP.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,12 @@ occupancy work. The GitHub Wiki roadmap still needs a synchronized update; see
66

77
## Available now
88

9-
- `nb_glm` and `clr_welch` remain the existing differential methods.
9+
- The differential-abundance config offers four per-taxon count models
10+
(`nb_glm`, `clr_welch`, `zinb`, `hurdle_nb`) and five compositional models
11+
(`ancom_bc`, `aldex2`, `songbird`, `multinomial`, `dirichlet_multinomial`);
12+
see [the differential-abundance section](README.md#differential-abundance).
13+
`songbird` and `multinomial` are two implementations of the same
14+
baseline-category multinomial model.
1015
- `zinb` and `hurdle_nb` are available in the differential-abundance config and
1116
UI. They report the count effect and zero/occurrence effect and test group
1217
effects jointly. The UI surfaces model cautions; BH adjustment remains on.

‎docs/statistics/zero-handling.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,11 @@ log-ratio units, not a fold change. Replacement choice and parameters are
4040
returned in `config`; zero counts, imputed-mass summaries, warnings, and (for
4141
GBM) concentration information are returned in `diagnostics.zero_replacement`.
4242

43+
The multinomial models (`multinomial`, `dirichlet_multinomial`) replace no
44+
zero: a zero count is part of the likelihood, and because the model conditions on
45+
each sample's total every effect is a log ratio against a reference taxon rather
46+
than an absolute abundance.
47+
4348
Every zero-replacement method introduces assumptions and can bias results,
4449
especially when zeros are common or sample sizes are small. Do not choose a
4550
replacement or `delta` to obtain a preferred p-value. Consider the negative

‎src/MetaManifold.jl‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,9 @@ include("canonical.jl")
4444
include("analysis/clade_cumulus.jl")
4545
include("analysis/diversity.jl")
4646
include("analysis/analysis.jl")
47-
include("analysis/clade_cumulus.jl")
47+
# The compositional (multinomial / Dirichlet-multinomial) fits are included
48+
# before differential.jl, which imports CompositionalModels for those methods.
49+
include("analysis/compositional.jl")
4850
include("analysis/differential.jl")
4951
# Opt-in validated descriptive statistics (issue #1). Ordinary workflows above
5052
# remain unchanged; these modes must be selected explicitly.

‎src/core/doi.jl‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@ module DOI
1818
# * Durable write-ahead journal; immutable DOI/provenance receipts
1919
# * DOI insertion never rewrites scientific objects
2020

21-
module DOI
2221

2322
using Dates, SHA, HTTP, JSON3, Logging
2423

0 commit comments

Comments
 (0)