feat(analysis): complete occupancy models and imperfect-detection workflow #205

Goal

Complete the end-user occupancy / imperfect-detection workflow that was deliberately left outside the first implementation slice for #18. The merged #196 slice supplies selectable zinb and hurdle_nb two-part differential fits and a lower-level repeated-visit occupancy likelihood, but it is not yet a complete occupancy feature.

Remaining scope

• Add first-class occupancy to AnalysisConfig and the analysis endpoint, accepting a real repeated-visit design rather than inferring repeat visits from ordinary samples.
• Add validated occupancy_detection_formula, zinb_zero_formula, and hurdle_count_dist (nb, lognormal, poisson) settings; update Nickel/DEED contracts, JSON/config schemas, config cascade validation, API types, and provenance/DOI bundles.
• Allow detection covariates (at minimum log library size and batch/primer where present) separately from biological occupancy predictors. Validate formula syntax, column existence, weak/non-identifiability risks, and warn/refuse when the occupancy and detection designs cannot be distinguished.
• Add the Advanced Analysis UI: occupancy vs logistic vs ZINB vs hurdle explanations and citations; repeated-visit input/design; detection-formula editor with library-size autocomplete; estimated runtime; conspicuous scientific caution/identifiability messages. Keep BH mandatory.
• Add supported excess-zero diagnostics/model comparison (including a justified ZINB-vs-NB test or calibrated alternative); do not default to zero inflation merely because a table is sparse. Preserve the distinction between NB overdispersion, structural zeros, and hurdle occurrence.
• Add simulation/reference validation: recover ψ≈0.7 and p≈0.5 to the stated tolerance across 100 taxa; compare against a documented reference implementation and data fixtures; validate ZINB/hurdle inference and false-positive behaviour.
• Benchmark runtime and memory at 100 and 1,000 taxa. Warn on runs expected to exceed five minutes; gate CI against >10% regression in existing analysis methods.
• Document closure, destructive microbiome sampling, the controversy around single-visit occupancy, and interpretation of ψ and p. Cite MacKenzie et al. (2002), Martin et al. (2005), and an appropriate microbiome occupancy reference.
• Refresh the repository Wiki roadmap/user/statistics/developer pages, README and EXPLAINME receipts so they state which portion is implemented and which remains planned; remove stale issue-number/status claims.

Acceptance

A user can configure and run occupancy on an explicitly valid repeated-detection design, interpret ψ and p with the stated assumptions, reproduce the exact formulas and count distribution from stored provenance, and receives refusal/warnings for invalid or weakly identifiable designs. ZINB/hurdle choices have documented diagnostics and calibrated tests. CI passes Julia/R, frontend, schema/contracts and benchmark/regression checks.

Related

Follow-up to #18 and partial implementation in #196. Keep this issue open until the acceptance criteria above are met.