Repository navigation
Add a validated Julia statistics layer with exact counts/rationals and higher precision #1
Description
Suggested labels: enhancement, statistics, scientific-validation.
Companion: Symbolic-engine proposal, which is blocked by this work and a separate approval gate.
User direction and scope
Add an optional Julia-owned statistical-analysis layer to MetaManifold, surfaced through Stipple/Vue. Preserve every existing feature and scientific workflow; do not silently change existing pipeline algorithms or previously saved analyses.
The user is interested in the first three forms of “exact” capability discussed, but prioritises only the first two for initial implementation:
- Exact integer counts and rational arithmetic where mathematically appropriate.
- Higher-precision floating-point computation for explicitly supported operations.
- Exact statistical tests: a later, separately validated extension, not part of the initial implementation acceptance gate.
Higher precision is not exact arithmetic. Neither removes measurement error, sampling uncertainty, model misspecification or biological bias. Symbolic mathematics is explicitly out of scope for this issue.
Proposed work
Numeric integrity
- Preserve integer counts; detect overflow and use checked/wider or arbitrary-size integers where justified.
- Support exact rational transformations for selected operations, with explicit handling of zero denominators and resource limits.
- Introduce an explicit numeric policy covering supported ordinary precision and arbitrary precision, precision settings, rounding/conversion rules and provenance.
- Audit CSV, JSON, database, browser, R and external-tool boundaries for precision loss, integer range and serialization compatibility. Never silently downgrade a requested numeric mode.
- Isolate precision configuration across concurrent analyses; avoid shared mutable precision settings leaking between users or jobs.
- Clearly distinguish exact stored/computed values, numerical approximations and rounded display values.
Statistical methods
- Define a bounded initial catalogue of methods before implementation, documenting supported response types and study designs.
- Support maximum-likelihood fitting for justified models, with explicit parameter constraints, convergence, identifiability and boundary-estimate checks.
- Support appropriate parametric and nonparametric analyses. Nonparametric is a class of methods, not a universal safe distribution or an assumption-free alternative.
- Treat Poisson/negative-binomial and binomial/beta-binomial models as candidates for review, not automatically approved choices for sequencing data.
- Require appropriate handling of overdispersion, zeros, sequencing depth, compositionality, covariates, pairing, blocking and repeated measurements where relevant.
- For permutation/bootstrap methods, validate exchangeability or resampling units, record seeds and resampling settings, and report uncertainty and Monte Carlo limitations.
- Specify uncertainty estimation, diagnostics, effect sizes, multiple-testing policy and warnings for every supported method.
Safe configuration and user experience
- Attach versioned configurations to immutable source runs; create linked derived analyses rather than overwrite existing results.
- Default to descriptive summaries when the information needed for valid inference is unavailable. Do not guess missing study design.
- Never silently switch statistical methods or select a method solely by a normality test or favourable p-value.
- Failed fits, unsupported precision, invalid input and resource exhaustion must produce explicit unsuccessful states, not plausible-looking results.
- Record inputs, transformations, model, settings, seeds, software/reference versions and diagnostics for reproducibility.
- Explain assumptions and numerical limitations in accessible user-facing language.
- Keep application logic Julia-owned; no new application TypeScript. Small necessary JavaScript adapters and Bun-managed tooling remain permitted.
Required deep validation and acceptance criteria
This is a scientific capability, not merely a form and an optimiser. Define method-specific acceptance tolerances and study-design coverage before claiming support.
- Approve the bounded method catalogue and publish supported/unsupported conditions.
- Publish numeric conversion and cross-language/storage contracts, including integer/rational round trips and explicit rejection of unsupported representations.
- Test known-answer cases and independent reference implementations; shared implementation assumptions alone are not independent validation.
- Test numerical stability across precision levels and adversarial cases: huge counts, sparse/all-zero tables, tiny probabilities, near-singular fits, boundary parameters, overflow/underflow and invalid/nonfinite inputs.
- Add property/generative tests for meaningful invariants and metamorphic relationships, plus fuzzing at parsing/configuration boundaries.
- Demonstrate negative controls and mutation/sensitivity tests: intentionally wrong inputs or implementations must be detected. Record commands and failing evidence.
- Use simulation studies to assess estimator bias, uncertainty-interval coverage and relevant error rates across supported conditions. Set justified tolerances in advance.
- Validate representative biological datasets and appropriate null/alternative cases; document where model assumptions fail.
- Verify reproducibility, serialization, concurrent precision isolation, cancellation, resource limits and failure recovery.
- Test UI-to-backend-to-result workflows, accessibility, provenance display and clear unsuccessful states.
- Verify existing scientific behaviour remains unchanged when the new analysis layer is not selected.
- Benchmark time/memory and bound expensive precision/resampling operations on supported platforms.
- Run the applicable repository/owner standards gates in CI, retaining results, coverage and benchmark artifacts; justify genuine N/A entries.
- Obtain independent statistical/scientific review and record limitations and release approval.
“Fully and deeply tested” means documented coverage of the declared scope, independent review and explicit residual-risk acceptance—not a claim that exhaustive testing proves all possible behaviour correct. Critical scientific-correctness, security or data-integrity defects must be resolved before release or before unblocking symbolic work.
Follow-up: exact statistical tests
Exact statistical tests remain desired, but are deferred beyond the initial integer/rational and higher-precision work. Each future test needs its own assumptions, supported designs, computational limits, known-answer cases and validation. Do not substitute an approximate method silently when an exact method is infeasible.
Dependency gate for symbolic mathematics
The companion symbolic-engine issue must stay blocked until the statistics layer completes the above validation, evidence is reviewed, blockers are resolved, and the owner explicitly approves moving to the symbolic stage. Shipping a UI, obtaining green unit tests, or closing this issue administratively is not sufficient to open that gate.
Activity
hyperpolymath commentedon Sep 22, 2026
Started, foundationally, in #49 — the numeric-integrity half of the layer, which the
methods half has to build on.
Delivered: exact integer counts and
Rational{BigInt}proportions, aBigFloatprecision scope, and an explicit
NumericPolicySpec(immutable, fingerprinted forprovenance) covering the numeric policy, rounding/conversion rules and the exact/
approximate/rounded distinction. Nothing existing changes behaviour:
:ordinaryremains Float64 throughout, so saved analyses are untouched.
The refusals matter more than the arithmetic, and each is a real hazard rather than
a hypothetical: Julia answers
1//0for a non-zero numerator, so an all-zero samplewould otherwise yield a non-finite
Rationalthat renders as a number — that case nowreturns
nothing, an explicit unsuccessful state. Counts arriving as Float64 beyond2^53 are refused, since such a value cannot prove which integer it was and may already
have been rounded by R, a CSV reader or a JavaScript client. Sums raise
CountOverflowErrorinstead of wrapping, rational growth is bounded by a stated budgetrather than an OOM, and NaN/Inf are refused at the JSON boundary. Precision isolation
across concurrent analyses is by serialisation in a lock, because Julia's BigFloat
precision is a process global; six concurrent tasks at six precisions are asserted to
each observe their own.
Published:
docs/statistics/numeric-contracts.md(the cross-language and storagecontracts, boundary by boundary: JSON, CSV, R, DuckDB, the browser — criterion 2) and
docs/statistics/method-catalogue-v1-draft.md(a bounded catalogue withsupported/unsupported conditions, drafted for your approval — criterion 1, which
this issue requires before methods are implemented).
Still open, and not claimed by #49: catalogue approval, per-method known-answer and
simulation validation, adversarial and property coverage at method level, UI-to-backend
workflow tests, and independent statistical review. The negative and metamorphic
controls do exist at the numeric layer (144 tests: overflowing sums, all-zero tables,
denominator exhaustion, non-finite input, 50 random tables whose proportions must sum
to exactly 1, and controls asserting the lossy alternatives really are lossy).
The symbolic gate in #2 stays blocked.
feat(analysis): exact counts and rationals with an explicit numeric p…
hyperpolymath commentedon Sep 22, 2026
Second numeric-integrity step, in #52: the boundary audit this issue asks for,
done by crossing each boundary with a real partner rather than by restating the
contract.
9007199254740993as a JSON numberarrives as
9007199254740992(one short, unmarked); as the encoded string itcrosses byte for byte.
as.integer(2^31)isNA) andevery number is a double; neither
format()noras.character()renders that countback to its digits. So a count that has been through R is unproven, and the
contract's route (send and read it as a string) is asserted by round trip.
BIGINTcarries the count exactly,DOUBLEsilently changes it; pastBIGINTaBIGINTcolumn refuses rather than rounds;HUGEINTis exact to2^127; beyond that, the string form."9.007199254740992e15"and"9007199254740993.0"are refused by name.
parse_exact_count/parse_exact_rationalnow close the round trip as an executableinverse of
to_storage, because a value that has passed through a float cannot beinspected for the damage — the string form is the only representation allowed to claim
to be a count. 45 boundary assertions alongside the existing 144.
Still outstanding on #1, unchanged and not claimed: approval of the bounded method
catalogue (
docs/statistics/method-catalogue-v1-draft.md— your call), then per-methodknown-answer and simulation validation, adversarial and property coverage at method
level, UI-to-backend workflow tests, benchmarks for the expensive precision paths, and
independent statistical review. The symbolic gate (#2) stays blocked.
test(analysis): measure the numeric boundaries instead of assuming them
test(analysis): measure the numeric boundaries instead of assuming them
hyperpolymath commentedon Sep 22, 2026
Verification note — the boundary tests are not dead code in CI. Worth stating
because the CI log does not name them: Julia's test summary collapses passing nested
testsets into the parent row, so
test_numeric_boundaries.jlcontributes toMetabarcodingPipeline | 1993without ever being printed. I checked that this iscollapsing rather than absence, arithmetically:
2f59384(before this work): 1938 passing1a71048(after ci(tools): retry pinned downloads and name a TLS failure as one #51 + test(analysis): measure the numeric boundaries instead of assuming them #52): 1993 passingassertions (ci(tools): retry pinned downloads and name a TLS failure as one #51). Accounted for exactly, to the assertion.
Both summary rows show
Pass / Totalwith no Broken column, and1993 == 1993.That matters because the boundary file skips loudly by name when a partner is missing
(
@test_skip), so a missing node, Rscript or DuckDB would have appeared as a Brokencount. It did not: in the CI runner all three partners were present and every boundary
was crossed for real.
The alternative check — push a deliberately failing assertion and watch CI go red —
turned out to be unnecessary once the counts reconciled, so no mutation branch was left
behind.
State: #49
e1856af, #510ef5eee, #52387fff5, #531a71048. CI on the currentmain tip is green (Julia 1993/1993, hygiene green, gate triage skipped by design).
hyperpolymath commentedon Sep 22, 2026
The bounded method catalogue is approved — recorded in
docs/statistics/method-catalogue-v1.md(PR #55), approved without amendment by therepository owner on 2026-09-22.
That satisfies this issue's requirement that the catalogue and its supported/unsupported
conditions be agreed before methods are implemented. What the approval covers, in the
document's own words:
explicit constraint and convergence reporting, (3) nonparametric permutation/bootstrap
with stated exchangeability and seeds.
needed for valid inference is missing.
What it does not cover: exact statistical tests (item 4, tracked in #3), any method
outside the list, or the symbolic work (#2), which stays blocked. Approval is of scope and
gate, not of method behaviour — each method still publishes its conditions, evidence and
negative controls before it is implemented, and nothing is implemented merely by
appearing in the list.
Next increment, now unblocked: item 1 — exact descriptive summaries (counts and
proportions as exact integers/rationals, no inference claimed), building on
numeric_policy.jland the boundary audit already onmain. It will arrive with thatmethod's published conditions, known-answer and independent-reference evidence, and its
negative controls.
docs(statistics): publish the conditions for exact descriptive summar…
feat(analysis): exact descriptive summaries (catalogue item 1)
docs(statistics): approve the v1 method catalogue (#55)
feat(analysis): exact descriptive summaries (catalogue item 1) (#57)
hyperpolymath commentedon Sep 22, 2026
Catalogue item 1 has shipped — exact descriptive summaries.
5441bd99(main tip); PR docs(statistics): approve the v1 method catalogue #55, the catalogue approval, asab68a6fa.src/analysis/exact_summaries.jl+ 97 assertions intest/unit/test_exact_summaries.jl,held to the conditions published in
docs/statistics/method-conditions/exact-descriptive-summaries.md(merged in docs(statistics): publish the conditions for exact descriptive summaries #56, before the implementation existed).
Counts are exact integers, proportions are rationals, a zero-total sample carries
nothingrather than0(the distinction a single number cannot express — the caseExecution.jlwrites0.0for today, deliberately left unchanged), floats above 2^53 arerefused rather than trusted, and group summaries aggregate counts rather than averaging
proportions.
Three things worth recording, because they are the expensive kind of progress:
text written beforehand exposed two defects in the numeric layer underneath: a rounding
labelled "6dp" that printed ~80 digits (
roundon a BigFloat keeps its significand),and exact rationals rendered as Julia's
2//3in text a person reads. Both fixed. Therounding now happens in integer arithmetic — correct past Float64, and dependent on
nothing outside Base.
import Printfpassed every localharness, because a harness mounts source into an ad-hoc module and cannot enforce
package dependency rules; precompilation failed with
Package MetaManifold does not have Printf in its dependencies. Fixed by removing thedependency rather than declaring it.
"stdlibs need no
[deps]entry" exemption list that is factually wrong (onlyBase,CoreandMainare bound without a declaration; verified by experiment), and itexamined only the first name in a comma list, so
using JSON3, YAMLnever checkedYAML. Rewritten to read the AST, with a self-test that replays the exact input which
defeated it and the inverses that must not be reported — mutation-checked, because a
gate that cannot fail is decoration.
Next, under the approved catalogue order: item 2, parametric fits with explicit
constraint and convergence reporting. Per the catalogue's own rule, the conditions
document comes first and the implementation is held to it.
fix(analysis): heal zero-depth samples before the transform (#58)
fix(analysis): the analysis path stops returning placeholder statisti…
feat(proofs): formal verification of the numeric core in Agda (issue #1)
feat(proofs): Agda formal verification of the numeric core (issue #1) (…
feat(proofs): unify Agda suite, audit gates, and run-notices catalogue (
feat(occupancy): grade the probe's samples against the HWM algebra (#105
hyperpolymath commentedon Oct 16, 2026
Triage note, 2026-10-15 (backlog sort). The sort is done around this issue, and it sharpens what lands here: with the register's re-measurement (his Oct-7 chain absorbed
clr_welch, the PERMANOVA terms, the significance statuses, and the BH/scaling proofs), this layer is now a re-derivation on live ground, not a port of the oldAnalysisConfig.jlcluster — those paths exist nowhere (measured; register unit statuses 01–09, 13). The exact-rational summaries slice is the entry point (no upstream collision, clearest proof obligations); the higher-precision float path second; exact tests third per the issue's own priority. Two backlog families wait on this one's first slice: the external packages of #17/#5 (his #22/#23 shapes).Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe vibe@mistral.ai
docs(milestones): restore the deferred-features spec the backlog issu…