Skip to content

fix(analysis): the analysis path stops returning placeholder statistics - #60

Merged
hyperpolymath merged 1 commit into
mainfrom
arena/01a0da3a-metamanifold-webui
Sep 25, 2026
Merged

hyperpolymath merged 1 commit into
mainfrom
arena/01a0da3a-metamanifold-webui

Conversation

@arena-ai-coding-agent

Copy link
Copy Markdown

The problem this fixes

Execution.run_analysis returned, per feature:

p_val   = 0.01 + (h % 100) / 1000.0     # h = hash(taxon_id)
padj    = p_val * 1.5                   # "mock BH"
log2fc  = (h % 20) / 10.0 - 1.0
baseMean= 100.0 + (h % 1000)

Every number is a deterministic function of the feature's name. Nothing in the
run touches the counts. The rows render in a table as p-values, fold changes and
means, and a caller has no way to tell them from a fit. The source called it a stub;
the caller received it as a result.

What replaces it

src/analysis/estimation.jl (catalogue item 2 — conditions published before the
implementation, in docs/statistics/method-conditions/parametric-fits.md):

method fit note
nb_glm MASS::glm.nb, per feature offset required; glmGamPoi/local/mean/pooled refused by name (issue #21)
clr_lm, ilr_lm stats::lm, per feature offset refused: it means nothing to a Gaussian fit on CLR units
logistic stats::glm(binomial), per feature 0/1 response required; proportions refused rather than given invented weights
  • Unsuccessful states are states. A failed fit gets status = "failed", a note
    saying why and null for every statistic; it is excluded from the BH family and
    counted. Nothing that cannot be fitted gets a number.
  • A run that cannot happen says so. No design, no metadata column, single-level
    grouping, RBusyError past the timeout, R unreachable → status = "not_run" with a
    reason and an empty result set, and a warning on the diagnostics. An empty result is
    never dressed up as "no significant features".
  • Refusals, not substitutions: interactions/transformations/random effects/nesting
    in a formula, $/backticks/; in a formula, offsets on non-count models, count models
    without one, binomial fits on proportions.
  • Provenance: R and MASS versions, formula terms and the contrast actually tested,
    offset kind and its SHA-256, BH family size, exclusion count, and the SHA-256 of the
    fits and coefficients tables as they crossed the R boundary.
  • BH implemented in Julia and tested against R's p.adjust(method = "BH").

Evidence (test/unit/test_estimation.jl)

  • Known answers written into the data (four-fold up, no effect, ten-fold down) rather
    than recorded from a previous run.
  • Independent reference: the same counts fitted directly in R, compared coefficient by
    coefficient, SE by SE, p by p.
  • Negative controls for every refusal, and for not_run/failed shapes.
  • A source-level guard that hash(taxon_id), hash( and rand( are gone from the
    execution path.

Refs #1 (catalogue item 2 delivered; item 3's nonparametric tests and the independent
statistical review are still outstanding there). Issue #21 stays open: glmGamPoi is
refused rather than aliased, and the refusal says so.

run_analysis derived its p-values from 0.01 + (hash(taxon_id) % 100)/1000.0,
padj from pvalue * 1.5 and log2FoldChange from (hash % 20)/10 - 1: every one of
those is a function of the feature's NAME, not of the counts, and they were
returned to callers as results.

The placeholder is deleted rather than deprecated. src/analysis/estimation.jl
runs the declared model for real (MASS::glm.nb with a required offset for
nb_glm, stats::lm for CLR/ILR, stats::glm binomial for logistic on a 0/1
response), records R and MASS versions, offset hash and result hashes, excludes
failed fits from the BH family while counting them, and returns an explicit
not_run with a reason — never an empty table that reads as 'nothing was
significant' — when no design or no runtime is available.

Conditions published in docs/statistics/method-conditions/parametric-fits.md
(catalogue item 2). Evidence in test/unit/test_estimation.jl: written-in known
effects, a direct R reference compared coefficient by coefficient, BH against
R's p.adjust, negative controls for every refusal, and a source-level guard that
the placeholder does not come back.

Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Sep 25, 2026

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: cf416472-674e-45e0-b8b3-81527cfea448

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@hyperpolymath
hyperpolymath merged commit 3e3af79 into main Sep 25, 2026
5 of 6 checks passed
@hyperpolymath
hyperpolymath deleted the arena/01a0da3a-metamanifold-webui branch September 25, 2026 20:36
hyperpolymath added a commit that referenced this pull request Sep 25, 2026
## Why

Julia tests on `main` have been red since #60, and the reason was never
visible: the job log is served from a blob host that some environments
cannot reach, and the one annotation the step posted (the last 6000
bytes of output) was truncated by the checks API at 4096 bytes — which
cut it off exactly before the failing testset. For `e26d6f2` (#63) all
it shows is that `Scaling — TSS, CSS, RSS/TMM and size factors (issue
#16)` has a failure.

## What

The "Run tests" step, on failure only, now posts:

- one `::error` annotation per failing assertion (the Test.jl block
starting at `Test Failed at` / `Error During Test at`, up to 60 lines /
3500 bytes, max 8), and
- one summary annotation holding only the `Test Summary` rows whose Fail
or Error column is non-empty (found by the header's column positions),
plus the `ERROR:` line.

If no failure block is found (e.g. a load error), the old tail
annotation is posted as a fallback. The test command and its exit status
are unchanged.

## Verified

- YAML re-parses; spdx/format/blob-hygiene gates clean.
- The extraction shell was run against a synthetic Test.jl log (one
`Test Failed`, one `Error During Test` with `Got exception outside of a
@test`, nested summary) and produced exactly two failure annotations and
a summary with only the failing rows.

## Not claimed

This does **not** fix the red test. It makes the failure readable so the
next PR can fix it rather than guess. This PR's own CI run should fail
the same way as `main` and name the assertion.

Co-authored-by: hyperpolymath <6759885+hyperpolymath@users.noreply.github.com>
hyperpolymath added a commit that referenced this pull request Oct 17, 2026
… work (#178)

## Purpose

Make the fork's anticipated upstream state concrete while Joshua's
reviews/merges remain pending. This is a **draft baseline proposal**,
not evidence that upstream has merged anything.

The proposed file tree is exactly JoshuaJewell/main at
`52feb605176a4b140f4b40c2b74d2f18adec1f72` plus:
- upstream #59 at `3c9d352e1f6c85353c2d78ef1ebf28e5fb64c263`
- upstream #60 at `ea974a2328f7f44ad9afeb2f10e5803cd4e31e62`
- upstream #61 at `d9ea982dc97c958405e1ff6f57f1b0d8d9b221d1`

Resulting tree: `b27daa7f1aa5fa84f320ddae522fa5f2a15ebef5`.

## Effect on fork main

This deliberately removes fork-only content from the proposed active
baseline: 74 files differ from current main. The existing main commit
`491a3e0702ca8ca40e411860d6166523b298b3ac` is the first parent, and all
three repair heads are additional parents, preserving the work in
history. No existing branch is deleted or rewritten. Do not submit this
integration branch upstream.

**Before merging:** decide whether main should be the upstream-aligned
baseline, with unfinished work isolated onto feature branches. This PR
removes real fork features from the active tree, not just stale merge
metadata. Recheck upstream and repair heads after Joshua's final
decisions; their eventual file tree may change.

## Branch audit

There were seven total branches when inspected: main and six additional
branches.

| Branches | Finding | Disposition |
| --- | --- | --- |
| repair/permanova-proofs | Clean pending upstream #59 | Keep until
merged |
| repair/documentation | Clean pending upstream #60 | Keep until merged
|
| repair/dada2-base-r | Clean pending upstream #61 | Keep until merged |
| upstream/permanova-proofs, upstream/docs-community,
upstream/dada2-base-r | All three have identical tree
2361ab8; old contaminated submissions,
not independent WIP | Retire after baseline decision; no unique feature
work needs completing on these branches |
| main | Contains additional work beyond the repaired upstream
contributions | Preserve and isolate selected changes |

Compared with the old three branches, main changes only .gitignore, the
differential fixture, the restored deferred-issues specification and
removal of the duplicate root code of conduct. Their substantive work is
preserved.

## Genuine work to isolate and finish

These are code-review findings, not claims that the features have passed
a fresh full validation.

| Work | Evidence / remaining work |
| --- | --- |
| glmGamPoi dispersion | Actual implementation and configuration exist,
but glmGamPoi is absent from renv.lock. Existing tests do not establish
successful reference fits through this path. Pin dependencies and
validate against trusted R fits before treating it as complete. |
| Significance proof model | The Agda model makes noncomputed numbers
unrepresentable, while the Julia documentation explicitly permits
retained pairwise rows after an omnibus failure. Reconcile model and
implementation; do not claim the proof establishes the Julia
representation's behavior. |
| Canonical JSON/JCS | Utility and fixtures exist; no production callers
found beyond module inclusion/export. Establish conformance and choose
an actual integration point before claiming end-to-end provenance. |
| Memory occupancy | Measurement, algebra/proof integration, tests and
scripts exist. Isolate the work, verify sibling dependency pins and real
runtime measurements. This is distinct from biological occupancy
modelling in issue #18. |
| Wildcard-IP origin handling | Separate behavioral change in server
middleware. Review address/port parsing and trust assumptions, then run
host/origin, DNS-rebinding and CSRF regressions before submission. |
| Plot title markup stripping | Small independent helper and usage
change; suitable for a focused bugfix after checking its tests and
behavior. |

Repository layout changes, frontend major upgrades, community policies,
automation and other tooling are separate fork-maintenance decisions,
not prerequisites for the three upstream repairs. Do not carry forward
the obsolete claim in fork AGENTS.md that locked ggplot2 requires
tibble: the retained tibble dependency is used by installation/CI
tooling.

Open planning issues are not evidence of implemented branch WIP; several
overlap or describe functionality already upstream.

## Validation

- All six repair commits applied together without conflicts.
- git diff --cached --check passes.
- Local combined tree and GitHub-created tree match exactly.
- Latest individual PR CI runs passed: #59 run 38976014775; #60 run
38998820680; #61 run 38998823755.
- Combined baseline CI is separate and must be inspected before merging.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Joshua Jewell <developer@joshuajewell.dev>
Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant