Skip to content

Prepare upstream PR 62 review fixes: theta, label-safe glmGamPoi contrast - #192

Merged
hyperpolymath merged 3 commits into
upstream/glmgampoi-rfrom
claude/elegant-rubin-z92rx4
Oct 20, 2026
Merged

hyperpolymath merged 3 commits into
upstream/glmgampoi-rfrom
claude/elegant-rubin-z92rx4

Conversation

@hyperpolymath

Copy link
Copy Markdown
Owner

Change and reason

Prepares the next push to upstream JoshuaJewell/MetaManifold-WebUI#62, the glmGamPoi dispersion option for JoshuaJewell#37. The branch is upstream PR 62's reviewed head (efe1df1) plus three new commits, so the update fast-forwards. Under upstream CONTRIBUTING item 8, nothing is rebased once review has started.

  1. Report glm_gp's negative-binomial overdispersion as theta. Your wip/pr62-theta commit, cherry-picked unchanged. It covers both of Joshua's code requests from Oct 19 (theta is 1 / overdispersions; the Poisson boundary is read from it).
  2. Pass the glmGamPoi contrast to test_de as numeric weights. glmGamPoi::test_de parses and evaluates a text contrast as R code. The contrast label is built from run, group and sub-group names joined by /, and those names may contain -. So the glmGamPoi path failed every taxon for comparisons such as run-A vs run-B or grpA/run_1 vs grpB/run_2, and label text that reads as R code was executed. The commit adds a regression test that covers both dispersions.
  3. State what glmGamPoi's theta, standard error and boundary mean. These are precision fixes, checked against the glmGamPoi 1.22.0 source:
    • overdispersions is the per-taxon estimate before shrinkage. The coefficients and the F-test use the cross-taxon trend, and only lfc_se reads overdispersions.
    • The boundary note no longer says the fit is Poisson.
    • The README now lists dispersion with the other study-scoped settings.

Do not merge this PR on GitHub. Merging would put unsigned commits, or a merge commit, onto upstream PR 62. Re-sign on your machine and push instead:

git fetch origin claude/elegant-rubin-z92rx4 upstream/glmgampoi-r
git switch -c pr62-next origin/claude/elegant-rubin-z92rx4
git rebase --exec 'git commit --amend --no-edit -S' efe1df1   # re-signs only the 3 new commits
git log --format='%h %G? %s' efe1df1..HEAD                     # expect G or E, never N
git push origin pr62-next:upstream/glmgampoi-r                  # fast-forward: no --force needed

When Joshua asks for a rebase (upstream main is now ce2c645), the five-commit stack rebases cleanly onto it. That was checked in a scratch worktree.

Validation

Environment: Julia 1.12.5 (the CI pin, from the official image), R 4.3.3, MASS 7.3-60.0.1, glmGamPoi 1.14.3, bun 1.4.2.

glmGamPoi 1.14.3 lacks compute_lfc_se, so 1.22.0's pure-R test_de, test_de_q and predict.glmGamPoi were loaded over it. Diffing the 1.14.0 and 1.22.0 sources shows no other change on these paths. Upstream CI runs the real pins (R 4.5, glmGamPoi 1.22.0).

  • CI=true julia --project=. test/runtests.jl test_differential.jl:

    • head: 253 passed, 0 failed, 5 skipped. The skips are clr_welch tests that need zCompositions. CI=true makes a missing glmGamPoi fail rather than skip, so every glmGamPoi test ran.
    • commits 1, 2 and 3 on their own: 202, 253 and 253 passed, 0 failed.
    • the new label test on the pre-fix code: 5 failures, as expected. The four label sets fail every taxon, and the R-code label ran.
  • Label experiments (the fit_nb R block, run directly):

    • Before: run-B, grpB/run_2 and 2024-02 failed every taxon, and a label of [file.create(...)] created the file.
    • After: every label set gives rows identical to A/B, the file is not created, and glm_nb is unchanged.
  • Null calibration (share of p < 0.05, binomial SE about 0.003):

    samples per group glm_nb (Wald) glmGamPoi (QL F)
    4 0.132 0.046
    12 0.069 0.048
  • Frontend: bun test src 39 pass; typecheck clean; build succeeds.

Not run: the integration and server suites, the DADA2 taxonomy and phylogeny paths, and the clr_welch tests (zCompositions unavailable; they skip loudly).

Scope

No lockfile change. The upstream PR description still has to be replaced; Joshua's third request asks for that. A draft follows, together with the summary comment for upstream issue 37.

Draft replacement description for upstream PR 62
## Change and reason

Adds an optional `analysis.differential.dispersion` setting for the `nb_glm` method (issue #37). `glm_nb` stays the default and is unchanged: `MASS::glm.nb` fits each taxon with its own maximum-likelihood theta, and the group coefficient gets a Wald test.

`glmGamPoi` fits one `glmGamPoi::glm_gp` across the taxa that pass `min_prevalence`, with `overdispersion_shrinkage = TRUE`. It reports `glmGamPoi::test_de`'s quasi-likelihood F-test of the group contrast (Lund et al., 2012), which is glmGamPoi's own published inference. The trend-plus-Wald design in the first version of this PR has been replaced. In that path:

- **estimate and se**: `lfc` and `lfc_se`, converted from log2 to the natural-log scale. `lfc_se` comes from `predict()` and does not feed the p-value.
- **statistic and pvalue**: the F statistic and its p-value. The F-test refits the reduced model at the cross-taxon dispersion trend and divides by the shrunken quasi-likelihood dispersion.
- **dispersion_theta**: `1 / overdispersions`. This is glm_gp's negative-binomial overdispersion, the taxon's own estimate before shrinkage, and the same parameter `MASS::glm.nb`'s theta inverts (requested in review).
- **Poisson boundary**: based on that overdispersion (requested in review). A zero estimate marks the row `boundary` and keeps its statistics, since the test does not read that estimate. A negative or non-finite estimate fails only its own taxon.
- **contrast**: passed to `test_de` as numeric weights on the design's columns. `test_de` parses and evaluates a text contrast as R code. Before this change, a contrast label containing `-` or `/` (e.g. `run-B`, or `grpB/run_2`, which the app builds when conditions differ by group and run) made every taxon fail. A label that reads as R code was executed.
- **failures**: a missing glmGamPoi raises `GlmGamPoiUnavailable` (HTTP 503). A `glm_gp` or `test_de` error fails every taxon, with no fallback to `glm_nb`. `glmGamPoi` is refused for `clr_welch`.

Config and UI: the config editor and `PATCH /api/v1/config` accept the new enum (`glm_nb`, `glmGamPoi`). The constructor refuses an invalid method/dispersion pair when the analysis runs (HTTP 400 `invalid_config`). There is no other config rewrite; the first version's description wrongly claimed one. The README lists the option with the other study-scoped settings.

Commits, oldest first (the last three are new since your review):

1. `02ee915` Add glmGamPoi to the pinned R dependencies
2. `efe1df1` Add glmGamPoi dispersion option to nb_glm differential abundance
3. Report glm_gp's negative-binomial overdispersion as theta (review: theta and boundary)
4. Pass the glmGamPoi contrast to test_de as numeric weights (labels with `-` or `/`; label text evaluated as R)
5. State what glmGamPoi's theta, standard error and boundary mean (comments, note text, README, result type)

Closes #37.

## Validation

Run in a cloud container with Julia 1.12.5 (the CI pin), R 4.3.3, MASS 7.3-60.0.1, glmGamPoi 1.14.3, and bun 1.4.2. glmGamPoi 1.14.3 lacks `compute_lfc_se`, so 1.22.0's `test_de`, `test_de_q` and `predict.glmGamPoi` (pure R) were loaded over it. Diffing the 1.14.0 and 1.22.0 sources shows no other change on these code paths: `glm_gp_impl`, `parse_contrast`, the overdispersion estimate and the quasi-likelihood shrinkage are the same. CI on this PR runs the pinned R 4.5 with glmGamPoi 1.22.0.

- `CI=true julia --project=. test/runtests.jl test_differential.jl`:
  - **branch head:** 253 passed, 0 failed, 5 skipped. The skips are the `clr_welch` tests, which need zCompositions. Under `CI=true` a missing glmGamPoi fails rather than skips, so the glmGamPoi tests all ran.
  - **each new commit on its own:** 202, 253 and 253 passed, 0 failed.
  - **the new label test against the pre-fix code:** it fails as intended. Each of the four label sets fails every taxon, and the R-code label's code runs.
- The `fit_nb` R block, run directly in R on the `effect` fixture:
  - **labels**: `run-A`/`run-B`, `grpA/run_1`/`grpB/run_2` and `2024-01`/`2024-02` failed every taxon before the fix. With it, every label set, including one containing R code, gives rows identical to `A`/`B`.
  - **injection**: a label of `[file.create(...)]` created the file before the fix and does not after it.
  - **glm_nb**: rows are unchanged.
  - **field check**: setting `overdispersions[1] <- 0` changes `lfc_se[1]` only; `pval`, `f_statistic` and `lfc` stay the same.
- Null calibration through the same R block (no taxon differs; TSS offsets; dispersion falling with the mean). Share of p < 0.05:

  | samples per group | taxa tested | glm_nb (Wald) | glmGamPoi (QL F) |
  |---|---|---|---|
  | 4 | ~5,900 | 0.132 | 0.046 |
  | 12 | 4,000 | 0.069 | 0.048 |

  The binomial SE is about 0.003.
- Frontend (`frontend/`): `bun test src` 39 pass; `bun run typecheck` clean; `bun run build` succeeds.

Not run here: the integration suite (`--integration`, `--server`), the DADA2 taxonomy and phylogeny paths, and the `clr_welch` tests (zCompositions was not available in the container; they skip loudly).

## Scope

Left out: a native Julia gamma-Poisson fit (#37 settled on R), zero-inflated and occupancy models, any change to the `glm_nb` default or its results, and any UI beyond the one enum.

Lockfile: `renv.lock` adds glmGamPoi 1.22.0 and the eleven packages it needs that were not already pinned. Existing pins are unchanged. The option cannot run without them, and CI and `install.jl` check that glmGamPoi loads.

Noted for a separate issue, not this PR: in the calibration above, the default `glm_nb` Wald test rejects too often at small n.

## Contribution checks

- [x] I followed [CONTRIBUTING.md](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/CONTRIBUTING.md).
- [ ] This branch starts at current upstream main and contains no merges from fork main. It starts at `029fa61` and contains no merges. Since review has started, it gains commits rather than being rebased (item 8). It rebases cleanly onto `ce2c645` whenever you ask.
- [ ] Every commit is signed and shows as Verified. *(tick after re-signing)*
- [ ] CI has run and passes. *(tick after the push)*
- [x] Existing licensing, citation and acknowledgement text is unchanged.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01HtyPfZ1MWBB94RawgzQgz7
Draft summary comment for upstream issue 37
**Summary for the record**

**Question (Oct 15):** take glmGamPoi's dispersion shrinkage for `nb_glm` (a) in R through RCall, or (b) as a native Julia gamma-Poisson fit with R kept as the oracle?

**Answer (@JoshuaJewell, Oct 17):** (a). "glmGamPoi stays R, since there is no reputable Julia equivalent."

**What follows from that, in #62:**

- **Option:** `analysis.differential.dispersion: glmGamPoi` is optional. The default stays `glm_nb` (`MASS::glm.nb`), and its results are unchanged.
- **Test:** inference is glmGamPoi's own quasi-likelihood F-test (`glm_gp` + `test_de`), not the trend-plus-Wald plan from Oct 15.
- **Review fixes:** theta is `1 / overdispersions` (glm_gp's NB overdispersion), and the Poisson boundary is read from it, as requested. The contrast is now passed as numeric weights. Before that, any label containing `-` or `/` (e.g. `grpB/run_2`) made every taxon fail, and label text was evaluated as R code.
- **Dependencies:** `renv.lock` adds glmGamPoi 1.22.0 and the eleven packages it needs; existing pins are unchanged.
- **Evidence:** in a null simulation, the share of p < 0.05 was:

  | samples per group | `glm_nb` (Wald) | glmGamPoi (QL F) |
  |---|---|---|
  | 4 | 13.2% | 4.6% |
  | 12 | 6.9% | 4.8% |

  The `glm_nb` excess at small n is outside this issue's scope and is worth its own issue.

**Good to go.** The R/RCall route is settled, and #62 is ready for re-review once the three new commits are signed and pushed.

Generated by Claude Code

hyperpolymath and others added 3 commits October 20, 2026 12:42
The glmGamPoi path reported 1 / ql_disp_shrunken as the dispersion
theta, but that is the quasi-likelihood scale test_de divides by, not
the negative-binomial parameter MASS::glm.nb's theta inverts. Read
glm_gp's overdispersions element instead (variance = mu + mu^2 *
overdispersion) and report its inverse, so dispersion_theta means one
thing under both dispersion settings.

The Poisson boundary moves with it: a taxon whose negative-binomial
overdispersion is zero is marked boundary, as under glm_nb, and keeps
its statistics, because the F-test's denominator is the shrunken
quasi-likelihood dispersion and stays defined there. The earlier
withholding was justified only by a zero quasi-likelihood scale, which
this check no longer reads. A negative or non-finite overdispersion
still fails only its own taxon. theta is nothing where it is unbounded.

The direct R reference in the tests inverts gp$overdispersions, and the
isolation test overrides that element, which test_de reads neither for
its refit nor for its F ratio, so the test under comparison is unchanged.

Requested in review on #62.
glmGamPoi::test_de parses a text contrast with rlang::parse_expr and
evaluates it, so the coefficient name built from the contrast label
reached R as code. Condition labels join group, run and sub-group
names with "/", and those names may carry "-", so comparing
"grpA/run_1" with "grpB/run_2", or "run-A" with "run-B", stopped
test_de and failed every taxon, while glm_nb fitted the same labels.
A label that reads as R code ran: with glmGamPoi 1.14.3 carrying the
1.22.0 test_de, a label of [file.create("<path>")] created the file
before test_de stopped.

The design matrix is now built once, and test_de receives the numeric
contrast that selects the contrast column, which it uses without
parsing. For labels that were already syntactic the rows are identical.

A test fits both dispersions under labels with "-", "/" and R code, and
requires the A/B results and that the code never ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtyPfZ1MWBB94RawgzQgz7
With overdispersion_shrinkage = TRUE, glm_gp returns each taxon's own
overdispersion estimate in `overdispersions` but refits the
coefficients at the cross-taxon trend, and test_de refits the reduced
model at that trend and divides by the shrunken quasi-likelihood
dispersion; only predict(), which supplies lfc_se, reads
`overdispersions`. The comments, the boundary note, the isolation
test's comment, the README and the result type now say so: theta is
the taxon's estimate before shrinkage, and a zero estimate leaves the
test's statistics standing because the test used the trend, so the fit
there is not Poisson.

The README lists dispersion with the other study-scoped
analysis.differential settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtyPfZ1MWBB94RawgzQgz7
@coderabbitai

coderabbitai Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 38f51354-a3b0-4db1-8b1c-c99918c2d03d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sonarqubecloud

Copy link
Copy Markdown

@hyperpolymath
hyperpolymath marked this pull request as ready for review October 20, 2026 13:15
@hyperpolymath
hyperpolymath merged commit e28e832 into upstream/glmgampoi-r Oct 20, 2026
3 checks passed
@hyperpolymath
hyperpolymath deleted the claude/elegant-rubin-z92rx4 branch October 20, 2026 13:15
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T13:18:23.383875Z 10e099b Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

hyperpolymath pushed a commit that referenced this pull request Oct 20, 2026
The owner chose to replace the squash commit e28e832 on
upstream/glmgampoi-r with #192's three commits, re-signed. Those
commits are still reachable as refs/pull/192/head after the branch was
deleted, so the entry gives the fetch, the re-sign from the reviewed
head, an empty-diff check against e28e832, and a push whose lease names
e28e832. The sequence was rehearsed in a scratch clone with a
no-signature amend and `git push --dry-run`: three commits rewritten,
no diff, lease accepted, nothing pushed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HtyPfZ1MWBB94RawgzQgz7
hyperpolymath added a commit that referenced this pull request Oct 20, 2026
…h path (#194)

## Change and reason

This is a fork-only note, not meant for upstream. It corrects
`notes/commit-signing.md` (#191) in three places.

1. **A PR already under review.** The routine's `git rebase --exec ...
upstream/main` replays every commit after upstream's tip. On a PR branch
that started from an older `main`, that is itself a rebase, which
upstream CONTRIBUTING item 8 allows only when Joshua asks. A new section
re-signs only the commits added since the reviewed head, after checking
that the branch contains that head, and pushes them as a fast-forward:

   ```sh
   git merge-base --is-ancestor <reviewed-head> HEAD &&
     git rebase --exec 'git commit --amend --no-edit -S' <reviewed-head>
   git push origin <local>:<pr-branch>   # no --force
   ```

Without the check, a branch that lacks the reviewed head would be
replayed onto it, upstream commits included, and still push as a
fast-forward. On `wip/pr62-theta` that would replay upstream's `78ffc2f`
re-signed.

2. **Squash-merging the fork PR instead.** This is the path #192 took.
GitHub signs the squash commit and the upstream PR moves forward, but:
- the commit's message defaults to the fork PR's description, and it
lands on Joshua's `main` unchanged (`e28e832` carries #192's whole
description);
   - the prepared commits collapse into one;
- the signature is GitHub's key, not one registered to your account.
Upstream CONTRIBUTING item 3 asks for the latter, and has not been asked
about this.

3. **Branch lists.** You chose to replace `e28e832` with #192's three
commits, re-signed. They are still reachable as `refs/pull/192/head`:

   ```sh
   git fetch origin pull/192/head:pr62-next upstream/glmgampoi-r
   git switch pr62-next
   git merge-base --is-ancestor efe1df1 HEAD &&
     git rebase --exec 'git commit --amend --no-edit -S' efe1df1
git diff --stat origin/upstream/glmgampoi-r HEAD # expect no output
git push
--force-with-lease=upstream/glmgampoi-r:e28e832fd52466f5192406025fe6b16d9fd99ae1
origin pr62-next:upstream/glmgampoi-r
   ```

`wip/pr62-theta` moves to a new "Superseded branches" section, kept
rather than deleted.

## Validation

All of this ran in scratch clones or worktrees, with a no-signature
`--amend` standing in for `-S`, since this container has no key.

- **Old routine** (`--exec ... upstream/main`) on upstream PR 62's
stack: rewrote all five commits, the two reviewed ones included.
- **New routine** (`--exec ... efe1df1`): rewrote the three new commits
only, with the tree unchanged.
- **Guard:**
- on `wip/pr62-theta`, the check failed, the rebase did not run, and
HEAD was unchanged;
- on `refs/pull/192/head`, it passed, and three commits were rewritten
with the tree unchanged.
- **Restore sequence:**
  - the diff against `e28e832` was empty;
- `git push --dry-run` with the lease reported a forced update of
`e28e832` only;
  - `upstream/glmgampoi-r` was unchanged afterwards.

Documentation only: no Julia, R or bun code changed, and no tests were
run.

## Scope

Leaves out any change to signing configuration. Nothing is pushed to
`upstream/glmgampoi-r` here; that push is yours. No lockfile change.

## Contribution checks

- [x] I followed [CONTRIBUTING.md](../CONTRIBUTING.md).
- [ ] This branch starts at current upstream main and contains no merges
from fork main. *(n/a: fork-only note, branched from fork `main`)*
- [ ] CI has run and passes. *(pending on b18c3b2)*
- [x] Existing licensing, citation and acknowledgement text is
unchanged.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01HtyPfZ1MWBB94RawgzQgz7

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants