Repository navigation
Fix the unplaced-reads regression #236 introduced on main - #237
Conversation
…ance #236 relaxed `PLACEMENT_WEIGHT_SUM_TOL` from 1e-5 to 0.05 so that a candidate list truncated by the placement tool still counts as `:valid`. The `unplaced_reads` field of the clade-cumulus diversity payload reused that same constant as its reporting floor, so a placement at LWR 0.99 — whose remaining 1% genuinely never reached the tree — was silently reported as 0 unplaced reads. `Julia tests` on the merged #234 tree caught it: 5679 pass, 2 fail, both in `test/unit/test_clade_cumulus.jl` ("6. POST combined mode against a study `.jplace` artifact"), expecting 0.05 and 0.10 and getting 0. The push-run CI for #236 itself was cancelled by the workflow concurrency group, so the regression reached `main` unverified. Reporting now uses `PLACEMENT_WEIGHT_OVER_TOL` (1e-9) as its floor, which is what the metric actually means, and the comment records why the two tolerances are not interchangeable. The existing assertions cover it; no test change. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 10 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Fork governance
🟣 critical · commit
Hypatia scan: 141 finding(s)
panic-attack assail: 19 finding(s)
pons asinorum: 10 finding(s)
rhodibot (RSR compliance): not_run — enrolled, but no report-back was received
panicbot (panic recovery, error handling): not_run — enrolled, but no report-back was received
echidnabot (formal verification: proofs/): not_run — enrolled, but no report-back was received
seambot (integration health: API routes): not_run — enrolled, but no report-back was received
finishingbot (release readiness): not_run — enrolled, but no report-back was received
accessibilitybot (WCAG on the React UI): not_run — enrolled, but no report-back was received
glambot (presentation quality): not_run — enrolled, but no report-back was received
cipherbot: not_applicable — no cryptography is implemented in this repository
sustainabot: not_applicable — fleet adapter slot reserved; no adapter exists yet (gitbot-fleet README)
Scope: static findings reported by each row's tool at the named commit. Purple, red, yellow and blue grade the worst finding present; orange is evidence missing, malformed or a scanner that did not run; grey is an explicit scope exclusion with its reason. No row is a security certification. report.json, badge.json and the raw findings are in the verdict-grid artifact on the linked run. |
|
`main` landed its own answer to most of #230 while this branch was open: #236 adopted the asymmetric placement tolerance (shortfall tolerated to 0.05, excess rejected at 1e-9) and #237 added the unplaced-reads noise floor. So this merge gives up re-litigating the tolerance and keeps `main`'s model verbatim: `PLACEMENT_WEIGHT_SUM_TOL = 0.05`, `PLACEMENT_WEIGHT_OVER_TOL = 1e-9`, the asymmetric parse branch, and the partition expectations that follow from it (0.96 is a truncated candidate list and is `:valid`; 1.04 is overweight evidence and stays `:undecided`). What is kept from this branch is what `main` does not have: * the library-size denominators (`_sample_total`, `totals` / `total_a` / `total_b`) that stop weighted UniFrac renormalising unplaced mass back onto the tips that were placed -- the Codex P1 finding; * the route's whole-table `unplaced_reads`, which counts an ASV the tree does not cover at all -- the Codex P2 finding -- instead of summing per-row placement shortfalls. It subsumes #237's noise floor: subtracting the placed mass from the library size reports a 0.99 placement's 1% exactly, with no threshold to tune; * the emptied-clade fix (`own_count`, `own_residual`) and its regression test; * the exact method+path route assertions; * the corrected unplaced-reads wording in the panel and the API types. The single-candidate reason stays `single_candidate_below_min_lwr (<lwr>)`: under `main`'s asymmetric rule a lone candidate's sum is in tolerance by construction, so the old wording named a weight-sum error that had not happened. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
…ngs, harden frontend parsers (#239) Supersedes #230 and #235. This branch is #235 (which already carried #230 merged onto `main`) fast-forwarded onto current `main` (`1f8701d`, incl. #237), plus the fixes below. No merge conflicts remain: the seven-file conflict set documented on #235 was resolved there, and this branch merges cleanly. ## Disposition of every app finding on #230 / #235 | Finding | Status | |---|---| | Codex P1 — unplaced mass renormalised away by UniFrac (#230) | Fixed on #230/#235 (`totals`/`total_a`/`total_b` library-size denominators); carried here, verified present | | Codex P2 — fully unmapped ASVs not counted as unplaced (#230) | Fixed on #230/#235 (`unplaced_reads` = library size − tree-carried mass); carried here | | CodeRabbit — unplaced-reads copy reversed the calculation (#230) | Fixed on #235 (panel + `api/types.ts` now say: excluded from tip abundances, included in the library-size denominator); carried here | | CodeRabbit — emptied clade kept stale activity (#230) | Fixed on #235 (`_reaggregate_after_move` recomputes from `direct_count`); carried here | | Codex P2 — vacated clade kept stale `residual_count` (#235) | Already fixed on #235 by `58848d9`; verified present here, no duplicate change | | CodeRabbit — emptied internal clade still counted as a taxon tip (#235) | **Fixed in this PR**: `count_leaf_taxa!` and the leaf branch of `_reaggregate_after_move` now share one tip-eligibility rule (direct reads, or terminal `tip`/`ASV` rank). New regression assertion: two-domain move scenario where root purity is 0.5 with the fix vs 0.6 if the wrapper counts as a fifth tip | | CodeRabbit `occursin` route assertions (#235) | Addressed on #235 (whole method+path pairs); carried here | The unresolved review threads on #230 and #235 are being resolved with pointers to this PR; Codex is at its usage limit (per its own comments on #230), so no re-review was available from it. ## Hypatia / DEED (K9-format estate governance) - **DEED lane: 🟢 conformant** on #235's head and unchanged here (no `.deed` file touched). Local contract check: `scripts/check-governance-contract.sh` — all 20 checks pass. - **Hypatia grid: 🟣 critical (45C/61H/35M)** — unchanged in kind. The criticals are `cicd_rules/banned_language_file` against the upstream TypeScript frontend: this fork's deed states the frontend is upstream-owned, so rewriting it is out of scope and a `.hypatia-baseline.json` acknowledgement is an owner decision — deliberately **not** committed here, because committing one (with `scripts/apply-baseline.sh`) arms the blocking gate in `hypatia-scan-reusable.yml`, and that gate cannot be exercised locally (no Elixir/Julia toolchain in this environment). Arming an untested blocking gate would risk turning every PR red. - Real findings fixed here anyway (panic-attack `InputBoundary`, advisory): SSE handlers in `frontend/src/api/events.ts` no longer throw on malformed frames; `parseJplace` in `frontend/src/tree/model.ts` reports invalid JSON as a jplace error. Also fixed the pons `string-concat-in-loop` in the Newick quoted-label reader. - Left as documented false positives / follow-ups: `JSON.parse(JSON.stringify(...))` deep-clones in `charts.ts`/`alphaMetrics.tsx` (cannot throw on malformed input — the input is a live object), the remaining pons string-concat heuristics in `TreeShapes.tsx` path building, and panic-attack's per-file `JSON3.read` try-block counts on Julia routes (requests are handled behind the routes' error boundary; changing six route files blind, without a Julia runtime, is riskier than the finding). ## Validation - Frontend: `tsc --noEmit -p .` clean. - `git diff --check` clean; governance contract script passes (20/20). - Julia is not installable in this environment (as on #230/#235), so the Julia changes are static-checked and rely on this PR's Julia CI job as the real check — watching it before merge. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com> Co-authored-by: Arena Agent <agent@arena.ai> Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
`main` now holds all of it: #236/#237 the asymmetric placement tolerance and the unplaced-reads accounting, #239 the #235 work itself -- the library-size denominators (`_sample_total`, `totals`), the route's whole-table `unplaced_reads`, the emptied-clade `own_count`/`own_residual` fix with its regression test, the exact method+path route assertions, the corrected unplaced-reads wording, and the vacated-clade tip-count rule. So this merge takes `main`'s side throughout, including over the two CodeRabbit autofix commits that landed on the branch afterwards: `main`'s in-tree tip rule is the same rule as the branch's `counts_as_tip` helper, and the branch's extra 'zero-read terminal' testset asserted a target purity of 0.5 where the code computes 2/3 (two GenusA tips out of three, the unassigned tip included), so it could not have passed. Nothing here needs re-landing. The branch tree is now identical to `main`. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
…N with 400 (#244) ## Change and reason Three things, in the order the linked review asked for them. ### 1. PR #228's merge conflicts, resolved (and shown to be a no-op) PR #228 (`arena/7805092e`) branched from `8df1041`; all six files it touched have since been rewritten on `main` by #231, #233, #234, #236, #237 and #239. Resolving the six conflicted files in favour of `main` produces a tree that is **byte-identical** to `main` (`git diff --stat origin/main` is empty), so the merge commit records the resolution and changes no content. Concretely: * `ci.yml` - main publishes a per-assertion digest of every failing validated-statistics test plus a log tail; #228's awk windows are superseded. * `src/analysis/clade_cumulus.jl` - main already reads `PhyloNode.edge_number` in `_graft_jplace_queries`, already sums `rk.subtree_pd` plus the node's own branch, and additionally recomputes an emptied clade from `direct_count`. * `src/server/routes/clade_cumulus.jl` - main already uses `with_results_db` and `String(strip(...))`, and deliberately does **not** renormalise placement like-weights by the retained total: `parse_jplace` already rejects ratios outside [0, 1], a multi-candidate placement already partitions its mass, and dividing a lone 0.6 candidate by its own total would move the 0.4 the tool left unplaced onto the reported edge (#236/#237 report it as `unplaced_reads` instead). * `src/server/routes/validated_statistics.jl` - main validates raw tokens for the whole body after `ijson_read` proves it I-JSON, and streams bounded rows instead of materialising a DataFrame. Both are strict supersets. * `test/unit/*` - main's suites are far larger; the one test #228 had that main lacked ("Nested-clade drag-and-drop counts each active branch once") is restored here. ### 2. Codex P2 on PR #228: leaf subtree PD `_reaggregate_after_move` credits an active leaf with its own incoming branch, so a parent can sum `subtree_pd` alone and count each active edge once. The ordinary builder (`build_node`) hard-coded `0.0` for leaves, so validating a drag-and-drop changed the reported PD of leaves whose topology had not changed and made the proposed tree's metrics incomparable with the original. Both builders now take a leaf's PD from `clade_faith_pd` - the function the internal-node branch already used - so the conventions cannot drift again. Internal nodes and inactive tips are unchanged. ### 3. panic-attack `InputBoundary`: malformed bodies are 400s 35 unguarded `JSON3.read(String(req.body))` call sites across eight route files raised a JSON3 parse error out of the handler, so a truncated or hostile body produced a 500. Every route now parses through `Server._request_json`, which throws the existing `BadRequest`. The helper copies `req.body`, because `String(::Vector{UInt8})` can consume the buffer. Rescuing that turned out to need one more step, which CI caught: Oxygen builds its chain as `custom(serializer(router))` and its serializer catches every handler exception and answers 500 before our middleware runs, so `_bad_request_middleware` never saw the throw. Oxygen's error mapper is the hook inside the serializer, and it is already a dispatch point (`handle_error(::ValidationError)` answers 400). Extending it for `BadRequest` keeps this server's `{"error", "message"}` shape and makes the test's `status == 400` true. ### 4. pons asinorum: tree path building is linear The six `string-concat-in-loop` hits in `TreeShapes.tsx` were real: every node appended to four shared strings inside a `useMemo` over the whole layout, which is quadratic in the node count on a view that renders every node. The fragments are collected per node and joined once; the emitted order is unchanged. ### 5. panic-attack `ProofDrift` and a guarded benchmark baseline The `@test x isa Y` assertions the rule flagged were real tautologies: they passed for any string. The occupancy pair now asserts the text the code emits (`measured 201 KiB above the predicted 101 KiB`) and the evidence round trip asserts the ISO-8601 UTC stamp `save_document` writes. The advanced zero-handling benchmark parsed its baseline outside any guard, so a truncated baseline surfaced as a JSON3 failure from inside the timing loop; it now names the file it could not read. ## Validation - `cd frontend && npx tsc --noEmit -p .` - clean. - `cd frontend && npx vite build` - clean (chunk-size warning only, pre-existing). - `cd frontend && bun test src` - **72 pass, 0 fail** (11 files). - `bash scripts/check-governance-contract.sh` - **all 20 checks passed**. - `bash scripts/lint-sonar-reliability.sh` - clean. - `bash -n` on every `scripts/*.sh` - clean. - All four workflows parse as YAML; the DEED lints clean with the pinned `deed_lint.js` from `hyperpolymath/standards@75bd5994` (see below). - **Not run locally:** the Julia suite (no Julia runtime in this environment; the two changed Julia files are covered by CI's `Julia tests` job). ## Hypatia and K9/DEED conformance * **DEED** - `MetaManifold-WebUI_chora.deed` lints clean with the exact linter the reusable lane runs (`bun 1-formats/deed/tools/deed_lint.js`). Note the stdlib Python port in the same directory is older and rejects `#u8`; the JS linter the workflow uses accepts it, so the deed is correct as written. * **K9** - this repository has no `.machine_readable/contractiles/` registry or Mustfile, and `governance.yml` does not call `k9-contractile.yml`, so the K9 contractile lane is not in scope here. What *was* wrong is that the deed declared `:report-backs ".gitbot-fleet/report-backs/"` while the directory did not exist; it now does, with the envelope contract documented, and the grid keeps rendering those members orange until a real report-back lands. * **Hypatia** - the 43 critical findings are all `cicd_rules/banned_language_file` for `frontend/src/**/*.ts`. That rule is a repo-side-unfixable policy finding: TypeScript's only carve-outs are narrow path prefixes (`bindings/typescript/`, `.d.ts`, `vite.config.ts`, ...), none of which cover an upstream-authored React frontend, and the rule cannot be suppressed from the repository. Migrating upstream's frontend to AffineScript is an estate project, not a change this fork can make. * **panic-attack / pons asinorum** - most remaining findings are stale or false positives at `main`: `api/events.ts` and `tree/model.ts` already wrap `JSON.parse` in try/catch (hardened in #239), the `DynamicCodeExecution` hits are `RCall.reval` (R source strings, not Julia `eval`), and the `unreachable-after-jump` hits point at plain arrow-function bodies. The two classes that were real - unguarded request-body parsing and quadratic path building - are fixed above. The grid's residual after this PR: Hypatia 45 critical (all `banned_language_file` on upstream TypeScript), 61 high, 35 medium; panic-attack 6 high (all `RCall.reval` false positives), 7 medium; pons asinorum 2 medium (`unreachable-after-jump` on a `useMemo` return and a `useEffect` cleanup, both false positives). ## Scope - [x] Focused on the linked review plus conformance evidence - [x] Does not alter dependencies, lockfiles, generated assets, or vendored files - [x] Secrets, local paths, and run outputs are not committed ## Contribution checks - [x] I followed CONTRIBUTING.md - [x] My commits include `Signed-off-by:` (DCO) and the licence is unchanged --------- Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: CodeRabbit <noreply@coderabbit.ai>



mainis currently red onJulia tests— 5679 pass, 2 fail, 0 error — and both failures come from #236, whose push-run CI was cancelled by the workflow concurrency group before the suite ran.Cause
#236 relaxed
PLACEMENT_WEIGHT_SUM_TOLfrom1e-5to0.05so that a candidate list truncated by the placement tool still parses as:valid— a correct change for validity. But theunplaced_readsfield of the clade-cumulus diversity payload reused the same constant as its reporting floor:With
TOL = 0.05, a row at weight0.99no longer qualified as short, so a placement at LWR 0.99 — whose remaining 1% genuinely never reached the tree — reported0unplaced reads instead of0.01 × reads. The test's placement (ASV_4at LWR 0.99, 5 and 10 reads) therefore expected0.05and0.10and got0.Fix
Reporting now uses the noise floor the metric actually means,
PLACEMENT_WEIGHT_OVER_TOL(1e-9, added by #236 for the opposite bound of the same check):1.0 - wsum > 1e-9iswsum < 1.0 - 1e-9for all shortfalls and additionally excludes overweight rows, which a shortfall comparison cannot express. The comment records why the two tolerances are not interchangeable, so the next change to the placement tolerance cannot silently re-break the report.No test changes: the two existing assertions are exactly the regression guard, and they were the only failures in the suite — my new indexed-tree testset passed (
CladeCumulus & Phylogenetic Integration: 196 pass, 0 fail).Also worth knowing for future merges:
actions/runs/39967407149(the push run forb8b6c9e) showscancelled, which is why #236 could merge without its suite running.