Repository navigation
Description
ACTIVE PRIORITY RESET — 30 October 2026: Joshua wants a scientifically robust, fast, dependable, single-user local application first, with organic use by his lab before any wider hosted deployment. The multi-user/zero-trust/Wolfi/jump server programme in #301 is parked. Do not promote architectural security work as current critical path. Immediate order: (1) complete and scientifically validate #64/PR #67, including signed commits and pinned R check; (2) recover interrupted SSH runs #66 in local single-user mode; (3) selectively consider #65 validated descriptive statistics (scientific review gates intact); (4) installation, UI, performance and reproducibility; (5) additional mathematics/evidence features only at maintainer request. Security bugs in the existing local deployment remain actionable, but no new login/SSO complexity. Previously recorded B0–B7 plans are retained for historical reference only.
Priority correction (2026-10-30): Jonathan relayed Joshua's explicit product requirement: MetaManifold must be a secure multi-user service. This supersedes any old assumption that authenticated multi-user deployment is optional. Core identity, backend authorisation, ownership, cross-tenant execution/storage isolation and attributable audit form a security-critical parallel delivery critical path alongside scientific #64/#65 and operational #66. In particular, #66 resume/cancel must not be implemented without principal/job binding. See #301. Details of deployment/SSO and sharing remain to be confirmed with Joshua.
Purpose
One living, human-reviewed coordination register. Not a request to merge fork main into Joshua's repository, and not a reason to duplicate upstream issues. The aim is to make contribution relevance visible despite legitimate divergence and to deliver small, validated changes under upstream CONTRIBUTING.
Baseline — measured 2026-10-30
| Reference | Commit |
|---|---|
Joshua main |
a325589d3d318847c15b21cd34fb7d6e7a6d2f7b |
This fork main |
fb10d66410ac018b41553d2b1c9de143744de2fb |
| Merge base | 257bcdec2db0d4231ed9d5b160b5caa623485681 |
Fork ahead 128 / behind 49 by commit reachability. Net path delta: 277 paths when comparing upstream to fork, versus 85 for the reciprocal comparison. This is not a count of defects, pending tasks or cherry-picks. No attempt should be made to drive these numbers to zero.
Existing upstream-based branches:
work/issue64-reproduction: ahead 2, behind 0 versus Joshuamain; contains reproducer and README for upstream #64. Preserve; runtime/pinned verification remains required.review-base/upstream-a325589: ahead 1, behind 0, but not an exact clean mirror: it includes the glmGamPoi zero-read correction from fork PR #298. Reclassify as upstream-candidate patch, not a tracking reference.governance-badges: orphan badge data lane, not an upstream contribution.
Transfer statuses (apply to each item separately)
shared-unmodified, upstream-accepted, fork-integrated, candidate, upstream-discussion, owner-declined, fork-only-by-design, external-independent, blocked-validation, superseded. No automatic 'stale' verdict merely because a declared lane differs from fork main.
Contribution blocks / owner discussion register
| Block | Upstream issue / fork evidence | What is actually available | Benefit if Joshua accepts | Risk/cost or owner choice | Next state |
|---|---|---|---|---|---|
| B0: SSH interruption/restart | upstream #66 | src/pipeline/remote_exec.jl is identical in both tips |
Restartability and less lost remote computation | Remote subprocess ownership, staging/retention, concurrency, cancellation, SSH failure modes | Upstream-first implementation with fault-injection tests, then fork integration |
| B1: glm_nb calibration | upstream #64, clean-base work/issue64-reproduction |
Reproducer script and write-up; not a confirmed default-method fix | Scientific honesty about small-sample false positives | Different generators, version parity, bias from dependent taxa, method change risks | Run pinned R suite and publish evidence, no default change |
| B2: constrained glmGamPoi correction | fork PR #298, clean-base candidate | Fixes zero-read taxa in shared fit; restores Wald wording | Prevents all-zero taxa from influencing other taxa's QL trend | Behaviour/test changes, R/glmGamPoi pinned versions; do not conflate with B1 | Verify in upstream CI; offer separately after review |
| B3: exact descriptive statistics | upstream #65; fork PRs #199, #202, #257, #259 | Implemented in fork only; NOT upstream/released as independently biologically validated | Exact counts/rationals, explicit precision and refusals | Large scientific and API surface, R/Julia/JS boundaries, persistence/GUI, proof and validation load | Selective sub-block port after agreement and evidence |
| B4: placement confidence | upstream #27, upstream main includes placement LWR from #63 |
First LWR subset upstream; broader evidence overlay still fork/independent work | Observable confidence, stable IDs and provenance | Sidecar contract, migration and scientific interpretation | Mark initial feature accepted, separate future sidecar design |
| B5: compositional MN/DM | upstream #22 | Independent CompositionalCounts.jl |
New modelling option | Joshua decided MGLM via RCall; pure-Julia route is not accepted upstream |
Retain pure-Julia package independent; only offer approved R route |
| B6: ALDEx2/ANCOM-BC | upstream #23 | Independent CompositionalDA.jl, incomplete methods/evidence |
Richer differential-abundance options | Joshua said no URL-pinned dependency; registration and validation required | Keep independent until criteria met |
| B7: mandatory multi-user security | fork security dossier #295 and core architecture #301 | Static review only; no authentication, authorisation, tenant isolation or audited ownership implementation verified | Required for Joshua's stated secure multi-user service; scientific integrity of user/study/job access | Product architecture / deployment gate, not optional feature; identity-provider, tenancy, migration, job/SSH isolation and testing decisions required | Top-tier critical path: confirm trusted SSO/session boundary and resource ownership with Joshua, implement in independently verifiable bounded increments; no secure multi-user deployment claim until evidence |
| B8: structural/security scan | tracking #300 | Tool capability/coverage reviewed; no runs yet | Better source-flow and provenance evidence beyond string comparison | Julia/R not covered by Pons grammar; false positives; scan runtime | Run scanners on both pinned snapshots; classify evidence |
| F1: occupancy | fork #205 following #18 and PR #196 | Partial ZINB/hurdle/low-level fitter; full repeated-detection workflow not done | Scientific option on fork | Identifiability and independent biological validation; not automatically offered upstream | Fork-only pending validation |
| F2: further exact/symbolic | fork #1, #3, #2, #4 | Descriptive layer exists; broader scientific review not closed | Future research | Exact inferential tests deferred, symbolic gated on deep validation and owner sign-off | Do not work around gates |
| G1: estate-specific governance | fork #294 | K9, Hypatia, DEED, CI, badges, proof infrastructure | Value to fork/estate | Joshua's requested stack and CI should not inherit this by default | Fork-only-by-design, not sync debt |
B3 port order — selective source extraction, not one giant cherry-pick
- D1 numeric core: numeric policy, exact summaries, precision boundaries and their negative/known-answer tests from Validated exact and high-precision descriptive statistics (#1) #199. Rebuild from current upstream base; isolate dependencies/format changes.
- D2 data/API boundary: only necessary API, persistence, conversion and independent-reference tests from Validated statistics API, persistence, and validation evidence (#1) #202.
- D3 optional integration: bounded method catalogue, Julia-owned UI and cancellation from Validated statistics layer: catalogue, cancellation, Julia UI and evidence (issue #1) #257 only after Joshua accepts the UI/API shape. Avoid an unrelated UI/framework change.
- D4 repair/test debt: applicable fixes from Fix the runtime faults the first complete statistics CI run exposed #259; explicitly exclude unrelated #260.
- D5 scientific release gate: pinned pipeline with real counts, independent biological checks, dataset rights/attributions, limitations and named scientific sign-off. A green build is not D5.
For each cherry-pick/adaptation: record source PR/commit → new upstream branch/commit → files → tests + precise before/after behaviour → maintainer decision → fork integration. Direct git cherry-pick is a technique, not a promise of compatibility. Do not transfer gitbot, K9, unrelated frontend, lockfile or test-framework changes in the same patch.
Branch-hygiene change requested
Current #294 measures branches against fork main and exempts named lanes. Extend the classification to target base:
upstream/*/ upstream contribution candidates → Joshuamain(current fetched upstream ref), not forkmain.work/*and normal fork PRs → forkmain.archive/*,governance-badges→ intentionally retained; measured and labelled but never candidates for upstream merge.- Require declared purpose + source issue + baseline SHA + disposition for intentional differences, rather than a naked branch-name exemption.
- Keep an exact upstream mirror reference without commits of its own; use separate topic branches for review.
- New UI summaries: Shared/accepted, Offerable, Fork-only by design, Awaiting Joshua, Needs validation, True drift/conflict. Never interpret ahead/behind alone as unfinished work.
- Preserve branch/PR refs; do not auto-delete work merely because its file paths overlap. Diff/semantic provenance and a human decision are required.
Joshua's contribution contract — mandatory
- Architectural decisions: discuss in an issue before code.
- One logical change per PR; branch from current upstream
main; behind_by=0; no merge commits. - Every commit Verified with the contributor's registered key; each commit self-contained and tested. Follow his fast-forward (no squash) merge process and rebase only as requested during review.
- Use pinned Julia/R/bun and existing Oxygen/React/RCall/DuckDB stack. No unwanted URL-pinned dependencies or licence/CITATION changes.
- Tests must run and pass; report unrun tests and separate statistical validation from test-green.
- Never place private vulnerability evidence, credentials or identifiable research data in public issues/PRs.
Working rhythm
Use this issue as the index; retain technical discussions in their existing issue, not a parallel thread in Joshua's repo. After each conversation with Joshua, update the block's owner decision and gate, then create one appropriately scoped upstream PR (if accepted). No CI job may edit this human decision table based on heuristics alone. The programme's goal is transparent transfer and intentional independence, not a zero ahead/behind count.
Activity
coderabbitai commentedon Oct 30, 2026
🔗 Related PRs
#191 - notes: how branches for upstream PRs get signed [merged]
#295 - docs(security): publish the security dossier, standards assessment, proposal and remedy ultraplan [closed]
#304 - security(B0): read-only native Linux capability preflight [open]
#224 - Resolve
#221against main's validated-statistics fixes [merged]#286 - fix(hygiene): lanes are not drift, and keep Dependabot PRs fresh [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
hyperpolymath commentedon Oct 30, 2026
Owner priority alignment / ultraplan steering note — 2026-10-30
The two programmes have different priority functions and must not be collapsed:
Joshua's priorities: (1) researcher dependability — scientifically valid, reliable, reproducible results and clear limitations; (2) industry-facing security, data processing and operational assurance. Upstream proposals must show scientific value, evidence, disruption cost, regression/operations risk, and rollback, not merely feature completeness.
Fork/research priorities: (1) mathematical accuracy/optimality and independent proofs; (2) refactoring/architecture, especially Marid, as a fork-side optional experiment because upstream CONTRIBUTING currently fixes Oxygen.jl/React/RCall/DuckDB and does not welcome replatforming; (3) long-term tree inference/exact NJ/sparse scaling (research charter #303); (4) identify the proposed FRED/fuzzy genomic method without assuming semantics; (5) general symbolic engine much later, after the existing scientific validation and explicit design gate.
Cross-cutting Mathesis: currently a repo-level census and catalogue (proof locations, prover, holes/axioms/policy candidates), not a completed theorem-level service or new owner of proofs. Extend it generically to index per-claim statements, assumptions, witnesses, tools/versions, independent checks, tests, and source provenance; original repositories remain authoritative. Agda theory runs with
agda --safe --without-k --cubical-compatible, followed by independent Rocq and Lean witness inspection, with Echidna future infrastructure. Do not embed Mathesis as a compulsory MetaManifold dependency.Near-term delivery order: upstream #66 robust remote job recovery; #64 full pinned/null-calibration evidence; #65 staged validated-descriptive port with independent biological sign-off; security issue #301 bounded to deployment model; #27 expanded evidence overlay only by agreement. Other research proceeds independently without silently altering Joshua's defaults.
Complexity truth condition: Reading a fully materialised n×n distance matrix costs Ω(n²); a claim of O(n) total time for arbitrary full-matrix exact NJ is incompatible with that input model. SwiftNJ (May 2026 preprint, DOI 10.64898/2026.05.28.728410) reports a constant-factor speedup, not a new asymptotic class. Compare RapidNJ, SwiftNJ, Dynamic NJ and Sparse NJ with precise equality/accuracy/complexity contracts. Veblen/ordinal work may help well-founded proof measures or representations, but should not be described as an established path to linear-time inference. FRED remains unidentified; no implementation gate is opened by a guessed acronym.
Coordination model: #302 is the stable cross-repo decision index. Work-specific technical issues and ChatGPT threads own implementation. Use a common handoff receipt:
block / scope / code owner / source SHA / upstream decision / claims / evidence / risks / acceptance / residual uncertainty / next action. After each owner discussion, update the decision here and move only the approved small change through verified-commit, fresh-upstream-branch, single-logical-change, pinned-CI/fast-forward contribution rules. Fork intentional differences are not 'desync failures'. Security vulnerabilities need private disclosure.This is a planning update, not new proof or benchmark evidence; no upstream file, package or architecture was changed.
hyperpolymath commentedon Oct 30, 2026
Correction to the master research register (30 Oct 2026). Owner's hypothesis is explicitly two-stage: (Q) exact/rational, certified RapidNJ search-space elimination as a proposed route from O(n³) to O(n²); (L) independent Veblen-based research as a proposed route from O(n²) to O(n). SwiftNJ is only a separate speed benchmark, not the claimed asymptotic breakthrough. I located relevant pre-existing Agda Q-bound/skip-warrant proofs in
hyperpolymath/epistemic-typesand separate ordinal proofs inhyperpolymath/ordinal-notation; neither currently proves the whole-algorithm asymptotic claim, and the rational rescaling, update invariants and Veblen→NJ algorithm connection remain unverified. A dense n×n input matrix already requires Ω(n²) reading time in n, so the second step must state an appropriately restricted/implicit input model or a different complexity variable. Full sources, open obligations and independent-review brief are now tracked in research issue #303. Preserve both hypotheses for examination; do not equate 'not established' with 'refuted'.hyperpolymath commentedon Oct 30, 2026
Constraint update for B0 and B7 (2026-10-30)
Joshua's target is a secure multi-user service delivered on ordinary Linux/SSH compute hosts with no required containers, VMs, hypervisor changes or root-controlled host reconfiguration. The previous container/Wolfi/VM-backed default-worker suggestion is not the upstream plan. Native application identity, per-study authorization, attributable persistent jobs, owner-aware SSH staging/cancellation and tenant-aware audit/quota controls are the critical path; see #301 updated architecture note.
This changes B0 #66: first define an identity-bound job record and security semantics for reattachment/cancellation; a content-based cache key alone does not authorize another researcher to resume/fetch the same computation. Implement reproducible resume tests before upstream PR.
BOINC is a pattern reference for durable work-unit state/checkpointing, not a VM requirement. Use native signed/pinned releases with SBOM/provenance and host capability checks instead of mandating Wolfi.
Joshua still needs to settle identity source, tenant sharing, trusted vs arbitrary user-supplied code, and what remote OS accounts, proxy/TLS termination and sandbox primitives host operators actually provide. Do not promise secure deployment where the necessary boundary is unavailable. The #64 PR and #65 science work remain separate and can continue without waiting on architecture.
hyperpolymath commentedon Oct 30, 2026
Secure native Linux implementation map (2026-10-30; detailed architecture in #301)
The current upstream single-user in-memory
JobQueue/shared SSH execution is not a tenant boundary. No VMs/containers/root/host-policy changes may be required. Work in explicit blocks:B0 read-only host and deployment-capability preflight (SELinux
getenforceand actual process label, AppArmor/LSM, Landlock, rlimits, cgroup delegation, SSH KEX, TLS/identity), B1 app HTTP controls and bounded inputs, B2 verified identities and default-deny owner/resource-action authorization, B3 durable owner-bound SSH jobs (#66), B4 tenant-scoped files/DB/execution, B5 audit/quotas and recovery, B6 native signed provenance/SBOM/releases, B7 opt-in host-admin SELinux profiles and post-quantum interop. B0/B1/B6 can proceed in parallel with scientific PR #67/#64/#65; B2 design precedes B3 implementation; B3/B4 need cross-tenant denial tests. No unconditional claim that SELinux is enabled or that kernel facilities supply isolation on remote hosts. Host-admin TLS/identity integration is a deployment prerequisite, not guaranteed by app code..well-known/security.txtand DNS records are optional when origin/domain is controlled; security headers are generally under app control.Crypto default: existing SHA-256 pins + signed provenance; prefer standard existing TLS/SSH libraries and OpenSSH PQ-hybrid KEX on both-end support rather than adding Ed448/BLAKE3/SHAKE/Kyber-1024 for their own sake; distinguish E2E encryption from server-readable computation. See full architecture decision note.
hyperpolymath commentedon Oct 30, 2026
Execution update — 2026-10-30
1c149285f86e62a93c828bf896e5fbe62115d3f4. Code review thread 4427476001 answered and marked resolved. Latest GitHub CI: Julia tests, Agda proofs and Codecov succeeded. Still blocked on unsigned commits: user must re-sign branch with their registered key and push (which retriggers checks); optionalCOMPARE_GLMGAMPOI=1path not yet independently run. No merge requested.Both tracks follow small reversible changes; identity/tenant isolation remains a mandatory release gate and is not delivered by B0 or #67.
hyperpolymath commentedon Oct 30, 2026
Active science-first progress — 30 October 2026
hyperpolymath commentedon Oct 30, 2026
Standing merge-readiness protocol — applied to MetaManifold (2026-10-30)
Canonical proposal: metadatastician/standards#1379. It states the evidence-based PR completion contract and the opportunity rule for shared tools (cicd-suite, proven-tests-and-benches, Hypatia, gitbot-fleet, squabbler, K9/DEED and proofs). This is durable tracking, not yet a signed, merged standards policy: attempts to commit the file were rejected by the standards repo's Verified signature gate, correctly. The temporary
policy/merge-readiness-evidence-gatebranch has no successful files.For this local-first upstream contribution path:
Current concrete demonstration: PR #306 remains draft. CodeRabbit's unequal-cluster test, SonarCloud's cognitive complexity/nested-with findings and Codex's four data-integrity findings have been addressed in scoped commits. The equivalent local Python implementation passed nine synthetic tests and reproduced all 12 original numeric comparator rows. Current-head GitHub CI, signed-commit gate and complete review are still outstanding. Upstream PR #67's unsigned commits are unchanged.
This protocol does not configure background monitoring or automatically run tools between conversations; we can adopt a future automation after its scope and permissions are established.
hyperpolymath commentedon Oct 30, 2026
Language and format-policy correction (30 October 2026)
Owner clarification: Python is prohibited for new work throughout Hyperpolymath. Preserve Joshua's existing React/TypeScript frontend as an explicitly transitional upstream allowance until Marid is viable; Julia/R remain MetaManifold's current scientific stack. Do not substitute Python under any circumstances to expedite a test or CI repair.
Formats: use canonical KYAML and I-JSON/JCS semantics, respecting
metadatastician/kyqas the planned four-format gateway (TOML, Dhall, KYAML and I-JSON/JCS) andhyperpolymath/ijson-jcsas the actual JSON canonicalizer. Do not invent another YAML or JSON processor for this PR. Existing R CSV benchmark files are not a new YAML/JSON format policy.Verified present status: as of latest checked
metadatastician/kyq/maincommit9a446862, its docs say all 11 AffineScript source files pass check, and a separate conformance harness exists, but compiled build, WASM, host I/O, actual CLI execution and benchmark/test maturity remain open. The README and Justfile explicitly refuse to claim implementedjust build/test/wasm; see kyq#11 and kyq#16. Revisit kyq in a separate thread/workstream, not on MetaManifold's science critical path.Actual work:
docstrings.enabled: truepolicy (draft; full CI/signatures pending).startup_failurefrom current PRs. Never treat its absence from regular check-run lists as success.Keep PRs draft until their actual mandatory checks and reviews are complete; do not weaken governance or signature rules.