Validated statistics layer: catalogue, cancellation, Julia UI and evidence (issue #1)

.github/workflows/ci.yml

@@ -252,22 +252,7 @@ jobs:
       # R.Version in renv.lock; CRAN's Ubuntu repository retains older revisions, so
       # this resolves rather than merely requesting the newest.
       - name: Set up R
-        run: |
-          wget --https-only -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc \
-            | sudo gpg --dearmor -o /usr/share/keyrings/r-project.gpg
-          echo "deb [signed-by=/usr/share/keyrings/r-project.gpg] https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/" \
-            | sudo tee /etc/apt/sources.list.d/r-project.list
-          sudo apt-get update -qq
-          sudo apt-get install -y \
-            r-base-core="$R_APT_VERSION" \
-            r-base-dev="$R_APT_VERSION" \
-            r-base-html="$R_APT_VERSION" \
-            r-recommended="$R_APT_VERSION" \
-            r-base="$R_APT_VERSION"
-          Rscript -e 'cat(R.version.string, "\n")'
-
-      - name: Install R system dependencies
-        run: sudo apt-get install -y libcurl4-openssl-dev libssl-dev libxml2-dev libfontconfig1-dev
+        run: bash scripts/ci/install-r-runtime.sh full
 
       # The R packages are restored from renv.lock, the same pin a developer restores
       # from, rather than resolved against whatever the repositories serve that day.
@@ -417,7 +402,7 @@ jobs:
         shell: bash
         run: |
           set +e
-          julia --project=. -t2 --compiled-modules=no test/runtests.jl validated_statistics numeric_policy exact_summaries high_precision derived_analyses 2>&1 | tee /tmp/validated-statistics-tests.log
+          julia --project=. -t2 --compiled-modules=no test/runtests.jl validated_statistics numeric_policy exact_summaries high_precision derived_analyses method_catalogue statistics_boundaries statistics_adversarial statistics_properties statistics_ui 2>&1 | tee /tmp/validated-statistics-tests.log
           # Preserve both exits immediately: a successful Julia run must not
           # hide a failed log write (e.g. a full runner filesystem).
           statuses=("${PIPESTATUS[@]}")
@@ -517,6 +502,105 @@ jobs:
           slug: JoshuaJewell/MetaManifold-WebUI
           fail_ci_if_error: false
 
+  # Issue #1 evidence that cannot live in the Julia suite: the two standards
+  # scripts run from the pinned standards commit, a seeded simulation study with
+  # pre-declared tolerances, and a mutation campaign that proves the focused
+  # tests can fail. The tools are pinned by commit and by SHA-256; a missing
+  # tool is exit 2 in both standards scripts, so this job cannot pass by skipping.
+  standards:
+    name: Statistics standards and evidence
+    needs: gate
+    if: needs.gate.outputs.relevant == 'true'
+    runs-on: ubuntu-24.04
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262  # v4.4.0
+
+      - name: Set up Julia
+        run: bash scripts/install-ci-julia.sh
+
+      # The versions, the HTTPS restriction and the exact-commit fetches live in
+      # the script (one reviewed place), not interpolated into this workflow.
+      - name: Install pinned standards tools
+        run: bash scripts/ci/fetch-statistics-tools.sh
+
+      # MetaManifold includes RCall at module load, and RCall's build step needs
+      # an R installation even though this job runs no R code. The version comes
+      # from the committed pin file, not from whatever apt calls current.
+      - name: Install the pinned R runtime
+        run: |
+          R_APT_VERSION="$(yq -r '.runtimes.r.apt_version' config/defaults/tool_versions.yml)"
+          R_APT_VERSION="$R_APT_VERSION" bash scripts/ci/install-r-runtime.sh core
+
+      # The evidence scripts are ordinary project code: they load JSON3 and the
+      # statistics modules, so the environment must be instantiated after R.
+      - name: Instantiate Julia project
+        run: julia --project=. -e 'import Pkg; Pkg.instantiate()'
+
+
+
+      # YAML-POLICY Y-3 / D307. --check exits 1 on drift and 2 when yq is
+      # missing or the file does not parse, so neither outcome can read as a pass.
+      - name: KYAML formatting gate
+        run: bash "$RUNNER_TEMP/standards/scripts/kyaml-format.sh" --check config/defaults/validated_statistics.kyaml
+
+      # JSON-POLICY J-1 / D306. Enforcement is scoped to the documents this
+      # branch is responsible for; the repository-wide pass is report-only
+      # because the policy calls unconverted files migration backlog, not a
+      # violation of a new change. Both outputs are retained.
+      - name: I-JSON/JCS gate
+        env:
+          IJSON_JCS: ${{ runner.temp }}/ijson-jcs/target/release/ijson-jcs
+        run: |
+          set -euo pipefail
+          bash "$RUNNER_TEMP/standards/scripts/check-ijson-jcs.sh" --enforce \
+            frontend/public/config.json
+          # test/fixtures/jcs_oracle.jsonl is a vector file: its inputs are
+          # deliberately NOT canonical and its expected outputs are the
+          # canonical form, so enforcing on it would be a category error.
+          bash "$RUNNER_TEMP/standards/scripts/check-ijson-jcs.sh" . \
+            | tee "$RUNNER_TEMP/ijson-jcs-report.txt"
+
+      # A failing study must say why in the checks API, not only in a log the
+      # reviewer cannot open from every client. Annotations are the same channel
+      # the Julia suite uses for failing assertions.
+      # run-with-log.sh keeps the command's own exit status and annotates a
+      # failure with the tail of its log. The workflow carries no copy of that
+      # logic, so the exit-preserving pipeline stays a single tested construct.
+      - name: Seeded simulation study
+        env:
+          RUN_LOG_TITLE: Seeded simulation study
+        run: |
+          bash scripts/ci/run-with-log.sh "$RUNNER_TEMP/simulation.log" -- \
+            julia --project=. -t2 scripts/statistics/simulation_study.jl \
+              --output "$RUNNER_TEMP/statistics-simulation-report.json"
+
+      # Six mutants are defined; four keep the campaign inside the job timeout.
+      # The script restores every file and re-verifies its digest, so a failed
+      # run cannot leave a planted defect behind.
+      - name: Mutation campaign
+        if: always()
+        env:
+          RUN_LOG_TITLE: Mutation campaign
+        run: |
+          bash scripts/ci/run-with-log.sh "$RUNNER_TEMP/mutation.log" -- \
+            julia --project=. -t2 scripts/statistics/mutation_check.jl --max-mutants 4 \
+              --output "$RUNNER_TEMP/statistics-mutation-report.json" \
+              --keep-logs "$RUNNER_TEMP/mutation-logs"
+
+      - name: Retain statistics evidence
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02  # v4.6.2
+        with:
+          name: statistics-standards
+          path: |
+            ${{ runner.temp }}/statistics-simulation-report.json
+            ${{ runner.temp }}/statistics-mutation-report.json
+            ${{ runner.temp }}/ijson-jcs-report.txt
+            ${{ runner.temp }}/mutation-logs
+          if-no-files-found: warn
+          retention-days: 30
+
   # Machine-checked proofs of the Benjamini-Hochberg adjustment and the TSS/RLE
   # scaling algebra (proofs/agda, proofs/agda/README.md).
   # Agda publishes no 2.6.4.3 binary, and agda-stdlib 2.1 requires Agda 2.6.4.x,

VALIDATED_STATISTICS.md

@@ -88,6 +88,64 @@ The matrix orientation is **features by samples**. For example:
 to `false` for direct input. If enabled, the persisted source includes the exact
 input matrix as strings, making the direct derived record self-contained.
 
+### Optional deadline and cancellation
+
+An optional `"time_limit_seconds"` field (a whole number of seconds greater
+than 0 and at most 3600) arms a cooperative deadline for that request. Decimal
+and exponent tokens are refused by the same raw-number rule as counts, so the
+JSON boundary accepts whole seconds only. Cancellation is checked
+between cells, samples, DuckDB rows and streamed chunks, so a bounded request
+stops rather than pinning a worker. A cancelled or expired request returns:
+
+```json
+{"error": "statistics_cancelled", "message": "…; no partial result was returned"}
+```
+
+with HTTP `408`. The deadline is operational only: it never changes a stored
+value, is never part of the numeric policy fingerprint, and is never written to
+a derived record. A request that times out publishes nothing. Missing or `null`
+means no deadline; zero, negative, non-finite, boolean or non-numeric values are
+`400`. The same contract applies to the run/DuckDB endpoint.
+
+## Method catalogue
+
+`GET /api/v1/statistics/catalogue`
+
+Returns the approved, bounded catalogue as canonical JCS. It contains the
+approval record (`approved_by`, `approved_on`, `approval_reference`), the
+`methods` array with a status of `supported`, `unsupported`, `deferred` or
+`out_of_scope`, the enforced `limits`, and response-only fields
+`catalogue_sha256`, `supported_method_ids`, `method_count` and `file`.
+
+A method that is not `supported` cannot produce a result: `parametric-model-fits`,
+`nonparametric-resampling` and `exact-statistical-tests` are published so that
+their absence is explicit, and `symbolic-mathematics` is out of scope. The
+machine-readable source of truth is `config/defaults/validated_statistics.kyaml`
+(KYAML); the loader refuses a catalogue whose status is not `approved` or whose
+limits differ from the constants the code enforces. See
+[docs/statistics/method-catalogue-v1.md](docs/statistics/method-catalogue-v1.md).
+
+## Julia-rendered statistics page
+
+`GET /statistics` and `POST /statistics` are a server-rendered, JavaScript-free
+surface for the same layer. The form accepts the numeric policy, an optional
+time limit and a features × samples count table, and renders:
+
+- the approved catalogue block with its fingerprint and approval reference;
+- per-sample totals, status and exact proportions (or MPFR metrics);
+- group summaries in `exact_counts` mode;
+- warnings, diagnostics and a provenance block with the numeric-policy
+  fingerprint and `input_sha256`;
+- explicit failure states with the documented status and "no result" wording;
+- an optional immutable derived record when `persist` is checked.
+
+The page is served with `Content-Security-Policy: default-src 'none';
+style-src 'unsafe-inline'; form-action 'self'; base-uri 'none';
+frame-ancestors 'none'` and contains no script. It is not a replacement for the
+React application and introduces no application TypeScript. The JSON endpoints
+remain the machine-readable contract. Per-cell tables are capped at 20,000
+cells for display; the summary and provenance stay authoritative.
+
 ### JSON number boundary
 
 - A non-negative integer JSON token is accepted only through `2^53 - 1`, the
@@ -125,8 +183,9 @@ The implementation intentionally does **not** call the legacy
 2. Sample columns discovered by the existing schema rules must have DuckDB type
    `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`, or `HUGEINT`.
    `DOUBLE`/`FLOAT` storage is rejected, not rounded or relabelled as exact.
-3. SQL performs row selection, `GROUP BY`, and deterministic feature ordering
-   only. Missing and negative count cells are rejected with feature/sample
+3. SQL performs row selection and column projection only. Rows are streamed
+   and grouped in Julia, so no SQL `SUM` and no arbitrary per-group value is
+   involved. Missing and negative count cells are rejected with feature/sample
    coordinates.
 4. Integer cells are converted to `BigInt`; taxon and sample totals are then
    summed in Julia. This avoids both `Float64` rounding and fixed-width SQL sum
@@ -215,7 +274,9 @@ Focused tests:
 
 ```bash
 julia --project=. -t2 test/runtests.jl numeric_policy exact_summaries \
-  high_precision derived_analyses validated_statistics
+  high_precision derived_analyses validated_statistics \
+  method_catalogue statistics_boundaries statistics_adversarial \
+  statistics_properties statistics_ui
 ```
 
 Full suite (with the repository's integration and server gates):
@@ -237,6 +298,24 @@ python3 test/references/check_high_precision_references.py --check \
   test/unit/test_high_precision.jl
 ```
 
+Seeded simulation study with pre-declared tolerances, and the mutation
+campaign that proves the tests can fail:
+
+```bash
+julia --project=. -t2 scripts/statistics/simulation_study.jl \
+  --output simulation-report.json
+julia --project=. -t2 scripts/statistics/mutation_check.jl --output mutation-report.json
+```
+
+A representative biological count matrix is validated against the existing
+`DiversityMetrics` implementation (a separate Float64 code path) with a
+pre-declared relative tolerance:
+
+```bash
+julia --project=. -t2 scripts/statistics/biological_validation.jl \
+  --input counts.csv --source "study/run or DOI" --output biological-report.json
+```
+
 Benchmark and retained CI artifact:
 
 ```bash
@@ -252,13 +331,20 @@ of equal timing on another platform. CI retains the JSON as
 
 ## Remaining acceptance work
 
-See also the [method-by-method validation matrix](VALIDATED_STATISTICS_VALIDATION.md).
-
-This increment does not self-certify issue #1. Before closure, the project still
-needs the applicable representative biological comparisons and method-by-method
-acceptance table, UI/workflow validation for the new modes, retained review
-artifacts, independent statistical/scientific review, and explicit release
-approval. Expensive-operation cancellation and broader recovery/soak testing
-also remain validation work; bounded inputs and isolated precision/persistence
-are implemented, but are not a claim of full production certification. Exact
-statistical tests and symbolic mathematics remain out of scope.
+See the acceptance-criterion map in
+[docs/statistics/validation-evidence.md](docs/statistics/validation-evidence.md)
+and the [method-by-method validation matrix](VALIDATED_STATISTICS_VALIDATION.md).
+
+Implemented and evidenced here: the approved catalogue, numeric/cross-language
+and storage contracts, known-answer and independent references, adversarial and
+property-based tests, a seeded simulation study with pre-declared tolerances, a
+mutation campaign, cancellation and resource limits, the Julia-rendered UI
+surface with provenance and failure states, the regression suite for existing
+behaviour, the benchmark, and the CI jobs that rerun the standards gates.
+
+Not self-certified by this increment: the representative biological comparison
+(runner supplied by `scripts/statistics/biological_validation.jl`; it requires a
+real, licensed count matrix and a provenance string, and refuses to run without
+one) and independent statistical/scientific review with explicit release
+approval. Those are owner and reviewer gates. Exact statistical tests are
+deferred and symbolic mathematics remains out of scope; issue #2 stays blocked.