Increment 10: measured performance and operations

.github/workflows/operations-monitor.yml

@@ -0,0 +1,39 @@
+name: Operations Monitor
+on:
+  schedule:
+    # Four checks a day; avoid keeping free services continuously awake.
+    - cron: '15 */6 * * *'
+  workflow_dispatch:
+permissions:
+  contents: read
+concurrency:
+  group: operations-monitor
+  cancel-in-progress: false
+jobs:
+  observe:
+    name: Observe ${{ matrix.environment }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 7
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - environment: production
+            api: https://chainlens-4qvy.onrender.com
+            indexer: disabled
+          - environment: testing
+            api: https://veritras-api-testing.onrender.com
+            indexer: enabled
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Check sustained readiness, API errors and SSE heartbeat
+        env:
+          API: ${{ matrix.api }}
+          ENVIRONMENT: ${{ matrix.environment }}
+          INDEXER: ${{ matrix.indexer }}
+        run: python backend/monitor_operations.py --url "$API" --environment "$ENVIRONMENT" --expect-indexer "$INDEXER"

IMPLEMENTATION_PLAN.md

@@ -1,6 +1,6 @@
 # Veritras implementation plan
 
-Originally prepared October 17, 2026 from [PROJECT_REVIEW.md](PROJECT_REVIEW.md). **Status updated October 19, 2026 for increment 9.** Integration recovery checks, typed contracts, protected-main merge controls and production admission gates are implemented. Final CI/deployment evidence is recorded in the increment 7 pull request; see [release controls](RELEASE_CONTROLS.md) and the [release/recovery runbook](RELEASE_RUNBOOK.md).
+Originally prepared October 17, 2026 from [PROJECT_REVIEW.md](PROJECT_REVIEW.md). **Status updated October 20, 2026 for increment 10.** Integration recovery checks, typed contracts, protected-main merge controls and production admission gates are implemented. Final CI/deployment evidence is recorded in the increment 7 pull request; see [release controls](RELEASE_CONTROLS.md) and the [release/recovery runbook](RELEASE_RUNBOOK.md).
 
 ## Status at a glance
 
@@ -22,7 +22,7 @@ Priority meanings: **P0 — release blocker** must be resolved before enabling p
 | Integration and release controls (increment 7) | **Done** | **P1 addressed** | Real write-failure/restart, process-loss, reorg and backup/restore rehearsals; typed detail/verification; protected main and exact-commit production admission; see [runbook](RELEASE_RUNBOOK.md) |
 | QR verification, shipment detail and guided demo (increment 8) | **Done** | **P2 addressed** | Direct public links and SVG QR labels, canonical paginated timeline, shipment detail with contract-checked transitions, isolated guided demo, mobile wallet/keyboard access and route metadata; see [guide](PRODUCT_JOURNEY.md) |
 | Shipment exceptions (increment 9) | **Done** | **P2 addressed** | Configurable owner policy, canonical arrival/checkpoint/custody rules, durable deduplicated occurrences, acknowledgement/history, synchronized evaluation and public mobile list; see [guide](SHIPMENT_EXCEPTIONS.md) |
-| Performance and operations (increment 10) | **Pending** | P2 | Existing lazy routes/Lighthouse checks provide a baseline; measured optimization and operational completion remain |
+| Performance and operations (increment 10) | **Done** | **P2 addressed** | Measured mobile LCP 11.33→5.37 s, deferred SDK/chart loading, private diagnostics, stream recovery, scheduled operations checks and migration-window safeguards; validation and release evidence in [PR #28](https://github.com/Oionos/Veritras/pull/28) |
 | Business pilot integrations | **Deferred** | Later | Organizations, evidence files, imports, standards integrations and repeated journeys |
 
 ### Urgent safeguards and next actions
@@ -33,6 +33,8 @@ Priority meanings: **P0 — release blocker** must be resolved before enabling p
 4. **Maintain release enforcement.** Main reports protected and the owner confirmed all eight required checks, up-to-date PRs and no bypasses/force pushes/deletions. Production Docker and frontend builds require successful exact-main-commit CI; API startup validates its built admission proof without GitHub access; frontend also waits for a compatible ready API. API rule-detail readback remains forbidden to this integration. See [RELEASE_CONTROLS.md](RELEASE_CONTROLS.md).
 5. **Preserve testing isolation and free-tier limits.** Use the stable `testing` preview for full-stack checks. Production and testing Render services share the workspace's 750 free instance hours/month; avoid continuous keep-alive pings. See [HOSTED_TESTING.md](HOSTED_TESTING.md).
 
+6. **Maintain operations and collect field evidence.** Increment 10 adds private diagnostics, mobile budgets and four scheduled checks per day. Field percentiles and personal failure-notification delivery remain unvalidated; see [operations](OPERATIONS.md) and [measurements](PERFORMANCE.md). Use the coordinated migration preflight for future DDL; never migrate production while exact-main CI is pending.
+
 ### Completed releases and evidence
 
 | Release | Main commit | Completed work |
@@ -85,9 +87,9 @@ The numbers below are **plan increment IDs**, not actual GitHub pull request num
 | 5 | **Done / P1 addressed** | `codex/registration-lifecycle-evidence` | Maintain durable transaction recovery, checked evidence and metadata history; release tracked in PR #21 | 2–4 |
 | 6 | **Done / P1 addressed** | `codex/truthful-analytics-and-exports` | Maintain canonical cohorts, SLA exclusions and safe matching exports; release tracked in PR #22 | 3–5 delivered |
 | 7 | **Done / P1 addressed** | `codex/integration-release-controls` | Maintain real recovery drills, protected-main checks and production admission; see [runbook](RELEASE_RUNBOOK.md) | 2–6 delivered |
-| 8 | Pending / P2 | `feat/qr-verification-and-demo` | QR links, unified shipment detail, mobile and guided demo | 5 and 7 |
-| 9 | Pending / P2 | `feat/shipment-exceptions` | Expected arrival, overdue and dwell exception list | 6–8 |
-| 10 | Pending / P2 | `perf/frontend-and-operations` | Measured loading improvements and operational cleanup | Stable reliable-demo milestone |
+| 8 | **Done / P2 addressed** | Released in #25 | Maintain QR links, shipment detail, mobile and guided demo; see [guide](PRODUCT_JOURNEY.md) | 5 and 7 |
+| 9 | **Done / P2 addressed** | Released in #27 | Maintain canonical exception rules, history and acknowledgement; see [guide](SHIPMENT_EXCEPTIONS.md) | 6–8 |
+| 10 | **Done / P2 addressed** | `codex/performance-and-operations`, PR #28 | Maintain loading budgets, private diagnostics, periodic health checks and recovery procedures; field comparison awaits real traffic | 5–9 delivered |
 
 Tests and migration checks belong in each implementation PR. PR 7 assembles the complete flow and release gates; it is not the first testing work.
 
@@ -296,19 +298,19 @@ Completed in increment 7: actual PostgreSQL checkpoint failure after a metadata
 
 ### PR 10: Performance and operations
 
-**Status: pending completion. Priority: P2.** Existing lazy routes, error reporting configuration and Lighthouse CI are a baseline. Measured optimization, environment-aware observability and readiness alerts remain. Increment 7 supplies the local backup/restore rehearsal and recovery runbook; hosted restore acceptance belongs to the separate production enablement gate.
+**Status: done for implementation and acceptance scope. Priority: P2 addressed.** Three controlled mobile samples per build show median LCP 11.33→5.37 seconds, transfer −60%, initial gzip JavaScript −69%, with unchanged CLS. See [performance evidence](PERFORMANCE.md) and [operations](OPERATIONS.md). Private error/route telemetry, replay disabled, API/SSE counters, heartbeat recovery, scheduled cold-start-aware monitoring and coordinated migration preflight are implemented. Valid production field percentiles require real traffic; no historical field comparison is claimed. Local backup/restore is rehearsed; hosted production restore acceptance remains in the separate enablement gate.
 
 - Measure mobile loading and interaction before changing bundles; investigate wallet/provider startup, chart imports, and unnecessary SDK loading.
 - Defer optional wallet/chart code where measurements justify it. Compare before/after Lighthouse and field metrics using the same scenario.
 - Configure release/environment labels for error tracking, appropriate replay masking, and monitoring for indexer lag, API failures, and event-stream health.
 - Document backup/restore and single-indexer ownership. Split the indexer into a worker only if uptime/scaling requirements justify the additional service.
 
-**Acceptance:** performance improvements have measured evidence; error reporting excludes sensitive input; indexing and API degradation are actionable and observable.
+**Acceptance completed:** three-run before/after mobile evidence; regression-tested privacy allowlists and replay disabled; real wallet/reconnect and recovery/restore journeys; hosted testing mobile/bundle/privacy/SSE checks; sustained testing readiness with zero lag and an actual heartbeat. Exact-main CI, provider IDs and final production checks are retained in [PR #28](https://github.com/Oionos/Veritras/pull/28). Ongoing field comparison requires real production samples; native notification delivery and hosted production restore/backfill are explicit separate limits.
 
 ## Migration, backfill, and release procedure
 
 1. Inspect production deployment/schema state and record a restorable database checkpoint. Rehearse on an isolated database copy first.
-2. Apply additive migrations and deploy compatible backend code. Keep any old cursor/projection writer paused while reconciling identities or rebuilding event projections.
+2. After protected merge, exact-main CI and candidate image admission, confirm the prior API is already awake/ready and run the short-lived migration-window preflight. Coordinate additive DDL with immediate compatible runtime rollout; never apply production DDL while CI is pending. See [operations](OPERATIONS.md#coordinated-migration-window). Keep any old cursor/projection writer paused while reconciling identities or rebuilding event projections.
 3. Run reconciliation in dry-run mode, inspect conflicts, then apply validated changes. Add uniqueness/non-null constraints only after conflicting data is resolved.
 4. Load/replay confirmed canonical logs in bounded ranges from each configured deployment block, validating receipt identities and projection counts. Persist progress transactionally.
 5. Confirm readiness, indexing progress, permissions, verification results, and analytics. Deploy the compatible frontend against the intended environment.
@@ -330,7 +332,7 @@ After the reliable demo is complete and a specific user workflow has been valida
 
 ## Next implementation session
 
-The next product increment is **9: shipment exception list**. Increments 5–8 are implemented, including truthful analytics, enforced release controls, public QR verification and the guided/mobile journey. **Production indexing enablement validation remains important and pending:** review the deployment anchor, rehearse identity/history reconciliation and hosted restore, then obtain explicit enablement before backfill. Keep that gate separate from ordinary application releases.
+Increment **10: performance and operations** is delivered; final protected-main and provider evidence is recorded in [PR #28](https://github.com/Oionos/Veritras/pull/28). Increments 5–9 are delivered. After increment 10, prioritize field observation and a defined business pilot; there is no automatically authorized production backfill. **Production indexing enablement validation remains important and pending:** review the deployment anchor, rehearse identity/history reconciliation and hosted restore, then obtain explicit enablement before backfill. Keep that gate separate from ordinary application releases.
 
 Preserve analytics definitions and sample/exclusion counts during subsequent work. Keep production indexing enablement and any change to release-gate configuration as explicit reviewed changes.