Increment 9: durable shipment exceptions and acknowledgement

IMPLEMENTATION_PLAN.md

@@ -1,6 +1,6 @@
 # Veritras implementation plan
 
-Originally prepared October 17, 2026 from [PROJECT_REVIEW.md](PROJECT_REVIEW.md). **Status updated October 19, 2026 for increment 7, following analytics release PR #22 (`ff88bcc`).** Integration recovery checks, typed contracts, protected-main merge controls and production admission gates are implemented. Final CI/deployment evidence is recorded in the increment 7 pull request; see [release controls](RELEASE_CONTROLS.md) and the [release/recovery runbook](RELEASE_RUNBOOK.md).
+Originally prepared October 17, 2026 from [PROJECT_REVIEW.md](PROJECT_REVIEW.md). **Status updated October 19, 2026 for increment 9.** Integration recovery checks, typed contracts, protected-main merge controls and production admission gates are implemented. Final CI/deployment evidence is recorded in the increment 7 pull request; see [release controls](RELEASE_CONTROLS.md) and the [release/recovery runbook](RELEASE_RUNBOOK.md).
 
 ## Status at a glance
 
@@ -21,7 +21,7 @@ Priority meanings: **P0 — release blocker** must be resolved before enabling p
 | Analytics and exports (increment 6) | **Done** | **P1 addressed** | Canonical shipment cohorts/lifecycle, optional locked SLA, real route/sample metrics, UTC filters and safe matching exports; see [runbook](ANALYTICS.md) |
 | Integration and release controls (increment 7) | **Done** | **P1 addressed** | Real write-failure/restart, process-loss, reorg and backup/restore rehearsals; typed detail/verification; protected main and exact-commit production admission; see [runbook](RELEASE_RUNBOOK.md) |
 | QR verification, shipment detail and guided demo (increment 8) | **Done** | **P2 addressed** | Direct public links and SVG QR labels, canonical paginated timeline, shipment detail with contract-checked transitions, isolated guided demo, mobile wallet/keyboard access and route metadata; see [guide](PRODUCT_JOURNEY.md) |
-| Shipment exceptions (increment 9) | **Pending** | P2 | Depends on trustworthy lifecycle timestamps, SLA definitions and synchronization |
+| Shipment exceptions (increment 9) | **Done** | **P2 addressed** | Configurable owner policy, canonical arrival/checkpoint/custody rules, durable deduplicated occurrences, acknowledgement/history, synchronized evaluation and public mobile list; see [guide](SHIPMENT_EXCEPTIONS.md) |
 | Performance and operations (increment 10) | **Pending** | P2 | Existing lazy routes/Lighthouse checks provide a baseline; measured optimization and operational completion remain |
 | Business pilot integrations | **Deferred** | Later | Organizations, evidence files, imports, standards integrations and repeated journeys |
 
@@ -43,7 +43,7 @@ Priority meanings: **P0 — release blocker** must be resolved before enabling p
 
 Historical increment 4 validation: **236 backend tests, 192 frontend tests, 23 Playwright regressions and 40 contract tests passed**, plus builds, generated-type checks, Lighthouse and real local integration/browser journeys. Hosted Sepolia registration → checkpoints → custody → delivery passed; a same-code testing redeploy preserved event counts. The backend matrix covers SQLite and PostgreSQL, with two intentional skips for PostgreSQL-only checks in the SQLite variant. A real local chain fork/recovery and hosted same-code redeploy also passed.
 
-Current released schema is `c62a904df318`; increments 7–8 add no migration. Historical durable-indexing rollout is tracked by [PR #18](https://github.com/Oionos/Veritras/pull/18); analytics was released in [PR #22](https://github.com/Oionos/Veritras/pull/22). Production indexing remains explicitly disabled. Increment 7 adds real PostgreSQL failure/process-loss and restart checks, local reorganization recovery, a locked backup/restore and read-only reconciliation rehearsal, and production gate rejection regressions. Test counts and final commit/deployment evidence are recorded in its pull request. Testing uses a dedicated Neon project/database (`Veritras Testing` / `veritras_preview`), a separate Render API and distinct Sepolia contracts. Setup and resource details: [TESTING.md](TESTING.md), [HOSTED_TESTING.md](HOSTED_TESTING.md), [AUTHENTICATION.md](AUTHENTICATION.md).
+Current schema is `e93a107bd624`; increments 7–8 add no migration. Historical durable-indexing rollout is tracked by [PR #18](https://github.com/Oionos/Veritras/pull/18); analytics was released in [PR #22](https://github.com/Oionos/Veritras/pull/22). Production indexing remains explicitly disabled. Increment 7 adds real PostgreSQL failure/process-loss and restart checks, local reorganization recovery, a locked backup/restore and read-only reconciliation rehearsal, and production gate rejection regressions. Test counts and final commit/deployment evidence are recorded in its pull request. Testing uses a dedicated Neon project/database (`Veritras Testing` / `veritras_preview`), a separate Render API and distinct Sepolia contracts. Setup and resource details: [TESTING.md](TESTING.md), [HOSTED_TESTING.md](HOSTED_TESTING.md), [AUTHENTICATION.md](AUTHENTICATION.md).
 
 ## Goal and completion criteria
 
@@ -285,7 +285,7 @@ Completed in increment 7: actual PostgreSQL checkpoint failure after a metadata
 
 ### PR 9: Shipment exception list
 
-**Status: pending. Priority: P2, after trustworthy metrics and synchronization.** Existing anomaly calculations do not satisfy the planned deduplicated, acknowledged exception lifecycle.
+**Status: implemented (increment 9). Priority: P2 addressed.** Configurable rules, canonical/synchronization gates, durable episodes and acknowledgement/history are delivered. See [SHIPMENT_EXCEPTIONS.md](SHIPMENT_EXCEPTIONS.md) for definitions, defaults and recovery behavior.
 
 - Add configurable expected arrival and checkpoint-interval/dwell thresholds with explicit meanings.
 - Show overdue shipments, stale checkpoint activity, and unresolved custody handoffs in an in-app exception list.

README.md

@@ -91,3 +91,7 @@ Veritras was built as a **portfolio project** to demonstrate full-stack blockcha
 <p align="center">
   <a href="https://veritras.vercel.app">Try the Live Demo</a>
 </p>
+
+## Shipment exceptions
+
+See [the exception guide](SHIPMENT_EXCEPTIONS.md) for configurable rules, synchronization gates, acknowledgement/history, owner permissions and additive schema revision `e93a107bd624`. Open `/exceptions` or a shipment detail page. Production evaluation remains paused while indexing is disabled.

RELEASE_RUNBOOK.md

@@ -32,7 +32,7 @@ Use `TESTING_COMPOSE_PROJECT=<existing-isolated-project>` for an existing named
 
 The full journey uses actual wallet signatures, transactions, PostgreSQL and API calls: draft registration, custody A to B, checkpoints, delivery, verification and matching analytics/exports. Unauthorized contract writes and duplicate registration must revert. The reorganization drill replaces a mined branch and checks restored metadata, history, custody and shipment projections. The write-failure drill makes PostgreSQL reject a checkpoint in a batch that also updates metadata; journal, cursor and projections must all remain at the preceding commit across an API restart. Removing the fault replays the batch once; a second restart cannot duplicate it. A separate PostgreSQL regression kills a real writer process with uncommitted SQL and verifies lock release and complete rollback.
 
-The backup rehearsal validates the local chain/database before DDL, holds the indexer writer lock, uses `pg_dump`, creates a fresh clone and restores with `pg_restore --exit-on-error`. It checks schema head and runs the identity/history reconciliation functions read-only against the actual chain. Before/after fingerprints cover products, shipments, checkpoints, transfers, canonical/noncanonical journal, anchors, cursor, baselines, aliases, registration attempts and schema version. Source, backup and restored fingerprints must match exactly. The source is never reset. An error releases the writer lock on connection close. Private backups and named clones are retained for inspection; remove only a specifically identified disposable rehearsal artifact when finished, after checking it is not needed.
+The backup rehearsal validates the local chain/database before DDL, holds the indexer writer lock, uses `pg_dump`, creates a fresh clone and restores with `pg_restore --exit-on-error`. It checks schema head and runs the identity/history reconciliation functions read-only against the actual chain. Before/after fingerprints cover products, shipments, checkpoints, transfers, canonical/noncanonical journal, anchors, cursor, baselines, aliases, registration attempts, exception policies/occurrences/evaluation state and schema version. Source, backup and restored fingerprints must match exactly. The source is never reset. An error releases the writer lock on connection close. Private backups and named clones are retained for inspection; remove only a specifically identified disposable rehearsal artifact when finished, after checking it is not needed.
 
 ## Separate production indexing/backfill gate
 
@@ -58,3 +58,7 @@ This section is a future operator procedure, not authorization to enable product
 | Database restore required | Pause writers and writes; preserve a fresh snapshot of the failing state. Restore the rehearsed backup into a new isolated target first, verify schema and fingerprints, reconcile confirmed chain events since the backup, and review counts/identity before a separately approved connection switch. Restoring a database does not undo mined transactions. |
 
 Application rollback retains the additive schema, aliases, registration attempts and event journal. Never run a destructive schema downgrade or delete canonical history as a deployment rollback. Recheck API readiness and frontend configuration/attestation after recovery. A healthy old frontend can remain on a compatible API while a corrective backend release is prepared.
+
+## Increment 9 schema compatibility
+
+Shipment exceptions add revision `e93a107bd624`; see [SHIPMENT_EXCEPTIONS.md](SHIPMENT_EXCEPTIONS.md). Rehearse the generated additive SQL on an isolated copy and retain a pre-migration checkpoint before explicitly applying it. Old running handlers remain compatible, but an older backend artifact cannot restart against a schema head it does not recognize. Use a compatible corrective release and retain all exception history. Production indexing stays disabled, so no production exception inference or backfill is enabled by this release.

SHIPMENT_EXCEPTIONS.md

@@ -0,0 +1,46 @@
+# Shipment exceptions (increment 9)
+
+`/exceptions` lists durable operational observations and resolved history. Each entry links to the shipment's canonical evidence. Shipment detail shows open observations and the owner's exception policy. These observations describe missing **confirmed indexed records**; they do not prove a physical shipment delay or authenticate an item.
+
+## Rules and defaults
+
+| Rule | Trigger | Default | Clears when |
+| --- | --- | --- | --- |
+| Planned arrival overdue | A planned arrival has passed after canonical departure, without canonical delivery. The plan must have been recorded on or before that departure. | Enabled; uses the shipment's optional UTC arrival plan | Delivery, valid plan/eligibility change, disabled rule or withdrawn departure evidence |
+| Checkpoint activity overdue | No checkpoint follows the latest checkpoint during a journey with an indexed departure | 24 hours | A new checkpoint resets the interval, delivery, disabled rule or evidence withdrawal |
+| Checkpoint follow-up overdue | No checkpoint follows the latest **At Checkpoint** record | 24 hours | A new checkpoint, delivery, disabled rule or evidence withdrawal |
+| Receiving handler follow-up overdue | The latest custody transfer has no subsequent checkpoint by its receiving handler | 6 hours | A checkpoint by that recipient, a newer transfer, delivery, disabled rule or evidence withdrawal |
+
+A threshold is reached at `now >= anchor timestamp + threshold`. At Checkpoint uses its own interval instead of also raising the general inactivity rule. It measures time since a **record**, not physical dwell. Custody transfers already take effect on-chain; follow-up does not represent a pending acceptance transaction. A checkpoint by a different wallet does not satisfy receiving-handler follow-up. Transaction/log order decides which event follows another, including equal timestamps.
+
+Owners can set intervals from 0.25 to 8760 hours or leave an interval blank to disable its rule. Arrival planning remains on the shipment record and is locked before indexed departure. Unknown historical plans are not invented. Drafts, unreconciled scope, superseded products and products without a canonical registration cannot establish exceptions.
+
+## Lifecycle and permissions
+
+- An occurrence persists with its threshold-crossing time, first/last observation, anchor event and occurrence identities, policy revision and evidence summary.
+- A partial unique index permits one open occurrence per scope, shipment and rule. Repeated evaluations update that occurrence; they do not duplicate it.
+- Material changes to a still-overdue rule/anchor resolve the prior occurrence as `condition_replaced` and open a new one. An unrelated policy change preserves acknowledgements. Newly cleared conditions use `condition_cleared`; removed canonical anchors use `evidence_withdrawn`.
+- A previously resolved condition that recurs opens a new occurrence. Its acknowledgement starts empty. Closed history and earlier acknowledgements remain intact across restarts and reorg recovery.
+- Only an authenticated, established product owner in the current contract scope may change policy or acknowledge. Current chain custody alone does not grant these API permissions. Legacy manufacturer text grants no rights.
+- Policy PUTs require `expected_revision`; stale saves return 409. Identical saves are idempotent. Acknowledgement POSTs are idempotent and preserve the first actor, timestamp and note. Resolved occurrences return 409.
+- Notes are public, plain text and limited to 500 characters. Acknowledgement records review; it does not resolve or suppress a condition. No email, webhook, external delivery or retry queue is configured.
+
+## Synchronization and operation
+
+The API process runs a lightweight evaluator approximately every 60 seconds when indexing is enabled. No additional paid service is required. Disabled indexing starts no evaluator. Render sleep naturally pauses assessment; the page reports the stored assessment time.
+
+Evaluation requires cached indexer status `current`, readiness, zero lag and zero finality gap. Inside the indexer's shared transaction writer lock, the evaluator checks the configured scope's durable cursor and block anchor against cached progress. It rechecks indexer health before committing. Journal reorgs, concurrent evaluators, policy writes and acknowledgements cannot interleave their write transactions. Failure rolls back the entire assessment. No RPC is performed by exception rules or public reads.
+
+Disabled, stale, catching-up, changed or unavailable indexing opens/resolves nothing. Invalid/future event timestamps and missing recipient evidence retain that shipment's previous observations; the response reports an uncertain-shipment count. A regressed clock cannot overwrite a later assessment. The UI labels paused evaluation and retains prior observations without treating missing data as proof of delay. Assessments older than three minutes are labelled stale even if indexer health is current. Public GET requests never evaluate or mutate lifecycle state.
+
+The list polls every 30 seconds, supports open/unacknowledged/acknowledged/resolved/all filters, shipment filtering and bounded pagination (maximum 100 via API). No count is inferred from client time. Existing anomaly analytics remain a separate statistical view.
+
+**Production indexing remains disabled.** Its exception list therefore shows paused evaluation. Production reconciliation/backfill and enablement retain their separate reviewed gate in [RELEASE_RUNBOOK.md](RELEASE_RUNBOOK.md).
+
+## Schema and release
+
+Additive Alembic revision `e93a107bd624` follows `c62a904df318`. It adds `exception_policies`, `shipment_exceptions` and `exception_evaluations` with constraints and indexes. There is no business-data backfill or canonical-journal modification. Rehearse on isolated copies, retain a pre-migration checkpoint, compare existing table fingerprints, and apply the same generated SQL transaction to the explicitly selected target before starting the new backend.
+
+October 19 rehearsals preserved fingerprints for products, shipments, checkpoints, transfers, canonical/orphan journal, block anchors, cursor, bindings, aliases and registration attempts on copies of both hosted databases. Retained `before-shipment-exceptions-20261019` checkpoint branches expire October 26; disposable rehearsal branches expire October 20. No database credentials were changed. Keep the additive schema on rollback. An old API artifact with an earlier schema head cannot pass its strict startup admission against this revision; use a corrective compatible release instead of a destructive downgrade.
+
+Recovery/backup fingerprints now include all three exception tables. Local PostgreSQL regressions use disposable schemas, preserving the shared regression database. Controlled-clock tests cover threshold crossings, custody source order, deduplication, material policy changes, acknowledgements, delivery, orphan recovery, uncertain evidence, paused/changed indexing, interrupted commits, restart persistence and concurrent writers. Browser coverage includes paused states, public mobile history, shipment links, retry and owner policy persistence/permission denial in the actual local wallet journey.

backend/README.md

@@ -147,3 +147,7 @@ MIT
 ## Indexer startup and health
 
 Use `/health/live` for hosting HTTP checks and `/health/ready` for dependency/indexer readiness. Initialization, confirmed-target lag, cursor progress and head freshness are reported separately. See [INDEXER_HEALTH.md](../INDEXER_HEALTH.md) for configuration, capped retries, safe diagnostics and bounded release probes. Production indexing remains disabled pending its separate enablement gate.
+
+## Shipment exceptions
+
+See [the exception guide](../SHIPMENT_EXCEPTIONS.md) for configurable rules, synchronization gates, acknowledgement/history, owner permissions and additive schema revision `e93a107bd624`. Open `/exceptions` or a shipment detail page. Production evaluation remains paused while indexing is disabled.