feat(ledger): structure audited failure outcomes

[gfyrag]

What changed

Persist each definitive audited business failure as one typed FailureDetails variant. Its oneof arm determines the stable code, and its failure_reason annotation determines the coarse reason. Audit, frozen idempotency outcomes, and skipped-order logs no longer store separate copies of either identifier. The FSM checks that a producer's reason agrees with its variant before persistence; the checker validates replayed details against the hash-bound audit entry.

Keep technical and retryable failures outside the definitive business-failure audit path. Map exported Numscript v0.2.0 interpreter, VM, and compiler failures to Ledger-owned typed variants. Structured facts have a fixed presence-checked shape; diagnostic prose stays out of persisted outcomes and public error metadata.

The request's skippable_reasons whitelist still uses coarse reasons. HTTP, JSON, events, and gRPC ErrorInfo continue to expose reason, code, and facts derived from the typed variant. The service protocol is revision 39. Ledger v3 is unreleased, so this changed persisted and gRPC shape requires fresh development state and matching revision-39 clients and servers.

Validation

• Protobuf regenerated with the pinned Nix toolchain; repository invariants, build, lint, and root-module go test -race ./... passed.
• Business and cluster E2E race suites passed locally.
• Schemathesis exercised 64 endpoints twice: 63 passed, none failed a contract check, and POST /v3/{ledgerName}/indexes had a connection reset on both runs. The same endpoint had produced this local network error before this commit; its server log is available in /tmp/schemathesis-server.txt.
• The exact local review loop approved commit 8dd7b3988 with no actionable findings.

Review focus

• Oneof arm → reason/code mapping and producer agreement.
• Audit hashing, idempotency replay, skipped-order verification, and protocol revision 39.
• Business failures versus retryable technical failures.

[NumaryBot]

docs/ops/deployment.md:761

🟠 [major] Exclude revision 29 from the no-wipe error-identity upgrade guarantee

An operator upgrading to revision 29 is still told that no persisted value encoding changes and existing entries remain verifiable after restarting all nodes. The revision-29 protocol documentation explicitly replaces persisted message/context and message/metadata fields with code/facts. Retaining pre-29 failures under this guarantee leaves incompatible state; a coordinated restart does not convert it. The intentional pre-release replacement is acceptable, but deployment instructions must not promise preservation.

Suggestion: Explicitly exclude revision 29 from the no-wipe guarantee and document its supported state replacement/recovery procedure. Keep the coordinated-restart-only guidance scoped to error-identity changes that preserve the persisted representation.

[gfyrag]

Fixed the high-review and CI findings in f70f500. The order UTF-8 scan now walks protobuf messages iteratively, so valid nested requests are no longer rejected with request is nested too deeply (the first error in the failed Tests-Model job). The restore E2E now expects PRELOAD_UNAVAILABLE to leave no audit entry, matching its retryable pre-audit path. Dirty was caused by formatter changes in validate_order.go; the file is formatted in this commit. Local validation: admission tests, agent-check, full tagged lint (0 issues), and the 180-second three-node model run pass. The targeted S3 restore suite compiles locally; its execution requires Docker/Testcontainers and is left for CI.

[gfyrag]

Separated internal proposal rejections from audited business failures in 7deae00 and 08e181c:

• Cluster policy validation/revision conflicts and preload coverage misses now return without an audit entry or frozen idempotency result. Their producers own the UncommittedFailure disposition.
• Removed their variants from persisted FailureFacts and advanced the shared protocol revision to 36.
• Preserved the FSM HLC on every unaudited return, including idempotency replay and preload unavailability, so incremental restore reconstructs the same durable clock.
• Added FSM retry tests and a real checkpoint + incremental restore parity test. Targeted state/backup suites and agent-check pass; the local review loop approved the exact pushed head with no findings.

The full local PR gate passed Go race tests and business/cluster E2E tests. Its Schemathesis step reported a connection reset on POST /v3/{ledgerName}/indexes; a standalone rerun reproduced the same network error without a server panic. CI is now running on this head, including Schemathesis.

[paul-nicolas]

R2 — [P2][blocking] Update the Numscript ErrorInfo contract and example
Location: docs/technical/architecture/subsystems/api/grpc-api.md:665–667, 698–699
Evidence: The table promises diagnostic details/detail fields, and the example reads Metadata["details"]. This PR removes those fields: parse failures have empty facts, while compile/execution failures expose variant-specific facts.
Impact: Clients following the example display an empty script-error explanation.
Resolution: Update the table and example to use the current facts or rendered status message.

[gfyrag]

Addressed both points in 4fbab16. The audit-chain docs, protobuf comment, and related audit manifests now distinguish definitive business outcomes from UncommittedFailure: the latter produces no audit entry, items, logs, or frozen idempotency result, and restores the prior HLC. The gRPC API docs and example now reflect that Numscript parse errors have no details metadata and that compile/execution facts depend on the typed variant. Local scripts/agent-check and the review loop passed.