Repository navigation
Commit 6883126
feat(arbitration): ECH-05 fidelity evidence and belief/epi/sound verdict levels (#430)
## Summary
Implements ECH-05 (drafted in `issues/echidna-05-fidelity-evidence.md`;
not yet filed). When backends agree, that doesn't show they proved the
same statement. So the arbiter now takes per-attempt **fidelity
evidence** and gives each fused verdict a **level**: `belief` < `epi` <
`sound`. A synonym collision (statements aligned only by name) can no
longer reach `sound`, however many backends agree. The numeric fusion is
unchanged; a regression fixture taken before the change shows this.
The ECH-05 plan asks for three stacked PRs. This session can push to one
branch only, so the three parts are three separate commits here, and
each can be reviewed on its own:
| Commit | Part |
|---|---|
| `6b6f54a` | **A**: `FidelityEvidence` on `ProverAttempt`; `level` /
`retracted` / `warranted_term` on `ArbitratedVerdict`; the policy;
regression fixture |
| `48e8034` | **B**: `src/abi/EchidnaABI/Arbitration.idr`, plus an
accept/reject fixture falsifier wired into `idris2-abi-ci.yml` |
| `33a3dc7` | **C**: `tests/arbitration_fidelity_test.rs`, ROADMAP stage
6 acceptance, backend-pair table, `issues/README.md` row and issue draft
|
## Changes
**A: input and output types**
(`src/rust/verification/result_arbiter.rs`, `src/rust/exchange/mod.rs`)
- `FidelityEvidence` = `KernelTranslation { format, checked_by,
term_hash }` | `DeclaredCorrespondence { consequence_tests, failures }`
| `None`. The default is `None`, and `ProverAttempt::new` keeps its
signature.
- `EpistemicLevel { Belief, Epi, Sound }` (ordered); `fidelity_level()`
is the policy:
- **`belief`** if the verdict needs review, rests on a retracted
backend, fused nothing, or **any** attempt has `None` evidence.
- **`sound`** only if every fused attempt is a `KernelTranslation` to
the same non-empty `term_hash`.
- **`epi`** if every fused attempt is a kernel translation or a
correspondence with **≥ 1** consequence test and 0 failures, and the
kernel hashes agree.
- **`belief`** otherwise, e.g. two kernel translations to different
terms.
- `ResultArbiter::retract(prover)` marks a backend unsound. Any verdict
that fuses one of its attempts drops to `belief`, and `retracted` lists
the backend.
- `exchange::ExchangeFormat` (one variant per bridge) and
`exchange::term_hash`: a BLAKE3 hash of the rendered term, separated by
format, with no canonicalisation. The failure mode is a *missed*
`sound`, never a false one.
- `tests/fixtures/arbitration_regression.json` was generated at
`29b18ea` *before* the change (16 scenarios × 4 policies).
`tests/arbitration_regression_test.rs` replays it field by field (floats
to 1e-12) and checks every verdict is at `belief`.
**B: Idris2 ABI** (`EchidnaABI.Arbitration`)
- `Level fused` is a closed sum indexed by the fused evidence. `Sound h`
requires an `All (KernelAt h) fused` proof, and each `KernelAt` holds
the hash-equality proof. So a collision, a correspondence or missing
evidence in `fused` cannot be typed `Sound`, whatever produced the value
(`collisionNotSound`, `correspondenceNotSound`, `noEvidenceNotSound`).
- The classifier `level` comes with `blockedIsBelief`,
`noEvidenceIsBelief`, `failedCorrespondenceIsBelief`,
`untestedCorrespondenceIsBelief`, `emptyFusedIsBelief`,
`blockingNeverRaises`, and evaluation controls (`alignedPairIsSound`,
`collisionPairIsBelief`, `mixedPairIsEpi`). It is total and needs only
`DecEq` on the hash type.
- epistemic-types `6a68de3` and residual-evidence-types `97244f5` are
cited by pin in the module header, not imported. The cited lemma names
were checked at those commits.
- Falsifier: `tests/abi/accept/ArbitrationAligned.idr` must type-check,
and `tests/abi/reject/ArbitrationCollision.idr` (identical apart from
one hash) must fail with `Mismatch between`.
- `scripts/abi-fixtures.sh` judges by output, because `idris2 --check`
exits 0 on a missing import.
- It checks each fixture in a fresh temp directory, because a stale
`build/` cache can silently skip the check (this bit me locally).
- It runs in `idris2-abi-ci.yml`, whose path filter now includes
`tests/abi/` and the script, and as `just abi-fixtures`.
**C: acceptance tests and docs**
- Synonym collision (Lean `ℕ` vs set.mm `NN`), under the portfolio,
Bayesian and Dempster–Shafer policies. Four cases all stay at `belief`
while `verified = true` / `CrossChecked`: lexical only; kernel
translations to different terms; a correspondence with a failed
consequence test; an untested correspondence.
- MiniF2F `mathd_numbertheory_3` positive control: the Lean and Isabelle
statements were checked against `openai/miniF2F@4e433ff` and translate
to one term, which reaches `sound`. Negative control: the same pair with
an off-by-one translation does not.
- Retraction: retracting Lean lowers the two verdicts that rested on it.
The Isabelle + HOL Light verdict stays `sound`, and the numbers don't
change.
- The exchange-layer terms are hand-written translation targets. These
tests check the arbiter *given* certificates, not that certificates can
be produced (that is stage 6).
- Docs:
- ROADMAP stage 6 acceptance (§2) and a "Cross-prover semantics" row
(§3).
- Gap-analysis table "Backend pairs that can share a term": every row is
DESIGNED in ECHIDNA, and external routes are named but not run.
- Short notes in the comparison doc and Guides.
- `issues/README.md` gains the ECH-05 row and lists its four labels
missing on this repo.
- CHANGELOG `[Unreleased]` entry.
## RSR Quality Checklist
### Required
- [x] Tests pass: `cargo test --lib` (1208 passed, 3 ignored) plus both
new integration tests; full `--workspace` run is left to CI (see
Testing)
- [x] Code is formatted (`cargo fmt --all -- --check`)
- [x] Linter is clean (`cargo clippy --locked --workspace --exclude
echidna-rest --all-targets -- -D warnings`; see Testing for the
exclusion)
- [x] No banned language patterns (Rust, Idris2, Bash only)
- [x] No `unsafe` blocks
- [x] No banned functions (the ABI dangerous-pattern grep over `src/abi`
and `tests/abi` is empty)
- [x] SPDX license headers present on all new/modified source files
- [x] No secrets, credentials, or `.env` files included
### As Applicable
- [ ] Repo deed `(status)` / `(maturity)` updated: not touched; nothing
ships to dispatch yet (every dispatched verdict is `belief`)
- [ ] Repo deed `(ecosystem)` updated: n/a
- [ ] Repo deed `(meta)` and/or `docs/decisions/` updated: policy
decisions are listed below for the owner
- [x] `docs/ROADMAP.adoc` updated (stage 6 acceptance + §3 row)
- [x] Documentation updated for user-facing changes (Guides, gap
analysis, comparison doc)
- [ ] `TOPOLOGY.md` updated: n/a
- [x] `CHANGELOG` updated
- [x] New dependencies reviewed: none added (`blake3` already a
dependency)
- [x] ABI changes validated: new module has no foreign declarations;
`ffi/zig` untouched
## Testing
```
$ cargo test --lib -- result_arbiter term_hash → 23 passed
$ cargo test --test arbitration_regression_test → 1 passed, 1 ignored (the generator)
$ cargo test --test arbitration_fidelity_test → 4 passed
$ cargo test --lib → 1208 passed; 0 failed; 3 ignored
$ cd src/abi && idris2 --build echidnaabi.ipkg → 20/20 modules, exit 0 (Idris2 0.8.0)
$ bash scripts/abi-fixtures.sh
ok accept: tests/abi/accept/ArbitrationAligned.idr
ok reject: tests/abi/reject/ArbitrationCollision.idr
Mismatch between: True and False.
```
Negative controls (applied, observed, then reverted):
- A 1e-6 relative perturbation of the Bayesian conflict metric fails the
regression test at `unanimous_proven_three / bayesian(0.5):
verdict.conflict`.
- Removing the term-hash equality check in `fidelity_level` fails
`synonym_collision_cannot_reach_sound` and
`minif2f_mistranslated_pair_does_not_reach_sound`.
- Making the reject fixture aligned makes the falsifier fail
("type-checked"); breaking its import makes it fail ("non-type reason").
Not run locally:
- `echidna-rest` builds `utoipa-swagger-ui`, whose build script
downloads Swagger UI from GitHub, and the sandbox proxy blocks that. It
is the only crate excluded from local clippy; CI builds the full
workspace.
- The full `cargo test --workspace --all-targets` was not run here; that
is left to CI.
## Decisions for the owner
1. **Stacked PRs**: one branch with three commits, because of this
session's single-branch constraint. Happy to split into A → B → C
stacked PRs.
2. **Policy choices beyond the spec's text**:
- A correspondence needs ≥ 1 consequence test (otherwise zero failures
is vacuous).
- Mismatched kernel hashes among fused attempts give `belief`, not
`epi`.
- A verdict with `needs_review` (which covers inconsistent premises) is
`belief`.
- An empty `term_hash` is not a certificate.
3. **Stage 6 "pair X"**: proposed HOL4 ↔ HOL Light via OpenTheory first,
then Lean 4 ↔ Rocq via Dedukti.
4. **Labels** `area:arbitration`, `area:exchange`, `epistemic-types`,
`residual-evidence-types` don't exist on this repo yet. They need
creating before ECH-05 is filed.
5. **Open questions** from the issue are unchanged: whether
Dempster–Shafer stays limited to same-encoding provers, and who runs
consequence testing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_019a95kk88TZ1iegsBK3kox3
---
_Generated by [Claude
Code](https://claude.ai/code/session_019a95kk88TZ1iegsBK3kox3)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 29b18ea commit 6883126
20 files changed
Lines changed: 3697 additions & 8 deletions
File tree
- .github/workflows
- docs
- design
- wikis
- issues
- scripts
- src
- abi
- EchidnaABI
- rust
- exchange
- verification
- tests
- abi
- accept
- reject
- fixtures
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
| 15 | + | |
14 | 16 | | |
15 | 17 | | |
16 | 18 | | |
| |||
44 | 46 | | |
45 | 47 | | |
46 | 48 | | |
47 | | - | |
| 49 | + | |
48 | 50 | | |
49 | 51 | | |
50 | 52 | | |
| |||
93 | 95 | | |
94 | 96 | | |
95 | 97 | | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
96 | 105 | | |
97 | 106 | | |
98 | 107 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
11 | 29 | | |
12 | 30 | | |
13 | 31 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
311 | 311 | | |
312 | 312 | | |
313 | 313 | | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
314 | 319 | | |
315 | 320 | | |
316 | 321 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
108 | 108 | | |
109 | 109 | | |
110 | 110 | | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
111 | 117 | | |
112 | 118 | | |
113 | 119 | | |
| |||
198 | 204 | | |
199 | 205 | | |
200 | 206 | | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
201 | 219 | | |
202 | 220 | | |
203 | 221 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
98 | 104 | | |
99 | 105 | | |
100 | 106 | | |
| |||
216 | 222 | | |
217 | 223 | | |
218 | 224 | | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
219 | 269 | | |
220 | 270 | | |
221 | 271 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
194 | 194 | | |
195 | 195 | | |
196 | 196 | | |
197 | | - | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
198 | 202 | | |
199 | 203 | | |
200 | 204 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
| 108 | + | |
| 109 | + | |
108 | 110 | | |
109 | 111 | | |
110 | 112 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| |||
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
| |||
28 | 30 | | |
29 | 31 | | |
30 | 32 | | |
| 33 | + | |
31 | 34 | | |
32 | 35 | | |
33 | 36 | | |
| |||
46 | 49 | | |
47 | 50 | | |
48 | 51 | | |
| 52 | + | |
49 | 53 | | |
50 | | - | |
| 54 | + | |
0 commit comments