Phase 6e fuzz budget

Cargo.toml

@@ -75,18 +75,33 @@ overflow-checks = true
 debug = true
 overflow-checks = true
 
-# `rustreexo` 0.6.0 as published generates *invalid inclusion proofs* for any
-# leaf whose sibling has been deleted, which breaks the transparent half of this
-# project outright — a bridge node cannot serve proofs across blocks
-# (`docs/design.md` D10). This pins a fork of v0.6.0 carrying the one-line fix
-# from upstream PR mit-dci/rustreexo#152, verified here against a control:
-# 1,848 of 4,671 proofs fail on stock, 0 on the fork (D25).
+# This pin now carries **three** upstream fixes, not one. Two of them are ours.
+#
+# **D10 — invalid proofs after deletion.** `rustreexo` 0.6.0 as published
+# generates invalid inclusion proofs for any leaf whose sibling has been
+# deleted, which breaks the transparent half of this project outright: a bridge
+# node cannot serve proofs across blocks. The fork carries the one-line fix from
+# upstream PR mit-dci/rustreexo#152, verified against a control — 1,848 of 4,671
+# proofs fail on stock, 0 on the fork (D25).
+#
+# **D33 — `MemForest::deserialize` is not total on malformed input.** Two
+# separate defects, both reachable from a snapshot file a peer supplied:
+# an unrecognised node-type byte panics, and deeply nested input recurses until
+# the stack overflows. The second is the worse one and was found while fixing
+# the first: a stack overflow **aborts rather than unwinds**, so the
+# `catch_unwind` in `UtxoForest::from_bytes` never contained it.
+#
+# Moving the pin from `dc368cc` to `8931ab8` is what lets `forest_decode` and
+# `snapshot_decode` be fuzzed at all — both died within seconds before, and
+# `scripts/fuzz_72h.sh` excluded them for that reason.
 #
 # A `[patch]` rather than a git dependency on the crate itself, so the version
 # requirement above stays visible and **removing these three lines restores the
 # published crate** the moment upstream merges. When that happens, delete this
 # block, drop the `allow-git` entry in `deny.toml`, and expect
-# `tests/upstream_rustreexo.rs` to keep passing — it asserts the fixed
-# behaviour, so it is the guard on this pin.
+# `tests/upstream_rustreexo.rs` to keep passing — it asserts the fixed D10
+# behaviour, so it is the guard on this pin. The D33 fixes are guarded by
+# `fuzz/artifacts/{forest,snapshot}_decode/crash-*`, replayed as regression
+# seeds.
 [patch.crates-io]
-rustreexo = { git = "https://github.com/USCMig/rustreexo", rev = "dc368ccc6988f85c0754ef0b1dff9258f2789123" }
+rustreexo = { git = "https://github.com/USCMig/rustreexo", rev = "8931ab8b6a01c75bfb784d128b9c20df83217aee" }

PLAN.md

@@ -40,9 +40,10 @@ infrastructure, `fix/<topic>` for defects.
 | 4b | Sparse wire format, bridge service, served IBD | `phase-4a-bundle` | **complete** — Phase 4 DoD met over a real socket; sparse paths cut wallet proofs 53.2% and bundle overhead 170.5% to 152.6%. Not gRPC ([D27](docs/design.md)) |
 | 5a | Headline measurement: nullifier-check cost vs gap length | `phase-4a-bundle` | **complete** — the claim holds decisively for spend-status queries (317x to 31,705x at a year's gap) and barely at all for full sync (<=14.7% of bytes, 0% of trial decryption) |
 | 5b | Shadow-mode CSN against Zebra, remaining Phase 5 axes | `phase-5b-shadow` | **complete** — storage measured at last (31.7 GiB vs 693 B, ~49M:1), latency p50/p99 over 60k sandblasting blocks and 1,021 at tip, and 500 blocks shadowed at the live tip with zero divergences. Reverses the narrow-or-keep direction: **keep the transparent forest** (below). Shadow is external to Zebra ([D30](docs/design.md)); reorg recovery is a queue for a compact node ([D31](docs/design.md)) |
-| 6 | Fuzzing, DoS analysis, privacy review | `phase-6-adversarial` | **in progress** — **72 h fuzz run complete 2026-08-25: 206 billion executions, zero crashes across all 5 targets**, so the DoD's fuzzing half is met for those five; two excluded on an upstream panic ([D33](docs/design.md)) — the fork fix is written and verified against all five crash artifacts but **not pushed**, and writing it turned up a second, worse bug in the same function: unbounded recursion overflows the stack on ~4 MB of input, which aborts rather than unwinds, so the `catch_unwind` D33 relied on never covered it. Bridge hardened: slowloris was total denial from one idle socket, now closed ([D34](docs/design.md)). **Privacy review complete and negative** — Phase 5a's headline query cannot be made privately ([D35](docs/design.md)). External review dropped (below). **Fuzz budgeting was wrong in both directions**: three targets gained zero edges in the whole run and `utxo_proof_decode` found its last at execution 101, while `bundle_decode` was **still finding edges 25 minutes before the clock ran out**. `scripts/fuzz_72h.sh` now budgets per target (7 d + `-fork=8` for `bundle_decode`, 24 h for the rest) and runs `scripts/fuzz_saturation.py` itself ([D36](docs/design.md)) |
+| 6 | Fuzzing, DoS analysis, privacy review | `phase-6-adversarial` | **in progress** — **72 h fuzz run complete 2026-08-25: 206 billion executions, zero crashes across all 5 targets**, so the DoD's fuzzing half is met for those five; two excluded on an upstream panic ([D33](docs/design.md)) — the fork fix is written, verified against all five crash artifacts, **pushed 2026-09-07 and pinned 2026-09-08 — both targets now fuzz** ([D33](docs/design.md)), and writing it turned up a second, worse bug in the same function: unbounded recursion overflows the stack on ~4 MB of input, which aborts rather than unwinds, so the `catch_unwind` D33 relied on never covered it. Bridge hardened: slowloris was total denial from one idle socket, now closed ([D34](docs/design.md)). **Privacy review complete and negative** — Phase 5a's headline query cannot be made privately ([D35](docs/design.md)). External review dropped (below). **Fuzz budgeting was wrong in both directions**: three targets gained zero edges in the whole run and `utxo_proof_decode` found its last at execution 101, while `bundle_decode` was **still finding edges 25 minutes before the clock ran out**. `scripts/fuzz_72h.sh` now budgets per target and runs `scripts/fuzz_saturation.py` itself ([D36](docs/design.md)). **Seven-target run complete 2026-09-17: 103 billion executions, zero crashes** ([D45](docs/design.md)). It had the budgets backwards. `bundle_decode` hit its ceiling in 2.1 h of its 7 d. `snapshot_decode` was still discovering at 18.1 h of its 24 h. The three zero-gain targets are finished, not under-seeded: every reachable region of their decoders is covered. **The DoD's fuzzing half is still open for 3 of 7 targets**: `forest_decode`, `snapshot_decode`, and `wire_request_decode` since wire v2 have never had 72 h. The next run (7 d on 8 forks for `snapshot_decode`, 72 h for the rest) closes it |
 | 6b | Prefix cohorts: can the headline query be made private? | `phase-6b-prefix-cohort` | **step 1 complete** — the one mitigation [D35](docs/design.md) left standing is built and measured. At Orchard's real 50.4M nullifiers a 16-bit prefix gives a **768-member anonymity set for 449.7 KB**, 498× a single proof and still 98× cheaper than scanning a year. Path dedup is worth 35.2%, not the ~20% predicted. **Found that the published sparse proof size was measured on a tree 768× too small**, overstating every Phase 5a ratio by 1.452×; all three eras re-measured ([D37](docs/design.md)). **Step 2 complete**: an epoch-sorted snapshot gives the chosen target of **12,288 members for 384.9 KB**, 14.2x better, at 32 bytes a member with an O(log n) proof that stays ~25 siblings whatever the cohort size ([D38](docs/design.md)). It turned out **not** to touch the Phase 3 format at all -- the sorted tree is derived, additive, bridge-side state and the IMT is untouched. **Step 3 complete**: no per-pool split is needed -- every pool reaches the target with a cohort that beats shipping its whole set, Ironwood's by 4x, and cost tracks the anonymity asked for rather than the pool asked about ([D39](docs/design.md)). **Step 4 measured the intersection question and the answer is bad**: at the operating point **100% of simulated wallets are uniquely fingerprinted** by their bucket set -- per-note anonymity 12,302, per-wallet anonymity 1. Bandwidth cannot fix it (a wallet anonymity set needs 734 MB a session, 193x), decoy buckets are stripped by intersection in 2-3 sessions however many are added, and the only mitigation that works is **one non-colluding bridge per note** ([D40](docs/design.md)). **Step 5 corrected that**: the mitigation is one *unlinkable session* per bucket, not one operator per note, which Tor already meets -- but it holds only above a user base and never in a burst. Ten queries in one second are recovered in 1-3 days at every population; the required spread is `10 x buckets x 86400 / (wallets x notes)` seconds, and at 1,000 users no setting is safe (crowd per bucket: 2) ([D41](docs/design.md)). This is linkability, not disclosure -- the note values stay hidden. **Re-scoped 2026-09-01 ([D44](docs/design.md)): the privacy purpose this stage was built for is dominated.** `valargroup/spendability-pir` and its Ironwood fork already do private spendability with SimplePIR over a bucketed hash table -- structurally the same partition-and-scan, but the bucket index is hidden by cryptography rather than by crowd size, so D40's per-wallet anonymity of 1 simply does not arise. It costs **1.78x** the cohort's bandwidth, not the orders of magnitude [D35](docs/design.md) assumed when it dismissed PIR unpriced. What the cohort still owns is narrow and stated as such: the beyond-window tail (their coverage is ~289 days; Phase 5a's headline was measured at a year), authenticity (they serve answers, we serve proofs), and non-membership as a primitive a validator can use. The diurnal-traffic refinement D41 flags is **dropped** -- it refines a mechanism that is no longer the answer |
 | 6c | Serving cohorts: the bridge side, and the epoch policy | `phase-6c-cohort-service` | **complete** — until now no wallet could make a private query: `zutreexo-bridge` had three methods and the private-looking one named the nullifier outright, and the word "cohort" appeared nowhere in the crate. Two methods added (`PrefixCohort`, `EpochManifest`, `WIRE_VERSION` 1→2); the request carries the *bucket*, never the value, asserted on the encoded bytes rather than argued from the type. A prefix under the anonymity floor is **refused, not widened**, and `max_bits` is published per epoch so a client never learns the floor by being refused ([D42](docs/design.md)). A cohort is 385 KB, so counting requests stopped being a bandwidth control: a peer inside the 600/min default pulls 231 MB/min, now bounded by a per-peer byte budget. **The epoch measurement changed a default**: `keep` was 2 on plausible reasoning, priced at **6 GB against one extra round trip per client per day**, and is now 1 — which is also the better privacy answer, since several live epochs give the bridge a second coordinate to group on. Interval stays 1,000: the bridge's duty cycle is 0.03% and irrelevant, and what bounds it is the client's delta scan against a 1,330-block break-even ([D43](docs/design.md)). 13 mutants, 13 killed; four of them found real gaps, and one found that the socket test helper **hung on failure instead of failing**. **Re-scoped 2026-09-01 ([D44](docs/design.md)) along with 6b**: the service is correct, tested and measured, and it is not the best available answer for a light client. It stands as the *authenticated, full-history* path -- the one thing deployed PIR does not offer -- and no longer as "the private query". Two things worth stealing from their design are recorded in D44 |
+| 6d | Unblock the last two fuzz targets | `phase-6d-fuzz-unblock` | **complete** — the D33 fork fix is pushed and the pin moved to it, so `forest_decode` and `snapshot_decode` are back in `scripts/fuzz_72h.sh` after four phases out. Corpora tell the story of the exclusion: `forest_decode` had **8** inputs and reached **514** in one 10M-run smoke test, `snapshot_decode` 100 to 361, no crashes in either. All five committed crash artifacts replay clean. **The fix walked into its own trap first**: `fuzz/` is outside the workspace and carries a second `[patch.crates-io]`, so bumping only the workspace pin left the fuzzer on the old rev and every artifact still reproduced — against a bug already fixed, which reads as a falsified fix rather than a stale pin. `crates/zutreexo-testkit/tests/pins_agree.rs` now guards both revs and rejects abbreviated hashes |
 | 7 | ZIP draft — gated on 5 and 6 | — | **gated shut, and the attempt to open it failed** — CLAUDE.md requires "the measured benefit is real **and** the privacy review is clean". [D35](docs/design.md) is negative. Phase 6b/6c was the attempt to change that and it did not: [D40](docs/design.md) found per-wallet anonymity of 1, [D41](docs/design.md) recovered it only above a user base the network does not have, and [D44](docs/design.md) found that deployed PIR does the job properly. The gate holds, for a better-understood reason than when it was first shut |
 
 ## Known gaps, carried deliberately

crates/zutreexo-testkit/tests/pins_agree.rs

@@ -0,0 +1,93 @@
+//! The `rustreexo` fork pin exists in two files, and they must not drift.
+//!
+//! `fuzz/` is deliberately outside the workspace — `cargo fuzz` builds with
+//! `-Zsanitizer=address` and its own profile, and pulling it in would put those
+//! flags on every ordinary build. The cost of that separation is a **second
+//! copy of the `[patch.crates-io]` pin**, and a second copy is a second thing
+//! to forget.
+//!
+//! It was forgotten. Bumping the workspace pin to the D33 fix
+//! (`MemForest::deserialize` no longer panics on a bad node-type byte, no
+//! longer overflows the stack on nested input) left `fuzz/Cargo.toml` at the
+//! old `dc368cc`. All five committed crash artifacts still reproduced, against
+//! a bug that had already been fixed, and the replay read as "the fix does not
+//! work" rather than "you are testing the wrong code". The give-away was the
+//! build path in the panic — `.../rustreexo-.../dc368cc/src/mem_forest/mod.rs`.
+//!
+//! This test is cheap and it is the guard on that. It compares the two revs
+//! textually rather than resolving them, because the failure mode is a stale
+//! literal, not a bad resolution.
+
+#![allow(
+    clippy::expect_used,
+    clippy::unwrap_used,
+    clippy::panic,
+    clippy::indexing_slicing
+)]
+
+use std::path::{Path, PathBuf};
+
+/// Repository root, from this crate's manifest directory.
+fn repo_root() -> PathBuf {
+    Path::new(env!("CARGO_MANIFEST_DIR"))
+        .join("..")
+        .join("..")
+        .canonicalize()
+        .expect("repo root")
+}
+
+/// The `rev = "..."` on the `rustreexo` patch line in one manifest.
+///
+/// Parsed with a string search rather than a TOML crate: this test guards a
+/// dependency pin, so it should not itself acquire a dependency that could
+/// need pinning.
+fn rustreexo_rev(manifest: &Path) -> String {
+    let text = std::fs::read_to_string(manifest)
+        .unwrap_or_else(|e| panic!("read {}: {e}", manifest.display()));
+    let line = text
+        .lines()
+        .find(|l| l.trim_start().starts_with("rustreexo = { git"))
+        .unwrap_or_else(|| panic!("no rustreexo patch line in {}", manifest.display()));
+    let at = line
+        .find("rev = \"")
+        .unwrap_or_else(|| panic!("no rev in {line}"));
+    let rest = &line[at + 7..];
+    let end = rest
+        .find('"')
+        .unwrap_or_else(|| panic!("unterminated rev in {line}"));
+    rest[..end].to_owned()
+}
+
+#[test]
+fn the_workspace_and_fuzz_crate_pin_the_same_rustreexo() {
+    let root = repo_root();
+    let workspace = rustreexo_rev(&root.join("Cargo.toml"));
+    let fuzz = rustreexo_rev(&root.join("fuzz").join("Cargo.toml"));
+    assert_eq!(
+        workspace, fuzz,
+        "\nthe fork pin has drifted between the two manifests:\n  \
+         Cargo.toml       {workspace}\n  fuzz/Cargo.toml  {fuzz}\n\n\
+         The fuzz targets would be exercising different code from the one the \
+         workspace ships, and a crash artifact replayed against the wrong rev \
+         reads as a failed fix rather than a stale pin."
+    );
+}
+
+#[test]
+fn both_pins_are_full_length_commit_hashes() {
+    // An abbreviated rev resolves today and can become ambiguous later, and it
+    // makes the two literals harder to compare by eye in review. Cargo accepts
+    // a short rev, so nothing else would complain.
+    for manifest in ["Cargo.toml", "fuzz/Cargo.toml"] {
+        let rev = rustreexo_rev(&repo_root().join(manifest));
+        assert_eq!(
+            rev.len(),
+            40,
+            "{manifest} pins an abbreviated rev {rev:?}; use the full 40-character hash"
+        );
+        assert!(
+            rev.chars().all(|c| c.is_ascii_hexdigit()),
+            "{manifest} rev {rev:?} is not hex"
+        );
+    }
+}