Skip to content

fix(rpc): queue Merkle proof requests instead of refusing them - #164

Merged
nol4lej merged 1 commit into
mainfrom
fix/queued-merkle-proofs
Oct 10, 2026
Merged

nol4lej merged 1 commit into
mainfrom
fix/queued-merkle-proofs

Conversation

@nol4lej

@nol4lej nol4lej commented Oct 10, 2026

Copy link
Copy Markdown
Member

fix(rpc): queue Merkle proof requests instead of refusing them

The privacy_getMerkleProof* RPCs refused any request past 4 in flight, with no queue: a burst of 16 simultaneous requests lost 12, although each proof takes ~2.5 ms and the node could have served all of them in ~10 ms. Every client had to retry around a refusal that should not happen under normal load.

What changes (client/rpc-v2/src/privacy.rs, ProofGate)

Before After
Slots 4, fixed half the node's cores, at least 1 (available_parallelism), automatic. The rest stay free for importing and authoring blocks, GRANDPA, networking and the other RPCs. 4 cores → 2, 16 → 8
When all slots are busy refused at once waits for a slot
Queue none at most 128 waiting, each for at most 2 s
Refused (-32009) as soon as 4 were running only when the queue is full (abuse) or the wait runs out (a saturated node)
  • Both proof RPCs are blocking, so each request waits on its own blocking thread: std Mutex + Condvar, no new dependency.
  • 128 stays well below tokio's blocking pool (512 by default).
  • A slot is freed when its permit drops, even if the proof panics.
  • A poisoned mutex is recovered, never fatal.

Measured (dev node, sealed 2^20-leaf tree: the most expensive path; 16 WebSocket connections, 10 cores → 5 slots)

Burst Before: served / refused After: served / refused After: p50 / p99 After: burst time
16 4 / 12 16 / 0 11.6 / 17.6 ms 18 ms
64 4 / 60 64 / 0 50.5 / 73.5 ms 75 ms
256 4 / 252 135 / 121 65 / 109 ms 110 ms
512 4 / 508 137 / 375 58 / 106 ms 107 ms
  • Bursts of up to ~130 simultaneous requests to one node are now fully served, in about a tenth of a second.
  • Beyond that the excess is refused at once, without saturating the node.
  • During every burst privacy_getMerkleRoot (not gated) answered in ≤ 10 ms, and the node kept authoring blocks, with no errors in the log.

Testing

fc-rpc-v2: 44 tests. Clippy -D warnings and fmt clean; the node builds. The gate tests (zero failures over 30 consecutive runs; they use threads) cover:

  • slots from cores;
  • a free slot taken without waiting;
  • a full queue refused at once;
  • a waiter taking the freed slot;
  • a wait past the deadline refused;
  • a burst of 64 all served, never more than slots at once;
  • a burst past the queue refusing only the excess;
  • a panicking proof freeing its slot.

Live regression: e2e spends against the new node, 20/20.

No runtime change: node only.

@nol4lej
nol4lej merged commit 9adb31b into main Oct 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant