Skip to content

Serve compute-index over a range, and measure the read path honestly - #3

Merged
ponthief merged 3 commits into
masterfrom
claude/blindbit-oracle-indexer-perf-y29o9e
Sep 12, 2026
Merged

ponthief merged 3 commits into
masterfrom
claude/blindbit-oracle-indexer-perf-y29o9e

Conversation

@ponthief

Copy link
Copy Markdown
Owner

One production change, plus the measurement work that justified it and corrected an earlier claim of mine.

Pairs with ponthief/siLNt PR setavenger#54, which uses the new endpoint. Merge order doesn't matter: without this, the scanner detects the 404 once per scan and falls back to forward matching.

/range/compute-index

/tweaks returns tweaks and nothing else, so a scanner can't tell which transaction a tweak belongs to. Lacking that, it must enumerate forwards — build every output the tweak could produce (the plain one, plus each label added and negated) and look all of them up. With four labels that's nine curve operations per tweak, and it's the largest single cost in a scan.

The compute index already carries (txid, tweak) together. Knowing the transaction, a scanner can instead subtract the plain candidate from that transaction's own outputs and look the difference up among the labels — one curve operation per output plus a map hit. Measured against the real matcher (800 tweaks, 4 labels, 2 outputs/tx): 144.6 ms → 94.6 ms per block.

Only the per-block form existed, which is no use to a batched scan. This adds the ranged form alongside the other three, with the same streaming, the same max_range_blocks cap, and the same rule that an unindexed height is omitted rather than returned empty.

Correcting my earlier benchmark

The read-path numbers I quoted when the first performance work landed were measured on an eight-block store. That isn't a database — it's one or two LSM levels entirely in cache, and it flatters every access pattern equally. It was the wrong instrument, and I shouldn't have presented it as describing a production oracle.

Two test-only benchmarks that do:

scaling_bench_test.go varies the store size. The per-block cost of /tweaks and /utxos grows with it; the compute index's doesn't:

store size tweaks+utxos (by txid) compute index (by height)
8 blocks 1.24 ms 0.082 ms
64 blocks 1.29 ms 0.085 ms
256 blocks 2.34 ms 0.114 ms

That's structural. Outputs and tweaks are keyed by txid, so reading one block means a seek per transaction scattered across the whole keyspace, and each gets dearer as the tree deepens. The compute index is keyed by height — a contiguous scan, flat. A production oracle holds every block since BIP-352 activation, orders of magnitude past the largest store here, and that first column keeps climbing.

http_e2e_test.go serves real requests through the real handler stack, gzip included. For a 100-block batch at 500 tweaks/block:

server time raw gzipped
/range/tweaks 66 ms 3.5 MB 155 KB
/range/utxos 437 ms 17.6 MB 870 KB
/range/spent-outputs 83 ms 11.4 MB 1.5 MB
one batch 587 ms 31.0 MB

317 KB of JSON per block, and /utxos is 57% of the bytes and 75% of the server time — for data the scanner uses only to test set membership. Where the oracle and the wallet share a machine there's no round-trip latency to hide behind, and this is what a scan's "request time" actually is.

transport_bench_test.go separates the transport question from the index question, because "would gRPC be faster" is really two questions:

per block, on the wire bytes
REST today: /tweaks + /utxos (hex JSON) 207,500 1.0×
same data, compute index as JSON 101,000 2.1× smaller
gRPC: compute index + spent (protobuf) 92,542 2.2× smaller

Nearly all of it is the index, not the encoding — hex JSON to protobuf is only ~1.09× once both carry the compute index. Server-side the gap is larger and matters more on a shared machine: building the full protobuf message including the DB read costs 139 µs, against ~3.4 ms for the three REST handlers.

No production code changes in those three files. They exist so the next decision about the read path is made against measurements rather than against a benchmark too small to disagree.

Verification

go build, go vet, go test ./... all green. New endpoint covered for correct per-block data, oversized spans rejected, and the existing range-endpoint tests extended.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CUvwa6Sa45bAudBMbZc3Yc


Generated by Claude Code

…mething

The earlier read-path benchmarks used an eight-block store. That is not a
database — it is one or two LSM levels with everything in cache, and it
flatters any access pattern equally. It was the wrong instrument for
deciding that the read path was fixed, and the numbers it produced should
not have been quoted as if they described a real oracle.

Two benchmarks that do describe one:

scaling_bench_test.go varies the store size. The per-block cost of /tweaks
and /utxos grows with it while the compute index's does not, which is a
property of how they are keyed rather than of how much data they return:

  store size   tweaks+utxos (by txid)   compute index (by height)
  8 blocks           1.24 ms                   0.082 ms
  64 blocks          1.29 ms                   0.085 ms
  256 blocks         2.34 ms                   0.114 ms

Outputs and tweaks are keyed by txid, so reading one block means a seek per
transaction scattered across the whole keyspace, and the deeper the tree the
more each seek costs. The compute index is keyed by height, so reading one
block is a contiguous scan and stays flat. A production oracle holds every
block since BIP-352 activation, which is orders of magnitude past the
largest store here, and that is the direction the first column keeps moving
in.

http_e2e_test.go serves real requests through the real handler stack, gzip
middleware included, and reports time and payload. For a 100-block batch at
500 tweaks per block:

  /range/tweaks           66 ms     3.5 MB raw     155 KB gzipped
  /range/utxos           437 ms    17.6 MB raw     870 KB gzipped
  /range/spent-outputs    83 ms    11.4 MB raw     1.5 MB gzipped
  ---
  one batch              587 ms    31.0 MB raw

That is 317 KB of JSON per block, and /utxos is most of it: 57% of the
bytes and 75% of the server time, for data the scanner uses only to test
whether a pubkey is in a set. Where the oracle and the wallet share a
machine there is no round-trip latency to hide behind, and this is what a
scan's "request time" actually is.

No production code changes here. These exist so the next decision about the
read path is made against measurements rather than against a benchmark that
was too small to disagree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CUvwa6Sa45bAudBMbZc3Yc
"Would gRPC be faster for scanning" turns out to be two questions wearing
one name, and they have very different answers. The gRPC service is not
another encoding of the REST API: it has no equivalent of /tweaks or
/utxos at all. What it exposes is StreamBlockScanDataShort, which is the
compute index — keyed by height, not by txid — plus the spent outputs, as
protobuf, ranged and streaming.

So switching transports also switches indexes, and this measures the two
apart. Per block, 3000 txs with 500 tweaked:

  REST today: /tweaks + /utxos (hex JSON)     207,500 bytes
  same data, compute index as JSON            101,000 bytes   2.1x smaller
  gRPC: compute index + spent (protobuf)       92,542 bytes   2.2x smaller

Nearly all of the difference is the index, not the encoding: hex JSON to
protobuf is only about 1.09x once both are carrying the compute index. The
index is what halves the payload, and it is already reachable over plain
HTTP.

The larger gap is CPU, which matters more here than bytes do — where the
oracle and the wallet share a machine, every core the oracle spends
serialising is a core the scanner is not matching on, and on loopback the
bytes themselves are close to free. Building the full protobuf message for
a block, database read included, costs 139 us. The three REST handlers for
the same block cost about 3.4 ms between them. That is the compute index
being a contiguous scan rather than three thousand seeks, and it is the
part that gets worse as the store grows.

What none of this touches is the matching, which is client-side elliptic
curve work and was the larger half of the reported scan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CUvwa6Sa45bAudBMbZc3Yc
…xids

/tweaks returns tweaks and nothing else, so a scanner cannot tell which
transaction a tweak belongs to. Lacking that, it has to enumerate every
output the tweak could possibly produce — the plain one, plus each label
added and negated — and look all of them up. With the default four labels
that is nine curve operations per tweak, and it is the largest single cost
in a scan.

The compute index already carries (txid, tweak) together. Knowing the
transaction, a scanner can instead subtract the plain candidate from that
transaction's own outputs and look the difference up among the labels,
which is one curve operation per output and a map hit. Measured against the
real matcher over 800 tweaks with four labels and two outputs per
transaction, that is 144.6 ms per block against 94.6 ms.

Only the per-block form existed, which is no use to a batched scan. This
adds the ranged form alongside the other three, using the same streaming,
the same cap, and the same rule that an unindexed height is omitted rather
than returned empty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CUvwa6Sa45bAudBMbZc3Yc
@ponthief
ponthief merged commit 2c27a6c into master Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants