Repository navigation
Serve compute-index over a range, and measure the read path honestly - #3
Merged
Merged
Conversation
…mething The earlier read-path benchmarks used an eight-block store. That is not a database — it is one or two LSM levels with everything in cache, and it flatters any access pattern equally. It was the wrong instrument for deciding that the read path was fixed, and the numbers it produced should not have been quoted as if they described a real oracle. Two benchmarks that do describe one: scaling_bench_test.go varies the store size. The per-block cost of /tweaks and /utxos grows with it while the compute index's does not, which is a property of how they are keyed rather than of how much data they return: store size tweaks+utxos (by txid) compute index (by height) 8 blocks 1.24 ms 0.082 ms 64 blocks 1.29 ms 0.085 ms 256 blocks 2.34 ms 0.114 ms Outputs and tweaks are keyed by txid, so reading one block means a seek per transaction scattered across the whole keyspace, and the deeper the tree the more each seek costs. The compute index is keyed by height, so reading one block is a contiguous scan and stays flat. A production oracle holds every block since BIP-352 activation, which is orders of magnitude past the largest store here, and that is the direction the first column keeps moving in. http_e2e_test.go serves real requests through the real handler stack, gzip middleware included, and reports time and payload. For a 100-block batch at 500 tweaks per block: /range/tweaks 66 ms 3.5 MB raw 155 KB gzipped /range/utxos 437 ms 17.6 MB raw 870 KB gzipped /range/spent-outputs 83 ms 11.4 MB raw 1.5 MB gzipped --- one batch 587 ms 31.0 MB raw That is 317 KB of JSON per block, and /utxos is most of it: 57% of the bytes and 75% of the server time, for data the scanner uses only to test whether a pubkey is in a set. Where the oracle and the wallet share a machine there is no round-trip latency to hide behind, and this is what a scan's "request time" actually is. No production code changes here. These exist so the next decision about the read path is made against measurements rather than against a benchmark that was too small to disagree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CUvwa6Sa45bAudBMbZc3Yc
"Would gRPC be faster for scanning" turns out to be two questions wearing one name, and they have very different answers. The gRPC service is not another encoding of the REST API: it has no equivalent of /tweaks or /utxos at all. What it exposes is StreamBlockScanDataShort, which is the compute index — keyed by height, not by txid — plus the spent outputs, as protobuf, ranged and streaming. So switching transports also switches indexes, and this measures the two apart. Per block, 3000 txs with 500 tweaked: REST today: /tweaks + /utxos (hex JSON) 207,500 bytes same data, compute index as JSON 101,000 bytes 2.1x smaller gRPC: compute index + spent (protobuf) 92,542 bytes 2.2x smaller Nearly all of the difference is the index, not the encoding: hex JSON to protobuf is only about 1.09x once both are carrying the compute index. The index is what halves the payload, and it is already reachable over plain HTTP. The larger gap is CPU, which matters more here than bytes do — where the oracle and the wallet share a machine, every core the oracle spends serialising is a core the scanner is not matching on, and on loopback the bytes themselves are close to free. Building the full protobuf message for a block, database read included, costs 139 us. The three REST handlers for the same block cost about 3.4 ms between them. That is the compute index being a contiguous scan rather than three thousand seeks, and it is the part that gets worse as the store grows. What none of this touches is the matching, which is client-side elliptic curve work and was the larger half of the reported scan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CUvwa6Sa45bAudBMbZc3Yc
…xids /tweaks returns tweaks and nothing else, so a scanner cannot tell which transaction a tweak belongs to. Lacking that, it has to enumerate every output the tweak could possibly produce — the plain one, plus each label added and negated — and look all of them up. With the default four labels that is nine curve operations per tweak, and it is the largest single cost in a scan. The compute index already carries (txid, tweak) together. Knowing the transaction, a scanner can instead subtract the plain candidate from that transaction's own outputs and look the difference up among the labels, which is one curve operation per output and a map hit. Measured against the real matcher over 800 tweaks with four labels and two outputs per transaction, that is 144.6 ms per block against 94.6 ms. Only the per-block form existed, which is no use to a batched scan. This adds the ranged form alongside the other three, using the same streaming, the same cap, and the same rule that an unindexed height is omitted rather than returned empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CUvwa6Sa45bAudBMbZc3Yc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One production change, plus the measurement work that justified it and corrected an earlier claim of mine.
Pairs with
ponthief/siLNtPR setavenger#54, which uses the new endpoint. Merge order doesn't matter: without this, the scanner detects the 404 once per scan and falls back to forward matching./range/compute-index/tweaksreturns tweaks and nothing else, so a scanner can't tell which transaction a tweak belongs to. Lacking that, it must enumerate forwards — build every output the tweak could produce (the plain one, plus each label added and negated) and look all of them up. With four labels that's nine curve operations per tweak, and it's the largest single cost in a scan.The compute index already carries
(txid, tweak)together. Knowing the transaction, a scanner can instead subtract the plain candidate from that transaction's own outputs and look the difference up among the labels — one curve operation per output plus a map hit. Measured against the real matcher (800 tweaks, 4 labels, 2 outputs/tx): 144.6 ms → 94.6 ms per block.Only the per-block form existed, which is no use to a batched scan. This adds the ranged form alongside the other three, with the same streaming, the same
max_range_blockscap, and the same rule that an unindexed height is omitted rather than returned empty.Correcting my earlier benchmark
The read-path numbers I quoted when the first performance work landed were measured on an eight-block store. That isn't a database — it's one or two LSM levels entirely in cache, and it flatters every access pattern equally. It was the wrong instrument, and I shouldn't have presented it as describing a production oracle.
Two test-only benchmarks that do:
scaling_bench_test.govaries the store size. The per-block cost of/tweaksand/utxosgrows with it; the compute index's doesn't:That's structural. Outputs and tweaks are keyed by txid, so reading one block means a seek per transaction scattered across the whole keyspace, and each gets dearer as the tree deepens. The compute index is keyed by height — a contiguous scan, flat. A production oracle holds every block since BIP-352 activation, orders of magnitude past the largest store here, and that first column keeps climbing.
http_e2e_test.goserves real requests through the real handler stack, gzip included. For a 100-block batch at 500 tweaks/block:/range/tweaks/range/utxos/range/spent-outputs317 KB of JSON per block, and
/utxosis 57% of the bytes and 75% of the server time — for data the scanner uses only to test set membership. Where the oracle and the wallet share a machine there's no round-trip latency to hide behind, and this is what a scan's "request time" actually is.transport_bench_test.goseparates the transport question from the index question, because "would gRPC be faster" is really two questions:/tweaks+/utxos(hex JSON)Nearly all of it is the index, not the encoding — hex JSON to protobuf is only ~1.09× once both carry the compute index. Server-side the gap is larger and matters more on a shared machine: building the full protobuf message including the DB read costs 139 µs, against ~3.4 ms for the three REST handlers.
No production code changes in those three files. They exist so the next decision about the read path is made against measurements rather than against a benchmark too small to disagree.
Verification
go build,go vet,go test ./...all green. New endpoint covered for correct per-block data, oversized spans rejected, and the existing range-endpoint tests extended.🤖 Generated with Claude Code
https://claude.ai/code/session_01CUvwa6Sa45bAudBMbZc3Yc
Generated by Claude Code