Skip to content

assume_utxo: much fast, super snappy snapshot load - #3

Closed
Sjors wants to merge 6 commits into
fjahr:2026-02-utxo-set-share-safe-take-2from
Sjors:2026/10/utxo-coin-chunks
Closed

Sjors wants to merge 6 commits into
fjahr:2026-02-utxo-set-share-safe-take-2from
Sjors:2026/10/utxo-coin-chunks

Conversation

@Sjors

@Sjors Sjors commented Oct 17, 2026 •

Copy link
Copy Markdown

This (vibe coded) branch adds three performance enhancements on top of bitcoin#35054:

  1. Align chunks with coins, so they can processed in parallel. As specified in BIP452: specify ordered independently decodable snapshot chunks bips#7. Requires a new v3 snapshot serialization.
  2. Skip cache and write directly to disk (we should do this anyway?)
  3. Something codex calls "bounded write pipeline" :-)

(1) and (2) have the biggest impact

Implementation SSD import
PR bitcoin#35054 baseline, v2 snapshot 232.4 s
v3 + incremental import 196.9 s
+ Direct database batches 66.0 s
+ Bounded write pipeline 48.1 s

Sjors added 6 commits October 17, 2026 08:05
Start a dedicated import worker when a chunk list is accepted. Feed it the
verified, contiguous chunk prefix, decoding coins and filling an inactive
chainstate while later chunks download. Out-of-order chunks remain on disk.
Serialize snapshot imports and interrupt/join the worker before shutdown
destroys its dependencies. Remove partial downloads on cancellation.

Use version 3's strict global outpoint order to compute HASH_SERIALIZED
during insertion, removing the database scan at the end of both downloaded
and file-based imports. Activation still requires the complete snapshot and
the existing assumeutxo commitment to match.

Expand the existing snapshot tests to cover processing before completion,
out-of-order and duplicate chunks, cancellation, restart and agreement with
an independent database hash. Register snapshot request messages in the
functional test peer to exercise a paused download.

An offline optimized benchmark of the 165,734,792-coin mainnet snapshot at
height 965000 took 190.0 seconds with the database hashing pass and 155.7
seconds with incremental hashing. Final flush/verification took 42.4 and
0.41 seconds respectively. Both matched Core's mainnet commitment. These
are single warm-cache runs with a tmpfs chainstate and a 450 MiB coin cache;
they exclude P2P transfer and download-file writes.

Validation: 158 native suites; 454 functional cases across the extended
run and corrected P2P recheck (7 environment skips; pruning and crash
stress tests excluded); snapshot fuzz replays; Clang thread-safety checks
on the changed import code and unit test; container lint.
Bypass the coins hash-map cache when importing a fresh, private snapshot
chainstate. Serialize validated coins into bounded database batches and
publish the best block only after the complete snapshot hash matches.

Extend the existing flush test to populate its database using many small
snapshot batches, check every coin, then exercise ordinary cache writes.
Keep two reusable database batches: the importer decodes, validates and
hashes coins into one while a worker writes the other. Backpressure at
each batch boundary bounds buffering independently of snapshot size.

Wait for the final database write before activation and propagate worker
errors to the importer. On cancellation, join any outstanding write and
discard the incomplete batch before the private database is destroyed.
Extend the existing flush test to exercise an abandoned import.
Measure a version 3 snapshot import into a fresh on-disk database using the
bounded snapshot writer. Verify the full coin count, ordering and UTXO
commitment to detect regressions in the complete import path.
Include database close in the measured iteration and document SSD invocation.

Keep the height-965000 fixture confined to the benchmark; production snapshot
acceptance is unchanged. Skip the import unless BITCOIN_BENCH_SNAPSHOT is set.
@Sjors

Sjors commented Oct 24, 2026

Copy link
Copy Markdown
Author

It looks like v3 offers no benefit on top of @ajtowns's recent effort. So I'm going to close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant