Skip to content

assume_utxo: slightly more speedy - #15

Closed
Sjors wants to merge 11 commits into
ajtowns:202610-loadtxoutset-speedyfrom
Sjors:2026/10/loadtxoutset-speedy
Closed

Sjors wants to merge 11 commits into
ajtowns:202610-loadtxoutset-speedyfrom
Sjors:2026/10/loadtxoutset-speedy

Conversation

@Sjors

@Sjors Sjors commented Oct 24, 2026

Copy link
Copy Markdown

The first commit allegedly fixes a hashing issue:

Variant Median import Median CPU time
AJ + hashing fix 46.64 s 109.79 s
Our v2 improvements 39.80 s 112.76 s
Improvements + v3 40.37 s 101.03 s

The second commit speeds up 202610-loadtxoutset-speedy a little on my setup (ssd drive).

The third commit was my original suggestion from fjahr#3, to align the chunks on transaction-group boundaries. But it adds no benefit.

Both are medians of three runs:

  • Median import: elapsed wall-clock time to read and decode the snapshot, verify its commitment, build and close the database, sync its files to SSD, and write the completion marker. Includes I/O waits.
  • Median CPU time: user + kernel CPU-seconds consumed by the process across all its threads. Two cores working for one second contribute two CPU-seconds, so this can exceed elapsed time.
    Snapshot input was warmed in the filesystem cache before each run. The separate database-verification scan is excluded.

One measurement detail: CPU time also includes harness startup and header parsing; the elapsed timer starts after those steps.

ajtowns and others added 11 commits October 23, 2026 11:35
This populates a chainstate leveldb directly from byte buffers (generated
from Coin objects via RawCoinDataDBWriter::Batch). This avoids the
overhead of going through a CoinsView map, and allows splitting the Coin
serialization step and the leveldb writes across different threads.
Multi-producer, multi-consumer thread-safe queue with maximum capacity
for handing out jobs to threads.
Return cleared CDBBatch buffers from a database worker so constructing and
obfuscating the next batch can overlap LevelDB inserting the previous one.
Reuse two buffers, wait for both before recording the best block, and
propagate worker exceptions through the snapshot writer's error handling.

Keep the serialized-record stage and configured database flush size, and
reduce decoded and serialized queue depths to one to bound memory alongside
the extra encoded buffer.
Pack complete coins into payloads of at most 256 KiB, each prefixed by a
CompactSize length. Repeat the transaction-group header when its remaining
outputs move to the next chunk. Specify and check global ordering by txid
bytes and numeric output index, including across chunks.

Read chunk payloads in bulk into memory while retaining the existing
hashing, serialization and database-write pipeline. Keep the transaction
resource bound across chunk boundaries.

Update existing snapshot tests, fuzz inputs and the SQLite decoder for the
new format, including a transaction that spans multiple chunks.
Allows importing snapshots directly into leveldb (dbwrapper), without
passing through Chainstate or CoinsView first. Supports multithreaded
processing, with one thread reading, one thread hashing, one thread
serializing for the database, and one thread populating the database.

Adds an additional check that the coins in the snapshot are sorted in
txid order.

Nothing calls it yet, so behaviour is unchanged.
ActivateSnapshot and, therefore, the loadtxoutset RPC now use the new
utxo_snapshot.cpp pipeline.

The dump is no longer staged in a CCoinsViewCache and then hashed and
flushed out of it, so `-dbcache` worth of cached coins during a mainnet
load are gone along with the flush cycles. In addition, the hashing
of the dump takes place in parallel with the import, rather than after
it completes.

The old loader is left in place until the next commit removes it;
nothing calls it any more.
After the previous two commits, this code is no longer used, so remove it.

This also removes the circular dependency between node/utxo_snapshot
and validation.
Adds a subcommand for quickly generating the summary of a serialized
utxo set for verification purposes (including various hashes, coin count
and value).
Version 2 dumps follow the database key order, which encodes output
indices as VARINTs. This can differ from the numeric ordering used by
the UTXO commitment, for example at indices 16511 and 16512.

Retain references to the decoded batches backing one transaction and
hash its outputs in numeric order, including when it spans batches.
Compare against database statistics in the regression test.

Bound the total serialized transaction outputs by the consensus block
weight limit before retaining decoded groups for sorting. Test groups
on both sides of the limit to bound memory use for corrupt files.

@ajtowns ajtowns left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Squashed commit one and added as co-author; reworked commit two.

* writes get flushed by the writer. */
static constexpr size_t SNAPSHOT_DATA_BATCH_BYTES{4_MiB};
static constexpr size_t SNAPSHOT_DB_BATCHES{8};
static constexpr size_t SNAPSHOT_DB_BATCHES{1};

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think dropping these all the way to 1 is appropriate for machines with slower disk where reads (of the snapshot) and writes (of the db) might be bursty and cause stalls? Are you seeing significant memory usage or slowdowns because of the higher values?

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, this comment should have been against:

-/** How many batches each of the reader's queues may hold. */
-static constexpr size_t SNAPSHOT_QUEUE_BATCHES{16};
+//! Bound decoded look-ahead alongside the two encoded database batches.
+static constexpr size_t SNAPSHOT_QUEUE_BATCHES{1};

Comment thread src/txdb.cpp
m_flush_size{flush_size}
{
m_available.Push(std::make_unique<CDBBatch>(db));
m_available.Push(std::make_unique<CDBBatch>(db));

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I did try separating out the xor, but ended up with an extra copy which took the same amount of time. I didn't think about separating the CDBBatch construction from the flush, that works nicely and simplifies things a bunch as well as being a speedup.

@ajtowns ajtowns closed this Oct 25, 2026
@Sjors

Sjors commented Oct 25, 2026

Copy link
Copy Markdown
Author

I forgot to mention in this PR that it's all vibe coded and I only looked at the performance diff, so if something doesn't make sense like #15 (comment), just ignore it.

@ajtowns

ajtowns commented Oct 25, 2026

Copy link
Copy Markdown
Owner

Yeah, "allegedly fixes a hashing issue" and "Our v2 improvements" gave that away :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants