Repository navigation
Conversation
This populates a chainstate leveldb directly from byte buffers (generated from Coin objects via RawCoinDataDBWriter::Batch). This avoids the overhead of going through a CoinsView map, and allows splitting the Coin serialization step and the leveldb writes across different threads.
Multi-producer, multi-consumer thread-safe queue with maximum capacity for handing out jobs to threads.
Return cleared CDBBatch buffers from a database worker so constructing and obfuscating the next batch can overlap LevelDB inserting the previous one. Reuse two buffers, wait for both before recording the best block, and propagate worker exceptions through the snapshot writer's error handling. Keep the serialized-record stage and configured database flush size, and reduce decoded and serialized queue depths to one to bound memory alongside the extra encoded buffer.
Pack complete coins into payloads of at most 256 KiB, each prefixed by a CompactSize length. Repeat the transaction-group header when its remaining outputs move to the next chunk. Specify and check global ordering by txid bytes and numeric output index, including across chunks. Read chunk payloads in bulk into memory while retaining the existing hashing, serialization and database-write pipeline. Keep the transaction resource bound across chunk boundaries. Update existing snapshot tests, fuzz inputs and the SQLite decoder for the new format, including a transaction that spans multiple chunks.
Allows importing snapshots directly into leveldb (dbwrapper), without passing through Chainstate or CoinsView first. Supports multithreaded processing, with one thread reading, one thread hashing, one thread serializing for the database, and one thread populating the database. Adds an additional check that the coins in the snapshot are sorted in txid order. Nothing calls it yet, so behaviour is unchanged.
ActivateSnapshot and, therefore, the loadtxoutset RPC now use the new utxo_snapshot.cpp pipeline. The dump is no longer staged in a CCoinsViewCache and then hashed and flushed out of it, so `-dbcache` worth of cached coins during a mainnet load are gone along with the flush cycles. In addition, the hashing of the dump takes place in parallel with the import, rather than after it completes. The old loader is left in place until the next commit removes it; nothing calls it any more.
After the previous two commits, this code is no longer used, so remove it. This also removes the circular dependency between node/utxo_snapshot and validation.
Adds a subcommand for quickly generating the summary of a serialized utxo set for verification purposes (including various hashes, coin count and value).
Version 2 dumps follow the database key order, which encodes output indices as VARINTs. This can differ from the numeric ordering used by the UTXO commitment, for example at indices 16511 and 16512. Retain references to the decoded batches backing one transaction and hash its outputs in numeric order, including when it spans batches. Compare against database statistics in the regression test. Bound the total serialized transaction outputs by the consensus block weight limit before retaining decoded groups for sorting. Test groups on both sides of the limit to bound memory use for corrupt files.
e81eeb8 to
4840412
Compare
ajtowns
left a comment
There was a problem hiding this comment.
Nice. Squashed commit one and added as co-author; reworked commit two.
| * writes get flushed by the writer. */ | ||
| static constexpr size_t SNAPSHOT_DATA_BATCH_BYTES{4_MiB}; | ||
| static constexpr size_t SNAPSHOT_DB_BATCHES{8}; | ||
| static constexpr size_t SNAPSHOT_DB_BATCHES{1}; |
There was a problem hiding this comment.
I don't think dropping these all the way to 1 is appropriate for machines with slower disk where reads (of the snapshot) and writes (of the db) might be bursty and cause stalls? Are you seeing significant memory usage or slowdowns because of the higher values?
There was a problem hiding this comment.
Oh, this comment should have been against:
-/** How many batches each of the reader's queues may hold. */
-static constexpr size_t SNAPSHOT_QUEUE_BATCHES{16};
+//! Bound decoded look-ahead alongside the two encoded database batches.
+static constexpr size_t SNAPSHOT_QUEUE_BATCHES{1};| m_flush_size{flush_size} | ||
| { | ||
| m_available.Push(std::make_unique<CDBBatch>(db)); | ||
| m_available.Push(std::make_unique<CDBBatch>(db)); |
There was a problem hiding this comment.
Ah, I did try separating out the xor, but ended up with an extra copy which took the same amount of time. I didn't think about separating the CDBBatch construction from the flush, that works nicely and simplifies things a bunch as well as being a speedup.
|
I forgot to mention in this PR that it's all vibe coded and I only looked at the performance diff, so if something doesn't make sense like #15 (comment), just ignore it. |
|
Yeah, "allegedly fixes a hashing issue" and "Our v2 improvements" gave that away :) |
The first commit allegedly fixes a hashing issue:
The second commit speeds up
202610-loadtxoutset-speedya little on my setup (ssd drive).The third commit was my original suggestion from fjahr#3, to align the chunks on transaction-group boundaries. But it adds no benefit.