Repository navigation
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. External sites
ReviewsSee the guideline and AI policy for information on the review process. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. LLM Linter (✨ experimental)Possible places where named args for integral literals may be used (e.g.
2026-10-28 02:47:34 |
|
The performance change I see on a Ryzen 9 3900X system with RAID HDD storage is ~11minutes with to ~1m35s for a load, consisting of ~1m15s of CPU time followed by ~20s of disk flushing; with SSD and a slightly faster CPU, total time is reduced from ~5m30s to ~50s (no significant flush delay after the CPU work is completed here). These numbers are with the block 965000 utxo set. |
|
🚧 At least one of the CI tasks failed. HintsTry to run the tests locally, according to the documentation. However, a CI failure may still
Leave a comment here, if you need help tracking down a confusing failure. |
51d7d9e to
32575f0
Compare
|
Can you please give us more context? Why do you consider an 11-minute load too slow? Is there a reason to think this is what's preventing people from using it - enough to justify yet more complexity in the most "load-bearing" parts of the code? Or is this just a demo to explore the possibilities? Over the past years we've discussed how relying on someone else's UTXO set makes us uncomfortable (see #32377 and #36598). |
32575f0 to
e81eeb8
Compare
In bips#2137 Sjors suggested splitting up the snapshot by coin to improve performance, which didn't seem sensible to me.
I don't consider it too slow; though it's possible it could be on slower systems than the ones I test on. But the reason it is that slow is because the database gets compacted multiple times, and it also seems to use excessive amounts of memory while loading, and there isn't really a good reason to do that. The difference between checking whether better performance is possible and actually making it available to users isn't that large.
As I've said on the list, I think the assumeutxo approach is a much better approach to resolving the "fast startup of a new node" challenge than trying to get a few extra percent out of IBD performance (which introduces its own large sets of complexities). If it's been decided that that's not one of this project's goals, that's fine I guess, but I haven't seen a project statement along those lines. Making assumeutxo a reasonable default, rather than something for experts, does require significantly more complexity (ie, the various stuff for the BIP above, cf #35054). Likewise if we want to do FEC encoding so pruned nodes can help share utxo snapshots or support IBD. |
4840412 to
92c6b60
Compare
|
🚧 At least one of the CI tasks failed. HintsTry to run the tests locally, according to the documentation. However, a CI failure may still
Leave a comment here, if you need help tracking down a confusing failure. |
The snapshot file format is not node state: reading a snapshot needs no chainstate, so move it out. This prepares the way for adding snapshot parsing to bitcoin-util in a future commit. Mostly move-only. The commit can be reviewed with the --color-moved=dimmed-zebra option for an easier review.
SnapshotMetadata took the expected network in its constructor and compared it while parsing, throwing on mismatch. Split that check out as a separate `CheckNetwork` method, so that callers have the option to read snapshots for arbitrary networks. Also replace the per-object `const std::set` for checking supported versions with an inline function.
Normally, we want updates to the database to be fsynced to disk so that the database is in a consistent state even after a crash. This adds an option to drop that guarantee, in favour of faster bulk writes, relying on the caller to make other arrangements to ensure the resulting database is flushed to disk.
This populates a chainstate leveldb (as a CDBWrapper) directly from a batch (generated from Coin objects). This avoids the overhead of going through a CoinsView map, and allows splitting the batch creation and the batch write/flush across different threads.
Multi-producer, multi-consumer thread-safe queue with maximum capacity for handing out jobs to threads.
Allows importing snapshots directly into leveldb (dbwrapper), without passing through Chainstate or CoinsView first. Supports multithreaded processing, with one thread reading, one thread hashing, one thread serializing for the database, and one thread populating the database. Adds an additional check that the coins in the snapshot are sorted in txid order. Note that because v2 dump follow the database key order (which encodes output indices as VARINTs), but the UTXO commitment requires outputs to be ordered numerically by output index, a sorting step is required (eg VARINT(16512) sorts less than VARINT(16511). We bound the total serialized transaction outputs by the consensus block weight limit to bound the amount of date being sorted in case of corrupt snapshot files. Nothing calls it yet, so behaviour is unchanged. Co-Authored-By: Sjors Provoost <sjors@sprovoost.nl>
ActivateSnapshot and, therefore, the loadtxoutset RPC now use the new utxo_snapshot.cpp pipeline. The dump is no longer staged in a CCoinsViewCache and then hashed and flushed out of it, so `-dbcache` worth of cached coins during a mainnet load are gone along with the flush cycles. In addition, the hashing of the dump takes place in parallel with the import, rather than after it completes. The old loader is left in place until the next commit removes it; nothing calls it any more.
After the previous two commits, this code is no longer used, so remove it. This also removes the circular dependency between node/utxo_snapshot and validation.
Adds a subcommand for quickly generating the summary of a serialized utxo set for verification purposes (including various hashes, coin count and value).
92c6b60 to
fbda3ac
Compare
|
🐙 This pull request conflicts with the target branch and needs rebase. |

Significantly improves loadtxoutset performance via a number of changes:
Also adds a subcommand to bitcoin-util for getting a snapshot's hash_serialized_3, muhash, coin count and total amount values without importing to leveldb first.
On my systems, performance ends up bottlenecked by leveldb CPU usage.