Repository navigation
Conversation
|
I'll wait for review from maintainers about the concept and then I may change the implementation if requested. I've written this in the simplest way I could think of to fix the issue that I previously had here. |
Roasbeef
left a comment
There was a problem hiding this comment.
The storage reduction is useful, but unconditional journal pruning breaks a later creation/rebuild of input-dependent indexes on an otherwise archival node. We need to preserve a supported rebuild path before merging. See the inline finding.
The root go test ./... suite passed. A focused test through the new depth-pruner reproduced the missing-journal assertion; a full node restart/index-manager rebuild test was not run.
| } | ||
| hashes = append(hashes, *hash) | ||
| } | ||
| err = dbPruneSpendJournalEntry(dbTx, hashes) |
There was a problem hiding this comment.
[P1] Preserve the spend data required by later index creation/rebuild.
After this pruning has run, restarting with a newly enabled addrindex or dropping/rebuilding cfindex requires Manager.Init to catch up through historical blocks. Both indexes implement NeedsInputs, and the catch-up loop calls chain.FetchSpendJournal for those blocks. Once a block with spends has lost its journal, that fetch fails with assertion failed: mismatched spend journal serialization - no serialization for expected ... stxos, preventing index initialization.
Index initialization precedes setupSpendJournalPruning in New, so the affected case is a later restart/rebuild after journals were pruned on an earlier run, not necessarily the first startup.
I reproduced the mechanism at this head by seeding a journal and height mapping at height 1, setting nextHeight=1, and calling dbPruneSpendJournalDepth at bestHeight=290. It deletes height 1, after which FetchSpendJournal fails. Please retain a supported way to create/rebuild these indexes after pruning, and add an index-manager catch-up regression test. Merely checking whether indexes are currently caught up won't restore journals already deleted.
There was a problem hiding this comment.
Makes sense, but what if we make the spend journal pruning opt in, with a flag --prunespendjournal=<nblocks> and disable the indexers (like --prune does)? To me, this would already be useful, as my archive btcd node has cfindex, txindex and addrindex disabled to reduce the storage usage while keeping all the blocks to be able to bootstrap new nodes even with no network connection.
In the first startup with the --prunespendjournal the journal is pruned if needed and we set the spendJournalStats in the DB, this way if the node is later started without the flag, we just start the node as if --prunespendjournal flag was used instead of failing to start because the spend journal was previously pruned and the flag was not passed, which I think that provides better UX than the behavior of --prune, that fails to start btcd if we don't pass the flag after pruning the blocks.
There was a problem hiding this comment.
I've added the option --prunespendjournal, now, pruning the journal is exclusive against cfindex, txindex and addrindex. The user must manually drop the indexes if they already exist when prunespendjournal is enabled.
Previously, archive nodes stored all the spend journal entries forever, ~75GiB of the ~90GiB used by leveldb was being used by the spend journal. This patch makes possible to run archive nodes with better performance in devices constrained in RAM and SSD by keeping leveldb in a small SSD to reduce IO latency, while keeping the blocks in an HDD (point --datadir to an HDD with enough available storage to store all blocks, then move the metadata directory to the SSD and replace it with a symlink). spend journal pruning can be enabled with the flag --prunespendjournal=<pruneDepthBlocks>, where the user can pass a prune depth of at least 288 blocks, which is enough to handle an unlikely chain reorg of ~2 days. If a reorg of more than prune depth blocks happen, a clean IBD is required to reach the chain tip again. Spend journal pruning is exclusive against cfindex, txindex and addrindex, this means that at least for now, these indexes must not be enabled/initialized to enable spend journal pruning. --prunespendjournal may be passed only once, as the config will be saved in the database and next btcd initializations will keep pruning the journal with the last depth passed to --prunespendjournal. As of now, after enabling spend journal pruning, it can only be disabled by deleting all node data and performing IBD again.
78ace52 to
029c474
Compare
Change Description
Closes #2579.
This patch provides ~80% leveldb storage usage reduction in archive nodes.
Currently, the master branch of btcd saves all the data needed to process chain
re-orgs for all achive node blocks in the leveldb database, causing leveldb to
use ~90GiB of storage by default, plus blocks in flat files.
From the ~90GiB of storage used by the leveldb database, ~75GiB is just the
spend journal.
In computers constrained in RAM, IBD time is dominated by I/O bottlenecks,
specially by fetching outputs being spent that are not cached. A way to reduce
the elapsed time significantly is to move the leveldb to a small and fast SSD
while keeping the blocks in a huge, slow HDD, as SSD's deal with random reads
better. But 70GiB is a lot for machines with small SSD's, so this patch fixes
this by only keeping the last N blocks of spent journal, where N is now 288.
We assume that a 288 block reorg is unlikely to happen, and if this happens,
performing IBD again will be the least of the problems, as there will be chaos
in several payment processing backends.
I'm already using the patch in my main btcd instance for some months.
As of today, RAM, SSD and HDD prices are sky high, so I think this is a valuable
optimization.
Steps to Test
Just run IBD in a new node or start a synced node with the patch, note that first
initialization in synced nodes may take a long time due to the pruning of the spend
journal. Also, in synced nodes a manual leveldb compaction must be triggered to
release the storage used by the deleted spend journal entries.
Pull Request Checklist
Testing
Code Style and Documentation
📝 Please see our Contribution Guidelines for further guidance.