Repository navigation
feat: introduce batched dirty recovery spdk - #85
davidcheng0922 wants to merge 5 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Standard replay scheduling scales quadratically, and default-enabled batched snapshot recovery lacks crash-test coverage.
2 open issues
What changed in this PR
Introduces batched blobstore recovery to reduce Longhorn V2 disk startup delays described in #12837.
Changes:
- Adds configurable recovery batching and enables it for lvol stores.
- Strengthens recovery validation, failure handling, and cluster accounting.
- Expands recovery and fault-injection tests.
| File | Description |
|---|---|
| test/unit/lib/lvol/lvol.c/lvol_ut.c | Verifies lvol recovery configuration. |
| test/unit/lib/blob/blob.c/blob_ut.c | Adds recovery and failure-path tests. |
| lib/lvol/lvol.c | Enables batched recovery by default. |
| lib/blob/request.c | Preserves earlier batch errors. |
| lib/blob/blobstore.c | Implements batched recovery and safety checks. |
| include/spdk/blob.h | Exposes the recovery queue-depth option. |
Review effort: Balanced
|
https://ci.longhorn.io/job/private/job/longhorn-e2e-test/10554/ |
|
SPDK manages code review in https://review.spdk.io. This PR will be closed automatically. |
|
SPDK manages code review in https://review.spdk.io. This PR will be closed automatically. |
When a batch created with bs_sequence_to_batch() completes from bs_batch_completion(), the batch callback receives the status of the last I/O instead of the accumulated set->bserrno. If an earlier I/O in the batch failed and the last one succeeded, the callback is told the whole batch succeeded and goes on to consume a buffer that was never filled. bs_batch_close() already passes set->bserrno when the batch completes there; do the same in bs_batch_completion(). Signed-off-by: David Cheng <david.cheng@suse.com> Change-Id: I843767a1c47b45cf89f1851a51394e238ce4b1ae
2501c67 to
369a9be
Compare
Memory usageThe chain pages only grow with the number of blobs, and they are much smaller than the memory those blobs use once they are opened. Most blobs use one metadata page. A blob only needs a second page once it has more than ~250 GiB allocated (with 1 MiB clusters). After load, every lvol is opened, and each one keeps a cluster map of
Recovery needs 20 to 200 times less memory for these pages than the opened lvols, and it frees them before opening any lvol. If a node runs out of memory here, it would also run out when opening the lvols, so limiting these pages would not help. The two other allocations have a fixed size:
Example: 1 TiB lvstore, 10 × 100 GiB volumes, 1 MiB clusters
The chain pages this comment is about stay around 2 MiB even with 50 snapshots per volume, about 0.5% of the resident memory, so bounding them would not reduce peak memory in a meaningful way. |
There was a problem hiding this comment.
🟡 Changes recommended
Recovery can write masks before rejecting invalid extent metadata, inline completions can overflow the stack, and dump failures can invoke callbacks before device cleanup.
4 open issues
2 resolved since last review
Review effort: Balanced
369a9be to
e932b42
Compare
Dirty recovery trusts metadata pages too much. When a page fails validation, recovery may skip it or count its clusters twice, and still write the rebuilt masks. Make the replay path stricter: - Check descriptor lengths before dereferencing them. Reject truncated extent RLE descriptors and extents beyond the end of the blobstore. - Count each cluster only once. An interrupted snapshot operation can leave the same cluster referenced by two blobs, which used to decrement num_free_clusters twice. - Fail the load with -EILSEQ when a page referenced by a live chain is invalid, belongs to another blob, has an unexpected sequence number, or points outside the metadata region or to an already claimed page. Previously, such a chain was silently truncated. - Check extent page numbers before submitting the reads, and index the extent page buffer by md_page_size instead of sizeof(struct spdk_blob_md_page). - Fail the load if blob data claims a cluster that backs metadata. - When recovery starts from a clean superblock (force_recover or a version 2 store without a used blobid mask), mark the superblock dirty before rewriting the masks. If the load fails partway, the next load replays metadata again instead of trusting partially written masks. - Handle a failed used blobid mask read on a clean load, and spdk_bit_pool_create_from_array() failures. - Complete a failed spdk_bs_dump() only after destroying the device, as a successful dump does. - Free the recovery page buffers on every failure path. Signed-off-by: David Cheng <david.cheng@suse.com> Change-Id: I0beded0e748ecc8af3cb8735619d056ebbb639a8
e932b42 to
bad71d6
Compare
Dirty recovery reads the metadata region one page at a time and follows each blob's metadata chain with dependent reads. On large blobstores, recovery time is dominated by per-I/O latency. In Longhorn, recovering an NVMe-backed lvstore after an unclean shutdown took about 12 minutes per disk. Add a batched recovery path: 1. Scan the metadata region in chunks of up to 1 MiB, with up to recovery_qd reads in flight, to find blob root pages. If a chunk read fails, retry it one page at a time. 2. Read the metadata chains (and extent pages) of the discovered blobs, with up to recovery_qd chains in flight. Keep the pages in memory and validate each chain before using it. 3. Rebuild the used page, blobid and cluster masks serially, in root page order, and write them using the existing serial-path code. Nothing is written until every chain has been read and validated. If a read fails or referenced metadata is invalid, the load fails without rewriting the masks. If a memory allocation fails, the partially built masks are cleared and the load falls back to serial recovery. Add a recovery_qd field to struct spdk_bs_opts to select the batched path. The default is 0, which keeps the existing serial recovery. Values greater than 1 enable batched recovery. Link: longhorn/longhorn#12837 Signed-off-by: David Cheng <david.cheng@suse.com> Change-Id: I0562a9803dcb524bcb2671da182a06592f601643
Lvol stores are loaded by the lvol library, and in SPDK applications usually by vbdev_lvol examine, so users cannot set struct spdk_bs_opts fields such as recovery_qd. Without this change, dirty lvol stores always use serial recovery. Set recovery_qd to 16 in lvs_bs_opts_init(), as is already done for max_channel_ops. The blobstore default stays 0, so direct users of the blob API are not affected. Signed-off-by: David Cheng <david.cheng@suse.com> Change-Id: I41e895829ab8fb4b2bd0cfc9b737afc601f39771
The snapshot create/delete power-failure tests only exercised serial dirty recovery. Batched recovery is enabled by default for lvol stores, and snapshot crashes are the only case where a clone and its snapshot share extent pages and clusters during recovery. Run both tests with recovery_qd 0 and 16, keeping the same checks for snapshot relationships, allocation masks and free cluster counts after every crash point. Change-Id: I1112122fdd68263c33564d59f27f650ce919692d Signed-off-by: David Cheng <david.cheng@suse.com>
bad71d6 to
ee6bb27
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The new default recovery path rebuilds allocation masks and repairs snapshots, requiring human validation of crash consistency and startup performance on real devices and large stores.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
|
Run e2e test for verifying data correctness: Longhorn Test PR: |



Which issue(s) this PR fixes:
longhorn/longhorn#12837
What this PR does / why we need it:
Special notes for your reviewer:
Additional documentation or context
Test by e2e: longhorn/longhorn-tests#3932