Skip to content

feat: introduce batched dirty recovery spdk - #85

Open
davidcheng0922 wants to merge 5 commits into
longhorn:longhorn-v26.05from
davidcheng0922:issue-12837-batch-dirty-recovery-v26.05
Open

davidcheng0922 wants to merge 5 commits into
longhorn:longhorn-v26.05from
davidcheng0922:issue-12837-batch-dirty-recovery-v26.05

Conversation

@davidcheng0922

@davidcheng0922 davidcheng0922 commented Oct 21, 2026 •

Copy link
Copy Markdown

Which issue(s) this PR fixes:

longhorn/longhorn#12837

What this PR does / why we need it:

Special notes for your reviewer:

Additional documentation or context

Test by e2e: longhorn/longhorn-tests#3932

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Standard replay scheduling scales quadratically, and default-enabled batched snapshot recovery lacks crash-test coverage.

2 open issues
What changed in this PR

Introduces batched blobstore recovery to reduce Longhorn V2 disk startup delays described in #12837.

Changes:

  • Adds configurable recovery batching and enables it for lvol stores.
  • Strengthens recovery validation, failure handling, and cluster accounting.
  • Expands recovery and fault-injection tests.
File Description
test/​unit/​lib/​lvol/​lvol.c/​lvol_ut.c Verifies lvol recovery configuration.
test/​unit/​lib/​blob/​blob.c/​blob_ut.c Adds recovery and failure-path tests.
lib/​lvol/​lvol.c Enables batched recovery by default.
lib/​blob/​request.c Preserves earlier batch errors.
lib/​blob/​blobstore.c Implements batched recovery and safety checks.
include/​spdk/​blob.h Exposes the recovery queue-depth option.

Review effort: Balanced

Comment thread lib/blob/blobstore.c Resolved
Comment thread lib/blob/blobstore.c Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The default-enabled crash-recovery rewrite affects allocation reconstruction and persistence, requiring human review and runtime validation.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@davidcheng0922

Copy link
Copy Markdown
Author

-i "single-replicaANDnode-reboot" -v RETRY_COUNT:1200 -v DATA_ENGINE:v2

https://ci.longhorn.io/job/private/job/longhorn-e2e-test/10554/

@github-actions

Copy link
Copy Markdown

SPDK manages code review in https://review.spdk.io. This PR will be closed automatically.

@github-actions github-actions Bot added the Stale label Oct 23, 2026
@derekbit derekbit removed the Stale label Oct 24, 2026
@github-actions

Copy link
Copy Markdown

SPDK manages code review in https://review.spdk.io. This PR will be closed automatically.

When a batch created with bs_sequence_to_batch() completes from
bs_batch_completion(), the batch callback receives the status of the
last I/O instead of the accumulated set->bserrno. If an earlier I/O in
the batch failed and the last one succeeded, the callback is told the
whole batch succeeded and goes on to consume a buffer that was never
filled.

bs_batch_close() already passes set->bserrno when the batch completes
there; do the same in bs_batch_completion().

Signed-off-by: David Cheng <david.cheng@suse.com>
Change-Id: I843767a1c47b45cf89f1851a51394e238ce4b1ae
@davidcheng0922
davidcheng0922 force-pushed the issue-12837-batch-dirty-recovery-v26.05 branch from 2501c67 to 369a9be Compare November 18, 2026 01:24
@davidcheng0922
davidcheng0922 requested a balanced review from Copilot November 18, 2026 01:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Recovery can trust partially rebuilt masks after a power loss because the dirty superblock is not flushed first, and unbounded chain-page retention can exhaust memory before fallback runs.

2 open issues

Review effort: Balanced

Comment thread lib/blob/blobstore.c Resolved
Comment thread lib/blob/blobstore.c Resolved
@davidcheng0922

Copy link
Copy Markdown
Author

Memory usage

The chain pages only grow with the number of blobs, and they are much smaller than the memory those blobs use once they are opened.

Most blobs use one metadata page. A blob only needs a second page once it has more than ~250 GiB allocated (with 1 MiB clusters).

After load, every lvol is opened, and each one keeps a cluster map of num_clusters × 8 bytes. Snapshots are the same size as their volume, so they need the same.

Blob with one metadata page During recovery After load
10 GiB (1 MiB clusters) 4 KiB 80 KiB
100 GiB 4 KiB 800 KiB

Recovery needs 20 to 200 times less memory for these pages than the opened lvols, and it frees them before opening any lvol. If a node runs out of memory here, it would also run out when opening the lvols, so limiting these pages would not help.

The two other allocations have a fixed size:

  • page_summaries: about 24 MiB per TiB of lvstore, no matter how many blobs there are.
  • Scan buffer: 16 MiB of DMA memory at recovery_qd = 16.

Example: 1 TiB lvstore, 10 × 100 GiB volumes, 1 MiB clusters

No snapshots 50 snapshots per volume
Blobs (including the lvstore super blob) 11 511
Chain pages (1 page per blob) ~44 KiB ~2 MiB
page_summaries (1,048,576 md pages × 24 B) 24 MiB 24 MiB
Scan buffer (DMA) 16 MiB 16 MiB
Resident active.clusters after load (102,400 clusters × 8 B per lvol) ~8 MiB ~398 MiB

The chain pages this comment is about stay around 2 MiB even with 50 snapshots per volume, about 0.5% of the resident memory, so bounding them would not reduce peak memory in a meaningful way.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Extent-page reads can exceed the configured recovery depth and risk memory-pressure failures, while replay concurrency is unnecessarily constrained by scan chunk count.

2 open issues
2 resolved since last review

Review effort: Balanced

Comment thread lib/blob/blobstore.c Resolved
Comment thread lib/blob/blobstore.c Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Recovery can write masks before rejecting invalid extent metadata, inline completions can overflow the stack, and dump failures can invoke callbacks before device cleanup.

4 open issues
2 resolved since last review

Review effort: Balanced

Comment thread lib/blob/blobstore.c Resolved
Comment thread lib/blob/blobstore.c Resolved
Comment thread lib/blob/blobstore.c Resolved
Comment thread lib/blob/blobstore.c Resolved
@davidcheng0922
davidcheng0922 force-pushed the issue-12837-batch-dirty-recovery-v26.05 branch from 369a9be to e932b42 Compare November 18, 2026 03:48
@davidcheng0922
davidcheng0922 requested a balanced review from Copilot November 18, 2026 03:49

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The dump error path invokes its callback before device cleanup, and the page-summary allocation fallback lacks coverage for successful standard replay.

2 open issues
4 resolved since last review

Review effort: Balanced

Comment thread lib/blob/blobstore.c Resolved
Comment thread lib/blob/blobstore.c Resolved
Dirty recovery trusts metadata pages too much. When a page fails
validation, recovery may skip it or count its clusters twice, and still
write the rebuilt masks.

Make the replay path stricter:

- Check descriptor lengths before dereferencing them. Reject truncated
  extent RLE descriptors and extents beyond the end of the blobstore.
- Count each cluster only once. An interrupted snapshot operation can
  leave the same cluster referenced by two blobs, which used to
  decrement num_free_clusters twice.
- Fail the load with -EILSEQ when a page referenced by a live chain is
  invalid, belongs to another blob, has an unexpected sequence number,
  or points outside the metadata region or to an already claimed page.
  Previously, such a chain was silently truncated.
- Check extent page numbers before submitting the reads, and index the
  extent page buffer by md_page_size instead of
  sizeof(struct spdk_blob_md_page).
- Fail the load if blob data claims a cluster that backs metadata.
- When recovery starts from a clean superblock (force_recover or a
  version 2 store without a used blobid mask), mark the superblock dirty
  before rewriting the masks. If the load fails partway, the next load
  replays metadata again instead of trusting partially written masks.
- Handle a failed used blobid mask read on a clean load, and
  spdk_bit_pool_create_from_array() failures.
- Complete a failed spdk_bs_dump() only after destroying the device, as
  a successful dump does.
- Free the recovery page buffers on every failure path.

Signed-off-by: David Cheng <david.cheng@suse.com>
Change-Id: I0beded0e748ecc8af3cb8735619d056ebbb639a8
@davidcheng0922
davidcheng0922 force-pushed the issue-12837-batch-dirty-recovery-v26.05 branch from e932b42 to bad71d6 Compare November 18, 2026 04:34
@davidcheng0922
davidcheng0922 requested a balanced review from Copilot November 18, 2026 04:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Transient scan-read corruption can cause a live root to be skipped, marking its metadata and clusters free and potentially allowing blob data to be overwritten.

1 open issue
2 resolved since last review

Review effort: Balanced

Comment thread lib/blob/blobstore.c Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

One or more reviewer sessions failed, leaving the review evidence incomplete.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Recovery can allocate 256 MiB of DMA buffers, causing a serial-scan fallback if allocation fails; scan slots need a byte-based cap.

1 open issue

Review effort: Balanced

Comment thread lib/blob/blobstore.c Resolved
Dirty recovery reads the metadata region one page at a time and follows
each blob's metadata chain with dependent reads. On large blobstores,
recovery time is dominated by per-I/O latency. In Longhorn, recovering
an NVMe-backed lvstore after an unclean shutdown took about 12 minutes
per disk.

Add a batched recovery path:

1. Scan the metadata region in chunks of up to 1 MiB, with up to
   recovery_qd reads in flight, to find blob root pages. If a chunk read
   fails, retry it one page at a time.
2. Read the metadata chains (and extent pages) of the discovered blobs,
   with up to recovery_qd chains in flight. Keep the pages in memory and
   validate each chain before using it.
3. Rebuild the used page, blobid and cluster masks serially, in root page
   order, and write them using the existing serial-path code.

Nothing is written until every chain has been read and validated. If a
read fails or referenced metadata is invalid, the load fails without
rewriting the masks. If a memory allocation fails, the partially built
masks are cleared and the load falls back to serial recovery.

Add a recovery_qd field to struct spdk_bs_opts to select the batched
path. The default is 0, which keeps the existing serial recovery. Values
greater than 1 enable batched recovery.

Link: longhorn/longhorn#12837
Signed-off-by: David Cheng <david.cheng@suse.com>
Change-Id: I0562a9803dcb524bcb2671da182a06592f601643
Lvol stores are loaded by the lvol library, and in SPDK applications
usually by vbdev_lvol examine, so users cannot set struct spdk_bs_opts
fields such as recovery_qd. Without this change, dirty lvol stores
always use serial recovery.

Set recovery_qd to 16 in lvs_bs_opts_init(), as is already done for
max_channel_ops. The blobstore default stays 0, so direct users of the
blob API are not affected.

Signed-off-by: David Cheng <david.cheng@suse.com>
Change-Id: I41e895829ab8fb4b2bd0cfc9b737afc601f39771
The snapshot create/delete power-failure tests only exercised serial
dirty recovery. Batched recovery is enabled by default for lvol stores,
and snapshot crashes are the only case where a clone and its snapshot
share extent pages and clusters during recovery.

Run both tests with recovery_qd 0 and 16, keeping the same checks for
snapshot relationships, allocation masks and free cluster counts after
every crash point.

Change-Id: I1112122fdd68263c33564d59f27f650ce919692d
Signed-off-by: David Cheng <david.cheng@suse.com>
@davidcheng0922
davidcheng0922 force-pushed the issue-12837-batch-dirty-recovery-v26.05 branch from bad71d6 to ee6bb27 Compare November 18, 2026 05:13
@davidcheng0922
davidcheng0922 requested a balanced review from Copilot November 18, 2026 05:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new default recovery path rebuilds allocation masks and repairs snapshots, requiring human validation of crash consistency and startup performance on real devices and large stores.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@davidcheng0922

Copy link
Copy Markdown
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants