Skip to content

blob, lvol: allocate per-blob I/O channels to isolate lvol I/O - #96

Merged
derekbit merged 1 commit into
longhorn:longhorn-v26.05from
derekbit:fix-lvol-per-blob-io-channel
Nov 19, 2026
Merged

derekbit merged 1 commit into
longhorn:longhorn-v26.05from
derekbit:fix-lvol-per-blob-io-channel

Conversation

@derekbit

@derekbit derekbit commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Which issue(s) this PR fixes:

Issue longhorn/longhorn#14589

What this PR does / why we need it:

Special notes for your reviewer:

Additional documentation or context

@derekbit
derekbit force-pushed the fix-lvol-per-blob-io-channel branch from 65cb652 to d0d5c20 Compare November 18, 2026 14:37
@derekbit
derekbit requested a balanced review from Copilot November 18, 2026 14:38
@derekbit derekbit self-assigned this Nov 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Per-blob channels alter cross-thread storage lifetimes and asynchronous teardown, requiring human review and real-device stress validation of CoW stalls and close/unload races beyond the mocked unit coverage.

No open issues 🎉

What changed in this PR

This PR introduces per-blob I/O channels and adopts them for logical volumes, isolating request pools and copy-on-write queues so stalled I/O on one volume does not exhaust another volume’s channel resources.

Changes:

  • Add and export a per-blob channel API with lifecycle management and shared external-snapshot channel handling.
  • Route lvol I/O and maintenance operations through dedicated channels, handling allocation failures.
  • Add tests for channel isolation, freeze/unfreeze, external-snapshot replacement, and asynchronous teardown.
File Description
test/​unit/​lib/​lvol/​lvol.c/​lvol_ut.c Test lvol channel allocation and blob selection.
test/​unit/​lib/​blob/​blob.c/​esnap_dev.c Resolve external-snapshot test channels through the root channel.
test/​unit/​lib/​blob/​blob.c/​blob_ut.c Test isolation, freeze/unfreeze, parent replacement, and teardown.
test/​unit/​lib/​bdev/​vbdev_lvol.c/​vbdev_lvol_ut.c Test handling of lvol channel allocation failure.
module/​bdev/​lvol/​vbdev_lvol.c Reject channel creation when allocation fails.
lib/​lvol/​lvol.c Use per-blob channels for lvol I/O and maintenance operations.
lib/​blob/​spdk_blob.map Export the new channel allocation API.
lib/​blob/​blobstore.h Add channel tracking and root-channel lookup support.
lib/​blob/​blobstore.c Implement per-blob channels and integrate lifecycle and queued-I/O handling.
include/​spdk/​lvol.h Document dedicated lvol channels and release requirements.
include/​spdk/​blob.h Declare and document the per-blob channel API.

Review effort: Balanced

@derekbit

Copy link
Copy Markdown
Member Author

Before the improvement

poll-group thread T
 └─ spdk_bs_channel (root)
      ├─ reqs[512]            ← shared by all lvols
      └─ need_cluster_alloc   ← shared by all lvos, and only one COW at the same time.

After the improvement

poll-group thread T
 └─ root spdk_bs_channel (bs io_device)
      ├─ esnap_channels (RB tree, still on the root)
      └─ thread_channels: [root, ch(X), ch(Y), ch(S)]
                                                             │       │       │
             per-blob channels: each has its own reqs[512], need_cluster_alloc,
             new_cluster_page, release_cluster_page and queued_io,
             and each holds a reference to the root channel

Comment thread test/unit/lib/blob/blob.c/blob_ut.c Resolved
Comment thread lib/blob/blobstore.c
Comment on lines +4184 to +4185
if (!__atomic_load_n(&blob->io_device_registered, __ATOMIC_ACQUIRE) ||
bs_channel_init(blob->bs, channel) != 0) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So IIUC memory usage now scales with the number of lvols that have an open IO channel times the number of threads? Worth adding to the doc comment?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I created a page for the memory usage.
longhorn/website#1480

spdk_lvol_get_io_channel() previously returned the blobstore-wide I/O
channel from spdk_bs_alloc_io_channel(). Every lvol on an lvstore
therefore shared a single spdk_bs_channel per poll-group thread,
including its 512-entry spdk_bs_request_set pool (reqs) and its
need_cluster_alloc serialization queue.

When an esnap-backed lvol (such as a rebuilding Longhorn V2 replica)
writes to an unallocated cluster, bs_allocate_and_copy_cluster() holds
the active allocation while reading from the remote external snapshot
and queues subsequent unallocated-cluster writes on that channel in
need_cluster_alloc, including writes to unrelated healthy lvols on the
same lvstore. Under heavy write load, nodes rebuilding replicas from
each other's snapshots exhaust all 512 request sets in
need_cluster_alloc and reject incoming snapshot reads with -ENOMEM,
deadlocking I/O across all lvols on the lvstore.

Add spdk_blob_alloc_io_channel() by registering each opened spdk_blob
as an I/O device when opened and deferring final blob memory release to
the spdk_io_device_unregister() completion callback:

- Each open blob gets its own spdk_bs_channel per thread with an
  isolated 512-entry reqs pool and need_cluster_alloc queue, so slow
  or stalled esnap CoW reads on one lvol cannot block cluster
  allocations or starve request sets for snapshot reads and other
  lvols.

- Guard blob->io_device_registered with acquire/release atomics and
  free struct spdk_blob only from blob_io_device_unregister_cpl() once
  all per-blob channel creations and asynchronous put_io_channel()
  teardowns across threads have completed.

- Each per-blob channel holds a reference to the thread's root
  blobstore channel (bs_io_channel) and links into
  root->thread_channels. This preserves spdk_bs_unload() teardown
  ordering, keeps esnap_channels on the root channel, and allows
  blob_execute_queued_io() and blob_esnap_destroy_one_channel() to
  walk all channels on the thread during freeze/unfreeze and parent
  replacement.

- Update spdk_lvol_get_io_channel() and internal lvol copy, inflate,
  decouple, and checksum paths to allocate per-blob channels, and
  check for channel allocation failure in vbdev_lvol. Document that
  the returned channel must be released before the lvol or blob is
  closed.

Add blob and lvol unit tests covering channel isolation under a
stalled esnap CoW read, freeze/unfreeze and esnap parent replacement
on per-blob channels, asynchronous per-blob channel close/teardown,
per-lvol channel allocation, and vbdev_lvol channel creation failure.

Related: longhorn/longhorn#14589

Signed-off-by: Derek Su <derek.su@suse.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@derekbit
derekbit force-pushed the fix-lvol-per-blob-io-channel branch from d0d5c20 to d7d05d1 Compare November 19, 2026 08:25
@derekbit
derekbit requested a review from c3y1huang November 19, 2026 09:01
@derekbit
derekbit merged commit f4213db into longhorn:longhorn-v26.05 Nov 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants