Repository navigation
blob, lvol: allocate per-blob I/O channels to isolate lvol I/O - #96
Conversation
65cb652 to
d0d5c20
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Per-blob channels alter cross-thread storage lifetimes and asynchronous teardown, requiring human review and real-device stress validation of CoW stalls and close/unload races beyond the mocked unit coverage.
No open issues 🎉
What changed in this PR
This PR introduces per-blob I/O channels and adopts them for logical volumes, isolating request pools and copy-on-write queues so stalled I/O on one volume does not exhaust another volume’s channel resources.
Changes:
- Add and export a per-blob channel API with lifecycle management and shared external-snapshot channel handling.
- Route lvol I/O and maintenance operations through dedicated channels, handling allocation failures.
- Add tests for channel isolation, freeze/unfreeze, external-snapshot replacement, and asynchronous teardown.
| File | Description |
|---|---|
test/unit/lib/lvol/lvol.c/lvol_ut.c |
Test lvol channel allocation and blob selection. |
test/unit/lib/blob/blob.c/esnap_dev.c |
Resolve external-snapshot test channels through the root channel. |
test/unit/lib/blob/blob.c/blob_ut.c |
Test isolation, freeze/unfreeze, parent replacement, and teardown. |
test/unit/lib/bdev/vbdev_lvol.c/vbdev_lvol_ut.c |
Test handling of lvol channel allocation failure. |
module/bdev/lvol/vbdev_lvol.c |
Reject channel creation when allocation fails. |
lib/lvol/lvol.c |
Use per-blob channels for lvol I/O and maintenance operations. |
lib/blob/spdk_blob.map |
Export the new channel allocation API. |
lib/blob/blobstore.h |
Add channel tracking and root-channel lookup support. |
lib/blob/blobstore.c |
Implement per-blob channels and integrate lifecycle and queued-I/O handling. |
include/spdk/lvol.h |
Document dedicated lvol channels and release requirements. |
include/spdk/blob.h |
Declare and document the per-blob channel API. |
Review effort: Balanced
Before the improvementAfter the improvement |
| if (!__atomic_load_n(&blob->io_device_registered, __ATOMIC_ACQUIRE) || | ||
| bs_channel_init(blob->bs, channel) != 0) { |
There was a problem hiding this comment.
So IIUC memory usage now scales with the number of lvols that have an open IO channel times the number of threads? Worth adding to the doc comment?
There was a problem hiding this comment.
Yes, I created a page for the memory usage.
longhorn/website#1480
spdk_lvol_get_io_channel() previously returned the blobstore-wide I/O channel from spdk_bs_alloc_io_channel(). Every lvol on an lvstore therefore shared a single spdk_bs_channel per poll-group thread, including its 512-entry spdk_bs_request_set pool (reqs) and its need_cluster_alloc serialization queue. When an esnap-backed lvol (such as a rebuilding Longhorn V2 replica) writes to an unallocated cluster, bs_allocate_and_copy_cluster() holds the active allocation while reading from the remote external snapshot and queues subsequent unallocated-cluster writes on that channel in need_cluster_alloc, including writes to unrelated healthy lvols on the same lvstore. Under heavy write load, nodes rebuilding replicas from each other's snapshots exhaust all 512 request sets in need_cluster_alloc and reject incoming snapshot reads with -ENOMEM, deadlocking I/O across all lvols on the lvstore. Add spdk_blob_alloc_io_channel() by registering each opened spdk_blob as an I/O device when opened and deferring final blob memory release to the spdk_io_device_unregister() completion callback: - Each open blob gets its own spdk_bs_channel per thread with an isolated 512-entry reqs pool and need_cluster_alloc queue, so slow or stalled esnap CoW reads on one lvol cannot block cluster allocations or starve request sets for snapshot reads and other lvols. - Guard blob->io_device_registered with acquire/release atomics and free struct spdk_blob only from blob_io_device_unregister_cpl() once all per-blob channel creations and asynchronous put_io_channel() teardowns across threads have completed. - Each per-blob channel holds a reference to the thread's root blobstore channel (bs_io_channel) and links into root->thread_channels. This preserves spdk_bs_unload() teardown ordering, keeps esnap_channels on the root channel, and allows blob_execute_queued_io() and blob_esnap_destroy_one_channel() to walk all channels on the thread during freeze/unfreeze and parent replacement. - Update spdk_lvol_get_io_channel() and internal lvol copy, inflate, decouple, and checksum paths to allocate per-blob channels, and check for channel allocation failure in vbdev_lvol. Document that the returned channel must be released before the lvol or blob is closed. Add blob and lvol unit tests covering channel isolation under a stalled esnap CoW read, freeze/unfreeze and esnap parent replacement on per-blob channels, asynchronous per-blob channel close/teardown, per-lvol channel allocation, and vbdev_lvol channel creation failure. Related: longhorn/longhorn#14589 Signed-off-by: Derek Su <derek.su@suse.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
d0d5c20 to
d7d05d1
Compare
Which issue(s) this PR fixes:
Issue longhorn/longhorn#14589
What this PR does / why we need it:
Special notes for your reviewer:
Additional documentation or context