Repository navigation
[Roadmap] Payload Transfer in TransferQueue #173
Description
This roadmap tracks the development of Payload Transfer in TransferQueue. Feedback, discussion, and contributions are welcome.
If you plan to work on an item, please comment below before starting to help coordinate the work and avoid duplication.
Foundation
-
Payload Transfer abstraction and NIXL-UCX backend
Introduce the Payload Transfer abstraction and the initial NIXL-UCX PUT/GET data path, while keeping ZMQ as the control plane and default payload path.
Reliability
- Improve transfer lifecycle management
Safely manage MR, buffers, pending state, and peer metadata across PREPARE, CANCEL, transfer failure, retry, and cleanup paths.
Performance
-
Optimize buffer pooling and memory registration
Reduce buffer allocation and memory registration overhead through better reuse. -
Reduce unnecessary data copies
Remove avoidable copies in the receive path and process data directly from registered buffers where possible. -
Improve payload transfer concurrency
Avoid serializing independent transfers through a single send executor (for example,max_workers=1) when one manager communicates with multiple StorageUnits, and avoid blocking the StorageUnit control worker while waiting for long-running NIXL GET transfers to complete. Preserve transfer lifecycle correctness while allowing concurrent data-plane progress and reducing head-of-line blocking in RL workloads.
Backend Expansion
- Add more Payload Transfer backends
Extend the abstraction to support additional backends such as HIXL.
Validation
- Expand correctness, stability, and performance validation
Cover larger-scale, sustained, and multi-node workloads.
Documentation
- Complete user-facing tutorials and documentation
Add usage guides, configuration examples, and related feature documentation.
Mainline Integration
- Prepare for integration into
main
Move toward mainline integration once the feature is sufficiently stable and validated.
Activity
xupinjie commentedon Sep 18, 2026
It is a clear technical roadmap. May I ask whether anyone is currently working on these three performance areas? I’d be happy to help with the optimization if that would be useful.
Ginray commentedon Sep 22, 2026
Sorry that I didn’t see your reply earlier. We’ve also been working on another project in parallel recently, so I missed your message. We already have some work in progress on our side, but it is still under validation. To avoid duplicated work, we submitted PR [WIP][feat] optimize NIXL-UCX payload transfer for SimpleStorage #176 with our current implementation. We are still evaluating whether the current design is reasonable enough. You likely have much more experience with performance optimization in this area, so if you have time, we’d really appreciate it if you could take a look. If there are no major issues with the overall architecture, you are also welcome to submit PRs directly to continue improving it.
Our follow-up work will mainly focus on larger-scale, sustained, and multi-node workloads, and another colleague will also be involved. Apart from that, we currently do not plan to integrate more NIXL backends or capabilities, such as GDR. I heard that you may be interested in this area. If so, we can sync on what each side plans to work on in advance, so that we can avoid duplicated work as much as possible.
Again, sorry that I didn’t see your reply earlier. I hope this hasn’t caused any trouble with the work you may have already started.