Avoid Vec's allocation for `PostRecvGuard` with new lowlevel API

[IgorErin]

Hello! Thank you so much for the project!

I am interested in the possibility to avoid allocations when creating work requests in PostRecvGuard.

The fact is that PostRecvGuard now contains Vec's that are dynamically filled in when constructing new work requests. I would like to avoid this, as it may contribute to memory fragmentation and may incur additional overhead on the data path.

I suggest adding an additional low-level API to work with already allocated elements, for example:

pub unsafe fn post_lowlevel(qp: NonNull<ibv_qp>, wrs: &mut [ibv_recv_wr], sges: &mut [ibv_sge]) -> Result<(), PostRecvError>

It could be used in existing code and offered to the user so that he can use it with preallocated arrays on the stack, for example.

Thank you in advance!

[dragonJACson]

Hello! Thanks for reporting this.

The same allocation issue also affects PostSendGuard in the basic QP API, so we would like to find a unified solution for both PostSendGuard and PostRecvGuard. We still need some time to agree on the design of the allocation-free interface.

In the meantime, you could use rdma-mummy-sys as a temporary workaround. However, that would require working with the raw bindings (the ibv_post_send and ibv_post_recv) directly, which somewhat defeats the purpose of this project. :(

[heroin-moose]

Actually same goes for inline data or connection parameters where copying also happens. Perhaps it's worth to introduce design constraint that only permits copying/allocations when no other way possible?

[dragonJACson]

Sorry, I’ve been rather busy lately, so I haven’t made much progress on the project.

Yes, I think that would be a reasonable design constraint: copies and allocations should only be used when they are unavoidable.

With the ibv_wr_* interface, ibv_wr_set_inline_data copies the inline data during the setter call, allowing the source buffer to be reused immediately. In contrast, when using ibv_post_send, the provider does not copy the inline data referenced by the SGEs into its WQEs until ibv_post_send is invoked by post. Therefore, avoiding an intermediate copy would require the caller-provided buffers to remain valid and unchanged until that point.

We may eventually provide a separate, unsafe low-level interface based on ibv_post_send as a higher-performance path, rather than trying to fit it into the unified interface.

[heroin-moose]

The thing is, inline data for protocols like NVMe starts from 512 or 4096 (disk block size) and copying is a HUGE performance hit here.

[heroin-moose]

We may eventually provide a separate, unsafe low-level interface based on ibv_post_send as a higher-performance path, rather than trying to fit it into the unified interface.

Actually it's quite possible to create an interface that accepts &'a buf: [u8] and makes sure no one touches that until send is complete.

[dragonJACson]

We may eventually provide a separate, unsafe low-level interface based on ibv_post_send as a higher-performance path, rather than trying to fit it into the unified interface.

Actually it's quite possible to create an interface that accepts &'a buf: [u8] and makes sure no one touches that until send is complete.

We would prefer to expose this as an additional API rather than modify the existing one. ExtendedPostSendGuard does not require such a lifetime constraint because ibv_wr_set_inline_data copies the data immediately, so imposing the constraint there would be unnecessarily restrictive.

Also, IBV_SEND_INLINE inherently requires the provider to copy the payload into the WQE. For example, both mlx5 and RXE perform a memcpy. BasicPostSendGuard currently introduces an additional copy, which a borrow-based API could eliminate, but the provider-side copy would remain.

For inline payloads of 512 or 4096 bytes, wouldn’t that unavoidable provider-side memcpy still be a significant performance cost?

[heroin-moose]

Hmm... yeah, you probably right, I was thinking kernel API.