ibverbs: wrap CQ event notifications

[perbu]

CompletionChannel gives the fd (AsRawFd) and the unsafe comp_channel() handle, and CompletionQueue::cq() gives the unsafe ibv_cq handle but there's no safe ibv_req_notify_cq / ibv_get_cq_event / ibv_ack_cq_events. So event-driven completion handling requires dropping to rdma-mummy-sys.

The arm → wait-on-fd → get/ack → drain → re-arm loop is the standard low-CPU alternative to busy-polling start_poll, and it's what lets an idle connection park instead of spinning a core.

Suggested API

impl CompletionQueue {                 // trait or per-CQ
    pub fn request_notify(&self, solicited_only: bool) -> Result<(), ...>;
    pub fn ack_events(&self, n: u32);
}
impl CompletionChannel {
    pub fn get_event(&self) -> Result<(), ...>; // ibv_get_cq_event
}

[dragonJACson]

Thanks for raising this. One design question that would help shape the API is how you expect CompletionQueue and CompletionChannel to be associated.

Do you have a 1:1 mapping in mind, where each CQ owns or is paired with its own completion channel? Or do you want to support multiple CQs sharing a single completion channel?

That choice would affect how flexible or complex the API needs to be.

[perbu]

I'd design for the support multiple CQs sharing a single completion channel. I think this aligns well with ibverbs.