Re: [PATCH for-next] RDMA/efa: Add CQ with external memory support

["Margolin, Michael" <mrgolin@amazon.com>]

On 6/11/2025 10:47 PM, Jason Gunthorpe wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> On Mon, Jun 09, 2025 at 06:03:32PM +0300, Margolin, Michael wrote:
>> On 5/26/2025 7:17 PM, Margolin, Michael wrote:
>>> On 5/26/2025 7:08 PM, Jason Gunthorpe wrote:
>>>> On Mon, May 26, 2025 at 06:45:59PM +0300, Margolin, Michael wrote:
>>>>
>>>>> Are you suggesting turning mlx5dv_devx_umem_reg into a common
>>>>> verb including
>>>>> the kernel part or some kind of rdma-core level abstraction for passing
>>>>> dmabuf+offset+length / address+length to a create CQ/QP function?
>>>> I think Leon was, but I'm not sure that is so worthwhile.
>>>>
>>>> I was thinking more of having the ioctls for things like QP/CQ/MR
>>>> accept a more standard common set of attributes to describe the buffer
>>>> memory and then making it simpler for the driver to get a umem from
>>>> those common attributes.
>>>>
>>>> But EFA is alread sort of different because it normally uses a kernel
>>>> allocated buffer, right?
>>>>
>>>> Jason
>>> Yes, EFA is an example for a driver that doesn't need this on the
>>> "standard" flow.
>>>
>>> Michael
>>>
>> How can we move forward with this patch? It's possible to add additional
>> attributes to the common create CQ ioctl and use it for EFA direct verbs but
>> it won't be easy to move existing drivers to use it.
> Then it becomes hard to make the other drivers reject those new common
> attributes :\
>
> Maybe what you have here is the best, but it really does seem
> unsatisfying from a kernel POV..
>
> Jason

Apologies for a delayed response, I want to make sure we are aligned on 
the solution.

I'm going to add a common option in libibverbs for providers to pass 
buffer info for create CQ ioctl and when the new ioctl attributes are 
set to the kernel, handle it in core code by creating a umem object and 
storing it in CQ context for drivers use.

Does this make sense?


Michael