Re: [PATCH for-next] RDMA/efa: Add CQ with external memory support

["Margolin, Michael" <mrgolin@amazon.com>]

On 5/25/2025 8:52 PM, Jason Gunthorpe wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> On Wed, May 21, 2025 at 06:19:51PM +0300, Margolin, Michael wrote:
>> On 5/20/2025 12:16 PM, Leon Romanovsky wrote:
>>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>>
>>>
>>>
>>> On Sun, May 18, 2025 at 11:56:56AM +0300, Margolin, Michael wrote:
>>>> On 5/18/2025 9:42 AM, Leon Romanovsky wrote:
>>>>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, May 15, 2025 at 02:50:40PM +0000, Michael Margolin wrote:
>>>>>> Add an option to create CQ using external memory instead of allocating
>>>>>> in the driver. The memory can be passed from userspace by dmabuf fd and
>>>>>> an offset.
>>>>> EFA is unique here. This patch is missing description of why it is
>>>>> needed, and why existing solutions if any exist, can't be used.
>>>>>
>>>>> Thanks
>>>> I probably should have explained more, the purpose is creating CQs that
>>>> reside in GPU HBM enabling low latency polling directly by the GPU. EFA
>>>> isn't unique in receiving pre-allocated memory from userspace, the extension
>>>> here is the use of dmabuf for that purpose as a general mechanism that
>>>> allows using memory independent of its source. I will add more info in the
>>>> commit message.
>>> I think that this functionality is worth to have as general verb and not DV.
>>> mlx5 has something similar and now EFA needs it too.
>>>
>>> Let's wait for Jason's response before rushing to implement it.
>>>
>>> Thanks
>> Jason, any thoughts on this?
>>
>> We can probably add optional attributes to create CQ command, but most of
>> the handling will remain vendor specific. I'm not convinced it's beneficial
>> enough.
> I don't know how a general verb would work, CQs in non-dv are polled
> in software in userspace and the userspace will have trouble reaching
> into a dmabuf. Plus the entire point of this is usually to write the
> polling code in a GPU language and run it on a GPU processor.
>
> Meaning I think all users of this will want to use a DV interface from
> verbs.
>
> At that point, is it worth adding more common verbs support?
>
> Though it is becoming a bit messy that drivers are all open coding
> creating normal or dmabuf umems. That part might be worth generalizing
> some more, and then doing QP as well.
>
> Jason

Are you suggesting turning mlx5dv_devx_umem_reg into a common verb 
including the kernel part or some kind of rdma-core level abstraction 
for passing dmabuf+offset+length / address+length to a create CQ/QP 
function?

Michael