Re: [PATCH for-next] RDMA/efa: Add CQ with external memory support

[Jason Gunthorpe <jgg@nvidia.com>]

On Wed, May 21, 2025 at 06:19:51PM +0300, Margolin, Michael wrote:
> 
> On 5/20/2025 12:16 PM, Leon Romanovsky wrote:
> > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> > 
> > 
> > 
> > On Sun, May 18, 2025 at 11:56:56AM +0300, Margolin, Michael wrote:
> > > On 5/18/2025 9:42 AM, Leon Romanovsky wrote:
> > > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> > > > 
> > > > 
> > > > 
> > > > On Thu, May 15, 2025 at 02:50:40PM +0000, Michael Margolin wrote:
> > > > > Add an option to create CQ using external memory instead of allocating
> > > > > in the driver. The memory can be passed from userspace by dmabuf fd and
> > > > > an offset.
> > > > EFA is unique here. This patch is missing description of why it is
> > > > needed, and why existing solutions if any exist, can't be used.
> > > > 
> > > > Thanks
> > > I probably should have explained more, the purpose is creating CQs that
> > > reside in GPU HBM enabling low latency polling directly by the GPU. EFA
> > > isn't unique in receiving pre-allocated memory from userspace, the extension
> > > here is the use of dmabuf for that purpose as a general mechanism that
> > > allows using memory independent of its source. I will add more info in the
> > > commit message.
> > I think that this functionality is worth to have as general verb and not DV.
> > mlx5 has something similar and now EFA needs it too.
> > 
> > Let's wait for Jason's response before rushing to implement it.
> > 
> > Thanks
> 
> Jason, any thoughts on this?
> 
> We can probably add optional attributes to create CQ command, but most of
> the handling will remain vendor specific. I'm not convinced it's beneficial
> enough.

I don't know how a general verb would work, CQs in non-dv are polled
in software in userspace and the userspace will have trouble reaching
into a dmabuf. Plus the entire point of this is usually to write the
polling code in a GPU language and run it on a GPU processor.

Meaning I think all users of this will want to use a DV interface from
verbs.

At that point, is it worth adding more common verbs support?

Though it is becoming a bit messy that drivers are all open coding
creating normal or dmabuf umems. That part might be worth generalizing
some more, and then doing QP as well.

Jason