another change breaks rxe

another change breaks rxe

[Bob Pearson]

Jason,

commit 628c02bf38aa42c09c3dde61284ba348290e6650
Author: Jason Gunthorpe <jgg@ziepe.ca>
Date:   Sat Oct 3 20:20:10 2020 -0300

    RDMA: Remove uverbs cmds from drivers that don't use them
    
    Allowing userspace to invoke these commands is probably going to crash
    these drivers as they are not tested and not expecting to use them on a
    user object.
    
    For example pvrdma touches cq->ring_state which is not initialized for
    user QPs.
    
    These commands are effected:
    
    - IB_USER_VERBS_CMD_REQ_NOTIFY_CQ is ibv_cmd_req_notify_cq() in
      rdma-core, only hfi1, ipath and rxe calls it.
    
    - IB_USER_VERBS_CMD_POLL_CQ is ibv_cmd_poll_cq() in rdma-core, only
      ipath and hfi1 calls it.
    
    - IB_USER_VERBS_CMD_POST_SEND/RECV is ibv_cmd_post_send/recv() in
      rdma-core, only ipath and hfi1 call them.

breaks rxe because it does use IB_USER_VERBS_CMD_POST_SEND. rxe posts wqes to a work queue in shared memory and then calls ibv_post_send with zero wqes as a doorbell to the kernel which uses this as a hint to go read the shared memory.

Bob

Re: another change breaks rxe

[Jason Gunthorpe]

On Fri, Oct 30, 2020 at 12:41:12AM -0500, Bob Pearson wrote:

> breaks rxe because it does use IB_USER_VERBS_CMD_POST_SEND. rxe
> posts wqes to a work queue in shared memory and then calls
> ibv_post_send with zero wqes as a doorbell to the kernel which uses
> this as a hint to go read the shared memory.

Gah, siw and rxe are open coding this stuff (wrongly!) in rdma-core

This would be a lot better to use an eventfd

Jason

Re: Re: another change breaks rxe

[Bernard Metzler]

-----"Jason Gunthorpe" <jgg@nvidia.com> wrote: -----

>To: "Bob Pearson" <rpearsonhpe@gmail.com>
>From: "Jason Gunthorpe" <jgg@nvidia.com>
>Date: 10/30/2020 12:47PM
>Cc: "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
>Subject: [EXTERNAL] Re: another change breaks rxe
>
>On Fri, Oct 30, 2020 at 12:41:12AM -0500, Bob Pearson wrote:
>
>> breaks rxe because it does use IB_USER_VERBS_CMD_POST_SEND. rxe
>> posts wqes to a work queue in shared memory and then calls
>> ibv_post_send with zero wqes as a doorbell to the kernel which uses
>> this as a hint to go read the shared memory.
>
>Gah, siw and rxe are open coding this stuff (wrongly!) in rdma-core
>
>This would be a lot better to use an eventfd
>
>Jason
>
Yes, siw does it the same.

I see the hfi and ipath drivers similarly using ibv_cmd_post_send
to push the wqe's (while not having them in shared mem). Do they
brake as well?
Do we have an example of decent eventfd usage where I could learn
from how to use it? Though I think having another fd open per context
might become a scaling issue.

Thanks and best regards,
Bernard.

Re: Re: another change breaks rxe

[Jason Gunthorpe]

On Fri, Oct 30, 2020 at 12:50:01PM +0000, Bernard Metzler wrote:

> I see the hfi and ipath drivers similarly using ibv_cmd_post_send
> to push the wqe's (while not having them in shared mem). Do they
> brake as well?
> Do we have an example of decent eventfd usage where I could learn
> from how to use it?

You'd add a uAPI accepting a eventfd and then call a sequence like:

        init_waitqueue_func_entry(.., my_func);
        init_poll_funcptr(..);

        events = vfs_poll(evfile, ..);

And my_func will be called when the user does write() on the event fd.

drivers/vfio/virqfd.c has the simplest version of this

It could be some core code to do this setup with a new driver callback
'user_doorbell_trigger' or something like that.

However, I see at closer inspection that rxe and siw both want the QPN
to be passed in, so this doesn't look like it would work - an eventfd
per QP is too much overhead.

But calling post_send, even with a 0 length, is really slow.

Jason