Re: Creating new RDMA driver for habanalabs

[Jason Gunthorpe <jgg@ziepe.ca>]

On Wed, Jul 06, 2022 at 11:59:14AM +0300, Oded Gabbay wrote:

> Due to that, we would want to put all the ports under a single struct ib_device,
> as you said it yourself in your original email a year ago.

Yes

> The major constraints are:
> 
> 1. Support only RDMA WRITE operation. We do not support READ, SEND or RECV.
>     This means that many existing open source tests in rdma-core are not
>     compatible. e.g. rc_pingpong.c will not work. I guess we will need to
>     implement different tests and submit them ? Do you have a
> different idea/suggestion ?

I would suggest following what EFA did and just using your own unique
QP with dv accessors to create it. A QP that can only do RDMA WRITE is
not IBA compliant and shouldn't be created by a standard verbs call.
 
> 2. As you mentioned in the original email, we support only a single PD.
>    I don't see any major implication regarding this constraint but please
>    correct me if you think otherwise.

Seems fine

> 3. MR limitation on the rkey that is received from the remote connection
>    during connection creation. The limitation is that our h/w extracts
>    the rkey from the QP h/w context and not from the WQE when sending packets.
>    This means that we may associate only a single remote MR per QP.

It seems OK in the context above where you have your own QP type and
obviouly your specila RDMA WRITE poster will not take in an rkey as
any argument.

>    Do you see any issue here with these two limitations ? One thing we noted is
>    that we need to somehow configure the rkey in our h/w QP context, while today
>    the API doesn't allow it.

When you add your own dv qp create function it will take in the
required rkey during qp creation.
 
>    These limitations are not relevant to a deployment where all the NICs are
>    Gaudi NICs, because we can use a single rkey for all MRs.

Er, that is weird, did you mean to say you have only one MR per PD and
that it always has a fixed value?
 
> 4. We do not support all the flags in the reg_mr API. e.g. we don't
>    support IBV_ACCESS_LOCAL_WRITE. I'm not sure what the
>    implication is here.

It is OK, since you can't issue a local operation WQE anyhow you can
just ignore the flag.

> 5. Our h/w contains several accelerations we would like to utilize.
>    e.g. we have a h/w mechanism for accelerating collective operations
>    on multiple RDMA NICs. These accelerations will require either extensions
>    to current APIs, or some dedicated APIs. For example, one of the
>    accelerations requires that the user will create a QP with the same
>    index on all the Gaudi NICs.

Use your DV interface to do these kinds of things

Thanks,
Jason