Re: [PATCH] RDMA/efa: Add rdma write capability to device caps

["Nachum, Yonatan" <ynachum@amazon.com>]

>>>>>>>>
>>>>>>>>       access_flags &= ~IB_ACCESS_OPTIONAL;
>>>>>>>>       if (access_flags & ~supp_access_flags) {
>>>>>>>> diff --git a/include/uapi/rdma/efa-abi.h b/include/uapi/rdma/efa-abi.h
>>>>>>>> index 74406b4817ce..d94c32f28804 100644
>>>>>>>> --- a/include/uapi/rdma/efa-abi.h
>>>>>>>> +++ b/include/uapi/rdma/efa-abi.h
>>>>>>>> @@ -121,6 +121,7 @@ enum {
>>>>>>>>       EFA_QUERY_DEVICE_CAPS_CQ_NOTIFICATIONS = 1 << 2,
>>>>>>>>       EFA_QUERY_DEVICE_CAPS_CQ_WITH_SGID     = 1 << 3,
>>>>>>>>       EFA_QUERY_DEVICE_CAPS_DATA_POLLING_128 = 1 << 4,
>>>>>>>> +     EFA_QUERY_DEVICE_CAPS_RDMA_WRITE = 1 << 5,
>>>>>>>
>>>>>>> Why do you need special device capability while all rdma-core users
>>>>>>> set IBV_ACCESS_REMOTE_WRITE anyway without relying on anything from
>>>>>>> providers?
>>>>>>>
>>>>>>> Thanks
>>>>>>
>>>>>> We need to query the device because not every device supprort the same RDMA capabilities. Upper layers in the SW stack needs this supported flags to indicate which flows they can use. In addition this is identical to the existing RDMA read support in our code.
>>>>>
>>>>> Nice, but it doesn't answer my question. Please pay attention to the
>>>>> second part of my question "while all rdma-core ....".
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>>> There are rdma-core users that doesn’t fail on unsupported features but fallback to supported ones. One example is Libfabric EFA provider that emulates RDMA write by read if device write isn’t supported but there are other similar examples. Correct way doing this in user code is by querying rdma-core for device supported capabilities, then selecting a suitable work flow. This is why existing and the additional capability bits are required.
>>>
>>> AFAIK, RDMA write is different here as fallback is performed in the kernel and
>>> not in the rdma-core provider. So why should EFA be different here?
>>>
>>> BTW, Please fix your mailer to break lines, so we will be able to reply
>>> on specific sentence with less effort.
>>>
>>> Thanks
>>
>> Can you please elaborate more on the fallback performed in the kernel?
>> What kind of fallback being performed? Is it in create MR/QP?
>> Does the fallback happens when providing unsupported write capability
>> and to what it fallback to?
> 
> OK, looked again, "Fallback" was in my imagination, sorry about that.
> 
> But my main question is continued to be, how other vendors which support
> RDMA write work without capability?
> 
> Thanks

Vendors that always support RDMA write don’t need a query for this capability.
Some EFA devices don’t support write capability so we provide the ability to
query the device to know if write is supported.
It is like mlx5 support query capabilities through direct verb.

Thanks