Re: [PATCH for-next] IB/cma: Introduce rdma_set_min_rnr_timer()

Haakon Bugge
2021-03-31T13:35:03Z

> On 31 Mar 2021, at 15:15, Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> On Wed, Mar 31, 2021 at 12:58:41PM +0000, Haakon Bugge wrote:
>>
>>
>>> On 31 Mar 2021, at 14:00, Jason Gunthorpe <jgg@nvidia.com> wrote:
>>>
>>> On Wed, Mar 31, 2021 at 10:38:02AM +0000, Haakon Bugge wrote:
>>>>
>>>>
>>>>> On 31 Mar 2021, at 01:12, Jason Gunthorpe <jgg@nvidia.com> wrote:
>>>>>
>>>>> On Thu, Mar 25, 2021 at 02:05:47PM +0100, Håkon Bugge wrote:
>>>>>> Introduce the ability for both user-space and kernel ULPs to adjust
>>>>>> the minimum RNR Retry timer. The INIT -> RTR transition executed by
>>>>>> RDMA CM will be used for this adjustment. This avoids an additional
>>>>>> ib_modify_qp() call.
>>>>>
>>>>> Can't userspace override the ibv_modify_qp() call the librdmacm wants
>>>>> to make to do this?
>>>>
>>>> Not sure I understand. The point is, that user-land which intends to
>>>> set said timer, can do so without an additional ibv_modify_qp()
>>>> call. May be I should have added:
>>>
>>> IIRC in userspace the application has the option to call
>>> ibv_modify_qp() so it can just change it before it makes the call?
>>
>> User-space can call ibv_modify_qp, but that call is inherently
>> expensive on some HCA implementations running virtualized.
>
> you are not following.
>
> In rdmacm userspace *always* calls the ibv_modify_qp
>
> rdmacm has a 'helper' mode where it hides the call inside its logic in
> librdmacm.
>
> But, IIRC, a ULP can get some event and do the ibv_modify_qp in its
> logic instead of using the hidden version inside rdma cm. Though that
> may only be possible if you eliminate the entire librdmacm hidden logic
>
> It just seems wrong to send data to the kernel just to have the kernel
> copy that same data out to another system call and never use it at
> all.
>
> Actually I bet you could do this same thing entirely in userspace by
> adjusting rdma_init_qp_attr() to copy the data that would be stored in
> the cm_id.. ??

This will definitely not solve the issue for kernel ULP, e.g., RDS. Further, why do we have rdma_set_option() with option RDMA_OPTION_ID_ACK_TIMEOUT ?

Let me dig into what you're saying.

Thxs, Håkon

>
> Jason