Re: [PATCH rdma-next 0/3] Support out of order data placement

Tom Talpey
2017-06-12T23:44:12Z

On 6/12/2017 6:54 PM, Parav Pandit wrote:
> Hi Tom,
>
>> -----Original Message-----
>> From: Tom Talpey [mailto:tom-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org]
>> Sent: Monday, June 12, 2017 5:20 PM
>> To: Parav Pandit <parav-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>; Jason Gunthorpe
>> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
>> Cc: Bart Van Assche <Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>; leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org;
>> dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Idan Burstein
>> <idanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>> Subject: Re: [PATCH rdma-next 0/3] Support out of order data placement
>>
>> On 6/12/2017 5:32 PM, Parav Pandit wrote:
>>> Hi Tom,
>> ...
>>>>
>>>> I agree with Jason, the bit should be 1 by default, if defined as you
>> propose.
>>>> Out-of-order is the norm, not the exception, for ULPs.
>>>> Honestly, I think you should perhaps consider making it the default
>>>> on your devices, and allowing only MLX-aware ULPs to turn it off.
>>>>
>>>
>>> There can be cases in deployment where responder has support for
>> receiving out-of-order, but requester doesn't.
>>
>> Yuck! So this needs to be negotiated end-to-end, and by the upper layer?
>> Talk about barriers to adoption, and opportunities for disaster.
>>
> As Jason confirmed that all Linux kernel consumers are coded to be compliant to o9-20 requirement,
> So I think kernel based rdma-cm consumers can be transparently enabled end-to-end without ULP's involvement with rdma_accept() and rdma_connect().

I have two thoughts here.

1) You seem to assume all consumers are Linux, and do not
need to negotiate. This is a dangerous assumption.

2) I assume that there is some performance benefit to toggling
this setting to non-strict. So, how do existing consumers get
this advantage, especially since they don't need strict
semantics? Bearing in mind that they do have to negotiate
this end-to-end, meaning they require a protocol extension.

Actually. I have a third thought. Since this is an attribute
to qp creation, performed even before establishing a connection,
how does the upper layer know when to set it?

Tom.