Re: [PATCH rdma-next 0/3] Support out of order data placement

Tom Talpey
2017-06-12T21:09:10Z

On 6/12/2017 12:55 PM, Jason Gunthorpe wrote:
> On Mon, Jun 12, 2017 at 04:53:28PM +0000, Parav Pandit wrote:
>>
>>
>>> From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
>>> Sent: Monday, June 12, 2017 11:44 AM
>>> To: Parav Pandit <parav-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>> Cc: Bart Van Assche <Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>; leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org;
>>> dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Idan Burstein
>>> <idanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>> Subject: Re: [PATCH rdma-next 0/3] Support out of order data placement
>>>
>>> On Mon, Jun 12, 2017 at 04:42:27PM +0000, Parav Pandit wrote:
>>>
>>>>> I would suggest at least using the inverted sense like Bart
>>>>> describes in the kernel - every kernel ULP is safe.
>>>
>>>> I don't see a need to use inverted sense in code. I can surely make
>>> documentation more descriptive as Bart suggested.
>>>
>>> If this is 'better' then it should be on as much as possible, and I certianly
>>> don't want to see kernel ULPs query caps and other pointless things when
>>> they already, necessarily, deal with out of order.
>>
>> Sure. Kernel ULPs and any other user ULPs can skip query caps.
>
> No, they can't because only mlx5 accepts the new flag.
>
> This is why inverting the flag in the kernel makes much more sense.

Absolutely. All iWARP adapters support multiple RDMA Writes in flight,
and their placement is not ordered. It's a quirk of the MLX devices,
and in turn their implementation of the IB protocol, that they place
writes in-order. I'm happy to see Mellanox become aware of this, it
limits write performance on networks with any significant latencies.

I agree with Jason, the bit should be 1 by default, if defined as you
propose. Out-of-order is the norm, not the exception, for ULPs.
Honestly, I think you should perhaps consider making it the default
on your devices, and allowing only MLX-aware ULPs to turn it off.

The Windows NDK API has a similar concept, by the way, it flips your
meaning and defaults to 0. Also, it's not settable by the ULP. The
flag isn't used for much, as a result.

NDK_ADAPTER_FLAG_IN_ORDER_DMA_SUPPORTED

https://msdn.microsoft.com/en-us/library/windows/hardware/hh439851(v=vs.85).aspx

Tom.