RE: [PATCH rdma-next 0/3] Support out of order data placement

Parav Pandit
2017-06-23T16:03:38Z

Hi Tom, Jason,

I will get back on updated v1 documentation and answers to below questions once I get some more details internally.

Parav

> -----Original Message-----
> From: Tom Talpey [mailto:tom@talpey.com]
> Sent: Monday, June 12, 2017 8:30 PM
> To: Parav Pandit <parav@mellanox.com>; Jason Gunthorpe
> <jgunthorpe@obsidianresearch.com>
> Cc: Bart Van Assche <Bart.VanAssche@sandisk.com>; leon@kernel.org;
> dledford@redhat.com; linux-rdma@vger.kernel.org; Idan Burstein
> <idanb@mellanox.com>
> Subject: Re: [PATCH rdma-next 0/3] Support out of order data placement
>
> On 6/12/2017 8:36 PM, Parav Pandit wrote:
> >> -----Original Message-----
> >> From: Tom Talpey [mailto:tom@talpey.com]
> >> Sent: Monday, June 12, 2017 7:12 PM
> >> To: Parav Pandit <parav@mellanox.com>; Jason Gunthorpe
> >> <jgunthorpe@obsidianresearch.com>
> >> Cc: Bart Van Assche <Bart.VanAssche@sandisk.com>; leon@kernel.org;
> >> dledford@redhat.com; linux-rdma@vger.kernel.org; Idan Burstein
> >> <idanb@mellanox.com>
> >> Subject: Re: [PATCH rdma-next 0/3] Support out of order data
> >> placement
> >>
> >> On 6/12/2017 7:59 PM, Parav Pandit wrote:
> >>>> -----Original Message-----
> >>>> From: Tom Talpey [mailto:tom@talpey.com]
> >>>> Sent: Monday, June 12, 2017 6:44 PM
> >>>> To: Parav Pandit <parav@mellanox.com>; Jason Gunthorpe
> >>>> <jgunthorpe@obsidianresearch.com>
> >>>> Cc: Bart Van Assche <Bart.VanAssche@sandisk.com>; leon@kernel.org;
> >>>> dledford@redhat.com; linux-rdma@vger.kernel.org; Idan Burstein
> >>>> <idanb@mellanox.com>
> >>>> Subject: Re: [PATCH rdma-next 0/3] Support out of order data
> >>>> placement
> >>>>
> >>>> On 6/12/2017 6:54 PM, Parav Pandit wrote:
> >>>>> Hi Tom,
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Tom Talpey [mailto:tom@talpey.com]
> >>>>>> Sent: Monday, June 12, 2017 5:20 PM
> >>>>>> To: Parav Pandit <parav@mellanox.com>; Jason Gunthorpe
> >>>>>> <jgunthorpe@obsidianresearch.com>
> >>>>>> Cc: Bart Van Assche <Bart.VanAssche@sandisk.com>;
> >> leon@kernel.org;
> >>>>>> dledford@redhat.com; linux-rdma@vger.kernel.org; Idan Burstein
> >>>>>> <idanb@mellanox.com>
> >>>>>> Subject: Re: [PATCH rdma-next 0/3] Support out of order data
> >>>>>> placement
> >>>>>>
> >>>>>> On 6/12/2017 5:32 PM, Parav Pandit wrote:
> >>>>>>> Hi Tom,
> >>>>>> ...
> >>>>>>>>
> >>>>>>>> I agree with Jason, the bit should be 1 by default, if defined
> >>>>>>>> as you
> >>>>>> propose.
> >>>>>>>> Out-of-order is the norm, not the exception, for ULPs.
> >>>>>>>> Honestly, I think you should perhaps consider making it the
> >>>>>>>> default on your devices, and allowing only MLX-aware ULPs to
> >>>>>>>> turn
> >> it off.
> >>>>>>>>
> >>>>>>>
> >>>>>>> There can be cases in deployment where responder has support for
> >>>>>> receiving out-of-order, but requester doesn't.
> >>>>>>
> >>>>>> Yuck! So this needs to be negotiated end-to-end, and by the upper
> >> layer?
> >>>>>> Talk about barriers to adoption, and opportunities for disaster.
> >>>>>>
> >>>>> As Jason confirmed that all Linux kernel consumers are coded to be
> >>>>> compliant to o9-20 requirement, So I think kernel based rdma-cm
> >>>> consumers can be transparently enabled end-to-end without ULP's
> >>>> involvement with rdma_accept() and rdma_connect().
> >>>>
> >>>> I have two thoughts here.
> >>>>
> >>>> 1) You seem to assume all consumers are Linux, and do not need to
> >>>> negotiate. This is a dangerous assumption.
> >>> Certainly not. I didn't assume that. I just gave one example that
> >>> known
> >> consumers can be done without modifying the ULP.
> >>> Explained further in 3rd question.
> >>> Even other consumers can work with this solution.
> >>> For example Linux rdmacm based client and Other OS based server.
> >>> Client is ooo capable.
> >>> Server is ooo not capable.
> >>> Once you follow below rdmacm based sequence, it will be clear how
> >>> this
> >> will works.
> >>
> >> Oh, so there's a MAD protocol change under the hood.
> > No. There is no change under the hood.
> > Your question was how can we avoid ULP change and still they can benefit
> of this feature?
> > So I said rdmacm based Linux kernel consumers that we know of comply to
> o9-20, can take the benefit once rdmacm is extended as below example.
> >
> >> Well, that's a wider
> >> question. And I still don't understand how existing,
> >> non-strict-requiring protocols can take advantage of this.
> >> Nor how this works for non-Mellanox, non-IB/RoCE implementations.
> >
> > Device capability indicates that which device supports this. Explained in
> Documentation/out_of_order.txt usage section.
> > So whichever vendor supports it, whichever protocol supports it, can set
> this optional device capability.
> >
> >>
> >> Again, I'd be a lot less concerned if non-strict were the default,
> >> and strict mode was negotiated. It's all just so upside-down.
> >
> > In IB spec, in-order delivery is default.
>
> I don't agree. Requests are sent in-order, and the responder processes them
> in-order, but the bytes thenselves are not guaranteed to appear in-order.
> Additionally, if retries occur, this is most definitely not the case.
>
> Section 9.5 Transaction Ordering, I believe, covers these requirements. Can
> you tell me where I misunderstand them?
> In fact, c9-28 explicitly warns:
>
> • An application shall not depend upon the order of data writes to
> memory within a message. For example, if an application sets up
> data buffers that overlap, for separate data segments within a
> message, it is not guaranteed that the last sent data will always
> overwrite the earlier.
>
> My guess is that this bit overrides the MLX behavior of never pipelining RDMA
> Write requests, allowing more packets to be queued at the responder and
> making better use of the network. This is not at all prohibited by the spec, nor
> is it unexpected by properly-coded upper layers, which all the kernel
> consumers are.
>
> I have one other question on the Documentation out-of-order.txt.
> It states the fence bit can be used to force ordering on a non-strict
> connection. But fence doesn't apply to RDMA Write?
> It only applies to operations which produce a reply, such as RDMA Read or
> Atomic. Have you changed the semantic?
>
> Tom.
>
>
>
> So can you suggest how can we change default IB behavior without breaking
> anything?
> > Adding optional attribute seems the right way that ensures compatibility.
> >>
> >> Tom.
> >>
> >>>> 2) I assume that there is some performance benefit to toggling this
> >>>> setting to non-strict. So, how do existing consumers get this
> >>>> advantage, especially since they don't need strict semantics?
> >>>> Bearing in mind that they do have to negotiate this end-to-end,
> >>>> meaning they
> >> require a protocol extension.
> >>> I don't have completely transparent upstream solution for existing
> >> consumers yet.
> >>>>
> >>>> Actually. I have a third thought. Since this is an attribute to qp
> >>>> creation, performed even before establishing a connection, how does
> >>>> the upper layer know when to set it?
> >>> This is not at QP creation time. I have described in
> >> Documentation/out_of_order.txt in usage section 3.
> >>> This is at QP state transition from INIT to RTR.
> >>> Here is the flow. It's just not coded enough for posting patches.
> >>>
> >>> 1. When rdmacm active side creates the QP, It is INIT state.
> >>> 2. Send MAD_Req msg (indicating ooo_requested=1) 3. When rdmacm
> >>> passive side receives the message, it looks up device_cap attribute
> >>> and
> >> matches it against ooo_requested flag.
> >>> 4. when device supports it, MAD_Rsp msg sets ooo_enabled=1, if it
> >>> doesn't support it, ooo_enabled=0 5. rdmacm passive side creates the
> >>> QP
> >> and moves to RTR state (with QP ooo enabled bit set).
> >>> 6. active side receives the message and puts the QP to RTR, RTS
> >>> state
> >> based on received bit setting from passive side.
> >>>
> >>> Flow is no different than how rest of the connection specific
> >>> parameters
> >> are shared such as IRD/ORD, PSN, timeouts, mtu etc.
> >>>
> >>>
> >>>
> >>>>
> >>>> Tom.
> >>> N r y b X ǧv ^ )޺{.n + { ٚ {ay \x1dʇڙ ,j f h z \x1e w
> >>
> >> j:+v w j m zZ+ ݢj" !tml=
> >>>
> > N r y b X ǧv ^ )޺{.n + { ٚ {ay \x1dʇڙ ,j f h z \x1e w
>
> j:+v w j m zZ+ ݢj" !tml=
> >