Re: [PATCH rdma-next 0/3] Support out of order data placement

Bart Van Assche
2017-06-12T16:29:17Z

On Mon, 2017-06-12 at 16:19 +0000, Parav Pandit wrote:
> Hi Bart,
>
> > -----Original Message-----
> > From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Bart Van Assche
> > Sent: Monday, June 12, 2017 10:28 AM
> > To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org; dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> > Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Idan Burstein <idanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Subject: Re: [PATCH rdma-next 0/3] Support out of order data placement
> >
> > On Mon, 2017-06-12 at 09:49 +0300, Leon Romanovsky wrote:
> > > Out of order data placement capability indicates that if HCA receives
> > > out of order RDMA packets, their data placement can be done at the
> > > desired memory destination given in the packet(s). This is applicable
> > > to RDMA read and write operations.
> >
> > Hello Leon and Parav,
> >
> > Since PCIe writes can be executed out of order, shouldn't that be mentioned
> > in Documentation/infiniband/out_of_order.txt? See also the
> > documentation of the Device Control Register and the Enable Relaxed
> > Ordering bit in the PCIe spec.
>
> There is no change in the way PCIe writes are done with respect to this per QP
> bit. Meaning, if this bit is cleared, PCIe subsystem can still do out of order
> writes depending on relaxed ordering flag.

Hello Parav,

That's why I asked to mention PCIe write reordering in out_of_order.txt. Someone
who is reading that document could be mislead to assume that if the HCA does not
reorder writes that no reordering will occur.

> > Additionally, since not handling out-of-order RDMA writes correctly is an
> > ULP bug and since there are more ULPs that handle out-of-order writes
> > correctly than ULPs that don't handle out-of-order writes correctly, if a new
> > flag is introduced, shouldn't that be a flag to disable out-of-order writes?
>
> Not sure I understood correctly. This bit is not a bug fix for ULPs who don't
> handle out-of-order writes. As I described in Documentation, "Out of order data
> placement capability indicates that if HCA receives out of order RDMA packets,
> their data placement can be done at the desired memory destination given in the
> packet(s). This is applicable to RDMA read and write operations." This flag
> indicates that - in above condition, HCA can do data placement out-of-order.
> Without enabling this flag, when HCA receives out of order packets, it would drop
> them due to PSN sequence error.
>
> So, - to your question - shouldn't that be a flag to disable out-of-order writes?
> By default, its disabled at RDMA level.

I don't think that your last two paragraphs mention anything that had not yet
been mentioned in the four e-mails constituting your patch series. Additionally,
I think my question was clear and unambiguous. So please reread my question.

Thanks,

Bart.