RE: [PATCH rdma-next 0/3] Support out of order data placement

Parav Pandit
2017-06-12T20:14:04Z

Hi Denny, Sean,

As Sean explained, relaxed ordering/out-of-order packets, allows making efficient use of network resources.
When transmitter and receiver is enabled to do so, as I described in overview section of Documentation, it helps
(a) to avoid retransmission - improves network utilization
(b) reduces latency due to timers not kicking in.

More comments to Sean's question below.

> -----Original Message-----
> From: Hefty, Sean [mailto:sean.hefty@intel.com]
> Sent: Monday, June 12, 2017 2:19 PM
> To: Dalessandro, Dennis <dennis.dalessandro@intel.com>; Parav Pandit
> <parav@mellanox.com>; Steve Wise <swise@opengridcomputing.com>;
> 'Leon Romanovsky' <leon@kernel.org>; 'Doug Ledford'
> <dledford@redhat.com>
> Cc: linux-rdma@vger.kernel.org; Idan Burstein <idanb@mellanox.com>
> Subject: RE: [PATCH rdma-next 0/3] Support out of order data placement
>
> > I don't understand why the application has to care about this at all.
> > If
> > the HW wants to place things out of order. Ok go for it. Applications
> > should be depending on the correct mechanism to know when it's OK to
> > look at the data. Why do they care what order it arrived in?
> >
> > The only reason to care is if the applications wants to do something
> > that is not compliant and look at parts of the data early. Perhaps you
> > can explain *how* an application may benefit from out-of-order
> > placement and likewise, why it wouldn't want to benefit.
>
> Relaxed ordering can improve performance by allowing packets to take
> multiple paths from the source to destination. FWIW, the ofiwg actually
> discussed ordering at length, and libfabric exposes several ordering related
> attributes.
>
> The application usually needs to know what sort of ordering the network is
> providing for correct operation.
Yes. This flag defines it how RW data can done in relaxed order manner.

> For example, can sends be re-ordered by
> the network, such that they may be received in the opposite order that they
> were sent?

No. Only rdma reads and writes. Table 79 of IB spec still holds true.
In internal patch review I had the table, but since there was no change, I just removed it.
I think it's worth to mention in the out_of_order.txt that it still holds true. I will add this line.

I would also document which compliant statements are not applicable when such flag is set.
How about having responder side table, similar to table 79 in Documentation/out_of_order.txt
That makes it more readable for users.

> Can rdma writes be processed out of order, such that the target
> data ends up with some sort of interleaving of the written data?
Yes.

> Are there size limits to ordering constraints?
No.

>
> It's hard to capture this level of detail with a single flag. There's ordering of
> data within a single message, as well as ordering of data between messages.
Good documentation should describe what a flag does. Flag name itself has RW prefix in it.
Are you suggesting there should be two flags - one for within message, one for across message?
If so, shouldn't we add those flag when such different implementation arise.
I cannot define something that cannot be implemented or tested at this point. This is across the messages.

> I don't know how this proposal works with explicit pathing that IB defines,
> or path failover.

It doesn't change with path failover or explicit path to my knowledge, but I will confirm if there is any change in v1.

>
> - Sean