Re: [PATCH RFC 1/1] Add support for ZONE_DEVICE IO memory with struct pages.

Jason Gunthorpe
2016-03-17T16:11:21Z

On Thu, Mar 17, 2016 at 05:18:11PM +0200, Haggai Eran wrote:
> On 3/14/2016 11:57 PM, Jason Gunthorpe wrote:
> > The other issue is that the fencing mechanism RDMA uses to create
> > ordering with system memory is not good enough to fence peer-peer
> > transactions in the general case. It is only possibly good enough if
> > all the transactions run through the root complex.
>
> Are you sure this is a problem? I'm not sure it is clear in the PCIe
> specs, but I thought that for transactions that are not relaxed-ordered
> and don't use ID-based ordering, a PCIe switch must prevent reads and
> writes from passing writes.

Yes, that is right, and that is good enough if the PCI-E fabric is a
simple acyclic configuration (ie the common case).

There are fringe cases that are more complex, and maybe the correct
reading of the spec is to setup routing to avoid optimal paths, but it
certainly is possible to configure switches in a way that could not
guarentee global ordering.

Jason