Re: [RFC/PARTIAL PATCH 0/3] dma: passing "attributes" to dma_map_* routines

[Roland Dreier <rdreier@cisco.com>]

 > What it wants to do is set strict ordering for the bus ... well, that is
 > an attribute in the PCIe standard (it just happens to be the default one
 > for a standard bus, whereas relaxed is the default for altix).  However,
 > set bus attribute strict would be a simple no-op for a standard bus (and
 > set attribute relaxed a no-op for altix).

I don't think that this can work.  As Arthur and I have said several
times, the Altix issue is within the system NUMA interconnect --
nothing to do with the PCI bus.  And as I understand things, to get
good performance, we have to allow reordering within the NUMA
interconnect except that certain transactions need to flush all
earlier writes.  So this attribute needs to be set per-mapping, not
per-bus.

If you have a cleaner way to specify this attribute that Arthur's
idea, then it would be very useful to share that.  If we were starting
from scratch, then probably adding another "flags" parameter to the
DMA mapping functions would be the way to go, but given the number of
calls to dma_map_xxx all around, changing the signature now doesn't
look very appealing.

The reason this hasn't been an issue until now is that almost all
drivers work correctly if the Altix code just sets the "flush" bit for
memory allocated via the consistent/coherent allocators.  However, if
we want the device to write to userspace memory, this doesn't work
(and mapping coherent memory allocated in the kernel into userspace is
a mess on other platforms, because it unnecessarily consumes lowmem
and/or kernel address space).

Also, all of this is quite separate from the PCIe "loose ordering"
stuff.  In fact, it's quite conceivable that SGI could hook up a PCIe
3.0 host bridge to the Altix NUMA interconnect, so that you could have
a PCI bus that supported loose ordering and also a system interconnect
that allowed a different type of reordering too.

 - R.