Re: [RFC/PARTIAL PATCH 0/3] dma: passing "attributes" to dma_map_* routines

[Grant Grundler <grundler@parisc-linux.org>]

On Wed, Jan 09, 2008 at 03:30:58PM -0600, James Bottomley wrote:
...
> > Also, all of this is quite separate from the PCIe "loose ordering"
> > stuff.  In fact, it's quite conceivable that SGI could hook up a PCIe
> > 3.0 host bridge to the Altix NUMA interconnect, so that you could have
> > a PCI bus that supported loose ordering and also a system interconnect
> > that allowed a different type of reordering too.
> 
> Actually, no ... there'd just be an even weirder attribute, I suspect.
> The attributes will be set per *transaction* not per bus.  A transaction
> is an operation (DMA, PIO, config space, etc) from the actual bus to the
> CPU.  It doesn't matter how many physical bus segments this traverses
> and whether they're different bus types; all that matters for the
> attributes are the characteristics that are CPU visible.

James is right. Setting the PCI-e "Relaxed Ordering" bit in config space
allows the device to set the "Relaxed Order" in specific DMA transactions.
Upstream bridges may choose to ignore the bit or follow well defined
transaction flushing/ordering rules if they do implement "Relaxed Ordering".
Key thing here is the device decides when to set RO bit in each transaction.
This is completely different than the re-ordering which occurs in Altix
NUMA bus for _all_ (default config) transactions.

Given SGI/Altix only cares about a limited number of drivers and only a
subset of those support RDMA (or something like it), would it be reasonable
to add the new API to the RDMA code instead of the generic DMA API?

Please don't shoot me for suggesting that...just thinking "out loud" here.

thanks,
grant