Re: [RFC] Relaxed PIO read vs. DMA write ordering

[jbarnes@sgi.com (Jesse Barnes)]

On Wed, Jan 07, 2004 at 03:21:42PM -0700, Grant Grundler wrote:
> > How about always setting the bit in readb() and having a readb_ordered()
> > which doesn't set the bit in the transaction?
> 
> I was under the impression the driver can't control RO for
> each transaction though. The PCI-X device controls which
> transactions set RO "signal" in the PCI-X command on read-return.
> The Read-Return is a seperate transaction from the Read-Request.

My understanding is that you need both.  And if we used the
pci_enable_relaxed() routine, we'd have to add a check to readX() for it
so that we don't accidentally set the RO bit in the transaction when the
command word has it clear.

> If anyone has data that specific devices are "smart" and set/clear
> RO appropriately, it would be safe to enable RO for them.

I don't know of any that do it automatically...

> On HP ZX1, the "Allow Relaxed Ordering" is only implemented for outbound
> DMA/PIO Writes *while they pass through the ZX1 chip*. Ie RO bit settings
> don't explicitly apply since we aren't talking about PCI-X bus transactions
> even though the system chipset needs to honor PCI-X rules.

So this wouldn't be helpful for your chipset then.

> > That way, drivers which
> > call pci_set_relaxed() have the responsibility to verify they're not
> > relying on these semantics and use readb_ordered() in any places that
> > they are.
> 
> if new variants of readb() are ok, then yours sounds better.
> 
> But I wasn't too keen on introducing readb variants to solve what
> looks like a DMA flushing problem. I've come to the conclusion
> that systems which implement (and enable) RO for inbound DMA are
> effectively not coherent. The data the CPU expects to be visible is not.

Ahh... that's a bit of a stretch of the definition of non-coherence I
think, but it might be close enough to use the sync semantics.

> DMA-mapping.txt already has support (pci_dma_sync_xx() or
> pci_dma_unmap_xx()) to deal with common forms off non-coherence and
> syncronize caches for streaming mappings but not for consistent
> mappings.  DMA-ABI.txt (2.6 only) has a method to handle non-coherent

Right, that's another option--adding a pci_sync_consistent() call.

> systems and I have to reread/study it to see if the provided interface
> is sufficient for the case of relaxed ordering.  Jesse, have you
> looked at this already?

All of them are pretty easy enough to do...  so I see our options as one
of the following:

  1) add pcix_enable_relaxed() and read_relaxed() (read() would always be
     ordered)
  2) add pcix_enable_relaxed() and read_ordered() (read() would be
     relaxed after the pcix_enable_relaxed() call)
  3) add pcix_enable_relaxed() and pci_sync_consistent() (read() would
     be relaxed after the pcix_enable_relaxed() call)

Thanks,
Jesse