Re: I/O write ordering

[Benjamin Herrenschmidt <benh@kernel.crashing.org>]

On Thu, 2004-09-23 at 01:43, James Bottomley wrote:
> On Wed, 2004-09-22 at 11:28, Benjamin Herrenschmidt wrote:
> >  - consistent memory is either cacheable or non-cacheable depending on
> > the CPU type (some embedded CPUs who don't implement cache coherency
> > will map consistent memory as non-cacheable, most desktop CPUs and all
> > 64 bits CPU don't mind and will map it cacheable).
> 
> Yes, this was what I feared, since streaming memory has exact ownership
> and handoff paradigms for handling it.
> 
> Your problem case is consistent memory access vs readX/writeX then.
> 
> Could you not just make consistent memory always uncacheable ... then
> you'd look identical to everyone else...

Nope :) That would be a big waste, and it's not simple as that would
also prevent us from using large pages on the linear mapping unless
we start playing horrible tricks with where consistent memory is
allocated from etc... we have a fully coherent DMA architecture, we'd
rather use it !

And that wouldn't help vs. spinlocks which are in real RAM
(we have a similar issue with IO leaking potentially out of locks, 
if we don't use some heavy barriers there too).

> The alternative in I/O terms is to implement an ownership model for
> consistent PCI memory itself...which is sort of what I had to do in the
> dma_noncoherent_alloc() API, but to make that more general would be
> quite a bit of work.

The alternative is to have proper barrier abstractions to use between
MMIO read & consistent memory read, and between consistent memory write
& MMIO write ... or get the architecture designers at IBM change the
way our barriers operate in future chips ;) (which is less likely ? :)

Ben.