Re: SCSI QLA not working on latest *-mm SN2

[Grant Grundler <grundler@parisc-linux.org>]

Jesse Barnes wrote:
...
> Btw Andrew (Vasquez), there's a small doc I put together that should describe 
> when you have to worry about PCI posting.  It's in the tree:  
> Documentation/io_ordering.txt.  If it's incomplete or confusing, just let me 
> know and I'll update it.

Jesse,
Both. incomplete and confusing.
"concrete example of a hypothetical driver" wasn't my first warning
this document needed work. :^)

I've hacked up the 2.6.9 version and even what I did still needs more work.
Have time to correct my mistakes and answer the questions I ask?

I'd be happy to review it again after you've done another round on it.

[]'s should all go away - used those to mark editorial notes.

hth,
grant


--------------------- cut here ------------------

Weakly Ordered Memory Mapped IO
-------------------------------

SGI Altix chipset implements weakly ordered Memory-Mapped I/O writes.
On this platform, driver writers are responsible for ensuring I/O writes
to memory-mapped addresses arrive in the order intended.

Like for PCI write posting problems, this is done by reading
a 'safe' device or bridge register, causing the I/O chipset to
flush pending writes to the device before any reads are issued.
A driver would issue the "safe" read immediately prior to the exit
of a critical section of code protected by spinlocks.  This would
ensure subsequent writes to I/O space arrived only after all prior
writes (much like a memory barrier op, mb(), only with respect to I/O).

Note: MMIO reads are expensive! Don't add MMIO reads after *every* MMIO
      write unless the device programming model absolutely requires it.


An example from a hypothetical device driver might help:

        ...
CPU A:  spin_lock_irqsave(&dev_lock, flags)
CPU A:  val = readl(my_status);
CPU A:  ...
CPU A:  writel(newval, ring_ptr);
CPU A:  spin_unlock_irqrestore(&dev_lock, flags)
        ...
CPU B:  spin_lock_irqsave(&dev_lock, flags)
CPU B:  val = readl(my_status);
CPU B:  ...
CPU B:  writel(newval2, ring_ptr);
CPU B:  spin_unlock_irqrestore(&dev_lock, flags)
        ...

In the case above, the device may receive newval2 before it receives newval,
which could cause problems.

[ Is this example broken or am I just staying up too late?
  The example is doing a readl() in the second critical section.
  Shouldn't that enforce the write ordering?
]

Fixing it is easy enough though:

        ...
CPU A:  spin_lock_irqsave(&dev_lock, flags)
CPU A:  val = readl(my_status);
CPU A:  ...
CPU A:  writel(newval, ring_ptr);
CPU A:  (void)readl(safe_register); /* maybe a config register? */
CPU A:  spin_unlock_irqrestore(&dev_lock, flags)
        ...
CPU B:  spin_lock_irqsave(&dev_lock, flags)
CPU B:  val = readl(my_status);
CPU B:  ...
CPU B:  writel(newval2, ring_ptr);
CPU B:  (void)readl(safe_register); /* maybe a config register? */
CPU B:  spin_unlock_irqrestore(&dev_lock, flags)

The reads from safe_register will cause the I/O chipset to flush any
pending writes before actually posting the read to the chipset, preventing
possible data corruption.

[ How about interactions with:
  o read_relaxed()?
  o DMA?
  o IO Port space reads?
  o IO Port space writes?
]