Re: SCSI QLA not working on latest *-mm SN2

Grant Grundler
2004-09-21T19:06:27Z

On Tue, Sep 21, 2004 at 02:09:10PM -0400, Jesse Barnes wrote:
> Grant, you say that I/O writes can't possibly arrive out of order if issued
> from different CPUs on any machines you're aware of?

Well, that's what I believe right now based on my experience.
Re-reading different parts of the PCI local bus spec doesn't give
me the warm fuzzies - especially in regard to retries.

The qla example Jeremy gave earlier wasn't clear on where the
mmio write reordering was taking place. Based on my understanding
(and I've been wrong before) of ia64 .rel/.acq semantics the reordering
didn't happen in the CPU coherency - ie writes are ordered WRT
to each other out to the Mckinley bus. That really only leaves
the chipset suspect - ie anything between Mckinley bus and PCI bus.

And this chipset is already known to violate DMA writes and
MMIO read return ordering rules...which led to read_relaxed().
We've agreed DMA reads bypassing MMIO writes violates the spec
but is "mostly harmless" (I haven't seen a case in real life where
it matters). HP parisc and ia64 platforms also implement
this "optimization" (bug) and I've not seen a problem with it yet.

> If not, it may be that the API will only benefit SGI machines.

Or hurt them.
We haven't talked about the flip side of this workaround much.

Normally, I expect the chipset is responsible for maintaining
order of MMIO writes - though that sounds near impossible on
a large fabric where the spinlock transactions may take a different
path than the IO transactions. But allowing out of order MMIO
write transactions is a big deal if we want high performance
devices that can operate correctly by only consuming MMIO
writes and doing DMA for everything else. Adding the MMIO
reads to enforcing MMIO write ordering will set us all back
a few years in terms of performance.

This behavior (bug?) essentially means drivers that only do MMIO
writes during normal operation (and MMIO reads during set up) will
not operate correctly on Altix (or similar if they exist) boxes.

hth,
grant