Re: Ordering between PCI config space writes and MMIO reads?

John Partridge
2006-11-01T17:09:13Z

Matthew,

So, if I understand correctly, you are saying because we cannot guarantee
the "flush" a config write even by doing a config read of the same register
(because the PPB can re-order) we have to make sure we block or spin on the
config write completion at the lowest level of the config write ?

Thanks
John

Matthew Wilcox wrote:
> On Wed, Nov 01, 2006 at 10:27:19AM -0600, John Partridge wrote:
>
>>Sorry, but I find this change a bit puzzling. The problem is particular to
>>the PPB on the HCA and not Altix.
>
>
> That's not true; it's more likely on Altix, but it's not unique. *any*
> PCI-PCI bridge can reorder pci config reads and writes. Apparently the
> normal PCI host bridge implementation avoids this problem by blocking
> until the completion comes back. If you put a quad-port tulip card into
> an Altix, you could experience the same problem (but it would be
> massively unlikely. You'd probably have to bring up three interfaces,
> saturate them with traffic, then bring up the fourth to see it. And
> even then it would be rare).
>
>
>>I can't see anywhere that a PCI Config
>>Write
>>is required to block until completion, it is the driver and the HCA ,not the
>>Altix hardware that requires the Config Write to have completed before we
>>leave mthca_reset()
>
>
> There's several places in the PCI midlayer that require the config write
> to have completed before we do a config read. The MWI code relies on
> this to see if the device supports MWI. If it gets out of order, we'll
> think that the device doesn't support MWI when it thinks it's been told
> to use MWI. Data corruption could result.
>
>
>>Changing pci_write_config_xxx() will change the behavior
>>for ALL drivers and the possibility of breaking something else. The fix was
>>very low risk in mthca_reset(), changing the PCI code to fix this is much
>>more onerous.
>
>
> I really don't think so. At worst you'll be changing the timing.

--
John Partridge

Silicon Graphics Inc
Tel: 651-683-3428
Vnet: 233-3428
E-Mail: johnip@sgi.com