Re: MMIO and gcc re-ordering issue

Jeremy Higdon
2008-05-31T07:52:50Z

On Thu, May 29, 2008 at 10:47:18AM -0400, Jes Sorensen wrote:
> Thats not going to solve the problem on Altix. On Altix the issue is
> that there can be multiple paths through the NUMA fabric from cpuX to
> PCI bridge Y.
>
> Consider this uber-cool<tm> ascii art - NR is my abbrevation for NUMA
> router:
>
> ------- -------
> |cpu X| |cpu Y|
> ------- -------
> | \____ ____/ |
> | \/ |
> | ____/\____ |
> | / \ |
> ----- ------
> |NR 1| |NR 2|
> ------ ------
> \ /
> \ /
> -------
> | PCI |
> -------
>
> The problem is that your two writel's, despite being both issued on
> cpu X, due to the spin lock, in your example, can end up with the
> first one going through NR 1 and the second one going through NR 2. If
> there's contention on NR 1, the write going via NR 2 may hit the PCI
> bridge prior to the one going via NR 1.

We don't actually have that problem on the Altix. All writes issued
by CPU X will be ordered with respect to each other. But writes by
CPU X and CPU Y will not be, unless an mmiowb() is done by the
original CPU before the second CPU writes. I.e.

CPU X writel
CPU X writel
CPU X mmiowb

CPU Y writel
...

Note that this implies some sort of locking. Also note that if in
the above, CPU Y did the mmiowb, that would not work.

jeremy