Re: 48GB NUMA-Q boots, with major IO-APIC hassles

[Anton Blanchard <anton@samba.org>]

> PCI config cycles must be qualified by segment here just to get the
> right address, so there's definitely a requirement for stuff. One
> "advantage" of NUMA-Q (if it can be called that) is that firmware/BIOS
> and hardware pushes a bus number mangling scheme that is more or less
> mandatory, so things can at largely implicitly taken care of with the
> bus number and the firmware's mapping of the mangled bus numbers to the
> cross-quad portio area. But this scheme does not have cooperation from
> PCI-PCI bridges, where NUMA-Q mangling scheme -noncompliant physical
> bus ID's are kept in the bus number registers.

Can you store the segment in pci_bus->sysdata and then use that in
your config read/write macros? What do you mean by the mangling?
Does each host bridge have a separate segment identifier? If so why
would the PCI-PCI bridges below it need to have incorrect IDs?

> I have no idea what to do about these; I just sort of hope and pray the
> backward-compatibility constraints won't hurt me. At the level of things
> exported to userspace my main concern is largely that things like disk
> arrays will actually be accessible as raw devices or mountable as fs's,
> cooperation with userspace drivers and accurate reporting is kind of
> secondary to me aside from satisfying backward-compatibility concerns
> from Pee Cee -centric sides of things.

Its possible customers will run X on their server, thats when getting
things like /proc/bus/pci/* right becomes important :) In fact on
sparc64 and ppc64 X should work with domains fairly easily because
they are already use /proc/bus/pci/* to mmap PCI BARs.

I think X should always use this method, reading PCI BARs and mmap'ing
/dev/mem is just awful.

Anton