Re: FYI: 2.6.16-smp: DMA memory inbalance for NUMA?

[Robert Hancock <hancockr@shaw.ca>]

Ulrich Windl wrote:
> Hi,
> 
> my apologies if this is an issue already solved:
> I have no idea what these messages exactly say, but for reasons of symmetry I 
> think there's something wrong:
> For a Sun Fire X4100 with two Dual_Core Operon processors, the kernel (SLES10 
> kernel (x86_64, 2.6.16.21-0.15-smp)) says during boot:
> 
> <6>SRAT: PXM 0 -> APIC 0 -> Node 0
> <6>SRAT: PXM 0 -> APIC 1 -> Node 0
> <6>SRAT: PXM 1 -> APIC 2 -> Node 1
> <6>SRAT: PXM 1 -> APIC 3 -> Node 1
> <6>SRAT: Node 0 PXM 0 100000-f4000000
> <6>SRAT: Node 1 PXM 1 20c000000-40c000000
> <6>SRAT: Node 0 PXM 0 100000-20c000000
> <6>SRAT: Node 0 PXM 0 0-20c000000
> 
> [[ Note: "Node 0" is mentioned three times, but "Node 1" is only mentioned once. 
> If this is intended to be some address assignments, they look quite odd to me. 
> Does the following really mean that only one node can do DMA (the other has zero 
> DMA pages)? ]]

DMA is not done only to the DMA zones. The DMA zone is for devices that 
can only address 24-bit addresses (ISA/LPC devices). The DMA32 zone is 
for devices that can only address 4GB of memory.

I suppose you could say there is a bit of an imbalance between nodes but 
hopefully these are both (especially ZONE_DMA) rarely used in any sane 
setup so it wouldn't make much difference. Since these zones are based 
on bus-visible physical addresses I'm not sure if this could be fixed in 
any case.

> 
> <6>ACPI: SLIT table looks invalid. Not used.
> <7>NUMA: Using 26 for the hash shift.
> <6>Bootmem setup node 0 0000000000000000-000000020c000000
> <6>Bootmem setup node 1 000000020c000000-000000040c000000
> <7>On node 0 totalpages: 2066745
> <7>  DMA zone: 2993 pages, LIFO batch:0
> <7>  DMA32 zone: 981032 pages, LIFO batch:31
> <7>  Normal zone: 1082720 pages, LIFO batch:31
> <7>  HighMem zone: 0 pages, LIFO batch:0
> <7>On node 1 totalpages: 2068480
> <7>  DMA zone: 0 pages, LIFO batch:0
> <7>  DMA32 zone: 0 pages, LIFO batch:0
> <7>  Normal zone: 2068480 pages, LIFO batch:31
> <7>  HighMem zone: 0 pages, LIFO batch:0

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/