FYI: 2.6.16-smp: DMA memory inbalance for NUMA?

FYI: 2.6.16-smp: DMA memory inbalance for NUMA?

[Ulrich Windl]

Hi,

my apologies if this is an issue already solved:
I have no idea what these messages exactly say, but for reasons of symmetry I 
think there's something wrong:
For a Sun Fire X4100 with two Dual_Core Operon processors, the kernel (SLES10 
kernel (x86_64, 2.6.16.21-0.15-smp)) says during boot:

<6>SRAT: PXM 0 -> APIC 0 -> Node 0
<6>SRAT: PXM 0 -> APIC 1 -> Node 0
<6>SRAT: PXM 1 -> APIC 2 -> Node 1
<6>SRAT: PXM 1 -> APIC 3 -> Node 1
<6>SRAT: Node 0 PXM 0 100000-f4000000
<6>SRAT: Node 1 PXM 1 20c000000-40c000000
<6>SRAT: Node 0 PXM 0 100000-20c000000
<6>SRAT: Node 0 PXM 0 0-20c000000

[[ Note: "Node 0" is mentioned three times, but "Node 1" is only mentioned once. 
If this is intended to be some address assignments, they look quite odd to me. 
Does the following really mean that only one node can do DMA (the other has zero 
DMA pages)? ]]

<6>ACPI: SLIT table looks invalid. Not used.
<7>NUMA: Using 26 for the hash shift.
<6>Bootmem setup node 0 0000000000000000-000000020c000000
<6>Bootmem setup node 1 000000020c000000-000000040c000000
<7>On node 0 totalpages: 2066745
<7>  DMA zone: 2993 pages, LIFO batch:0
<7>  DMA32 zone: 981032 pages, LIFO batch:31
<7>  Normal zone: 1082720 pages, LIFO batch:31
<7>  HighMem zone: 0 pages, LIFO batch:0
<7>On node 1 totalpages: 2068480
<7>  DMA zone: 0 pages, LIFO batch:0
<7>  DMA32 zone: 0 pages, LIFO batch:0
<7>  Normal zone: 2068480 pages, LIFO batch:31
<7>  HighMem zone: 0 pages, LIFO batch:0

As usual, please CC: replies to my address as I'm not subscribed here.

Regards,
Ulrich

Re: FYI: 2.6.16-smp: DMA memory inbalance for NUMA?

[Robert Hancock]

Ulrich Windl wrote:
> Hi,
> 
> my apologies if this is an issue already solved:
> I have no idea what these messages exactly say, but for reasons of symmetry I 
> think there's something wrong:
> For a Sun Fire X4100 with two Dual_Core Operon processors, the kernel (SLES10 
> kernel (x86_64, 2.6.16.21-0.15-smp)) says during boot:
> 
> <6>SRAT: PXM 0 -> APIC 0 -> Node 0
> <6>SRAT: PXM 0 -> APIC 1 -> Node 0
> <6>SRAT: PXM 1 -> APIC 2 -> Node 1
> <6>SRAT: PXM 1 -> APIC 3 -> Node 1
> <6>SRAT: Node 0 PXM 0 100000-f4000000
> <6>SRAT: Node 1 PXM 1 20c000000-40c000000
> <6>SRAT: Node 0 PXM 0 100000-20c000000
> <6>SRAT: Node 0 PXM 0 0-20c000000
> 
> [[ Note: "Node 0" is mentioned three times, but "Node 1" is only mentioned once. 
> If this is intended to be some address assignments, they look quite odd to me. 
> Does the following really mean that only one node can do DMA (the other has zero 
> DMA pages)? ]]

DMA is not done only to the DMA zones. The DMA zone is for devices that 
can only address 24-bit addresses (ISA/LPC devices). The DMA32 zone is 
for devices that can only address 4GB of memory.

I suppose you could say there is a bit of an imbalance between nodes but 
hopefully these are both (especially ZONE_DMA) rarely used in any sane 
setup so it wouldn't make much difference. Since these zones are based 
on bus-visible physical addresses I'm not sure if this could be fixed in 
any case.

> 
> <6>ACPI: SLIT table looks invalid. Not used.
> <7>NUMA: Using 26 for the hash shift.
> <6>Bootmem setup node 0 0000000000000000-000000020c000000
> <6>Bootmem setup node 1 000000020c000000-000000040c000000
> <7>On node 0 totalpages: 2066745
> <7>  DMA zone: 2993 pages, LIFO batch:0
> <7>  DMA32 zone: 981032 pages, LIFO batch:31
> <7>  Normal zone: 1082720 pages, LIFO batch:31
> <7>  HighMem zone: 0 pages, LIFO batch:0
> <7>On node 1 totalpages: 2068480
> <7>  DMA zone: 0 pages, LIFO batch:0
> <7>  DMA32 zone: 0 pages, LIFO batch:0
> <7>  Normal zone: 2068480 pages, LIFO batch:31
> <7>  HighMem zone: 0 pages, LIFO batch:0

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/