Opteron 128GB NODMAPSIZE too small?

Opteron 128GB NODMAPSIZE too small?

[Oliver Weihe]

While running SuSE Linux 10.0 (x86_64) with a vanilla 2.6.16.1 on an
8way (8 sockets) Opteron equipped with 128GB (16GB per socket) of memory
I found this in dmesg.

Any guesses to which value I should set NODEMAPSIZE?
Currently it is 0xfff (from 'include/asm-x86_64/mmzone.h')

Scanning NUMA topology in Northbridge 24
Number of nodes 8
Node 0 MemBase 0000000000000000 Limit 0000000422000000
Node 1 MemBase 0000000422000000 Limit 0000000822000000
Node 2 MemBase 0000000822000000 Limit 0000000c22000000
Node 3 MemBase 0000000c22000000 Limit 0000001022000000
Node 4 MemBase 0000001022000000 Limit 0000001422000000
Node 5 MemBase 0000001422000000 Limit 0000001822000000
Node 6 MemBase 0000001822000000 Limit 0000001c22000000
Node 7 MemBase 0000001c22000000 Limit 0000002022000000
NUMA: Using 25 for the hash shift.
Your memory is not aligned you need to rebuild your kernel with a bigger
NODEMAPSIZE shift=25
No NUMA node hash function found. Contact maintainer
No NUMA configuration found
Faking a node at 0000000000000000-0000002022000000
Bootmem setup node 0 0000000000000000-0000002022000000


Regards,
  Oliver Weihe

Re: Opteron 128GB NODMAPSIZE too small?

[Andi Kleen]

On Wednesday 12 April 2006 19:33, Oliver Weihe wrote:
> While running SuSE Linux 10.0 (x86_64) with a vanilla 2.6.16.1 on an
> 8way (8 sockets) Opteron equipped with 128GB (16GB per socket) of memory
> I found this in dmesg.
> 
> Any guesses to which value I should set NODEMAPSIZE?
> Currently it is 0xfff (from 'include/asm-x86_64/mmzone.h')

0x2fff. Or update to a newer kernel - it should
have that problem fixed by finding a better hash shift that works
with a smaller table too.

-Andi

Re: Re: Opteron 128GB NODMAPSIZE too small?

[Oliver Weihe]

Dear Andi,

thx for your quick reply.

> > While running SuSE Linux 10.0 (x86_64) with a vanilla 2.6.16.1 on an
> > 8way (8 sockets) Opteron equipped with 128GB (16GB per socket) of
> > memory
> > I found this in dmesg.
> > 
> > Any guesses to which value I should set NODEMAPSIZE?
> > Currently it is 0xfff (from 'include/asm-x86_64/mmzone.h')
> 
> 0x2fff. Or update to a newer kernel - it should
> have that problem fixed by finding a better hash shift that works
> with a smaller table too.

A newer Kernel (vanilla 2.6.16.5) doens't solve the problem (hash-shift
is still 25bit and with the default 12bit (0xfff) in NODEMAPSIZE there
is still one bit missing to adress the whole memory).
Anyway, increasing NODEMAPSIZE solves the problem. I wanted to ask you
before doing this because I'm not familar with the code and can't imagin
if this has any sideeffects.

Regards,
  Oliver Weihe