NUMA processor numbering

NUMA processor numbering

[Stephan von Krawczynski]

Hello all,

I have a box with this kind of processor (0-31) and 128 GB RAM:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
stepping        : 7
microcode       : 0x70d
cpu MHz         : 2486.000
cache size      : 20480 KB
physical id     : 0
siblings        : 16
core id         : 0
cpu cores       : 8
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc aperfmperf eagerfp u pni pclmulqdq dtes64 monitor ds_cpl vmx smx
est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt
tsc_deadline_tim er aes xsave avx lahf_lm ida arat xsaveopt pln pts dtherm
tpr_shadow vnmi flexpriority ept vpid 
bogomips        : 4400.12
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:


Now, numactl --hardware shows this:

available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
node 0 size: 64581 MB
node 0 free: 12676 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31
node 1 size: 64637 MB
node 1 free: 10660 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

Physically these are two processors with 8 Cores and 8 HT each.
Does the above output mean that the cores are numbered right across the two
physical cpus? Does this mean one has to pin processes to 0,2,4,... to stay in
"short distance" to node 0 RAM?
If so, it would be a lot better to have them numbered 0-15 and 16-31 for pinning.
Is there a way to achieve this?
Please cc me when answering.

-- 
Regards,
Stephan

Re: NUMA processor numbering

[Henrique de Moraes Holschuh]

On Thu, 03 Oct 2013, Stephan von Krawczynski wrote:
> Does the above output mean that the cores are numbered right across the two
> physical cpus? Does this mean one has to pin processes to 0,2,4,... to stay in
> "short distance" to node 0 RAM?

...

> If so, it would be a lot better to have them numbered 0-15 and 16-31 for pinning.
> Is there a way to achieve this?

Yes, use hwloc to get the pinning masks for whatever property you want (e.g.
all threads in a given core, all threads in a given node, all threads that
share a given L3 cache...).

http://www.open-mpi.org/projects/hwloc/

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: NUMA processor numbering

[Stephan von Krawczynski]

On Thu, 3 Oct 2013 07:22:55 -0300
Henrique de Moraes Holschuh <hmh@hmh.eng.br> wrote:

> On Thu, 03 Oct 2013, Stephan von Krawczynski wrote:
> > Does the above output mean that the cores are numbered right across the two
> > physical cpus? Does this mean one has to pin processes to 0,2,4,... to stay in
> > "short distance" to node 0 RAM?
> 
> ...
> 
> > If so, it would be a lot better to have them numbered 0-15 and 16-31 for pinning.
> > Is there a way to achieve this?
> 
> Yes, use hwloc to get the pinning masks for whatever property you want (e.g.
> all threads in a given core, all threads in a given node, all threads that
> share a given L3 cache...).
> 
> http://www.open-mpi.org/projects/hwloc/

Ok, let me re-phrase the question a bit.
Is it really possible what you see here:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
stepping        : 7
microcode       : 0x70d
cpu MHz         : 2002.000
cache size      : 20480 KB
physical id     : 0
siblings        : 16
core id         : 0
cpu cores       : 8
apicid          : 0
initial apicid  : 0
[...]

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
stepping        : 7
microcode       : 0x70d
cpu MHz         : 1518.000
cache size      : 20480 KB
physical id     : 1
siblings        : 16
core id         : 0
cpu cores       : 8
apicid          : 32
initial apicid  : 32
[...]

These are the first two in the cpu list. If you look at that they are both on
core id 0, but have different physical ids. Up to now I thought that processor
1 is the HT of core id 0. But with a different physical id this would mean
that they are different NUMA nodes, right? How can that be? Someone from Intel
with a hint?

-- 
Regards,
Stephan

Re: NUMA processor numbering

[Brice Goglin]

Le 03/10/2013 12:46, Stephan von Krawczynski a écrit :
> Ok, let me re-phrase the question a bit.
> Is it really possible what you see here:
>
> processor       : 0
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 45
> model name      : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
> stepping        : 7
> microcode       : 0x70d
> cpu MHz         : 2002.000
> cache size      : 20480 KB
> physical id     : 0
> siblings        : 16
> core id         : 0
> cpu cores       : 8
> apicid          : 0
> initial apicid  : 0
> [...]
>
> processor       : 1
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 45
> model name      : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
> stepping        : 7
> microcode       : 0x70d
> cpu MHz         : 1518.000
> cache size      : 20480 KB
> physical id     : 1
> siblings        : 16
> core id         : 0
> cpu cores       : 8
> apicid          : 32
> initial apicid  : 32
> [...]
>
> These are the first two in the cpu list. If you look at that they are both on
> core id 0, but have different physical ids. Up to now I thought that processor
> 1 is the HT of core id 0. But with a different physical id this would mean
> that they are different NUMA nodes, right? How can that be? Someone from Intel
> with a hint?

Such "unexpected" numberings are very common. The BIOS is free to number
processors in many different orders, including NUMA first, core first,
HT first, etc.

Having the two hyperthreads of the first core physically numbered 0 and
1 doesn't seem very common on current Intel platforms. Most Xeon E5
machines I've seen put the second hyperthreads of all cores at the end
of range. But there's no guarantee about this.

Use hwloc just like Henrique said, it will take care of virtually
reordering objects in a logical manner.

Brice