NUMA problem with AMD G34 system

NUMA problem with AMD G34 system

[Brian Marcotte]

I'm having a problem with NUMA on an AMD G34 system. Xen seems to think
that out of the 16 total cores, there are 12 on node0 and 4 on node3.

The box is:

    SuperMicro H8DGU (dual socket G34)
    Pair of AMD 6128 CPUs (8 core each)
    32 GB of RAM.
    BIOS has been upgraded three times - no help.

    xen-unstable from Feb 12 (same problem on 3.4.x).

I'm including the output from the "u" command below to show Xen's idea
of the CPU->node mapping. Complete boot log is here:

    http://www.panix.com/~marcotte/g34/boot.txt

Any help would be appreciated.

Thanks.

- Brian

(XEN) Xen version 4.1.0-rc5-pre (root@nyc.access.net) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) Sat Feb 12 18:53:49 EST 2011
(XEN) Command line: dom0_mem=600M com1=57600,8n1 console=com1 no-mce loglvl=debug

(XEN) 'u' pressed -> dumping numa info (now-0x2:5A61B881)
(XEN) idx0 -> NODE0 start->0 size->4325376
(XEN) phys_to_nid(0000000000001000) -> 0 should be 0
(XEN) idx3 -> NODE3 start->4325376 size->4194304
(XEN) phys_to_nid(0000000420001000) -> 3 should be 3
(XEN) CPU0 -> NODE0
(XEN) CPU1 -> NODE0
(XEN) CPU2 -> NODE0
(XEN) CPU3 -> NODE0
(XEN) CPU4 -> NODE0
(XEN) CPU5 -> NODE0
(XEN) CPU6 -> NODE0
(XEN) CPU7 -> NODE0
(XEN) CPU8 -> NODE3
(XEN) CPU9 -> NODE3
(XEN) CPU10 -> NODE3
(XEN) CPU11 -> NODE3
(XEN) CPU12 -> NODE0
(XEN) CPU13 -> NODE0
(XEN) CPU14 -> NODE0
(XEN) CPU15 -> NODE0
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 153600):
(XEN)     Node 0: 132232
(XEN)     Node 3: 21368

Re: NUMA problem with AMD G34 system

[Andre Przywara]

Brian Marcotte wrote:
> I'm having a problem with NUMA on an AMD G34 system. Xen seems to think
> that out of the 16 total cores, there are 12 on node0 and 4 on node3.
> 
> The box is:
> 
>     SuperMicro H8DGU (dual socket G34)
>     Pair of AMD 6128 CPUs (8 core each)
>     32 GB of RAM.
>     BIOS has been upgraded three times - no help.
> 
>     xen-unstable from Feb 12 (same problem on 3.4.x).
> 
> I'm including the output from the "u" command below to show Xen's idea
> of the CPU->node mapping. Complete boot log is here:
> 
>     http://www.panix.com/~marcotte/g34/boot.txt
> 
> Any help would be appreciated.

Have you checked that you populated the DIMM slots properly?
The SRAT CPU table seems sane, but the memory table seems to say that 
there is no memory on node 1 and 3, thus Xen will simply put those CPUs 
to node 0 as it cannot cope with memory-less nodes.
If I understand the manual right, you first have to populate all the 
blue slots, then the black ones.

Hope that helps,
Andre.


> 
> Thanks.
> 
> - Brian
> 
> (XEN) Xen version 4.1.0-rc5-pre (root@nyc.access.net) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) Sat Feb 12 18:53:49 EST 2011
> (XEN) Command line: dom0_mem=600M com1=57600,8n1 console=com1 no-mce loglvl=debug
> 
> (XEN) 'u' pressed -> dumping numa info (now-0x2:5A61B881)
> (XEN) idx0 -> NODE0 start->0 size->4325376
> (XEN) phys_to_nid(0000000000001000) -> 0 should be 0
> (XEN) idx3 -> NODE3 start->4325376 size->4194304
> (XEN) phys_to_nid(0000000420001000) -> 3 should be 3
> (XEN) CPU0 -> NODE0
> (XEN) CPU1 -> NODE0
> (XEN) CPU2 -> NODE0
> (XEN) CPU3 -> NODE0
> (XEN) CPU4 -> NODE0
> (XEN) CPU5 -> NODE0
> (XEN) CPU6 -> NODE0
> (XEN) CPU7 -> NODE0
> (XEN) CPU8 -> NODE3
> (XEN) CPU9 -> NODE3
> (XEN) CPU10 -> NODE3
> (XEN) CPU11 -> NODE3
> (XEN) CPU12 -> NODE0
> (XEN) CPU13 -> NODE0
> (XEN) CPU14 -> NODE0
> (XEN) CPU15 -> NODE0
> (XEN) Memory location of each domain:
> (XEN) Domain 0 (total: 153600):
> (XEN)     Node 0: 132232
> (XEN)     Node 3: 21368
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 448-3567-12

Re: NUMA problem with AMD G34 system

[Brian Marcotte]

> The SRAT CPU table seems sane, but the memory table seems to say
> that there is no memory on node 1 and 3, thus Xen will simply put
> those CPUs to node 0 as it cannot cope with memory-less nodes.

Ah. I think that's what was causing my confusion. Since the cpu-to-node
mapping was unbalanced and Xen wasn't complaining, I thought for sure
this was a problem with Xen and newer CPUs and I'd need to help with
debugging the problem.

In this case, I suppose I'll remove a CPU and rearrange the memory
until it's time to add more memory.

Thanks.

--
- Brian

Re: NUMA problem with AMD G34 system

[Andre Przywara]

Brian Marcotte wrote:
>> The SRAT CPU table seems sane, but the memory table seems to say
>> that there is no memory on node 1 and 3, thus Xen will simply put
>> those CPUs to node 0 as it cannot cope with memory-less nodes.
> 
> Ah. I think that's what was causing my confusion. Since the cpu-to-node
> mapping was unbalanced and Xen wasn't complaining, I thought for sure
> this was a problem with Xen and newer CPUs and I'd need to help with
> debugging the problem.
I prepared Xen to handle this situation (nodes without memory) before 
the M-C launch, before that it simply crashed. I agree that it is not 
obvious that you need to populate at least 4 DIMMs per CPU.

> In this case, I suppose I'll remove a CPU and rearrange the memory
> until it's time to add more memory.
Do you have only 4 DIMMs? If yes, you could do so if you can cope with 
half of the CPU power.
If you don't want to sacrifice the second CPU, you could go ahead and 
use the system with this kind of strange NUMA setup. Depending on your 
workload you will benefit more from this setup, as the impact of remote 
memory on most application is not that huge (compared to less cores, at 
least).
But if you have 8 DIMMs, you should keep the second CPU and put all 
DIMMs in the blue slots.

Regards,
Andre.

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany

Re: NUMA problem with AMD G34 system

[Brian Marcotte]

> I prepared Xen to handle this situation (nodes without memory)
> before the M-C launch, before that it simply crashed. I agree that
> it is not obvious that you need to populate at least 4 DIMMs per
> CPU.

I suppose that Xen could arrange the cpu to node mapping such that
the memoryless CPUs were assigned to the "nearest" node making it so
you could benefit from the shared L3 cache.

I don't know that it's worth anyone's time though. Perhaps a warning
would suffice?

> Do you have only 4 DIMMs?

Yes, but I don't expect that to last too long. We find that over the
lifetime of a machine, we max out the RAM slots long before we run out
of CPU. The second CPU is mostly just so we can (eventually) fill all
the RAM slots. It's not a problem to run with one CPU for while. I may
also just pin the domains to particular CPU cores rather than pull the
CPU.

Thanks for your help.

--
- Brian

Re: NUMA problem with AMD G34 system

[Andre Przywara]

Brian Marcotte wrote:
>> I prepared Xen to handle this situation (nodes without memory)
>> before the M-C launch, before that it simply crashed. I agree that
>> it is not obvious that you need to populate at least 4 DIMMs per
>> CPU.
> 
> I suppose that Xen could arrange the cpu to node mapping such that
> the memoryless CPUs were assigned to the "nearest" node making it so
> you could benefit from the shared L3 cache.
> 
> I don't know that it's worth anyone's time though. Perhaps a warning
> would suffice?
I also considered more possible action back then, but found that it is 
probably not worth the effort, as this kind of configuration was 
considered kind of bogus. But a warning message sounds like a good idea.

> 
>> Do you have only 4 DIMMs?
> 
> Yes, but I don't expect that to last too long. We find that over the
> lifetime of a machine, we max out the RAM slots long before we run out
> of CPU. The second CPU is mostly just so we can (eventually) fill all
> the RAM slots. It's not a problem to run with one CPU for while. I may
> also just pin the domains to particular CPU cores rather than pull the
> CPU.
Yes, just put all the DIMMs in the blue slots of the first CPU and boot 
Xen with maxcpus=8. This should give you the best results without much 
fiddling with the hardware.

Regards,
Andre.

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany