KVM and NUMA

[Ralf Spenneberg <software@opensource-security.de>]

Hi,

I just had a chance to play with KVM on Ubuntu 10.04 LTS on some new HP
360 g6 with Nehalem processors. I have a feeling that KVM and NUMA on
these machines do not play well together. 

Doing some benchmarks I got bizarre numbers. Sometimes the VMs were
performing fine and some times the performance was very bad! Apparently
KVM does not recognize the NUMA-architecture and places memory and
process randomly and therefore often on different numa cells.

First a couple of specs of the machine:
Two Nehalem sockets with E5520, Hyperthreading turned off, 4 cores per
socket, all in all 8 processors.

# cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
stepping        : 5
...


Linux recognizes the NUMA-architecture:
# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6
node 0 size: 12277 MB
node 0 free: 9183 MB
node 1 cpus: 1 3 5 7
node 1 size: 12287 MB
node 1 free: 8533 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

So I have got two cells with 4 cores each.

Virsh does not recognize the topology:
# virsh capabilities
<capabilities>
  <host>
    <cpu>
      <arch>x86_64</arch>
      <model>core2duo</model>
      <topology sockets='2' cores='4' threads='1'/>
      <feature name='lahf_lm'/>
..

I guess this is the fact, because QEMU does not recognize the
NUMA-Architecture (QEMU-Monitor):
(qemu) info numa
0 nodes


This is done on an Ubuntu 10.04 LTS:
Linux lxkvm01 2.6.32-23-server #37-Ubuntu SMP Fri Jun 11 09:11:11 UTC
2010 x86_64 GNU/Linux

The package used was:
qemu-kvm 0.12.3+noroms-0ubuntu9.2

When doing benchmarks numastat shows a lot of misses when the
performance is bad. 
# numastat
                           node0           node1
numa_hit                15527158        17015505
numa_miss                7032982         3512950
numa_foreign             3512950         7032982
interleave_hit              8078            8264
local_node              15525187        17006655
other_node               7034953         3521800

So apparently KVM does not utilize the NUMA-architecture. Did I do
something wrong. Is KVM missing a patch? Do I need to activate something
in KVM to recognize the NUMA-Architecture? 
I tried the newest Kernel Module 2.6.32.15 and Qemu-kvm 0.12.4 without a
change.

Any hints?

Kind regards,

Ralf