From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ralf Spenneberg Subject: KVM and NUMA Date: Thu, 15 Jul 2010 19:10:35 +0200 Message-ID: <1279213835.21655.77.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-15" Content-Transfer-Encoding: 7bit To: kvm@vger.kernel.org Return-path: Received: from mail2.spenneberg.net ([87.106.54.221]:33251 "EHLO mail2.spenneberg.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933080Ab0GORVQ (ORCPT ); Thu, 15 Jul 2010 13:21:16 -0400 Received: from [192.168.255.105] (p4FC456EC.dip.t-dialin.net [79.196.86.236]) (Authenticated sender: ralf@mail2.spenneberg.net) by mail2.spenneberg.net (Postfix) with ESMTPA id 62AAE40C146 for ; Thu, 15 Jul 2010 17:10:36 +0000 (UTC) Sender: kvm-owner@vger.kernel.org List-ID: Hi, I just had a chance to play with KVM on Ubuntu 10.04 LTS on some new HP 360 g6 with Nehalem processors. I have a feeling that KVM and NUMA on these machines do not play well together. Doing some benchmarks I got bizarre numbers. Sometimes the VMs were performing fine and some times the performance was very bad! Apparently KVM does not recognize the NUMA-architecture and places memory and process randomly and therefore often on different numa cells. First a couple of specs of the machine: Two Nehalem sockets with E5520, Hyperthreading turned off, 4 cores per socket, all in all 8 processors. # cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU E5520 @ 2.27GHz stepping : 5 ... Linux recognizes the NUMA-architecture: # numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 2 4 6 node 0 size: 12277 MB node 0 free: 9183 MB node 1 cpus: 1 3 5 7 node 1 size: 12287 MB node 1 free: 8533 MB node distances: node 0 1 0: 10 20 1: 20 10 So I have got two cells with 4 cores each. Virsh does not recognize the topology: # virsh capabilities x86_64 core2duo .. I guess this is the fact, because QEMU does not recognize the NUMA-Architecture (QEMU-Monitor): (qemu) info numa 0 nodes This is done on an Ubuntu 10.04 LTS: Linux lxkvm01 2.6.32-23-server #37-Ubuntu SMP Fri Jun 11 09:11:11 UTC 2010 x86_64 GNU/Linux The package used was: qemu-kvm 0.12.3+noroms-0ubuntu9.2 When doing benchmarks numastat shows a lot of misses when the performance is bad. # numastat node0 node1 numa_hit 15527158 17015505 numa_miss 7032982 3512950 numa_foreign 3512950 7032982 interleave_hit 8078 8264 local_node 15525187 17006655 other_node 7034953 3521800 So apparently KVM does not utilize the NUMA-architecture. Did I do something wrong. Is KVM missing a patch? Do I need to activate something in KVM to recognize the NUMA-Architecture? I tried the newest Kernel Module 2.6.32.15 and Qemu-kvm 0.12.4 without a change. Any hints? Kind regards, Ralf