From mboxrd@z Thu Jan 1 00:00:00 1970 From: Anthony Liguori Subject: Re: [PATCH 0/3] KVM-userspace: add NUMA support for guests Date: Mon, 01 Dec 2008 09:49:04 -0600 Message-ID: <49340770.9000908@codemonkey.ws> References: <492F1DD9.8030901@amd.com> <49318A10.7080801@redhat.com> <4933F177.5040802@amd.com> <4933F4B0.7040500@redhat.com> <4934027D.8080904@codemonkey.ws> <493404CA.3060404@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Andre Przywara , kvm@vger.kernel.org, "Daniel P. Berrange" , Andi Kleen To: Avi Kivity Return-path: Received: from an-out-0708.google.com ([209.85.132.240]:37386 "EHLO an-out-0708.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751252AbYLAPtK (ORCPT ); Mon, 1 Dec 2008 10:49:10 -0500 Received: by an-out-0708.google.com with SMTP id d40so921706and.1 for ; Mon, 01 Dec 2008 07:49:09 -0800 (PST) In-Reply-To: <493404CA.3060404@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: Avi Kivity wrote: > Anthony Liguori wrote: >> >> I see no compelling reason to do cpu placement internally. It can be >> done quite effectively externally. >> >> Memory allocation is tough, but I don't think it's out of reach. >> Looking at the numactl man page, you can do: >> >> numactl --offset=1G --length=1G --membind=1 --file /dev/shm/A --touch >> Bind the second gigabyte in the tmpfs file /dev/shm/A to node 1. >> >> >> Since we can already create VM's with the -mem-path argument, if you >> create a 2GB guest and want it to span two numa nodes, you could do: >> >> numactl --offset=0G --length=1G --membind=0 --file /dev/shm/A --touch >> numactl --offset=1G --length=1G --membind=1 --file /dev/shm/A --touch >> >> And then create the VM with: >> >> qemu-system-x86_64 -mem-path /dev/shm/A -mem 2G ... >> >> What's best about this approach, is that you get full access to what >> numactl is capable of. Interleaving, rebalancing, etc. > > It looks horribly difficult and unintuitive. It forces you to use > -mem-path (which is an abomination; the only reason it lives is that > we can't allocate large pages with it). As opposed to inventing new options for QEMU that convey all of the same information a slightly different way? We're stuck with -mem-path so we might as well make good use of it. The proposed syntax is: qemu -numanode node=1,cpu=2,cpu=3,start=1G,size=1G,hostnode=3 The new syntax would be: qemu -smp 4 -numa nodes=2,cpus=1:2:3:4,mem=1G:1G -mem-path /dev/hugetlbfs/foo Then you would have to look up the thread ids, and do taskset taskset taskset taskset numactl -o 1G -l 1G -m 0 -f /dev/hugetlbfs/foo numactl -o 1G -l 1G -m 1 -f /dev/hugetlbfs/foo This may look like a lot more, but it's not going to be nearly enough to specify a NUMA placement on startup. What if you have a very large NUMA system and want to rebalance virtual machines? You need a mechanism to do this that now has to be exposed through the monitor. In fact, you'll almost certainly introduce a taskset-like monitor command and a numactl-like monitor command. Why reinvent the wheel? Plus, taskset and numactl gives you a lot of flexibility. All we're going to do by cooking this stuff into QEMU is artificially limit ourselves. Regards, Anthony LIguori