From mboxrd@z Thu Jan 1 00:00:00 1970 From: Avi Kivity Subject: Re: KVM performance vs. Xen Date: Thu, 30 Apr 2009 11:56:14 +0300 Message-ID: <49F967AE.4040905@redhat.com> References: <49F8672E.5080507@linux.vnet.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: kvm-devel To: Andrew Theurer Return-path: Received: from mx2.redhat.com ([66.187.237.31]:50191 "EHLO mx2.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1761551AbZD3I4R (ORCPT ); Thu, 30 Apr 2009 04:56:17 -0400 In-Reply-To: <49F8672E.5080507@linux.vnet.ibm.com> Sender: kvm-owner@vger.kernel.org List-ID: Andrew Theurer wrote: > I wanted to share some performance data for KVM and Xen. I thought it > would be interesting to share some performance results especially > compared to Xen, using a more complex situation like heterogeneous > server consolidation. > > The Workload: > The workload is one that simulates a consolidation of servers on to a > single host. There are 3 server types: web, imap, and app (j2ee). In > addition, there are other "helper" servers which are also consolidated: > a db server, which helps out with the app server, and an nfs server, > which helps out with the web server (a portion of the docroot is nfs > mounted). There is also one other server that is simply idle. All 6 > servers make up one set. The first 3 server types are sent requests, > which in turn may send requests to the db and nfs helper servers. The > request rate is throttled to produce a fixed amount of work. In order > to increase utilization on the host, more sets of these servers are > used. The clients which send requests also have a response time > requirement which is monitored. The following results have passed the > response time requirements. > What's the typical I/O load (disk and network bandwidth) while the tests are running? > The host hardware: > A 2 socket, 8 core Nehalem with SMT, and EPT enabled, lots of disks, 4 x > 1 GB Ethenret CPU time measurements with SMT can vary wildly if the system is not fully loaded. If the scheduler happens to schedule two threads on a single core, both of these threads will generate less work compared to if they were scheduled on different cores. > Test Results: > The throughput is equal in these tests, as the clients throttle the work > (this is assuming you don't run out of a resource on the host). What's > telling is the CPU used to do the same amount of work: > > Xen: 52.85% > KVM: 66.93% > > So, KVM requires 66.93/52.85 = 26.6% more CPU to do the same amount of > work. Here's the breakdown: > > total user nice system irq softirq guest > 66.90 7.20 0.00 12.94 0.35 3.39 43.02 > > Comparing guest time to all other busy time, that's a 23.88/43.02 = 55% > overhead for virtualization. I certainly don't expect it to be 0, but > 55% seems a bit high. So, what's the reason for this overhead? At the > bottom is oprofile output of top functions for KVM. Some observations: > > 1) I'm seeing about 2.3% in scheduler functions [that I recognize]. > Does that seems a bit excessive? Yes, it is. If there is a lot of I/O, this might be due to the thread pool used for I/O. > 2) cpu_physical_memory_rw due to not using preadv/pwritev? I think both virtio-net and virtio-blk use memcpy(). > 3) vmx_[save|load]_host_state: I take it this is from guest switches? These are called when you context-switch from a guest, and, much more frequently, when you enter qemu. > We have 180,000 context switches a second. Is this more than expected? Way more. Across 16 logical cpus, this is >10,000 cs/sec/cpu. > I wonder if schedstats can show why we context switch (need to let > someone else run, yielded, waiting on io, etc). > Yes, there is a scheduler tracer, though I have no idea how to operate it. Do you have kvm_stat logs? -- error compiling committee.c: too many arguments to function