From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Theurer Subject: Windows Server 2008 VM performance Date: Tue, 02 Jun 2009 16:05:47 -0500 Message-ID: <4A25942B.7010503@linux.vnet.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit To: KVM list Return-path: Received: from e2.ny.us.ibm.com ([32.97.182.142]:37454 "EHLO e2.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751652AbZFBVFr (ORCPT ); Tue, 2 Jun 2009 17:05:47 -0400 Received: from d01relay02.pok.ibm.com (d01relay02.pok.ibm.com [9.56.227.234]) by e2.ny.us.ibm.com (8.13.1/8.13.1) with ESMTP id n52L1Svp017001 for ; Tue, 2 Jun 2009 17:01:28 -0400 Received: from d01av04.pok.ibm.com (d01av04.pok.ibm.com [9.56.224.64]) by d01relay02.pok.ibm.com (8.13.8/8.13.8/NCO v9.2) with ESMTP id n52L5mNU257980 for ; Tue, 2 Jun 2009 17:05:48 -0400 Received: from d01av04.pok.ibm.com (loopback [127.0.0.1]) by d01av04.pok.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id n52L5mQe003418 for ; Tue, 2 Jun 2009 17:05:48 -0400 Received: from twinturbo.austin.ibm.com (twinturbo.austin.ibm.com [9.41.41.35]) by d01av04.pok.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id n52L5l6M003297 for ; Tue, 2 Jun 2009 17:05:47 -0400 Sender: kvm-owner@vger.kernel.org List-ID: I've been looking at how KVM handles windows guests, and I am a little concerned with the CPU overhead. My test case is as follows: I am running 4 instances of a J2EE benchmark. Each instance needs one application server and one DB server. 8 VMs in total are used. I have the same App and DB software for Linux and Windows (and same versions) so I can compare between Linux and Windows. I also have another hypervisor which I can test both Windows and Linux VMs. The host has EPT capable processors. VMs in KVM are backed with large pages. Test results: Config CPU utilization ------ ----- KVM-85 Windows Server 2008 64-bit VMs 44.84 RedHat 5.3 w/ 2.6.29 64-bit VMs 24.56 Other-Hypervisor Windows Server 2008 64-bit VMs 30.63 RedHat 5.3 w/ 2.6.18 64-bit VMs 27.13 -KVM running Windows VMs uses 46% more CPU than the Other-Hypervisor -The Other-Hypervisor provides an optimized virtual network driver -KVM results listed above did not use virtio_net or virtio_disk for Windows, but do for Linux -One extra KVM run (not listed above) was made with virtio_net for Windows VMs but only reduced CPU by 2% -Most of the CPU overhead could be attributed to the DB VMs, where there is about 5 MB/sec writes per VM -I don't have a virtio_block driver for Windows to test. Does one exist? -All tests above had 2 vCPUS per VM Here's a comparison of kvm_stat between Windows (run1) and Linux (run2): run1 run2 run1/run2 ---- ---- --------- efer_relo: 0 0 1 exits : 1206880 121916 9.899 fpu_reloa: 210969 20863 10.112 halt_exit: 15092 13222 1.141 halt_wake: 14466 9294 1.556 host_stat: 211066 45117 4.678 hypercall: 0 0 1 insn_emul: 119582 38126 3.136 insn_emul: 0 0 1 invlpg : 0 0 1 io_exits : 131051 26349 4.974 irq_exits: 8128 12937 0.628 irq_injec: 29955 21825 1.373 irq_windo: 2504 2022 1.238 kvm_reque: 0 0 1 largepage: 1 64 0.009 mmio_exit: 59224 0 Inf mmu_cache: 0 3 0.000 mmu_flood: 0 0 1 mmu_pde_z: 0 0 1 mmu_pte_u: 0 0 1 mmu_pte_w: 0 0 1 mmu_recyc: 0 0 1 mmu_shado: 0 0 1 mmu_unsyn: 0 0 1 mmu_unsyn: 0 0 1 nmi_injec: 0 0 1 nmi_windo: 0 0 1 pf_fixed : 1 67 0.009 pf_guest : 0 0 1 remote_tl: 0 0 1 request_n: 0 0 1 signal_ex: 0 0 1 tlb_flush: 220 14037 0.016 10x the number of exits, a problem? I happened to try just one vCPU per VM for KVM/Windows VMs, and I was surprised how much of a difference it made: Config CPU utilization ------ ----- KVM-85 Windows Server 2008 64-bit VMs, 2 vCPU per VM 44.84 Windows Server 2008 64-bit VMs, 1 vCPU per VM 36.44 A 19% reduction in CPU utilization vs KVM/Windows-2vCPU! Does not explain all the overhead (vs Other-Hypervisor, 2 vCPUs per VM) but, that sure seems like a lot between 1 to 2 vCPUs for KVM/Windows-VMs. I have not run with 1 vCPU per VM with Other-Hypervisor, but I will soon. Anyway, I also collected kvm_stat for the 1 vCPU case, and here it is compared to KVM/Linux VMs with 2 vCPUs: run1 run2 run1/run2 ---- ---- --------- efer_relo: 0 0 1 exits : 1184471 121916 9.715 fpu_reloa: 192766 20863 9.240 halt_exit: 4697 13222 0.355 halt_wake: 4360 9294 0.469 host_stat: 192828 45117 4.274 hypercall: 0 0 1 insn_emul: 130487 38126 3.422 insn_emul: 0 0 1 invlpg : 0 0 1 io_exits : 114430 26349 4.343 irq_exits: 7075 12937 0.547 irq_injec: 29930 21825 1.371 irq_windo: 2391 2022 1.182 kvm_reque: 0 0 1 largepage: 0 64 0.001 mmio_exit: 69028 0 Inf mmu_cache: 0 3 0.000 mmu_flood: 0 0 1 mmu_pde_z: 0 0 1 mmu_pte_u: 0 0 1 mmu_pte_w: 0 0 1 mmu_recyc: 0 0 1 mmu_shado: 0 0 1 mmu_unsyn: 0 0 1 mmu_unsyn: 0 0 1 nmi_injec: 0 0 1 nmi_windo: 0 0 1 pf_fixed : 0 67 0.001 pf_guest : 0 0 1 remote_tl: 0 0 1 request_n: 0 0 1 signal_ex: 0 0 1 tlb_flush: 124 14037 0.009 Still see the huge difference in vm_exits, so I guess not all is great yet. So, what would be the best course of action for this? Is there a virtio_block driver to test? Can we find the root cause of the exits (is there a way to get stack dump or something that can show where there are coming from)? Thanks! -Andrew P.S. Here is the qemu cmd line for the windows VMs: /usr/local/bin/qemu-system-x86_64 -name newcastle-xdbt01 -hda /dev/disk/by-id/scsi-3600a0b80000f1eb10000074f4a02b08a -net nic,model=e1000,vlan=0,macaddr=00:50:56:00:00:03 -m 1024 -mempath /hugetlbfs -net tap,vlan=0,ifname=tap3,script=/etc/qemu-ifup -vnc 127.0.0.1:3 -smp 2 -daemonize