From mboxrd@z Thu Jan 1 00:00:00 1970 From: Brian Rak Subject: Windows Server 2008R2 KVM guest performance issues Date: Mon, 26 Aug 2013 15:15:00 -0400 Message-ID: <521BA934.3050507@gameservers.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit To: kvm@vger.kernel.org Return-path: Received: from mail.choopa.net ([216.155.136.52]:46772 "EHLO mail.choopa.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751852Ab3HZTY1 (ORCPT ); Mon, 26 Aug 2013 15:24:27 -0400 Received: from localhost (unknown [127.0.0.1]) by mail.choopa.net (iRedMail) with ESMTP id C7DAA4AEDD8 for ; Mon, 26 Aug 2013 15:15:00 -0400 (EDT) Received: from mail.choopa.net ([127.0.0.1]) by localhost (mail.choopa.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cIglkn0sXi0a for ; Mon, 26 Aug 2013 15:15:00 -0400 (EDT) Received: from [10.5.5.117] (office-nat.choopa.net [208.167.225.40]) by mail.choopa.net (iRedMail) with ESMTPSA id 4CFF04AEDC4 for ; Mon, 26 Aug 2013 15:15:00 -0400 (EDT) Sender: kvm-owner@vger.kernel.org List-ID: I've been trying to track down the cause of some serious performance issues with a Windows 2008R2 KVM guest. So far, I've been unable to determine what exactly is causing the issue. When the guest is under load, I see very high kernel CPU usage, as well as terrible guest performance. The workload on the guest is approximately 1/4 of what we'd run unvirtualized on the same hardware. Even at that level, we max out every vCPU in the guest. While the guest runs, I see very high kernel CPU usage (based on `htop` output). Host setup: Linux nj1058 3.10.8-1.el6.elrepo.x86_64 #1 SMP Tue Aug 20 18:48:29 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux CentOS 6 qemu 1.6.0 2x Intel E5-2630 (virtualization extensions turned on, total of 24 cores including hyperthread cores) 24GB memory swap file is enabled, but unused Guest setup: Windows Server 2008R2 (64 bit) 24 vCPUs 16 GB memory VirtIO disk and network drivers installed /qemu16/bin/qemu-system-x86_64 -name VMID100 -S -machine pc-i440fx-1.6,accel=kvm,usb=off -cpu host,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -m 15259 -smp 24,sockets=1,cores=12,threads=2 -uuid 90301200-8d47-6bb3-0623-bed7c8b1dd7c -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/libvirt111/var/lib/libvirt/qemu/VMID100.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc,driftfix=slew -no-hpet -boot c -usb -drive file=/dev/vmimages/VMID100,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=18,id=hostnet0,vhost=on,vhostfd=19 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:00:2c:6d,bus=pci.0,addr=0x3 -vnc 127.0.0.1:100 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 The beginning of `perf top` output: Samples: 62M of event 'cycles', Event count (approx.): 642019289177 64.69% [kernel] [k] _raw_spin_lock 2.59% qemu-system-x86_64 [.] 0x00000000001e688d 1.90% [kernel] [k] native_write_msr_safe 0.84% [kvm] [k] vcpu_enter_guest 0.80% [kernel] [k] __schedule 0.77% [kvm_intel] [k] vmx_vcpu_run 0.68% [kernel] [k] effective_load 0.65% [kernel] [k] update_cfs_shares 0.62% [kernel] [k] _raw_spin_lock_irq 0.61% [kernel] [k] native_read_msr_safe 0.56% [kernel] [k] enqueue_entity I've captured 20,000 lines of kvm trace output. This can be found https://gist.github.com/devicenull/fa8f49d4366060029ee4/raw/fb89720d34b43920be22e3e9a1d88962bf305da8/trace So far, I've tried the following with very little effect: * Disable HPET on the guest * Enable hv_relaxed, hv_vapic, hv_spinlocks * Enable SR-IOV * Pin vCPUs to physical CPUs * Forcing x2apic enabled in the guest (bcdedit /set x2apicpolicy yes) * bcdedit /set useplatformclock yes and no Any suggestions as to what I can do to get better performance out of ths guest? Or reasons why I'm seeing such high kernel cpu usage with it?