From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marcelo Tosatti Subject: Re: Guest performance is reduced after live migration Date: Wed, 2 Jan 2013 22:48:46 -0200 Message-ID: <20130103004846.GA8845@amt.cnet> References: <443D5D30774F8E4F8B2D304016C65D7A655AA85B@sswchi5pmbx1.peak6.net> <20130102215041.GA21706@amt.cnet> <443D5D30774F8E4F8B2D304016C65D7A655C0223@sswchi5pmbx1.peak6.net> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: "kvm@vger.kernel.org" , "Shouta.Uehara@jp.yokogawa.com" To: Mark Petersen Return-path: Received: from mx1.redhat.com ([209.132.183.28]:16028 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753058Ab3ACAs7 (ORCPT ); Wed, 2 Jan 2013 19:48:59 -0500 Content-Disposition: inline In-Reply-To: <443D5D30774F8E4F8B2D304016C65D7A655C0223@sswchi5pmbx1.peak6.net> Sender: kvm-owner@vger.kernel.org List-ID: On Wed, Jan 02, 2013 at 11:56:11PM +0000, Mark Petersen wrote: > I don't think it's related to huge pages... >=20 > I was using phoronix-test-suite to run benchmarks. The 'batch/compil= ation' group shows the slowdown for all tests, the 'batch/computation' = show some performance degradation, but not nearly as significant. Huge pages in the host, for the qemu-kvm process, i mean. Without huge pages backing guest memory in the host, 4k EPT TLB entries will be used instead of 2MB EPT TLB entries. > You could probably easily test this way without phoronix - Start a V= M with almost nothing running. Download mainline Linux kernel, compile= =2E This takes about 45 seconds in my case (72GB memory, 16 virtual CP= Us, idle physical host running this VM.) Run as many times as you want= , still takes ~45 seconds. >=20 > Migrate to a new idle host, kernel compile now takes ~90 seconds, wai= t 3 hours (should give khugepaged a change to do its thing I imagine), Please verify its the case (by checking how much memory is backed by hugepages). http://www.mjmwired.net/kernel/Documentation/vm/transhuge.txt "Monitoring Usage". > kernel compiles still take 90 seconds. Reboot virtual machine (run '= shutdown -r now', reboot, whatever.) First compile will take ~45 secon= ds after reboot. You don't even need to reset/destroy/shutdown the VM,= just a reboot in the guest fixes the issue. What is the qemu command line? > I'm going to test more with qemu-kvm 1.3 tomorrow as I have a new/ded= icated lab setup and recently built the 1.3 code base. I'd be happy to= run any test that would help in diagnosing the real issue here, I'm ju= st not sure how to best diagnose this issue. >=20 > Thanks, > Mark > =20 > -----Original Message----- >=20 > Can you describe more details of the test you are performing?=20 >=20 > If transparent hugepages are being used then there is the possibility= that there has been no time for khugepaged to back guest memory with h= uge pages, in the destination (don't recall the interface for retrievin= g number of hugepages for a given process, probably somewhere in /proc/= pid/). >=20 > On Wed, Dec 19, 2012 at 12:43:37AM +0000, Mark Petersen wrote: > > Hello KVM, > >=20 > > I'm seeing something similar to this (http://thread.gmane.org/gmane= =2Ecomp.emulators.kvm.devel/100592) as well when doing live migrations = on Ubuntu 12.04 (Host and Guest) with a backported libvirt 1.0 and qemu= -kvm 1.2 (improved performance for live migrations on guests with large= memory guests is great!)=A0 The default libvirt =A00.9.8 and qemu-kvm = 1.0 have the same issue. > >=20 > > Kernel is 3.2.0-34-generic and eglicb 2.15 on both host/guest.=A0 I= 'm seeing similar issues with both virtio and ide bus.=A0 Hugetblfs is = not used, but transparent hugepages are.=A0 Host machines are dual core= Xeon E5-2660 processors.=A0 I tried disabling EPT but that doesn't see= m to make a difference so I don't think it's a requirement to reproduce= =2E > >=20 > > If I use Ubuntu 10.04 guest with eglibc 2.11 and any of these kerne= ls I don't seem to have the issue: > >=20 > > linux-image-2.6.32-32-server - 2.6.32-32.62=20 > > linux-image-2.6.32-38-server - 2.6.32-38.83=20 > > linux-image-2.6.32-43-server - 2.6.32-43.97=20 > > linux-image-2.6.35-32-server - 2.6.35-32.68~lucid1=20 > > linux-image-2.6.38-16-server - 2.6.38-16.67~lucid1=20 > > linux-image-3.0.0-26-server - 3.0.0-26.43~lucid1 > > linux-image-3.2-5 - mainline 3.2.5 kernel > >=20 > > I'm guess it's a libc issue (or at least a libc change causing the = issue) as it doesn't seem to a be kernel related. > >=20 > > I'll try other distributions as a guest (probably Debian/Ubuntu) wi= th newer libc's and see if I can pinpoint the issue to a libc version. = Any other ideas? > >=20 > > Shared disk backend is clvm/LV via FC to EMC SAN, not sure what els= e might be relevant. > >=20 > > Thanks, > > Mark > >=20 > >=20 > > ______________________________________________ > >=20 > > See http://www.peak6.com/email_disclaimer/ for terms and conditions= =20 > > related to this email > > -- > > To unsubscribe from this list: send the line "unsubscribe kvm" in t= he=20 > > body of a message to majordomo@vger.kernel.org More majordomo info = at =20 > > http://vger.kernel.org/majordomo-info.html