From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:40787) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SeT7I-0001WK-FM for qemu-devel@nongnu.org; Tue, 12 Jun 2012 11:34:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1SeT7D-0003ph-FY for qemu-devel@nongnu.org; Tue, 12 Jun 2012 11:34:47 -0400 Received: from g6t0184.atlanta.hp.com ([15.193.32.61]:26103) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SeT7D-0003jb-AA for qemu-devel@nongnu.org; Tue, 12 Jun 2012 11:34:43 -0400 Message-ID: <4FD76167.4080705@hp.com> Date: Tue, 12 Jun 2012 08:33:59 -0700 From: Chegu Vinod MIME-Version: 1.0 References: <1339173966.26976.95.camel@ul30vt> <4FD23221.5090208@hp.com> <1339177340.26976.111.camel@ul30vt> <4FD23CF9.5000203@hp.com> <4FD23FA0.6060202@siemens.com> <4FD24285.1060702@hp.com> <4FD2467B.1080005@siemens.com> In-Reply-To: <4FD2467B.1080005@siemens.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Large sized guest taking for ever to boot... Reply-To: chegu_vinod@hp.com List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jan Kiszka Cc: Alex Williamson , qemu-devel , "kvm@vger.kernel.org" On 6/8/2012 11:37 AM, Jan Kiszka wrote: > On 2012-06-08 20:20, Chegu Vinod wrote: >> On 6/8/2012 11:08 AM, Jan Kiszka wrote: >>> [CC'ing qemu as this discusses its code base] >>> >>> On 2012-06-08 19:57, Chegu Vinod wrote: >>>> On 6/8/2012 10:42 AM, Alex Williamson wrote: >>>>> On Fri, 2012-06-08 at 10:10 -0700, Chegu Vinod wrote: >>>>>> On 6/8/2012 9:46 AM, Alex Williamson wrote: >>>>>>> On Fri, 2012-06-08 at 16:29 +0000, Chegu Vinod wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> I picked up a recent version of the qemu (1.0.92 with some fixes) >>>>>>>> and tried it >>>>>>>> on x86_64 server (with host and the guest running 3.4.1 kernel). >>>>>> BTW, I observe the same thing if i were to use 1.1.50 version of the >>>>>> qemu... not sure if this is really >>>>>> related to qemu... >>>>>> >>>>>>>> While trying to boot a large guest (80 vcpus + 512GB) I observed >>>>>>>> that the guest >>>>>>>> took for ever to boot up... ~1 hr or even more. [This wasn't the >>>>>>>> case when I >>>>>>>> was using RHEL 6.x related bits] >>>>>>> Was either case using device assignment? Device assignment will map >>>>>>> and >>>>>>> pin each page of guest memory before startup, which can be a noticeable >>>>>>> pause on smallish (<16GB) guests. That should be linear scaling though >>>>>>> and if you're using qemu and not qemu-kvm, not related. Thanks, >>>>>> I don't have any device assignment at this point . Yes I am using qemu >>>>>> (not qemu-kvm)... >>>>> Just to be safe, are you using --enable-kvm with qemu? >>>> Yes... >>> Unless you are using current qemu.git master (where it is enabled by >>> default), --enable-kvm does not activate the in-kernel irqchip support >>> for you. Not sure if that can make such a huge difference, but it is a >>> variation from qemu-kvm. You can enable it in qemu-1.1 with -machine >>> kernel_irqchip=on. >> Thanks for pointing this out...i will add that. >> >> I was using qemu.git not the master > With "qemu.git master" I meant the latest version you can pull from the > master branch or qemu.git. If you are using an older version, please > specify the hash. > > BTW, you can check if irqchip is on by looking at the out of "info > qtree" in the qemu monitor. One of the last devices listed must be > called "kvm-apic". Sorry for the confusion. I was using the qemu.git and I do see the "kvm-apic" stuff (in the info qtree output) without specifying the -machine kernel_irqchip=on option.. > >> >>>> ----- >>>> >>>> /etc/qemu-ifup tap0 >>>> >>>> /usr/local/bin/qemu-system-x86_64 -enable-kvm \ >>>> >>>> -cpu >>>> Westmere,+rdtscp,+pdpe1gb,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+s >>>> >>>> s,+acpi,+ds,+vme \ >>>> -m 524288 -smp 80,sockets=80,cores=1,threads=1 \ >>>> -name vm1 \ >>>> -chardev >>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait >>>> \ >>>> -drive >>>> file=/dev/libvirt_lvm/vm.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native >>>> \ >>>> -device >>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>> \ >>>> -monitor stdio \ >>>> -net nic,macaddr=52:54:00:71:01:01 \ >>>> -net tap,ifname=tap0,script=no,downscript=no \ >>>> -vnc :4 >>>> >>>> /etc/qemu-ifdown tap0 >>>> >>>> ---- >>>> >>>>>> The issue seems very basic... 'was earlier running RHEL6.3 RC1 on the >>>>>> host and the guest and the host and the guest seemed to boot fine.. >>>>> Note that RHEL is based on qemu-kvm. Thanks, >>>> Yep..knew that :) >>>> >>>> I was using upstream qemu-kvm and was encouraged to move away from >>>> it...to qemu. >>> And that is good. :) >>> >>> Is the problem present in current qemu-kvm.git? If yes, can you bisect >>> when it was introduced? >> Shall try out the qemu-kvm.git (after finishing some experiments..). > Yes, please. Just did that and below are the results... > >> BTW, another data point ...if I try to boot a the RHEL6.3 kernel in the >> guest (with the latest qemu.git and the 3.4.1 on the host) it boots just >> fine.... >> >> So something to do with the 3.4.1 kernel in the guest and the existing >> udev... Need to dig >> deeper. > Maybe. But lets focus on getting the problematic guest running fine in > some configuration. If that turns out to be impossible, we may see a > guest issue. Host config. : 80 Cores + 1TB. Guest config : 40VCPUs + 512GB. I rebuilt the 3.4.1 kernel in the guest from scratch and retried my experiments and measured the boot times... a) Host : RHEL6.3 RC1 + qemu-kvm (that came with it) & Guest : RHEL6.3 RC1: ~1 min b) Host :3.4.1 + qemu-kvm.git & Guest : RHEL6.3RC1 - ~1 min c) Host :3.4.1 + qemu-kvm.git & Guest : 3.4.1 - ~10 mins d) Host :3.4.1 + qemu.git & Guest : RHEL6.3 RC1 - ~1 min e) Host :3.4.1 + qemu.git & Guest : 3.4.1 - ~14 mins In both the case (c) & (e) had quite a few warning/error messages from udevd. FYI.. Vinod PS: Haven't had the time to do any further analysis...as the machine is being used for other experiments... > > Jan >