Re: [Qemu-devel] Windows slow boot: contractor wanted

[Troy Benjegerdes <hozer@hozed.org>]

I'd be interested in working on this.. What I'd like to propose is to write
an automated regression test harness that will reboot the host hardware, and
start booting up guest VMs and report the time-to-boot, as well as relative
performance of the running VMs.

For best results, I'd need access to the specific hardware you are using.

I'd also like to release the test harness back to the community, so I would
like some feedback from the mailing list on what kinds of tests should be
written that would provide the best information for the KVM developers.

What do you want to know, and what is the most usefull data to record to
debug this and future performance regressions?

On Thu, Aug 16, 2012 at 11:47:27AM +0100, Richard Davies wrote:
> Hi,
> 
> We run a cloud hosting provider using qemu-kvm 1.1, and are keen to find a
> contractor to track down and fix problems we have with large memory Windows
> guests booting very slowly - they can take several hours.
> 
> We previously reported these problems in July (copied below) and they are
> still present with Linux kernel 3.5.1 and qemu-kvm 1.1.1.
> 
> This is a serious issue for us which is causing significant pain to our
> larger Windows VM customers when their servers are offline for many hours
> during boot.
> 
> If anyone knowledgeable in the area would be interested in being paid to
> work on this, or if you know someone who might be, I would be delighted to
> hear from you.
> 
> Cheers,
> 
> Richard.
> 
> 
> ===== Previous bug report
> 
> http://marc.info/?l=qemu-devel&m=134304194329745
> 
> 
> We have been experiencing this problem for a while now too, using qemu-kvm
> (currently at 1.1.1).
> 
> Unfortunately, hv_relaxed doesn't seem to fix it. The following command line
> produces the issue:
> 
> qemu-kvm -nodefaults -m 4096 -smp 8 -cpu host,hv_relaxed -vga cirrus -usbdevice tablet -vnc :99 -monitor stdio -hda test.img
> 
> The hardware consists of dual AMD Opteron 6128 processors (16 cores in
> total) and 64GB of memory. This command line was tested on kernel 3.1.4. 
> 
> I've also tested with -no-hpet.
> 
> What I have seen is much as described: the memory fills out slowly, and top
> on the host will show the process using 100% on all allocated CPU cores. The
> most extreme case was a machine which took something between 6 and 8 hours
> to boot.
> 
> This seems to be related to the assigned memory, as described, but also the
> number of processor cores (which makes sense if we believe it's a timing
> issue?). I have seen slow-booting guests improved by switching down to a
> single or even two cores.
> 
> Matthew, I agree that this seems to be linked to the number of VMs running -
> in fact, shutting down other VMs on a dedicated test host caused the machine
> to start booting at a normal speed (with no reboot required).
> 
> However, the level of contention is never such that this could be explained
> by the host simply being overcommitted.
> 
> If it helps anyone, there's an image of the hard drive I've been using to
> test at:
> 
> http://46.20.114.253/
> 
> It's 5G of gzip file containing a fairly standard Windows 2008 trial
> installation. Since it's in the trial period, anyone who wants to use it may
> have to re-arm the trial: http://support.microsoft.com/kb/948472
> 
> Please let me know if I can provide any more information, or test anything.
> 
> Best wishes,
> 
> Owen Tuz
>