Re: Windows slow boot: contractor wanted

[Richard Davies <richard@arachsys.com>]

Avi Kivity wrote:
> Richard Davies wrote:
> > Hi Avi,
> >
> > Thanks to you and several others for offering help. We will work with Avi at
> > first, but are grateful for all the other offers of help. We have a number
> > of other qemu-related projects which we'd be interested in getting done, and
> > will get in touch with these names (and anyone else who comes forward) to
> > see if any are of interest to you.
> >
> >
> > This slow boot problem is intermittent and varys in how slow the boots are,
> > but I managed to trigger it this morning with medium slow booting (5-10
> > minutes) and link to the requested traces below.
> >
> > The host in question has 128GB RAM and dual AMD Opteron 6128 (16 cores
> > total). It is running kernel 3.5.1 and qemu-kvm 1.1.1.
> >
> > In this morning's test, we have 3 guests, all booting Windows with 40GB RAM
> > and 8 cores each (we have seen small VMs go slow as I originally said, but
> > it is easier to trigger with big VMs):
> >
> > pid 15665: qemu-kvm -nodefaults -m 40960 -smp 8 -cpu host,hv_relaxed \
> >   -vga cirrus -usbdevice tablet -vnc :99 -monitor stdio -hda test1.raw
> > pid 15676: qemu-kvm -nodefaults -m 40960 -smp 8 -cpu host,hv_relaxed \
> >   -vga cirrus -usbdevice tablet -vnc :98 -monitor stdio -hda test2.raw
> > pid 15653: qemu-kvm -nodefaults -m 40960 -smp 8 -cpu host,hv_relaxed \
> >   -vga cirrus -usbdevice tablet -vnc :97 -monitor stdio -hda test3.raw
> >
> > We are running with hv_relaxed since this was suggested in the previous
> > thread, but we see intermittent slow boots with and without this flag.
> >
> >
> > All 3 VMs are booting slowly for most of the attached capture, which I
> > started after confirming the slow boots and stopped as soon as the first of
> > them (15665) had booted. In terms of visible symptoms, the VMs are showing
> > the Windows boot progress bar, which is moving very slowly. In top, the VMs
> > are at 400% CPU and their resident state size (RES) memory is slowly
> > counting up until it reaches the full VM size, at which point they finish
> > booting.
> >
> >
> > Here are the trace files:
> >
> > http://users.org.uk/slow-win-boot-1/ps.txt (ps auxwwwf as root)
> > http://users.org.uk/slow-win-boot-1/top.txt (top with 2 VMs still slow)
> > http://users.org.uk/slow-win-boot-1/trace-console.txt (running trace-cmd)
> > http://users.org.uk/slow-win-boot-1/trace.dat (the 1.7G trace data file)
> > http://users.org.uk/slow-win-boot-1/trace-report.txt (the 4G trace report)
> >
> >
> > Please let me know if there is anything else which I can provide?
>
>
> There are tons of PAUSE exits indicating cpu overcommit (and indeed you
> are overcommitted by about 50%).
>
> What host kernel version are you running?
>
> Does this reproduce without overcommit?


We're running host kernel 3.5.1 and qemu-kvm 1.1.1.

I hadn't though about it, but I agree this is related to cpu overcommit. The
slow boots are intermittent (and infrequent) with cpu overcommit whereas I
don't think it occurs without cpu overcommit.

In addition, if there is a slow boot ongoing, and you kill some other VMs to
reduce cpu overcommit then this will sometimes speed it up.

I guess the question is why even with overcommit most boots are fine, but
some small fraction then go slow?

Richard.