[Qemu-devel] kvm suspend performance

[Qemu-devel] kvm suspend performance

[Thomas Knauth]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]

Dear all,

lately I've been playing around with qemu's/kvm's suspend (to disk) and
resume. My initial expectation was that both operations are I/O bound. So
it surprised me to see that suspend to disk seems to be CPU-bound.
Suspending a VM with 1.5 GB memory takes 55 seconds. This works out to less
than 30 MB/s. Again, I was expecting to be I/O bound and reach 100 MB/s
(and more). I am talking to qemu/kvm via libvirt. Not sure if this matters.

I am looking for a hint what the issue could be here. Hopefully with
pointers to the (offending) code.

Thanks,
Thomas.

[-- Attachment #2: Type: text/html, Size: 723 bytes --]

Re: [Qemu-devel] kvm suspend performance

[Stefan Hajnoczi]

On Tue, Mar 19, 2013 at 05:24:59PM +0100, Thomas Knauth wrote:
> lately I've been playing around with qemu's/kvm's suspend (to disk) and
> resume. My initial expectation was that both operations are I/O bound. So
> it surprised me to see that suspend to disk seems to be CPU-bound.
> Suspending a VM with 1.5 GB memory takes 55 seconds. This works out to less
> than 30 MB/s. Again, I was expecting to be I/O bound and reach 100 MB/s
> (and more). I am talking to qemu/kvm via libvirt. Not sure if this matters.
> 
> I am looking for a hint what the issue could be here. Hopefully with
> pointers to the (offending) code.

Hi Thomas,
Please provide more details about the issue:

Which QEMU or libvirt command are you using to suspend the guest to
disk?

Why do you say it is CPU-bound?  Did you use a tool like vmstat or
simply because it does 30 MB/s instead of the expected 100 MB/s?

Which versions of libvirt and QEMU are you using?

Stefan

Re: [Qemu-devel] kvm suspend performance

[Thomas Knauth]

[-- Attachment #1: Type: text/plain, Size: 1504 bytes --]

Hi Stefan,

thanks for taking the time to reply.

On Wed, Mar 20, 2013 at 9:11 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:

> Which QEMU or libvirt command are you using to suspend the guest to
> disk?
>

virsh save <name> <file>


> Why do you say it is CPU-bound?  Did you use a tool like vmstat or
> simply because it does 30 MB/s instead of the expected 100 MB/s?
>

I monitor the system with top/vmstat. The kvm process shows a utilization
of 16% while the libvirtd process has a utilization of 4-5%. As this is a
4-core machine, a utilization of 25% would indicate that one core is fully
utilized. vmstat also gives the number of context switches. I see a
background context switch rate of about 60. This increases by 10x, to
around 600, during the save. But even 600 cxt switches per second doesn't
feel like a bottleneck.

So while I agree, that I might have jumped to quick to a conclusion about
the CPU-boundedness, there has to be some bottleneck. I would like to know
where and why. In my naive view we are just reading memory and writing it
to disk. This should progress faster than the 30 MB/s I am seeing right now.

Which versions of libvirt and QEMU are you using?


I've tried this on a couple of machines with differing versions of Ubuntu
(12.04, 10.10, and 10.04). They are all showing the same performance. The
machine where I did most of my measurements was the 12.04 machine. The
versions are libvirt 0.9.8-2ubuntu17 and
qemu-kvm 1.0+noroms-0ubuntu14.7.

Kind regards,
Thomas.

[-- Attachment #2: Type: text/html, Size: 2601 bytes --]

Re: [Qemu-devel] kvm suspend performance

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2095 bytes --]

On 03/20/2013 04:44 PM, Thomas Knauth wrote:
> Hi Stefan,
> 
> thanks for taking the time to reply.
> 
> On Wed, Mar 20, 2013 at 9:11 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> 
>> Which QEMU or libvirt command are you using to suspend the guest to
>> disk?
>>
> 
> virsh save <name> <file>

Then this is as much a libvirt question as a qemu question.  Which
version of libvirt?  'virsh save' maps to the qemu 'migrate' monitor
command, with a destination of migration to file.  On older libvirt, we
just relied on qemu's default throttling during migration.  Remember,
qemu throttles migration by default, so that it is less likely to
saturate a network link; but while this makes sense for a migration to a
remote network location, it gets in the way of a migration to a local file.

In newer libvirt, we added code so that when migrating to a file, we
temporarily raise the qemu throttling to be effectively unlimited.  In
my own testing, it provided nearly a 10x speedup on the time for 'virsh
save' on the guests that I was testing on my own machine.

Meanwhile, if your libvirt is too old to have the automatic temporary
change to the throttling limit, you should be able to issue 'virsh
migrate-setspeed $dom 1000000' (or some other large number), to raise
the qemu throttling limits to a much higher rate of MiB/s permitted
during the migration to file.

> Which versions of libvirt and QEMU are you using?
> 
> 
> I've tried this on a couple of machines with differing versions of Ubuntu
> (12.04, 10.10, and 10.04). They are all showing the same performance. The
> machine where I did most of my measurements was the 12.04 machine. The
> versions are libvirt 0.9.8-2ubuntu17 and
> qemu-kvm 1.0+noroms-0ubuntu14.7.

That explains it.  Libvirt migration-to-file with unlimited speed wasn't
until upstream commit v0.10.0-rc0-62-g6cfdeaa.  I really wish the Ubuntu
distro would quit shipping what feels like stone-age libvirt.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]

Re: [Qemu-devel] kvm suspend performance

[Thomas Knauth]

[-- Attachment #1: Type: text/plain, Size: 703 bytes --]

Hi Eric,

thanks for the reply. This indeed solved my issue. Suspending is much
faster without the artificial throttle.

On a related note: I'm curious about the baseline resume latency. It takes
about 5 seconds to resume an instance with a tiny amount of state (500 MB
dump size). The data is all in the page cache. I measure the time it takes
the 'virsh restore <dump file>' command to return.

Any opinions on what could be the issue here? I've also linked to a figure
illustrating the problem. Resume speed scales nicely with increased dump
size, but the baseline penalty feels awfully high

​https://www.dropbox.com/s/nhosknq02lnn4hz/resumetime-vs-dumpsize.pdf

Thanks,
Thomas.

[-- Attachment #2: Type: text/html, Size: 985 bytes --]

Re: [Qemu-devel] kvm suspend performance

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

On 03/23/2013 06:42 AM, Thomas Knauth wrote:
> Hi Eric,
> 
> thanks for the reply. This indeed solved my issue. Suspending is much
> faster without the artificial throttle.
> 
> On a related note: I'm curious about the baseline resume latency. It takes
> about 5 seconds to resume an instance with a tiny amount of state (500 MB
> dump size). The data is all in the page cache. I measure the time it takes
> the 'virsh restore <dump file>' command to return.
> 
> Any opinions on what could be the issue here? I've also linked to a figure
> illustrating the problem. Resume speed scales nicely with increased dump
> size, but the baseline penalty feels awfully high

Sorry, but I haven't ever profiled it to see where the problems might
lie.  It sounds like it might be an interesting project if you want to
take it on, but I don't have the time to spend on it myself at the moment.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]