[libvirt] libvirk KVM save (migrate) terribly slow

[libvirt] libvirk KVM save (migrate) terribly slow

[Nikola Ciprich]

Hi,
I noticed that using libvirt to save running domain is terribly slow,
I mean it can take even 10 minutes to save VM with 2GB RAM, although
the host is almost idle (8CPU, 16GB RAM). If I understand correctly, 
libvirt first pauses the vm and then migrates it to file (optionally through gzip).
Restoring it back to running state is almost instant.
I guess this is not what should be expected, is there something particular 
I should check?
Thanks a lot in advance
with best regards
nik

-- 
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[libvirt] Re: libvirk KVM save (migrate) terribly slow

[Daniel P. Berrange]

On Thu, Dec 10, 2009 at 08:05:12AM +0100, Nikola Ciprich wrote:
> Hi,
> I noticed that using libvirt to save running domain is terribly slow,
> I mean it can take even 10 minutes to save VM with 2GB RAM, although
> the host is almost idle (8CPU, 16GB RAM). If I understand correctly, 
> libvirt first pauses the vm and then migrates it to file (optionally through gzip).
> Restoring it back to running state is almost instant.
> I guess this is not what should be expected, is there something particular 
> I should check?

This matches behaviour that I see when saving VMs. QEMU takes a really
very long time to save itself using migrate exec:.  It is not CPU limited
even in gzip mode - there is near zero CPU usage while this is going on.
QEMU just seems to be very slow at sending the data. I've not had time to
investigate whether this is a flaw in QEMU's exec: migration handling,
or whether it is being hurt by too small pipe buffers, or something else
entirely.  I must say I'm not entirely convinced that it is a good idea
for QEMU to be mixing use of FILE * / popen, with non-blocking I/O, but
that could be a red-herring.

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: libvirk KVM save (migrate) terribly slow

[Pierre Riteau]

On 10 déc. 2009, at 12:44, Daniel P. Berrange wrote:

> On Thu, Dec 10, 2009 at 08:05:12AM +0100, Nikola Ciprich wrote:
>> Hi,
>> I noticed that using libvirt to save running domain is terribly slow,
>> I mean it can take even 10 minutes to save VM with 2GB RAM, although
>> the host is almost idle (8CPU, 16GB RAM). If I understand correctly, 
>> libvirt first pauses the vm and then migrates it to file (optionally through gzip).
>> Restoring it back to running state is almost instant.
>> I guess this is not what should be expected, is there something particular 
>> I should check?
> 
> This matches behaviour that I see when saving VMs. QEMU takes a really
> very long time to save itself using migrate exec:.  It is not CPU limited
> even in gzip mode - there is near zero CPU usage while this is going on.
> QEMU just seems to be very slow at sending the data. I've not had time to
> investigate whether this is a flaw in QEMU's exec: migration handling,
> or whether it is being hurt by too small pipe buffers, or something else
> entirely.  I must say I'm not entirely convinced that it is a good idea
> for QEMU to be mixing use of FILE * / popen, with non-blocking I/O, but
> that could be a red-herring.
> 
> Daniel

I've reported this issue back when the regression was introduced in qemu.
Anthony had the same idea (not mixing popen with non-blocking I/O), but no solution was found at the time.
I haven't tried recently (I'm using TCP migration only) but the problem must still be here.

The thread on qemu-devel:
http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg01557.html
http://lists.gnu.org/archive/html/qemu-devel/2009-09/msg00020.html

-- 
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/