From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1MsuwX-0001qJ-2L for qemu-devel@nongnu.org; Wed, 30 Sep 2009 04:53:49 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1MsuwR-0001nU-VL for qemu-devel@nongnu.org; Wed, 30 Sep 2009 04:53:48 -0400 Received: from [199.232.76.173] (port=56666 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MsuwR-0001nE-O2 for qemu-devel@nongnu.org; Wed, 30 Sep 2009 04:53:43 -0400 Received: from mail.corp.accelance.fr ([213.162.48.15]:61893) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1MsuwQ-0001iO-W7 for qemu-devel@nongnu.org; Wed, 30 Sep 2009 04:53:43 -0400 Subject: Re: [Qemu-devel] Qemu savevm and CPU soft lockup From: Benjamin Cleyet-Marrel In-Reply-To: <1253744460.1993.4.camel@athens.parenthephobia.org.uk> References: <1253267801.9686.159.camel@bcm-portable> <1253721904.16114.101.camel@bcm-portable> <20090923181945.GB23822@shareable.org> <7F8DD534-BEFE-4D77-B1AC-A24A0403CCAF@accelance.fr> <1253744460.1993.4.camel@athens.parenthephobia.org.uk> Content-Type: multipart/alternative; boundary="=-2i5zO1X+5H7PQj0V6Mvb" Date: Wed, 30 Sep 2009 10:53:38 +0200 Message-Id: <1254300818.5918.107.camel@bcm-portable> Mime-Version: 1.0 List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Nathan Baum Cc: "qemu-devel@nongnu.org" --=-2i5zO1X+5H7PQj0V6Mvb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable -------- Message initial -------- De: Nathan Baum =C3=80: Ben Accelance Cc: qemu-devel@nongnu.org Sujet: Re: [Qemu-devel] Qemu savevm and CPU soft lockup Date: Wed, 23 Sep 2009 23:21:00 +0100 On Wed, 2009-09-23 at 20:52 +0200, Ben Accelance wrote: >=20 >=20 > Le 23 sept. 2009 =C3=A0 20:19, Jamie Lokier a =C3= =A9crit : >=20 > > Benjamin Cleyet-Marrel wrote: > >> > >> Hi, > >> After further investigation, I figured out that when issuing a =20 > >> savevm > >> command > >> the entire qemu process gets stuck on IO wait. > >> I can't issue any other commands on the monitor the process is =20 > >> shown > >> as D and the guest is in softlockup state. > >> Looking at the way migrate (and the -d for detach I presume) is > >> working I suppose the same behaviour would be expected from the =20 > >> savevm > >> function. > >> A savevm -d so that the qemu process would not be freezed while =20 > >> saving > >> the data. > >> Sorry if I am just talking non sense but my snapshot on iscsi =20 > >> storage > >> takes about 1 minutes. > >> which means that my guest are down for a minute or so which is not > >> ideal. > > > > This is normal savevm behaviour, and it is exactly the reason why > > migrate-to-file is useful. I would not be surprised if savevm is > > changed to use migrate-to-file internally at some point, but it does > > not look like happening soon. > > > > You might avoid the guest softlockup state by stopping the guest > > ("stop" command) before savevm, and "cont" afterwards? > > > > Or the guest might get just as confused, as the clock still advances. > > > > -- Jamie > > > > > Thanks for this, at least it clarifies things. Now sorry for the =20 > pain , but how do you migrate to a file and then how do you restore =20 > from the file ? > Thanks > Cheers >The way I migrate-to-file is with > > (qemu) migrate -d "exec:cat>file". > >and then restore with > > $ qemu ... -incoming "exec:cat file" > Hi,=20 I finaly found some time to investigate deeper on this. this migrate_to_file is definitly not cutting it. I am trying to do a daily snapshot of my vm without any interruption. the migrate to file by itself is not doing an synchroneous disque image snapshot (provided that image file are qcow2) so restoring the status without the proper data is useless. The only way I found is to embedd all the save function and qemu-img snapshot -c call in a shell script. but it is not very clean I would say. the dosavevm script looklike this :=20 #!/bin/sh qemu-img snapshot -c statefile /vm/img/test2_20G_vdc.img & qemu-img snapshot -c statefile /vm/img/test2_2G_swap.img & qemu-img snapshot -c statefile /vm/img/test2_5G_vda.img & cat > /tmp/test.statefile and to save my disk I do:=20 #!/bin/sh echo -e "\001c\nmigrate -d \"exec:/root/dosavevm\"\n\001c\n" | socat STDIO UNIX:/var/run/libvirt/qemu/test.serial=20 satus=3D"" while [ -z "$status" ] do status=3D`echo -e "\001c\ninfo migrate\n\001c\n" | socat STDIO UNIX:/var/run/libvirt/qemu/test.serial | grep completed` sleep 1 done echo -e "\001c\ncont\n\001c\n" | socat STDIO UNIX:/var/run/libvirt/qemu/test.serial=20 Which is as hugly as it can get. Finnaly restoring the data needs: shutdown the guest=20 applying the snapshot=20 starting a new process with the correct parameters=20 So back to my original problem. Is there a way to perform a consitent backup of kvm vm without 60s downtime an hugly CPU soft lockup ? Thanks Cheers Ben=20 --=-2i5zO1X+5H7PQj0V6Mvb Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: 7bit



-------- Message initial --------
De: Nathan Baum <nathan@parenthephobia.org.uk>
À: Ben Accelance <bcm@accelance.fr>
Cc: qemu-devel@nongnu.org <qemu-devel@nongnu.org>
Sujet: Re: [Qemu-devel] Qemu savevm and CPU soft lockup
Date: Wed, 23 Sep 2009 23:21:00 +0100

On Wed, 2009-09-23 at 20:52 +0200, Ben Accelance wrote:
> 
> 
> Le 23 sept. 2009 à 20:19, Jamie Lokier <jamie@shareable.org> a écrit :
> 
> > Benjamin Cleyet-Marrel wrote:
> >>
> >>   Hi,
> >>   After further investigation, I figured out that when issuing a  
> >> savevm
> >>   command
> >>   the entire qemu process gets stuck on IO wait.
> >>   I can't issue any other commands on the monitor the process is  
> >> shown
> >>   as D and the guest is in softlockup state.
> >>   Looking at the way migrate (and the -d for detach I presume) is
> >>   working I suppose the same behaviour would be expected from the  
> >> savevm
> >>   function.
> >>   A savevm -d so that the qemu process would not be freezed while  
> >> saving
> >>   the data.
> >>   Sorry if I am just talking non sense but my snapshot on iscsi  
> >> storage
> >>   takes about 1 minutes.
> >>   which means that my guest are down for a minute or so which is not
> >>   ideal.
> >
> > This is normal savevm behaviour, and it is exactly the reason why
> > migrate-to-file is useful.  I would not be surprised if savevm is
> > changed to use migrate-to-file internally at some point, but it does
> > not look like happening soon.
> >
> > You might avoid the guest softlockup state by stopping the guest
> > ("stop" command) before savevm, and "cont" afterwards?
> >
> > Or the guest might get just as confused, as the clock still advances.
> >
> > -- Jamie
> >
> >
> Thanks for this, at least it clarifies things. Now sorry for the  
> pain , but how do you migrate to a file and then how do you restore  
> from the file ?
> Thanks
> Cheers

>The way I migrate-to-file is with
>
>  (qemu) migrate -d "exec:cat>file".
>
>and then restore with
>
>  $ qemu ... -incoming "exec:cat file"
>



Hi,
I finaly found some time to investigate deeper on this.
this migrate_to_file is definitly not cutting it.
I am trying to do a daily snapshot of my vm without any interruption.
the migrate to file by itself  is not doing an synchroneous disque image snapshot (provided that image file are qcow2)
so restoring the status without the proper data is useless.
The only way I found is to embedd all the save function and qemu-img snapshot -c call in a shell script. but it is not very clean I would say.

the dosavevm script  looklike this :

#!/bin/sh
qemu-img snapshot -c statefile /vm/img/test2_20G_vdc.img &
qemu-img snapshot -c statefile /vm/img/test2_2G_swap.img &
qemu-img snapshot -c statefile /vm/img/test2_5G_vda.img &
cat  > /tmp/test.statefile


and to save my disk I do:

#!/bin/sh
echo -e "\001c\nmigrate -d \"exec:/root/dosavevm\"\n\001c\n" | socat STDIO UNIX:/var/run/libvirt/qemu/test.serial
satus=""
while [ -z "$status" ]
do
status=`echo -e "\001c\ninfo migrate\n\001c\n" | socat STDIO UNIX:/var/run/libvirt/qemu/test.serial | grep completed`
sleep 1
done
echo -e "\001c\ncont\n\001c\n" | socat STDIO UNIX:/var/run/libvirt/qemu/test.serial


Which is as hugly as it can get.
Finnaly restoring the data needs:
shutdown the guest
applying the snapshot
starting a new process with the correct parameters

So back to my original problem.

Is there a way to perform a consitent backup of kvm vm without 60s downtime an hugly CPU soft lockup ?

Thanks

Cheers
Ben
--=-2i5zO1X+5H7PQj0V6Mvb--