[Qemu-devel] Qemu savevm and CPU soft lockup

[Qemu-devel] Qemu savevm and CPU soft lockup

[Benjamin Cleyet-Marrel]

[-- Attachment #1: Type: text/plain, Size: 2418 bytes --]

Hi, 

First post to this list so sorry if I am mistaken.
I am using qemu-kvm-0.11.0-rc2 I've googled quite a lot and could not
find anyone having or reporting the same problem.
The guest is a Centos5.3

The savevm/loadvm seems to be broken:

When issuing a savevm in the monitor, 
Guest get stuck for a few seconds and I get the infamous soft lockup a
couple of time. (see bellow)

In the end, the snapshot is working.

When doing a loadvm I often get weird behaviour on the console.
this can be completly removed if I issue a stop and cont before and
after the loadvm.

Question is:
Am I doing something wrong or is this a bug ?

Thanks
Cheers
Ben



Sep 18 10:58:48 dhcp155 kernel: BUG: soft lockup - CPU#0 stuck for 10s!
[swapper:0]
Sep 18 10:58:48 dhcp155 kernel: 
Sep 18 10:58:48 dhcp155 kernel: Pid: 0, comm:              swapper
Sep 18 10:58:48 dhcp155 kernel: EIP: 0060:[<c044d219>] CPU: 0
Sep 18 10:58:48 dhcp155 kernel: EIP is at handle_IRQ_event+0x39/0x8c
Sep 18 10:58:48 dhcp155 kernel:  EFLAGS: 00000246    Not tainted
(2.6.18-128.el5 #1)
Sep 18 10:58:48 dhcp155 kernel: EAX: 00000001 EBX: c06e6f00 ECX:
f7c9bf80 EDX: c0754f9c
Sep 18 10:58:48 dhcp155 kernel: ESI: f7c9bf80 EDI: 00000001 EBP:
00000000 DS: 007b ES: 007b
Sep 18 10:58:48 dhcp155 kernel: CR0: 8005003b CR2: 080c3e80 CR3:
376ad000 CR4: 00000690
Sep 18 10:58:48 dhcp155 kernel:  [<c044d2f0>] __do_IRQ+0x84/0xd6
Sep 18 10:58:48 dhcp155 kernel:  [<c04074e5>] do_IRQ+0xb0/0xc3
Sep 18 10:58:48 dhcp155 kernel:  [<c0405946>] common_interrupt+0x1a/0x20
Sep 18 10:58:48 dhcp155 kernel:  [<c044d219>] handle_IRQ_event+0x39/0x8c
Sep 18 10:58:48 dhcp155 kernel:  [<c044d2f0>] __do_IRQ+0x84/0xd6
Sep 18 10:58:48 dhcp155 kernel:  [<c044d26c>] __do_IRQ+0x0/0xd6
Sep 18 10:58:48 dhcp155 kernel:  [<c04074ce>] do_IRQ+0x99/0xc3
Sep 18 10:58:48 dhcp155 kernel:  [<c0405946>] common_interrupt+0x1a/0x20
Sep 18 10:58:48 dhcp155 kernel:  [<c0428ba7>] __do_softirq+0x57/0x114
Sep 18 10:58:48 dhcp155 kernel:  [<c04073eb>] do_softirq+0x52/0x9c
Sep 18 10:58:48 dhcp155 kernel:  [<c04059d7>] apic_timer_interrupt
+0x1f/0x24
Sep 18 10:58:48 dhcp155 kernel:  [<c0403bb0>] default_idle+0x0/0x59
Sep 18 10:58:48 dhcp155 kernel:  [<c0403be1>] default_idle+0x31/0x59
Sep 18 10:58:48 dhcp155 kernel:  [<c0403ca8>] cpu_idle+0x9f/0xb9
Sep 18 10:58:48 dhcp155 kernel:  [<c06f59ee>] start_kernel+0x379/0x380
Sep 18 10:58:48 dhcp155 kernel:  =======================


in dmesg




[-- Attachment #2: Type: text/html, Size: 3137 bytes --]

Re: [Qemu-devel] Qemu savevm and CPU soft lockup

[Benjamin Cleyet-Marrel]

[-- Attachment #1: Type: text/plain, Size: 3385 bytes --]

Hi, 

After further investigation, I figured out that when issuing a savevm
command
the entire qemu process gets stuck on IO wait.
I can't issue any other commands on the monitor the process is shown as
D and the guest is in softlockup state.

Looking at the way migrate (and the -d for detach I presume) is working
I suppose the same behaviour would be expected from the savevm function.

A savevm -d so that the qemu process would not be freezed while saving
the data.

Sorry if I am just talking non sense but my snapshot on iscsi storage
takes about 1 minutes. 
which means that my guest are down for a minute or so which is not
ideal.

Thanks for your time
Cheers
Ben


-------- Message initial --------
De: Benjamin Cleyet-Marrel <bcm@accelance.fr>
À: qemu-devel@nongnu.org
Sujet: [Qemu-devel] Qemu savevm and CPU soft lockup
Date: Fri, 18 Sep 2009 11:56:41 +0200

Hi, 

First post to this list so sorry if I am mistaken.
I am using qemu-kvm-0.11.0-rc2 I've googled quite a lot and could not
find anyone having or reporting the same problem.
The guest is a Centos5.3

The savevm/loadvm seems to be broken:

When issuing a savevm in the monitor, 
Guest get stuck for a few seconds and I get the infamous soft lockup a
couple of time. (see bellow)

In the end, the snapshot is working.

When doing a loadvm I often get weird behaviour on the console.
this can be completly removed if I issue a stop and cont before and
after the loadvm.

Question is:
Am I doing something wrong or is this a bug ?

Thanks
Cheers
Ben



Sep 18 10:58:48 dhcp155 kernel: BUG: soft lockup - CPU#0 stuck for 10s!
[swapper:0]
Sep 18 10:58:48 dhcp155 kernel: 
Sep 18 10:58:48 dhcp155 kernel: Pid: 0, comm:              swapper
Sep 18 10:58:48 dhcp155 kernel: EIP: 0060:[<c044d219>] CPU: 0
Sep 18 10:58:48 dhcp155 kernel: EIP is at handle_IRQ_event+0x39/0x8c
Sep 18 10:58:48 dhcp155 kernel:  EFLAGS: 00000246    Not tainted
(2.6.18-128.el5 #1)
Sep 18 10:58:48 dhcp155 kernel: EAX: 00000001 EBX: c06e6f00 ECX:
f7c9bf80 EDX: c0754f9c
Sep 18 10:58:48 dhcp155 kernel: ESI: f7c9bf80 EDI: 00000001 EBP:
00000000 DS: 007b ES: 007b
Sep 18 10:58:48 dhcp155 kernel: CR0: 8005003b CR2: 080c3e80 CR3:
376ad000 CR4: 00000690
Sep 18 10:58:48 dhcp155 kernel:  [<c044d2f0>] __do_IRQ+0x84/0xd6
Sep 18 10:58:48 dhcp155 kernel:  [<c04074e5>] do_IRQ+0xb0/0xc3
Sep 18 10:58:48 dhcp155 kernel:  [<c0405946>] common_interrupt+0x1a/0x20
Sep 18 10:58:48 dhcp155 kernel:  [<c044d219>] handle_IRQ_event+0x39/0x8c
Sep 18 10:58:48 dhcp155 kernel:  [<c044d2f0>] __do_IRQ+0x84/0xd6
Sep 18 10:58:48 dhcp155 kernel:  [<c044d26c>] __do_IRQ+0x0/0xd6
Sep 18 10:58:48 dhcp155 kernel:  [<c04074ce>] do_IRQ+0x99/0xc3
Sep 18 10:58:48 dhcp155 kernel:  [<c0405946>] common_interrupt+0x1a/0x20
Sep 18 10:58:48 dhcp155 kernel:  [<c0428ba7>] __do_softirq+0x57/0x114
Sep 18 10:58:48 dhcp155 kernel:  [<c04073eb>] do_softirq+0x52/0x9c
Sep 18 10:58:48 dhcp155 kernel:  [<c04059d7>] apic_timer_interrupt
+0x1f/0x24
Sep 18 10:58:48 dhcp155 kernel:  [<c0403bb0>] default_idle+0x0/0x59
Sep 18 10:58:48 dhcp155 kernel:  [<c0403be1>] default_idle+0x31/0x59
Sep 18 10:58:48 dhcp155 kernel:  [<c0403ca8>] cpu_idle+0x9f/0xb9
Sep 18 10:58:48 dhcp155 kernel:  [<c06f59ee>] start_kernel+0x379/0x380
Sep 18 10:58:48 dhcp155 kernel:  =======================


in dmesg




[-- Attachment #2: Type: text/html, Size: 4257 bytes --]

Re: [Qemu-devel] Qemu savevm and CPU soft lockup

[Jamie Lokier]

Benjamin Cleyet-Marrel wrote:
> 
>    Hi,
>    After further investigation, I figured out that when issuing a savevm
>    command
>    the entire qemu process gets stuck on IO wait.
>    I can't issue any other commands on the monitor the process is shown
>    as D and the guest is in softlockup state.
>    Looking at the way migrate (and the -d for detach I presume) is
>    working I suppose the same behaviour would be expected from the savevm
>    function.
>    A savevm -d so that the qemu process would not be freezed while saving
>    the data.
>    Sorry if I am just talking non sense but my snapshot on iscsi storage
>    takes about 1 minutes.
>    which means that my guest are down for a minute or so which is not
>    ideal.

This is normal savevm behaviour, and it is exactly the reason why
migrate-to-file is useful.  I would not be surprised if savevm is
changed to use migrate-to-file internally at some point, but it does
not look like happening soon.

You might avoid the guest softlockup state by stopping the guest
("stop" command) before savevm, and "cont" afterwards?

Or the guest might get just as confused, as the clock still advances.

-- Jamie

Re: [Qemu-devel] Qemu savevm and CPU soft lockup

[Ben Accelance]

Le 23 sept. 2009 à 20:19, Jamie Lokier <jamie@shareable.org> a écrit :

> Benjamin Cleyet-Marrel wrote:
>>
>>   Hi,
>>   After further investigation, I figured out that when issuing a  
>> savevm
>>   command
>>   the entire qemu process gets stuck on IO wait.
>>   I can't issue any other commands on the monitor the process is  
>> shown
>>   as D and the guest is in softlockup state.
>>   Looking at the way migrate (and the -d for detach I presume) is
>>   working I suppose the same behaviour would be expected from the  
>> savevm
>>   function.
>>   A savevm -d so that the qemu process would not be freezed while  
>> saving
>>   the data.
>>   Sorry if I am just talking non sense but my snapshot on iscsi  
>> storage
>>   takes about 1 minutes.
>>   which means that my guest are down for a minute or so which is not
>>   ideal.
>
> This is normal savevm behaviour, and it is exactly the reason why
> migrate-to-file is useful.  I would not be surprised if savevm is
> changed to use migrate-to-file internally at some point, but it does
> not look like happening soon.
>
> You might avoid the guest softlockup state by stopping the guest
> ("stop" command) before savevm, and "cont" afterwards?
>
> Or the guest might get just as confused, as the clock still advances.
>
> -- Jamie
>
>
Thanks for this, at least it clarifies things. Now sorry for the  
pain , but how do you migrate to a file and then how do you restore  
from the file ?
Thanks
Cheers

Benjamin Cleyet-Marrel

Re: [Qemu-devel] Qemu savevm and CPU soft lockup

[Nathan Baum]

On Wed, 2009-09-23 at 20:52 +0200, Ben Accelance wrote:
> 
> 
> Le 23 sept. 2009 à 20:19, Jamie Lokier <jamie@shareable.org> a écrit :
> 
> > Benjamin Cleyet-Marrel wrote:
> >>
> >>   Hi,
> >>   After further investigation, I figured out that when issuing a  
> >> savevm
> >>   command
> >>   the entire qemu process gets stuck on IO wait.
> >>   I can't issue any other commands on the monitor the process is  
> >> shown
> >>   as D and the guest is in softlockup state.
> >>   Looking at the way migrate (and the -d for detach I presume) is
> >>   working I suppose the same behaviour would be expected from the  
> >> savevm
> >>   function.
> >>   A savevm -d so that the qemu process would not be freezed while  
> >> saving
> >>   the data.
> >>   Sorry if I am just talking non sense but my snapshot on iscsi  
> >> storage
> >>   takes about 1 minutes.
> >>   which means that my guest are down for a minute or so which is not
> >>   ideal.
> >
> > This is normal savevm behaviour, and it is exactly the reason why
> > migrate-to-file is useful.  I would not be surprised if savevm is
> > changed to use migrate-to-file internally at some point, but it does
> > not look like happening soon.
> >
> > You might avoid the guest softlockup state by stopping the guest
> > ("stop" command) before savevm, and "cont" afterwards?
> >
> > Or the guest might get just as confused, as the clock still advances.
> >
> > -- Jamie
> >
> >
> Thanks for this, at least it clarifies things. Now sorry for the  
> pain , but how do you migrate to a file and then how do you restore  
> from the file ?
> Thanks
> Cheers

The way I migrate-to-file is with

  (qemu) migrate -d "exec:cat>file".

and then restore with

  $ qemu ... -incoming "exec:cat file"

Re: [Qemu-devel] Qemu savevm and CPU soft lockup

[Benjamin Cleyet-Marrel]

[-- Attachment #1: Type: text/plain, Size: 3649 bytes --]





-------- Message initial --------
De: Nathan Baum <nathan@parenthephobia.org.uk>
À: Ben Accelance <bcm@accelance.fr>
Cc: qemu-devel@nongnu.org <qemu-devel@nongnu.org>
Sujet: Re: [Qemu-devel] Qemu savevm and CPU soft lockup
Date: Wed, 23 Sep 2009 23:21:00 +0100


On Wed, 2009-09-23 at 20:52 +0200, Ben Accelance wrote:
> 
> 
> Le 23 sept. 2009 à 20:19, Jamie Lokier <jamie@shareable.org> a écrit :
> 
> > Benjamin Cleyet-Marrel wrote:
> >>
> >>   Hi,
> >>   After further investigation, I figured out that when issuing a  
> >> savevm
> >>   command
> >>   the entire qemu process gets stuck on IO wait.
> >>   I can't issue any other commands on the monitor the process is  
> >> shown
> >>   as D and the guest is in softlockup state.
> >>   Looking at the way migrate (and the -d for detach I presume) is
> >>   working I suppose the same behaviour would be expected from the  
> >> savevm
> >>   function.
> >>   A savevm -d so that the qemu process would not be freezed while  
> >> saving
> >>   the data.
> >>   Sorry if I am just talking non sense but my snapshot on iscsi  
> >> storage
> >>   takes about 1 minutes.
> >>   which means that my guest are down for a minute or so which is not
> >>   ideal.
> >
> > This is normal savevm behaviour, and it is exactly the reason why
> > migrate-to-file is useful.  I would not be surprised if savevm is
> > changed to use migrate-to-file internally at some point, but it does
> > not look like happening soon.
> >
> > You might avoid the guest softlockup state by stopping the guest
> > ("stop" command) before savevm, and "cont" afterwards?
> >
> > Or the guest might get just as confused, as the clock still advances.
> >
> > -- Jamie
> >
> >
> Thanks for this, at least it clarifies things. Now sorry for the  
> pain , but how do you migrate to a file and then how do you restore  
> from the file ?
> Thanks
> Cheers

>The way I migrate-to-file is with
>
>  (qemu) migrate -d "exec:cat>file".
>
>and then restore with
>
>  $ qemu ... -incoming "exec:cat file"
>



Hi, 

I finaly found some time to investigate deeper on this.
this migrate_to_file is definitly not cutting it.
I am trying to do a daily snapshot of my vm without any interruption.
the migrate to file by itself  is not doing an synchroneous disque image
snapshot (provided that image file are qcow2)
so restoring the status without the proper data is useless.
The only way I found is to embedd all the save function and qemu-img
snapshot -c call in a shell script. but it is not very clean I would
say.

the dosavevm script  looklike this : 

#!/bin/sh
qemu-img snapshot -c statefile /vm/img/test2_20G_vdc.img &
qemu-img snapshot -c statefile /vm/img/test2_2G_swap.img &
qemu-img snapshot -c statefile /vm/img/test2_5G_vda.img &
cat  > /tmp/test.statefile


and to save my disk I do: 

#!/bin/sh
echo -e "\001c\nmigrate -d \"exec:/root/dosavevm\"\n\001c\n" | socat
STDIO UNIX:/var/run/libvirt/qemu/test.serial 
satus=""
while [ -z "$status" ]
do
	status=`echo -e "\001c\ninfo migrate\n\001c\n" | socat STDIO
UNIX:/var/run/libvirt/qemu/test.serial | grep completed`
	sleep 1
done
echo -e "\001c\ncont\n\001c\n" | socat STDIO
UNIX:/var/run/libvirt/qemu/test.serial 


Which is as hugly as it can get.
Finnaly restoring the data needs:
shutdown the guest 
applying the snapshot 
starting a new process with the correct parameters 

So back to my original problem.

Is there a way to perform a consitent backup of kvm vm without 60s
downtime an hugly CPU soft lockup ?

Thanks

Cheers
Ben 





[-- Attachment #2: Type: text/html, Size: 4785 bytes --]

Re: [Qemu-devel] Qemu savevm and CPU soft lockup

[Anthony Liguori]

Jamie Lokier wrote:
> Benjamin Cleyet-Marrel wrote:
>   
>>    Hi,
>>    After further investigation, I figured out that when issuing a savevm
>>    command
>>    the entire qemu process gets stuck on IO wait.
>>    I can't issue any other commands on the monitor the process is shown
>>    as D and the guest is in softlockup state.
>>    Looking at the way migrate (and the -d for detach I presume) is
>>    working I suppose the same behaviour would be expected from the savevm
>>    function.
>>    A savevm -d so that the qemu process would not be freezed while saving
>>    the data.
>>    Sorry if I am just talking non sense but my snapshot on iscsi storage
>>    takes about 1 minutes.
>>    which means that my guest are down for a minute or so which is not
>>    ideal.
>>     
>
> This is normal savevm behaviour, and it is exactly the reason why
> migrate-to-file is useful.  I would not be surprised if savevm is
> changed to use migrate-to-file internally at some point, but it does
> not look like happening soon.
>   

It's the same infrastructure.  The reason savevm isn't live is that 
savevm stores it's data in a qcow2 file.  Right now the way qcow2 is 
structured, the snapshot has to be a fixed size and allocated at once.  
In order to make savevm live, we need a method to stream savevm data to 
a qcow2 file while still allowing other IO operations to that qcow2 file.

I'm fairly sure this will require a change to the qcow2 format in order 
to support this.

Regards,

Anthony Liguori

Re: [Qemu-devel] Qemu savevm and CPU soft lockup

[Kevin Wolf]

Am 24.09.2009 00:28, schrieb Anthony Liguori:
> Jamie Lokier wrote:
>> This is normal savevm behaviour, and it is exactly the reason why
>> migrate-to-file is useful.  I would not be surprised if savevm is
>> changed to use migrate-to-file internally at some point, but it does
>> not look like happening soon.
>>   
> 
> It's the same infrastructure.  The reason savevm isn't live is that 
> savevm stores it's data in a qcow2 file.  Right now the way qcow2 is 
> structured, the snapshot has to be a fixed size and allocated at once.  
> In order to make savevm live, we need a method to stream savevm data to 
> a qcow2 file while still allowing other IO operations to that qcow2 file.
> 
> I'm fairly sure this will require a change to the qcow2 format in order 
> to support this.

Hm, snapshots are nothing complicated from qcow2 perspective. Why do you
think data needs to be fixed size? In qcow2 a snapshot consists of some
fixed-size metadata plus the save/load_vmstate area which is just disk
space after the end of the virtual disk and could be extended in theory
until the 64 bits of the offset are used up.

I'm not sure what the internal interfaces look like and if they would
need to be changed, but for the format I don't see any problems. You
only need to set the right VM state size in the snapshot metadata when
you are done.

Kevin

Re: [Qemu-devel] Qemu savevm and CPU soft lockup

[Anthony Liguori]

Kevin Wolf wrote:
> Am 24.09.2009 00:28, schrieb Anthony Liguori:
>   
>> Jamie Lokier wrote:
>>     
>>> This is normal savevm behaviour, and it is exactly the reason why
>>> migrate-to-file is useful.  I would not be surprised if savevm is
>>> changed to use migrate-to-file internally at some point, but it does
>>> not look like happening soon.
>>>   
>>>       
>> It's the same infrastructure.  The reason savevm isn't live is that 
>> savevm stores it's data in a qcow2 file.  Right now the way qcow2 is 
>> structured, the snapshot has to be a fixed size and allocated at once.  
>> In order to make savevm live, we need a method to stream savevm data to 
>> a qcow2 file while still allowing other IO operations to that qcow2 file.
>>
>> I'm fairly sure this will require a change to the qcow2 format in order 
>> to support this.
>>     
>
> Hm, snapshots are nothing complicated from qcow2 perspective. Why do you
> think data needs to be fixed size?

What happens if you're in the middle of writing snapshot data and 
another cluster has to be allocated?  You'll need a way to store the 
snapshot data discontinuously.

Regards,

Anthony Liguori

Re: [Qemu-devel] Qemu savevm and CPU soft lockup

[Kevin Wolf]

Am 24.09.2009 18:20, schrieb Anthony Liguori:
> Kevin Wolf wrote:
>> Am 24.09.2009 00:28, schrieb Anthony Liguori:
>>   
>>> Jamie Lokier wrote:
>>>     
>>>> This is normal savevm behaviour, and it is exactly the reason why
>>>> migrate-to-file is useful.  I would not be surprised if savevm is
>>>> changed to use migrate-to-file internally at some point, but it does
>>>> not look like happening soon.
>>>>   
>>>>       
>>> It's the same infrastructure.  The reason savevm isn't live is that 
>>> savevm stores it's data in a qcow2 file.  Right now the way qcow2 is 
>>> structured, the snapshot has to be a fixed size and allocated at once.  
>>> In order to make savevm live, we need a method to stream savevm data to 
>>> a qcow2 file while still allowing other IO operations to that qcow2 file.
>>>
>>> I'm fairly sure this will require a change to the qcow2 format in order 
>>> to support this.
>>>     
>>
>> Hm, snapshots are nothing complicated from qcow2 perspective. Why do you
>> think data needs to be fixed size?
> 
> What happens if you're in the middle of writing snapshot data and 
> another cluster has to be allocated?  You'll need a way to store the 
> snapshot data discontinuously.

Well, mapping virtually continuous data to discontinuous areas in the
image file is just how qcow2 works, right?

If you look at qcow_vmstate_load/save(offset), this is basically just a
call to bdrv_pread/pwrite(virtual_disk_size + offset). So for snapshots
it works exactly the same way as with regular data (which usually isn't
preallocated in qcow2 images).

Kevin

Re: [Qemu-devel] Qemu savevm and CPU soft lockup

[Ben Accelance]

Benjamin Cleyet-Marrel


Le 25 sept. 2009 à 09:40, Kevin Wolf <kwolf@redhat.com> a écrit :

> Am 24.09.2009 18:20, schrieb Anthony Liguori:
>> Kevin Wolf wrote:
>>> Am 24.09.2009 00:28, schrieb Anthony Liguori:
>>>
>>>> Jamie Lokier wrote:
>>>>
>>>>> This is normal savevm behaviour, and it is exactly the reason why
>>>>> migrate-to-file is useful.  I would not be surprised if savevm is
>>>>> changed to use migrate-to-file internally at some point, but it  
>>>>> does
>>>>> not look like happening soon.
>>>>>
>>>>>
>>>> It's the same infrastructure.  The reason savevm isn't live is that
>>>> savevm stores it's data in a qcow2 file.  Right now the way qcow2  
>>>> is
>>>> structured, the snapshot has to be a fixed size and allocated at  
>>>> once.
>>>> In order to make savevm live, we need a method to stream savevm  
>>>> data to
>>>> a qcow2 file while still allowing other IO operations to that  
>>>> qcow2 file.
>>>>
>>>> I'm fairly sure this will require a change to the qcow2 format in  
>>>> order
>>>> to support this.
>>>>
>>>
>>> Hm, snapshots are nothing complicated from qcow2 perspective. Why  
>>> do you
>>> think data needs to be fixed size?
>>
>> What happens if you're in the middle of writing snapshot data and
>> another cluster has to be allocated?  You'll need a way to store the
>> snapshot data discontinuously.
>
> Well, mapping virtually continuous data to discontinuous areas in the
> image file is just how qcow2 works, right?
>
> If you look at qcow_vmstate_load/save(offset), this is basically  
> just a
> call to bdrv_pread/pwrite(virtual_disk_size + offset). So for  
> snapshots
> it works exactly the same way as with regular data (which usually  
> isn't
> preallocated in qcow2 images).
>
> Kevin

Thanks for this information,
Having a live save function would be tremendous and a huge gain  
compare to other solutions.
The migrate to file using exec is just not cutting it as I have to be  
in sync with image snapshot.
Having said that I am still not sure I understand why savevm isn't  
live . or should it be ? And then it is a bug ?

Cheers
And thanks for your Time.

Ben