[libvirt] qemu-kvm-0.10.5, kvm-kmod-2.6.30, libvirt-0.6.4 - save/restore still unreliable

[libvirt] qemu-kvm-0.10.5, kvm-kmod-2.6.30, libvirt-0.6.4 - save/restore still unreliable

[Nikola Ciprich]

Hi,
I'm very sorry for crossposting, but I'm not sure whether my problem
is related to libvirt or kvm itself.
I've spent some time testing save/restore and it seems it's still not 
really reliable. Starting guest, loading it with some applications 
(ie kernel compilation) and saving/restoring it for few times usually leads to error:

[root@vbox4 hibernated]# virsh save vmroo55 /home/LVS/hibernated/vmroo55.vm
Domain vmroo55 saved to /home/LVS/hibernated/vmroo55.vm

[root@vbox4 hibernated]# virsh restore /home/LVS/hibernated/vmroo55.vm
Domain restored from /home/LVS/hibernated/vmroo55.vm

[root@vbox4 hibernated]# virsh save vmroo55 /home/LVS/hibernated/vmroo55.vm
Domain vmroo55 saved to /home/LVS/hibernated/vmroo55.vm

[root@vbox4 hibernated]# virsh restore /home/LVS/hibernated/vmroo55.vm
error: Failed to restore domain from /home/LVS/hibernated/vmroo55.vm
error: internal error unable to start guest: char device redirected to /dev/pts/2

in /var/log/messages, i see:
Jun 12 15:18:30 vbox4 libvirtd: 15:18:30.793: error : qemudReadMonitorOutput:696 : internal error Timed out while reading monitor startup output
Jun 12 15:18:30 vbox4 libvirtd: 15:18:30.820: error : qemudWaitForMonitor:994 : internal error unable to start guest: char device redirected to /dev/pts/2

/var/log/libvirt/vmroo55.log:
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/ /usr/bin/qemu-kvm -S -M pc -m 768 -smp 2 -name vmroo55 -uuid 987e7a85-0e15-11de-a522-00304892f672 -monitor pty -boot c -kernel /home/LVS/boot/vmlinuz-2.6.29lb.00_01_PRE14-x86_64 -initrd /home/LVS/boot/kvm-initrd-2.6.29lb.00_01_PRE14-x86_64.cpio.gz -append root=/dev/sda ro vmhostname=vmroo55 -drive file=/dev/vgshared/vmroo55-1,if=ide,index=0,boot=on -drive file=/dev/vgshared/vmroo55-2,if=ide,index=1 -drive file=/dev/vgshared/vmroo55-3,if=virtio,index=0 -net nic,macaddr=00:16:3e:18:37:00,vlan=0,model=virtio -net tap,fd=15,vlan=0 -serial none -parallel none -usb -vnc 0.0.0.0:24155 -incoming exec:cat
char device redirected to /dev/pts/2
qemu_popen: returning result of qemu_fopen_ops
qemu_popen: returning result of qemu_fopen_ops

I wanted to try it using just kvm, but for some reason, I can't figure the proper way to execute it, is the
-incoming exec:cat somewhere properly documented? I tried doing cat .../img.vm | qemu-kvm ... -incoming exec:cat
but this doesn't seem to be the proper way.

Could somebody give me a hint on how to debug this?

thanks a lot in advance!

nik

-- 
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: [libvirt] qemu-kvm-0.10.5, kvm-kmod-2.6.30, libvirt-0.6.4 - save/restore still unreliable

[Charles Duffy]

Nikola Ciprich wrote:
> I wanted to try it using just kvm, but for some reason, I can't figure the proper way to execute it, is the
> -incoming exec:cat somewhere properly documented? I tried doing cat .../img.vm | qemu-kvm ... -incoming exec:cat
> but this doesn't seem to be the proper way.

That should work fine. Your "..." sets up the drives, memory amount, 
etc., right? Why do you say it "doesn't seem to be the proper way"? What 
fails?

BTW, you might also do this: -incoming 'exec:cat .../img.vm', avoiding 
the need to pipe through stdin.

Re: [libvirt] qemu-kvm-0.10.5, kvm-kmod-2.6.30, libvirt-0.6.4 - save/restore still unreliable

[Nikola Ciprich]

Hello Charlie,
thanks for the reply
> That should work fine. Your "..." sets up the drives, memory amount,  
> etc., right? Why do you say it "doesn't seem to be the proper way"? What  
> fails?
sure, I'm (hopefully) providing all the needed arguments.

>
> BTW, you might also do this: -incoming 'exec:cat .../img.vm', avoiding  
> the need to pipe through stdin.
yup, I tried that, but it allways ends with following:

cat  /home/LVS/hibernated/vmroo55.vm | /usr/bin/qemu-kvm -M pc -m 768 -smp 2 -name vmroo55 -uuid 987e7a85-0e15-11de-a522-00304892f672 -monitor pty -boot c -kernel /home/LVS/boot/vmlinuz-2.6.29lb.00_01_PRE14-x86_64 -initrd /home/LVS/boot/kvm-initrd-2.6.29lb.00_01_PRE14-x86_64.cpio.gz -append "root=/dev/sda ro vmhostname=vmroo55" -drive file=/dev/vgshared/vmroo55-1,if=ide,index=0,boot=on -drive file=/dev/vgshared/vmroo55-2,if=ide,index=1 -drive file=/dev/vgshared/vmroo55-3,if=virtio,index=0 -net nic,macaddr=00:16:3e:18:37:00,vlan=0,model=virtio -net tap,script=/etc/qemu-ifup.sh,vlan=0 -serial none -parallel none -usb -vnc0.0.0.0:24155 -incoming exec:cat

Executing /etc/qemu-ifup
Bringing up tap0 for bridged mode...
Adding tap0 to br0...
char device redirected to /dev/pts/7
qemu_popen: returning result of qemu_fopen_ops
load of migration failed
cat: write error: Broken pipe
/etc/qemu-ifdown: could not launch network script


using 
... -incoming exec:cat /home/...image.vm
seems to start the machine, but it seems to be stuck, even though I removed -S parameter.

>
> --
> Libvir-list mailing list
> Libvir-list@redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list
>

-- 
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: [libvirt] qemu-kvm-0.10.5, kvm-kmod-2.6.30, libvirt-0.6.4 - save/restore still unreliable

[Charles Duffy]

Nikola Ciprich wrote:
> using 
> ... -incoming exec:cat /home/...image.vm
> seems to start the machine, but it seems to be stuck, even though I removed -S parameter.

That may well be a guest issue rather than a qemu migration problem -- 
I've seen similar difficulties for guests being resumed via incoming 
migration when using guest kernels with TSC as their clocksource.

You might try interrogating the monitor to see your guest's state.

Re: [libvirt] qemu-kvm-0.10.5, kvm-kmod-2.6.30, libvirt-0.6.4 - save/restore still unreliable

[Nikola Ciprich]

Hmm,
I just noticed that dump file saved using virsh contains some libvirt 
related headers, so I guess I can't just feed it directly to qemu-kvm.
So I guess best way to test save/restore without libvirt is to save 
vm state using monitor and then restore it again.
But I can't find any clear documentation on how to do this.
Could anyone give me a hint?
thanks a lot!
nik


On Fri, Jun 12, 2009 at 03:17:18PM -0500, Charles Duffy wrote:
> That may well be a guest issue rather than a qemu migration problem --  
> I've seen similar difficulties for guests being resumed via incoming  
> migration when using guest kernels with TSC as their clocksource.
>
> You might try interrogating the monitor to see your guest's state.
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: [libvirt] qemu-kvm-0.10.5, kvm-kmod-2.6.30, libvirt-0.6.4 - save/restore still unreliable

[Avi Kivity]

Nikola Ciprich wrote:
> Hmm,
> I just noticed that dump file saved using virsh contains some libvirt 
> related headers, so I guess I can't just feed it directly to qemu-kvm.
> So I guess best way to test save/restore without libvirt is to save 
> vm state using monitor and then restore it again.
> But I can't find any clear documentation on how to do this.
> Could anyone give me a hint?
> thanks a lot!
>   


There's savevm and loadvm in the monitor.

-- 
error compiling committee.c: too many arguments to function

Re: [libvirt] qemu-kvm-0.10.5, kvm-kmod-2.6.30, libvirt-0.6.4 -save/restore still unreliable

[Nikola Ciprich]

Hello Charles,

> Otherwise -- at the monitor, migrate "exec:cat >somefile.out"
I tried that, saving seems to work OK,

but if I restore the domain, kvm always seems to be stuck, I'm not able to swith
to monitor to do anything. running strace on the kvm process show that
following is repeating:

clock_gettime(CLOCK_MONOTONIC, {453254, 437209647}) = 0
clock_gettime(CLOCK_MONOTONIC, {453254, 437244463}) = 0
clock_gettime(CLOCK_MONOTONIC, {453254, 437295938}) = 0
select(20, [5 10 17 19], [], [], {1, 0}) = 1 (in [5], left {0, 999998})
read(5, "\0", 512)                      = 1
read(5, 0x7fff60506840, 512)            = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_MONOTONIC, {453254, 437458602}) = 0
clock_gettime(CLOCK_MONOTONIC, {453254, 437492440}) = 0
timer_gettime(0, {it_interval={0, 0}, it_value={0, 0}}) = 0
timer_settime(0, 0, {it_interval={0, 0}, it_value={0, 30000000}}, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {453254, 437605596}) = 0
clock_gettime(CLOCK_MONOTONIC, {453254, 437639489}) = 0
select(20, [5 10 17 19], [], [], {1, 0}) = 1 (in [19], left {0, 970104})
read(19, "\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 128) = 128
rt_sigaction(SIGALRM, NULL, {0x4083b0, ~[KILL STOP RTMIN RT_1], SA_RESTORER, 0x7f25576e94c0}, 8) = 0
write(6, "\0", 1)                       = 1
read(19, 0x7fff605069c0, 128)           = -1 EAGAIN (Resource temporarily unavailable)

I guess I maybe need to unfreeze the domain somehow after loading it?

> 
> Replace cat with gzip -c or other compression tool (and reverse on
> migrate in) for more efficient on-disk storage.
> 
> -----Original Message-----
> From: Nikola Ciprich [mailto:extmaillist@linuxbox.cz] 
> Sent: Saturday, June 13, 2009 1:32 AM
> To: CHARLESDUFFY
> Cc: kvm@vger.kernel.org; nikola.ciprich@linuxbox.cz
> Subject: Re: [libvirt] qemu-kvm-0.10.5, kvm-kmod-2.6.30, libvirt-0.6.4
> -save/restore still unreliable
> 
> Hmm,
> I just noticed that dump file saved using virsh contains some libvirt 
> related headers, so I guess I can't just feed it directly to qemu-kvm.
> So I guess best way to test save/restore without libvirt is to save 
> vm state using monitor and then restore it again.
> But I can't find any clear documentation on how to do this.
> Could anyone give me a hint?
> thanks a lot!
> nik
> 
> 
> On Fri, Jun 12, 2009 at 03:17:18PM -0500, Charles Duffy wrote:
> > That may well be a guest issue rather than a qemu migration problem --
> 
> > I've seen similar difficulties for guests being resumed via incoming  
> > migration when using guest kernels with TSC as their clocksource.
> >
> > You might try interrogating the monitor to see your guest's state.
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> -- 
> -------------------------------------
> Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28. rijna 168, 709 01 Ostrava
> 
> tel.:   +420 596 603 142
> fax:    +420 596 621 273
> mobil:  +420 777 093 799
> 
> www.linuxbox.cz
> 
> mobil servis: +420 737 238 656
> email servis: servis@linuxbox.cz
> -------------------------------------
> 

-- 
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------