XenCD xm save problem

XenCD xm save problem

[Jared Rhine]

[Paul == i_p_a_u_l@yahoo.com on Tue, 08 Feb 2005 01:09:57 +0200]

  Paul> I tried with XenCD 1.0 rc1 in both vmware environments, and it
  Paul> did not work, but I suspect there is something else fishy,
  Paul> because "xm save" did not work either.

Thanks for the report.  I can duplicate the problem that xm save does
not work on XenCD.  Currently, I'm treating this as a release blocker
for 1.0final.

I'm unable at determine a fix for the problem and am soliciting help
from anyone who thinks they could contribute.

After a fresh boot of XenCD 1.0rc01, if I do:

  xm save 1 /tmp/ttylinux1.save

I get:

  Error: Error: [Failure instance: Traceback: \
    xen.xend.XendError.XendError, save failed

The /var/log/xend-debug.log in enclosed below.  I see:

  /dev/loop: Is a directory
  ioctl: LOOP_SET_FD: Device or resource busy
  ioctl: LOOP_SET_FD: Device or resource busy

Relevant to that, I'm using udev with a plain Debian sarge install.
/dev/loop does indeed exist as a directory.  My other udev box,
Gentoo, also has a /dev/loop directory, with 8 block devices inside.
XenCD has on 1 block device.  My Debian woody devfs box has no
/dev/loop directory entry.

dom0 is booted with max_loop=64.

Oddly, there are possible loop leaks in my startup as there are
processes and lsof entries for loop0, loop1, loop2, loop3, loop10, and
loop11, which seems like a few too many.  Only ttylinux-1 and 2's
loopback rootfs should be using up loops, I think.

Possibilities:

Do I need to patch sarge's udev to create a /dev/loop device?

Should I be digging into xend code to see how it uses /dev/loop?  I'm
still wrapping my head around xend and I can't find "loop" in a grep
of the tree.

Is there something about tmpfs that makes xm save work for others but
not XenCD?  XenCD's Xen installation is as absolutely vanilla as I can
make it, other than tmpfs.  And ttylinux is running on a loopback
ext2.

Apologies if I'm missing something obvious.

-- begin /var/log/xend-debug.log --

network start bridge=xen-br0 netdev=eth0 antispoof=no
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c034f6c4
 Init. ramdisk: c0350000->c0350000
 Phys-Mach map: c0350000->c0358000
 Page tables:   c0358000->c035a000
 Start info:    c035a000->c035b000
 Boot stack:    c035b000->c035c000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
 VCPUS:         1
/dev/loop: Is a directory
ioctl: LOOP_SET_FD: Device or resource busy
ioctl: LOOP_SET_FD: Device or resource busy
vif-bridge up vif=vif1.0 domain=ttylinux-1 mac=aa:00:00:69:13:b5 bridge=xen-br0
recv_fe_driver_status> {'status': 1}

recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 23907, 'rx_shmem_frame': 23906, 'handle': 0}
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0100000->c034f6c4
 Init. ramdisk: c0350000->c0350000
 Phys-Mach map: c0350000->c0358000
 Page tables:   c0358000->c035a000
 Start info:    c035a000->c035b000
 Boot stack:    c035b000->c035c000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0100000
 VCPUS:         1
/dev/loop: Is a directory
ioctl: LOOP_SET_FD: Device or resource busy
ioctl: LOOP_SET_FD: Device or resource busy
ioctl: LOOP_SET_FD: Device or resource busy
vif-bridge up vif=vif2.0 domain=ttylinux-2 mac=aa:00:00:56:44:c6 bridge=xen-br0
recv_fe_driver_status> {'status': 1}

recv_fe_driver_status>

recv_fe_interface_connect {'tx_shmem_frame': 5204, 'rx_shmem_frame': 5203, 'handle': 0}
sync_session> <type 'str'> 1 ['save', ['id', '1'], ['state', 'begin'], ['domain', '1'], ['file', '/tmp/ttylinux1.save']]
Started to connect self= <xen.xend.XendMigrate.XfrdClientFactory instance at 0xb78fbdec> connector= <twisted.internet.tcp.Connector instance at 0xb78fbe0c>
buildProtocol> IPv4Address(TCP, 'localhost', 8002)
***request> (domain (id 1) (name ttylinux-1) (memory 31) (maxmem 32768) (state -b---) (cpu 0) (cpu_time 81.206402435) (up_time 59103.1672821) (start_time 1107920300.57) (console (status listening) (id 12) (domain 1) (local_port 12) (remote_port 1) (console_port 9601)) (devices (vif (idx 0) (vif 0) (mac aa:00:00:69:13:b5) (evtchn 14 4) (index 0)) (vbd (idx 0) (vdev 2049) (device 1802) (mode w) (dev sda1) (uname file:/tmp/xen-rootfs.ttylinux-1) (node /dev/loop10) (index 0))) (config (vm (name ttylinux-1) (memory 32) (cpu -1) (restart always) (image (linux (kernel /media/cdrom/boot/xenU-kernel) (root '/dev/sda1 ro') (vcpus 1))) (device (vbd (uname file:/tmp/xen-rootfs.ttylinux-1) (dev sda1) (mode w))) (device (vif (mac aa:00:00:69:13:b5))) (memmap) (device_model) (device_config))))
***request> begin
xfr_err> ['xfr.err', '0']
xfr_err> <type 'str'> 0
Xfrd>connectionLost> [Failure instance: Traceback: twisted.internet.error.ConnectionDone, Connection was closed cleanly.
]
XfrdSaveInfo>connectionLost> [Failure instance: Traceback: twisted.internet.error.ConnectionDone, Connection was closed cleanly.
]
XfrdInfo>connectionLost> [Failure instance: Traceback: twisted.internet.error.ConnectionDone, Connection was closed cleanly.
]
Error> save failed
Error> calling errback
***cbremove> [Failure instance: Traceback: xen.xend.XendError.XendError, save failed
]
***_delete_session> 1
clientConnectionLost> connector= <twisted.internet.tcp.Connector instance at 0xb78fbe0c> reason= [Failure instance: Traceback: twisted.internet.error.ConnectionDone, Connection was closed cleanly.
]

-- end --

-- jared@wordzoo.com

"A black hole is where God is dividing by zero."
        -- attributed to Roger Smith


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

XenCD xm save problem

[Jared Rhine]

[Jared == jared@wordzoo.com on Wed, 09 Feb 2005 16:05:18 -0800]

  Jared> I can duplicate the problem that xm save does not work on
  Jared> XenCD.

Others have reported success after switching to Xen 2.0.4, but my
XenCD save problem still exists with a 2.0.4-built snapshot.

From my last email:

-- begin --

Possibilities:

Do I need to patch sarge's udev to create a /dev/loop device?

Should I be digging into xend code to see how it uses /dev/loop?  I'm
still wrapping my head around xend and I can't find "loop" in a grep
of the tree.

Is there something about tmpfs that makes xm save work for others but
not XenCD?  XenCD's Xen installation is as absolutely vanilla as I can
make it, other than tmpfs.  And ttylinux is running on a loopback
ext2.

Apologies if I'm missing something obvious.

-- end --

Details available in my last message.

-- jared@wordzoo.com

"Better to be of a rare breed than a long line." -- TDK


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

Re: XenCD xm save problem

[Arthur Bergman]

On 10 Feb 2005, at 22:50, Jared Rhine wrote:

> [Jared == jared@wordzoo.com on Wed, 09 Feb 2005 16:05:18 -0800]
>
>   Jared> I can duplicate the problem that xm save does not work on
>   Jared> XenCD.
>
> Others have reported success after switching to Xen 2.0.4, but my
> XenCD save problem still exists with a 2.0.4-built snapshot.
>
> From my last email:

If you run xfrd under strace -fF does it help?

What versions of python/twisted do you have on it?

Cheers
Arthur

-----
CTO @ Fotango Ltd
+447834716919
http://www.fotango.com/


-----
CTO @ Fotango Ltd
+447834716919
http://www.fotango.com/


-----
CTO @ Fotango Ltd
+447834716919
http://www.fotango.com/



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

XenCD xm save problem

[Jared Rhine]

[Arthur == abergman@fotango.com on Thu, 10 Feb 2005 23:31:21 +0000]

  Arthur> On 10 Feb 2005, at 22:50, Jared Rhine wrote:

  Jared> Others have reported success after switching to Xen 2.0.4,
  Jared> but my XenCD save problem still exists with a 2.0.4-built
  Jared> snapshot.

  Arthur> If you run xfrd under strace -fF does it help?

Ah, should have mentioned.  No, it does not.  At least on Xen
unstable, I will try one more time with 2.0.4+strace xfrd.

  Arthur> What versions of python/twisted do you have on it?

Debian sarge.  Looking...

  Python 2.3.4
  Twisted 1.3.0

I presume others can replicate my results exactly just by using the CD
and the command sequence posted.  I replicated results reported by
someone else.

-- jared@wordzoo.com

"A pessimist is one who has been intimately acquainted with an optimist."
        -- Elbert Hubbard


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: XenCD xm save problem

[Arthur Bergman]

On 11 Feb 2005, at 00:40, Jared Rhine wrote:

>   Arthur> What versions of python/twisted do you have on it?
>
> Debian sarge.  Looking...
>
>   Python 2.3.4
>   Twisted 1.3.0
>
> I presume others can replicate my results exactly just by using the CD
> and the command sequence posted.  I replicated results reported by
> someone else.
>

Thanks, now can someone where save/migrate works flawless please tell 
us the python version, twisted version and gcc version :) (and possibly 
also distro)

Cheers
Arthur


-----
CTO @ Fotango Ltd
+447834716919
http://www.fotango.com/



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

XenCD xm save problem: solved by 2.0.4

[Jared Rhine]

[Jared == jared@wordzoo.com on Thu, 10 Feb 2005 16:40:05 -0800]

  Jared> Others have reported success after switching to Xen 2.0.4,
  Jared> but my XenCD save problem still exists with a 2.0.4-built
  Jared> snapshot.

My sincere thanks to Paul Ionescu who pointed out that my 2.0.4 build
wasn't actually 2.0.4 but rather xeno-unstable.

After fixing this build bug and retrying the resulting XenCD inside
qemu, xm save/restore now works properly!  (Fresh out of the box (no
strace xfrd), to both hard disks and tmpfs).

I believe that now every recent report of xm save/restore actually
works under 2.0.4, while current xen-unstable installations are still
broken (perhaps in only some cases?)

Hopefully, this community testing provides the Xen team a more clear
problem report and points to xm save/restore reversion between 2.0.4
and current unstable.

My box is chugging away on building some new XenCD snapshots and the
resulting snapshot should be available for download in a few hours.
(If I remember, I'll report on xen-testing's save/restore
success/failure as well).

This issue is important enough for demo purposes that XenCD 1.0rc01 is
revoked as a release candidate, and I'll need to release a 1.0rc02
within a few days.

-- jared@wordzoo.com

"We suffer primarily not from our vices or our weaknesses, but from our
 illusions.  We are haunted, not by reality, but by those images we have put
 in place of reality." - Daniel J. Boorstin


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: XenCD xm save problem: solved by 2.0.4

[Ian Pratt]

 

> My sincere thanks to Paul Ionescu who pointed out that my 2.0.4 build
> wasn't actually 2.0.4 but rather xeno-unstable.
> 
> After fixing this build bug and retrying the resulting XenCD inside
> qemu, xm save/restore now works properly!  (Fresh out of the box (no
> strace xfrd), to both hard disks and tmpfs).
> 
> I believe that now every recent report of xm save/restore actually
> works under 2.0.4, while current xen-unstable installations are still
> broken (perhaps in only some cases?)

Save/restore doesn't work at all in unstable as the tools haven't been
updated to cope with multiple VCPUs per domain. It's on the todo list...

> Hopefully, this community testing provides the Xen team a more clear
> problem report and points to xm save/restore reversion between 2.0.4
> and current unstable.

Yep, it's great to have the testing feedback, thanks.

Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

XenCD xm save problem: solved by 2.0.4

[Jared Rhine]

[Ian == m+Ian.Pratt@cl.cam.ac.uk on Sat, 12 Feb 2005 10:02:35 -0000]

  >> I believe that now every recent report of xm save/restore
  >> actually works under 2.0.4, while current xen-unstable
  >> installations are still broken (perhaps in only some cases?)

  Ian> Save/restore doesn't work at all in unstable as the tools
  Ian> haven't been updated to cope with multiple VCPUs per
  Ian> domain. It's on the todo list...

Oh my, that tidbit of info would have saved a number of people quite a
few hours of experimentation.  Apologies for all the chatter about an
already known issue.

Thanks.

-- jared@wordzoo.com

"Truth is a great flirt." -- Franz Liszt


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: XenCD xm save problem: solved by 2.0.4

[Arthur Bergman]

On 12 Feb 2005, at 07:56, Jared Rhine wrote:

>
> After fixing this build bug and retrying the resulting XenCD inside
> qemu, xm save/restore now works properly!  (Fresh out of the box (no
> strace xfrd), to both hard disks and tmpfs).
>
> I believe that now every recent report of xm save/restore actually
> works under 2.0.4, while current xen-unstable installations are still
> broken (perhaps in only some cases?)
>

Please try it multiple times

Compiling xfrd with debugging it took my 36 save restore cycles to hit 
the bug. Anything that SLOWS xfrd down seems to fix the bug. Which is 
odd since it is blocking in a read.

Anyway... this is where it is blocking. It seems to be waiting for a 
response that it never gets.

Looking closer at this.

Cheers
Arthur

(gdb) bt
#0  0x402acb48 in read () from /lib/libc.so.6
#1  0x403125e0 in ?? () from /lib/libc.so.6
#2  0x4024e30b in _IO_file_read () from /lib/libc.so.6
#3  0x4024e79f in _IO_file_xsputn () from /lib/libc.so.6
#4  0x4024f4e8 in _IO_sgetn () from /lib/libc.so.6
#5  0x402434b2 in fread () from /lib/libc.so.6
#6  0x0804fb22 in file_read (s=0x1, buf=0xfffffe00, n=4294966784) at 
../../tools/libxutil/file_stream.c:77
#7  0x0804f499 in Conn_sxpr (conn=0xbffff900, sxpr=0xfffffe00) at 
iostream.h:114
#8  0x0804dd6c in xfr_vm_suspend (xend=0xbffff900, vmid=1) at xfrd.c:451
#9  0x40026524 in suspend_and_state () from /usr/lib/libxc.so.2.0
#10 0x40027c60 in xc_linux_save () from /usr/lib/libxc.so.2.0
#11 0x0804e9e4 in xen_domain_snd (xend=0xfffffe00, io=0xfffffe00, 
dom=4294966784, vmconfig=0xfffffe00 <Address 0xfffffe00 out of bounds>,
     vmconfig_n=-512, live=0, resource=-512) at xen_domain.c:39
#12 0x0804b79a in xfr_save (args=0x805d3b0, state=0xbffff820, 
xend=0xbffff900, file=0x805f9a8 "/tmp/foo") at xfrd.c:774
#13 0x0804c454 in xfrd_service (args=0x805d3b0, peersock=-512, peer_in=
       {sin_family = 2, sin_port = 12294, sin_addr = {s_addr = 
16777343}, sin_zero = "\000\000\000\000\000\000\000"}) at xfrd.c:950
#14 0x0804c8c8 in xfrd_accept (args=0xfffffe00, sock=2) at xfrd.c:1024
#15 0x0804c9b5 in xfrd_select (args=0x805d3b0, listen_sock=2) at 
xfrd.c:1054
#16 0x0804d1e8 in main (argc=1, argv=0xbffffe44) at xfrd.c:1210



-----
CTO @ Fotango Ltd
+447834716919
http://www.fotango.com/




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click