Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0

[Joshua West <jwest@brandeis.edu>]

Hey all,

So it looks like the xc_save utility has issues saving a 32-bit 
paravirtualized domU on a 64-bit dom0.  As a result, live migration of 
32-bit PV VM's (between 64-bit dom0's) also does not work.  However, I 
am able to "xm save" and "xm migrate" 64-bit PV domU's on those same 
64-bit dom0's without encountering issues.

Has anybody else run into save/migrate issues of 32-bit domU's on 64-bit 
dom0's?  Do you know of a fix or workaround?

The 32-bit domU's I'm working with have kernels which are compiled with 
PAE (HIGHMEM64G=y) support.  Additionally, both of these machines are of 
identical hardware configuration (Dell PowerEdge 1950's) and have been 
setup exactly the same.

Here is a snippit of output from /var/log/xen/xend.log:

#----------------------------------------------------------#
[2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:88) [xc_save]: 
/usr/lib64/xen/bin/xc_save 30 1 0 0 0
[2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:333) suspend
[2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:91) In saveInputHandler 
suspend
[2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:93) Suspending 1 ...
[2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:451) 
XendDomainInfo.shutdown(suspend)
[2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:991) 
XendDomainInfo.handleShutdownWatch
[2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:991) 
XendDomainInfo.handleShutdownWatch
[2008-01-17 19:18:08 8353] INFO (XendCheckpoint:98) Domain 1 suspended.
[2008-01-17 19:18:08 8353] INFO (XendDomainInfo:1179) Domain has 
shutdown: name=migrating-rhel5-test id=1 reason=suspend.
[2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:107) Written done
[2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) ERROR Internal 
error: Timed out waiting for frame list updated.
[2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) ERROR Internal 
error: Failed to map/save the p2m frame list
[2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) Save exit rc=1
[2008-01-17 19:18:09 8353] ERROR (XendCheckpoint:140) Save failed on 
domain rhel5-test (1).
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 109, in save
    forkHelper(cmd, fd, saveInputHandler, False)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 350, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_save 30 1 0 0 0 failed
[2008-01-17 19:18:09 8353] DEBUG (XendDomainInfo:1793) 
XendDomainInfo.resumeDomain(1)
[2008-01-17 19:18:09 8353] DEBUG (XendCheckpoint:143) 
XendCheckpoint.save: resumeDomain
#----------------------------------------------------------#

I'm not sure if this is related, but I also see messages such as the 
following in "xm dmesg":
(XEN) event_channel.c:231:d3 EVTCHNOP failure: domain 1, error -17, line 231
(XEN) event_channel.c:231:d4 EVTCHNOP failure: domain 4, error -17, line 231

I suspect they're just a result of the save failures.

Any help would be *greatly* appreciated!

Thanks!

-- 
Joshua West
Systems Engineer
Brandeis University
http://www.brandeis.edu