Live Migration problems using xenapi with xm

Live Migration problems using xenapi with xm

[Ben Guthro]

I am seeing a problem when live migrating a guest under unstable when I 
have xm configured to XenAPI.

If my xm-config.xml file has the XenAPI stuff commented out - live 
migrate (xm migrate --live <domid> <dest IP>) seems to be working in my 
test environment.

However, if I enable it - I get failures when migrating the same guest 
(see dump from xend.log below.)  They obviously go down quite different 
code paths in these two cases.

Being somewhat new to the xm, and xenapi control stacks - I was hoping 
someone might point me in the right direction to track down what might 
be happening here...

Any help is appreciated.

Ben

Xend log snippet:

2007-11-07 09:01:35 15422] DEBUG (XendCheckpoint:88) [xc_save]: 
/usr/lib64/xen/bin/xc_save 20 2 0 0 5
[2007-11-07 09:01:35 15422] INFO (XendCheckpoint:362) Saving memory 
pages: iter 1   0%ERROR Internal error: Error when writing to state file 
(5) (errno 104)
[2007-11-07 09:01:35 15422] INFO (XendCheckpoint:362) Save exit rc=1
[2007-11-07 09:01:36 15422] ERROR (XendCheckpoint:140) Save failed on 
domain VS022 (2).
Traceback (most recent call last):
 File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 108, in save
   forkHelper(cmd, fd, saveInputHandler, False)
 File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 350, in 
forkHelper
   raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_save 20 2 0 0 5 failed
[2007-11-07 09:01:36 15422] DEBUG (XendDomainInfo:1856) 
XendDomainInfo.resumeDomain(2)
[2007-11-07 09:01:36 15422] ERROR (xmlrpclib2:166) Internal error 
handling VM.migrate
Traceback (most recent call last):
 File "/usr/lib64/python/xen/util/xmlrpclib2.py", line 131, in 
_marshaled_dispatch
   response = self._dispatch(method, params)
 File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch
   return func(*params)
 File "/usr/lib64/python/xen/xend/XendAPI.py", line 221, in f
   return func(self, *args, **kwargs)
 File "/usr/lib64/python/xen/xend/XendAPI.py", line 256, in check_session
   return func(self, session, *args, **kwargs)
 File "/usr/lib64/python/xen/xend/XendAPI.py", line 312, in <lambda>
   'VM', func, *args, **kwargs)
 File "/usr/lib64/python/xen/xend/XendAPI.py", line 268, in _check_ref
   return func(api, session, ref, *args, **kwargs)
 File "/usr/lib64/python/xen/xend/XendAPI.py", line 1763, in VM_migrate
   bool(live), resource, port)
 File "/usr/lib64/python/xen/xend/XendDomain.py", line 1282, in 
domain_migrate
   XendCheckpoint.save(sock.fileno(), dominfo, True, live, dst)
 File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 142, in save
   dominfo.resumeDomain()
 File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 1885, in 
resumeDomain
   if self.is_hvm():
 File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 2139, in 
__getattr__
   raise AttributeError()
AttributeError

Re: Live Migration problems using xenapi with xm

[Ben Guthro]

I've managed to track this down to being a bug in HVM migration using 
XenAPI.

It seems that restore is trying to restore a PV guest, while save is 
trying to save an HVM guest... so the protocol of how many mytes to read 
on the socket is off, and naturally fails when it tries to read much 
more data than the max batch size allows.

Tracing this back, it seems to come down to the "is_hvm" method of the 
XendConfig class, which checks the "HVM_boot_policy" parameter. For some 
reason, it seems this parameter is not being preserved during domain 
creation with xenapi, because by the time migration occurs, the 
dictionary item no longer exists.

I'll continue to dig further, but if anyone has any thoughts on this, 
I'd welcome pointers.



Ben Guthro wrote:
> I am seeing a problem when live migrating a guest under unstable when 
> I have xm configured to XenAPI.
>
> If my xm-config.xml file has the XenAPI stuff commented out - live 
> migrate (xm migrate --live <domid> <dest IP>) seems to be working in 
> my test environment.
>
> However, if I enable it - I get failures when migrating the same guest 
> (see dump from xend.log below.)  They obviously go down quite 
> different code paths in these two cases.
>
> Being somewhat new to the xm, and xenapi control stacks - I was hoping 
> someone might point me in the right direction to track down what might 
> be happening here...
>
> Any help is appreciated.
>
> Ben
>
> Xend log snippet:
>
> 2007-11-07 09:01:35 15422] DEBUG (XendCheckpoint:88) [xc_save]: 
> /usr/lib64/xen/bin/xc_save 20 2 0 0 5
> [2007-11-07 09:01:35 15422] INFO (XendCheckpoint:362) Saving memory 
> pages: iter 1   0%ERROR Internal error: Error when writing to state 
> file (5) (errno 104)
> [2007-11-07 09:01:35 15422] INFO (XendCheckpoint:362) Save exit rc=1
> [2007-11-07 09:01:36 15422] ERROR (XendCheckpoint:140) Save failed on 
> domain VS022 (2).
> Traceback (most recent call last):
> File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 108, in save
>   forkHelper(cmd, fd, saveInputHandler, False)
> File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 350, in 
> forkHelper
>   raise XendError("%s failed" % string.join(cmd))
> XendError: /usr/lib64/xen/bin/xc_save 20 2 0 0 5 failed
> [2007-11-07 09:01:36 15422] DEBUG (XendDomainInfo:1856) 
> XendDomainInfo.resumeDomain(2)
> [2007-11-07 09:01:36 15422] ERROR (xmlrpclib2:166) Internal error 
> handling VM.migrate
> Traceback (most recent call last):
> File "/usr/lib64/python/xen/util/xmlrpclib2.py", line 131, in 
> _marshaled_dispatch
>   response = self._dispatch(method, params)
> File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch
>   return func(*params)
> File "/usr/lib64/python/xen/xend/XendAPI.py", line 221, in f
>   return func(self, *args, **kwargs)
> File "/usr/lib64/python/xen/xend/XendAPI.py", line 256, in check_session
>   return func(self, session, *args, **kwargs)
> File "/usr/lib64/python/xen/xend/XendAPI.py", line 312, in <lambda>
>   'VM', func, *args, **kwargs)
> File "/usr/lib64/python/xen/xend/XendAPI.py", line 268, in _check_ref
>   return func(api, session, ref, *args, **kwargs)
> File "/usr/lib64/python/xen/xend/XendAPI.py", line 1763, in VM_migrate
>   bool(live), resource, port)
> File "/usr/lib64/python/xen/xend/XendDomain.py", line 1282, in 
> domain_migrate
>   XendCheckpoint.save(sock.fileno(), dominfo, True, live, dst)
> File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 142, in save
>   dominfo.resumeDomain()
> File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 1885, in 
> resumeDomain
>   if self.is_hvm():
> File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 2139, in 
> __getattr__
>   raise AttributeError()
> AttributeError
>
>
>

Re: Re: Live Migration problems using xenapi with xm

[Ian Jackson]

Ben Guthro writes ("[Xen-devel] Re: Live Migration problems using xenapi with xm"):
> I've managed to track this down to being a bug in HVM migration using 
> XenAPI.

Aha.

> It seems that restore is trying to restore a PV guest, while save is 
> trying to save an HVM guest...  [ etc. ]

Thanks for investigating.

> I'll continue to dig further, but if anyone has any thoughts on this, 
> I'd welcome pointers.

I don't have any thoughts right now, I'm afraid.  I hope to be in a
better position to help out very shortly; There's quite a lot of
review, testing and general sorting out of XenAPI needed which I'll be
looking into.

In the meantime, good luck and do let us know what more you find.

Regards,
Ian.

Re: Re: Live Migration problems using xenapi with xm

[Ben Guthro]

[-- Attachment #1.1: Type: text/plain, Size: 1403 bytes --]

Thanks for your reply,

I'm wondering - is this uncharted territory, or has migration on HVM 
(with xenapi) worked in the past? We'll be happy to help fix this...but 
it would go a long way to know what is thought to be working.

Strangely, I have found that I can migrate guests with xenapi that were 
not created with xenapi.

Just as a hack, I also tried always returning True for is_hvm. This 
seemed to get the migration process further, but seemed to get hung up 
on closing(or connecting) the vkbd device - leaving the domain in a 
broken state at both the sender, and receiver

Ben

Ian Jackson wrote:
> Ben Guthro writes ("[Xen-devel] Re: Live Migration problems using xenapi with xm"):
>   
>> I've managed to track this down to being a bug in HVM migration using 
>> XenAPI.
>>     
>
> Aha.
>
>   
>> It seems that restore is trying to restore a PV guest, while save is 
>> trying to save an HVM guest...  [ etc. ]
>>     
>
> Thanks for investigating.
>
>   
>> I'll continue to dig further, but if anyone has any thoughts on this, 
>> I'd welcome pointers.
>>     
>
> I don't have any thoughts right now, I'm afraid.  I hope to be in a
> better position to help out very shortly; There's quite a lot of
> review, testing and general sorting out of XenAPI needed which I'll be
> looking into.
>
> In the meantime, good luck and do let us know what more you find.
>
> Regards,
> Ian.
>   


[-- Attachment #1.2: Type: text/html, Size: 1999 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Live Migration problems using xenapi with xm

[Jim Fehlig]

Ben Guthro wrote:
> Thanks for your reply,
>
> I'm wondering - is this uncharted territory, or has migration on HVM
> (with xenapi) worked in the past? We'll be happy to help fix
> this...but it would go a long way to know what is thought to be working.

I would suspect you are in uncharted territory.  Currently I don't think
there are a lot of xenapi users.  In the past I have generally had to
fix bugs or provide implementation for xenapi features previously not
consumed by my app.  You are perhaps the first to attempt migration
(almost certainly of HVM guests) through xenapi.

Regards,
Jim

Re: Re: Live Migration problems using xenapi with xm

[Ian Jackson]

Jim Fehlig writes ("Re: [Xen-devel] Re: Live Migration problems using xenapi with xm"):
> I would suspect you are in uncharted territory.  Currently I don't think
> there are a lot of xenapi users.  In the past I have generally had to
> fix bugs or provide implementation for xenapi features previously not
> consumed by my app.  You are perhaps the first to attempt migration
> (almost certainly of HVM guests) through xenapi.

Jim may well be right, I'm afraid.  I've only just arrived so to speak
but I think the implementation in xend certainly needs work.

Thanks,
Ian.

Re: Re: Live Migration problems using xenapi with xm

[Ben Guthro]

[-- Attachment #1.1: Type: text/plain, Size: 1357 bytes --]

I think I have hacked together something that seems to work...

HVM Live migration seems to succeed if I do the following:
a.) Ignore the fact that receive thinks it is a PV guest (force is_hvm 
to return True)
b.) Modify XendDomainInfo.waitForDevices to ignore "vkbd" and "vfb" 
devices for HVM guests

We don't really do much PV guest stuff here, so I'm not real sure what 
the vfb, and vkbd are in the PV world...
However, I'm not sure that they are necessary in HVM (though please 
correct me if I'm wrong)

I'd like to come up with a patch for this, but would like to hear 
thoughts on what the right way to go about this is.
Should I ignore those devices in waitForDevices?
Should they not be there in the first place?


Ian Jackson wrote:
> Jim Fehlig writes ("Re: [Xen-devel] Re: Live Migration problems using xenapi with xm"):
>   
>> I would suspect you are in uncharted territory.  Currently I don't think
>> there are a lot of xenapi users.  In the past I have generally had to
>> fix bugs or provide implementation for xenapi features previously not
>> consumed by my app.  You are perhaps the first to attempt migration
>> (almost certainly of HVM guests) through xenapi.
>>     
>
> Jim may well be right, I'm afraid.  I've only just arrived so to speak
> but I think the implementation in xend certainly needs work.
>
> Thanks,
> Ian.
>   


[-- Attachment #1.2: Type: text/html, Size: 1815 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel