Xen-3.0.3 migration problems

Xen-3.0.3 migration problems

[Bhatia, Nikhil]

[-- Attachment #1.1: Type: text/plain, Size: 3159 bytes --]

Hello Xen Users and Developers, 

 

I am running 3 guest domains on a node. Following is the information: 

 

[root@dmz02 fedora]# xm list

Name                                      ID Mem(MiB) VCPUs State
Time(s)

Domain-0                                   0      255     4 r-----
66.2

vcl-dmz02                                  1      256     1 -b----
1.1

vcn-dmz02                                  6      256     1 -b----
0.3

vcs-dmz02                                  7      256     1 -b----
0.3

 

I am trying to perform live migration using the command: 

 

(dmz03.ornl.gov is also running Xen-3.0.3 and is running 3 guest domains
of it's own) 

 

xm migrate --live vcs-dmz02 dmz03.ornl.gov

Error: can't connect: No route to host

Usage: xm migrate <Domain> <Host>

 

Migrate a domain to another machine.

 

Options:

 

-h, --help           Print this help.

-l, --live           Use live migration.

-p=portnum, --port=portnum

                     Use specified port for migration.

-r=MBIT, --resource=MBIT

                     Set level of resource usage for migration.

 

 

 

So, apparently dmz03.ornl.gov cannot be seen from dmz02.ornl.gov.
However, when I run "ping", I GET THE FOLLOWING OUTPUT: 

 

[root@dmz02 fedora]# ping dmz03.ornl.gov

PING dmz03.ornl.gov (160.91.40.231) 56(84) bytes of data.

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=0 ttl=64
time=0.142 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=1 ttl=64
time=0.099 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=2 ttl=64
time=0.104 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=3 ttl=64
time=0.099 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=4 ttl=64
time=0.098 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=5 ttl=64
time=0.099 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=6 ttl=64
time=0.122 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=7 ttl=64
time=0.098 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=8 ttl=64
time=0.099 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=9 ttl=64
time=0.098 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=10 ttl=64
time=0.098 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=11 ttl=64
time=0.098 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=12 ttl=64
time=0.102 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=13 ttl=64
time=0.099 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=14 ttl=64
time=0.099 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=15 ttl=64
time=0.098 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=16 ttl=64
time=0.097 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=17 ttl=64
time=0.098 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=18 ttl=64
time=0.098 ms

64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=19 ttl=64
time=0.097 ms

 

Can someone please suggest what the problem might be? 

 

 

Thanks,

Nikhil

 


[-- Attachment #1.2: Type: text/html, Size: 14758 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [Xen-devel] Xen-3.0.3 migration problems

[Ian Pratt]

> xm migrate --live vcs-dmz02 dmz03.ornl.gov
> 
> Error: can't connect: No route to host
> 
> Usage: xm migrate <Domain> <Host>


Have you edited /etc/xen/xend-config.sxp on both hosts to enable the
relocation service?

Best,
Ian
 
> 
> 
> Migrate a domain to another machine.
> 
> 
> 
> Options:
> 
> 
> 
> -h, --help           Print this help.
> 
> -l, --live           Use live migration.
> 
> -p=portnum, --port=portnum
> 
>                      Use specified port for migration.
> 
> -r=MBIT, --resource=MBIT
> 
>                      Set level of resource usage for migration.
> 
> 
> 
> 
> 
> 
> 
> So, apparently dmz03.ornl.gov cannot be seen from dmz02.ornl.gov.
However,
> when I run "ping", I GET THE FOLLOWING OUTPUT:
> 
> 
> 
> [root@dmz02 fedora]# ping dmz03.ornl.gov
> 
> PING dmz03.ornl.gov (160.91.40.231) 56(84) bytes of data.
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=0 ttl=64
time=0.142
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=1 ttl=64
time=0.099
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=2 ttl=64
time=0.104
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=3 ttl=64
time=0.099
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=4 ttl=64
time=0.098
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=5 ttl=64
time=0.099
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=6 ttl=64
time=0.122
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=7 ttl=64
time=0.098
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=8 ttl=64
time=0.099
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=9 ttl=64
time=0.098
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=10 ttl=64
time=0.098
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=11 ttl=64
time=0.098
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=12 ttl=64
time=0.102
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=13 ttl=64
time=0.099
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=14 ttl=64
time=0.099
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=15 ttl=64
time=0.098
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=16 ttl=64
time=0.097
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=17 ttl=64
time=0.098
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=18 ttl=64
time=0.098
> ms
> 
> 64 bytes from dmz03.ornl.gov (160.91.40.231): icmp_seq=19 ttl=64
time=0.097
> ms
> 
> 
> 
> Can someone please suggest what the problem might be?
> 
> 
> 
> 
> 
> Thanks,
> 
> Nikhil
> 
> 

Re: RE: [Xen-devel] Xen-3.0.3 migration problems

[Nico Kadel-Garcia]

Ian Pratt wrote:
>> xm migrate --live vcs-dmz02 dmz03.ornl.gov
>>
>> Error: can't connect: No route to host
>>
>> Usage: xm migrate <Domain> <Host>
>>     
>
>
> Have you edited /etc/xen/xend-config.sxp on both hosts to enable the
> relocation service?
>
> Best,
> Ian
>   
Does it need to be on both hosts? And do there need to be image files 
*already* in place on the target host in /var/lib/xen/images/? And what 
if you've been using LVM managed disk partitions for your working Xen 
images?

Unfortunately, I'm finding the documentation for Xen troublesomely poor.

Re: RE: [Xen-devel] Xen-3.0.3 migration problems

[Adam Seering]

Hi,
	I'm having this same problem.  The relocation service is enabled on  
both machines; neither has a firewall (sure, I care about  
security...), and there is no firewall between them.

	For a live migration, do the dom0 kernels and the source/target PC's  
hardware have to be the same?  I'm migrating from a stock Debian Etch- 
based Xen AMD64 system on a Pentium D 8xx (no HVM) to a Fedora Core 6  
Xen AMD64 system on a Pentium D 9xx, with HVM support enabled.

Adam


On Dec 28, 2006, at 6:32 AM, Nico Kadel-Garcia wrote:

> Ian Pratt wrote:
>>> xm migrate --live vcs-dmz02 dmz03.ornl.gov
>>>
>>> Error: can't connect: No route to host
>>>
>>> Usage: xm migrate <Domain> <Host>
>>>
>>
>>
>> Have you edited /etc/xen/xend-config.sxp on both hosts to enable the
>> relocation service?
>>
>> Best,
>> Ian
>>
> Does it need to be on both hosts? And do there need to be image  
> files *already* in place on the target host in /var/lib/xen/ 
> images/? And what if you've been using LVM managed disk partitions  
> for your working Xen images?
>
> Unfortunately, I'm finding the documentation for Xen troublesomely  
> poor.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

Re: RE: [Xen-devel] Xen-3.0.3 migration problems

[Nico Kadel-Garcia]

Adam Seering wrote:
> Hi,
>     I'm having this same problem.  The relocation service is enabled 
> on both machines; neither has a firewall (sure, I care about 
> security...), and there is no firewall between them.
>
>     For a live migration, do the dom0 kernels and the source/target 
> PC's hardware have to be the same?  I'm migrating from a stock Debian 
> Etch-based Xen AMD64 system on a Pentium D 8xx (no HVM) to a Fedora 
> Core 6 Xen AMD64 system on a Pentium D 9xx, with HVM support enabled.
>
> Adam
I've duplicated my relocation domain to both Xen Dom0 servers, and am 
waiting to reboot one of them to make sure it gets all the options set 
correctly. (I had to schedule a shutdown anyway.) Unfortunately, it 
seems to fail silently. "xm list" reports a "migrating-DOMAIN" domain on 
the source host, and a "DOMAIN" domain on the other target host, but the 
transfer eventually fails silently and leaves the DOMAIN in question 
dead on the first host, which is really, really bad practice.

I'm working on an ASUS RS120-E3 system, without VT, all 32-bit systems. 
I'm also tyring to play with a RHEL 4.92 beta system, but it's been a 
source of adventures.

>
> On Dec 28, 2006, at 6:32 AM, Nico Kadel-Garcia wrote:
>
>> Ian Pratt wrote:
>>>> xm migrate --live vcs-dmz02 dmz03.ornl.gov
>>>>
>>>> Error: can't connect: No route to host
>>>>
>>>> Usage: xm migrate <Domain> <Host>
>>>>
>>>
>>>
>>> Have you edited /etc/xen/xend-config.sxp on both hosts to enable the
>>> relocation service?
>>>
>>> Best,
>>> Ian
>>>
>> Does it need to be on both hosts? And do there need to be image files 
>> *already* in place on the target host in /var/lib/xen/images/? And 
>> what if you've been using LVM managed disk partitions for your 
>> working Xen images?
>>
>> Unfortunately, I'm finding the documentation for Xen troublesomely poor.

Re: RE: [Xen-devel] Xen-3.0.3 migration problems

[Keir Fraser]

On 28/12/06 5:15 pm, "Adam Seering" <aseering@gmail.com> wrote:

> I'm having this same problem.  The relocation service is enabled on
> both machines; neither has a firewall (sure, I care about
> security...), and there is no firewall between them.
> 
> For a live migration, do the dom0 kernels and the source/target PC's
> hardware have to be the same?  I'm migrating from a stock Debian Etch-
> based Xen AMD64 system on a Pentium D 8xx (no HVM) to a Fedora Core 6
> Xen AMD64 system on a Pentium D 9xx, with HVM support enabled.

Migrating to a newer (more capable) processor should work. It's when
features silently disappear that guests can get upset. The dom0 kernel does
not matter but the version of the Xen toolstack might -- did you install Xen
and tools yourself on both boxes from a consistent version?

/var/log/xen/xend.log and /var/log/xen/xend-debug.log probably have
interesting info if things are going wrong (the logs may be big: if so it's
the tail that's interesting).

 -- Keir

Re: RE: [Xen-devel] Xen-3.0.3 migration problems

[Adam Seering]

On Dec 29, 2006, at 5:05 AM, Keir Fraser wrote:

> On 28/12/06 5:15 pm, "Adam Seering" <aseering@gmail.com> wrote:
>
>> I'm having this same problem.  The relocation service is enabled on
>> both machines; neither has a firewall (sure, I care about
>> security...), and there is no firewall between them.
>>
>> For a live migration, do the dom0 kernels and the source/target PC's
>> hardware have to be the same?  I'm migrating from a stock Debian  
>> Etch-
>> based Xen AMD64 system on a Pentium D 8xx (no HVM) to a Fedora Core 6
>> Xen AMD64 system on a Pentium D 9xx, with HVM support enabled.
>
> Migrating to a newer (more capable) processor should work. It's when
> features silently disappear that guests can get upset. The dom0  
> kernel does
> not matter but the version of the Xen toolstack might -- did you  
> install Xen
> and tools yourself on both boxes from a consistent version?

I did not, actually; I'm using the stock versions from the FC6 /  
Debian Etch repo's.  I could try that.

One of these machines just lost a hard disk; I'm not eager to try  
anything interesting until I can replace it, so I'll play around with  
this when the new drive comes in.

A recent e-mail mentioned that migration machines have to be on the  
same L2 subnet (I missed that line in the Xen manual).  These  
machines aren't; they're in different buildings, with IP routing  
between them (though the connection speed and latency are very  
good).  I'll try putting the machines on the same subnet.


> /var/log/xen/xend.log and /var/log/xen/xend-debug.log probably have
> interesting info if things are going wrong (the logs may be big: if  
> so it's
> the tail that's interesting).

OK; I'll take a look at those.

Thanks,
Adam

RE: [Xen-users] RE: Xen-3.0.3 migration problems

[Bhatia, Nikhil]

Hello All,

Thanks a lot for your quick and extremely useful responses. Although, I
have not been able to solve the problem but I would like o mention some
things about my configuration setup for the guest domains and the host
domain. As mentioned earlier, I have two nodes running (dmz02 and
dmz03). 

DMZ02 is using xen-3.0.3 version which I grabbed from the Xensource
website a couple of days back. The guest domain configuration file has
been setup so as to use DHCP protocol for assigning the IP addresses to
the guest domains. Now one of the problems that I am facing is that the
Xen guest domains are not being able to see the DHCP server on creation.
Therefore, the domains are created in a "blocked" state. That might
explain why things are not working for me. 

DMZ03 is using xen-3.0.3-testing.hg source obtained 3-4 weeks back. Can
that be a potential problem too? Now, DMZ-03 is been used by one of my
other colleagues, so it would take some handshaking for me to get the
versions compatible. Also, the guest domains on this node are created in
the "blocked" state. 

So, I understand that the guest domains should be in a "running" state
for migration to take place. Is that a safe statement to say? Anyways, I
think my problem is the "blocked" state and the unreachable DHCP server.


I have checked that my /etc/xen/xend.config has been configured properly
on both sides. 

Thanks for all your suggestions.

Wish you all a Happy new Year. 

Thanks,
Nikhil




-----Original Message-----
From: Adam Seering [mailto:aseering@gmail.com] 
Sent: Friday, December 29, 2006 11:34 AM
To: Keir Fraser
Cc: Ian Pratt; xen-devel@lists.xensource.com;
xen-users@lists.xensource.com; Bhatia, Nikhil; Nico Kadel-Garcia
Subject: Re: [Xen-users] RE: [Xen-devel] Xen-3.0.3 migration problems


On Dec 29, 2006, at 5:05 AM, Keir Fraser wrote:

> On 28/12/06 5:15 pm, "Adam Seering" <aseering@gmail.com> wrote:
>
>> I'm having this same problem.  The relocation service is enabled on
>> both machines; neither has a firewall (sure, I care about
>> security...), and there is no firewall between them.
>>
>> For a live migration, do the dom0 kernels and the source/target PC's
>> hardware have to be the same?  I'm migrating from a stock Debian  
>> Etch-
>> based Xen AMD64 system on a Pentium D 8xx (no HVM) to a Fedora Core 6
>> Xen AMD64 system on a Pentium D 9xx, with HVM support enabled.
>
> Migrating to a newer (more capable) processor should work. It's when
> features silently disappear that guests can get upset. The dom0  
> kernel does
> not matter but the version of the Xen toolstack might -- did you  
> install Xen
> and tools yourself on both boxes from a consistent version?

I did not, actually; I'm using the stock versions from the FC6 /  
Debian Etch repo's.  I could try that.

One of these machines just lost a hard disk; I'm not eager to try  
anything interesting until I can replace it, so I'll play around with  
this when the new drive comes in.

A recent e-mail mentioned that migration machines have to be on the  
same L2 subnet (I missed that line in the Xen manual).  These  
machines aren't; they're in different buildings, with IP routing  
between them (though the connection speed and latency are very  
good).  I'll try putting the machines on the same subnet.


> /var/log/xen/xend.log and /var/log/xen/xend-debug.log probably have
> interesting info if things are going wrong (the logs may be big: if  
> so it's
> the tail that's interesting).

OK; I'll take a look at those.

Thanks,
Adam

Re: [Xen-users] RE: Xen-3.0.3 migration problems

[Keir Fraser]

On 29/12/06 4:34 pm, "Adam Seering" <aseering@gmail.com> wrote:

> A recent e-mail mentioned that migration machines have to be on the
> same L2 subnet (I missed that line in the Xen manual).  These
> machines aren't; they're in different buildings, with IP routing
> between them (though the connection speed and latency are very
> good).  I'll try putting the machines on the same subnet.

The guest's IP address will not change across the migration, so the original
IP address needs to be routable to the target host.

 -- Keir

Re: RE: [Xen-devel] Xen-3.0.3 migration problems

[Ulrich Windl]

On 29 Dec 2006 at 18:34, Keir Fraser wrote:

> On 29/12/06 4:34 pm, "Adam Seering" <aseering@gmail.com> wrote:
> 
> > A recent e-mail mentioned that migration machines have to be on the
> > same L2 subnet (I missed that line in the Xen manual).  These
> > machines aren't; they're in different buildings, with IP routing
> > between them (though the connection speed and latency are very
> > good).  I'll try putting the machines on the same subnet.
> 
> The guest's IP address will not change across the migration, so the original
> IP address needs to be routable to the target host.

??? Migration is communication between Dom0s, right? Can't Dom0 ensure that DomU 
IPs are routable?

Maybe a warning would be preferrable to an error (do-nothing).

Regards,
Ulrich