xfrd inifinite loop reloading pages

xfrd inifinite loop reloading pages

[Charles Coffing]

I'm seeing an infinite loop in xfrd on the destination physical machine,
when migrating a domain.  It passes 100% while reloading memory pages,
and keeps going.  I've seen this on an older build (an early 2.0.x IIRC)
but have never yet had time to dig into it.  Anyone else seen this?

Here's /var/log/xfrd.log:


9657 [INF] XFRD> Accepted connection from 151.155.190.17:32859 on 2
10346 [INF] XFRD> Xfr service for 151.155.190.17:32859
[DEBUG] Conn_init> flags=1
[DEBUG] Conn_init> write stream...
[DEBUG] stream_init>mode=w flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_init> read stream...
[DEBUG] stream_init>mode=r flags=1 compress=0
[DEBUG] stream_init> unbuffer...
[DEBUG] stream_init< err=0
[DEBUG] Conn_sxpr>
(xfr.hello 1 0)[DEBUG] Conn_sxpr< err=0
[DEBUG] Conn_sxpr>
(xfr.xfr 8)[DEBUG] Conn_sxpr< err=0
[1111078930.273966] xc_linux_restore start
 
xc_linux_restore start
[1111078930.287904] Created domain 1
 
Created domain 1
(Domain-1 Domain-0)'domain id=1 name=SuSE.33 memory=256 console=9601
image=/shared/boot/vmlinuz-2.6.10-xenU'[1111078930.389380] Reloading
memory pages:   0%
Reloading memory pages:   0  6%
^H^H^H^H  6 12%
^H^H^H^H 12 17%
^H^H^H^H 17 23%
^H^H^H^H 23 28%
^H^H^H^H 28 34%
^H^H^H^H 34 39%
^H^H^H^H 39 45%
^H^H^H^H 45 50%
^H^H^H^H 50 56%
^H^H^H^H 56 62%
^H^H^H^H 62 67%
^H^H^H^H 67 73%
^H^H^H^H 73 78%
^H^H^H^H 78 84%
^H^H^H^H 84 89%
^H^H^H^H 89 95%
^H^H^H^H 95100%
^H^H^H^H100105%
^H^H^H^H105110%
^H^H^H^H110115%
^H^H^H^H115120%
^H^H^H^H120125%
^H^H^H^H125130%
^H^H^H^H130135%
^H^H^H^H135140%
^H^H^H^H140145%
^H^H^H^H145150%
^H^H^H^H150155%

[chomp]


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: xfrd inifinite loop reloading pages

[Charles Coffing]

Forgot to mention -- I'm seeing this on 2.0.5.


>>>"Charles Coffing" <ccoffing@novell.com> 03/17/05 10:14 am >>> 
I'm seeing an infinite loop in xfrd on the destination physical machine,

when migrating a domain.  It passes 100% while reloading memory pages, 
and keeps going.  I've seen this on an older build (an early 2.0.x IIRC)

but have never yet had time to dig into it.  Anyone else seen this? 
 
Here's /var/log/xfrd.log: 
 
 
9657 [INF] XFRD> Accepted connection from 151.155.190.17:32859 on 2 
10346 [INF] XFRD> Xfr service for 151.155.190.17:32859 
[DEBUG] Conn_init> flags=1 
[DEBUG] Conn_init> write stream... 
[DEBUG] stream_init>mode=w flags=1 compress=0 
[DEBUG] stream_init> unbuffer... 
[DEBUG] stream_init< err=0 
[DEBUG] Conn_init> read stream... 
[DEBUG] stream_init>mode=r flags=1 compress=0 
[DEBUG] stream_init> unbuffer... 
[DEBUG] stream_init< err=0 
[DEBUG] Conn_sxpr> 
(xfr.hello 1 0)[DEBUG] Conn_sxpr< err=0 
[DEBUG] Conn_sxpr> 
(xfr.xfr 8)[DEBUG] Conn_sxpr< err=0 
[1111078930.273966] xc_linux_restore start 
 
xc_linux_restore start 
[1111078930.287904] Created domain 1 
 
Created domain 1 
(Domain-1 Domain-0)'domain id=1 name=SuSE.33 memory=256 console=9601 
image=/shared/boot/vmlinuz-2.6.10-xenU'[1111078930.389380] Reloading 
memory pages:   0% 
Reloading memory pages:   0  6% 
^H^H^H^H  6 12% 
^H^H^H^H 12 17% 
^H^H^H^H 17 23% 
^H^H^H^H 23 28% 
^H^H^H^H 28 34% 
^H^H^H^H 34 39% 
^H^H^H^H 39 45% 
^H^H^H^H 45 50% 
^H^H^H^H 50 56% 
^H^H^H^H 56 62% 
^H^H^H^H 62 67% 
^H^H^H^H 67 73% 
^H^H^H^H 73 78% 
^H^H^H^H 78 84% 
^H^H^H^H 84 89% 
^H^H^H^H 89 95% 
^H^H^H^H 95100% 
^H^H^H^H100105% 
^H^H^H^H105110% 
^H^H^H^H110115% 
^H^H^H^H115120% 
^H^H^H^H120125% 
^H^H^H^H125130% 
^H^H^H^H130135% 
^H^H^H^H135140% 
^H^H^H^H140145% 
^H^H^H^H145150% 
^H^H^H^H150155% 
 
[chomp] 
 
 
------------------------------------------------------- 
SF email is sponsored by - The IT Product Guide 
Read honest & candid reviews on hundreds of IT Products from real users.

Discover which products truly live up to the hype. Start reading now. 
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click 
 
Xen-devel mailing list 
Xen-devel@lists.sourceforge.net 
https://lists.sourceforge.net/lists/listinfo/xen-devel 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: xfrd inifinite loop reloading pages

[Ian Pratt]

> I'm seeing an infinite loop in xfrd on the destination 
> physical machine,
> when migrating a domain.  It passes 100% while reloading memory pages,
> and keeps going.  I've seen this on an older build (an early 
> 2.0.x IIRC)
> but have never yet had time to dig into it.  Anyone else seen this?

Are you using the '-l' live option to migrate? If you are, going past
100% isn't actually a bug (though looping forever obviously is).

What are you seeing in the sending xfrd log? Have you used the ballon
driver or mem= command line options to the domain?

Also, what's running in the domain you are trying to migrate?

Best,
Ian


 
> Here's /var/log/xfrd.log:
> 
> 
> 9657 [INF] XFRD> Accepted connection from 151.155.190.17:32859 on 2
> 10346 [INF] XFRD> Xfr service for 151.155.190.17:32859
> [DEBUG] Conn_init> flags=1
> [DEBUG] Conn_init> write stream...
> [DEBUG] stream_init>mode=w flags=1 compress=0
> [DEBUG] stream_init> unbuffer...
> [DEBUG] stream_init< err=0
> [DEBUG] Conn_init> read stream...
> [DEBUG] stream_init>mode=r flags=1 compress=0
> [DEBUG] stream_init> unbuffer...
> [DEBUG] stream_init< err=0
> [DEBUG] Conn_sxpr>
> (xfr.hello 1 0)[DEBUG] Conn_sxpr< err=0
> [DEBUG] Conn_sxpr>
> (xfr.xfr 8)[DEBUG] Conn_sxpr< err=0
> [1111078930.273966] xc_linux_restore start
>  
> xc_linux_restore start
> [1111078930.287904] Created domain 1
>  
> Created domain 1
> (Domain-1 Domain-0)'domain id=1 name=SuSE.33 memory=256 console=9601
> image=/shared/boot/vmlinuz-2.6.10-xenU'[1111078930.389380] Reloading
> memory pages:   0%
> Reloading memory pages:   0  6%
> ^H^H^H^H  6 12%
> ^H^H^H^H 12 17%
> ^H^H^H^H 17 23%
> ^H^H^H^H 23 28%
> ^H^H^H^H 28 34%
> ^H^H^H^H 34 39%
> ^H^H^H^H 39 45%
> ^H^H^H^H 45 50%
> ^H^H^H^H 50 56%
> ^H^H^H^H 56 62%
> ^H^H^H^H 62 67%
> ^H^H^H^H 67 73%
> ^H^H^H^H 73 78%
> ^H^H^H^H 78 84%
> ^H^H^H^H 84 89%
> ^H^H^H^H 89 95%
> ^H^H^H^H 95100%
> ^H^H^H^H100105%
> ^H^H^H^H105110%
> ^H^H^H^H110115%
> ^H^H^H^H115120%
> ^H^H^H^H120125%
> ^H^H^H^H125130%
> ^H^H^H^H130135%
> ^H^H^H^H135140%
> ^H^H^H^H140145%
> ^H^H^H^H145150%
> ^H^H^H^H150155%
> 
> [chomp]
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from 
> real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

RE: xfrd inifinite loop reloading pages

[Charles Coffing]

It's a live migration.  The domain is idle -- booted, but basically 0%
cpu usage.  I gave the domain 256MB, but hadn't used the balloon driver.
 I left one machine receiving the migration for well over 10 minutes,
but it never unpaused the domain.  The xfrd log was just curning with
ever increasing percentages, no other messages.  

Also, it appears as if there is some hysteresis at work here.  Migration
failed on my first two attempts, but all subsequent attempts today have
worked.


Aaahhh... now I'm remembering better:  When I first saw this, months
back, I poked around and found some possible corner cases where EOF
could be missed in libxutil/file_stream.c and elsewhere.  I suspect
that's related.  Let me see if I can get a patch together for that...

Thanks.

 
>>>"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> 03/17/05 1:41 pm >>> 
>I'm seeing an infinite loop in xfrd on the destination 
>physical machine, 
>when migrating a domain.  It passes 100% while reloading memory pages, 
>and keeps going.  I've seen this on an older build (an early 
>2.0.x IIRC) 
>but have never yet had time to dig into it.  Anyone else seen this? 
 
Are you using the '-l' live option to migrate? If you are, going past 
100% isn't actually a bug (though looping forever obviously is). 
 
What are you seeing in the sending xfrd log? Have you used the ballon 
driver or mem= command line options to the domain? 
 
Also, what's running in the domain you are trying to migrate? 
 
Best, 
Ian 
 
 
 
>Here's /var/log/xfrd.log: 
> 
> 
>9657 [INF] XFRD> Accepted connection from 151.155.190.17:32859 on 2 
>10346 [INF] XFRD> Xfr service for 151.155.190.17:32859 
>[DEBUG] Conn_init> flags=1 
>[DEBUG] Conn_init> write stream... 
>[DEBUG] stream_init>mode=w flags=1 compress=0 
>[DEBUG] stream_init> unbuffer... 
>[DEBUG] stream_init< err=0 
>[DEBUG] Conn_init> read stream... 
>[DEBUG] stream_init>mode=r flags=1 compress=0 
>[DEBUG] stream_init> unbuffer... 
>[DEBUG] stream_init< err=0 
>[DEBUG] Conn_sxpr> 
>(xfr.hello 1 0)[DEBUG] Conn_sxpr< err=0 
>[DEBUG] Conn_sxpr> 
>(xfr.xfr 8)[DEBUG] Conn_sxpr< err=0 
>[1111078930.273966] xc_linux_restore start 
>  
>xc_linux_restore start 
>[1111078930.287904] Created domain 1 
>  
>Created domain 1 
>(Domain-1 Domain-0)'domain id=1 name=SuSE.33 memory=256 console=9601 
>image=/shared/boot/vmlinuz-2.6.10-xenU'[1111078930.389380] Reloading 
>memory pages:   0% 
>Reloading memory pages:   0  6% 
>^H^H^H^H  6 12% 
>^H^H^H^H 12 17% 
>^H^H^H^H 17 23% 
>^H^H^H^H 23 28% 
>^H^H^H^H 28 34% 
>^H^H^H^H 34 39% 
>^H^H^H^H 39 45% 
>^H^H^H^H 45 50% 
>^H^H^H^H 50 56% 
>^H^H^H^H 56 62% 
>^H^H^H^H 62 67% 
>^H^H^H^H 67 73% 
>^H^H^H^H 73 78% 
>^H^H^H^H 78 84% 
>^H^H^H^H 84 89% 
>^H^H^H^H 89 95% 
>^H^H^H^H 95100% 
>^H^H^H^H100105% 
>^H^H^H^H105110% 
>^H^H^H^H110115% 
>^H^H^H^H115120% 
>^H^H^H^H120125% 
>^H^H^H^H125130% 
>^H^H^H^H130135% 
>^H^H^H^H135140% 
>^H^H^H^H140145% 
>^H^H^H^H145150% 
>^H^H^H^H150155% 
> 
>[chomp] 
> 
> 
>------------------------------------------------------- 
>SF email is sponsored by - The IT Product Guide 
>Read honest & candid reviews on hundreds of IT Products from 
>real users. 
>Discover which products truly live up to the hype. Start reading now. 
>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click 
> 
>Xen-devel mailing list 
>Xen-devel@lists.sourceforge.net 
>https://lists.sourceforge.net/lists/listinfo/xen-devel 
> 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click