stubdom fs-backend issue

stubdom fs-backend issue

[Keith Coleman]

[-- Attachment #1.1: Type: text/plain, Size: 2160 bytes --]

I initially encountered this issue testing live migration with stubdoms. The
first 19 live migrations work but the domU and domU-dm freeze on the 20th
attempt. The same behavior is observed when doing save/restore cycles.

The issue seems to be related to fs-backend. After fs-backend has launched
20 stubdoms it is no longer able to restore a domU.

Restarting fs-backend after 19 uses avoids this issue but orphans any
running stubdoms, causing them to lock up.

I have tried various flavors of linux and windows in domU. On dom0 I
have tried xen 3.3.0, 3.3.1, 3.3.2-rc1-pre on the official 2.6.18.8 and the
latest lenny dom0 kernel. I tried 32bit and amd64. The issue exists across
the board.

To help debug the problem I am attaching several logs. Here are the steps I
took:
strace fs-backend 1>fs-backend-strace-1 2>fs-backend-strace-2
xm create win3-stub
xm save win3 win3.save; sleep 2; xm restore win3.save   (did this 19 times)
xm save win3 win3.save
xm restore win3.save
At this point the domU and domU-dm are not functioning:
xm list
Name                                        ID   Mem VCPUs      State
Time(s)
Domain-0                                     0   512     2     r-----
254.6
win3                                        41   512     1     ------
0.0
win3-dm                                     42    32     1     -b----
0.0

destroyed win3 and killed fs-backend

I have attached the last several qemu logs that show the domUs working and
the last one that didn't work, xend.log (starting when I booted the dom0
machine), domain-builder-ng.log and the output of stracing fs-backend:

qemu-dm-win3-dm.log
qemu-dm-win3-dm.log.1
qemu-dm-win3-dm.log.2
qemu-dm-win3-dm.log.3
qemu-dm-win3.log
qemu-dm-win3.log.1
qemu-dm-win3.log.2
qemu-dm-win3.log.3
xend.log
domain-builder-ng.log
fs-backend-strace-1 (~90MB uncompressed)
fs-backend-strace-2
I hope someone can take a few minutes to reproduce this issue. I mentioned
it on the users list but got no responses and haven't been able to find
anyone successfully using stubdoms that has time to test. When stubdoms are
working they work so well that I don't want to give up.

Thanks,
Keith Coleman

[-- Attachment #1.2: Type: text/html, Size: 3707 bytes --]

[-- Attachment #2: fs-backend-error-00.tgz.sha1sum --]
[-- Type: text/plain, Size: 66 bytes --]

1614629183c8677016fe7e3f24e15c3a22ef7310  fs-backend-error-00.tgz

[-- Attachment #3: fs-backend-error-00.tgz --]
[-- Type: application/x-gzip, Size: 3551741 bytes --]

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

stubdom fs-backend issue

[Keith Coleman]

I initially encountered this issue testing live migration with
stubdoms. The first 19 live migrations work but the domU and domU-dm
freeze on the 20th attempt. The same behavior is observed when doing
save/restore cycles.

The issue seems to be related to fs-backend. After fs-backend has
launched 20 stubdoms it is no longer able to restore a domU.

Restarting fs-backend after 19 uses avoids this issue but orphans any
running stubdoms, causing them to lock up.

I have tried various flavors of linux and windows in domU. On dom0 I
have tried xen 3.3.0, 3.3.1, 3.3.2-rc1-pre on the official 2.6.18.8
and the latest lenny dom0 kernel. I tried 32bit and amd64. The issue
exists across the board.

To help debug the problem I am attaching several logs. Here are the
steps I took:
strace fs-backend 1>fs-backend-strace-1 2>fs-backend-strace-2
xm create win3-stub
xm save win3 win3.save; sleep 2; xm restore win3.save   (did this 19 times)
xm save win3 win3.save
xm restore win3.save
At this point the domU and domU-dm are not functioning:
xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   512     2     r-----    254.6
win3                                        41   512     1     ------      0.0
win3-dm                                     42    32     1     -b----      0.0

destroyed win3 and killed fs-backend

I have uploaded the last several qemu logs that show the domUs working
and the last one that didn't work, xend.log (starting when I booted
the dom0 machine), domain-builder-ng.log and the output of stracing
fs-backend:

https://svn.8svn.com/fs-backend-error-00.tgz (3.5 MB)
sha1sum: 1614629183c8677016fe7e3f24e15c3a22ef7310  fs-backend-error-00.tgz

qemu-dm-win3-dm.log
qemu-dm-win3-dm.log.1
qemu-dm-win3-dm.log.2
qemu-dm-win3-dm.log.3
qemu-dm-win3.log
qemu-dm-win3.log.1
qemu-dm-win3.log.2
qemu-dm-win3.log.3
xend.log
domain-builder-ng.log
fs-backend-strace-1 (~90MB uncompressed)
fs-backend-strace-2

I hope someone can take a few minutes to reproduce this issue. I
mentioned it on the users list but got no responses and haven't been
able to find anyone successfully using stubdoms that has time to test.
When stubdoms are working they work so well that I don't want to give
up.

Thanks,
Keith Coleman

Re: stubdom fs-backend issue

[Stefano Stabellini]

Keith Coleman wrote:

> I initially encountered this issue testing live migration with
> stubdoms. The first 19 live migrations work but the domU and domU-dm
> freeze on the 20th attempt. The same behavior is observed when doing
> save/restore cycles.


I am able to reproduce the issue, I'll try to come up with a fix.

Re: stubdom fs-backend issue

[Keith Coleman]

On Fri, Feb 20, 2009 at 12:05 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> Keith Coleman wrote:
>
>> I initially encountered this issue testing live migration with
>> stubdoms. The first 19 live migrations work but the domU and domU-dm
>> freeze on the 20th attempt. The same behavior is observed when doing
>> save/restore cycles.
>
>
> I am able to reproduce the issue, I'll try to come up with a fix.
>
>

Do we need to file a bug report for this issue?

Keith Coleman

Re: stubdom fs-backend issue

[Stefano Stabellini]

Keith Coleman wrote:

> On Fri, Feb 20, 2009 at 12:05 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
>> Keith Coleman wrote:
>>
>>> I initially encountered this issue testing live migration with
>>> stubdoms. The first 19 live migrations work but the domU and domU-dm
>>> freeze on the 20th attempt. The same behavior is observed when doing
>>> save/restore cycles.
>>
>> I am able to reproduce the issue, I'll try to come up with a fix.
>>
>>
> 
> Do we need to file a bug report for this issue?
> 

No need: I am fixing this issue right now.
Unfortunately fs-backend has more problems than I though, so the fix is
taking time.