unpredictable Xen crash w NetBSD 5.0.2(XEN3PAE_DOMU)

unpredictable Xen crash w NetBSD 5.0.2(XEN3PAE_DOMU)

[彧]

Dear all:

I'm sorry I crossmail.

I try setup aoe-vblade server on netbsd 5.0.2(domU)
and I try to do some stress test with
for i in {65536}; do dd if=/dev/zero of=/dev/etherd/e?.? bs=4K;done
on a Linux box
Two Xen dom0 configurations I use:
1. 32bits SuSE Enterprise Linux 11sp1 2.6.32.12-0.7-xen with 32bits
Xen 4.0.0_21091_04-0.2.6
2. 64bits Gentoo 2.6.32-xen-r1 with 64bits Xen 4.0.0
The NIC is RTL-8169 on board

After 20~40mins later, whole system (include dom0 and Xen) will crash
and reboot.
xm dmesg
dmesg
system message log
xend.log
console
output nothing at crash

However if I replace NetBSD with FreeBSD 8.0p2 and SuSE JeOS 11.3
the stress test is stable (~6hours).

Can anyone give me sugguestions to figure out what's the problem?
Thank you

Angelo Liaw

Re: unpredictable Xen crash w NetBSD 5.0.2(XEN3PAE_DOMU)

[Jean-Yves Migeon]

On 25.09.2010 07:24, 彧 wrote:
> Dear all:
> 
> I'm sorry I crossmail.
> 
> I try setup aoe-vblade server on netbsd 5.0.2(domU)
> and I try to do some stress test with
> for i in {65536}; do dd if=/dev/zero of=/dev/etherd/e?.? bs=4K;done
> on a Linux box
> Two Xen dom0 configurations I use:
> 1. 32bits SuSE Enterprise Linux 11sp1 2.6.32.12-0.7-xen with 32bits
> Xen 4.0.0_21091_04-0.2.6
> 2. 64bits Gentoo 2.6.32-xen-r1 with 64bits Xen 4.0.0
> The NIC is RTL-8169 on board
> 
> After 20~40mins later, whole system (include dom0 and Xen) will crash
> and reboot.
> xm dmesg
> dmesg
> system message log
> xend.log
> console
> output nothing at crash
> 
> However if I replace NetBSD with FreeBSD 8.0p2 and SuSE JeOS 11.3
> the stress test is stable (~6hours).
> 
> Can anyone give me sugguestions to figure out what's the problem?

Only suggestions to isolate the problem first, sorry :/

I occasionally encountered weird reboots, due to (virtual) ethernet
problems. They were all memory related, under reasonable load in dom0,
with pages bouncing back and forth between dom0 and domU.

What I would try (first) is to keep the NetBSD (domU, correct?) running,
but comment out the vif line in the associated xmdomain.cfg file, or
detach it, before doing your stress tests.

I'd like to see your network setup in dom0 (brctl + ifconfig), if it's
possible, too.

-- 
Jean-Yves Migeon
jym@NetBSD.org

Re: unpredictable Xen crash w NetBSD 5.0.2(XEN3PAE_DOMU)

[彧]

Dear:
Finally, I downgrade Xen 4.0 to 3.4.2
And it stable, no more crash.

As Manuel Bouyer said, I believe it should be a bug in Xen4.0.
If I get more time, I'll try to find out what's the cause.

Thank you, all.

Angelo Liaw

2010/9/25 Jean-Yves Migeon <jym@netbsd.org>:
> On 25.09.2010 07:24, 彧 wrote:
>> Dear all:
>>
>> I'm sorry I crossmail.
>>
>> I try setup aoe-vblade server on netbsd 5.0.2(domU)
>> and I try to do some stress test with
>> for i in {65536}; do dd if=/dev/zero of=/dev/etherd/e?.? bs=4K;done
>> on a Linux box
>> Two Xen dom0 configurations I use:
>> 1. 32bits SuSE Enterprise Linux 11sp1 2.6.32.12-0.7-xen with 32bits
>> Xen 4.0.0_21091_04-0.2.6
>> 2. 64bits Gentoo 2.6.32-xen-r1 with 64bits Xen 4.0.0
>> The NIC is RTL-8169 on board
>>
>> After 20~40mins later, whole system (include dom0 and Xen) will crash
>> and reboot.
>> xm dmesg
>> dmesg
>> system message log
>> xend.log
>> console
>> output nothing at crash
>>
>> However if I replace NetBSD with FreeBSD 8.0p2 and SuSE JeOS 11.3
>> the stress test is stable (~6hours).
>>
>> Can anyone give me sugguestions to figure out what's the problem?
>
> Only suggestions to isolate the problem first, sorry :/
>
> I occasionally encountered weird reboots, due to (virtual) ethernet
> problems. They were all memory related, under reasonable load in dom0,
> with pages bouncing back and forth between dom0 and domU.
>
> What I would try (first) is to keep the NetBSD (domU, correct?) running,
> but comment out the vif line in the associated xmdomain.cfg file, or
> detach it, before doing your stress tests.
NetBSD works fine and lives long with/without vif and xbd, under
normal operating ^^
>
> I'd like to see your network setup in dom0 (brctl + ifconfig), if it's
> possible, too.
>
It's default bridge setting from SLES and Xen
It would be like this
brectl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.001d7d004f19       no              eth0
                                                        vif6.0

ifconfig
br0       Link encap:Ethernet  HWaddr 00:1D:7D:00:4F:19
          inet addr:140.112.90.52  Bcast:140.112.90.255  Mask:255.255.255.0
          inet6 addr: 2001:288:1001:28:21d:7dff:fe00:4f19/64 Scope:Global
          inet6 addr: fe80::21d:7dff:fe00:4f19/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:103545441 errors:0 dropped:0 overruns:0 frame:0
          TX packets:91095087 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1610450951 (1535.8 Mb)  TX bytes:3374297172 (3217.9 Mb)

eth0      Link encap:Ethernet  HWaddr 00:1D:7D:00:4F:19
          inet6 addr: fe80::21d:7dff:fe00:4f19/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:104136292 errors:0 dropped:0 overruns:0 frame:0
          TX packets:91107012 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3199000667 (3050.8 Mb)  TX bytes:3379922283 (3223.3 Mb)
          Interrupt:179 Base address:0x8000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:6839 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6839 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:137634 (134.4 Kb)  TX bytes:137634 (134.4 Kb)

vif6.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:20 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3627360 errors:0 dropped:29 overruns:0 carrier:0
          collisions:0 txqueuelen:32
          RX bytes:2656 (2.5 Kb)  TX bytes:342575888 (326.7 Mb)

However, as my downgrade test, I think it's Xen4.0's problem. :P
Thank you
> --
> Jean-Yves Migeon
> jym@NetBSD.org
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: unpredictable Xen crash w NetBSD 5.0.2(XEN3PAE_DOMU)

[Manuel Bouyer]

On Sat, Sep 25, 2010 at 01:24:32PM +0800, ? wrote:
> Dear all:
> 
> I'm sorry I crossmail.
> 
> I try setup aoe-vblade server on netbsd 5.0.2(domU)
> and I try to do some stress test with
> for i in {65536}; do dd if=/dev/zero of=/dev/etherd/e?.? bs=4K;done
> on a Linux box
> Two Xen dom0 configurations I use:
> 1. 32bits SuSE Enterprise Linux 11sp1 2.6.32.12-0.7-xen with 32bits
> Xen 4.0.0_21091_04-0.2.6
> 2. 64bits Gentoo 2.6.32-xen-r1 with 64bits Xen 4.0.0
> The NIC is RTL-8169 on board

Sorry, but I don't understand what your're trying to do.
What is /dev/etherd/e?.? ? how does it relate to the NetBSD domU ?

BTW, any chance to try NetBSD 5.1_RC4 instead of 5.0.2 ?

-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--

Re: unpredictable Xen crash w NetBSD 5.0.2(XEN3PAE_DOMU)

[彧]

Sorry for my English.
Let me explain more detail what I would like to do.
Here is my setup.

1. One Linux box(D0) acts as dom0 with Xen4.0

2. One NetBSD 5.0.2(Du502) act as domU on D0
2.1 pkg_add aoe-vblade in Du502 and run as an AoE target[1,2], AoE is
similar to iSCSI
2.2 ex: vblade 1 1 xennet0 /TEST.img

3. Another Linux box(C1) issues read/write requests with dd
3.1. the /dev/etherd/e?.?[3] is AoE target block device which is
exported by Du502 and
       is showed under C1's /dev/etherd path
3.2 ex: modprobe aoe # ls /dev/etherd will find /dev/etherd/e1.1
           for i in {1..65535}; dd if=/dev/zero of=/dev/etherd/e1.1 bs=4K; done

4. wait for 20~40mins, D0 reboot without any warning.
  dd of=/dev/zero if=/dev/etherd/e1.1

And Yes, I tried
ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-5.1_RC4/i386/binary/kernel/netbsd-XEN3PAE_DOMU.gz
I'll try ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-4.0.1/i386/binary/kernel/netbsd-XEN3_DOMU.gz
netbsd-current and Xen 3.4.2 later, but it would take a little time
And I try figure out how to remote debug netbsd kernel.(Is it possible?)

Angelo Liaw
--
*1 http://pkgsrc.se/net/aoe-vblade
*2 https://help.ubuntu.com/community/ATAOverEthernet  Section "#To
create an AoE device"
*3 https://help.ubuntu.com/community/ATAOverEthernet  Section "#Using AoE"


2010/9/25 Manuel Bouyer <bouyer@antioche.eu.org>:
> On Sat, Sep 25, 2010 at 01:24:32PM +0800, ? wrote:
>> Dear all:
>>
>> I'm sorry I crossmail.
>>
>> I try setup aoe-vblade server on netbsd 5.0.2(domU)
>> and I try to do some stress test with
>> for i in {65536}; do dd if=/dev/zero of=/dev/etherd/e?.? bs=4K;done
>> on a Linux box
>> Two Xen dom0 configurations I use:
>> 1. 32bits SuSE Enterprise Linux 11sp1 2.6.32.12-0.7-xen with 32bits
>> Xen 4.0.0_21091_04-0.2.6
>> 2. 64bits Gentoo 2.6.32-xen-r1 with 64bits Xen 4.0.0
>> The NIC is RTL-8169 on board
>
> Sorry, but I don't understand what your're trying to do.
> What is /dev/etherd/e?.? ? how does it relate to the NetBSD domU ?
>
> BTW, any chance to try NetBSD 5.1_RC4 instead of 5.0.2 ?
>
> --
> Manuel Bouyer <bouyer@antioche.eu.org>
>     NetBSD: 26 ans d'experience feront toujours la difference
> --
>

Re: unpredictable Xen crash w NetBSD 5.0.2(XEN3PAE_DOMU)

[Manuel Bouyer]

On Sat, Sep 25, 2010 at 09:36:16PM +0800, ? wrote:
> Sorry for my English.
> Let me explain more detail what I would like to do.
> Here is my setup.
> 
> 1. One Linux box(D0) acts as dom0 with Xen4.0
> 
> 2. One NetBSD 5.0.2(Du502) act as domU on D0
> 2.1 pkg_add aoe-vblade in Du502 and run as an AoE target[1,2], AoE is
> similar to iSCSI
> 2.2 ex: vblade 1 1 xennet0 /TEST.img
> 
> 3. Another Linux box(C1) issues read/write requests with dd
> 3.1. the /dev/etherd/e?.?[3] is AoE target block device which is
> exported by Du502 and
>        is showed under C1's /dev/etherd path
> 3.2 ex: modprobe aoe # ls /dev/etherd will find /dev/etherd/e1.1
>            for i in {1..65535}; dd if=/dev/zero of=/dev/etherd/e1.1 bs=4K; done
> 
> 4. wait for 20~40mins, D0 reboot without any warning.

looks like a Xen or dom0 bug. Even if NetBSD is doing something
forbidden, this should not crash the hypervisor or dom0.

>   dd of=/dev/zero if=/dev/etherd/e1.1
> 
> And Yes, I tried
> ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-5.1_RC4/i386/binary/kernel/netbsd-XEN3PAE_DOMU.gz
> I'll try ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-4.0.1/i386/binary/kernel/netbsd-XEN3_DOMU.gz
> netbsd-current and Xen 3.4.2 later, but it would take a little time
> And I try figure out how to remote debug netbsd kernel.(Is it possible?)

there is still some KGDB support around but I've never tried using it.
I usually use the in-kernel debugger on the console

-- 
Manuel Bouyer <bouyer@antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--