Kernel modules errors and weird xl behavior with Xen 4.1

Kernel modules errors and weird xl behavior with Xen 4.1

[Claudiu Curcă]

Hello,

I'm using Gentoo Linux amd64 with the 2.6.34-xen-r4 kernel from portage.
I've been successfully using Xen 4.0 for almost a year, but now it
seems that with the dawn of Xen 4.1, the package maintainers have
forces unstable users to upgrade from 4.0 to 4.1

Looking forward to try the new xl interface, I installed it, but there
seem to be a few problems with xen-4.1

First off, xl seems to behave weird. For example, I have the following
domU config:

builder = "linux"
bootloader = "/usr/bin/pygrub"
vcpus = 1
memory = 512
name = "centos5_vm1"
root = "/dev/xvda1 ro"
disk = [ "file:/storage/xen/images/centos5-vm1.img,xvda,w" ]
vif = [ "ip = 172.18.0.225, mac = 00:16:3e:00:00:25" ]
vfb = [ "type = vnc, vncdisplay = 25, vnclisten = 172.18.0.1" ]

There are two issues if I start the domain using "xl create". The
first issue is that the machine has no network connectivity at all.
The second one is that the VNC session is not started. If I try to
connect to 172.18.0.1:5925, I get a "connection refused" error, and
netstat doesn't show anyone listening at that location. The domain
shows up in "xl list" as running. These problems do not show up if
using the old-fashion "xm create". A minor inconvenience, compared to
what's to come.

The big showstopper for this were some kernel problems. Regardless if
I start any domain, or even xencommons/xend, at some random time (at
most 5 minutes), random hardware components will fail.

For reference, here's the setup:

MB: Intel S5520HCV Motherboard
CPUs: 2x Intel Xeon E5520
RAM: 16GB DDR3
Networking: 2x Intel Gigabit (eth1+eth2 using igb kernel driver) and
1x Linksys Fast PCI Ethernet Adapter (eth0 using tulip kernel driver)
Storage: 3ware 9650SE-8ML RAID controller (using 3w-9xxx kernel driver)

eth0 is the internet adapter, eth1 is the LAN adapter (bridged to
xenbr0) and eth2 is another LAN adapter on a different network,
unbridged.

For example, this kills my network connection on eth0 (external
connection): http://pastebin.com/x8E9bxUT
Sometimes, the RAID controller would randomly stop working:

Apr 10 05:16:07 localhost kernel: [  646.720086] sd 4:0:0:0: WARNING:
(0x06:0x002C): Command (0x28) timed out, resetting card.
Apr 10 05:16:45 localhost kernel: [  684.816086] 3w-9xxx: scsi4:
WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting
card.

One last issue, which is also quite annoying... in Xen 4.1, if I start
a VM via xm create, then shut it down, when Xen is deleting the vif,
it somehow makes dhcpcd (I use it on eth0 to get the internet IP)
start on all interfaces (even peth1, other vif interfaces and those
who already have a static IP assigned to them), obviously causing the
machine to lose connectivity, as it gets 169.254.x.y autoassigned IPs
on those interfaces. Is this intended, or is is a bug?

None of these issues occur with xen-4.0

Has anyone else experienced such things with xen-4.1? At the moment
I've reverted to xen-4.0 using some older ebuilds in order to keep the
machine in a working state, but I am willing to do more tests in the
following days, if anyone wants to help me debug these things.

Re: [Xen-devel] Kernel modules errors and weird xl behavior with Xen 4.1

[Konrad Rzeszutek Wilk]

On Sun, Apr 10, 2011 at 02:28:23PM +0300, Claudiu Curcă wrote:
> Hello,
> 
> I'm using Gentoo Linux amd64 with the 2.6.34-xen-r4 kernel from portage.
> I've been successfully using Xen 4.0 for almost a year, but now it
> seems that with the dawn of Xen 4.1, the package maintainers have
> forces unstable users to upgrade from 4.0 to 4.1
> 
> Looking forward to try the new xl interface, I installed it, but there
> seem to be a few problems with xen-4.1
> 
> First off, xl seems to behave weird. For example, I have the following
> domU config:
> 
> builder = "linux"
> bootloader = "/usr/bin/pygrub"
> vcpus = 1
> memory = 512
> name = "centos5_vm1"
> root = "/dev/xvda1 ro"
> disk = [ "file:/storage/xen/images/centos5-vm1.img,xvda,w" ]
> vif = [ "ip = 172.18.0.225, mac = 00:16:3e:00:00:25" ]
> vfb = [ "type = vnc, vncdisplay = 25, vnclisten = 172.18.0.1" ]
> 
> There are two issues if I start the domain using "xl create". The
> first issue is that the machine has no network connectivity at all.
> The second one is that the VNC session is not started. If I try to
> connect to 172.18.0.1:5925, I get a "connection refused" error, and
> netstat doesn't show anyone listening at that location. The domain
> shows up in "xl list" as running. These problems do not show up if
> using the old-fashion "xm create". A minor inconvenience, compared to
> what's to come.

That just looks like a bug.
> 
> The big showstopper for this were some kernel problems. Regardless if
> I start any domain, or even xencommons/xend, at some random time (at
> most 5 minutes), random hardware components will fail.
> 
> For reference, here's the setup:
> 
> MB: Intel S5520HCV Motherboard
> CPUs: 2x Intel Xeon E5520
> RAM: 16GB DDR3
> Networking: 2x Intel Gigabit (eth1+eth2 using igb kernel driver) and
> 1x Linksys Fast PCI Ethernet Adapter (eth0 using tulip kernel driver)
> Storage: 3ware 9650SE-8ML RAID controller (using 3w-9xxx kernel driver)

OK. Try 'x2apic=off'

and please attach your /proc/interrupts before and after the failure.
> 
> eth0 is the internet adapter, eth1 is the LAN adapter (bridged to
> xenbr0) and eth2 is another LAN adapter on a different network,
> unbridged.
> 
> For example, this kills my network connection on eth0 (external
> connection): http://pastebin.com/x8E9bxUT
> Sometimes, the RAID controller would randomly stop working:
> 
> Apr 10 05:16:07 localhost kernel: [  646.720086] sd 4:0:0:0: WARNING:
> (0x06:0x002C): Command (0x28) timed out, resetting card.
> Apr 10 05:16:45 localhost kernel: [  684.816086] 3w-9xxx: scsi4:
> WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting
> card.
> 
> One last issue, which is also quite annoying... in Xen 4.1, if I start
> a VM via xm create, then shut it down, when Xen is deleting the vif,
> it somehow makes dhcpcd (I use it on eth0 to get the internet IP)
> start on all interfaces (even peth1, other vif interfaces and those
> who already have a static IP assigned to them), obviously causing the
> machine to lose connectivity, as it gets 169.254.x.y autoassigned IPs
> on those interfaces. Is this intended, or is is a bug?

That looks like a bug with your distro. Somehow it is listening on the
udev and firring off the dhcpd daemon whenever it detects the device
going on/off.

> 
> None of these issues occur with xen-4.0
> 
> Has anyone else experienced such things with xen-4.1? At the moment
> I've reverted to xen-4.0 using some older ebuilds in order to keep the
> machine in a working state, but I am willing to do more tests in the
> following days, if anyone wants to help me debug these things.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel