Strange events leading to "Hotplug scripts not working" and xen death

Strange events leading to "Hotplug scripts not working" and xen death

[Avleen Vig]

I sincerely apologise for posting this message here, after posting it on
xen-users only a day ago. But the more I look at the problem, the more
I'm convinced it is a bug. Unfortunately my debugging abilities are
limited as I'm not really a developer, but I'm trying :-)

Does anyone here have insight into this?
I'm also trying to run xend under pdb but having a few problems with
"address in use" errors which cause xend to die when it's not
daemonised.



System: NetBSD 4.0 BETA2 as of ~1 week ago, Xen 3 compiled from pkgsrc.

I have Dom0 working just fine. It boots without a problem.
I also have a file-backed Linux DomU which starts. The config for this
is at the end.

I've found a rather.. strange series of events that causes Xen to stop
creating new DomUs. Even my working Linux DomUs fail to create.
The thing that makes this strange, is that there doesn't seem to be one
single thing thing in particular that causes this. For instance,
following the instructions here on creating a FreeBSD DomU work the
first time, and after that everything stops working:
  
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/virtualization-guest.html

Also, downloading CentOS 5 from jailtime.org causes everything to stop
working. The configs I used for FreeBSD and CentOS5 are at the end also.

When things "stop working", I see messages like this when trying to
create a domain:
  [root@xen1] /xen/configs # xm create -c freebsd1
  Using config file "./freebsd1".
  Error: Device 0 (vif) could not be connected. Hotplug scripts not working.

Some digging indicated that the scripts (eg,
/usr/pkg/etc/xen/scripts/vif-bridge) are not actually being accessed any
more - their access time does not change again until I reboot to fix
this problem. Restarting xend or any other xen daemons doesn't work.

*Finally* :-) commands like 'xenstore-ls' just hang too.
The xenstore commands seem to be hanging trying to read from the store.
/var/run/xenstored/tdb is getting updated though, before 'xm create'
hangs.

I believe the problem may have something to do with the storage, but I'm
not sure.
The end of the xenstore-trace.log looks like this:

OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_END (OK )
IN  0x805b0b0 20070513 19:53:35 TRANSACTION_START ( )
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_START (447 )
IN  0x805b0b0 20070513 19:53:35 READ (/local/domain/244/device/vif/0/backend )
OUT 0x805b0b0 20070513 19:53:35 READ (/local/domain/0/backend/vif/244/0)
IN  0x805b0b0 20070513 19:53:35 TRANSACTION_END (T )
DESTROY transaction 0x8063230
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_END (OK )
IN  0x805b0b0 20070513 19:53:35 WATCH 
(/local/domain/0/backend/vif/244/0/hotplug-status 138635916 )
CREATE watch 0x8062030
OUT 0x805b0b0 20070513 19:53:35 WATCH (OK )
OUT 0x805b0b0 20070513 19:53:35 WATCH_EVENT 
(/local/domain/0/backend/vif/244/0/hotplug-status 138635916 )
IN  0x805b0b0 20070513 19:53:35 TRANSACTION_START ( )
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_START (448 )
IN  0x805b0b0 20070513 19:53:35 READ 
(/local/domain/0/backend/vif/244/0/hotplug-status )
OUT 0x805b0b0 20070513 19:53:35 ERROR (ENOENT )
IN  0x805b0b0 20070513 19:53:35 TRANSACTION_END (T )
DESTROY transaction 0x8063230
OUT 0x805b0b0 20070513 19:53:35 TRANSACTION_END (OK )


Can anyone help? The only suggestion that the many Google results on
"Hotplug scripts not working" errors say to make sure udev is working,
but I don't think that applies to NetBSD just Linux.


Working Linux DomU config:
  kernel = "/xen/kernels/vmlinuz-2.6.16.33-xen"
  memory = 128
  name = "Debian_Sarge_Image"
  vif = [ 'mac=00:11:22:33:44:55, bridge=bridge0' ]
  disk = [ 'file:/xen/nfs_root/sarge1/sarge1.img,hda1,w' ]
  root = "/dev/hda1 ro"
  extra = "init=/bin/sh"

FreeBSD 7 which causes breakage:
  kernel = "/xen/kernels/freebsd7-kernel"
  memory = 256
  name = "freebsd_xen_INSTALL"
  vif = [ 'mac=00:11:22:33:44:66, bridge=bridge0' ]
  disk = [ 'file:/xen/nfs_root/freebsd1/mdroot-7.0,hda1,w' ]
  extra = "boot_verbose"
  extra += ",boot_single"
  extra += ",kern.hz=100"
  extra += ",vfs.root.mountfrom=ufs:/dev/xbd769a"

CentOS 5 which causes breakage:
  kernel = "/xen/kernels/vmlinuz-2.6.16.33-xen"
  memory = 128
  name = "centos.5-0"
  vif = [ '' ]
  dhcp = "dhcp"
  disk = ['file:/xen/nfs_root/centos50/centos.5-0.img,sda1,w', 
'file:/xen/nfs_root/centos50/centos.swap,sda2,w']
  root = "/dev/sda1 ro"


-- 
Avleen Vig
Systems Administrator
Personal: www.silverwraith.com

"It's not me that's crazy, it's you."
"Monkey?"

Re: Strange events leading to "Hotplug scripts not working"and xen death

[Keir Fraser]

On 15/5/07 04:29, "Avleen Vig" <lists-xen@silverwraith.com> wrote:

> I sincerely apologise for posting this message here, after posting it on
> xen-users only a day ago. But the more I look at the problem, the more
> I'm convinced it is a bug. Unfortunately my debugging abilities are
> limited as I'm not really a developer, but I'm trying :-)
> 
> Does anyone here have insight into this?
> I'm also trying to run xend under pdb but having a few problems with
> "address in use" errors which cause xend to die when it's not
> daemonised.
> 
> System: NetBSD 4.0 BETA2 as of ~1 week ago, Xen 3 compiled from pkgsrc.

You may get more joy contacting the netbsd xen maintainer (whoever that is).
There's unlikely to be many users of netbsd dom0 on this list.

 -- Keir