can't start domU

can't start domU

[Charles Coffing]

I've been getting hangs while attempting to start domUs on recent
changesets.  I get an error such as:
Error: Device 0 (vif) could not be connected. Hotplug scripts not
working.

Yet if I run diagnose.py on the now- paused domU, I get this:
# python diagnose.py 2
Domain ID is 2.
Domain name is ExampleDomain.
Found 2 device classes in use.
Found 1 vbd devices.
Found device vbd, 769.
Backend is stuck waiting for frontend for device vbd, 769.
Device vbd, 769 hotplugging has completed successfully.
Found 0 vif devices.

Any idea why the front end would be stuck?

I'm running udev 068.

Re: can't start domU

[Nivedita Singhvi]

Charles Coffing wrote:
> I've been getting hangs while attempting to start domUs on recent
> changesets.  I get an error such as:
> Error: Device 0 (vif) could not be connected. Hotplug scripts not
> working.
> 
> Yet if I run diagnose.py on the now- paused domU, I get this:
> # python diagnose.py 2
> Domain ID is 2.
> Domain name is ExampleDomain.
> Found 2 device classes in use.
> Found 1 vbd devices.
> Found device vbd, 769.
> Backend is stuck waiting for frontend for device vbd, 769.
> Device vbd, 769 hotplugging has completed successfully.
> Found 0 vif devices.
> 
> Any idea why the front end would be stuck?
> 
> I'm running udev 068.

All of these are failures to create the vif devices.
Please check /var/log/messages, xend-debug.log, etc.
You can put a set -x in your /etc/xen/vif-bridge (or equivalent)
script for more verbose detail.

Also, check to see that your install went successfully.

 From previous debugging over the past week, most of
these cases were the lack of brctl (or execute perms
for root), or the lack of other utilities (make install
should catch these).

thanks,
Nivedita

Re: can't start domU

[Charles Coffing]

>>> On Fri, Nov 11, 2005 at  3:44 pm, in message
<43751EDA.80304@us.ibm.com>,
Nivedita Singhvi <niv@us.ibm.com> wrote: 
> Charles Coffing wrote:
>> I've been getting hangs while attempting to start domUs on recent
>> changesets.  I get an error such as:
>> Error: Device 0 (vif) could not be connected. Hotplug scripts not
>> working.
[snip]
> 
> All of these are failures to create the vif devices.
> Please check /var/log/messages, xend- debug.log, etc.
> You can put a set - x in your /etc/xen/vif- bridge (or equivalent)
> script for more verbose detail.
> 
> Also, check to see that your install went successfully.
> 
>  From previous debugging over the past week, most of
> these cases were the lack of brctl (or execute perms
> for root), or the lack of other utilities (make install
> should catch these).

The obvious stuff looks good:  The install went okay; brctl exists; all
scripts are executable.

I'll look more closely at the bridge script.

Thanks,
Charles

Re: can't start domU

[Steve Dobbelstein]

"Charles Coffing" <ccoffing@novell.com> wrote on 11/11/2005 05:13:25 PM:

> >>> On Fri, Nov 11, 2005 at  3:44 pm, in message
> <43751EDA.80304@us.ibm.com>,
> Nivedita Singhvi <niv@us.ibm.com> wrote:
> > Charles Coffing wrote:
> >> I've been getting hangs while attempting to start domUs on recent
> >> changesets.  I get an error such as:
> >> Error: Device 0 (vif) could not be connected. Hotplug scripts not
> >> working.
> [snip]
> >
> > All of these are failures to create the vif devices.
> > Please check /var/log/messages, xend- debug.log, etc.
> > You can put a set - x in your /etc/xen/vif- bridge (or equivalent)
> > script for more verbose detail.
> >
> > Also, check to see that your install went successfully.
> >
> >  From previous debugging over the past week, most of
> > these cases were the lack of brctl (or execute perms
> > for root), or the lack of other utilities (make install
> > should catch these).
>
> The obvious stuff looks good:  The install went okay; brctl exists; all
> scripts are executable.
>
> I'll look more closely at the bridge script.

Has anyone found a solution to this problem?  I'm seeing the same thing but
didn't want to just post a "me, too".

My domU config file does not have a "vif" entry; the defaults are fine for
me.

/var/log/messages, /var/log/xend-debug.log, and /var/log/xend.log don't
contain any additional information; they look the same as the ones on a
system that is working.

I added lines to /etc/xen/scripts/vif-bridge:
echo "Turning debugging on"
echo "turning debugging on">/tmp/vif-bridge.out
set -x

I don't get any debug output upon running "xm create".  I don't get a file
/tmp/vif-bridge, so it appears that vif-bridge isn't even called.

hotplug is at version 058.  I tried the "echo /sbin/hotplug >
/proc/sys/kernel/hotplug" trick, but that didn't help.  Besides, the system
that works is also running hotplug 058 and it has /proc/sys/kernel/hotplug
set to /sbin/udevsend.

Apparently something is not creating the vif for the domain.  I have been
crawling through the xend scripts to try and figure out where vifs get
created in hopes of finding some clues as to what is going wrong, but being
barely a python newbie I get lost in the code.  :(  (Time to add another
programming language to my skill set.)

Any solutions or pointers on how to proceed would be appreciated.

Steve D.

Re: can't start domU

[Ewan Mellor]

On Tue, Nov 15, 2005 at 01:59:25PM -0600, Steve Dobbelstein wrote:

> "Charles Coffing" <ccoffing@novell.com> wrote on 11/11/2005 05:13:25 PM:
> 
> > >>> On Fri, Nov 11, 2005 at  3:44 pm, in message
> > <43751EDA.80304@us.ibm.com>,
> > Nivedita Singhvi <niv@us.ibm.com> wrote:
> > > Charles Coffing wrote:
> > >> I've been getting hangs while attempting to start domUs on recent
> > >> changesets.  I get an error such as:
> > >> Error: Device 0 (vif) could not be connected. Hotplug scripts not
> > >> working.
> > [snip]
> > >
> > > All of these are failures to create the vif devices.
> > > Please check /var/log/messages, xend- debug.log, etc.
> > > You can put a set - x in your /etc/xen/vif- bridge (or equivalent)
> > > script for more verbose detail.
> > >
> > > Also, check to see that your install went successfully.
> > >
> > >  From previous debugging over the past week, most of
> > > these cases were the lack of brctl (or execute perms
> > > for root), or the lack of other utilities (make install
> > > should catch these).
> >
> > The obvious stuff looks good:  The install went okay; brctl exists; all
> > scripts are executable.
> >
> > I'll look more closely at the bridge script.
> 
> Has anyone found a solution to this problem?  I'm seeing the same thing but
> didn't want to just post a "me, too".
> 
> My domU config file does not have a "vif" entry; the defaults are fine for
> me.
> 
> /var/log/messages, /var/log/xend-debug.log, and /var/log/xend.log don't
> contain any additional information; they look the same as the ones on a
> system that is working.
> 
> I added lines to /etc/xen/scripts/vif-bridge:
> echo "Turning debugging on"
> echo "turning debugging on">/tmp/vif-bridge.out
> set -x
> 
> I don't get any debug output upon running "xm create".  I don't get a file
> /tmp/vif-bridge, so it appears that vif-bridge isn't even called.
> 
> hotplug is at version 058.  I tried the "echo /sbin/hotplug >
> /proc/sys/kernel/hotplug" trick, but that didn't help.  Besides, the system
> that works is also running hotplug 058 and it has /proc/sys/kernel/hotplug
> set to /sbin/udevsend.
> 
> Apparently something is not creating the vif for the domain.  I have been
> crawling through the xend scripts to try and figure out where vifs get
> created in hopes of finding some clues as to what is going wrong, but being
> barely a python newbie I get lost in the code.  :(  (Time to add another
> programming language to my skill set.)

Have you tried today's changeset?  There have been plenty of changes in this
area in the last week, and a few today.

What distro?  (Not that this will help, but it seems a good idea to keep track
of problematic distro versions.)

Does /sbin/hotplug or /sbin/udevsend run at all?

If so, does it make it as far as /etc/hotplug/xen-backend.agent?

If so, how about /etc/xen/scripts/vif-common.sh?

What do you get if you run

PYTHONPATH=/usr/lib/python python /usr/lib/python/xen/util/diagnose.py <dom ID>

Thanks for your help,

Ewan.

Re: can't start domU

[Steve Dobbelstein]

Ewan Mellor <ewan@xensource.com> wrote on 11/15/2005 05:34:48 PM:

Thanks for your assitance, Ewan.

> On Tue, Nov 15, 2005 at 01:59:25PM -0600, Steve Dobbelstein wrote:
>
> > "Charles Coffing" <ccoffing@novell.com> wrote on 11/11/2005 05:13:25
PM:
> >
> > > >>> On Fri, Nov 11, 2005 at  3:44 pm, in message
> > > <43751EDA.80304@us.ibm.com>,
> > > Nivedita Singhvi <niv@us.ibm.com> wrote:
> > > > Charles Coffing wrote:
> > > >> I've been getting hangs while attempting to start domUs on recent
> > > >> changesets.  I get an error such as:
> > > >> Error: Device 0 (vif) could not be connected. Hotplug scripts not
> > > >> working.
> > > [snip]
> > > >
> > > > All of these are failures to create the vif devices.
> > > > Please check /var/log/messages, xend- debug.log, etc.
> > > > You can put a set - x in your /etc/xen/vif- bridge (or equivalent)
> > > > script for more verbose detail.
> > > >
> > > > Also, check to see that your install went successfully.
> > > >
> > > >  From previous debugging over the past week, most of
> > > > these cases were the lack of brctl (or execute perms
> > > > for root), or the lack of other utilities (make install
> > > > should catch these).
> > >
> > > The obvious stuff looks good:  The install went okay; brctl exists;
all
> > > scripts are executable.
> > >
> > > I'll look more closely at the bridge script.
> >
> > Has anyone found a solution to this problem?  I'm seeing the same thing
but
> > didn't want to just post a "me, too".
> >
> > My domU config file does not have a "vif" entry; the defaults are fine
for
> > me.
> >
> > /var/log/messages, /var/log/xend-debug.log, and /var/log/xend.log don't
> > contain any additional information; they look the same as the ones on a
> > system that is working.
> >
> > I added lines to /etc/xen/scripts/vif-bridge:
> > echo "Turning debugging on"
> > echo "turning debugging on">/tmp/vif-bridge.out
> > set -x
> >
> > I don't get any debug output upon running "xm create".  I don't get a
file
> > /tmp/vif-bridge, so it appears that vif-bridge isn't even called.
> >
> > hotplug is at version 058.  I tried the "echo /sbin/hotplug >
> > /proc/sys/kernel/hotplug" trick, but that didn't help.  Besides, the
system
> > that works is also running hotplug 058 and it has
/proc/sys/kernel/hotplug
> > set to /sbin/udevsend.
> >
> > Apparently something is not creating the vif for the domain.  I have
been
> > crawling through the xend scripts to try and figure out where vifs get
> > created in hopes of finding some clues as to what is going wrong, but
being
> > barely a python newbie I get lost in the code.  :(  (Time to add
another
> > programming language to my skill set.)
>
> Have you tried today's changeset?  There have been plenty of changes in
this
> area in the last week, and a few today.

I grabbed the latest source, 7826, and still have the problem.

> What distro?  (Not that this will help, but it seems a good idea to keep
track
> of problematic distro versions.)

Fedora Core 4 for both the dom0 and the domU

> Does /sbin/hotplug or /sbin/udevsend run at all?
>
> If so, does it make it as far as /etc/hotplug/xen-backend.agent?

I'm not sure how to tell if /sbin/udevsend binary gets run.  So I did the
"echo /sbin/hotplug > /proc/sys/kernel/hotplug" trick so that I could hack
debugging stuff into the /etc/hotplug/xen-backend.agent script.  It now
looks like:

-----------
#! /bin/sh

date=`date`
echo "$date $0 $*" >> /var/log/xen-backend.log
echo "XENBUS_TYPE = \"$XENBUS_TYPE\"  ACTION = \"$ACTION\"" >>
/var/log/xen-backend.log
PATH=/etc/xen/scripts:$PATH

case "$XENBUS_TYPE" in
  vbd)
    echo "/etc/xen/scripts/block $ACTION" >> /var/log/xen-backend.log
    /etc/xen/scripts/block "$ACTION"
    ;;
  vif)
    if [ -n "$script" ]; then
      echo "$script $ACTION" >> /var/log/xen-backend.log
    else
      echo "\$script is empty" >> /var/log/xen-backend.log
    fi
    [ -n "$script" ] && $script "$ACTION"
    ;;
esac

case "$ACTION" in
  add)
    ;;
  remove)
    # remove device backend store entries
    xenstore-rm -t "$XENBUS_PATH"       || true
    xenstore-rm -t "error/$XENBUS_PATH" || true
    ;;
  online)
    ;;
  offline)
    ;;
esac
-----------

After I run "xm create /tmp/dom1/cfg" /var/log/xen-backend.log has:

-----------
Wed Nov 16 10:54:37 CST 2005 /etc/hotplug/xen-backend.agent
XENBUS_TYPE = "vbd"  ACTION = "add"
/etc/xen/scripts/block add
Wed Nov 16 10:54:37 CST 2005 /etc/hotplug/xen-backend.agent
XENBUS_TYPE = "vbd"  ACTION = "add"
/etc/xen/scripts/block add
-----------

Looks like it doesn't get called to create the vif.

> If so, how about /etc/xen/scripts/vif-common.sh?

I added similar debugging statements to /etc/xen/scripts/vif-bridge and
/etc/xen/scripts/vif-common.sh.  I don't get any log file for either vif-*
script.  Looks like they are not called.

> What do you get if you run
>
> PYTHONPATH=/usr/lib/python python /usr/lib/python/xen/util/diagnose.
> py <dom ID>

-----------
[root@gir scripts]# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      768     4 r-----   112.4
dom1                               4      768     4 -b----    12.3
[root@gir scripts]# PYTHONPATH=/usr/lib/python python
/usr/lib/python/xen/util/diagnose.py 4
Domain ID is 4.
Domain name is dom1.
Found 1 device classes in use.
Found 2 vbd devices.
Found device vbd, 2049.
Backend is in state Connected.
Frontend is in state Connected.
Device vbd, 2049 hotplugging has completed successfully, and is connected
to physical device 0xfd01.
Found device vbd, 2050.
Backend is in state Connected.
Frontend is in state Connected.
Device vbd, 2050 hotplugging has completed successfully, and is connected
to physical device 0xfd00.
-----------

FWIW, /tmp/dom1.cfg says:

-----------
kernel = "/boot/vmlinuz-xenU-smp"
memory = 768
name = "dom1"
vcpus = 4
nics = 1
disk = ['phy:/dev/virt-blkdev-backend/dom1,sda1,w',
'phy:/dev/virt-blkdev-backend/usr,sda2,r']
root = "/dev/sda1 ro"
extra = " profile=1 GATEWAY=10.0.1.254 NETMASK=255.255.0.0
IPADDR=10.0.134.1 HOSTNAME=dom1"
on_poweroff = 'destroy'
on_reboot   = 'destroy'
on_crash    = 'destroy'
-----------

> Thanks for your help,

Thanks for yours.

> Ewan.

Steve D.

Re: can't start domU

[Ewan Mellor]

On Wed, Nov 16, 2005 at 11:19:12AM -0600, Steve Dobbelstein wrote:

> I'm not sure how to tell if /sbin/udevsend binary gets run.

mv /sbin/udevsend{,.real}
cat >/sbin/udevsend
#!/bin/sh
echo "udevsend running" >>/var/log/udev.log
exec /sbin/udevsend.real
^D

ought to do it.  I think that it's clear that the whole hotplug subsystem isn't
running in your case, so I wouldn't worry about it.

> [root@gir scripts]# PYTHONPATH=/usr/lib/python python
> /usr/lib/python/xen/util/diagnose.py 4
> Domain ID is 4.
> Domain name is dom1.
> Found 1 device classes in use.
> Found 2 vbd devices.
> Found device vbd, 2049.
> Backend is in state Connected.
> Frontend is in state Connected.
> Device vbd, 2049 hotplugging has completed successfully, and is connected
> to physical device 0xfd01.
> Found device vbd, 2050.
> Backend is in state Connected.
> Frontend is in state Connected.
> Device vbd, 2050 hotplugging has completed successfully, and is connected
> to physical device 0xfd00.

This suggests that the tools have not even got as far as trying to create a
vif for you.  Are you getting the "Device 0 (vif) could not be connected.
Hotplug scripts not working." message, same as Charles Coffing?  That really
would be very curious.  In any case, your config file ought to be causing Xend
to at least try to create a vif.  I will need to see your /var/log/xend.log to
pin this down, and if you can get a xenstore trace, that will be useful too.

Cheers,

Ewan.

Re: can't start domU

[Ewan Mellor]

On Wed, Nov 16, 2005 at 06:45:39PM +0000, Ewan Mellor wrote:

> On Wed, Nov 16, 2005 at 11:19:12AM -0600, Steve Dobbelstein wrote:
> 
> > I'm not sure how to tell if /sbin/udevsend binary gets run.
> 
> mv /sbin/udevsend{,.real}
> cat >/sbin/udevsend
> #!/bin/sh
> echo "udevsend running" >>/var/log/udev.log
> exec /sbin/udevsend.real
> ^D

Followed by 

chmod a+x /sbin/udevsend

of course.

Ewan.

Re: can't start domU

[peter Bier]

Ewan Mellor <ewan <at> xensource.com> writes:

> 
> On Wed, Nov 16, 2005 at 06:45:39PM +0000, Ewan Mellor wrote:
> 
> > On Wed, Nov 16, 2005 at 11:19:12AM -0600, Steve Dobbelstein wrote:
> > 
> > > I'm not sure how to tell if /sbin/udevsend binary gets run.
> > 
> > mv /sbin/udevsend{,.real}
> > cat >/sbin/udevsend
> > #!/bin/sh
> > echo "udevsend running" >>/var/log/udev.log
> > exec /sbin/udevsend.real
> > ^D
> 
> Followed by 
> 
> chmod a+x /sbin/udevsend
> 
> of course.
> 
> Ewan.
> 


I get the same error message when starting up a domain and I found out, that 
xenrestore says ENOENT when it gets the 
request "read /local/domain/0/backend/vif/20/0/hotplug-status", which I 
found out by setting the environment variable XENSTORED_TRACE and looking
into the file /var/log/senstored-trace.log

the output of the diagnose script is:

Domain ID is 20.
Domain name is ExampleDomain.
Found 2 device classes in use.
Found 4 vbd devices.
Found device vbd, 769.
Backend is in state Initialising.
Frontend is in state Initialising.
Device vbd, 769 hotplugging failed.
Found device vbd, 833.
Backend is in state Initialising.
Frontend is in state Initialising.
Device vbd, 833 hotplugging failed.
Found device vbd, 8705.
Backend is in state Initialising.
Frontend is in state Initialising.
Device vbd, 8705 hotplugging failed.
Found device vbd, 8451.
Backend is in state Initialising.
Frontend is in state Initialising.
Device vbd, 8451 hotplugging failed.
Found 1 vif devices.
Found device vif, 0.
Backend is in state Closing.
Frontend is in state Initialising.
Device vif, 0 hotplugging has completed successfully, and is using handle 0.



Peter Bier

Re: can't start domU

[Hien Nguyen]

[-- Attachment #1.1: Type: text/plain, Size: 2201 bytes --]






The problem occurred when configured xend-config.sxp with route. See
detalies in the bug #392:
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=392

# tail -20 /var/log/xend.log
[2005-11-16 13:53:05 xend.XendDomainInfo] DEBUG (XendDomainInfo:574)
Storing domain details: {'console/ring-ref': '122397', 'console/port': '2',
'name': 'x335-hien1-vm4', 'console/limit': '1048576', 'vm':
'/vm/32184982-022a-fddd-cc97-6bf93f56afcf', 'domid': '1',
'cpu/0/availability': 'online', 'memory/target': '1048576',
'store/ring-ref': '122398', 'store/port': '1'}
[2005-11-16 13:53:05 xend] DEBUG (DevController:82) Waiting for devices
vif.
[2005-11-16 13:53:05 xend] DEBUG (DevController:88) Waiting for 0.
[2005-11-16 13:53:05 xend] DEBUG (DevController:309) hotplugStatus 0
[2005-11-16 13:53:10 xend] ERROR (SrvBase:87) Request wait_for_devices
failed.
Traceback (most recent call last):
  File "/usr/lib/python/xen/web/SrvBase.py", line 85, in perform
    return op_method(op, req)
  File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 68, in
op_wait_for_devices
    return self.dom.waitForDevices()
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 1200, in
waitForDevices    self.waitForDevices_(c)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 856, in
waitForDevices_    return
self.getDeviceController(deviceClass).waitForDevices()
  File "/usr/lib/python/xen/xend/server/DevController.py", line 84, in
waitForDevices
    return map(self.waitForDevice, self.deviceIDs())
  File "/usr/lib/python/xen/xend/server/DevController.py", line 93, in
waitForDevice
    raise VmError( ("Device %s (%s) could not be connected. "
VmError: Device 0 (vif) could not be connected. Hotplug scripts not working
[2005-11-16 13:53:10 xend] DEBUG (DevController:309) hotplugStatus 0


changeset:   7793:090e44133d40
tag:         tip
user:        kaf24@firebug.cl.cam.ac.uk
date:        Mon Nov 14 11:13:38 2005 +0100
summary:     Use make_lowmem_page_readonly/writable() in preference to the

Regards,

 Hien Nguyen
 Linux Technology Center             AUSTIN
 Phone: (512) 838-4140            Tie Line: 678-4140
 e-mail: hien1@us.ibm.com

[-- Attachment #1.2: Type: text/html, Size: 2497 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: can't start domU

[Steve Dobbelstein]

[-- Attachment #1: Type: text/plain, Size: 2917 bytes --]

Ewan Mellor <ewan@xensource.com> wrote on 11/16/2005 12:45:39 PM:

> On Wed, Nov 16, 2005 at 11:19:12AM -0600, Steve Dobbelstein wrote:
>
> > I'm not sure how to tell if /sbin/udevsend binary gets run.
>
> mv /sbin/udevsend{,.real}
> cat >/sbin/udevsend
> #!/bin/sh
> echo "udevsend running" >>/var/log/udev.log
> exec /sbin/udevsend.real
> ^D
>
> ought to do it.

Of course.  My duh.  I have been scratching my head over this and
scrambling it with python code that I forgot the obvious.

> I think that it's clear that the whole hotplug
> subsystem isn't
> running in your case, so I wouldn't worry about it.

/var/log/udesend.log says:
--------
Wed Nov 16 16:54:21 CST 2005 udevsend net
Wed Nov 16 16:57:22 CST 2005 udevsend xen-backend
Wed Nov 16 16:57:22 CST 2005 udevsend xen-backend
--------

Looks like hotplug is initiated, but perhaps not for the vif stuff.

> > [root@gir scripts]# PYTHONPATH=/usr/lib/python python
> > /usr/lib/python/xen/util/diagnose.py 4
> > Domain ID is 4.
> > Domain name is dom1.
> > Found 1 device classes in use.
> > Found 2 vbd devices.
> > Found device vbd, 2049.
> > Backend is in state Connected.
> > Frontend is in state Connected.
> > Device vbd, 2049 hotplugging has completed successfully, and is
connected
> > to physical device 0xfd01.
> > Found device vbd, 2050.
> > Backend is in state Connected.
> > Frontend is in state Connected.
> > Device vbd, 2050 hotplugging has completed successfully, and is
connected
> > to physical device 0xfd00.
>
> This suggests that the tools have not even got as far as trying to create
a
> vif for you.  Are you getting the "Device 0 (vif) could not be connected.
> Hotplug scripts not working." message, same as Charles Coffing?  That
really
> would be very curious.  In any case, your config file ought to be causing
Xend
> to at least try to create a vif.  I will need to see your
/var/log/xend.log to
> pin this down, and if you can get a xenstore trace, that will be useful
too.

I had seen the "Device 0 (vif) could not be connected. Hotplug scripts not
working." message on occasion in the past.  It's not doing it now.

Here are some helpful debug files.

(See attached file: xend.log)(See attached file: xend.trace.gz)(See
attached file: xend-debug.log)(See attached file: xenstored-trace.log.gz)

I'm afraid this is going to end up being some user error because something
got changed elsewhere on the system, but I can't figure out what.  The
machine used to not exhibit the problem with build 7713.  I upgraded the
machine to a later build (can't remember which offhand) and saw the
problem.  I then backed down to 7713 but still saw the problem.  It has
been doing it on every build since then.  I have tried cleaning everything
out (make uninstall, rm -rf /etc/xen, etc.) and rebuilding and
reinstalling.  I am beginning to suspect something outside of Xen is
causing it to trip up.

Thanks a lot for your help.

Steve D.

[-- Attachment #2: xend.log --]
[-- Type: application/octet-stream, Size: 7059 bytes --]

[2005-11-16 16:54:21 xend] INFO (SrvDaemon:268) Xend Daemon started
[2005-11-16 16:54:21 xend] INFO (SrvDaemon:272) Xend changeset: Wed Nov 16 04:29:52 2005 7826:9bf6f907b3ff0261902f06d261f76c1bd.
[2005-11-16 16:54:21 xend.XendDomainInfo] DEBUG (XendDomainInfo:172) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 20775127839L, 'ssidref': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'shutdown_reason': 0, 'dying': 0, 'dom': 0, 'mem_kb': 786432, 'maxmem_kb': -4, 'max_vcpu_id': 3, 'crashed': 0, 'running': 1, 'shutdown': 0, 'online_vcpus': 1, 'blocked': 0})
[2005-11-16 16:54:21 xend.XendDomainInfo] INFO (XendDomainInfo:184) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.
[2005-11-16 16:54:21 xend] DEBUG (XendDomain:139) number of vcpus to use is 0
[2005-11-16 16:54:21 xend] INFO (SrvServer:112) unix path=/var/lib/xend/xend-socket
[2005-11-16 16:57:22 xend.XendDomainInfo] DEBUG (XendDomainInfo:151) XendDomainInfo.create(['vm', ['name', 'dom1'], ['memory', '768'], ['on_poweroff', 'destroy'], ['on_reboot', 'destroy'], ['on_crash', 'destroy'], ['image', ['linux', ['kernel', '/boot/vmlinuz-xenU-smp'], ['root', '/dev/sda1 ro'], ['args', ' profile=1 GATEWAY=10.0.1.254 NETMASK=255.255.0.0 IPADDR=10.0.134.1 HOSTNAME=dom1'], ['vcpus', '4'], ['vcpus', '4'], ['boot', 'c']]], ['device', ['vbd', ['uname', 'phy:/dev/virt-blkdev-backend/dom1'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/virt-blkdev-backend/usr'], ['dev', 'sda2'], ['mode', 'r']]], ['device', ['vif']]])
[2005-11-16 16:57:22 xend.XendDomainInfo] DEBUG (XendDomainInfo:254) parseConfig: config is ['vm', ['name', 'dom1'], ['memory', '768'], ['on_poweroff', 'destroy'], ['on_reboot', 'destroy'], ['on_crash', 'destroy'], ['image', ['linux', ['kernel', '/boot/vmlinuz-xenU-smp'], ['root', '/dev/sda1 ro'], ['args', ' profile=1 GATEWAY=10.0.1.254 NETMASK=255.255.0.0 IPADDR=10.0.134.1 HOSTNAME=dom1'], ['vcpus', '4'], ['vcpus', '4'], ['boot', 'c']]], ['device', ['vbd', ['uname', 'phy:/dev/virt-blkdev-backend/dom1'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/virt-blkdev-backend/usr'], ['dev', 'sda2'], ['mode', 'r']]], ['device', ['vif']]]
[2005-11-16 16:57:22 xend.XendDomainInfo] DEBUG (XendDomainInfo:308) parseConfig: result is {'ssidref': None, 'name': 'dom1', 'bootloader': None, 'on_reboot': 'destroy', 'image': ['linux', ['kernel', '/boot/vmlinuz-xenU-smp'], ['root', '/dev/sda1 ro'], ['args', ' profile=1 GATEWAY=10.0.1.254 NETMASK=255.255.0.0 IPADDR=10.0.134.1 HOSTNAME=dom1'], ['vcpus', '4'], ['vcpus', '4'], ['boot', 'c']], 'on_poweroff': 'destroy', 'on_crash': 'destroy', 'backend': [], 'vcpus': 4, 'cpu_weight': None, 'vcpu_avail': None, 'memory': 768, 'device': [('vbd', ['vbd', ['uname', 'phy:/dev/virt-blkdev-backend/dom1'], ['dev', 'sda1'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/virt-blkdev-backend/usr'], ['dev', 'sda2'], ['mode', 'r']]), ('vif', ['vif'])], 'maxmem': None, 'cpu': None, 'uuid': None}
[2005-11-16 16:57:22 xend.XendDomainInfo] DEBUG (XendDomainInfo:1016) XendDomainInfo.construct: None 0
[2005-11-16 16:57:22 xend.XendDomainInfo] DEBUG (XendDomainInfo:1048) XendDomainInfo.initDomain: 1 1.0
[2005-11-16 16:57:22 xend] INFO (image:132) buildDomain os=linux dom=1 vcpus=4
[2005-11-16 16:57:22 xend] DEBUG (image:170) dom            = 1
[2005-11-16 16:57:22 xend] DEBUG (image:171) image          = /boot/vmlinuz-xenU-smp
[2005-11-16 16:57:22 xend] DEBUG (image:172) store_evtchn   = 1
[2005-11-16 16:57:22 xend] DEBUG (image:173) console_evtchn = 2
[2005-11-16 16:57:22 xend] DEBUG (image:174) cmdline        =  root=/dev/sda1 ro  profile=1 GATEWAY=10.0.1.254 NETMASK=255.255.0.0 IPADDR=10.0.134.1 HOSTNAME=dom1
[2005-11-16 16:57:22 xend] DEBUG (image:175) ramdisk        = 
[2005-11-16 16:57:22 xend] DEBUG (image:176) vcpus          = 4
[2005-11-16 16:57:22 xend] DEBUG (DevController:302) DevController: writing {'virtual-device': '2049', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/1/2049'} to /local/domain/1/device/vbd/2049.
[2005-11-16 16:57:22 xend] DEBUG (DevController:304) DevController: writing {'domain': 'dom1', 'params': '/dev/virt-blkdev-backend/dom1', 'frontend-id': '1', 'type': 'phy', 'state': '1', 'dev': 'sda1', 'frontend': '/local/domain/1/device/vbd/2049'} to /local/domain/0/backend/vbd/1/2049.
[2005-11-16 16:57:22 xend] DEBUG (DevController:302) DevController: writing {'virtual-device': '2050', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/1/2050'} to /local/domain/1/device/vbd/2050.
[2005-11-16 16:57:22 xend] DEBUG (DevController:304) DevController: writing {'domain': 'dom1', 'frontend': '/local/domain/1/device/vbd/2050', 'dev': 'sda2', 'state': '1', 'params': '/dev/virt-blkdev-backend/usr', 'frontend-id': '1', 'type': 'phy', 'read-only': ''} to /local/domain/0/backend/vbd/1/2050.
[2005-11-16 16:57:22 xend.XendDomainInfo] DEBUG (XendDomainInfo:557) Storing VM details: {'ssidref': '0', 'uuid': '146b10f7-55a0-0d10-6103-18017d1217e7', 'on_reboot': 'destroy', 'image': "(linux (kernel /boot/vmlinuz-xenU-smp) (root '/dev/sda1 ro') (args ' profile=1 GATEWAY=10.0.1.254 NETMASK=255.255.0.0 IPADDR=10.0.134.1 HOSTNAME=dom1') (vcpus 4) (vcpus 4) (boot c))", 'on_poweroff': 'destroy', 'name': 'dom1', 'vcpus': '4', 'vcpu_avail': '15', 'memory': '768', 'on_crash': 'destroy', 'maxmem': '768'}
[2005-11-16 16:57:22 xend.XendDomainInfo] DEBUG (XendDomainInfo:582) Storing domain details: {'console/ring-ref': '782827', 'console/port': '2', 'cpu/3/availability': 'online', 'name': 'dom1', 'console/limit': '1048576', 'cpu/2/availability': 'online', 'vm': '/vm/146b10f7-55a0-0d10-6103-18017d1217e7', 'domid': '1', 'cpu/0/availability': 'online', 'memory/target': '786432', 'store/ring-ref': '782828', 'cpu/1/availability': 'online', 'store/port': '1'}
[2005-11-16 16:57:22 xend] DEBUG (DevController:88) Waiting for devices vif.
[2005-11-16 16:57:22 xend] DEBUG (DevController:88) Waiting for devices usb.
[2005-11-16 16:57:22 xend] DEBUG (DevController:88) Waiting for devices vbd.
[2005-11-16 16:57:22 xend] DEBUG (DevController:94) Waiting for 2049.
[2005-11-16 16:57:22 xend] DEBUG (DevController:361) hotplugStatusCallback /local/domain/0/backend/vbd/1/2049/hotplug-status.
[2005-11-16 16:57:22 xend] DEBUG (DevController:361) hotplugStatusCallback /local/domain/0/backend/vbd/1/2049/hotplug-status.
[2005-11-16 16:57:22 xend] DEBUG (DevController:376) hotplugStatusCallback 1.
[2005-11-16 16:57:22 xend] DEBUG (DevController:94) Waiting for 2050.
[2005-11-16 16:57:22 xend] DEBUG (DevController:361) hotplugStatusCallback /local/domain/0/backend/vbd/1/2050/hotplug-status.
[2005-11-16 16:57:22 xend] DEBUG (DevController:376) hotplugStatusCallback 1.
[2005-11-16 16:57:22 xend] DEBUG (DevController:88) Waiting for devices pci.
[2005-11-16 16:57:22 xend] DEBUG (DevController:88) Waiting for devices ioports.
[2005-11-16 16:57:22 xend] DEBUG (DevController:88) Waiting for devices vtpm.
[2005-11-16 16:57:22 xend] INFO (XendDomain:348) Domain dom1 (1) unpaused.

[-- Attachment #3: xend.trace.gz --]
[-- Type: application/octet-stream, Size: 33268 bytes --]

[-- Attachment #4: xend-debug.log --]
[-- Type: application/octet-stream, Size: 18 bytes --]

Nothing to flush.

[-- Attachment #5: xenstored-trace.log.gz --]
[-- Type: application/octet-stream, Size: 3762 bytes --]

[-- Attachment #6: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: can't start domU

[Ewan Mellor]

On Wed, Nov 16, 2005 at 01:23:22PM -0700, Hien Nguyen wrote:

> The problem occurred when configured xend-config.sxp with route. See
> detalies in the bug #392:
> [1]http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=392
> 
> changeset: 7793:090e44133d40
> tag: tip
> user: kaf24@firebug.cl.cam.ac.uk
> date: Mon Nov 14 11:13:38 2005 +0100
> summary: Use make_lowmem_page_readonly/writable() in preference to the

This is fixed in changeset 7814:a064c5804eae.

Ewan.

Re: can't start domU

[Hien Nguyen]

It still happend with 7826:

changeset:   7826:9bf6f907b3ff
tag:         tip
user:        kaf24@firebug.cl.cam.ac.uk
date:        Wed Nov 16 04:29:52 2005 +0100
summary:     Trivial coding style fix.

[root@x335-hien1 xen]# xm create -c x335-hien1-vm4.cfg.switch
Using config file "x335-hien1-vm4.cfg.switch".
Error: Device 0 (vif) could not be connected. Hotplug scripts not working
[root@x335-hien1 xen]# tail -30 /var/log/xend.log
[2005-11-17 10:25:32 xend] DEBUG (image:174) cmdline        = 
root=/dev/sde1 ro
ro selinux=0 3
[2005-11-17 10:25:32 xend] DEBUG (image:175) ramdisk        =
[2005-11-17 10:25:32 xend] DEBUG (image:176) vcpus          = 1
[2005-11-17 10:25:32 xend] DEBUG (blkif:24) exception looking up device 
number for /dev/sde1: [Errno 2] No such file or directory: '/dev/sde1'
[2005-11-17 10:25:32 xend] DEBUG (DevController:302) DevController: 
writing {'virtual-device': '2113', 'backend-id': '0', 'state': '1', 
'backend': '/local/domain/0/backend/vbd/23/2113'} to 
/local/domain/23/device/vbd/2113.
[2005-11-17 10:25:32 xend] DEBUG (DevController:304) DevController: 
writing {'domain': 'x335-rhel4-vm3', 'params': 
'/FC4-FILEVBD4/FC4-32-filevbd4', 'frontend-id': '23', 'type': 'file', 
'state': '1', 'dev': '/dev/sde1', 'frontend': 
'/local/domain/23/device/vbd/2113'} to 
/local/domain/0/backend/vbd/23/2113.
[2005-11-17 10:25:32 xend] DEBUG (DevController:302) DevController: 
writing {'virtual-device': '2096', 'backend-id': '0', 'state': '1', 
'backend': '/local/domain/0/backend/vbd/23/2096'} to 
/local/domain/23/device/vbd/2096.
[2005-11-17 10:25:32 xend] DEBUG (DevController:304) DevController: 
writing {'domain': 'x335-rhel4-vm3', 'params': 'sdd', 'frontend-id': '23', 
'type': 'phy', 'state': '1', 'dev': 'sdd', 'frontend': 
'/local/domain/23/device/vbd/2096'} to 
/local/domain/0/backend/vbd/23/2096.
[2005-11-17 10:25:32 xend] DEBUG (DevController:302) DevController: 
writing {'backend-id': '0', 'mac': 'AA:00:00:17:12:6D', 'handle': '0', 
'state': '1', 'backend': '/local/domain/0/backend/vif/23/0'} to 
/local/domain/23/device/vif/0.
[2005-11-17 10:25:32 xend] DEBUG (DevController:304) DevController: 
writing {'domain': 'x335-rhel4-vm3', 'handle': '0', 'script': 
'/etc/xen/scripts/vif-route', 'ip': '9.3.189.251', 'state': '1', 
'frontend': '/local/domain/23/device/vif/0', 'mac': 'AA:00:00:17:12:6D', 
'frontend-id': '23'} to /local/domain/0/backend/vif/23/0.
[2005-11-17 10:25:32 xend.XendDomainInfo] DEBUG (XendDomainInfo:557) 
Storing VM details: {'ssidref': '0', 'uuid': 
'358df4d7-27f6-cc69-21e1-11a97b441f46', 'on_reboot': 'restart', 'image': 
"(linux (kernel /boot/vmlinuz-2.6-xenU) (root '/dev/sde1 ro') (args 'ro 
selinux=0 3') (vcpus 1) (vcpus 1) (boot c) (display localhost:10.0))", 
'on_poweroff': 'destroy', 'name': 'x335-rhel4-vm3', 'vcpus': '1', 
'vcpu_avail': '1', 'memory': '512', 'on_crash': 'restart', 'maxmem': 
'512'}
[2005-11-17 10:25:32 xend.XendDomainInfo] DEBUG (XendDomainInfo:582) 
Storing domain details: {'console/ring-ref': '122612', 'console/port': 
'2', 'name': 'x335-rhel4-vm3', 'console/limit': '1048576', 'vm': 
'/vm/358df4d7-27f6-cc69-21e1-11a97b441f46', 'domid': '23', 
'cpu/0/availability': 'online', 'memory/target': '524288', 
'store/ring-ref': '122613', 'store/port': '1'}
[2005-11-17 10:25:32 xend] DEBUG (DevController:88) Waiting for devices 
vif.
[2005-11-17 10:25:32 xend] DEBUG (DevController:94) Waiting for 0.
[2005-11-17 10:25:32 xend] DEBUG (DevController:361) hotplugStatusCallback 
/local/domain/0/backend/vif/23/0/hotplug-status.
[2005-11-17 10:25:37 xend] ERROR (SrvBase:87) Request wait_for_devices 
failed.
Traceback (most recent call last):
  File "/usr/lib/python/xen/web/SrvBase.py", line 85, in perform
    return op_method(op, req)
  File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 68, in 
op_wait_for_devices
    return self.dom.waitForDevices()
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 1205, in 
waitForDevices    self.waitForDevices_(c)
  File "/usr/lib/python/xen/xend/XendDomainInfo.py", line 864, in 
waitForDevices_    return 
self.getDeviceController(deviceClass).waitForDevices()
  File "/usr/lib/python/xen/xend/server/DevController.py", line 90, in 
waitForDevices
    return map(self.waitForDevice, self.deviceIDs())
  File "/usr/lib/python/xen/xend/server/DevController.py", line 100, in 
waitForDevice
    raise VmError("Device %s (%s) could not be connected. "
VmError: Device 0 (vif) could not be connected. Hotplug scripts not 
working



Regards,

 Hien Nguyen
 Linux Technology Center             AUSTIN
 Phone: (512) 838-4140            Tie Line: 678-4140
 e-mail: hien1@us.ibm.com




Ewan Mellor <ewan@xensource.com> 
11/16/2005 05:39 PM

To
Hien Nguyen/Austin/IBM@IBMUS
cc
xen-devel@lists.xensource.com
Subject
Re: [Xen-devel] can't start domU






On Wed, Nov 16, 2005 at 01:23:22PM -0700, Hien Nguyen wrote:

> The problem occurred when configured xend-config.sxp with route. See
> detalies in the bug #392:
> [1]http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=392
> 
> changeset: 7793:090e44133d40
> tag: tip
> user: kaf24@firebug.cl.cam.ac.uk
> date: Mon Nov 14 11:13:38 2005 +0100
> summary: Use make_lowmem_page_readonly/writable() in preference to the

This is fixed in changeset 7814:a064c5804eae.

Ewan.

Re: can't start domU

[Ewan Mellor]

On Thu, Nov 17, 2005 at 10:39:50AM -0600, Hien Nguyen wrote:

> It still happend with 7826:
> 
> changeset:   7826:9bf6f907b3ff
> tag:         tip
> user:        kaf24@firebug.cl.cam.ac.uk
> date:        Wed Nov 16 04:29:52 2005 +0100
> summary:     Trivial coding style fix.
> 
> [root@x335-hien1 xen]# xm create -c x335-hien1-vm4.cfg.switch
> Using config file "x335-hien1-vm4.cfg.switch".
> Error: Device 0 (vif) could not be connected. Hotplug scripts not working

And what do you get in /var/log/{debug,messages,syslog} now?

Ewan.

Re: can't start domU

[Hien Nguyen]

[-- Attachment #1.1.1: Type: text/plain, Size: 2338 bytes --]






There are no new messages in /var/log/xend-debug.log and /var/log/messages
after trying to create domU. Only xend.log recorded errors.

Regards,

 Hien Nguyen
 Linux Technology Center             AUSTIN
 Phone: (512) 838-4140            Tie Line: 678-4140
 e-mail: hien1@us.ibm.com



                                                                       
             Ewan Mellor                                               
             <ewan@xensource.c                                         
             om>                                                        To
                                                                       
             11/17/2005 10:55          Hien Nguyen/Austin/IBM@IBMUS    
             AM                                                         cc
                                       xen-devel@lists.xensource.com   
                                                                       
                                                                       
                                                                       
                                                                       
                                                                   Subject
                                       Re: [Xen-devel] can't start domU
                                                                       
                                                                       
                                                                       
                                                                       
                                                                       
                                                                       




On Thu, Nov 17, 2005 at 10:39:50AM -0600, Hien Nguyen wrote:

> It still happend with 7826:
>
> changeset:   7826:9bf6f907b3ff
> tag:         tip
> user:        kaf24@firebug.cl.cam.ac.uk
> date:        Wed Nov 16 04:29:52 2005 +0100
> summary:     Trivial coding style fix.
>
> [root@x335-hien1 xen]# xm create -c x335-hien1-vm4.cfg.switch
> Using config file "x335-hien1-vm4.cfg.switch".
> Error: Device 0 (vif) could not be connected. Hotplug scripts not working

And what do you get in /var/log/{debug,messages,syslog} now?

Ewan.


[-- Attachment #1.1.2: Type: text/html, Size: 3926 bytes --]

[-- Attachment #1.2: graycol.gif --]
[-- Type: image/gif, Size: 105 bytes --]

[-- Attachment #1.3: pic31819.gif --]
[-- Type: image/gif, Size: 1255 bytes --]

[-- Attachment #1.4: ecblank.gif --]
[-- Type: image/gif, Size: 45 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: can't start domU

[Ewan Mellor]

On Wed, Nov 16, 2005 at 05:21:31PM -0600, Steve Dobbelstein wrote:

> [Re failure to create vifs]
>
> I'm afraid this is going to end up being some user error because something
> got changed elsewhere on the system, but I can't figure out what.  The
> machine used to not exhibit the problem with build 7713.  I upgraded the
> machine to a later build (can't remember which offhand) and saw the
> problem.  I then backed down to 7713 but still saw the problem.  It has
> been doing it on every build since then.  I have tried cleaning everything
> out (make uninstall, rm -rf /etc/xen, etc.) and rebuilding and
> reinstalling.  I am beginning to suspect something outside of Xen is
> causing it to trip up.

I've found your problem, in your /var/log/xend.trace file:

> |    |    | < server/netif.py:getDeviceDetails
> |    |    |  79         if devid == 0:
> |    |    |  80             return devid

These two lines in DevController.py have never existed in the Xen repository.
This was a patch posted to the xen-devel mailing list, and it was never right
in the first place.

Ewan.

Re: can't start domU

[Steve Dobbelstein]

Ewan Mellor <ewan@xensource.com> wrote on 11/18/2005 11:32:48 AM:

> On Wed, Nov 16, 2005 at 05:21:31PM -0600, Steve Dobbelstein wrote:
> I've found your problem, in your /var/log/xend.trace file:
>
> > |    |    | < server/netif.py:getDeviceDetails
> > |    |    |  79         if devid == 0:
> > |    |    |  80             return devid
>
> These two lines in DevController.py have never existed in the Xen
repository.
> This was a patch posted to the xen-devel mailing list, and it was never
right
> in the first place.
>
> Ewan.

Ugh.  Too many patches going around that are supposed to *fix* problems.

Thank you very much for your time, Ewan.  Sorry to bother you with a user
error.

Steve D.