xen_blk timeout connecting to device

xen_blk timeout connecting to device

[Himanshu Raj]

domU won't start. Attached is the log. Worth noticing is the xen_blk timeout
connecting to the device.

-------------------------------------------
Using config file "xmexample1".
Started domain fc4
Linux version 2.6.12-xenU (rhim@ilab15) (gcc version 3.4.3 20050227 (Red Hat 3.4.3-22.1)) #1 SMP Tue Oct 4 19:47:30 EDT 2005
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 0000000004000000 (usable)
0MB HIGHMEM available.
64MB LOWMEM available.
IRQ lockup detection disabled
Built 1 zonelists
Kernel command line:  ip=:1.2.3.4::::eth0:dhcp root=/dev/sda1 ro selinux=0
Initializing CPU#0
PID hash table entries: 512 (order: 9, 8192 bytes)
Xen reported: 2783.912 MHz processor.
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
vmalloc area: c4800000-fb7fe000, maxmem 34000000
Memory: 61696k/65536k available (1801k kernel code, 3672k reserved, 490k data, 140k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Mount-cache hash table entries: 512
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... disabled
CPU0: Intel(R) Xeon(TM) CPU 2.80GHz stepping 07
Total of 1 processors activated (5557.45 BogoMIPS).
Brought up 1 CPUs
NET: Registered protocol family 16
xen_mem: Initialising balloon driver.
Grant table initialized
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Xen virtual console successfully installed as tty1
Event-channel device installed.
xen_blk: Initialising virtual block device driver
xen_blk: Timeout connecting to device!
xen_net: Initialising virtual ethernet driver.
NET: Registered protocol family 2
IP: routing cache hash table of 256 buckets, 4Kbytes
TCP established hash table entries: 4096 (order: 4, 65536 bytes)
TCP bind hash table entries: 4096 (order: 3, 49152 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
NET: Registered protocol family 1
NET: Registered protocol family 17
IP-Config: Incomplete network configuration information.
Root-NFS: No NFS server available, giving up.
VFS: Unable to mount root fs via NFS, trying floppy.
VFS: Cannot open root device "sda1" or unknown-block(2,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
 
-------------------------

-Himanshu

-- 
-------------------------------------------------------------------------
Himanshu Raj
PhD Student, GaTech (www.cc.gatech.edu/~rhim)
I prefer to receive attachments in an open, non-proprietary format.
-------------------------------------------------------------------------

Re: xen_blk timeout connecting to device

[Ewan Mellor]

On Wed, Oct 05, 2005 at 02:38:52AM -0400, Himanshu Raj wrote:

> domU won't start. Attached is the log. Worth noticing is the xen_blk timeout
> connecting to the device.

Device initialisation now requires hotplug / udev.  Please check that you have
hotplug and udev installed (apt-get install hotplug udev on Debian, YMMV).

Cheers,

Ewan.

Re: xen_blk timeout connecting to device

[Jacob Gorm Hansen]

On 10/5/05, Ewan Mellor <ewan@xensource.com> wrote:
> On Wed, Oct 05, 2005 at 02:38:52AM -0400, Himanshu Raj wrote:
>
> > domU won't start. Attached is the log. Worth noticing is the xen_blk timeout
> > connecting to the device.
>
> Device initialisation now requires hotplug / udev.  Please check that you have
> hotplug and udev installed (apt-get install hotplug udev on Debian, YMMV).

Is this in both dom0 and domU? And is it necessary when using physical
devices rather than file-backed ones?

Jacob

Re: xen_blk timeout connecting to device

[Chris Bainbridge]

On 05/10/05, Jacob Gorm Hansen <jacobg@diku.dk> wrote:
> On 10/5/05, Ewan Mellor <ewan@xensource.com> wrote:
> > On Wed, Oct 05, 2005 at 02:38:52AM -0400, Himanshu Raj wrote:
> >
> > > domU won't start. Attached is the log. Worth noticing is the xen_blk timeout
> > > connecting to the device.
> >
> > Device initialisation now requires hotplug / udev.  Please check that you have
> > hotplug and udev installed (apt-get install hotplug udev on Debian, YMMV).
>
> Is this in both dom0 and domU? And is it necessary when using physical
> devices rather than file-backed ones?

dom0. Yes.

Re: xen_blk timeout connecting to device

[Ewan Mellor]

On Wed, Oct 05, 2005 at 11:34:28AM +0200, Jacob Gorm Hansen wrote:

> On 10/5/05, Ewan Mellor <ewan@xensource.com> wrote:
> > On Wed, Oct 05, 2005 at 02:38:52AM -0400, Himanshu Raj wrote:
> >
> > > domU won't start. Attached is the log. Worth noticing is the xen_blk timeout
> > > connecting to the device.
> >
> > Device initialisation now requires hotplug / udev.  Please check that you have
> > hotplug and udev installed (apt-get install hotplug udev on Debian, YMMV).
> 
> Is this in both dom0 and domU? And is it necessary when using physical
> devices rather than file-backed ones?

We use them in dom0.  However, I believe you are trying to bring up devices
without using Xend, so how you achieve that is up to you.  I don't believe
that hotplug is _required_ for bare Xen.

Ewan.

Re: xen_blk timeout connecting to device

[Himanshu Raj]

I am having problem with domU, and yes, I have both hotplug and udev installed.

-Himanshu
On Wed, Oct 05, 2005 at 10:57:44AM +0100, Ewan Mellor wrote:
> On Wed, Oct 05, 2005 at 11:34:28AM +0200, Jacob Gorm Hansen wrote:
> 
> > On 10/5/05, Ewan Mellor <ewan@xensource.com> wrote:
> > > On Wed, Oct 05, 2005 at 02:38:52AM -0400, Himanshu Raj wrote:
> > >
> > > > domU won't start. Attached is the log. Worth noticing is the xen_blk timeout
> > > > connecting to the device.
> > >
> > > Device initialisation now requires hotplug / udev.  Please check that you have
> > > hotplug and udev installed (apt-get install hotplug udev on Debian, YMMV).
> > 
> > Is this in both dom0 and domU? And is it necessary when using physical
> > devices rather than file-backed ones?
> 
> We use them in dom0.  However, I believe you are trying to bring up devices
> without using Xend, so how you achieve that is up to you.  I don't believe
> that hotplug is _required_ for bare Xen.
> 
> Ewan.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
-------------------------------------------------------------------------
Himanshu Raj
PhD Student, GaTech (www.cc.gatech.edu/~rhim)
I prefer to receive attachments in an open, non-proprietary format.
-------------------------------------------------------------------------

Re: xen_blk timeout connecting to device

[Ewan Mellor]

On Wed, Oct 05, 2005 at 09:24:04AM -0400, Himanshu Raj wrote:

> I am having problem with domU, and yes, I have both hotplug and udev
> installed.

The problem may manifest itself in domU, but the timeout is a failure to get a
response from dom0, and it's there that the hotplug tools are needed.
Assuming you've got them installed in dom0, then you'll have to give us some
information about the distro that you are using, and maybe poke around the
scripts in there to see if /etc/hotplug/xen-backend.agent is being run, and if
so what parameters it is receiving, that kind of thing.

It gets distro-specific at this point, and we've had a lot of trouble with
this recently, because the hotplug architecture is changing beneath our feet.

Ewan.

Re: xen_blk timeout connecting to device

[Ryan Grimm]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1319 bytes --]

Hi,

I am having similar problems starting a domU with domO running ubuntu
5.04.  Attached are the xm console log and message log after turning on
debug output in the default.hotplug script.  The changset is
7224:f98c8238e909.

In the console log, vbd and vif are reporting failure with error -22.

Ryan

sorry for repeat send ... malconfigured email client

On Wed, 5 Oct 2005, Ewan Mellor wrote:

> On Wed, Oct 05, 2005 at 09:24:04AM -0400, Himanshu Raj wrote:
>
>> I am having problem with domU, and yes, I have both hotplug and udev
>> installed.
>
> The problem may manifest itself in domU, but the timeout is a failure to get a
> response from dom0, and it's there that the hotplug tools are needed.
> Assuming you've got them installed in dom0, then you'll have to give us some
> information about the distro that you are using, and maybe poke around the
> scripts in there to see if /etc/hotplug/xen-backend.agent is being run, and if
> so what parameters it is receiving, that kind of thing.
>
> It gets distro-specific at this point, and we've had a lot of trouble with
> this recently, because the hotplug architecture is changing beneath our feet.
>
> Ewan.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

[-- Attachment #2: xm console log --]
[-- Type: TEXT/PLAIN, Size: 2533 bytes --]

Linux version 2.6.12-xenU (root@thinkvision-rpg) (gcc version 3.3.5 (Debian 1:3.3.5-8ubuntu2)) #2 SMP Wed Oct 5 17:31:02 CDT 2005
kernel direct mapping tables upto 4000000 @ 3f0000-412000
Built 1 zonelists
Kernel command line:  root=/dev/sda3 ro 4
Initializing CPU#0
PID hash table entries: 512 (order: 9, 16384 bytes)
Xen reported: 3600.122 MHz processor.
Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
Memory: 60836k/65536k available (1719k kernel code, 4540k reserved, 609k data, 136k init)
Mount-cache hash table entries: 256
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 3
Brought up 1 CPUs
WARNING: 1 siblings found for CPU0, should be 2
NET: Registered protocol family 16
xen_mem: Initialising balloon driver.
Grant table initialized
Number of CPUs sharing cache didn't match any known set of CPUs
Number of CPUs sharing cache didn't match any known set of CPUs
IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
audit: initializing netlink socket (disabled)
audit(1128616122.012:0): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
Xen virtual console successfully installed as tty1
Event-channel device installed.
xen_blk: Initialising virtual block device driver
vbd: probe of vbd-2051 failed with error -22
xen_blk: Timeout connecting to device!
xen_net: Initialising virtual ethernet driver.
vif: probe of vif-1 failed with error -22
xen_net: Timeout connecting to device!
md: md driver 0.90.1 MAX_MD_DEVS=256, MD_SB_DISKS=27
NET: Registered protocol family 2
IP: routing cache hash table of 256 buckets, 4Kbytes
TCP established hash table entries: 4096 (order: 4, 65536 bytes)
TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
VFS: Cannot open root device "sda3" or unknown-block(0,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

[-- Attachment #3: /var/log/messages --]
[-- Type: TEXT/PLAIN, Size: 2296 bytes --]

Oct  6 11:31:04 localhost default.hotplug[7403]: arguments (net) env (SUBSYSTEM=net OLDPWD=/ DEVPATH=/class/net/vif6.1 PATH=/bin:/sbin:/usr/sbin:/usr/bin ACTION=add PWD=/etc/hotplug UDEV_LOG=1 MANAGED_EVENT=1 SHLVL=1 HOME=/ INTERFACE=vif6.1 DEBUG=yes SEQNUM=988 _=/usr/bin/env)
Oct  6 11:31:04 localhost default.hotplug[7403]: invoke /etc/hotplug/net.agent ()
Oct  6 11:31:04 localhost net.agent[7403]: invoke ifrename for vif6.1
Oct  6 11:31:04 localhost net.ifup[7425]: Invoking ifup vif6.1=hotplug
Oct  6 11:31:04 localhost default.hotplug[7421]: arguments (xen-backend) env (SUBSYSTEM=xen-backend OLDPWD=/ XENBUS_TYPE=vbd DEVPATH=/devices/xen-backend/vbd-6-2051 PATH=/bin:/sbin:/usr/sbin:/usr/bin ACTION=add PWD=/etc/hotplug UDEV_LOG=1 XENBUS_PATH=backend/vbd/6/2051 MANAGED_EVENT=1 SHLVL=1 HOME=/ DEBUG=yes PHYSDEVBUS=xen-backend SEQNUM=986 _=/usr/bin/env)
Oct  6 11:31:04 localhost default.hotplug[7421]: invoke /etc/hotplug/xen-backend.agent ()
Oct  6 11:31:04 localhost default.hotplug[7428]: arguments (xen-backend) env (SUBSYSTEM=xen-backend OLDPWD=/ XENBUS_TYPE=vif DEVPATH=/devices/xen-backend/vif-6-1 PATH=/bin:/sbin:/usr/sbin:/usr/bin ACTION=add PWD=/etc/hotplug UDEV_LOG=1 XENBUS_PATH=backend/vif/6/1 MANAGED_EVENT=1 SHLVL=1 HOME=/ DEBUG=yes PHYSDEVBUS=xen-backend SEQNUM=987 _=/usr/bin/env)
Oct  6 11:31:04 localhost default.hotplug[7428]: invoke /etc/hotplug/xen-backend.agent ()
Oct  6 11:31:04 localhost default.hotplug[7452]: arguments (xen-backend) env (SUBSYSTEM=xen-backend OLDPWD=/ XENBUS_TYPE=vif DEVPATH=/devices/xen-backend/vif-6-1 script=/etc/xen/scripts/vif-bridge vif=vif6.1 PATH=/bin:/sbin:/usr/sbin:/usr/bin ACTION=online PWD=/etc/hotplug UDEV_LOG=1 mac=aa:00:00:44:01:b8 domain=ttylinux bridge=xen-br0 XENBUS_PATH=backend/vif/6/1 MANAGED_EVENT=1 SHLVL=1 HOME=/ PHYSDEVDRIVER=vif DEBUG=yes PHYSDEVBUS=xen-backend SEQNUM=989 _=/usr/bin/env)
Oct  6 11:31:04 localhost default.hotplug[7452]: invoke /etc/hotplug/xen-backend.agent ()
Oct  6 11:31:04 localhost kernel: device vif6.1 entered promiscuous mode
Oct  6 11:31:04 localhost kernel: xen-br0: port 7(vif6.1) entering learning state
Oct  6 11:31:04 localhost kernel: xen-br0: topology change detected, propagating
Oct  6 11:31:04 localhost kernel: xen-br0: port 7(vif6.1) entering forwarding state

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: xen_blk timeout connecting to device

[Ewan Mellor]

On Thu, Oct 06, 2005 at 12:15:28PM -0500, Ryan Grimm wrote:

> Hi,
> 
> I am having similar problems starting a domU with domO running ubuntu
> 5.04.  Attached are the xm console log and message log after turning on
> debug output in the default.hotplug script.  The changset is
> 7224:f98c8238e909.
> 
> In the console log, vbd and vif are reporting failure with error -22.

There was a fix for an error -22 failure today: changeset 7228.  Further
changes have been made in this area today too -- changesets 7236/7, so you're
probably better off with those.

Ewan.

Re: xen_blk timeout connecting to device

[Jeremy Katz]

On Wed, 2005-10-05 at 02:38 -0400, Himanshu Raj wrote:
> domU won't start. Attached is the log. Worth noticing is the xen_blk timeout
> connecting to the device.

What is dom0 running?  Anything running udev 059 or later requires a
different setup than the hotplug scripts that are currently present in
the tree.

Also, you should make sure that you're really getting the current
scripts and that you didn't have previous ones present (or, use a
changeset from today which has my patch to always install the current
scripts).

Jeremy

Re: xen_blk timeout connecting to device

[Himanshu Raj]

Dom0 is running RHEL4.

Apparently the problem is somewhere in NFS. As such with dom0, the nfs doesn't 
seem to be working properly (frequent disconnections to the server, long delays
re-establishing). xend took a good 5-6 mins to start, but when it did, I could
boot domU just fine.

I pulled the freshest updates. Your tools script must have fixed something, since
it is no more crying about permissions and such (I have a pseudo root access, so
trying to run stuff from my user home directory wasn't working).

-Himanshu
On Wed, Oct 05, 2005 at 11:07:58AM -0400, Jeremy Katz wrote:
> On Wed, 2005-10-05 at 02:38 -0400, Himanshu Raj wrote:
> > domU won't start. Attached is the log. Worth noticing is the xen_blk timeout
> > connecting to the device.
> 
> What is dom0 running?  Anything running udev 059 or later requires a
> different setup than the hotplug scripts that are currently present in
> the tree.
> 
> Also, you should make sure that you're really getting the current
> scripts and that you didn't have previous ones present (or, use a
> changeset from today which has my patch to always install the current
> scripts).
> 
> Jeremy

-- 
-------------------------------------------------------------------------
Himanshu Raj
PhD Student, GaTech (www.cc.gatech.edu/~rhim)
I prefer to receive attachments in an open, non-proprietary format.
-------------------------------------------------------------------------

Re: xen_blk timeout connecting to device

[Chris Bainbridge]

On 05/10/05, Himanshu Raj <rhim@cc.gatech.edu> wrote:
> Dom0 is running RHEL4.
>
> Apparently the problem is somewhere in NFS. As such with dom0, the nfs doesn't
> seem to be working properly (frequent disconnections to the server, long delays
> re-establishing). xend took a good 5-6 mins to start, but when it did, I could
> boot domU just fine.

I have noticed problems serving nfs from dom0 to the domUs. dom0 nfs
seems to hang, and I end up rebooting everything to get it working
again.

Re: xen_blk timeout connecting to device

[Tim Durack]

[-- Attachment #1.1: Type: text/plain, Size: 77 bytes --]

I have hotplug and udev installed, but am seeing the same problem.

Tim:>

[-- Attachment #1.2: Type: text/html, Size: 92 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: xen_blk timeout connecting to device

[Chris Bainbridge]

On 05/10/05, Tim Durack <tdurack@gmail.com> wrote:
> I have hotplug and udev installed, but am seeing the same problem.

"cat /proc/sys/kernel/hotplug". /sbin/hotplug seems to work for
everyone (?). /sbin/udevsend needs udev rules. The backward compatible
udev rules should work, but don't.

Some notes on debugging this stuff, in case anyone else has problems

in /etc/udev/udev.conf add:

udev_log="info"

In /etc/hotplug.d/default/default.hotplug:

DEBUG=yes export DEBUG

In /etc/hotplug/xen-backend.agent:

add "set -x" as 2nd line.

You should now get debug info in either /var/log/messages or
/var/log/udev (depends on syslog setup).

I gave up with the hotplug stuff, and used the udev rules I posted a
few days ago. If you go that route comment out this line from
/etc/udev/rules.d/50-udev.rules:

ENV{UDEVD_EVENT}=="1", RUN+="/sbin/udev_run_hotplugd"

That will disable the backward compatible hotplug stuff which seems to
be broken with respect to xen. You will now be able to see the udev
events and rules in your log file, and exactly what is going on in the
scripts you execute (echo to stdout will be syslogged by udev). After
that it's just a matter of tying together the unhandled events with
actions.

xen_blk: Timeout connecting to device

[Bonnell, David]

[-- Attachment #1.1: Type: text/plain, Size: 525 bytes --]

After upgrading to a recent snapshot (6831) I am unable to boot a guest
- I get the following messages when booting the guest domain:
 
Xen_blk: Initialising virtual block device driver
Xen_blk: Timeout connecting to device!
...
VFS: Cannot open root device "sda1" or unknown-block(0,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on
known-block(0,0)
 
The root fs of my guest domains is an LVM volume, as is the root of
dom0.
 
Any ideas?
 
 
-Dave

[-- Attachment #1.2: Type: text/html, Size: 8978 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel