Xen error report (as requested)

Xen error report (as requested)

[Kyler Laird]

This is with changeset 8365.  I was checking "xm info" because I
couldn't start a domU.  ("Error: Error creating domain:
'free_memory'.)

--kyler

========================================================================
# xm info
Unexpected error: xen.xend.XendProtocol.XendError

Please report to xen-devel@lists.xensource.com
Traceback (most recent call last):
  File "/usr/sbin/xm", line 10, in ?
    main.main(sys.argv)
  File "/usr/lib/python/xen/xm/main.py", line 897, in main
    rc = cmd(args)
  File "/usr/lib/python/xen/xm/main.py", line 626, in xm_info
    info = server.xend_node()
  File "/usr/lib/python/xen/xend/XendClient.py", line 171, in xend_node
    return self.xendGet(self.nodeurl())
  File "/usr/lib/python/xen/xend/XendClient.py", line 150, in xendGet
    return self.client.xendGet(url, args)
  File "/usr/lib/python/xen/xend/XendProtocol.py", line 90, in xendGet
    return self.xendRequest(url, "GET", args)
  File "/usr/lib/python/xen/xend/XendProtocol.py", line 174, in
xendRequest
    val = self.handleStatus(resp.version, resp.status, resp.reason)
  File "/usr/lib/python/xen/xend/XendProtocol.py", line 110, in
handleStatus
    return self.handleException(XendError(message))
  File "/usr/lib/python/xen/xend/XendProtocol.py", line 136, in
handleException
    raise err
xen.xend.XendProtocol.XendError: Internal server error

Re: Xen error report (as requested)

[Ewan Mellor]

On Wed, Dec 14, 2005 at 01:25:00PM -0500, Kyler Laird wrote:

> This is with changeset 8365.  I was checking "xm info" because I
> couldn't start a domU.  ("Error: Error creating domain:
> 'free_memory'.)

At a guess, I'd say that you had updated your Python scripts, but not the
extensions (written in C, and installed in /usr/lib/python/xen/lowlevel).
Could you check that everything's been built and installed cleanly?

Otherwise, what platform are you on?  xm info might not work on IA64 or others
maybe, but obviously it works fine for us on normal 32 bit machines.

Ewan.

Re: Xen error report (as requested)

[Kyler Laird]

On Thu, Dec 15, 2005 at 12:05:33PM +0000, Ewan Mellor wrote:

> > This is with changeset 8365.  I was checking "xm info" because I
> > couldn't start a domU.  ("Error: Error creating domain:
> > 'free_memory'.)
> 
> At a guess, I'd say that you had updated your Python scripts, but not the
> extensions (written in C, and installed in /usr/lib/python/xen/lowlevel).
> Could you check that everything's been built and installed cleanly?

A mismatch was my first thought too.  I've gotten fairly good at
clearing out all of the old files though.  (I've been banging on this
for awhile.  strace and I are buddies.)

> Otherwise, what platform are you on?

This is a dual Xeon system running in 64-bit mode.

I finally discovered that this error only seems to happen with recent
unstable changesets.  I switched to testing (8256) and did not have this
problem.

Thank you.

--kyler

Re: Xen error report (as requested)

[Ewan Mellor]

On Thu, Dec 15, 2005 at 08:42:30AM -0500, Kyler Laird wrote:

> On Thu, Dec 15, 2005 at 12:05:33PM +0000, Ewan Mellor wrote:
> 
> > > This is with changeset 8365.  I was checking "xm info" because I
> > > couldn't start a domU.  ("Error: Error creating domain:
> > > 'free_memory'.)
> > 
> > At a guess, I'd say that you had updated your Python scripts, but not the
> > extensions (written in C, and installed in /usr/lib/python/xen/lowlevel).
> > Could you check that everything's been built and installed cleanly?
> 
> A mismatch was my first thought too.  I've gotten fairly good at
> clearing out all of the old files though.  (I've been banging on this
> for awhile.  strace and I are buddies.)
> 
> > Otherwise, what platform are you on?
> 
> This is a dual Xeon system running in 64-bit mode.
> 
> I finally discovered that this error only seems to happen with recent
> unstable changesets.  I switched to testing (8256) and did not have this
> problem.

Could you run xen-bugtool then, and please post your logs so I can have a look
around.  It might fail, since yours is quite a low level problem, in which
case please send /var/log/{xend,xend-debug}.log at the least, but hopefully it
will be able to run and create a tarball of stuff for you to send.

Thanks,

Ewan.

Re: Xen error report (as requested)

[Kyler Laird]

On Thu, Dec 15, 2005 at 01:52:41PM +0000, Ewan Mellor wrote:

> Could you run xen-bugtool then, and please post your logs so I can have a look
> around.  It might fail, since yours is quite a low level problem, in which
> case please send /var/log/{xend,xend-debug}.log at the least, but hopefully it
> will be able to run and create a tarball of stuff for you to send.

This was at a time when I was having trouble starting a domU.  That
turned out to be an issue with my AoE initialization.  (Not all
partitions are recognized at first.)  I just tried switching back to
changeset 8365 and was not able to duplicate the problem. 

I suspect that you already know that Xen breaks apart if one of the
domUs fails to start because of a bad root filesystem.  I'd chalk
this up to a result of my attempt to start a domU using a root
filesystem that wasn't available.

Thank you for your attention.  Sorry I wasn't able to give you something
more helpful.

--kyler

Re: Xen error report (as requested)

[Ewan Mellor]

On Thu, Dec 15, 2005 at 09:31:29AM -0500, Kyler Laird wrote:

> On Thu, Dec 15, 2005 at 01:52:41PM +0000, Ewan Mellor wrote:
> 
> > Could you run xen-bugtool then, and please post your logs so I can have a look
> > around.  It might fail, since yours is quite a low level problem, in which
> > case please send /var/log/{xend,xend-debug}.log at the least, but hopefully it
> > will be able to run and create a tarball of stuff for you to send.
> 
> This was at a time when I was having trouble starting a domU.  That
> turned out to be an issue with my AoE initialization.  (Not all
> partitions are recognized at first.)  I just tried switching back to
> changeset 8365 and was not able to duplicate the problem. 
> 
> I suspect that you already know that Xen breaks apart if one of the
> domUs fails to start because of a bad root filesystem.  I'd chalk
> this up to a result of my attempt to start a domU using a root
> filesystem that wasn't available.

No, it's not meant to "break apart"!  If the domU fails to start, that
shouldn't break the rest of the system.  What do you mean by that?

Ewan.

Re: Xen error report (as requested)

[Kyler Laird]

On Thu, Dec 15, 2005 at 02:40:12PM +0000, Ewan Mellor wrote:

> > I suspect that you already know that Xen breaks apart if one of the
> > domUs fails to start because of a bad root filesystem.  I'd chalk
> > this up to a result of my attempt to start a domU using a root
> > filesystem that wasn't available.
> 
> No, it's not meant to "break apart"!  If the domU fails to start, that
> shouldn't break the rest of the system.  What do you mean by that?

After trying to start a domU with an unavailable root filesystem, I get
this error on subsequent "xm create" attempts.
	xenconsole: Could not read tty from store: No such file or directory	
The only way I know to clear it is to reboot.

Note that this is a result of *my* mistake in specifying an AoE
partition that the kernel didn't yet recognize.  The same thing happens
if another invalid device (like "/dev/sdh") is specified as the root
device.

Thank you.

--kyler

=========================================================================
ucmengcluster0000:~# aoe-stat
    e0.0            eth1              up
ucmengcluster0000:~# parted -s /dev/etherd/e0.0 print
Disk geometry for /dev/etherd/e0.0: 0kB - 5201GB
Disk label type: gpt
Number  Start   End     Size    File system  Name                  Flags
1       17kB    2000MB  2000MB
46      2000MB  22GB    20GB    ext3         foo0000.sda
2       40GB    60GB    20GB    ext3         bar0000.sda
ucmengcluster0000:~# xm create -c test
Using config file "/etc/xen/test".
disk: ['phy:/dev/etherd/e0.0p2,/dev/sda,w']
Started domain foo0000
Linux version 2.6.12.6-xenU (root@ucmengcluster0000) (gcc version 4.0.3
20051201 (prerelease) (Debian 4.0.2-5)) #7 SMP Thu Dec 15 14:42:17 UTC
2005
kernel direct mapping tables upto 10800000 @ 444000-4ca000
Built 1 zonelists
Kernel command line:  root=/dev/sda1 rw 4
Initializing CPU#0
PID hash table entries: 2048 (order: 11, 65536 bytes)
Xen reported: 2793.180 MHz processor.
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Memory: 252672k/270336k available (1685k kernel code, 16932k reserved,
561k data, 136k init)
Mount-cache hash table entries: 256
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
Brought up 1 CPUs
NET: Registered protocol family 16
xen_mem: Initialising balloon driver.
Grant table initialized
IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
audit: initializing netlink socket (disabled)
audit(1134659388.685:0): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
Xen virtual console successfully installed as tty1
Event-channel device installed.
netfront: Initialising virtual ethernet driver.
md: md driver 0.90.1 MAX_MD_DEVS=256, MD_SB_DISKS=27
NET: Registered protocol family 2
IP: routing cache hash table of 1024 buckets, 16Kbytes
TCP established hash table entries: 16384 (order: 6, 262144 bytes)
TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
VFS: Cannot open root device "sda1" or unknown-block(0,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on
unknown-block(0,0)

Message from syslogd@localhost at Thu Dec 15 15:09:48 2005 ...
localhost kernel: Oops: 0000 [1]

Message from syslogd@localhost at Thu Dec 15 15:09:48 2005 ...
localhost kernel: CR2: 0000000000000002

[================ ctrl-] break out of console ===============]

ucmengcluster0000:~# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      251     1 r-----     6.6
foo0000                            1      256     1 r-----     6.2
ucmengcluster0000:~# xm destroy foo0000
ucmengcluster0000:~# parted -s /dev/etherd/e0.0 set 1 boot off
ucmengcluster0000:~# xm create -c test
Using config file "/etc/xen/test".
disk: ['phy:/dev/etherd/e0.0p2,/dev/sda,w']
Started domain foo0000
xenconsole: Could not read tty from store: No such file or directory
ucmengcluster0000:~# reboot -f -n

[================ wait for reboot, kick self... ===============]

ucmengcluster0000:~# parted -s /dev/etherd/e0.0 set 1 boot off
ucmengcluster0000:~# xm create -c test
Using config file "/etc/xen/test".
disk: ['phy:/dev/etherd/e0.0p2,/dev/sda,w']
Started domain foo0000
Linux version 2.6.12.6-xenU (root@ucmengcluster0000) (gcc version 4.0.3
20051201 (prerelease) (Debian 4.0.2-5)) #7 SMP Thu Dec 15 14:42:17 UTC
2005

[================ continues to boot normally ===============]



======================= console messages =========================
ip_tables: (C) 2000-2002 Netfilter core team
Unable to handle kernel NULL pointer dereference at 0000000000000002
RIP:
<ffffffff80177eb3>{blkdev_put+35}
PGD d445067 PUD d446067 PMD 0
Oops: 0000 [1]
CPU 0
Modules linked in: ipt_physdev iptable_filter ip_tables
Pid: 3, comm: events/0 Not tainted 2.6.12.6-xen0
RIP: e030:[<ffffffff80177eb3>] <ffffffff80177eb3>{blkdev_put+35}
RSP: e02b:ffff880000699e18  EFLAGS: 00010296
RAX: 0000000000000010 RBX: fffffffffffffffa RCX: ffffffff8010dba4
RDX: 0000000000000004 RSI: 0000000000000103 RDI: fffffffffffffffa
RBP: 0000000000000000 R08: 0000000000000010 R09: ffff880000a8c6e8
R10: 00002aaaaaac379b R11: 0000000000000202 R12: 0000000000000012
R13: ffff88000c850e48 R14: ffffffff802f2200 R15: ffff8800006d4dd0
FS:  00002aaaaaf09c80(0000) GS:ffffffff8060b380(0000)
knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
Process events/0 (pid: 3, threadinfo ffff880000698000, task
ffff8800006806d0)
Stack: ffff88000c850e78 0000000000000000 ffff8800006d4dc0
ffff88000c850e48
       ffffffff802f2200 ffffffff802f24c2 ffff88000c850e48
ffffffff802f2229
       ffff88000c850ed8 ffffffff8014126c
Call Trace:<ffffffff802f2200>{free_blkif+0}
<ffffffff802f24c2>{vbd_free+18}
       <ffffffff802f2229>{free_blkif+41}
<ffffffff8014126c>{worker_thread+492}
       <ffffffff8012d0b0>{default_wake_function+0}
<ffffffff80141080>{worker_thread+0}
       <ffffffff80141080>{worker_thread+0}
<ffffffff801455dd>{kthread+205}
       <ffffffff80112083>{child_rip+8} <ffffffff80145510>{kthread+0}
       <ffffffff8011207b>{child_rip+0}

Code: 48 8b 6f 08 4c 8b af 88 00 00 00 4c 89 e7 ff 4b 18 0f 88 43
RIP <ffffffff80177eb3>{blkdev_put+35} RSP <ffff880000699e18>
CR2: 0000000000000002