Unable to create guest domain AND mm.c/grant_table.c errors

Unable to create guest domain AND mm.c/grant_table.c errors

[Bonnell, David]

[-- Attachment #1.1: Type: text/plain, Size: 1614 bytes --]

Since a few changesets back (currently using 6963) I am unable to create
a guest domain.  It appears the front-end VBD cannot connect to the
backend and as such the guest cannot mount its root filesystem and
panics.
 
The xen console log and xend logs are attached.  Below is the bottom
part of the guest's console output
 
...
Event-channel device ubstakked,
xen_blk: Initialising virtual block device driver
*** A FEW SECONDS DELAY ***
xen_blk: Timeout connecting to device!
xen_net: Initialising virtual Ethernet driver.
NET: Registered protocol family 2
...
Root-NFS: No NFS server available, giving up.
VFS: Unable to mount root fs via NFS, trying floppy.
VFS: Cannot open root device "sda1" or unknown-block(2,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on
unknown-block(2,0)
 
The "Unable to read memory/target" console message at the end of the Xen
console log is new for this changeset.  Also, every now and then a
failed attempt to create a guest domain ends with the following sequence
of errors on the Xen console and after such a failure the guest domain
is not automatically destroyed as it is normally and needs a manual "xm
destroy ???" command to remove it.
 
(XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address
c8cec000
(XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address
c8ced000
(XEN) (file=grant_table.c, line=1054) Grant unref rd(7) ld(0) frm(1ff51)
flgs(0).
(XEN) (file=grant_table.c, line=1054) Grant unref rd(7) ld(0) frm(1ff50)
flgs(0).
 
 
-Dave

[-- Attachment #1.2: Type: text/html, Size: 13242 bytes --]

[-- Attachment #2: xend.log --]
[-- Type: application/octet-stream, Size: 8360 bytes --]

[2005-09-21 13:55:40 xend] INFO (SrvDaemon:301) Xend Daemon started
[2005-09-21 13:55:42 xend] INFO (XendDomain:355) Creating entry for unknown xend domain: id=0 uuid=18ebf8f4-affb-4928-842d-d6b0b54063d9
[2005-09-21 13:55:42 xend] DEBUG (XendDomainInfo:172) info={'vcpu_to_cpu': [0, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], 'cpu_time': 25912083259L, 'ssidref': 0, 'shutdown_reason': 0, 'dying': 0, 'dom': 0, 'running': 1, 'maxmem_kb': -4, 'paused': 0, 'crashed': 0, 'vcpus': 1, 'shutdown': 0, 'mem_kb': 125952, 'cpumap': [-1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'blocked': 0}
[2005-09-21 13:55:42 xend] DEBUG (XendDomainInfo:173) config='None'

[2005-09-21 13:55:42 xend] INFO (XendRoot:141) EVENT> xend.domain.create ['Domain-0', 0]
[2005-09-21 13:55:42 xend] DEBUG (XendDomainInfo:962) number of vcpus to use is 0
[2005-09-21 13:55:42 xend] INFO (SrvServer:107) unix path=/var/lib/xend/xend-socket
[2005-09-21 13:55:42 xend] INFO (XendRoot:141) EVENT> xend.start 0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + set -e
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + OP=start
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + shift
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + for arg in '"$@"'
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + export bridge=xen-br0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + bridge=xen-br0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + for arg in '"$@"'
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + export antispoof=no
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + antispoof=no
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + bridge=xen-br0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + netdev=eth0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + antispoof=no
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + echo '*network start bridge=xen-br0 netdev=eth0 antispoof=no'
[2005-09-21 13:55:42 xend] INFO (process:35) [network-bridge] network start bridge=xen-br0 netdev=eth0 antispoof=no
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + case ${OP} in
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + op_start
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + '[' xen-br0 == null ']'
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + create_bridge xen-br0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + local bridge=xen-br0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + brctl show
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + grep -q xen-br0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + brctl addbr xen-br0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + brctl stp xen-br0 off
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + brctl setfd xen-br0 0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + ifconfig xen-br0 up
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + ifconfig
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + grep -q veth0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + ifconfig veth0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + grep -q veth0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] ++ ifconfig eth0
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] ++ grep HWadd
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] ++ sed -e 's/.*\(..:..:..:..:..:..\).*/\1/'
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + mac=00:E0:18:37:EF:B9
[2005-09-21 13:55:42 xend] ERROR (process:37) [network-bridge] + ifdown eth0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + ip link set eth0 name peth0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + ip link set veth0 name eth0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + ifconfig peth0 0.0.0.0 -arp down
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + ifconfig peth0 hw ether fe:ff:ff:ff:ff:ff
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + ifconfig eth0 hw ether 00:E0:18:37:EF:B9
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + add_to_bridge xen-br0 vif0.0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + local bridge=xen-br0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + local dev=vif0.0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + brctl show
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + grep -q vif0.0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + brctl addif xen-br0 vif0.0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + add_to_bridge xen-br0 peth0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + local bridge=xen-br0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + local dev=peth0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + brctl show
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + grep -q peth0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + brctl addif xen-br0 peth0
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + ip link set xen-br0 up
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + ip link set vif0.0 up
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + ip link set peth0 up
[2005-09-21 13:55:43 xend] ERROR (process:37) [network-bridge] + ifup eth0
[2005-09-21 13:56:00 xend] DEBUG (image:164) initDomain: cpu=-1 mem_kb=32768 ssidref=0 dom=1
[2005-09-21 13:56:00 xend] DEBUG (XendDomainInfo:702) init_domain> Created domain=1 name=vm1 memory=32
[2005-09-21 13:56:01 xend] INFO (image:202) buildDomain os=linux dom=1 vcpus=1
[2005-09-21 13:56:01 xend] DEBUG (image:246) dom            = 1
[2005-09-21 13:56:01 xend] DEBUG (image:247) image          = /boot/vmlinuz-2.6-xenU
[2005-09-21 13:56:01 xend] DEBUG (image:248) store_evtchn   = 1
[2005-09-21 13:56:01 xend] DEBUG (image:249) console_evtchn = 2
[2005-09-21 13:56:01 xend] DEBUG (image:250) cmdline        =  root=/dev/sda1 ro
[2005-09-21 13:56:01 xend] DEBUG (image:251) ramdisk        = 
[2005-09-21 13:56:01 xend] DEBUG (image:252) flags          = 0
[2005-09-21 13:56:01 xend] DEBUG (image:253) vcpus          = 1
[2005-09-21 13:56:01 xend] DEBUG (blkif:24) exception looking up device number for sda1: [Errno 2] No such file or directory: '/dev/sda1'
[2005-09-21 13:56:01 xend] DEBUG (DevController:181) DevController: writing {'virtual-device': '2049', 'backend-id': '0', 'backend': '/domain/18ebf8f4-affb-4928-842d-d6b0b54063d9/backend/vbd/0ac761ab-a54e-4e80-9705-5ca539720217/2049'} to /domain/0ac761ab-a54e-4e80-9705-5ca539720217/device/vbd/2049.
[2005-09-21 13:56:01 xend] DEBUG (DevController:183) DevController: writing {'params': 'VolGroup00/vm1disk1', 'domain': 'vm1', 'type': 'phy', 'frontend': '/domain/0ac761ab-a54e-4e80-9705-5ca539720217/device/vbd/2049', 'frontend-id': '1'} to /domain/18ebf8f4-affb-4928-842d-d6b0b54063d9/backend/vbd/0ac761ab-a54e-4e80-9705-5ca539720217/2049.
[2005-09-21 13:56:01 xend] DEBUG (DevController:181) DevController: writing {'backend-id': '0', 'mac': 'aa:00:00:03:8c:3a', 'handle': '1', 'backend': '/domain/18ebf8f4-affb-4928-842d-d6b0b54063d9/backend/vif/0ac761ab-a54e-4e80-9705-5ca539720217/1'} to /domain/0ac761ab-a54e-4e80-9705-5ca539720217/device/vif/1.
[2005-09-21 13:56:01 xend] DEBUG (DevController:183) DevController: writing {'bridge': 'xen-br0', 'mac': 'aa:00:00:03:8c:3a', 'handle': '1', 'script': '/etc/xen/scripts/vif-bridge', 'frontend-id': '1', 'domain': 'vm1', 'frontend': '/domain/0ac761ab-a54e-4e80-9705-5ca539720217/device/vif/1'} to /domain/18ebf8f4-affb-4928-842d-d6b0b54063d9/backend/vif/0ac761ab-a54e-4e80-9705-5ca539720217/1.
[2005-09-21 13:56:01 xend] INFO (XendRoot:141) EVENT> xend.domain.create ['vm1', 1]
[2005-09-21 13:56:01 xend] INFO (XendRoot:141) EVENT> xend.domain.unpause ['vm1', 1]
[2005-09-21 13:56:12 xend] DEBUG (XendDomain:232) domain died name=vm1 domid=1


[-- Attachment #3: xen.log --]
[-- Type: application/octet-stream, Size: 11939 bytes --]

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Chris Bainbridge]

[-- Attachment #1: Type: text/plain, Size: 7875 bytes --]

Hi,

I'm been struggling with this for a couple of days.. I'd assumed that
there was just something wrong with my setup rather than a bug. I've
seen the same PTE error. In trying to eliminate possible problems I've
started using the nightly binary xen builds. So I'm using the binary
precompiled hypervisor, dom0 and domU kernels.

The really weird thing is that I have another machine with the same
Xen build, same xen and kernel image binaries, and it works. I pulled
the hard disk and put it in the bad machine, and it worked. So it
appears there's something wrong with that particular domain 0 root fs
that is causing it to fail, rather than anything wrong with the Xen
binaries or hardware.

I've tried rm -rf /var/lib/xenstored but no luck - does Xen store
state anywhere else? Is there anything else in the domain 0 root fs
that could be causing this error? In fact I did find something, that
if we try to create a device (eg. /dev/hda1) in domU then it has to
exist in dom0, otherwise xend can't find a major/minor number. That
can be tricky with udev if you don't have a tarball of standard
devices, maybe it can be fixed?

The only other thing I could think of was that I built this root fs
with '-mno-tls-direct-seg-refs' in CFLAGS, and glibc with nptlonly (it
has no linuxthreads). AFAIK, this set up should work fine with Xen.
Can anyone confirm that?

domU:

Event-channel device installed.
xen_blk: Initialising virtual block device driver
xenbus: failed to write error node for device/vbd/2049 (22 writing ring-ref)
vbd: probe of vbd-2049 failed with error -22
xen_blk: Timeout connecting to device!
xen_net: Initialising virtual ethernet driver.
xenbus: failed to write error node for device/vif/1 (22 writing tx ring-ref)
vif: probe of vif-1 failed with error -22
xen_net: Timeout connecting to device!

xen console:
smp_send_stop disable_local_APIC
<Sometimes the PTE error appears after this smp msg>
(XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address d0d4c000
(XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address d0d4d000
(XEN) (file=grant_table.c, line=1054) Grant unref rd(1) ld(0) frm(eef9) flgs(0).
(XEN) (file=grant_table.c, line=1054) Grant unref rd(1) ld(0) frm(eef8) flgs(0).

xend-debug.log:
ERROR: do_evtchn_op: HYPERVISOR_event_channel_op failed: -1
ERROR: do_evtchn_op: HYPERVISOR_event_channel_op failed: -1

# xend.log
2005-09-21 11:36:44 xend] DEBUG (image:164) initDomain: cpu=-1
mem_kb=98304 ssidref=0 dom=1
[2005-09-21 11:36:44 xend] DEBUG (XendDomainInfo:702) init_domain>
Created domain=1 name=ubuntu memory=96
[2005-09-21 11:36:44 xend] INFO (image:202) buildDomain os=linux dom=1 vcpus=1
[2005-09-21 11:36:44 xend] DEBUG (image:246) dom            = 1
[2005-09-21 11:36:44 xend] DEBUG (image:247) image          =
/boot/vmlinuz-2.6-xenU
[2005-09-21 11:36:44 xend] DEBUG (image:248) store_evtchn   = 1
[2005-09-21 11:36:44 xend] DEBUG (image:249) console_evtchn = 2
[2005-09-21 11:36:44 xend] DEBUG (image:250) cmdline        =  root=/dev/sda1
[2005-09-21 11:36:44 xend] DEBUG (image:251) ramdisk        =
[2005-09-21 11:36:44 xend] DEBUG (image:252) flags          = 0
[2005-09-21 11:36:44 xend] DEBUG (image:253) vcpus          = 1
[2005-09-21 11:36:44 xend] DEBUG (DevController:181) DevController:
writing {'virtual-device': '2049', 'backend-id': '0', 'backend':
'/domain/0e9d1b5f-ae27-40c0-8f78-b1c1039815dc/backend/vbd/64dd317d-dbcf-4e\
9e-be5a-b392e83da1fa/2049'} to
/domain/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/device/vbd/2049.
[2005-09-21 11:36:44 xend] DEBUG (DevController:183) DevController:
writing {'params': '/var/tmp/ubuntu.img', 'domain': 'ubuntu', 'type':
'file', 'frontend': '/domain/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/dev\
ice/vbd/2049', 'frontend-id': '1'} to
/domain/0e9d1b5f-ae27-40c0-8f78-b1c1039815dc/backend/vbd/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/2049.
[2005-09-21 11:36:44 xend] DEBUG (DevController:181) DevController:
writing {'backend-id': '0', 'mac': 'aa:00:00:00:00:11', 'handle': '1',
'backend': '/domain/0e9d1b5f-ae27-40c0-8f78-b1c1039815dc/backend/vif\
/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/1'} to
/domain/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/device/vif/1.
[2005-09-21 11:36:44 xend] DEBUG (DevController:183) DevController:
writing {'bridge': 'xen-br0', 'mac': 'aa:00:00:00:00:11', 'handle':
'1', 'script': '/etc/xen/scripts/vif-bridge', 'frontend-id': '1',
'doma\
in': 'ubuntu', 'frontend':
'/domain/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/device/vif/1'} to
/domain/0e9d1b5f-ae27-40c0-8f78-b1c1039815dc/backend/vif/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/1.
[2005-09-21 11:36:44 xend] INFO (XendRoot:141) EVENT>
xend.domain.create ['ubuntu', 1]
[2005-09-21 11:36:44 xend] INFO (XendRoot:141) EVENT>
xend.domain.unpause ['ubuntu', 1]
[2005-09-21 11:37:05 xend] DEBUG (XendDomain:232) domain died
name=ubuntu domid=1
[2005-09-21 11:37:05 xend] INFO (XendRoot:141) EVENT> xend.domain.died
['ubuntu', None]

And I once saw this bug, could be related:

xen_blk: Initialising virtual block device driver
xen_blk: Timeout connecting to device!
xen_net: Initialising virtual ethernet driver.
XENBUS ack device/vif/1/tx-ring-ref fail -5
kernel BUG at drivers/xen/xenbus/xenbus_xs.c:151 (xs_talkv)!
 [<c0244c3d>] xs_talkv+0x15d/0x170
 [<c0245880>] watch_thread+0x0/0x180
 [<c0244c95>] xs_single+0x45/0x50
 [<c0244e43>] xenbus_directory+0x33/0x120
 [<c0245880>] watch_thread+0x0/0x180
 [<c0244f4f>] xenbus_exists+0x1f/0x40
 [<c0246711>] dev_changed+0x51/0x140
 [<c0245880>] watch_thread+0x0/0x180
 [<c0244c95>] xs_single+0x45/0x50
 [<c0245880>] watch_thread+0x0/0x180
 [<c024565e>] find_watch+0x1e/0x60
 [<c0245988>] watch_thread+0x108/0x180
 [<c0136110>] autoremove_wake_function+0x0/0x60
 [<c0136110>] autoremove_wake_function+0x0/0x60
 [<c0135c1a>] kthread+0xba/0xc0
 [<c0135b60>] kthread+0x0/0xc0
 [<c0107395>] kernel_thread_helper+0x5/0x10
Kernel panic - not syncing: BUG!


Chris


On 21/09/05, Bonnell, David <David_Bonnell@bmc.com> wrote:
>  
>  
> 
> Since a few changesets back (currently using 6963) I am unable to create a
> guest domain.  It appears the front-end VBD cannot connect to the backend
> and as such the guest cannot mount its root filesystem and panics. 
> 
>   
> 
> The xen console log and xend logs are attached.  Below is the bottom part of
> the guest's console output 
> 
>   
> 
> … 
> 
> Event-channel device ubstakked, 
> 
> xen_blk: Initialising virtual block device driver 
> 
> *** A FEW SECONDS DELAY *** 
> 
> xen_blk: Timeout connecting to device! 
> 
> xen_net: Initialising virtual Ethernet driver. 
> 
> NET: Registered protocol family 2 
> 
> … 
> 
> Root-NFS: No NFS server available, giving up. 
> 
> VFS: Unable to mount root fs via NFS, trying floppy. 
> 
> VFS: Cannot open root device "sda1" or unknown-block(2,0) 
> 
> Please append a correct "root=" boot option 
> 
> Kernel panic – not syncing: VFS: Unable to mount root fs on
> unknown-block(2,0) 
> 
>   
> 
> The "Unable to read memory/target" console message at the end of the Xen
> console log is new for this changeset.  Also, every now and then a failed
> attempt to create a guest domain ends with the following sequence of errors
> on the Xen console and after such a failure the guest domain is not
> automatically destroyed as it is normally and needs a manual "xm destroy
> ???" command to remove it. 
> 
>   
> 
> (XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address
> c8cec000 
> 
> (XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address
> c8ced000 
> 
> (XEN) (file=grant_table.c, line=1054) Grant unref rd(7) ld(0) frm(1ff51)
> flgs(0). 
> 
> (XEN) (file=grant_table.c, line=1054) Grant unref rd(7) ld(0) frm(1ff50)
> flgs(0). 
> 
>   
> 
>   
> 
> -Dave 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 
>

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Chris Bainbridge]

hmm, dmesg also says:

store_evtchn = 14
Unable to read memory/target

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Keir Fraser]

On 21 Sep 2005, at 15:18, Chris Bainbridge wrote:

> hmm, dmesg also says:
>
> store_evtchn = 14
> Unable to read memory/target

We have a patch about ready to check in for this one.

  -- Keir

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Keir Fraser]

On 21 Sep 2005, at 15:10, Chris Bainbridge wrote:

> I'm been struggling with this for a couple of days.. I'd assumed that
> there was just something wrong with my setup rather than a bug. I've
> seen the same PTE error. In trying to eliminate possible problems I've
> started using the nightly binary xen builds. So I'm using the binary
> precompiled hypervisor, dom0 and domU kernels.

Yeah this is a subtle bug in every backend driver. I'll see about 
making a patch.

  -- Keir

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Keir Fraser]

On 21 Sep 2005, at 15:46, Keir Fraser wrote:

>> I'm been struggling with this for a couple of days.. I'd assumed that
>> there was just something wrong with my setup rather than a bug. I've
>> seen the same PTE error. In trying to eliminate possible problems I've
>> started using the nightly binary xen builds. So I'm using the binary
>> precompiled hypervisor, dom0 and domU kernels.
>
> Yeah this is a subtle bug in every backend driver. I'll see about 
> making a patch.

Should now be fixed. (Also the 'unable to read memory/target' bug 
should also be gone.)

  -- Keir

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Chris Bainbridge]

On 21/09/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> On 21 Sep 2005, at 15:46, Keir Fraser wrote:
> 
> >> I'm been struggling with this for a couple of days.. I'd assumed that
> >> there was just something wrong with my setup rather than a bug. I've
> >> seen the same PTE error. In trying to eliminate possible problems I've
> >> started using the nightly binary xen builds. So I'm using the binary
> >> precompiled hypervisor, dom0 and domU kernels.
> >
> > Yeah this is a subtle bug in every backend driver. I'll see about
> > making a patch.
> 
> Should now be fixed. (Also the 'unable to read memory/target' bug
> should also be gone.)

Looks like a new problem? Changeset 6897:

 # xm create ubuntu -c
Using config file "ubuntu".
Error: Error creating domain: int argument required

 # cat xend-debug.log
Traceback (most recent call last):
  File "/usr/lib/python2.3/site-packages/xen/xend/XendDomainInfo.py",
line 719, in construct
    self.info['bootloader']))
  File "/usr/lib/python2.3/site-packages/xen/xend/image.py", line 158,
in initDomain
    log.debug("initDomain: cpu=%d mem_kb=%d ssidref=%d dom=%d", cpu,
mem_kb, ssidref, dom)
  File "/usr/lib/python2.3/logging/__init__.py", line 879, in debug
    apply(self._log, (DEBUG, msg, args), kwargs)
  File "/usr/lib/python2.3/logging/__init__.py", line 994, in _log
    self.handle(record)
  File "/usr/lib/python2.3/logging/__init__.py", line 1004, in handle
    self.callHandlers(record)
  File "/usr/lib/python2.3/logging/__init__.py", line 1037, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python2.3/logging/__init__.py", line 592, in handle
    self.emit(record)
  File "/usr/lib/python2.3/logging/handlers.py", line 102, in emit
    msg = "%s\n" % self.format(record)
  File "/usr/lib/python2.3/logging/__init__.py", line 567, in format
    return fmt.format(record)
  File "/usr/lib/python2.3/logging/__init__.py", line 362, in format
    record.message = record.getMessage()
  File "/usr/lib/python2.3/logging/__init__.py", line 233, in getMessage
    msg = msg % self.args
TypeError: int argument required

xend.log:
[2005-09-21 18:35:24 xend] DEBUG (XendDomainInfo:143)
XendDomainInfo.create(/domain, ...)
[2005-09-21 18:35:24 xend] DEBUG (XendDomainInfo:210) parseConfig:
config is ['vm', ['name', 'ttylinux'], ['memory', '64'], ['ssidref',
'0'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['ip',
'1.2.3.4:1.2.3.4::::eth0:off'], ['root', '/dev/sda1 ro'], ['vcpus',
'1'], ['vcpus', '1'], ['boot', 'c']]], ['device', ['vbd', ['uname',
'file:/root/ttylinux-xen'], ['dev', 'sda1'], ['mode', 'w']]],
['device', ['vif', ['mac', 'aa:00:00:5e:22:5f']]]]
[2005-09-21 18:35:24 xend] DEBUG (XendDomainInfo:250) parseConfig:
result is {'ssidref': 0, 'name': 'ttylinux', 'image': ['linux',
['kernel', '/boot/vmlinuz-2.6-xenU'], ['ip',
'1.2.3.4:1.2.3.4::::eth0:off'], ['root', '/dev/sda1 ro'], ['vcpus',
'1'], ['vcpus', '1'], ['boot', 'c']], 'memory': 64, 'maxmem_kb': None,
'backend': [], 'vcpus': 1, 'cpu_weight': None, 'mem_kb': None,
'device': [('vbd', ['vbd', ['uname', 'file:/root/ttylinux-xen'],
['dev', 'sda1'], ['mode', 'w']]), ('vif', ['vif', ['mac',
'aa:00:00:5e:22:5f']])], 'restart_mode': None, 'bootloader': None,
'cpu': None, 'maxmem': None}
[2005-09-21 18:35:24 xend] DEBUG (XendDomainInfo:712)
XendDomainInfo.construct: calling initDomain(None 65536 0 None 1.0)
[2005-09-21 18:35:24 xend] DEBUG (XendDomainInfo:802) XendDomainInfo.destroy
[2005-09-21 18:35:24 xend] DEBUG (XendDomainInfo:458) Storing
{'xend/state': 'terminated', 'memory_KiB': '65536', 'uuid':
'f915ea65-5ced-4f09-91d6-bc9ee7567351', 'bootloader': 'None', 'image':
"['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['ip',
'1.2.3.4:1.2.3.4::::eth0:off'], ['root', '/dev/sda1 ro'], ['vcpus',
'1'], ['vcpus', '1'], ['boot', 'c']]", 'cpu': 'None', 'maxmem_KiB':
'1073741824', 'memory/target': '65536', 'domid': 'None', 'name':
'ttylinux', 'xend/restart_count': '0', 'xend/restart_mode':
'onreboot', 'restart_time': 'None', 'restart_mode': 'onreboot',
'device': "[('vbd', ['vbd', ['uname', 'file:/root/ttylinux-xen'],
['dev', 'sda1'], ['mode', 'w']]), ('vif', ['vif', ['mac',
'aa:00:00:5e:22:5f']])]", 'cpu_weight': '1.0', 'cpumap': '[1]',
'backend': '[]', 'vcpus': '1', 'ssidref': '0'}

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Steven Hand]

> On 21/09/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> > 
> > On 21 Sep 2005, at 15:46, Keir Fraser wrote:
> > 
> > >> I'm been struggling with this for a couple of days.. I'd assumed that
> > >> there was just something wrong with my setup rather than a bug. I've
> > >> seen the same PTE error. In trying to eliminate possible problems I've
> > >> started using the nightly binary xen builds. So I'm using the binary
> > >> precompiled hypervisor, dom0 and domU kernels.
> > >
> > > Yeah this is a subtle bug in every backend driver. I'll see about
> > > making a patch.
> > 
> > Should now be fixed. (Also the 'unable to read memory/target' bug
> > should also be gone.)
> 
> Looks like a new problem? Changeset 6897:


Do you mean 6897? That's about 4 days ago... 

cheers,

S.

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Chris Bainbridge]

On 21/09/05, Steven Hand <Steven.Hand@cl.cam.ac.uk> wrote:
>
> > On 21/09/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> > >
> > > On 21 Sep 2005, at 15:46, Keir Fraser wrote:
> > >
> > > >> I'm been struggling with this for a couple of days.. I'd assumed that
> > > >> there was just something wrong with my setup rather than a bug. I've
> > > >> seen the same PTE error. In trying to eliminate possible problems I've
> > > >> started using the nightly binary xen builds. So I'm using the binary
> > > >> precompiled hypervisor, dom0 and domU kernels.
> > > >
> > > > Yeah this is a subtle bug in every backend driver. I'll see about
> > > > making a patch.
> > >
> > > Should now be fixed. (Also the 'unable to read memory/target' bug
> > > should also be gone.)
> >
> > Looks like a new problem? Changeset 6897:
>
>
> Do you mean 6897? That's about 4 days ago...

I meant 6987, from 15 hours ago.

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Chris Bainbridge]

changeset:   6987:76f4c7ea7602

On 21/09/05, Chris Bainbridge <chris.bainbridge@gmail.com> wrote:
>
>  # xm create ubuntu -c
> Using config file "ubuntu".
> Error: Error creating domain: int argument required
>
>  # cat xend-debug.log
> Traceback (most recent call last):
>   File "/usr/lib/python2.3/site-packages/xen/xend/XendDomainInfo.py",
> line 719, in construct
>     self.info['bootloader']))
>   File "/usr/lib/python2.3/site-packages/xen/xend/image.py", line 158,
> in initDomain
>     log.debug("initDomain: cpu=%d mem_kb=%d ssidref=%d dom=%d", cpu,
> mem_kb, ssidref, dom)

At this point, cpu==None. Confirmed by xend.log:

[2005-09-22 09:37:19 xend] DEBUG (XendDomainInfo:712)
XendDomainInfo.construct: calling initDomain(None 98304 0 None 1.0)

domid and cpu are both None there... the rest of xend.log is:

  2005-09-22 09:46:03 xend] DEBUG (XendDomainInfo:539)
XendDomainInfo.update({'vcpu_to_cpu': [0, 1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1], 'cpu_time': 15142445835L, 'ssidref': 0,
'shutdown_reason': 0, 'dying': 0, 'dom': 0, 'running': 1, 'maxmem_kb':
-4, 'paused': 0, 'crashed': 0, 'vcpus': 2, 'shutdown': 0, 'mem_kb':
93184, 'cpumap': [-1, -1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'blocked': 0}) on domain
0
[2005-09-22 09:46:03 xend] DEBUG (XendDomainInfo:550)
XendDomainInfo.update done on domain 0: {'paused': 0, 'shutdown': 0,
'backend': [], 'dying': 0, 'maxmem_KiB': 1073741824, 'cpu_weight':
1.0, 'restart_mode': 'onreboot', 'blocked': 0, 'shutdown_reason': 0,
'crashed': 0, 'running': 1, 'device': [], 'cpumap': [-1, -1, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0], 'bootloader': None, 'vcpu_to_cpu': [0, 1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1], 'cpu_time': 15142445835L,
'ssidref': 0, 'memory_KiB': 93184, 'name': 'Domain-0', 'dom': 0,
'vcpus': 2}
[2005-09-22 09:46:03 xend] DEBUG (XendDomainInfo:539)
XendDomainInfo.update(None) on domain 0
[2005-09-22 09:46:03 xend] DEBUG (XendDomainInfo:550)
XendDomainInfo.update done on domain 0: {'paused': 0, 'shutdown': 0,
'backend': [], 'dying': 0, 'maxmem_KiB': 1073741824, 'cpu_weight':
1.0, 'restart_mode': 'onreboot', 'blocked': 0, 'shutdown_reason': 0,
'crashed': 0, 'running': 1, 'device': [], 'cpumap': [-1, -1, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0], 'bootloader': None, 'vcpu_to_cpu': [0, 1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1], 'cpu_time': 15149955096L,
'ssidref': 0, 'memory_KiB': 93184, 'name': 'Domain-0', 'dom': 0,
'vcpus': 2}
[2005-09-22 09:46:19 xend] DEBUG (XendDomainInfo:143)
XendDomainInfo.create(/domain, ...)
[2005-09-22 09:46:19 xend] DEBUG (XendDomainInfo:210) parseConfig:
config is ['vm', ['name', 'ubuntu'], ['memory', '96'], ['ssidref',
'0'], ['image', ['linux', ['kernel', '/boot/vmlinuzU'], ['root',
'/dev/hda1'], ['vcpus', '1'], ['vcpus', '1'], ['boot', 'c']]],
['device', ['vbd', ['uname', 'file:/var/tmp/ubuntu.img'], ['dev',
'hda1'], ['mode', 'w']]], ['device', ['vif', ['mac',
'aa:00:00:00:00:11'], ['bridge', 'xen-br0']]]]
[2005-09-22 09:46:19 xend] DEBUG (XendDomainInfo:250) parseConfig:
result is {'ssidref': 0, 'name': 'ubuntu', 'image': ['linux',
['kernel', '/boot/vmlinuzU'], ['root', '/dev/hda1'], ['vcpus', '1'],
['vcpus', '1'], ['boot', 'c']], 'memory': 96, 'maxmem_kb': None,
'backend': [], 'vcpus': 1, 'cpu_weight': None, 'mem_kb': None,
'device': [('vbd', ['vbd', ['uname', 'file:/var/tmp/ubuntu.img'],
['dev', 'hda1'], ['mode', 'w']]), ('vif', ['vif', ['mac',
'aa:00:00:00:00:11'], ['bridge', 'xen-br0']])], 'restart_mode': None,
'bootloader': None, 'cpu': None, 'maxmem': None}
[2005-09-22 09:46:19 xend] DEBUG (XendDomainInfo:712)
XendDomainInfo.construct: calling initDomain(None 98304 0 None 1.0)
[2005-09-22 09:46:19 xend] DEBUG (XendDomainInfo:802) XendDomainInfo.destroy
[2005-09-22 09:46:19 xend] DEBUG (XendDomainInfo:458) Storing
{'xend/state': 'terminated', 'memory_KiB': '98304', 'uuid':
'4148da2c-bb0b-4afe-8585-aa8256ae5374', 'bootloader': 'None', 'image':
"['linux', ['kernel', '/boot/vmlinuzU'], ['root', '/dev/hda1'],
['vcpus', '1'], ['vcpus', '1'], ['boot', 'c']]", 'cpu': 'None',
'maxmem_KiB': '1073741824', 'memory/target': '98304', 'domid': 'None',
'name': 'ubuntu', 'xend/restart_count': '0', 'xend/restart_mode':
'onreboot', 'restart_time': 'None', 'restart_mode': 'onreboot',
'device': "[('vbd', ['vbd', ['uname', 'file:/var/tmp/ubuntu.img'],
['dev', 'hda1'], ['mode', 'w']]), ('vif', ['vif', ['mac',
'aa:00:00:00:00:11'], ['bridge', 'xen-br0']])]", 'cpu_weight': '1.0',
'cpumap': '[1]', 'backend': '[]', 'vcpus': '1', 'ssidref': '0'}

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Chris Bainbridge]

On 22/09/05, Chris Bainbridge <chris.bainbridge@gmail.com> wrote:
> changeset:   6987:76f4c7ea7602
>
> On 21/09/05, Chris Bainbridge <chris.bainbridge@gmail.com> wrote:
> >
> >  # xm create ubuntu -c
> > Using config file "ubuntu".
> > Error: Error creating domain: int argument required

ok, cpu=-1 in config fixed this error, as per Kips post.

But the original problem with the block device still persists!

# xm create ubuntu -c
store_evtchn = 1
xen_mem: Initialising balloon driver.
SCSI subsystem initialized
Grant table initialized
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
loop: loaded (max 8 devices)
Xen virtual console successfully installed as tty1
Event-channel device installed.
xen_blk: Initialising virtual block device driver
xen_blk: Timeout connecting to device!
xen_net: Initialising virtual ethernet driver.
NET: Registered protocol family 2

# xend-debug.log
ERROR: do_evtchn_op: HYPERVISOR_event_channel_op failed: -1
ERROR: do_evtchn_op: HYPERVISOR_event_channel_op failed: -1

# xend.log
[2005-09-22 09:58:07 xend] DEBUG (XendDomainInfo:143)
XendDomainInfo.create(/domain, ...)
[2005-09-22 09:58:07 xend] DEBUG (XendDomainInfo:210) parseConfig:
config is ['vm', ['name', 'ubuntu'], ['memory', '96'], ['ssidref',
'0'], ['cpu', '-1'], ['image', ['linux', ['kernel', '/boot/vmlinuzU'],
['root', '/dev/hda1'], ['vcpus', '1'], ['vcpus', '1'], ['boot',
'c']]], ['device', ['vbd', ['uname', 'file:/var/tmp/ubuntu.img'],
['dev', 'hda1'], ['mode', 'w']]], ['device', ['vif', ['mac',
'aa:00:00:00:00:11'], ['bridge', 'xen-br0']]]]
[2005-09-22 09:58:07 xend] DEBUG (XendDomainInfo:250) parseConfig:
result is {'ssidref': 0, 'name': 'ubuntu', 'image': ['linux',
['kernel', '/boot/vmlinuzU'], ['root', '/dev/hda1'], ['vcpus', '1'],
['vcpus', '1'], ['boot', 'c']], 'memory': 96, 'maxmem_kb': None,
'backend': [], 'vcpus': 1, 'cpu_weight': None, 'mem_kb': None,
'device': [('vbd', ['vbd', ['uname', 'file:/var/tmp/ubuntu.img'],
['dev', 'hda1'], ['mode', 'w']]), ('vif', ['vif', ['mac',
'aa:00:00:00:00:11'], ['bridge', 'xen-br0']])], 'restart_mode': None,
'bootloader': None, 'cpu': -1, 'maxmem': None}
[2005-09-22 09:58:07 xend] DEBUG (XendDomainInfo:712)
XendDomainInfo.construct: calling initDomain(None 98304 0 -1 1.0)
[2005-09-22 09:58:07 xend] DEBUG (image:158) initDomain: cpu=-1
mem_kb=98304 ssidref=0 dom=1
[2005-09-22 09:58:07 xend] DEBUG (XendDomainInfo:724) init_domain>
Created domain=1 name=ubuntu memory=98304
[2005-09-22 09:58:08 xend] INFO (image:193) buildDomain os=linux dom=1 vcpus=1
[2005-09-22 09:58:08 xend] DEBUG (image:239) dom            = 1
[2005-09-22 09:58:08 xend] DEBUG (image:240) image          = /boot/vmlinuzU
[2005-09-22 09:58:08 xend] DEBUG (image:241) store_evtchn   = 1
[2005-09-22 09:58:08 xend] DEBUG (image:242) console_evtchn = 2
[2005-09-22 09:58:08 xend] DEBUG (image:243) cmdline        =  root=/dev/hda1
[2005-09-22 09:58:08 xend] DEBUG (image:244) ramdisk        =
[2005-09-22 09:58:08 xend] DEBUG (image:245) flags          = 0
[2005-09-22 09:58:08 xend] DEBUG (image:246) vcpus          = 1
[2005-09-22 09:58:08 xend] DEBUG (XendDomainInfo:458) Storing
{'xend/state': 'ok', 'memory_KiB': '98304', 'uuid':
'bfe56e71-e231-48fe-8743-979575e94bd7', 'bootloader': 'None', 'vcpus':
'1', 'start_time': '1127379487.97', 'cpu': '-1', 'maxmem_KiB':
'1073741824', 'memory/target': '98304', 'domid': '1', 'name':
'ubuntu', 'xend/restart_count': '0', 'xend/restart_mode': 'onreboot',
'restart_time': 'None', 'restart_mode': 'onreboot', 'device':
"[('vbd', ['vbd', ['uname', 'file:/var/tmp/ubuntu.img'], ['dev',
'hda1'], ['mode', 'w']]), ('vif', ['vif', ['mac',
'aa:00:00:00:00:11'], ['bridge', 'xen-br0']])]", 'cpu_weight': '1.0',
'cpumap': '[1]', 'backend': '[]', 'image': "['linux', ['kernel',
'/boot/vmlinuzU'], ['root', '/dev/hda1'], ['vcpus', '1'], ['vcpus',
'1'], ['boot', 'c']]", 'ssidref': '0'}
[2005-09-22 09:58:08 xend] DEBUG (DevController:181) DevController:
writing {'virtual-device': '769', 'backend-id': '0', 'backend':
'/domain/2392ff85-b837-4d95-bbd6-9d76066f3eb1/backend/vbd/bfe56e71-e231-48fe-8743-979575e94bd7/769'}
to /domain/bfe56e71-e231-48fe-8743-979575e94bd7/device/vbd/769.
[2005-09-22 09:58:08 xend] DEBUG (DevController:183) DevController:
writing {'params': '/var/tmp/ubuntu.img', 'domain': 'ubuntu', 'type':
'file', 'frontend':
'/domain/bfe56e71-e231-48fe-8743-979575e94bd7/device/vbd/769',
'frontend-id': '1'} to
/domain/2392ff85-b837-4d95-bbd6-9d76066f3eb1/backend/vbd/bfe56e71-e231-48fe-8743-979575e94bd7/769.
[2005-09-22 09:58:08 xend] DEBUG (DevController:181) DevController:
writing {'backend-id': '0', 'mac': 'aa:00:00:00:00:11', 'handle': '1',
'backend': '/domain/2392ff85-b837-4d95-bbd6-9d76066f3eb1/backend/vif/bfe56e71-e231-48fe-8743-979575e94bd7/1'}
to /domain/bfe56e71-e231-48fe-8743-979575e94bd7/device/vif/1.
[2005-09-22 09:58:08 xend] DEBUG (DevController:183) DevController:
writing {'bridge': 'xen-br0', 'mac': 'aa:00:00:00:00:11', 'handle':
'1', 'script': '/etc/xen/scripts/vif-bridge', 'frontend-id': '1',
'domain': 'ubuntu', 'frontend':
'/domain/bfe56e71-e231-48fe-8743-979575e94bd7/device/vif/1'} to
/domain/2392ff85-b837-4d95-bbd6-9d76066f3eb1/backend/vif/bfe56e71-e231-48fe-8743-979575e94bd7/1.
[2005-09-22 09:58:08 xend] DEBUG (XendDomainInfo:458) Storing
{'xend/state': 'ok', 'memory_KiB': '98304', 'uuid':
'bfe56e71-e231-48fe-8743-979575e94bd7', 'bootloader': 'None', 'vcpus':
'1', 'start_time': '1127379487.97', 'cpu': '-1', 'maxmem_KiB':
'1073741824', 'memory/target': '98304', 'domid': '1', 'name':
'ubuntu', 'xend/restart_count': '0', 'xend/restart_mode': 'onreboot',
'restart_time': 'None', 'restart_mode': 'onreboot', 'device':
"[('vbd', ['vbd', ['uname', 'file:/var/tmp/ubuntu.img'], ['dev',
'hda1'], ['mode', 'w']]), ('vif', ['vif', ['mac',
'aa:00:00:00:00:11'], ['bridge', 'xen-br0']])]", 'cpu_weight': '1.0',
'cpumap': '[1]', 'backend': '[]', 'image': "['linux', ['kernel',
'/boot/vmlinuzU'], ['root', '/dev/hda1'], ['vcpus', '1'], ['vcpus',
'1'], ['boot', 'c']]", 'ssidref': '0'}
[2005-09-22 09:58:08 xend] DEBUG (XendDomainInfo:458) Storing
{'xend/state': 'ok', 'memory_KiB': '98304', 'uuid':
'bfe56e71-e231-48fe-8743-979575e94bd7', 'bootloader': 'None', 'vcpus':
'1', 'start_time': '1127379487.97', 'cpu': '-1', 'maxmem_KiB':
'1073741824', 'memory/target': '98304', 'domid': '1', 'name':
'ubuntu', 'xend/restart_count': '0', 'xend/restart_mode': 'onreboot',
'restart_time': 'None', 'restart_mode': 'onreboot', 'device':
"[('vbd', ['vbd', ['uname', 'file:/var/tmp/ubuntu.img'], ['dev',
'hda1'], ['mode', 'w']]), ('vif', ['vif', ['mac',
'aa:00:00:00:00:11'], ['bridge', 'xen-br0']])]", 'cpu_weight': '1.0',
'cpumap': '[1]', 'backend': '[]', 'image': "['linux', ['kernel',
'/boot/vmlinuzU'], ['root', '/dev/hda1'], ['vcpus', '1'], ['vcpus',
'1'], ['boot', 'c']]", 'ssidref': '0'}
[2005-09-22 09:58:08 xend] INFO (XendRoot:141) EVENT>
xend.domain.create ['ubuntu', 1]
[2005-09-22 09:58:08 xend] INFO (XendRoot:141) EVENT>
xend.domain.unpause ['ubuntu', 1]
[2005-09-22 09:58:18 xend] DEBUG (XendDomain:227) domain died
name=ubuntu domid=1
[2005-09-22 09:58:18 xend] DEBUG (XendDomainInfo:802) XendDomainInfo.destroy
[2005-09-22 09:58:18 xend] DEBUG (XendDomainInfo:458) Storing
{'xend/state': 'terminated', 'memory_KiB': '98304', 'uuid':
'bfe56e71-e231-48fe-8743-979575e94bd7', 'bootloader': 'None', 'vcpus':
'1', 'start_time': '1127379487.97', 'cpu': '-1', 'maxmem_KiB':
'1073741824', 'memory/target': '98304', 'domid': '1', 'name':
'ubuntu', 'xend/restart_count': '0', 'xend/restart_mode': 'onreboot',
'restart_time': 'None', 'restart_mode': 'onreboot', 'device':
"[('vbd', ['vbd', ['uname', 'file:/var/tmp/ubuntu.img'], ['dev',
'hda1'], ['mode', 'w']]), ('vif', ['vif', ['mac',
'aa:00:00:00:00:11'], ['bridge', 'xen-br0']])]", 'cpu_weight': '1.0',
'cpumap': '[1]', 'backend': '[]', 'image': "['linux', ['kernel',
'/boot/vmlinuzU'], ['root', '/dev/hda1'], ['vcpus', '1'], ['vcpus',
'1'], ['boot', 'c']]", 'ssidref': '0'}
[2005-09-22 09:58:18 xend] DEBUG (XendDomainInfo:539)
XendDomainInfo.update({'vcpu_to_cpu': [0, 1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1], 'cpu_time': 16817804111L, 'ssidref': 0,
'shutdown_reason': 0, 'dying': 0, 'dom': 0, 'running': 1, 'maxmem_kb':
-4, 'paused': 0, 'crashed': 0, 'vcpus': 2, 'shutdown': 0, 'mem_kb':
93184, 'cpumap': [-1, -1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'blocked': 0}) on domain
0
[2005-09-22 09:58:18 xend] DEBUG (XendDomainInfo:550)
XendDomainInfo.update done on domain 0: {'paused': 0, 'shutdown': 0,
'backend': [], 'dying': 0, 'maxmem_KiB': 1073741824, 'cpu_weight':
1.0, 'restart_mode': 'onreboot', 'blocked': 0, 'shutdown_reason': 0,
'crashed': 0, 'running': 1, 'device': [], 'cpumap': [-1, -1, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0], 'bootloader': None, 'vcpu_to_cpu': [0, 1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, -1], 'cpu_time': 16817804111L,
'ssidref': 0, 'memory_KiB': 93184, 'name': 'Domain-0', 'dom': 0,
'vcpus': 2}
[2005-09-22 09:58:18 xend] INFO (XendRoot:141) EVENT> xend.domain.died
['ubuntu', 1]

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Ewan Mellor]

On Thu, Sep 22, 2005 at 10:03:59AM +0100, Chris Bainbridge wrote:

> On 22/09/05, Chris Bainbridge <chris.bainbridge@gmail.com> wrote:
> > changeset:   6987:76f4c7ea7602
> >
> > On 21/09/05, Chris Bainbridge <chris.bainbridge@gmail.com> wrote:
> > >
> > >  # xm create ubuntu -c
> > > Using config file "ubuntu".
> > > Error: Error creating domain: int argument required
> 
> ok, cpu=-1 in config fixed this error, as per Kips post.

Thanks, to both you and Kip.  There's a fix on the way for this.

> But the original problem with the block device still persists!
> 
> # xm create ubuntu -c
> store_evtchn = 1
> xen_mem: Initialising balloon driver.
> SCSI subsystem initialized
> Grant table initialized
> io scheduler noop registered
> io scheduler anticipatory registered
> io scheduler deadline registered
> io scheduler cfq registered
> loop: loaded (max 8 devices)
> Xen virtual console successfully installed as tty1
> Event-channel device installed.
> xen_blk: Initialising virtual block device driver
> xen_blk: Timeout connecting to device!

When I see this, it clears after a reboot.  Does that work for you?  It would
certainly happen at the moment if you restart xenstored.  Are you doing that?
If not, then we have a problem to chase.

Ewan.

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Chris Bainbridge]

On 22/09/05, Ewan Mellor <ewan@xensource.com> wrote:
> On Thu, Sep 22, 2005 at 10:03:59AM +0100, Chris Bainbridge wrote:
> > But the original problem with the block device still persists!
> >
> > # xm create ubuntu -c
> > store_evtchn = 1
> > xen_mem: Initialising balloon driver.
> > SCSI subsystem initialized
> > Grant table initialized
> > io scheduler noop registered
> > io scheduler anticipatory registered
> > io scheduler deadline registered
> > io scheduler cfq registered
> > loop: loaded (max 8 devices)
> > Xen virtual console successfully installed as tty1
> > Event-channel device installed.
> > xen_blk: Initialising virtual block device driver
> > xen_blk: Timeout connecting to device!
>
> When I see this, it clears after a reboot.  Does that work for you?  It would
> certainly happen at the moment if you restart xenstored.  Are you doing that?
> If not, then we have a problem to chase.

I've tried rebooting, doesn't work, rm -rf /var/lib/xenstored, reboot,
doesn't work.

All I do after reboot is ssh in, cd /etc/xen/auto, xend start, xm
create ubuntu -c. I've built my own dom0 and domU kernels from
changeset 6987 and tried last nights binary release and they both fail
in this same way.

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Chris Bainbridge]

On 22/09/05, Chris Bainbridge <chris.bainbridge@gmail.com> wrote:
> On 22/09/05, Ewan Mellor <ewan@xensource.com> wrote:
> > On Thu, Sep 22, 2005 at 10:03:59AM +0100, Chris Bainbridge wrote:
> > > But the original problem with the block device still persists!

Here are two oops.

Oops: 0000 [#1]
SMP
CPU:    1
EIP:    0061:[<c014627a>]    Not tainted VLI
EFLAGS: 00010206   (2.6.12.5-xen)
EIP is at generic_page_range+0x23/0x16a
eax: 00000000   ebx: c6d2c000   ecx: 00000000   edx: 0000031b
esi: c6d2c000   edi: 00202000   ebp: c0fe5684   esp: c0fe5628
ds: 007b   es: 007b   ss: 0069
Process kxbwatch (pid: 2125, threadinfo=c0fe4000 task=c1381a20)
Stack: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c6d2d000
       00000000 00001000 c6d2c000 00202000 c0fe5684 c011587f c011570c c0fe5680
       00000000 c0fe5694 00001000 00020000 00001000 00000000 c0fe5684 00000000
Call Trace:
 [<c011587f>] __direct_remap_pfn_range+0x10c/0x14d
 [<c011570c>] direct_remap_area_pte_fn+0x0/0x67
 [<c01197e7>] __wake_up+0x38/0x4e
 [<c021f1fe>] wake_waiting+0x1e/0x27
 [<c0118b00>] find_busiest_group+0x1bc/0x301
 [<c0118e93>] load_balance_newidle+0x30/0x8c
 [<c010af82>] monotonic_clock+0x55/0x95
 [<c012e465>] search_exception_tables+0x11/0x12
 [<c0116cd3>] fixup_exception+0xb/0x20
 [<c011539c>] do_page_fault+0x2ec/0x5cf
 [<c02de1e7>] schedule+0x417/0xc34
 [<c010af82>] monotonic_clock+0x55/0x95
 [<c01197e7>] __wake_up+0x38/0x4e
 [<c021f482>] xs_input_avail+0x28/0x35
 [<c021f613>] xb_read+0x184/0x1c4
 [<c0130efe>] autoremove_wake_function+0x0/0x43
 [<c0108e1e>] page_fault+0x2e/0x34
 [<c02e0722>] iret_exc+0x89e/0xb64
 [<c0146357>] generic_page_range+0x100/0x16a
 [<c02249d3>] map_frontend_pages+0x33/0x75
 [<c0224a69>] netif_map+0x53/0x12e
 [<c022429e>] frontend_changed+0x190/0x222
 [<c0220471>] watch_thread+0xf2/0x168
 [<c0130efe>]_function+0x0/0x43
 [<c0130efe>] autoremove_wake_function+0x98cb>] complete+0x40/0x55
 [<c022037f>] watch_thread+0x0/0x16] kthread+0xa0/0xd4
 [<c01309c6>] kthread+0x0/0xd4
 [<c0106c79>] kernel_thread_helper+0x5/0xb
Code: b8 2e c0 e8 74 70 fd ff 55 8d 0c 0a 57 56 53 89 d3 83 ec 24 39
ca 89 44 24 20 89 4c 24 1c 0f 83 19 01 00 00 8b 4c 24 20 c1 ea 16 <8b>
41 1c 83 c1 3c 8d 2c 90 89 c8 89 4c 24 10 e8 c9 96 19 00 8b
 smp_send_stop disable_local_APIC


(XEN) *** Oops: 0000 [#1]
SMP
CPU:    1
EIP:    0061:[<c014627a>]    Not tainted VLI
EFLAGS: 00010206   (2.6.12.5-xen)
EIP is at generic_page_range+0x23/0x16a
eax: 00000000   ebx: c6d2c000   ecx: 00000000   edx: 0000031b
esi: c6d2c000   edi: 00202000   ebp: c5a0d684   esp: c5a0d628
ds: 007b   es: 007b   ss: 0069
Process kxbwatch (pid: 2098, threadinfo=c5a0c000 task=c2a04a20)
Stack: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c6d2d000
       00000000 00001000 c6d2c000 00202000 c5a0d684 c011587f c011570c c5a0d680
       00000000 c5a0d694 00001000 00010000 00001000 00000000 c5a0d684 00000000
Call Trace:
 [<c011587f>] __direct_remap_pfn_range+0x10c/0x14d
 [<c011570c>] direct_remap_area_pte_fn+0x0/0x67
 [<c01a464c>] search_by_key+0x41c/0xc48
 [<c015811e>] bh_lru_install+0xc8/0x125
 [<c0136ea8>] f39/0x4a
 [<c01a464c>] search_by_key+0x41c/0xc48
 [<c0158228>] __find_get_block+0xad/0x10c
 [<c01a464c>] search_by_key+0x41c/0xc48
 [<c01a245f>] leaf_paste_entries+0x108/0x1b5
 [<c020837c>] as_update_arq+0x1e/0x5e
 [<c020911d>] as_add_request+0x1a6/0x22d
 [<c0209c70>] as_set_request+0x14/0x63
 [<c0209c5c>] as_set_request+0x0/0x63
 [<c015811e>] bh_lru_install+0xc8/0x125
 [<c0136ea8>] find_get_page+0x39/0x4a
 [<c01a4ed8>] search_for_position_by_key+0x60/0x37a
 [<c0158228>] __find_get_block+0xad/0x10c
 [<c01a464c>] search_by_key+0x41c/0xc48
 [<c0130efe>] autoremove_wake_function+0x0/0x43
 [<c01900a1>] make_cpu_key+0x4b/0x5b
 [<c01a3ec5>] pathrelse+0x1e/0x2c
 [<c01907c0>] _get_block_create_0+0x4c6/0x698
 [<c01917dd>] reiserfs_get_block+0xbad/0x11b6
 [<c01900a1>] make_cpu_key+0x4b/0x5b
 [<c01907c0>] _get_block_create_0+0x4c6/0x698
 [<c019a357>] ip_check_balance+0x3f4/0xaf8
 [<c015811e>] bh_lru_install+0xc8/0x125
 [<c0136ea8>] find_get_page+0x39/0x4a
 [<c01a464c>] search_by_key+0x41c/0xc48
 [<c010af82>] monotonic_clock+0x55/0x95
 [<c01188e9>] move_tasks+0x1d4/0x22f
 [<c0118b00>] find_busiest_group+0x1bc/0x301
 [<c0118e93>] load_balance_newidle+0x30/0x8c
 [<c010af82>] monotonic_clock+0x55/0x95
 [<c012e465>] search_exception_tables+0x11/0x12
 [<c0116cd3>] fixup_exception+0xb/0x20
 [<c011539c>] do_page_fault+0x2ec/0x5cf
 [<c02de1e7>] schedule+0x417/0xc34
 [<c010af82>] monotonic_clock+0x55/0x95
 [<c01197e7>] __wake_up+0x38/0x4e
 [<c021f482>] xs_input_avail+0x28/0x35
 [<c021f613>] xb_read+0x184/0x1c4
 [<c0130efe>] autoremove_wake_function+0x0/0x43
 [<c0108e1e>] page_fault+0x2e/0x34
 [<c02e0722>] iret_exc+0x89e/0xb64
 [<c0146357>] generic_page_range+0x100/0x16a
 [<c02249d3>] map_frontend_pages+0x33/0x75
 [<c0224a69>] netif_map+0x53/0x12e
 [<c022429e>] frontend_changed+0x190/0x222
 [<c0220471>] watch_thread+0xf2/0x168
 [<c0130efe>] autoremove_wake_function+0x0/0x43
 [<c0130efe>] autoremove_wake_function+0x0/0x43
 [<c01198cb>] complete+0x40/0x55
 [<c022037f>] watch_thread+0x0/0x168
 [<c0130a66>] kthread+0xa0/0xd4
 [<c01309c6>] kthread+0x0/0xd4
 [<c0106c79>] kernel_thread_helper+0x5/0xb
Code: b8 2e c0 e8 74 70 fd ff 55 8d 0c 0a 57 56 53 89 d3 83 ec 24 39
ca 89 44 24 20 89 4c 24 1c 0f 83 19 01 00 00 8b 4c 24 20 c1 ea 16 <8b>
41 1c 83 c1 3c 8d 2c 90 89 c8 89 4c 24 10 e8 c9 96 19 00 8b
 smp_send_stop disable_local_APIC

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Jon Mason]

I have been seeing the exact same problem (xen_blk timing out) for a
couple of weeks now.  I used mercurial to go back and find which patch 
caused me to stop working (hg update -C <Node ID>).  Using this method,
I determined the patch to be causing the problem is "Move block device
bind/unbind into hotplug scripts" (e9d01c5dc7b4d6b7cda9ade0d137ddb89bb204cc), 
which was dropped on 12 Sep 2005.

Since then many changes have been made to these same files, and I lack
the understanding to be able to fix the problem myself.  I'll be happy
to test any patches or help out in any other way.

Thanks,
Jon


On Wed, Sep 21, 2005 at 08:29:04AM -0500, Bonnell, David wrote:
> Since a few changesets back (currently using 6963) I am unable to create
> a guest domain.  It appears the front-end VBD cannot connect to the
> backend and as such the guest cannot mount its root filesystem and
> panics.
>  
> The xen console log and xend logs are attached.  Below is the bottom
> part of the guest's console output
>  
> ...
> Event-channel device ubstakked,
> xen_blk: Initialising virtual block device driver
> *** A FEW SECONDS DELAY ***
> xen_blk: Timeout connecting to device!
> xen_net: Initialising virtual Ethernet driver.
> NET: Registered protocol family 2
> ...
> Root-NFS: No NFS server available, giving up.
> VFS: Unable to mount root fs via NFS, trying floppy.
> VFS: Cannot open root device "sda1" or unknown-block(2,0)
> Please append a correct "root=" boot option
> Kernel panic - not syncing: VFS: Unable to mount root fs on
> unknown-block(2,0)
>  
> The "Unable to read memory/target" console message at the end of the Xen
> console log is new for this changeset.  Also, every now and then a
> failed attempt to create a guest domain ends with the following sequence
> of errors on the Xen console and after such a failure the guest domain
> is not automatically destroyed as it is normally and needs a manual "xm
> destroy ???" command to remove it.
>  
> (XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address
> c8cec000
> (XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address
> c8ced000
> (XEN) (file=grant_table.c, line=1054) Grant unref rd(7) ld(0) frm(1ff51)
> flgs(0).
> (XEN) (file=grant_table.c, line=1054) Grant unref rd(7) ld(0) frm(1ff50)
> flgs(0).
>  
>  
> -Dave



> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Kip Macy]

[-- Attachment #1.1: Type: text/plain, Size: 894 bytes --]

Based on Michael Vrable's mail, I've installed hotplug. I'm still seeing
timeouts when using files (and after the inevitable panic an undestroyable
domain) and hangs when trying to export an unpartitioned loopback device.
Obviously I'm still missing something from my ramdisk. I would appreciate
pointers on tracking this down. I continue to only be able to mount the root
using Arun's workaround.

-Kip




On 9/22/05, Jon Mason <jdmason@us.ibm.com> wrote:
>
> I have been seeing the exact same problem (xen_blk timing out) for a
> couple of weeks now. I used mercurial to go back and find which patch
> caused me to stop working (hg update -C <Node ID>). Using this method,
> I determined the patch to be causing the problem is "Move block device
> bind/unbind into hotplug scripts"
> (e9d01c5dc7b4d6b7cda9ade0d137ddb89bb204cc),
> which was dropped on 12 Sep 2005.
>
>

[-- Attachment #1.2: Type: text/html, Size: 1319 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Chris Bainbridge]

On 26/09/05, Kip Macy <kip.macy@gmail.com> wrote:
> Based on Michael Vrable's mail, I've installed hotplug. I'm still seeing
> timeouts when using files (and after the inevitable panic an undestroyable
> domain) and hangs when trying to export an unpartitioned loopback device.
> Obviously I'm still missing something from my ramdisk. I would appreciate
> pointers on tracking this down. I continue to only be able to mount the root
> using Arun's workaround.

Did you set /proc/sys/kernel/hotplug to the hotplug executable?

Also what is Aruns workaround? There is no email from Arun in this thread.

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Kip Macy]

[-- Attachment #1.1: Type: text/plain, Size: 472 bytes --]

>
> Did you set /proc/sys/kernel/hotplug to the hotplug executable?


This is centos4.1 so it is enabled by default.

-bash-3.00# cat /proc/sys/kernel/hotplug
/sbin/hotplug


Also what is Aruns workaround? There is no email from Arun in this thread.


He sent out e-mail on August 24th when he had similar problems with
timeouts.

>From strace'ing udevd it is clear that xen-backend is getting called. I
can't tell anything more from the output.

-Kip

[-- Attachment #1.2: Type: text/html, Size: 940 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Unable to create guest domain AND mm.c/grant_table.c errors

[Kip Macy]

[-- Attachment #1.1: Type: text/plain, Size: 1058 bytes --]

I realized what the problem was: I was giving a relative path to the
loopback file. Even if I could be considered brain-damaged for trying that,
it would be nice if there were logging indicating wherethe failure occurred.


Exporting a whole device still causes a hang, but file: works for me now.

works:
disk = [ 'file:/usr/src/xen-unstable.hg.tot/testing/newfs.00.vbd,hda1,w' ]

hangs:
disk = [ 'file:/usr/src/xen-unstable.hg.tot/testing/newfs.00.vbd,hda,w' ]



-Kip



On 9/26/05, Kip Macy <kip.macy@gmail.com> wrote:
>
>
>
> > Did you set /proc/sys/kernel/hotplug to the hotplug executable?
>
>
> This is centos4.1 so it is enabled by default.
>
> -bash-3.00# cat /proc/sys/kernel/hotplug
> /sbin/hotplug
>
>
> Also what is Aruns workaround? There is no email from Arun in this thread.
>
>
> He sent out e-mail on August 24th when he had similar problems with
> timeouts.
>
> From strace'ing udevd it is clear that xen-backend is getting called. I
> can't tell anything more from the output.
>
> -Kip
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 1913 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel