can't mount root... devfs???

can't mount root... devfs???

[James Harper]

[-- Attachment #1: Type: text/plain, Size: 390 bytes --]

Domain1 is now not booting anymore, and the boot console tells me it's because it can't mount /dev/sda7.

My config files says sda7 maps to /dev/sda7 eg:

disk = [ 'phy:sda7,sda7,w' ]
root = "/dev/sda7 ro"

My only thought is that maybe devfs is confusing the issue... eg there are no scsi devices so sda7 doesn't appear in /dev.

Can anyone confirm/deny this?

thanks

James

[-- Attachment #2: Type: text/html, Size: 998 bytes --]

Re: can't mount root... devfs???

[Ian Pratt]

> Domain1 is now not booting anymore, and the boot console tells me it's because it can't mount /dev/sda7.
> 
> My config files says sda7 maps to /dev/sda7 eg:
> 
> disk = [ 'phy:sda7,sda7,w' ]
> root = "/dev/sda7 ro"
> 
> My only thought is that maybe devfs is confusing the
> issue... eg there are no scsi devices so sda7 doesn't appear in
> /dev.

I think you are the first person to try using devfs, so it might
take a bit of debugging, but should be possible to make work.

Are you using devfs in both dom0 and guest domains?

If dom0 hasn't got a /dev/sda7 I'm surprised the tools are
letting you proceed as they need to lookup /dev/sda7 to discover
the major,minor (8,7). I believe devfs creates links for the old
/dev/sdXX names though, right?

Within the new guest, it will have had a block device plumbed through
to it appearing as major,minor = 8,7.

When it parses it's root= command line, it will lookup the name
'sda' in the grim table appearing in init/do_mounts.c, and
associate that with 8,7 hence everything should work from a
kernel POV. When the startup scripts go through your fstab and
remount root that's a different matter, and it's possible that
the 8,7 device hasn't been properly registered with devfs.
Does it the guest get this far?


Ian


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: can't mount root... devfs???

[Keir Fraser]

> Domain1 is now not booting anymore, and the boot console tells me it's because it can't mount /dev/sda7.
> 
> My config files says sda7 maps to /dev/sda7 eg:
> 
> disk = [ 'phy:sda7,sda7,w' ]
> root = "/dev/sda7 ro"
> 
> My only thought is that maybe devfs is confusing the issue... eg there are no scsi devices so sda7 doesn't appear in /dev.
> 
> Can anyone confirm/deny this?

We make no effort to make 'logical' devices appear in devfs --- if
it doesn't happen automatically then code needs to be added to do it
manually.

Is devfs still alive in 2.6, or have they moved to some other scheme?

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

RE: can't mount root... devfs???

[James Harper]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

devfs is alive and well in 2.6. there is a udev(?) scheme i think which accomplishes the same sort of thing via a different mechanism, but that is about the extent of my knowledge on it and is probably wrong anyway!!!

it is the kernel failing to mount /dev/sda7, not userspace. It also fails if I specify root as 0807, so I guess it isn't a devfs issue on dom1. Dom0 also is using devfs, /dev/sda7 is a symlink to /dev/scsi/host0/bus0/target5/lun0/part7, which is in turn of the right major and minor number. Would xen be failing to work out the device numbers because of the symlink?

After my next reboot i'll try making /dev/sda7 a proper device node. After dom1 crashes and burns, I don't seem to be able to kill it (xm destroy 1), or create it again, so I have to reboot. doh.

Also, make menuconfig for xenU flashes an error on the screen but otherwise works unless I try to look in SCSI where I get this message:

Menuconfig has encountered a possible error in one of the kernel's
configuration files and is unable to continue.  Here is the error
report:
 Q> scripts/Menuconfig: line 832: MCmenu53: command not found
Please report this to the maintainer <mec@shout.net>.  You may also
send a problem report to <linux-kernel@vger.kernel.org>.
Please indicate the kernel version you are trying to configure and
which menu you were trying to enter when this error occurred.

James





From: Keir Fraser
Sent: Wed 30/06/2004 4:43 PM
To: James Harper
Cc: xen-devel@lists.sourceforge.net
Subject: Re: [Xen-devel] can't mount root... devfs???


> Domain1 is now not booting anymore, and the boot console tells me it's because it can't mount /dev/sda7.
> 
> My config files says sda7 maps to /dev/sda7 eg:
> 
> disk = [ 'phy:sda7,sda7,w' ]
> root = "/dev/sda7 ro"
> 
> My only thought is that maybe devfs is confusing the issue... eg there are no scsi devices so sda7 doesn't appear in /dev.
> 
> Can anyone confirm/deny this?

We make no effort to make 'logical' devices appear in devfs --- if
it doesn't happen automatically then code needs to be added to do it
manually.

Is devfs still alive in 2.6, or have they moved to some other scheme?

 -- Keir

[-- Attachment #2: Type: text/html, Size: 3320 bytes --]

Re: can't mount root... devfs???

[Keir Fraser]

Turns out devfs is obsolete in 2.6 to be replaced by udev, which is
supposed to be rather nicer to work with. I don't think I'll be trying
to emulate /dev/scsi/host0/bus0/target5/lun0/part7 style names with
the Xen backend driver.

As for your DOM0 issue, what do you get if you run
 '/sbin/sfdisk -d /dev/sda' ?

You should get lines of the form:
/dev/sda3 : start= 16948575, size=16836120, Id=83, bootable

Perhaps you are getting the primary devfs names, in which case we can
fix by modifying our matching regexp.

 -- Keir

> devfs is alive and well in 2.6. there is a udev(?) scheme i think which accomplishes the same sort of thing via a different mechanism, but that is about the extent of my knowledge on it and is probably wrong anyway!!!
> 
> it is the kernel failing to mount /dev/sda7, not userspace. It also fails if I specify root as 0807, so I guess it isn't a devfs issue on dom1. Dom0 also is using devfs, /dev/sda7 is a symlink to /dev/scsi/host0/bus0/target5/lun0/part7, which is in turn of the right major and minor number. Would xen be failing to work out the device numbers because of the symlink?
> 
> After my next reboot i'll try making /dev/sda7 a proper device node. After dom1 crashes and burns, I don't seem to be able to kill it (xm destroy 1), or create it again, so I have to reboot. doh.
> 
> Also, make menuconfig for xenU flashes an error on the screen but otherwise works unless I try to look in SCSI where I get this message:
> 
> Menuconfig has encountered a possible error in one of the kernel's
> configuration files and is unable to continue.  Here is the error
> report:
>  Q> scripts/Menuconfig: line 832: MCmenu53: command not found
> Please report this to the maintainer <mec@shout.net>.  You may also
> send a problem report to <linux-kernel@vger.kernel.org>.
> Please indicate the kernel version you are trying to configure and
> which menu you were trying to enter when this error occurred.
> 
> James
> 
> 
> 
> 
> 
> From: Keir Fraser
> Sent: Wed 30/06/2004 4:43 PM
> To: James Harper
> Cc: xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] can't mount root... devfs???
> 
> 
> > Domain1 is now not booting anymore, and the boot console tells me it's because it can't mount /dev/sda7.
> > 
> > My config files says sda7 maps to /dev/sda7 eg:
> > 
> > disk = [ 'phy:sda7,sda7,w' ]
> > root = "/dev/sda7 ro"
> > 
> > My only thought is that maybe devfs is confusing the issue... eg there are no scsi devices so sda7 doesn't appear in /dev.
> > 
> > Can anyone confirm/deny this?
> 
> We make no effort to make 'logical' devices appear in devfs --- if
> it doesn't happen automatically then code needs to be added to do it
> manually.
> 
> Is devfs still alive in 2.6, or have they moved to some other scheme?
> 
>  -- Keir
\x1f -=- MIME -=- \x1f\f

--_03EC8C21-3324-4C90-85C6-3C4D454655EF_
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

devfs is alive and well in 2.6. there is a udev(?) scheme i think which acc=
omplishes the same sort of thing via a different mechanism, but that is abo=
ut the extent of my knowledge on it and is probably wrong anyway!!!

it is the kernel failing to mount /dev/sda7, not userspace. It also fails i=
f I specify root as 0807, so I guess it isn't a devfs issue on dom1. Dom0 a=
lso is using devfs, /dev/sda7 is a symlink to /dev/scsi/host0/bus0/target5/=
lun0/part7, which is in turn of the right major and minor number. Would xen=
 be failing to work out the device numbers because of the symlink?

After my next reboot i'll try making /dev/sda7 a proper device node. After =
dom1 crashes and burns, I don't seem to be able to kill it (xm destroy 1), =
or create it again, so I have to reboot. doh.

Also, make menuconfig for xenU flashes an error on the screen but otherwise=
 works unless I try to look in SCSI where I get this message:

Menuconfig has encountered a possible error in one of the kernel's
configuration files and is unable to continue.  Here is the error
report:
 Q> scripts/Menuconfig: line 832: MCmenu53: command not found
Please report this to the maintainer <mec@shout.net>.  You may also
send a problem report to <linux-kernel@vger.kernel.org>.
Please indicate the kernel version you are trying to configure and
which menu you were trying to enter when this error occurred.

James





From: Keir Fraser
Sent: Wed 30/06/2004 4:43 PM
To: James Harper
Cc: xen-devel@lists.sourceforge.net
Subject: Re: [Xen-devel] can't mount root... devfs???


> Domain1 is now not booting anymore, and the boot console tells me it's be=
cause it can't mount /dev/sda7.
>=20
> My config files says sda7 maps to /dev/sda7 eg:
>=20
> disk =3D [ 'phy:sda7,sda7,w' ]
> root =3D "/dev/sda7 ro"
>=20
> My only thought is that maybe devfs is confusing the issue... eg there ar=
e no scsi devices so sda7 doesn't appear in /dev.
>=20
> Can anyone confirm/deny this?

We make no effort to make 'logical' devices appear in devfs --- if
it doesn't happen automatically then code needs to be added to do it
manually.

Is devfs still alive in 2.6, or have they moved to some other scheme?

 -- Keir

--_03EC8C21-3324-4C90-85C6-3C4D454655EF_
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML><HEAD></HEAD>
<BODY>
<DIV id=3DidOWAReplyText89384 dir=3Dltr>
<DIV dir=3Dltr><FONT face=3DArial color=3D#000000 size=3D2>devfs is alive a=
nd well in 2.6. there is a udev(?) scheme i think which accomplishes the sa=
me sort of thing via a different mechanism, but that is about the extent of=
 my knowledge on it and is probably wrong anyway!!!</FONT></DIV>
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV dir=3Dltr><FONT face=3DArial size=3D2>it is the kernel failing to moun=
t /dev/sda7, not userspace. It also fails if I specify root as 0807, so I g=
uess it isn't a devfs issue on dom1. Dom0 also is using devfs, /dev/sda7 is=
 a symlink to /dev/scsi/host0/bus0/target5/lun0/part7, which is in turn of =
the right major and minor number. Would xen be failing to work out the devi=
ce numbers because of the symlink?</FONT></DIV>
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV dir=3Dltr><FONT face=3DArial size=3D2>After my next reboot i'll try ma=
king /dev/sda7 a proper device node. After dom1 crashes and burns, I don't =
seem to be able to kill it (xm destroy 1), or create it again, so I have to=
 reboot. doh.</FONT></DIV>
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV dir=3Dltr><FONT face=3DArial size=3D2>Also, make menuconfig for xenU f=
lashes an error on the screen but otherwise works unless I try to look in S=
CSI where I get this message:</FONT></DIV>
<DIV dir=3Dltr><FONT face=3DArial size=3D2><BR>Menuconfig has encountered a=
 possible error in one of the kernel's<BR>configuration files and is unable=
 to continue.&nbsp; Here is the error<BR>report:</DIV>
<DIV dir=3Dltr>&nbsp;Q&gt; scripts/Menuconfig: line 832: MCmenu53: command =
not found</DIV>
<DIV dir=3Dltr>Please report this to the maintainer &lt;<A href=3D"mailto:m=
ec@shout.net" target=3D_blank>mec@shout.net</A>&gt;.&nbsp; You may also<BR>=
send a problem report to &lt;<A href=3D"mailto:linux-kernel@vger.kernel.org=
" target=3D_blank>linux-kernel@vger.kernel.org</A>&gt;.</DIV>
<DIV dir=3Dltr>Please indicate the kernel version you are trying to configu=
re and<BR>which menu you were trying to enter when this error occurred.</DI=
V>
<DIV dir=3Dltr></FONT>&nbsp;</DIV>
<DIV dir=3Dltr><FONT face=3DArial size=3D2>James</FONT></DIV>
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV></DIV>
<DIV dir=3Dltr><BR>
<HR tabIndex=3D-1>
<FONT face=3DTahoma size=3D2><B>From:</B> Keir Fraser<BR><B>Sent:</B> Wed 3=
0/06/2004 4:43 PM<BR><B>To:</B> James Harper<BR><B>Cc:</B> xen-devel@lists.=
sourceforge.net<BR><B>Subject:</B> Re: [Xen-devel] can't mount root... devf=
s???<BR></FONT><BR></DIV>
<DIV><PRE style=3D"WORD-WRAP: break-word">&gt; Domain1 is now not booting a=
nymore, and the boot console tells me it's because it can't mount /dev/sda7=
.
&gt;=20
&gt; My config files says sda7 maps to /dev/sda7 eg:
&gt;=20
&gt; disk =3D [ 'phy:sda7,sda7,w' ]
&gt; root =3D "/dev/sda7 ro"
&gt;=20
&gt; My only thought is that maybe devfs is confusing the issue... eg there=
 are no scsi devices so sda7 doesn't appear in /dev.
&gt;=20
&gt; Can anyone confirm/deny this?

We make no effort to make 'logical' devices appear in devfs --- if
it doesn't happen automatically then code needs to be added to do it
manually.

Is devfs still alive in 2.6, or have they moved to some other scheme?

 -- Keir
</PRE></DIV></BODY></HTML>

--_03EC8C21-3324-4C90-85C6-3C4D454655EF_--



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: can't mount root... devfs???

[Ian Pratt]

> devfs is alive and well in 2.6. there is a udev(?) scheme i think which accomplishes the same sort of thing via a different mechanism, but that is about the extent of my knowledge on it and is probably wrong anyway!!!
> 
> it is the kernel failing to mount /dev/sda7, not userspace. It also fails if I specify root as 0807, so I guess it isn't a devfs issue on dom1. Dom0 also is using devfs, /dev/sda7 is a symlink to /dev/scsi/host0/bus0/target5/lun0/part7, which is in turn of the right major and minor number. Would xen be failing to work out the device numbers because of the symlink?

Possibly, though we always used to look through links; don't know
about the new tools, but this could be the issue. Could you try a
devfs dom0 and standard dom1?
 
> After my next reboot i'll try making /dev/sda7 a proper device node. After dom1 crashes and burns, I don't seem to be able to kill it (xm destroy 1), or create it again, so I have to reboot. doh.

Is this with the latest repo? I think some changes that went in
last night finally got this functionality working again.

> Also, make menuconfig for xenU flashes an error on the screen but otherwise works unless I try to look in SCSI where I get this message:
> Menuconfig has encountered a possible error in one of the kernel's
> configuration files and is unable to continue.  Here is the
> error

There's something broken with the arch/xen/config.in but I just
can't spot it. The error messages from the Linux config system
are so spectacularly unhelpful that it's impossible to
debug. Fortunately, the error message doesn't actually seem to
obviously break anything, though I have noticed xconfig do
strange things if you ever visit the SCSI config pages.

I think we need to revert to the standard i386 one, and then just
add back in the Xen specific options.

Anyone fancy a really boring but important job? ;-)

Ian


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: can't mount root... devfs???

[Mark Williamson]

I believe DevFS is officially deprecated in 2.6.  They're looking to replace 
it with udev, which is a userspace daemon that creates device nodes on your 
existing filesystem when it detects and identifies devices using mechanisms 
including the new sysfs.  Amongst other things, this allows for more flexible 
naming policy for device nodes.

It is still in 2.6 but I think they want to get rid of it eventually.

Cheers,
Mark



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: can't mount root... devfs???

[Keir Fraser]

> I believe DevFS is officially deprecated in 2.6.  They're looking to replace 
> it with udev, which is a userspace daemon that creates device nodes on your 
> existing filesystem when it detects and identifies devices using mechanisms 
> including the new sysfs.  Amongst other things, this allows for more flexible 
> naming policy for device nodes.
> 
> It is still in 2.6 but I think they want to get rid of it eventually.

It turns out that most kernel developers hate devfs. And fortunately
it's rabid naming schemes (e.g., /dev/scsi/host/bus/lun/target/....)
are going with it. :-) udev appears to default to using traditional
/dev names [hopefully distros will stick to this too]. 

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: can't mount root... devfs???

[Ian Pratt]

> devfs is alive and well in 2.6. there is a udev(?) scheme i think which accomplishes the same sort of thing via a different mechanism, but that is about the extent of my knowledge on it and is probably wrong anyway!!!
> 
> it is the kernel failing to mount /dev/sda7, not userspace. It also fails if I specify root as 0807, so I guess it isn't a devfs issue on dom1. Dom0 also is using devfs, /dev/sda7 is a symlink to /dev/scsi/host0/bus0/target5/lun0/part7, which is in turn of the right major and minor number. Would xen be failing to work out the device numbers because of the symlink?

I've just done a simple test where I created a normal symlink
pointing to a device node and the control tools followed it fine.

I forget how devfs works, but is the major:minor of
/dev/scsi/host0/bus0/target5/lun0/part7 08:07 ? If so, I'd have
thought it would just work...

Could you do an experiment with a non devfs dom1 kernel just to
ascertain that the problem is actually with the tools or backend
driver in dom0 (or vice versa)?

Thanks,
Ian


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

RE: can't mount root... devfs???

[James Harper]

[-- Attachment #1: Type: text/plain, Size: 1876 bytes --]

I'm now recompiling both xen0 and xenU with devfs not compiled in (I had tried booting xenU with devfs=nomount and it didn't make a difference).

One thing I did just notice is now that the xenU config is fixed for SCSI (make menuconfig would crash if you tried to access SCSI), I can actually go in there and enable or disable scsi in the kernel. I assume that even though xen0 is using the scsi device directly, and that i'm exporting /dev/sda7 to xenU, xenU shouldn't need any scsi block devices at all, is that correct? SCSI was enabled, i'm now trying it with SCSI not selected at all.

thanks

James



From: Ian Pratt
Sent: Thu 1/07/2004 9:14 AM
To: James Harper
Cc: Keir Fraser; xen-devel@lists.sourceforge.net; Ian.Pratt@cl.cam.ac.uk
Subject: Re: [Xen-devel] can't mount root... devfs???


> devfs is alive and well in 2.6. there is a udev(?) scheme i think which accomplishes the same sort of thing via a different mechanism, but that is about the extent of my knowledge on it and is probably wrong anyway!!!
> 
> it is the kernel failing to mount /dev/sda7, not userspace. It also fails if I specify root as 0807, so I guess it isn't a devfs issue on dom1. Dom0 also is using devfs, /dev/sda7 is a symlink to /dev/scsi/host0/bus0/target5/lun0/part7, which is in turn of the right major and minor number. Would xen be failing to work out the device numbers because of the symlink?

I've just done a simple test where I created a normal symlink
pointing to a device node and the control tools followed it fine.

I forget how devfs works, but is the major:minor of
/dev/scsi/host0/bus0/target5/lun0/part7 08:07 ? If so, I'd have
thought it would just work...

Could you do an experiment with a non devfs dom1 kernel just to
ascertain that the problem is actually with the tools or backend
driver in dom0 (or vice versa)?

Thanks,
Ian

[-- Attachment #2: Type: text/html, Size: 2524 bytes --]

Re: can't mount root... devfs???

[Keir Fraser]

> I'm now recompiling both xen0 and xenU with devfs not compiled in (I had tried booting xenU with devfs=nomount and it didn't make a difference).
> 
> One thing I did just notice is now that the xenU config is fixed for SCSI (make menuconfig would crash if you tried to access SCSI), I can actually go in there and enable or disable scsi in the kernel. I assume that even though xen0 is using the scsi device directly, and that i'm exporting /dev/sda7 to xenU, xenU shouldn't need any scsi block devices at all, is that correct? SCSI was enabled, i'm now trying it with SCSI not selected at all.
> 
> thanks
> 
> James

Correct --- you don't need low-level device drivers in xenU. If we
were prepared to modify the drivers/scsi/config, we'd remove the
low-level driver options when configuring a non-privileged kernel.

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: can't mount root... devfs???

[Ian Pratt]

> > One thing I did just notice is now that the xenU config is fixed for SCSI (make menuconfig would crash if you tried to access SCSI), I can actually go in there and enable or disable scsi in the kernel. I assume that even though xen0 is using the scsi device directly, and that i'm exporting /dev/sda7 to xenU, xenU shouldn't need any scsi block devices at all, is that correct? SCSI was enabled, i'm now trying it with SCSI not selected at all.
> 
> Correct --- you don't need low-level device drivers in xenU. If we
> were prepared to modify the drivers/scsi/config, we'd remove the
> low-level driver options when configuring a non-privileged kernel.

Just to follow up, although you never need SCSI low-level drivers
in an unprivileged kernel, there are still occasions where you
might want the SCSI core and disk code, for instance when using
iSCSI within the domain.

Ian 


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

almost working

[James Harper]

[-- Attachment #1: Type: text/plain, Size: 2784 bytes --]

I am once again able to boot a xenU kernel after removing all traces of devfs. *yay*

can't say for sure if it was devfs though as i did a bk pull just before I sent my last email...

having booted my new domain, 'xm list' in dom0 hung for ages with 100% cpu time until I hit ctrl-C. trying it again worked fine though.

halting the new domain from the console causes it to disappear from 'xm list' correctly, but it won't restart properly, and then 'xm destroy, won't remove it from 'xm list'.

lastly, is there a plan to be able to export a filesystem rather than just block devices, like 'hostfs' under uml? It would make copying modules over a bit easier!

keep up the good work!

james



From: James Harper
Sent: Fri 2/07/2004 12:58 PM
To: Ian Pratt
Cc: Keir Fraser; xen-devel@lists.sourceforge.net; Ian.Pratt@cl.cam.ac.uk
Subject: RE: [Xen-devel] can't mount root... devfs???


I'm now recompiling both xen0 and xenU with devfs not compiled in (I had tried booting xenU with devfs=nomount and it didn't make a difference).

One thing I did just notice is now that the xenU config is fixed for SCSI (make menuconfig would crash if you tried to access SCSI), I can actually go in there and enable or disable scsi in the kernel. I assume that even though xen0 is using the scsi device directly, and that i'm exporting /dev/sda7 to xenU, xenU shouldn't need any scsi block devices at all, is that correct? SCSI was enabled, i'm now trying it with SCSI not selected at all.

thanks

James



From: Ian Pratt
Sent: Thu 1/07/2004 9:14 AM
To: James Harper
Cc: Keir Fraser; xen-devel@lists.sourceforge.net; Ian.Pratt@cl.cam.ac.uk
Subject: Re: [Xen-devel] can't mount root... devfs???


> devfs is alive and well in 2.6. there is a udev(?) scheme i think which accomplishes the same sort of thing via a different mechanism, but that is about the extent of my knowledge on it and is probably wrong anyway!!!
> 
> it is the kernel failing to mount /dev/sda7, not userspace. It also fails if I specify root as 0807, so I guess it isn't a devfs issue on dom1. Dom0 also is using devfs, /dev/sda7 is a symlink to /dev/scsi/host0/bus0/target5/lun0/part7, which is in turn of the right major and minor number. Would xen be failing to work out the device numbers because of the symlink?

I've just done a simple test where I created a normal symlink
pointing to a device node and the control tools followed it fine.

I forget how devfs works, but is the major:minor of
/dev/scsi/host0/bus0/target5/lun0/part7 08:07 ? If so, I'd have
thought it would just work...

Could you do an experiment with a non devfs dom1 kernel just to
ascertain that the problem is actually with the tools or backend
driver in dom0 (or vice versa)?

Thanks,
Ian

[-- Attachment #2: Type: text/html, Size: 4359 bytes --]

Re: almost working

[Ian Pratt]

> I am once again able to boot a xenU kernel after removing all traces of devfs. *yay*

Excellent. Success stories are good to hear.

> can't say for sure if it was devfs though as i did a bk pull just before I sent my last email...

I suspect adding devfs support wouldn't be too hard, but its
probably not high on the list of priorities for us. Do any
common distros use it by default?

> having booted my new domain, 'xm list' in dom0 hung for ages with 100% cpu time until I hit ctrl-C. trying it again worked fine though.

Not good. I think xend is getting more robust over time.

> halting the new domain from the console causes it to disappear from 'xm list' correctly, but it won't restart properly, and then 'xm destroy, won't remove it from 'xm list'.

So: you started a domain, halted it, and it correctly
disappeared. You then issued the same xm create command to
recreate it and it failed to come up.

Odd. Any error output from xend or xm?

> lastly, is there a plan to be able to export a filesystem rather than just block devices, like 'hostfs' under uml? It would make copying modules over a bit easier!

How about using NFS for the moment?

It would be nice to add a super-fast page flipping filesystem
interface, but its on the todo list.

Ian



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Niraj Tolia]

> I suspect adding devfs support wouldn't be too hard, but its
> probably not high on the list of priorities for us. Do any
> common distros use it by default?
> 
> > having booted my new domain, 'xm list' in dom0 hung for ages with 100% cpu time until I hit ctrl-C. trying it again worked fine though.
> 
> Not good. I think xend is getting more robust over time.
> 

I have seen a number of ''heisenbugs" with xend. They especially occur
when I stop xend and start it again. Every now and then a 'xend start'
does nothing and sends no output to console. Commands such as 'xm
list' will give a 'Connection refused' error. A 'xend stop' followed
by a 'xend start' usually fixes this but there have been a few times
where I have had to repeat the cycle a couple of times.

Also, sometimes after a reboot, a domain creation just fails and the
'xm create' command does not return. There are no error messages from
xend. Even though I suspect that things are hosed at this point in
time, a 'xend stop' works. However xend will not startup again and
gives a 'IOError: [Errno 16] Device or resource busy' error. The
problem cannot be reproduced as the  same create command works after
another reboot.

Niraj


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Ian Pratt]

> > I suspect adding devfs support wouldn't be too hard, but its
> > probably not high on the list of priorities for us. Do any
> > common distros use it by default?
> > 
> > > having booted my new domain, 'xm list' in dom0 hung for ages with 100% cpu time until I hit ctrl-C. trying it again worked fine though.
> > 
> > Not good. I think xend is getting more robust over time.
> > 
> 
> I have seen a number of ''heisenbugs" with xend. They especially occur
> when I stop xend and start it again. Every now and then a 'xend start'
> does nothing and sends no output to console. Commands such as 'xm
> list' will give a 'Connection refused' error. A 'xend stop' followed
> by a 'xend start' usually fixes this but there have been a few times
> where I have had to repeat the cycle a couple of times.

Could you try running xend with "xend start_trace". It might give
a clue as to what is going wrong if you compare traces.

It would be nice to get this nailed.

Thanks,
Ian

> Also, sometimes after a reboot, a domain creation just fails and the
> 'xm create' command does not return. There are no error messages from
> xend. Even though I suspect that things are hosed at this point in
> time, a 'xend stop' works. However xend will not startup again and
> gives a 'IOError: [Errno 16] Device or resource busy' error. The
> problem cannot be reproduced as the  same create command works after
> another reboot.
> 
> Niraj
> 
> 
> -------------------------------------------------------
> This SF.Net email sponsored by Black Hat Briefings & Training.
> Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
> digital self defense, top technical experts, no vendor pitches, 
> unmatched networking opportunities. Visit www.blackhat.com
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

RE: Re: almost working

[James Harper]

[-- Attachment #1: Type: text/plain, Size: 2757 bytes --]

If devfs is indeed disappearing in 2.6 then there's probably not a lot of point adding support for it. Debian can use it, but I don't think it's enabled by default. I like the way devfs works but obviously most people don't.

What are the meanings of 'state'? I see dom0 as 'r----', and dom1 (when it works) appears as '-b---', and when it doesn't it appears with a 'p' somewhere in there. I would have thought that 'b' means booting, and it should then progress to 'r' meaning running.

I was thinking about a hostfs option for automatically copying in the modules in at boot time. My network configuration doesn't actually enable networking until it has started the firewall, which it can't because the modules aren't loaded. Being on the todo list sounds promising!

James




From: Ian Pratt
Sent: Sat 3/07/2004 1:21 AM
To: James Harper
Cc: Ian Pratt; Keir Fraser; xen-devel@lists.sourceforge.net; Ian.Pratt@cl.cam.ac.uk
Subject: [Xen-devel] Re: almost working


> I am once again able to boot a xenU kernel after removing all traces of devfs. *yay*

Excellent. Success stories are good to hear.

> can't say for sure if it was devfs though as i did a bk pull just before I sent my last email...

I suspect adding devfs support wouldn't be too hard, but its
probably not high on the list of priorities for us. Do any
common distros use it by default?

> having booted my new domain, 'xm list' in dom0 hung for ages with 100% cpu time until I hit ctrl-C. trying it again worked fine though.

Not good. I think xend is getting more robust over time.

> halting the new domain from the console causes it to disappear from 'xm list' correctly, but it won't restart properly, and then 'xm destroy, won't remove it from 'xm list'.

So: you started a domain, halted it, and it correctly
disappeared. You then issued the same xm create command to
recreate it and it failed to come up.

Odd. Any error output from xend or xm?

> lastly, is there a plan to be able to export a filesystem rather than just block devices, like 'hostfs' under uml? It would make copying modules over a bit easier!

How about using NFS for the moment?

It would be nice to add a super-fast page flipping filesystem
interface, but its on the todo list.

Ian



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

[-- Attachment #2: Type: text/html, Size: 3506 bytes --]

Re: Re: almost working

[Keir Fraser]

> What are the meanings of 'state'? I see dom0 as 'r----', and dom1 (when it works) appears as '-b---', and when it doesn't it appears with a 'p' somewhere in there. I would have thought that 'b' mean
s booting, and it should then progress to 'r' meaning running.

I have some better state-output code that I'll check in for the 'xm'
tool. Currently, the flag array is 'rbpsc', where:
 r: Running
 b: Blocked (e.g., waiting for I/O)
 p: Paused DOM0 by control software
 s: Domain has shut itself down (probably awaiting cleanup by xend)
 c: Domain crashed inside Xen -- shutdown pending postmortem by xend

I have code that actually gives a useful phrase describing the state
of the domain in plain English, which might be more useful. :-)

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[xen]

On Sat, Jul 03, 2004 at 08:24:54PM +1000, James Harper wrote:
> If devfs is indeed disappearing in 2.6 then there's probably not a lot of point adding support for it. Debian can use it, but I don't think it's enabled by default. I like the way devfs works but obviously most people don't.

Just a note, I believe Andrew Morton confirmed that devfs definitely
won't be disappearing in 2.6. The earliest it would be removed is 2.7,
and even that's not certain.


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Jody Belka]

[-- Attachment #1: Type: text/plain, Size: 481 bytes --]

On Fri, Jul 02, 2004 at 04:21:32PM +0100, Ian Pratt wrote:
> So: you started a domain, halted it, and it correctly
> disappeared. You then issued the same xm create command to
> recreate it and it failed to come up.
> 
> Odd. Any error output from xend or xm?

Having just upgraded to r1056 i'm getting this now myself.
I'm including screenlogs from the xend tracing, and the
operations that caused it. It happens every time, without
fail.

-- 
Jody Belka
knew (at) pimb (dot) org

[-- Attachment #2: screenlog: commands --]
[-- Type: text/plain, Size: 10691 bytes --]

^[]0;root@iris: /home/jmb\aroot@iris:~# sudo reboot\b\b\b\b\b\b\b\b\b\b\b^[[5Preboot\b\b\b\b\b\b^[[4Pls\b\brm screenlog.1\b0\b\b\b\b\b\b\b\b\b\b\b\b\b\bls^[[K\b\bcat screenlog.1\rroot@iris:~# ^[[1@less screenlog.1\b0\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bs^[[K\b\bscreen -L sudo bash\rroot@iris:~# ^[[3Pxend trace_start\b\b\b\b\b\b\b\b\b\b\b^[[7Pstop\b\b\b\b\b\b\b\b\brm trace \b\b\b\b\b\b\b\b\bxm create -f eos -c\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[7Pdestroy 3\b\b\b\b\b\b\b\b\b^[[5Plist\b\b\b\bcreate -f eos -c\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b^[[7Pdestroy 2\b\b\b\b\b\b\b\b\b^[[5Plist\b\b\b\bcreate -f eos -c\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\blist^[[K\b\b\b\bcreate -f eos -c\rroot@iris:~# ^[[7@screen xm create -f eos -c\rroot@iris:~# xm list^[[K\b\b\b\b\b\b\bcd /etc/xen/machines/\rroot@iris:~# xend trace_start &> trace; tail -f trace\rroot@iris:~# ^[[19Pcd /etc/xen/machines/
^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# xm list
Dom  Name             Mem(MB)  CPU  State  Time(s)
0    Domain-0              55    0  r----     40.3
^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# xm cre\aate -f xe\b \b\b \be\aos -c
Started domain 1, console on port 9601
************ REMOTE CONSOLE: CTRL-] TO QUIT ********
Linux version 2.4.26-xenU (jmb@iris) (gcc version 2.95.4 20011002 (Debian prerelease)) #7 Sun Jul 4 01:59:27 CEST 2004
\rOn node 0 totalpages: 4096
\rzone(0): 4096 pages.
\rzone(1): 0 pages.
\rzone(2): 0 pages.
\rKernel command line:  ip=:::::eth0:off root=/dev/hda1 ro
\rInitializing CPU#0
\rXen reported: 1300.100 MHz processor.
\rCalibrating delay loop... 1287.78 BogoMIPS
\rMemory: 14524k/16384k available (1059k kernel code, 1860k reserved, 287k data, 44k init, 0k highmem)
\rDentry cache hash table entries: 2048 (order: 2, 16384 bytes)
\rInode cache hash table entries: 1024 (order: 1, 8192 bytes)
\rMount cache hash table entries: 512 (order: 0, 4096 bytes)
\rBuffer cache hash table entries: 1024 (order: 0, 4096 bytes)
\rPage-cache hash table entries: 4096 (order: 2, 16384 bytes)
\rCPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
\rCPU: L2 Cache: 64K (64 bytes/line)
\rCPU: AMD Duron(tm) processor stepping 01
\rPOSIX conformance testing by UNIFIX
\rLinux NET4.0 for Linux 2.4
\rBased upon Swansea University Computer Society NET3.039
\rInitializing RT netlink socket
\rStarting kswapd
\rJournalled Block Device driver loaded
\rEvent-channel device installed.
\rXen virtual console successfully installed as tty
\rStarting Xen Balloon driver
\rpty: 256 Unix98 ptys configured
\rInitialising Xen virtual block device
\rRAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
\rloop: loaded (max 8 devices)
\rInitialising Xen virtual ethernet frontend driver<6>NET4: Linux TCP/IP 1.0 for NET4.0
\rIP Protocols: ICMP, UDP, TCP
\rIP: routing cache hash table of 512 buckets, 4Kbytes
\rTCP: Hash tables configured (established 1024 bind 1024)
\rip_conntrack version 2.1 (128 buckets, 1024 max) - 288 bytes per conntrack
\rip_tables: (C) 2000-2002 Netfilter core team
\ript_recent v0.3.1: Stephen Frost <sfrost@snowman.net>.  http://snowman.net/projects/ipt_recent/
\rarp_tables: (C) 2002 David S. Miller
\rNET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
\rkjournald starting.  Commit interval 5 seconds
\rEXT3-fs: mounted filesystem with ordered data mode.
\rVFS: Mounted root (ext3 filesystem) readonly.
\rFreeing unused kernel memory: 44k freed
\rmodprobe: modprobe: cannot create /var/log/ksymoops/20040704.log Read-only file system
modprobe: modprobe: Can't locate module char-major-4
modprobe: modprobe: cannot create /var/log/ksymoops/20040704.log Read-only file system
\rINIT: version 2.84 booting
Activating swap.
Adding Swap: 16372k swap-space (priority -1)
\rChecking root file system...
fsck 1.27 (8-Mar-2002)
eos-root: clean, 18206/53248 files, 172868/212991 blocks
EXT3 FS 2.4-0.9.19, 19 August 2002 on hd(3,1), internal journal
\rSystem time was Sun Jul  4 14:21:41 UTC 2004.
Setting the System Clock using the Hardware Clock as reference...
modprobe: modprobe: Can't locate module char-major-10-135
hwclock is unable to get I/O port access:  the iopl(3) call failed.
modprobe: modprobe: Can't locate module char-major-10-135
System Clock set. System local time is now Sun Jul  4 14:21:41 UTC 2004.
Calculating module dependencies... done.
Loading modules: 
Checking all file systems...
fsck 1.27 (8-Mar-2002)
eos-dnsdata: clean, 11/4096 files, 1564/16383 blocks
Setting kernel variables.
Mounting local filesystems...
kjournald starting.  Commit interval 5 seconds
\rEXT3 FS 2.4-0.9.19, 19 August 2002 on hd(3,65), internal journal
\rEXT3-fs: mounted filesystem with ordered data mode.
\r/dev/hdb1 on /etc/dns type ext3 (rw)
Cleaning: /etc/network/ifstate.
Setting up IP spoofing protection: rp_filter.
Configuring network interfaces: done.

Setting the System Clock using the Hardware Clock as reference...
modprobe: modprobe: Can't locate module char-major-10-135
hwclock is unable to get I/O port access:  the iopl(3) call failed.
System Clock set. Local time: Sun Jul  4 16:21:42 CEST 2004

Cleaning: /tmp /var/lock /var/run.
Initializing random number generator... done.
\rINIT: Entering runlevel: 2
Starting system log daemon: syslogd.
Starting kernel log daemon: klogd.
Starting OpenBSD Secure Shell server: sshd.
Starting periodic command scheduler: cron.

Debian GNU/Linux 3.0 eos tty1

eos login: root
Password: 
Last login: Sun Jul  4 16:08:36 2004 on tty1
Linux eos 2.4.26-xenU #7 Sun Jul 4 01:59:27 CEST 2004 i686 unknown

Most of the programs included with the Debian GNU/Linux system are
freely redistributable; the exact distribution terms for each program
are described in the individual files in /usr/share/doc/*/copyright

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
eos:~# halt
\a
Broadcast message from root (tty1) (Sun Jul  4 16:23:36 2004):

\rThe system is going down for system halt NOW!\r\rINIT: Sending processes the TERM signal
eos:~# \rINIT: Sending processes the KILL signal
Stopping periodic command scheduler: cron.
Stopping OpenBSD Secure Shell server: sshd.
Saving the System Clock time to the Hardware Clock...
hwclock is unable to get I/O port access:  the iopl(3) call failed.
Hardware Clock updated to Sun Jul  4 16:23:42 CEST 2004.
Stopping kernel log daemon: klogd.
Stopping system log daemon: syslogd.
Sending all processes the TERM signal... done.
Sending all processes the KILL signal... done.
Saving random seed... done.
Unmounting remote filesystems... done.
Deconfiguring network interfaces: done.
Deactivating swap... done.
Unmounting local filesystems... done.
Power down.

************ REMOTE CONSOLE EXITED *****************
^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# xm list
Dom  Name             Mem(MB)  CPU  State  Time(s)
0    Domain-0              55    0  r----     45.9
^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# xm list\b\b\b\bcreate -f eos -c
Traceback (most recent call last):
  File "/usr/sbin/xm", line 6, in ?
    main.main(sys.argv)
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 448, in main
    xm.main(args)
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 79, in main
    p.main(args[1:])
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 153, in main
    create.main(args)
  File "/usr/lib/python2.2/site-packages/xen/xm/create.py", line 365, in main
    (d, c) = make_domain(opts, config)
  File "/usr/lib/python2.2/site-packages/xen/xm/create.py", line 334, in make_domain
    dominfo = server.xend_domain_create(config)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 184, in xend_domain_create
    {'op'      : 'create',
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 128, in xend_call
    return xend_request(url, "POST", data)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 102, in xend_request
    resp = conn.getresponse()
  File "/usr/lib/python2.2/httplib.py", line 595, in getresponse
    response.begin()
  File "/usr/lib/python2.2/httplib.py", line 119, in begin
    line = self.fp.readline()
KeyboardInterrupt
^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# xm create -f eos -c\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\blist^[[K
Dom  Name             Mem(MB)  CPU  State  Time(s)
0    Domain-0              55    0  r----     46.7
2    eos - dns             16    0  --p--      0.0
^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# xm list\b \b\b \b\b \b\b \bdestroy 2
Traceback (most recent call last):
  File "/usr/sbin/xm", line 6, in ?
    main.main(sys.argv)
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 448, in main
    xm.main(args)
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 79, in main
    p.main(args[1:])
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 267, in main
    server.xend_domain_destroy(dom)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 204, in xend_domain_destroy
    {'op'      : 'destroy'})
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 128, in xend_call
    return xend_request(url, "POST", data)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 108, in xend_request
    raise RuntimeError(resp.reason)
RuntimeError: Internal Server Error
^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# xm destroy 2\b\b\b\b\b\b\b\b\b^[[5Plist\b\b\b\bcreate -f eos -c
Traceback (most recent call last):
  File "/usr/sbin/xm", line 6, in ?
    main.main(sys.argv)
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 448, in main
    xm.main(args)
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 79, in main
    p.main(args[1:])
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 153, in main
    create.main(args)
  File "/usr/lib/python2.2/site-packages/xen/xm/create.py", line 365, in main
    (d, c) = make_domain(opts, config)
  File "/usr/lib/python2.2/site-packages/xen/xm/create.py", line 334, in make_domain
    dominfo = server.xend_domain_create(config)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 184, in xend_domain_create
    {'op'      : 'create',
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 128, in xend_call
    return xend_request(url, "POST", data)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 108, in xend_request
    raise RuntimeError(resp.reason)
RuntimeError: Internal Server Error
^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# 

[-- Attachment #3: screenlog: xend tracing --]
[-- Type: text/plain, Size: 14141 bytes --]

^[]0;root@iris: /home/jmb\aroot@iris:~# sudo reboot\b\b\b\b\b\b\b\b\b\b\b^[[5Preboot\b\b\b\b\b\b^[[4Pls\b\brm screenlog.1\b0\b\b\b\b\b\b\b\b\b\b\b\b\b\bls^[[K\b\bcat screenlog.1\rroot@iris:~# ^[[1@less screenlog.1\b0\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bs^[[K\b\bscreen -L sudo bash\rroot@iris:~# ^[[3Pxend trace_start\b\b\b\b\b\b\b\b\b\b\b^[[7Pstop\b\b\b\b\b\b\b\b\brm trace \b\b\b\b\b\b\b\b\bxend stop
^[]0;root@iris: /home/jmb\aroot@iris:~# xend stop\b\b\b\b\b\b\b\b\bsudo reboot\b\b\b\b\b\b\b\b\b\b\b^[[5Preboot\b\b\b\b\b\b^[[4Pls\b\brm screenlog.1\b0\b\b\b\b\b\b\b\b\b\b\b\b\b\bls^[[K\b\bcat screenlog.1\rroot@iris:~# ^[[1@less screenlog.1\b0\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bs^[[K\b\bscreen -L sudo bash\rroot@iris:~# ^[[3Pxend trace_start\b\b\b\b\b\b\b\b\b\b\b^[[7Pstop\b\b\b\btrace_start
^[]0;root@iris: /home/jmb\aroot@iris:~# brctl addbr nbe-br
device nbe-br already exists; can't create bridge with the same name
brctl sethello nbe-br 0
brctl setfd nbe-br 0
brctl stp nbe-br off
Interface not found: eth0
initial_refresh>
db dom= ['domain', ['id', '0'], ['name', 'Domain-0'], ['memory', '55']]
xc dom= {'cpu_time': 39919403308L, 'name': 'Domain-0', 'dying': 0, 'dom': 0, 'maxmem_kb': 65536, 'paused': 0, 'crashed': 0, 'running': 1, 'shutdown_reason': 0, 'shutdown': 0, 'mem_kb': 56320, 'cpu': 0, 'blocked': 0}
dom= 0 config= ['domain', ['id', '0'], ['name', 'Domain-0'], ['memory', '55']]
dom= 0 new
XendDomain>reap>
XendDomain>reap<
XendDomain>initial_refresh> doms:
dom domain id=0 name=Domain-0 memory=55
EVENT> xend.start 0
00\b \b\b \b\a\a0\b \bXendDomain>reap>
XendDomain>reap<
config: (vm (name 'eos - dns') (memory '16') (cpu '-1') (image (linux (kernel /boot/xen/xenU-2.4.26) (ip :::::eth0:off) (root '/dev/hda1 ro'))) (device (vbd (uname phy:/dev/evms/machines/eos/root) (dev hda1) (mode w))) (device (vbd (uname phy:/dev/evms/machines/eos/swap) (dev hda2) (mode w))) (device (vbd (uname phy:/dev/evms/machines/eos/dnsdata) (dev hdb1) (mode w))) (device (vif)))
vm_create>
create_domain> linux /boot/xen/xenU-2.4.26
create-domain> init_domain...
init_domain> 16 eos - dns -1
create_domain> dom= 1
build_domain> linux 1 /boot/xen/xenU-2.4.26  ip=:::::eth0:off root=/dev/hda1 ro 
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0000000->c0198a58
 Init. ramdisk: c0199000->c0199000
 Phys-Mach map: c0199000->c019d000
 Page tables:   c019d000->c019f000
 Start info:    c019f000->c01a0000
 Boot stack:    c01a0000->c01a1000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0000000
vm_create<
EVENT> xend.console.create [12, 1, 9601]
>create_devices
blkif_dev_create> 1 769 w {'device': 65034, 'type': 'Disk', 'start_sector': 0L, 'nr_sectors': 425982L}
blkif_dev_create> 1 770 w {'device': 65036, 'type': 'Disk', 'start_sector': 0L, 'nr_sectors': 32766L}
<create_devices
_vm_configure1> made devices...
_vm_configure1<
brctl addif nbe-br vif1.0
vm_dev_vif> created <xen.xend.server.netif.NetDev instance at 0x84e4664>
blkif_dev_create> 1 833 w {'device': 65032, 'type': 'Disk', 'start_sector': 0L, 'nr_sectors': 32766L}
_vm_configure1> cbok [(1, <xen.xend.server.blkif.BlkifController instance at 0x84df5c4>), (1, 0), (1, 0), (1, 0)]
>callback _vm_configure2...
_vm_configure2> cbok []
<_vm_configure2
vm_create> cbok domain id=1 name=eos - dns memory=16 console=12 image=/boot/xen/xenU-2.4.26
EVENT> xend.domain.created 1
EVENT> xend.domain.unpause 1
EVENT> xend.console.connect [12, '127.0.0.1', 1028]
responseReceived> No handler: Message type blkif_fe_interface_status_changed_t 2:0 <xen.xend.server.blkif.BlkifController instance at 0x84df5c4>
recv_fe_interface_connect>
(blkif (dom '1') (evtchn '13' '4'))
responseReceived> No handler: Message type blkif_fe_interface_status_changed_t 2:0 <xen.xend.server.blkif.BlkifController instance at 0x84df5c4>
responseReceived> No handler: Message type netif_fe_interface_status_changed_t 4:0 <xen.xend.server.netif.NetifController instance at 0x84e59a4>
responseReceived> No handler: Message type netif_fe_interface_status_changed_t 4:0 <xen.xend.server.netif.NetifController instance at 0x84e59a4>
VirqClient.virqReceived> 4
XendDomain> virq 4
XendDomain>reap>
XendDomain>reap> died id= 1 {'cpu_time': 1301179463L, 'name': 'eos - dns', 'dying': 0, 'dom': 1, 'maxmem_kb': 16384, 'paused': 0, 'crashed': 0, 'running': 0, 'shutdown_reason': 0, 'shutdown': 1, 'mem_kb': 15604, 'cpu': 0, 'blocked': 0}
cleanup> 1
release_devices> 1
release_vifs> 1
NetifController>destroy> dom= 1
NetDev>destroy> vif= 0
(netdev (vif 0) (mac aa:0:0:51:9d:7) (evtchn '14' '5'))
brctl delif nbe-br vif1.0
release_vbds> 1
BlkifController>destroy> dom= 1
>BlkifController>send_be_disconnect> dom= 1
BlkifController>lostChannel> dom= 1
NetifController>lostChannel> dom= 1
XendDomain>reap<
EVENT> xend.virq 4
EVENT> xend.domain.destroy 1
responseReceived> No handler: Message type netif_be_disconnect_t 3:3 <xen.xend.server.netif.NetifControllerFactory instance at 0x84477ec>
responseReceived> No handler: Message type blkif_be_disconnect_t 1:3 <xen.xend.server.blkif.BlkifControllerFactory instance at 0x842db24>
EVENT> xend.console.disconnect [12, '127.0.0.1', 1028]
XendDomain>reap>
XendDomain>reap<
EVENT> xend.domain.died 1
config: (vm (name 'eos - dns') (memory '16') (cpu '-1') (image (linux (kernel /boot/xen/xenU-2.4.26) (ip :::::eth0:off) (root '/dev/hda1 ro'))) (device (vbd (uname phy:/dev/evms/machines/eos/root) (dev hda1) (mode w))) (device (vbd (uname phy:/dev/evms/machines/eos/swap) (dev hda2) (mode w))) (device (vbd (uname phy:/dev/evms/machines/eos/dnsdata) (dev hdb1) (mode w))) (device (vif)))
vm_create>
create_domain> linux /boot/xen/xenU-2.4.26
create-domain> init_domain...
init_domain> 16 eos - dns -1
create_domain> dom= 2
build_domain> linux 2 /boot/xen/xenU-2.4.26  ip=:::::eth0:off root=/dev/hda1 ro 
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0000000->c0198a58
 Init. ramdisk: c0199000->c0199000
 Phys-Mach map: c0199000->c019d000
 Page tables:   c019d000->c019f000
 Start info:    c019f000->c01a0000
 Boot stack:    c01a0000->c01a1000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0000000
vm_create<
EVENT> xend.console.create [15, 2, 9602]
Traceback (most recent call last):
  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 145, in doPoll
    log.callWithLogger(selectable, _drdw, selectable, fd, event, POLLIN, POLLOUT, log)
  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 65, in callWithLogger
    callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 52, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib/python2.2/site-packages/twisted/python/context.py", line 43, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 160, in _doReadOrWrite
    why = selectable.doRead()
  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 252, in doRead
    self.protocol.notificationReceived(notification)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 176, in notificationReceived
    channel.notificationReceived()
  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 132, in notificationReceived
    self.handleNotification()
  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 296, in handleNotification
    work += self.handleResponses()
  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 331, in handleResponses
    self.responseReceived(msg)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 339, in responseReceived
    dev.responseReceived(msg, ty, subty)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 30, in responseReceived
    method(msg, 0)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 93, in recv_be_create
    self.callDeferred(blkif)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 118, in callDeferred
    d.callback(*args)
  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
    self._startRunCallbacks(result)
  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 299, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError: 
Unhandled error in Deferred:
Failure: twisted.internet.defer.TimeoutError: Callback timed out
XendDomain>reap>
XendDomain>reap<
EVENT> xend.domain.created 2
cleanup> 2
release_devices> 2
release_vifs> 2
release_vbds> 2
BlkifController>destroy> dom= 2
>BlkifController>send_be_disconnect> dom= 2
BlkifController>lostChannel> dom= 2
Traceback (most recent call last):
  File "/usr/lib/python2.2/site-packages/twisted/protocols/basic.py", line 229, in dataReceived
    return self.rawDataReceived(data)
  File "/usr/lib/python2.2/site-packages/twisted/protocols/http.py", line 1015, in rawDataReceived
    self.allContentReceived()
  File "/usr/lib/python2.2/site-packages/twisted/protocols/http.py", line 1006, in allContentReceived
    req.requestReceived(command, path, version)
  File "/usr/lib/python2.2/site-packages/twisted/protocols/http.py", line 557, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/usr/lib/python2.2/site-packages/twisted/web/server.py", line 165, in process
    self.render(resrc)
  File "/usr/lib/python2.2/site-packages/twisted/web/server.py", line 172, in render
    body = resrc.render(self)
  File "/usr/lib/python2.2/site-packages/twisted/web/resource.py", line 201, in render
    return m(request)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDomain.py", line 155, in render_POST
    return self.perform(req)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvBase.py", line 98, in perform
    val = op_method(op, req)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDomain.py", line 36, in op_destroy
    val = self.xd.domain_destroy(self.dom.id)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomain.py", line 292, in domain_destroy
    val = dominfo.destroy()
  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 518, in destroy
    return self.destroy_domain()
  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 528, in destroy_domain
    chan.close()
  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 243, in close
    self.factory.channelClosed(self)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 92, in channelClosed
    self.delChannel(channel.idx)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 46, in delChannel
    self.notifier.unbind(idx)
exceptions.IOError: [Errno 14] Bad address
EVENT> xend.domain.destroy 2
config: (vm (name 'eos - dns') (memory '16') (cpu '-1') (image (linux (kernel /boot/xen/xenU-2.4.26) (ip :::::eth0:off) (root '/dev/hda1 ro'))) (device (vbd (uname phy:/dev/evms/machines/eos/root) (dev hda1) (mode w))) (device (vbd (uname phy:/dev/evms/machines/eos/swap) (dev hda2) (mode w))) (device (vbd (uname phy:/dev/evms/machines/eos/dnsdata) (dev hdb1) (mode w))) (device (vif)))
vm_create>
create_domain> linux /boot/xen/xenU-2.4.26
create-domain> init_domain...
init_domain> 16 eos - dns -1
create_domain> dom= 3
cleanup> 3
release_devices> 3
release_vifs> 3
release_vbds> 3
Traceback (most recent call last):
  File "/usr/lib/python2.2/site-packages/twisted/protocols/basic.py", line 229, in dataReceived
    return self.rawDataReceived(data)
  File "/usr/lib/python2.2/site-packages/twisted/protocols/http.py", line 1015, in rawDataReceived
    self.allContentReceived()
  File "/usr/lib/python2.2/site-packages/twisted/protocols/http.py", line 1006, in allContentReceived
    req.requestReceived(command, path, version)
  File "/usr/lib/python2.2/site-packages/twisted/protocols/http.py", line 557, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/usr/lib/python2.2/site-packages/twisted/web/server.py", line 165, in process
    self.render(resrc)
  File "/usr/lib/python2.2/site-packages/twisted/web/server.py", line 172, in render
    body = resrc.render(self)
  File "/usr/lib/python2.2/site-packages/twisted/web/resource.py", line 201, in render
    return m(request)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDomainDir.py", line 99, in render_POST
    return self.perform(req)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvBase.py", line 98, in perform
    val = op_method(op, req)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDomainDir.py", line 59, in op_create
    deferred = self.xd.domain_create(config)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomain.py", line 231, in domain_create
    deferred = XendDomainInfo.vm_create(config)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 275, in vm_create
    return vm.construct(config)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 445, in construct
    self.destroy()
  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 518, in destroy
    return self.destroy_domain()
  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 528, in destroy_domain
    chan.close()
  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 243, in close
    self.factory.channelClosed(self)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 92, in channelClosed
    self.delChannel(channel.idx)
  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 46, in delChannel
    self.notifier.unbind(idx)
exceptions.IOError: [Errno 14] Bad address

^[]0;root@iris: /home/jmb\aroot@iris:~# exit

RE: Re: almost working

[ron minnich]

On Sat, 3 Jul 2004, James Harper wrote:

> If devfs is indeed disappearing in 2.6 then there's probably not a lot
> of point adding support for it. Debian can use it, but I don't think
> it's enabled by default. I like the way devfs works but obviously most
> people don't.

yes, on point in fooling with devfs. It's dead.

ron



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Keir Fraser]

> > Not good. I think xend is getting more robust over time.
> 
> I have seen a number of ''heisenbugs" with xend. They especially occur
> when I stop xend and start it again. Every now and then a 'xend start'
> does nothing and sends no output to console. Commands such as 'xm
> list' will give a 'Connection refused' error. A 'xend stop' followed
> by a 'xend start' usually fixes this but there have been a few times
> where I have had to repeat the cycle a couple of times.
> 
> Also, sometimes after a reboot, a domain creation just fails and the
> 'xm create' command does not return. There are no error messages from
> xend. Even though I suspect that things are hosed at this point in
> time, a 'xend stop' works. However xend will not startup again and
> gives a 'IOError: [Errno 16] Device or resource busy' error. The
> problem cannot be reproduced as the  same create command works after
> another reboot.

I think many people had been seeing problems of this ilk, so you will
probably be delighted to hear that these bugs have been squashed by
today's checkins.

It was a combination of a simple stupid bug in the device-driver code,
coupled with faulty error handling in xend.

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Jody Belka]

[-- Attachment #1: Type: text/plain, Size: 794 bytes --]

On Tue, Jul 06, 2004 at 05:11:04PM +0100, Keir Fraser wrote:
> I think many people had been seeing problems of this ilk, so you will
> probably be delighted to hear that these bugs have been squashed by
> today's checkins.
> 
> It was a combination of a simple stupid bug in the device-driver code,
> coupled with faulty error handling in xend.

Seeing this msg i cloned down the latest (at the time) rev from bk. After
rebuilding xen, my dom0 and domU kernels, and all the tools, I tried it
out. Unfortunately i couldn't even start an unpriveleged domain any more.
Logs of xend tracing and the relevant command window are attached; hope
they're helpful in tracking down the problem.

For reference, this was using rev 40eae953qXVB0VAXac-t2uDXq-tS9Q


-- 
Jody Belka
(knew) at (pimb) dot (org)

[-- Attachment #2: screenlog.0 --]
[-- Type: text/plain, Size: 31379 bytes --]

^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# /etc/init.d/xend-\b \b.real start_trace
/usr/sbin/xend
Starting Xen control daemon: xend.
^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# brctl addbr nbe-br
device nbe-br already exists; can't create bridge with the same name
brctl sethello nbe-br 0
brctl setfd nbe-br 0
brctl stp nbe-br off
Interface not found: eth0
initial_refresh>
db dom= ['domain', ['id', '0'], ['name', 'Domain-0'], ['memory', '39']]
xc dom= {'cpu_time': 76671513806L, 'name': 'Domain-0', 'dying': 0, 'dom': 0, 'maxmem_kb': 49152, 'paused': 0, 'crashed': 0, 'running': 1, 'shutdown_reason': 0, 'shutdown': 0, 'mem_kb': 39936, 'cpu': 0, 'blocked': 0}
dom= 0 config= ['domain', ['id', '0'], ['name', 'Domain-0'], ['memory', '39']]
dom= 0 new
XendDomain>refresh>
XendDomain>initial_refresh> doms:
dom domain id=0 name=Domain-0 memory=39
EVENT> xend.start 0
XendDomain>reap>
XendDomain>reap<
config: ('vm' ('name' 'eos - dns' ) ('memory' '16' ) ('cpu' '-1' ) ('image' ('linux' ('kernel' '/boot/xen/xenU-2.4.26-1' ) ('ip' ':::::eth0:off' ) ('root' '/dev/hda1 ro' ) ) ) ('device' ('vbd' ('uname' 'phy:/dev/evms/machines/eos/root' ) ('dev' 'hda1' ) ('mode' 'w' ) ) ) ('device' ('vbd' ('uname' 'phy:/dev/evms/machines/eos/swap' ) ('dev' 'hda2' ) ('mode' 'w' ) ) ) ('device' ('vbd' ('uname' 'phy:/dev/evms/machines/eos/dnsdata' ) ('dev' 'hdb1' ) ('mode' 'w' ) ) ) ('device' ('vif' ) ) )
vm_create>
create_domain> linux /boot/xen/xenU-2.4.26-1
create-domain> init_domain...
init_domain> 16 eos - dns -1
create_domain> dom= 3
build_domain> linux 3 /boot/xen/xenU-2.4.26-1  ip=:::::eth0:off root=/dev/hda1 ro 
VIRTUAL MEMORY ARRANGEMENT:
 Loaded kernel: c0000000->c0198a58
 Init. ramdisk: c0199000->c0199000
 Phys-Mach map: c0199000->c019d000
 Page tables:   c019d000->c019f000
 Start info:    c019f000->c01a0000
 Boot stack:    c01a0000->c01a1000
 TOTAL:         c0000000->c0400000
 ENTRY ADDRESS: c0000000
vm_create<
EVENT> xend.console.create [12, 3, 9603]
respond_be_create>
>create_devices
blkif_dev_create> 3 769 w {'device': 65034, 'type': 'Disk', 'start_sector': 0L, 'nr_sectors': 425982L}
blkif_dev_create> 3 770 w {'device': 65036, 'type': 'Disk', 'start_sector': 0L, 'nr_sectors': 65534L}
blkif_dev_create> 3 833 w {'device': 65032, 'type': 'Disk', 'start_sector': 0L, 'nr_sectors': 32766L}
<create_devices
_vm_configure1> made devices...
_vm_configure1<
recv_be_vbd_create> <xen.xend.server.blkif.BlkifControllerFactory instance at 0x8172e4c>
brctl addif nbe-br vif3.0
vm_dev_vif> created <xen.xend.server.netif.NetDev instance at 0x8547074>
cleanup> 3
release_devices> 3
release_vifs> 3
brctl delif nbe-br vif3.0
release_vbds> 3
>BlkifController>send_be_disconnect> dom= 3
op_create> Deferred Exception creating domain: [Failure instance: Traceback: <type 'tuple'>, (<twisted.python.failure.Failure twisted.internet.defer.TimeoutError>, 1)
]
Unhandled error in Deferred:
(debug:  C: Deferred was created:
 C:  File "/usr/sbin/xend", line 45, in ?
 C:    sys.exit(main())
 C:  File "/usr/sbin/xend", line 35, in main
 C:    return daemon.start(trace=1)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 521, in start
 C:    self.run()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 633, in run
 C:    reactor.run()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 126, in run
 C:    self.mainLoop()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 137, in mainLoop
 C:    self.doIteration(t)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 145, in doPoll
 C:    log.callWithLogger(selectable, _drdw, selectable, fd, event, POLLIN, POLLOUT, log)
 C:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 65, in callWithLogger
 C:    callWithContext({"system": lp}, func, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 52, in callWithContext
 C:    return context.call({ILogContext: newCtx}, func, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/twisted/python/context.py", line 43, in callWithContext
 C:    return func(*args,**kw)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 160, in _doReadOrWrite
 C:    why = selectable.doRead()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 253, in doRead
 C:    self.protocol.notificationReceived(notification)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 177, in notificationReceived
 C:    channel.notificationReceived()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 132, in notificationReceived
 C:    self.handleNotification()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 298, in handleNotification
 C:    work += self.handleResponses()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 335, in handleResponses
 C:    self.responseReceived(msg)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 343, in responseReceived
 C:    dev.responseReceived(msg, ty, subty)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 92, in responseReceived
 C:    if self.callResponders(msg):
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 140, in callResponders
 C:    resp.responseReceived(msg)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 31, in responseReceived
 C:    self.deferred.callback(msg)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 C:    self._startRunCallbacks(result)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 C:    self._runCallbacks()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 C:    self.result = callback(self.result, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 124, in respond_be_create
 C:    d.callback(blkif)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 C:    self._startRunCallbacks(result)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 C:    self._runCallbacks()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 C:    self.result = callback(self.result, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 331, in _vm_configure1
 C:    d = vm.create_devices()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 678, in create_devices
 C:    v = dev_handler(self, dev, dev_index)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 854, in vm_dev_vbd
 C:    defer = make_disk(vm.dom, uname, dev, mode, vm.recreate)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 156, in make_disk
 C:    ctrl = xend.blkif_create(dom, recreate=recreate)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 691, in blkif_create
 C:    d = self.blkifCF.createInstance(dom, recreate=recreate)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 35, in createInstance
 C:    d = defer.Deferred()
 I: First Invoker was:
 I:  File "/usr/sbin/xend", line 45, in ?
 I:    sys.exit(main())
 I:  File "/usr/sbin/xend", line 35, in main
 I:    return daemon.start(trace=1)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 521, in start
 I:    self.run()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 633, in run
 I:    reactor.run()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 126, in run
 I:    self.mainLoop()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 137, in mainLoop
 I:    self.doIteration(t)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 145, in doPoll
 I:    log.callWithLogger(selectable, _drdw, selectable, fd, event, POLLIN, POLLOUT, log)
 I:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 65, in callWithLogger
 I:    callWithContext({"system": lp}, func, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 52, in callWithContext
 I:    return context.call({ILogContext: newCtx}, func, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/twisted/python/context.py", line 43, in callWithContext
 I:    return func(*args,**kw)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 160, in _doReadOrWrite
 I:    why = selectable.doRead()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 253, in doRead
 I:    self.protocol.notificationReceived(notification)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 177, in notificationReceived
 I:    channel.notificationReceived()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 132, in notificationReceived
 I:    self.handleNotification()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 298, in handleNotification
 I:    work += self.handleResponses()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 335, in handleResponses
 I:    self.responseReceived(msg)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 343, in responseReceived
 I:    dev.responseReceived(msg, ty, subty)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 92, in responseReceived
 I:    if self.callResponders(msg):
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 140, in callResponders
 I:    resp.responseReceived(msg)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 31, in responseReceived
 I:    self.deferred.callback(msg)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 I:    self._startRunCallbacks(result)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 I:    self._runCallbacks()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 I:    self.result = callback(self.result, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 124, in respond_be_create
 I:    d.callback(blkif)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 I:    self._startRunCallbacks(result)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 I:    self._runCallbacks()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 I:    self.result = callback(self.result, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 331, in _vm_configure1
 I:    d = vm.create_devices()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 678, in create_devices
 I:    v = dev_handler(self, dev, dev_index)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 854, in vm_dev_vbd
 I:    defer = make_disk(vm.dom, uname, dev, mode, vm.recreate)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 156, in make_disk
 I:    ctrl = xend.blkif_create(dom, recreate=recreate)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 691, in blkif_create
 I:    d = self.blkifCF.createInstance(dom, recreate=recreate)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 38, in createInstance
 I:    d.callback(blkif)
)
Failure: twisted.internet.defer.TimeoutError: Callback timed out
Unhandled error in Deferred:
(debug:  C: Deferred was created:
 C:  File "/usr/sbin/xend", line 45, in ?
 C:    sys.exit(main())
 C:  File "/usr/sbin/xend", line 35, in main
 C:    return daemon.start(trace=1)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 521, in start
 C:    self.run()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 633, in run
 C:    reactor.run()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 126, in run
 C:    self.mainLoop()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 137, in mainLoop
 C:    self.doIteration(t)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 145, in doPoll
 C:    log.callWithLogger(selectable, _drdw, selectable, fd, event, POLLIN, POLLOUT, log)
 C:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 65, in callWithLogger
 C:    callWithContext({"system": lp}, func, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 52, in callWithContext
 C:    return context.call({ILogContext: newCtx}, func, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/twisted/python/context.py", line 43, in callWithContext
 C:    return func(*args,**kw)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 160, in _doReadOrWrite
 C:    why = selectable.doRead()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 253, in doRead
 C:    self.protocol.notificationReceived(notification)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 177, in notificationReceived
 C:    channel.notificationReceived()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 132, in notificationReceived
 C:    self.handleNotification()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 298, in handleNotification
 C:    work += self.handleResponses()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 335, in handleResponses
 C:    self.responseReceived(msg)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 343, in responseReceived
 C:    dev.responseReceived(msg, ty, subty)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 92, in responseReceived
 C:    if self.callResponders(msg):
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 140, in callResponders
 C:    resp.responseReceived(msg)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 31, in responseReceived
 C:    self.deferred.callback(msg)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 C:    self._startRunCallbacks(result)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 C:    self._runCallbacks()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 C:    self.result = callback(self.result, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 124, in respond_be_create
 C:    d.callback(blkif)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 C:    self._startRunCallbacks(result)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 C:    self._runCallbacks()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 C:    self.result = callback(self.result, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 331, in _vm_configure1
 C:    d = vm.create_devices()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 678, in create_devices
 C:    v = dev_handler(self, dev, dev_index)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 854, in vm_dev_vbd
 C:    defer = make_disk(vm.dom, uname, dev, mode, vm.recreate)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 156, in make_disk
 C:    ctrl = xend.blkif_create(dom, recreate=recreate)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 691, in blkif_create
 C:    d = self.blkifCF.createInstance(dom, recreate=recreate)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 35, in createInstance
 C:    d = defer.Deferred()
 I: First Invoker was:
 I:  File "/usr/sbin/xend", line 45, in ?
 I:    sys.exit(main())
 I:  File "/usr/sbin/xend", line 35, in main
 I:    return daemon.start(trace=1)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 521, in start
 I:    self.run()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 633, in run
 I:    reactor.run()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 126, in run
 I:    self.mainLoop()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 137, in mainLoop
 I:    self.doIteration(t)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 145, in doPoll
 I:    log.callWithLogger(selectable, _drdw, selectable, fd, event, POLLIN, POLLOUT, log)
 I:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 65, in callWithLogger
 I:    callWithContext({"system": lp}, func, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 52, in callWithContext
 I:    return context.call({ILogContext: newCtx}, func, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/twisted/python/context.py", line 43, in callWithContext
 I:    return func(*args,**kw)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 160, in _doReadOrWrite
 I:    why = selectable.doRead()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 253, in doRead
 I:    self.protocol.notificationReceived(notification)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 177, in notificationReceived
 I:    channel.notificationReceived()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 132, in notificationReceived
 I:    self.handleNotification()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 298, in handleNotification
 I:    work += self.handleResponses()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 335, in handleResponses
 I:    self.responseReceived(msg)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 343, in responseReceived
 I:    dev.responseReceived(msg, ty, subty)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 92, in responseReceived
 I:    if self.callResponders(msg):
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 140, in callResponders
 I:    resp.responseReceived(msg)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 31, in responseReceived
 I:    self.deferred.callback(msg)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 I:    self._startRunCallbacks(result)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 I:    self._runCallbacks()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 I:    self.result = callback(self.result, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 124, in respond_be_create
 I:    d.callback(blkif)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 I:    self._startRunCallbacks(result)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 I:    self._runCallbacks()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 I:    self.result = callback(self.result, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 331, in _vm_configure1
 I:    d = vm.create_devices()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 678, in create_devices
 I:    v = dev_handler(self, dev, dev_index)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 854, in vm_dev_vbd
 I:    defer = make_disk(vm.dom, uname, dev, mode, vm.recreate)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 156, in make_disk
 I:    ctrl = xend.blkif_create(dom, recreate=recreate)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 691, in blkif_create
 I:    d = self.blkifCF.createInstance(dom, recreate=recreate)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 38, in createInstance
 I:    d.callback(blkif)
)
Failure: twisted.internet.defer.TimeoutError: Callback timed out
Unhandled error in Deferred:
(debug:  C: Deferred was created:
 C:  File "/usr/sbin/xend", line 45, in ?
 C:    sys.exit(main())
 C:  File "/usr/sbin/xend", line 35, in main
 C:    return daemon.start(trace=1)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 521, in start
 C:    self.run()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 633, in run
 C:    reactor.run()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 126, in run
 C:    self.mainLoop()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 137, in mainLoop
 C:    self.doIteration(t)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 145, in doPoll
 C:    log.callWithLogger(selectable, _drdw, selectable, fd, event, POLLIN, POLLOUT, log)
 C:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 65, in callWithLogger
 C:    callWithContext({"system": lp}, func, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 52, in callWithContext
 C:    return context.call({ILogContext: newCtx}, func, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/twisted/python/context.py", line 43, in callWithContext
 C:    return func(*args,**kw)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 160, in _doReadOrWrite
 C:    why = selectable.doRead()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 253, in doRead
 C:    self.protocol.notificationReceived(notification)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 177, in notificationReceived
 C:    channel.notificationReceived()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 132, in notificationReceived
 C:    self.handleNotification()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 298, in handleNotification
 C:    work += self.handleResponses()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 335, in handleResponses
 C:    self.responseReceived(msg)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 343, in responseReceived
 C:    dev.responseReceived(msg, ty, subty)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 92, in responseReceived
 C:    if self.callResponders(msg):
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 140, in callResponders
 C:    resp.responseReceived(msg)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 31, in responseReceived
 C:    self.deferred.callback(msg)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 C:    self._startRunCallbacks(result)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 C:    self._runCallbacks()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 C:    self.result = callback(self.result, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 124, in respond_be_create
 C:    d.callback(blkif)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 C:    self._startRunCallbacks(result)
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 C:    self._runCallbacks()
 C:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 C:    self.result = callback(self.result, *args, **kw)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 331, in _vm_configure1
 C:    d = vm.create_devices()
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 678, in create_devices
 C:    v = dev_handler(self, dev, dev_index)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 854, in vm_dev_vbd
 C:    defer = make_disk(vm.dom, uname, dev, mode, vm.recreate)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 156, in make_disk
 C:    ctrl = xend.blkif_create(dom, recreate=recreate)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 691, in blkif_create
 C:    d = self.blkifCF.createInstance(dom, recreate=recreate)
 C:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 35, in createInstance
 C:    d = defer.Deferred()
 I: First Invoker was:
 I:  File "/usr/sbin/xend", line 45, in ?
 I:    sys.exit(main())
 I:  File "/usr/sbin/xend", line 35, in main
 I:    return daemon.start(trace=1)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 521, in start
 I:    self.run()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 633, in run
 I:    reactor.run()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 126, in run
 I:    self.mainLoop()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 137, in mainLoop
 I:    self.doIteration(t)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 145, in doPoll
 I:    log.callWithLogger(selectable, _drdw, selectable, fd, event, POLLIN, POLLOUT, log)
 I:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 65, in callWithLogger
 I:    callWithContext({"system": lp}, func, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 52, in callWithContext
 I:    return context.call({ILogContext: newCtx}, func, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/twisted/python/context.py", line 43, in callWithContext
 I:    return func(*args,**kw)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/pollreactor.py", line 160, in _doReadOrWrite
 I:    why = selectable.doRead()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 253, in doRead
 I:    self.protocol.notificationReceived(notification)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 177, in notificationReceived
 I:    channel.notificationReceived()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 132, in notificationReceived
 I:    self.handleNotification()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 298, in handleNotification
 I:    work += self.handleResponses()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 335, in handleResponses
 I:    self.responseReceived(msg)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/channel.py", line 343, in responseReceived
 I:    dev.responseReceived(msg, ty, subty)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 92, in responseReceived
 I:    if self.callResponders(msg):
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 140, in callResponders
 I:    resp.responseReceived(msg)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/controller.py", line 31, in responseReceived
 I:    self.deferred.callback(msg)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 I:    self._startRunCallbacks(result)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 I:    self._runCallbacks()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 I:    self.result = callback(self.result, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 124, in respond_be_create
 I:    d.callback(blkif)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 252, in callback
 I:    self._startRunCallbacks(result)
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 313, in _startRunCallbacks
 I:    self._runCallbacks()
 I:  File "/usr/lib/python2.2/site-packages/twisted/internet/defer.py", line 338, in _runCallbacks
 I:    self.result = callback(self.result, *args, **kw)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 331, in _vm_configure1
 I:    d = vm.create_devices()
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 678, in create_devices
 I:    v = dev_handler(self, dev, dev_index)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 854, in vm_dev_vbd
 I:    defer = make_disk(vm.dom, uname, dev, mode, vm.recreate)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/XendDomainInfo.py", line 156, in make_disk
 I:    ctrl = xend.blkif_create(dom, recreate=recreate)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/SrvDaemon.py", line 691, in blkif_create
 I:    d = self.blkifCF.createInstance(dom, recreate=recreate)
 I:  File "/usr/lib/python2.2/site-packages/xen/xend/server/blkif.py", line 38, in createInstance
 I:    d.callback(blkif)
)
Failure: twisted.internet.defer.TimeoutError: Callback timed out
exit

[-- Attachment #3: screenlog.1 --]
[-- Type: text/plain, Size: 1346 bytes --]

^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# xm create -f eos
Traceback (most recent call last):
  File "/usr/sbin/xm", line 6, in ?
    main.main(sys.argv)
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 473, in main
    xm.main(args)
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 71, in main
    self.main_call(args)
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 88, in main_call
    p.main(args[1:])
  File "/usr/lib/python2.2/site-packages/xen/xm/main.py", line 162, in main
    create.main(args)
  File "/usr/lib/python2.2/site-packages/xen/xm/create.py", line 401, in main
    (d, c) = make_domain(opts, config)
  File "/usr/lib/python2.2/site-packages/xen/xm/create.py", line 362, in make_domain
    dominfo = server.xend_domain_create(config)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 214, in xend_domain_create
    {'op'      : 'create',
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 150, in xend_call
    return xend_request(url, "POST", data)
  File "/usr/lib/python2.2/site-packages/xen/xend/XendClient.py", line 121, in xend_request
    raise RuntimeError(resp.reason)
RuntimeError: Error creating domain
^[]0;root@iris: /etc/xen/machines\aroot@iris:/etc/xen/machines# 

Re: Re: almost working

[Ian Pratt]

> Seeing this msg i cloned down the latest (at the time) rev from bk. After
> rebuilding xen, my dom0 and domU kernels, and all the tools, I tried it
> out. Unfortunately i couldn't even start an unpriveleged domain any more.
> Logs of xend tracing and the relevant command window are attached; hope
> they're helpful in tracking down the problem.

Oops. 

I can confirm I get the same error. 

And it was all working so well earlier this afternoon...

It looks like a timeout talking to the blk device backend driver,
which is a bug that was believed fixed earlier today -- sorry
folks. 

Not withstanding this bug, the good news is that we're beginning
to get the first inklings of warm fuzzy feelings about new xend's
stability. I'm afraid a fix will have to wait until tomorrow,
though.

Ian


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Keir Fraser]

> 
> > Seeing this msg i cloned down the latest (at the time) rev from bk. After
> > rebuilding xen, my dom0 and domU kernels, and all the tools, I tried it
> > out. Unfortunately i couldn't even start an unpriveleged domain any more.
> > Logs of xend tracing and the relevant command window are attached; hope
> > they're helpful in tracking down the problem.
> 
> Oops. 
> 
> I can confirm I get the same error. 
> 
> And it was all working so well earlier this afternoon...
> 
> It looks like a timeout talking to the blk device backend driver,
> which is a bug that was believed fixed earlier today -- sorry
> folks. 
> 
> Not withstanding this bug, the good news is that we're beginning
> to get the first inklings of warm fuzzy feelings about new xend's
> stability. I'm afraid a fix will have to wait until tomorrow,
> though.

Somehow related to specifying more than one VBD when creating a new
guest. If you specify just one then it works. Hopefully not hard to
track down.

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

No module named xen.xm error

[Rune Johan Andresen]

Hi, after booting Xen i am trying to start some domains with the domain 
control tools.
When i type "xend start"

File "usr/bin/xend", line 23, in ?
	from xen.xend.server import SrvDaemon
ImportError: no module named xen.xend.server

I guess the files are installed in the wrong place, but the "make 
world" should to this, right? I have manually tried to move the files 
under /install/ to their right places. Whar can be the error?

Regards
Rune A 



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: No module named xen.xm error

[Tom Wilkie]

You should go to xeno.bk/tools/ and type make install.  That will 
ensure file go in their correct place.

Cheers

Tom

On 7 Jul 2004, at 15:56, Rune Johan Andresen wrote:

> Hi, after booting Xen i am trying to start some domains with the 
> domain control tools.
> When i type "xend start"
>
> File "usr/bin/xend", line 23, in ?
> 	from xen.xend.server import SrvDaemon
> ImportError: no module named xen.xend.server
>
> I guess the files are installed in the wrong place, but the "make 
> world" should to this, right? I have manually tried to move the files 
> under /install/ to their right places. Whar can be the error?
>
> Regards
> Rune A
>
>
> -------------------------------------------------------
> This SF.Net email sponsored by Black Hat Briefings & Training.
> Attend Black Hat Briefings & Training, Las Vegas July 24-29 - digital 
> self defense, top technical experts, no vendor pitches, unmatched 
> networking opportunities. Visit www.blackhat.com
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: No module named xen.xm error

[Ian Pratt]

> Hi, after booting Xen i am trying to start some domains with the domain 
> control tools.
> When i type "xend start"
> 
> File "usr/bin/xend", line 23, in ?
> 	from xen.xend.server import SrvDaemon
> ImportError: no module named xen.xend.server
> 
> I guess the files are installed in the wrong place, but the "make 
> world" should to this, right? I have manually tried to move the files 
> under /install/ to their right places. Whar can be the error?

Please try 'make world' followed by 'make install'.

Ian



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: No module named xen.xm error

[Rune Johan Andresen]

Thanks! I must have missed this in the howtos. Sorry, the information is
a little distributed :)  Do you have any howtos or docs on the new 
features for the domain control tools?


Rune

On Jul 7, 2004, at 5:06 PM, Tom Wilkie wrote:

> You should go to xeno.bk/tools/ and type make install.  That will 
> ensure file go in their correct place.
>
> Cheers
>
> Tom
>
> On 7 Jul 2004, at 15:56, Rune Johan Andresen wrote:
>
>> Hi, after booting Xen i am trying to start some domains with the 
>> domain control tools.
>> When i type "xend start"
>>
>> File "usr/bin/xend", line 23, in ?
>> 	from xen.xend.server import SrvDaemon
>> ImportError: no module named xen.xend.server
>>
>> I guess the files are installed in the wrong place, but the "make 
>> world" should to this, right? I have manually tried to move the files 
>> under /install/ to their right places. Whar can be the error?
>>
>> Regards
>> Rune A
>>
>>
>> -------------------------------------------------------
>> This SF.Net email sponsored by Black Hat Briefings & Training.
>> Attend Black Hat Briefings & Training, Las Vegas July 24-29 - digital 
>> self defense, top technical experts, no vendor pitches, unmatched 
>> networking opportunities. Visit www.blackhat.com
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/xen-devel
>>
>
>
>
> -------------------------------------------------------
> This SF.Net email sponsored by Black Hat Briefings & Training.
> Attend Black Hat Briefings & Training, Las Vegas July 24-29 - digital 
> self defense, top technical experts, no vendor pitches, unmatched 
> networking opportunities. Visit www.blackhat.com
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Jody Belka]

On Wed, Jul 07, 2004 at 08:10:10AM +0100, Keir Fraser wrote:
> > > Seeing this msg i cloned down the latest (at the time) rev from bk. After
> > > rebuilding xen, my dom0 and domU kernels, and all the tools, I tried it
> > > out. Unfortunately i couldn't even start an unpriveleged domain any more.
> > > Logs of xend tracing and the relevant command window are attached; hope
> > > they're helpful in tracking down the problem.
> 
> Somehow related to specifying more than one VBD when creating a new
> guest. If you specify just one then it works. Hopefully not hard to
> track down.

After spending 2 solid hours slogging away at this in the early hours
of this morning (or late late last night if you prefer), i tracked it
down. To my eyes it looks like it had the potential to muck about with
more than just VBD's, and that's just what was triggering it in this case.

Here's a patch for it:

--- tools/python/xen/xend/server/controller.py  Fri Jul  9 08:06:52 2004
+++ tools/python/xen/xend/server/controller.py.new      Fri Jul  9 08:06:24 2004
@@ -178,7 +178,7 @@
             resp = self.responders[0]
             if resp.mid > mid:
                 break
-            self.responders.pop()
+            self.responders.pop(0)
             if resp.mid < mid:
                 print 'callResponders> Out of order:', resp.mid, mid
                 resp.error(OutOfOrderError())


On a seperate matter, i always have to apply this patch to a new clone of
xen-unstable or stuff doesn't work. Am i the only one having this trouble,
and if so, any ideas why?

--- tools/python/xen/xend/XendDomainInfo.py     Fri Jul  9 08:10:38 2004
+++ tools/python/xen/xend/XendDomainInfo.py.new Fri Jul  9 08:10:18 2004
@@ -49,6 +49,20 @@
         lines.append(line)
     return lines
 
+def readline(fd):
+    """Version of readline safe against EINTR.
+    """
+    import errno
+    
+    while 1:
+        try:
+            return fd.readline()
+        except IOError, ex:
+            if ex.errno == errno.EINTR:
+                continue
+            else:
+                raise
+
 class VmError(ValueError):
     """Vm construction error."""
 

-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

RE: Re: almost working

[James Harper]

[-- Attachment #1: Type: text/plain, Size: 3246 bytes --]

Jody,

This fixed my problem where I couldn't start a domain with >1 virtual network interface. I for one think your time was well spent!!!

re: your second patch, exactly what doesn't work without it? Apart from reboots under high load I don't seem to have any other problems, although it looks like something that would show up randomly.

James




From: Jody Belka
Sent: Fri 9/07/2004 4:19 PM
To: Keir Fraser
Cc: Ian Pratt; xen-devel@lists.sourceforge.net
Subject: Re: [Xen-devel] Re: almost working


On Wed, Jul 07, 2004 at 08:10:10AM +0100, Keir Fraser wrote:
> > > Seeing this msg i cloned down the latest (at the time) rev from bk. After
> > > rebuilding xen, my dom0 and domU kernels, and all the tools, I tried it
> > > out. Unfortunately i couldn't even start an unpriveleged domain any more.
> > > Logs of xend tracing and the relevant command window are attached; hope
> > > they're helpful in tracking down the problem.
> 
> Somehow related to specifying more than one VBD when creating a new
> guest. If you specify just one then it works. Hopefully not hard to
> track down.

After spending 2 solid hours slogging away at this in the early hours
of this morning (or late late last night if you prefer), i tracked it
down. To my eyes it looks like it had the potential to muck about with
more than just VBD's, and that's just what was triggering it in this case.

Here's a patch for it:

--- tools/python/xen/xend/server/controller.py  Fri Jul  9 08:06:52 2004
+++ tools/python/xen/xend/server/controller.py.new      Fri Jul  9 08:06:24 2004
@@ -178,7 +178,7 @@
             resp = self.responders[0]
             if resp.mid > mid:
                 break
-            self.responders.pop()
+            self.responders.pop(0)
             if resp.mid < mid:
                 print 'callResponders> Out of order:', resp.mid, mid
                 resp.error(OutOfOrderError())


On a seperate matter, i always have to apply this patch to a new clone of
xen-unstable or stuff doesn't work. Am i the only one having this trouble,
and if so, any ideas why?

--- tools/python/xen/xend/XendDomainInfo.py     Fri Jul  9 08:10:38 2004
+++ tools/python/xen/xend/XendDomainInfo.py.new Fri Jul  9 08:10:18 2004
@@ -49,6 +49,20 @@
         lines.append(line)
     return lines
 
+def readline(fd):
+    """Version of readline safe against EINTR.
+    """
+    import errno
+    
+    while 1:
+        try:
+            return fd.readline()
+        except IOError, ex:
+            if ex.errno == errno.EINTR:
+                continue
+            else:
+                raise
+
 class VmError(ValueError):
     """Vm construction error."""
 

-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

[-- Attachment #2: Type: text/html, Size: 4018 bytes --]

Re: Re: almost working

[Keir Fraser]

Thanks! I forwarded it to Mike Wray, so it should get checked in
fairly soon.

 -- Keir

> After spending 2 solid hours slogging away at this in the early hours
> of this morning (or late late last night if you prefer), i tracked it
> down. To my eyes it looks like it had the potential to muck about with
> more than just VBD's, and that's just what was triggering it in this case.


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Jody Belka]

On Fri, Jul 09, 2004 at 07:35:24PM +1000, James Harper wrote:
> This fixed my problem where I couldn't start a domain with >1 virtual network interface. I for one think your time was well spent!!!

Glad to have helped someone else at the same time.


> re: your second patch, exactly what doesn't work without it? Apart from reboots under high load I don't seem to have any other problems, although it looks like something that would show up randomly.
>> +def readline(fd):
>> +    """Version of readline safe against EINTR.

Well, creating a domain for one :/

Error: Error creating domain: [Failure instance: Traceback: exceptions.NameError, global name 'readline' is not defined


-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Ian Pratt]

> After spending 2 solid hours slogging away at this in the early hours
> of this morning (or late late last night if you prefer), i tracked it
> down. To my eyes it looks like it had the potential to muck about with
> more than just VBD's, and that's just what was triggering it in this case.

Thanks very much!  The code still needs a bit of further work as
the kernel is at liberty to issue responses out of order (though
this rarely happens in practice).

> On a seperate matter, i always have to apply this patch to a new clone of
> xen-unstable or stuff doesn't work. Am i the only one having this trouble,
> and if so, any ideas why?

Ah -- your config uses raw disks rather than partitions so you
hit this issue. Thanks for the patch.

Ian


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Jody Belka]

On Fri, Jul 09, 2004 at 12:19:05PM +0100, Ian Pratt wrote:
> Ah -- your config uses raw disks rather than partitions so you
> hit this issue. Thanks for the patch.

Well, neither of them to be precise ;)  I use EVMS to manage my
systems, so it's EVMS volumes that i'm passing in. Looking at the
code, luckily i don't have any digits in the volume names or
it'd assume i was passing in devices with dos partition tables
and fail to process them correctly.


-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Niraj Tolia]

On Tue, 06 Jul 2004 17:11:04 +0100, Keir Fraser
<keir.fraser@cl.cam.ac.uk> wrote:
> > > Not good. I think xend is getting more robust over time.
> >
> > I have seen a number of ''heisenbugs" with xend. They especially occur
> > when I stop xend and start it again. Every now and then a 'xend start'
> > does nothing and sends no output to console. Commands such as 'xm
> > list' will give a 'Connection refused' error. A 'xend stop' followed
> > by a 'xend start' usually fixes this but there have been a few times
> > where I have had to repeat the cycle a couple of times.
> >
> > Also, sometimes after a reboot, a domain creation just fails and the
> > 'xm create' command does not return. There are no error messages from
> > xend. Even though I suspect that things are hosed at this point in
> > time, a 'xend stop' works. However xend will not startup again and
> > gives a 'IOError: [Errno 16] Device or resource busy' error. The
> > problem cannot be reproduced as the  same create command works after
> > another reboot.
> 
> I think many people had been seeing problems of this ilk, so you will
> probably be delighted to hear that these bugs have been squashed by
> today's checkins.
> 
> It was a combination of a simple stupid bug in the device-driver code,
> coupled with faulty error handling in xend.
> 

Just FYI, I still see most of the above bugs. Running the latest and greatest.

BTW what has the start_trace option for xend been replaced by?

Niraj

>  -- Keir
>


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Ian Pratt]

> > > I have seen a number of ''heisenbugs" with xend. They especially occur
> > > when I stop xend and start it again. Every now and then a 'xend start'
> > > does nothing and sends no output to console. Commands such as 'xm
> > > list' will give a 'Connection refused' error. A 'xend stop' followed
> > > by a 'xend start' usually fixes this but there have been a few times
> > > where I have had to repeat the cycle a couple of times.
> > >
> > > Also, sometimes after a reboot, a domain creation just fails and the
> > > 'xm create' command does not return. There are no error messages from
> > > xend. Even though I suspect that things are hosed at this point in
> > > time, a 'xend stop' works. However xend will not startup again and
> > > gives a 'IOError: [Errno 16] Device or resource busy' error. The
> > > problem cannot be reproduced as the  same create command works after
> > > another reboot.

> Just FYI, I still see most of the above bugs. Running the latest and greatest.

I've actually found that xend has been very stable for me for the
last few days. How are other people finding it?

I think there's real merit in cleaning out old installations
before doing a new install e.g.:

 rm -rf "/etc/xen /usr/lib/python2.2/site-packages/xen* /usr/lib/libxc* /usr/lib/python2.2/site-packages/Xc*"

If you still get problems, please post any output.


I know of two outstanding Xen bugs, both of which Keir
understands and is about to vanquish:

 * instability after shutting a domain down on a CPU other than CPU 0.

 * dom0 oops when hitting local block devices hard from within
dom0 while doing block io from other domains.

Let's get these two fixed then see where we are.

Ian





-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Re: almost working

[Keir Fraser]

> I know of two outstanding Xen bugs, both of which Keir
> understands and is about to vanquish:
> 
>  * instability after shutting a domain down on a CPU other than CPU 0.
> 
>  * dom0 oops when hitting local block devices hard from within
> dom0 while doing block io from other domains.
> 
> Let's get these two fixed then see where we are.

Both of these are now fixed. Both of them were both nasty and fairly
difficult to hit --- but when you hit them they would take out the
machine by crashing either Xen or DOM0.

Hopefully occasional machine reboots will be rather more occasional
now. :-)

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

RE: Re: almost working

[James Harper]

[-- Attachment #1: Type: text/plain, Size: 2950 bytes --]

Just for feedback's sake - I've been running an unprivileged xen domain which has been happily processing spam for over 24 hours. It is configured as a backup mail server and because the primary server is up, it only really gets spam (spammers love to hit MX records other than the lowest as they often aren't configured for spam filtering). It get's a message every few minutes or so.

It's running on an SMP machine with two virtual interfaces mapped to two physical interfaces in dom0.

my xen tree is probably 36 hours old or so and therefore probably doesn't have any of the recent fixes in it which from what i've read fix quite a few issues - none of which are affecting me thus far except for a reboot problem I saw a few days ago.

just a suggestion, could you put those rm's which you mention in a make option, eg make installflush. Maybe exclude /etc/xen from that though or only remove specific files from there though, i've modified at least one config file in there.

James



From: Ian Pratt
Sent: Wed 14/07/2004 1:13 AM
To: Niraj Tolia
Cc: Keir Fraser; Ian Pratt; James Harper; xen-devel@lists.sourceforge.net; Ian.Pratt@cl.cam.ac.uk
Subject: Re: [Xen-devel] Re: almost working


> > > I have seen a number of ''heisenbugs" with xend. They especially occur
> > > when I stop xend and start it again. Every now and then a 'xend start'
> > > does nothing and sends no output to console. Commands such as 'xm
> > > list' will give a 'Connection refused' error. A 'xend stop' followed
> > > by a 'xend start' usually fixes this but there have been a few times
> > > where I have had to repeat the cycle a couple of times.
> > >
> > > Also, sometimes after a reboot, a domain creation just fails and the
> > > 'xm create' command does not return. There are no error messages from
> > > xend. Even though I suspect that things are hosed at this point in
> > > time, a 'xend stop' works. However xend will not startup again and
> > > gives a 'IOError: [Errno 16] Device or resource busy' error. The
> > > problem cannot be reproduced as the  same create command works after
> > > another reboot.

> Just FYI, I still see most of the above bugs. Running the latest and greatest.

I've actually found that xend has been very stable for me for the
last few days. How are other people finding it?

I think there's real merit in cleaning out old installations
before doing a new install e.g.:

 rm -rf "/etc/xen /usr/lib/python2.2/site-packages/xen* /usr/lib/libxc* /usr/lib/python2.2/site-packages/Xc*"

If you still get problems, please post any output.


I know of two outstanding Xen bugs, both of which Keir
understands and is about to vanquish:

 * instability after shutting a domain down on a CPU other than CPU 0.

 * dom0 oops when hitting local block devices hard from within
dom0 while doing block io from other domains.

Let's get these two fixed then see where we are.

Ian

[-- Attachment #2: Type: text/html, Size: 3835 bytes --]