qemu and xl semantics

qemu and xl semantics

[Christoph Egger]

Hi!

When I start a guest with xm  the disk startup script assigns a loopback 
device for qemu to open it.

Now it seems that qemu opens the disk image directly. Then when
the loopback device wants to open the disk image then that fails
with EBUSY.

How is the disk startup script supposed to work with the new
semantic for
a) HVM guests
b) PV guests
?

The network startup script adds the tap device to the bridge
or assigns an ip address.
With xl neither the disk nor the network script runs.
So when I start the guest with xl then I have
the tap device assigned to the guest but the
tap device is not configured in the dom0.

How does the 'xl' way work in respect to the network script
used with 'xm' ?


Christoph


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: qemu and xl semantics

[Ian Campbell]

On Fri, 2010-12-17 at 09:00 +0000, Christoph Egger wrote:
> Hi!
> 
> When I start a guest with xm  the disk startup script assigns a loopback 
> device for qemu to open it.
> 
> Now it seems that qemu opens the disk image directly. Then when
> the loopback device wants to open the disk image then that fails
> with EBUSY.

By "Now..." you mean "With xl..." ?

> How is the disk startup script supposed to work with the new
> semantic for
> a) HVM guests
> b) PV guests
> ?

I think this is all very specific to the precise disk type you have in
your config, i.e. tap: vs file: vs phy: etc. Which are you using?

> 
> The network startup script adds the tap device to the bridge
> or assigns an ip address.
> With xl neither the disk nor the network script runs.
> So when I start the guest with xl then I have
> the tap device assigned to the guest but the
> tap device is not configured in the dom0.
> 
> How does the 'xl' way work in respect to the network script
> used with 'xm' ?

On Linux these are run from the hotplug event, via the udev rules. I
presume you are talking about on NetBSD though?

Under Linux I think it was always the same under xm too although there
have been some tweaks recently, e.g. the vif script is now always
/etc/xen/scripts/vif-setup which handles the indirection to the script
in the domain config or the default. Previously xend the hotplug rules
called the configured script directly. This change was
21549:8bcaec29574e and was common to xm and xl I think.

Ian.

Re: qemu and xl semantics

[Christoph Egger]

On Friday 17 December 2010 10:15:14 Ian Campbell wrote:
> On Fri, 2010-12-17 at 09:00 +0000, Christoph Egger wrote:
> > Hi!
> >
> > When I start a guest with xm  the disk startup script assigns a loopback
> > device for qemu to open it.
> >
> > Now it seems that qemu opens the disk image directly. Then when
> > the loopback device wants to open the disk image then that fails
> > with EBUSY.
>
> By "Now..." you mean "With xl..." ?

no, I mean with xm.

>
> > How is the disk startup script supposed to work with the new
> > semantic for
> > a) HVM guests
> > b) PV guests
> > ?
>
> I think this is all very specific to the precise disk type you have in
> your config, i.e. tap: vs file: vs phy: etc. Which are you using?

I use 'file'.

> > The network startup script adds the tap device to the bridge
> > or assigns an ip address.
> > With xl neither the disk nor the network script runs.
> > So when I start the guest with xl then I have
> > the tap device assigned to the guest but the
> > tap device is not configured in the dom0.
> >
> > How does the 'xl' way work in respect to the network script
> > used with 'xm' ?
>
> On Linux these are run from the hotplug event, via the udev rules. I
> presume you are talking about on NetBSD though?

Yes.

> Under Linux I think it was always the same under xm too although there
> have been some tweaks recently, e.g. the vif script is now always
> /etc/xen/scripts/vif-setup which handles the indirection to the script
> in the domain config or the default. Previously xend the hotplug rules
> called the configured script directly. This change was
> 21549:8bcaec29574e and was common to xm and xl I think.

On NetBSD a xenbackendd starts along with xenstored.
xenbackendd watches on /local/domain/0/backend
and invokes the corresponding scripts when something changes
beneath that.

'xend' is doing that in DevController.py. Since 'xl' is not interacting
with 'xend' the scripts don't get launched at all.

Christoph


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: qemu and xl semantics

[Ian Campbell]

On Fri, 2010-12-17 at 09:49 +0000, Christoph Egger wrote:
> On Friday 17 December 2010 10:15:14 Ian Campbell wrote:
> > On Fri, 2010-12-17 at 09:00 +0000, Christoph Egger wrote:
> > > Hi!
> > >
> > > When I start a guest with xm  the disk startup script assigns a loopback
> > > device for qemu to open it.
> > >
> > > Now it seems that qemu opens the disk image directly. Then when
> > > the loopback device wants to open the disk image then that fails
> > > with EBUSY.
> >
> > By "Now..." you mean "With xl..." ?
> 
> no, I mean with xm.

Hrm, I didn't think xm had changed in this area for ages, but I don't
really keep an eye on xm/xend.

> > > How is the disk startup script supposed to work with the new
> > > semantic for
> > > a) HVM guests
> > > b) PV guests
> > > ?

I don't think there are new semantics with xm, you aren't talking about
xl here?

Can you identify a change to xend which you think changed the semantics?

> > I think this is all very specific to the precise disk type you have in
> > your config, i.e. tap: vs file: vs phy: etc. Which are you using?
> 
> I use 'file'.

If you are using xm and not xl this is not relevant but:

AIUI libxl treats file: as tap:aio:, in other words it will fire up
blktap to provide the device backend. xend probably used a loop device
and treated it more like phy: i.e. invoked blkback.

Since NetBSD doesn't have blktap perhaps libxl needs to be updated to do
the NetBSD equivalent? (I thought I'd seen a patch from you which did
this?)

Recently libxl was also changed to use the qemu disk backend in cases
where blktap is not available -- perhaps that had an undesired knockon
effect on NetBSD?

Further to this there was a patch floating around (from Stefano) which
ensured that a qemu was launched when it was necessary for PV guests to
provide the disk backend. I'm not sure this got merged but ion the
meantime the workaround is to add a vfb which also causes qemu to get
launched for a PV guest.

> > > The network startup script adds the tap device to the bridge
> > > or assigns an ip address.
> > > With xl neither the disk nor the network script runs.
> > > So when I start the guest with xl then I have
> > > the tap device assigned to the guest but the
> > > tap device is not configured in the dom0.
> > >
> > > How does the 'xl' way work in respect to the network script
> > > used with 'xm' ?
> >
> > On Linux these are run from the hotplug event, via the udev rules. I
> > presume you are talking about on NetBSD though?
> 
> Yes.
> 
> > Under Linux I think it was always the same under xm too although there
> > have been some tweaks recently, e.g. the vif script is now always
> > /etc/xen/scripts/vif-setup which handles the indirection to the script
> > in the domain config or the default. Previously xend the hotplug rules
> > called the configured script directly. This change was
> > 21549:8bcaec29574e and was common to xm and xl I think.
> 
> On NetBSD a xenbackendd starts along with xenstored.
> xenbackendd watches on /local/domain/0/backend
> and invokes the corresponding scripts when something changes
> beneath that.
> 
> 'xend' is doing that in DevController.py. Since 'xl' is not interacting
> with 'xend' the scripts don't get launched at all.

xl also creates stuff under /local/domain/0/backend (as it must) so why
is xenbackendd not firing up and running the scripts?

Perhaps xenbackendd is expecting some additional key which xend adds but
libxl doesn't?

I think this needs someone to debug from the NetBSD side, it's possible
libxl will need to be tweaked to work properly with xenbackendd or vice
versa.


Ian.

Re: qemu and xl semantics

[Christoph Egger]

On Friday 17 December 2010 11:32:41 Ian Campbell wrote:
> On Fri, 2010-12-17 at 09:49 +0000, Christoph Egger wrote:
> > On Friday 17 December 2010 10:15:14 Ian Campbell wrote:
> > > On Fri, 2010-12-17 at 09:00 +0000, Christoph Egger wrote:
> > > > Hi!
> > > >
> > > > When I start a guest with xm  the disk startup script assigns a
> > > > loopback device for qemu to open it.
> > > >
> > > > Now it seems that qemu opens the disk image directly. Then when
> > > > the loopback device wants to open the disk image then that fails
> > > > with EBUSY.
> > >
> > > By "Now..." you mean "With xl..." ?
> >
> > no, I mean with xm.
>
> Hrm, I didn't think xm had changed in this area for ages, but I don't
> really keep an eye on xm/xend.

It is not xm/xend that changed. It is qemu that changed and
the startup scripts need an adaption to the new behaviour.
Hence I am asking for the new semantics to know how to do that.

> > > > How is the disk startup script supposed to work with the new
> > > > semantic for
> > > > a) HVM guests
> > > > b) PV guests
> > > > ?
>
> I don't think there are new semantics with xm, you aren't talking about
> xl here?
>
> Can you identify a change to xend which you think changed the semantics?

See above.

> > > I think this is all very specific to the precise disk type you have in
> > > your config, i.e. tap: vs file: vs phy: etc. Which are you using?
> >
> > I use 'file'.
>
> If you are using xm and not xl this is not relevant but:
>
> AIUI libxl treats file: as tap:aio:, in other words it will fire up
> blktap to provide the device backend. xend probably used a loop device
> and treated it more like phy: i.e. invoked blkback.

Yes, I noticed that and I wonder how the algorithm works behind that.

> Since NetBSD doesn't have blktap perhaps libxl needs to be updated to do
> the NetBSD equivalent? (I thought I'd seen a patch from you which did
> this?)

The disk access works on NetBSD - don't ask me how - otherwise I wouldn't
be able to boot a guest at all. I have to run the network script manually
in the dom0 to have network access from and to the guest.

> Recently libxl was also changed to use the qemu disk backend in cases
> where blktap is not available -- perhaps that had an undesired knockon
> effect on NetBSD?

Yes. Recently this message appears from the NetBSD disk backend driver:

xbdback backend/vbd/1/832: can't VOP_OPEN device 0xe13: 16
16 means EBUSY.

> Further to this there was a patch floating around (from Stefano) which
> ensured that a qemu was launched when it was necessary for PV guests to
> provide the disk backend. I'm not sure this got merged but ion the
> meantime the workaround is to add a vfb which also causes qemu to get
> launched for a PV guest.

What is the longterm solution?

> > > > The network startup script adds the tap device to the bridge
> > > > or assigns an ip address.
> > > > With xl neither the disk nor the network script runs.
> > > > So when I start the guest with xl then I have
> > > > the tap device assigned to the guest but the
> > > > tap device is not configured in the dom0.
> > > >
> > > > How does the 'xl' way work in respect to the network script
> > > > used with 'xm' ?
> > >
> > > On Linux these are run from the hotplug event, via the udev rules. I
> > > presume you are talking about on NetBSD though?
> >
> > Yes.
> >
> > > Under Linux I think it was always the same under xm too although there
> > > have been some tweaks recently, e.g. the vif script is now always
> > > /etc/xen/scripts/vif-setup which handles the indirection to the script
> > > in the domain config or the default. Previously xend the hotplug rules
> > > called the configured script directly. This change was
> > > 21549:8bcaec29574e and was common to xm and xl I think.
> >
> > On NetBSD a xenbackendd starts along with xenstored.
> > xenbackendd watches on /local/domain/0/backend
> > and invokes the corresponding scripts when something changes
> > beneath that.
> >
> > 'xend' is doing that in DevController.py. Since 'xl' is not interacting
> > with 'xend' the scripts don't get launched at all.
>
> xl also creates stuff under /local/domain/0/backend (as it must) so why
> is xenbackendd not firing up and running the scripts?

xenbackendd is firing up for things 
like '/local/domain/0/backend/console/1/0/script'.

xenbackendd listens for changes on vec[XS_WATCH_PATH]/state.
Then it reads vec[XS_WATCH_PATH]/hotplug-status.

And then it looks at '/local/domain/0/backend/vbd' 
and '/local/domain/0/backend/vif' and starts the corresponding
scripts.

> Perhaps xenbackendd is expecting some additional key which xend adds but
> libxl doesn't?

Maybe. Is something missing from above?

Christoph

> I think this needs someone to debug from the NetBSD side, it's possible
> libxl will need to be tweaked to work properly with xenbackendd or vice
> versa.
>
>
> Ian.



-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: qemu and xl semantics

[Ian Campbell]

On Fri, 2010-12-17 at 17:21 +0000, Christoph Egger wrote:
> On Friday 17 December 2010 11:32:41 Ian Campbell wrote:
> > On Fri, 2010-12-17 at 09:49 +0000, Christoph Egger wrote:
> > > On Friday 17 December 2010 10:15:14 Ian Campbell wrote:
> > > > On Fri, 2010-12-17 at 09:00 +0000, Christoph Egger wrote:
> > > > > Hi!
> > > > >
> > > > > When I start a guest with xm  the disk startup script assigns a
> > > > > loopback device for qemu to open it.
> > > > >
> > > > > Now it seems that qemu opens the disk image directly. Then when
> > > > > the loopback device wants to open the disk image then that fails
> > > > > with EBUSY.
> > > >
> > > > By "Now..." you mean "With xl..." ?
> > >
> > > no, I mean with xm.
> >
> > Hrm, I didn't think xm had changed in this area for ages, but I don't
> > really keep an eye on xm/xend.
> 
> It is not xm/xend that changed. It is qemu that changed and
> the startup scripts need an adaption to the new behaviour.
> Hence I am asking for the new semantics to know how to do that.

Which qemu changeset? (you obviously had this information to hand, why
not pro-actively help me to help you? Don't make me guess what you know
or have discovered)

The only recent qemu change which I am aware of in this area is the
backport of the blkback functionality from upstream Qemu. However this
should only be enabled in cases where blkback or blktap are not
available and furthermore is tied to libxl/xl so shouldn't have done
anything on xend (although shouldn't != doesn't).

Perhaps the problem is that this stuff is erroneously getting activated
on NetBSD?

> > > > I think this is all very specific to the precise disk type you have in
> > > > your config, i.e. tap: vs file: vs phy: etc. Which are you using?
> > >
> > > I use 'file'.
> >
> > If you are using xm and not xl this is not relevant but:
> >
> > AIUI libxl treats file: as tap:aio:, in other words it will fire up
> > blktap to provide the device backend. xend probably used a loop device
> > and treated it more like phy: i.e. invoked blkback.
> 
> Yes, I noticed that and I wonder how the algorithm works behind that.

See libxl_device.c:libxl__device_disk_backend_type_of_phystype and
libxl.c:libxl_device_disk_add.

A file: type device is really just a "degenerate" thing in the style of
qcow or vhd so using tapdisk to export it makes some sort of sense and
when it came to implementing libxl it was the simpler option compared
with integrating with the loopback subsystem.

> > Since NetBSD doesn't have blktap perhaps libxl needs to be updated to do
> > the NetBSD equivalent? (I thought I'd seen a patch from you which did
> > this?)
> 
> The disk access works on NetBSD - don't ask me how - otherwise I wouldn't
> be able to boot a guest at all. I have to run the network script manually
> in the dom0 to have network access from and to the guest.
> 
> > Recently libxl was also changed to use the qemu disk backend in cases
> > where blktap is not available -- perhaps that had an undesired knockon
> > effect on NetBSD?
> 
> Yes. Recently this message appears from the NetBSD disk backend driver:
> 
> xbdback backend/vbd/1/832: can't VOP_OPEN device 0xe13: 16
> 16 means EBUSY.

But it works other than this message?

I don't have much insight into how the NetBSD side of things works, I'm
afraid I think you will need to dig in and debug it.

> > Further to this there was a patch floating around (from Stefano) which
> > ensured that a qemu was launched when it was necessary for PV guests to
> > provide the disk backend. I'm not sure this got merged but ion the
> > meantime the workaround is to add a vfb which also causes qemu to get
> > launched for a PV guest.
> 
> What is the longterm solution?

The patch from Stefano.

Even longer term we may improve the libxl interface to make it more
explicit to the calling toolstack when it needs to start a qemu.

> 
> > > > > The network startup script adds the tap device to the bridge
> > > > > or assigns an ip address.
> > > > > With xl neither the disk nor the network script runs.
> > > > > So when I start the guest with xl then I have
> > > > > the tap device assigned to the guest but the
> > > > > tap device is not configured in the dom0.
> > > > >
> > > > > How does the 'xl' way work in respect to the network script
> > > > > used with 'xm' ?
> > > >
> > > > On Linux these are run from the hotplug event, via the udev rules. I
> > > > presume you are talking about on NetBSD though?
> > >
> > > Yes.
> > >
> > > > Under Linux I think it was always the same under xm too although there
> > > > have been some tweaks recently, e.g. the vif script is now always
> > > > /etc/xen/scripts/vif-setup which handles the indirection to the script
> > > > in the domain config or the default. Previously xend the hotplug rules
> > > > called the configured script directly. This change was
> > > > 21549:8bcaec29574e and was common to xm and xl I think.
> > >
> > > On NetBSD a xenbackendd starts along with xenstored.
> > > xenbackendd watches on /local/domain/0/backend
> > > and invokes the corresponding scripts when something changes
> > > beneath that.
> > >
> > > 'xend' is doing that in DevController.py. Since 'xl' is not interacting
> > > with 'xend' the scripts don't get launched at all.
> >
> > xl also creates stuff under /local/domain/0/backend (as it must) so why
> > is xenbackendd not firing up and running the scripts?
> 
> xenbackendd is firing up for things 
> like '/local/domain/0/backend/console/1/0/script'.
> 
> xenbackendd listens for changes on vec[XS_WATCH_PATH]/state.
> Then it reads vec[XS_WATCH_PATH]/hotplug-status.
> 
> And then it looks at '/local/domain/0/backend/vbd' 
> and '/local/domain/0/backend/vif' and starts the corresponding
> scripts.
> 
> > Perhaps xenbackendd is expecting some additional key which xend adds but
> > libxl doesn't?
> 
> Maybe. Is something missing from above?

I've no idea -- what does xenbackendd expect? What do you get in
xenstore? Howe do the two match up?

Ian.

Re: qemu and xl semantics

[Christoph Egger]

On Monday 20 December 2010 11:23:59 Ian Campbell wrote:
> On Fri, 2010-12-17 at 17:21 +0000, Christoph Egger wrote:
> > On Friday 17 December 2010 11:32:41 Ian Campbell wrote:
> > > On Fri, 2010-12-17 at 09:49 +0000, Christoph Egger wrote:
> > > > On Friday 17 December 2010 10:15:14 Ian Campbell wrote:
> > > > > On Fri, 2010-12-17 at 09:00 +0000, Christoph Egger wrote:
> > > > > > Hi!
> > > > > >
> > > > > > When I start a guest with xm  the disk startup script assigns a
> > > > > > loopback device for qemu to open it.
> > > > > >
> > > > > > Now it seems that qemu opens the disk image directly. Then when
> > > > > > the loopback device wants to open the disk image then that fails
> > > > > > with EBUSY.
> > > > >
> > > > > By "Now..." you mean "With xl..." ?
> > > >
> > > > no, I mean with xm.
> > >
> > > Hrm, I didn't think xm had changed in this area for ages, but I don't
> > > really keep an eye on xm/xend.
> >
> > It is not xm/xend that changed. It is qemu that changed and
> > the startup scripts need an adaption to the new behaviour.
> > Hence I am asking for the new semantics to know how to do that.
>
> Which qemu changeset? (you obviously had this information to hand, why
> not pro-actively help me to help you? Don't make me guess what you know
> or have discovered)

It is git head or git tip - whatever latest c/s is called.

>
> The only recent qemu change which I am aware of in this area is the
> backport of the blkback functionality from upstream Qemu. However this
> should only be enabled in cases where blkback or blktap are not
> available and furthermore is tied to libxl/xl so shouldn't have done
> anything on xend (although shouldn't != doesn't).

Is this supposed to interact with the Dom0 PV disk backend driver ?

> Perhaps the problem is that this stuff is erroneously getting activated
> on NetBSD?
>
> > > > > I think this is all very specific to the precise disk type you have
> > > > > in your config, i.e. tap: vs file: vs phy: etc. Which are you
> > > > > using?
> > > >
> > > > I use 'file'.
> > >
> > > If you are using xm and not xl this is not relevant but:
> > >
> > > AIUI libxl treats file: as tap:aio:, in other words it will fire up
> > > blktap to provide the device backend. xend probably used a loop device
> > > and treated it more like phy: i.e. invoked blkback.
> >
> > Yes, I noticed that and I wonder how the algorithm works behind that.
>
> See libxl_device.c:libxl__device_disk_backend_type_of_phystype and
> libxl.c:libxl_device_disk_add.
>
> A file: type device is really just a "degenerate" thing in the style of
> qcow or vhd so using tapdisk to export it makes some sort of sense and
> when it came to implementing libxl it was the simpler option compared
> with integrating with the loopback subsystem.
>
> > > Since NetBSD doesn't have blktap perhaps libxl needs to be updated to
> > > do the NetBSD equivalent? (I thought I'd seen a patch from you which
> > > did this?)
> >
> > The disk access works on NetBSD - don't ask me how - otherwise I wouldn't
> > be able to boot a guest at all. I have to run the network script manually
> > in the dom0 to have network access from and to the guest.
> >
> > > Recently libxl was also changed to use the qemu disk backend in cases
> > > where blktap is not available -- perhaps that had an undesired knockon
> > > effect on NetBSD?
> >
> > Yes. Recently this message appears from the NetBSD disk backend driver:
> >
> > xbdback backend/vbd/1/832: can't VOP_OPEN device 0xe13: 16
> > 16 means EBUSY.
>
> But it works other than this message?

Well, that means that there are now two processes which want to open the vbd
simultaneously. The first one wins. And on guest shutdown the VOP_CLOSE fails.

> I don't have much insight into how the NetBSD side of things works, I'm
> afraid I think you will need to dig in and debug it.
>
> > > Further to this there was a patch floating around (from Stefano) which
> > > ensured that a qemu was launched when it was necessary for PV guests to
> > > provide the disk backend. I'm not sure this got merged but ion the
> > > meantime the workaround is to add a vfb which also causes qemu to get
> > > launched for a PV guest.
> >
> > What is the longterm solution?
>
> The patch from Stefano.
>
> Even longer term we may improve the libxl interface to make it more
> explicit to the calling toolstack when it needs to start a qemu.
>
> > > > > > The network startup script adds the tap device to the bridge
> > > > > > or assigns an ip address.
> > > > > > With xl neither the disk nor the network script runs.
> > > > > > So when I start the guest with xl then I have
> > > > > > the tap device assigned to the guest but the
> > > > > > tap device is not configured in the dom0.
> > > > > >
> > > > > > How does the 'xl' way work in respect to the network script
> > > > > > used with 'xm' ?
> > > > >
> > > > > On Linux these are run from the hotplug event, via the udev rules.
> > > > > I presume you are talking about on NetBSD though?
> > > >
> > > > Yes.
> > > >
> > > > > Under Linux I think it was always the same under xm too although
> > > > > there have been some tweaks recently, e.g. the vif script is now
> > > > > always /etc/xen/scripts/vif-setup which handles the indirection to
> > > > > the script in the domain config or the default. Previously xend the
> > > > > hotplug rules called the configured script directly. This change
> > > > > was
> > > > > 21549:8bcaec29574e and was common to xm and xl I think.
> > > >
> > > > On NetBSD a xenbackendd starts along with xenstored.
> > > > xenbackendd watches on /local/domain/0/backend
> > > > and invokes the corresponding scripts when something changes
> > > > beneath that.
> > > >
> > > > 'xend' is doing that in DevController.py. Since 'xl' is not
> > > > interacting with 'xend' the scripts don't get launched at all.
> > >
> > > xl also creates stuff under /local/domain/0/backend (as it must) so why
> > > is xenbackendd not firing up and running the scripts?
> >
> > xenbackendd is firing up for things
> > like '/local/domain/0/backend/console/1/0/script'.
> >
> > xenbackendd listens for changes on vec[XS_WATCH_PATH]/state.
> > Then it reads vec[XS_WATCH_PATH]/hotplug-status.
> >
> > And then it looks at '/local/domain/0/backend/vbd'
> > and '/local/domain/0/backend/vif' and starts the corresponding
> > scripts.

I did some more debugging and figured out xenbackendd runs the vif-bridge
script so network with a PV driver works. 

The scripts not running is the one associated with 'vbd' and qemu-ifup.
AFAIK  qemu-dm always run qemu-ifup which is no longer the case.

qemu-ifup adds tap interface to the bridge.

> > > Perhaps xenbackendd is expecting some additional key which xend adds
> > > but libxl doesn't?
> >
> > Maybe. Is something missing from above?
>
> I've no idea -- what does xenbackendd expect? What do you get in
> xenstore? Howe do the two match up?

This is the output of 'xenstore-ls':

tool = ""
 xenstored = ""
local = ""
 domain = ""
  0 = ""
   backend = ""
    qdisk = ""
     2 = ""
      768 = ""
       frontend = "/local/domain/2/device/vbd/768"
       params = "aio:/hvm-guest/netbsd_64b.img"
       frontend-id = "2"
       online = "1"
       removable = "1"
       bootable = "1"
       state = "1"
       dev = "hda"
       type = "tap"
       mode = "w"
      832 = ""
       frontend = "/local/domain/2/device/vbd/832"
       params = "aio:/hvm-guest/netbsd_disk2.img"
       frontend-id = "2"
       online = "1"
       removable = "1"
       bootable = "1"
       state = "1"
       dev = "hdb"
       type = "tap"
       mode = "w"
    vif = ""
     2 = ""
      0 = ""
       frontend = "/local/domain/2/device/vif/0"
       frontend-id = "2"
       online = "1"
       state = "2"
       script = "/usr/local.22573.netbsd/etc/xen/scripts/vif-bridge"
       mac = "00:16:3e:4d:e0:51"
       bridge = "bridge0"
       handle = "0"
       feature-rx-copy = "1"
       feature-rx-flip = "1"
       hotplug-status = "connected"
    console = ""
     2 = ""
      0 = ""
       frontend = "/local/domain/2/console"
       frontend-id = "2"
       online = "1"
       state = "1"
       domain = "HVM64-NetBSD"
       protocol = "vt100"
   device-model = ""
    2 = ""
     disable_pf = "0"
     state = "running"
  2 = ""
   vm = "/vm/50df51d0-d90d-e011-b0d4-00e081806fbe"
   name = "HVM64-NetBSD"
   device = ""
    suspend = ""
     event-channel = ""
    vbd = ""
     768 = ""
      backend = "/local/domain/0/backend/qdisk/2/768"
      backend-id = "0"
      state = "1"
      virtual-device = "768"
      device-type = "disk"
     832 = ""
      backend = "/local/domain/0/backend/qdisk/2/832"
      backend-id = "0"
      state = "1"
      virtual-device = "832"
      device-type = "disk"
    vif = ""
     0 = ""
      backend = "/local/domain/0/backend/vif/2/0"
      backend-id = "0"
      state = "1"
      handle = "0"
      mac = "00:16:3e:4d:e0:51"
   data = ""
   cpu = ""
    0 = ""
     availability = "online"
    1 = ""
     availability = "online"
    2 = ""
     availability = "online"
    3 = ""
     availability = "online"
   memory = ""
    static-max = "3145728"
    target = "3137536"
    videoram = "8192"
   error = ""
   drivers = ""
   control = ""
    platform-feature-multiprocessor-suspend = "1"
   attr = ""
   messages = ""
   domid = "2"
   store = ""
    port = "5"
    ring-ref = "1044476"
   console = ""
    backend = "/local/domain/0/backend/console/2/0"
    backend-id = "0"
    limit = "1048576"
    type = "xenconsoled"
    output = "pty"
    port = "6"
    ring-ref = "1044479"
    tty = "/dev/pts/1"
    vnc-port = "5900"
   image = ""
    device-model-pid = "558"
   serial = ""
    0 = ""
     tty = "/dev/pts/2"
vm = ""
 50df51d0-d90d-e011-b0d4-00e081806fbe = ""
  uuid = "50df51d0-d90d-e011-b0d4-00e081806fbe"
  name = "HVM64-NetBSD"
  pool_name = "Pool-0"
  rtc = ""
   timeoffset = ""
  image = ""
   ostype = "hvm"
  start_time = "1293028997.78"
  vncpasswd = "\000"


Christoph

-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: qemu and xl semantics

[Ian Campbell]

On Wed, 2010-12-22 at 15:42 +0000, Christoph Egger wrote:
> On Monday 20 December 2010 11:23:59 Ian Campbell wrote:
> > On Fri, 2010-12-17 at 17:21 +0000, Christoph Egger wrote:
> > > On Friday 17 December 2010 11:32:41 Ian Campbell wrote:
> > > > On Fri, 2010-12-17 at 09:49 +0000, Christoph Egger wrote:
> > > > > On Friday 17 December 2010 10:15:14 Ian Campbell wrote:
> > > > > > On Fri, 2010-12-17 at 09:00 +0000, Christoph Egger wrote:

> > The only recent qemu change which I am aware of in this area is the
> > backport of the blkback functionality from upstream Qemu. However this
> > should only be enabled in cases where blkback or blktap are not
> > available and furthermore is tied to libxl/xl so shouldn't have done
> > anything on xend (although shouldn't != doesn't).
> 
> Is this supposed to interact with the Dom0 PV disk backend driver ?

The qemu disk backend is intended to be used when there is no dom0 PV
disk backend driver in the kernel, if there is a driver in the kernel
then it is intended that the kernel driver should be used.

The background to this is that we have just gotten the basic dom0
support into the upstream Linux kernel and are about to start on
upstreaming the backend drivers. However we observed that there may not
be any need to upstream a block backend since a userspace implementation
may well suffice. It's not a done decision but the early signs are good,
especially compared with blktap which hits userspace anyway.

Even if we do eventually decide to upstream the Linux blkback using the
disk backend in qemu tides us over for the time being.

> > > xbdback backend/vbd/1/832: can't VOP_OPEN device 0xe13: 16
> > > 16 means EBUSY.
> >
> > But it works other than this message?
> 
> Well, that means that there are now two processes which want to open the vbd
> simultaneously. The first one wins. And on guest shutdown the VOP_CLOSE fails.

Right. This suggests that the qemu disk backend is being erroneously
activated on NetBSD when what you actually want is to use the xbdback
process. I think you likely need to patch libxl to do the right thing
for NetBSD, currently the decision is based on the presence or absence
of blktap, I guess it needs extending to detect NetBSD's backend too.

Alternatively perhaps NetBSD also wants to transition to using the qemu
based block backend, I suppose there is benefit to be had from sharing
this code between Linux and NetBSD dom0.

> I did some more debugging and figured out xenbackendd runs the vif-bridge
> script so network with a PV driver works. 
> 
> The scripts not running is the one associated with 'vbd' and qemu-ifup.
> AFAIK  qemu-dm always run qemu-ifup which is no longer the case.
> 
> qemu-ifup adds tap interface to the bridge.

This changed (on Linux) quite a while ago so I didn't think of it.
qemu-ifup is not used there any more and instead the vif-bridge script
is called for both PV and TAP devices. See xen-unstable.hg
21944:0232bc7c9544, I guess either some equivalent change is needed to
tools/hotplug/NetBSD or the libxl bit needs to be conditional on NetBSD.

> local = ""
>  domain = ""
>   0 = ""
>    backend = ""
>     qdisk = ""

You probably don't want this on NetBSD as described above.

However AFAIK only libxl will cause these to be created, so if you are
using xend then I am confused.

Ian.

Re: qemu and xl semantics

[Christoph Egger]

On Wednesday 22 December 2010 17:08:38 Ian Campbell wrote:
> On Wed, 2010-12-22 at 15:42 +0000, Christoph Egger wrote:
> > On Monday 20 December 2010 11:23:59 Ian Campbell wrote:
> > > On Fri, 2010-12-17 at 17:21 +0000, Christoph Egger wrote:
> > > > On Friday 17 December 2010 11:32:41 Ian Campbell wrote:
> > > > > On Fri, 2010-12-17 at 09:49 +0000, Christoph Egger wrote:
> > > > > > On Friday 17 December 2010 10:15:14 Ian Campbell wrote:
> > > > > > > On Fri, 2010-12-17 at 09:00 +0000, Christoph Egger wrote:
> > >
> > > The only recent qemu change which I am aware of in this area is the
> > > backport of the blkback functionality from upstream Qemu. However this
> > > should only be enabled in cases where blkback or blktap are not
> > > available and furthermore is tied to libxl/xl so shouldn't have done
> > > anything on xend (although shouldn't != doesn't).
> >
> > Is this supposed to interact with the Dom0 PV disk backend driver ?
>
> The qemu disk backend is intended to be used when there is no dom0 PV
> disk backend driver in the kernel, if there is a driver in the kernel
> then it is intended that the kernel driver should be used.
>
> The background to this is that we have just gotten the basic dom0
> support into the upstream Linux kernel and are about to start on
> upstreaming the backend drivers. However we observed that there may not
> be any need to upstream a block backend since a userspace implementation
> may well suffice. It's not a done decision but the early signs are good,
> especially compared with blktap which hits userspace anyway.
>
> Even if we do eventually decide to upstream the Linux blkback using the
> disk backend in qemu tides us over for the time being.

NetBSD has a kernel backend driver since Xen 1.x days ... upstream.
The last major change happened to it when it got updated to fit for Xen 2
and still fits well for Xen 3/4, too.

> > > > xbdback backend/vbd/1/832: can't VOP_OPEN device 0xe13: 16
> > > > 16 means EBUSY.
> > >
> > > But it works other than this message?
> >
> > Well, that means that there are now two processes which want to open the
> > vbd simultaneously. The first one wins. And on guest shutdown the
> > VOP_CLOSE fails.
>
> Right. This suggests that the qemu disk backend is being erroneously
> activated on NetBSD when what you actually want is to use the xbdback
> process. I think you likely need to patch libxl to do the right thing
> for NetBSD, currently the decision is based on the presence or absence
> of blktap, I guess it needs extending to detect NetBSD's backend too.

xbdback is actually the kernel backend driver. The conflicting processes
are qemu-dm and the script that assigns the disk through the loopback device.

It seems to work in either way depending on which process wins...

> Alternatively perhaps NetBSD also wants to transition to using the qemu
> based block backend, I suppose there is benefit to be had from sharing
> this code between Linux and NetBSD dom0.

Yes, I think so as long as it doesn't start requiring a kernel driver
other than xbdback.

> > I did some more debugging and figured out xenbackendd runs the vif-bridge
> > script so network with a PV driver works.
>
> > The scripts not running is the one associated with 'vbd' and qemu-ifup.
> > AFAIK  qemu-dm always run qemu-ifup which is no longer the case.
> >
> > qemu-ifup adds tap interface to the bridge.
>
> This changed (on Linux) quite a while ago so I didn't think of it.
> qemu-ifup is not used there any more and instead the vif-bridge script
> is called for both PV and TAP devices. See xen-unstable.hg
> 21944:0232bc7c9544, I guess either some equivalent change is needed to
> tools/hotplug/NetBSD or the libxl bit needs to be conditional on NetBSD.
>

hmm... so when I start the guest with 'xm' then qemu runs qemu-ifup.
when I start the guest with 'xl' then qemu does not.

So when I adjust the script then I fix one and break the other.
So to have it working on both the best way is to have the libxl bit 
conditional on NetBSD.

BTW: Using the hardcoded path of the libxl bit prior to c/s 21944
doesn't work for me since I install into /usr/local

----------------------------------------------------------------------------------------------
> > local = ""
> >  domain = ""
> >   0 = ""
> >    backend = ""
> >     qdisk = ""
>
> You probably don't want this on NetBSD as described above.
>
> However AFAIK only libxl will cause these to be created, so if you are
> using xend then I am confused.

Sorry, I confused myself. This snippet counts for using xl w/o xend.
-----------------------------------------------------------------------------------------


Again: The 'xenstore-ls' output with 'xm/xend':

tool = ""
 xenstored = ""
local = ""
 pool = ""
  0 = ""
   other_config = ""
   description = "Pool-0"
   uuid = "ea0f9675-726e-e493-8b79-2149f486cb00"
   name = "Pool-0"
 domain = ""
  0 = ""
   vm = "/vm/00000000-0000-0000-0000-000000000000"
   device = ""
   control = ""
    platform-feature-multiprocessor-suspend = "1"
   error = ""
   memory = ""
    target = "524288"
   guest = ""
   hvmpv = ""
   data = ""
   description = ""
   console = ""
    limit = "1048576"
    type = "xenconsoled"
   domid = "0"
   cpu = ""
    0 = ""
     availability = "online"
   name = "Domain-0"
   device-model = ""
    1 = ""
     disable_pf = "0"
     state = "running"
   backend = ""
    vfb = ""
     1 = ""
      0 = ""
       vncunused = "1"
       domain = "HVM64-NetBSD"
       vnc = "1"
       uuid = "154d563d-7482-11ff-e49c-fd15dc25959c"
       vnclisten = "0.0.0.0"
       frontend = "/local/domain/1/device/vfb/0"
       state = "1"
       online = "1"
       frontend-id = "1"
       location = "0.0.0.0:5900"
    vbd = ""
     1 = ""
      768 = ""
       domain = "HVM64-NetBSD"
       frontend = "/local/domain/1/device/vbd/768"
       uuid = "62bfd8c9-ca3e-a427-2890-adbf23bbfc17"
       bootable = "1"
       dev = "hda"
       state = "4"
       params = "/hvm-guest/netbsd_64b.img"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "file"
       vnd = "/dev/vnd0d"
       physical-device = "3587"
       hotplug-status = "connected"
       sectors = "2048000"
       info = "0"
       sector-size = "512"
       feature-flush-cache = "1"
      832 = ""
       domain = "HVM64-NetBSD"
       frontend = "/local/domain/1/device/vbd/832"
       uuid = "57629777-1611-18c7-66a1-5b7b5e4a3024"
       bootable = "0"
       dev = "hdb"
       state = "4"
       params = "/hvm-guest/netbsd_disk2.img"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "file"
       vnd = "/dev/vnd1d"
       physical-device = "3603"
       hotplug-status = "connected"
       sectors = "204800000"
       info = "0"
       sector-size = "512"
       feature-flush-cache = "1"
    vif = ""
     1 = ""
      0 = ""
       bridge = "bridge0"
       domain = "HVM64-NetBSD"
       handle = "0"
       uuid = "683d226b-6b3c-1e1a-16c1-cd9c866aec91"
       script = "/usr/local.22573.netbsd/etc/xen/scripts/vif-bridge"
       state = "2"
       frontend = "/local/domain/1/device/vif/0"
       mac = "00:16:3e:11:14:8c"
       online = "1"
       frontend-id = "1"
       model = "ne2k_pci"
       type = "ioemu"
       feature-rx-copy = "1"
       feature-rx-flip = "1"
       hotplug-status = "connected"
    console = ""
     1 = ""
      0 = ""
       domain = "HVM64-NetBSD"
       protocol = "vt100"
       uuid = "e5c50c5f-de58-36ca-afbe-ec6a6751ef4a"
       frontend = "/local/domain/1/device/console/0"
       state = "1"
       location = "6"
       online = "1"
       frontend-id = "1"
  1 = ""
   vm = "/vm/2085991b-e8f7-8cbf-4fe0-72ce3de73abe"
   device = ""
    vfb = ""
     0 = ""
      state = "1"
      backend-id = "0"
      backend = "/local/domain/0/backend/vfb/1/0"
    vbd = ""
     768 = ""
      backend-id = "0"
      virtual-device = "768"
      device-type = "disk"
      state = "1"
      backend = "/local/domain/0/backend/vbd/1/768"
     832 = ""
      backend-id = "0"
      virtual-device = "832"
      device-type = "disk"
      state = "1"
      backend = "/local/domain/0/backend/vbd/1/832"
    vif = ""
     0 = ""
      state = "1"
      backend-id = "0"
      backend = "/local/domain/0/backend/vif/1/0"
    console = ""
     0 = ""
      state = "1"
      backend-id = "0"
      backend = "/local/domain/0/backend/console/1/0"
   control = ""
    platform-feature-multiprocessor-suspend = "1"
   error = ""
   memory = ""
    target = "3145728"
   guest = ""
   hvmpv = ""
   data = ""
   device-misc = ""
    vif = ""
     nextDeviceID = "1"
    console = ""
     nextDeviceID = "1"
   image = ""
    device-model-fifo = "/var/run/xend/dm-1-1293031772.fifo"
    device-model-pid = "784"
    suspend-cancel = "1"
   console = ""
    vnc-port = "5900"
    port = "6"
    limit = "1048576"
    type = "ioemu"
   serial = ""
    0 = ""
     tty = "/dev/pts/1"
   description = ""
   cpu = ""
    2 = ""
     availability = "online"
    0 = ""
     availability = "online"
    3 = ""
     availability = "online"
    1 = ""
     availability = "online"
   domid = "1"
   store = ""
    ring-ref = "1044476"
    port = "5"
   name = "HVM64-NetBSD"
vm = ""
 00000000-0000-0000-0000-000000000000 = ""
  on_xend_stop = "ignore"
  pool_name = "Pool-0"
  shadow_memory = "0"
  uuid = "00000000-0000-0000-0000-000000000000"
  on_reboot = "restart"
  image = "(linux (kernel '') (superpages 0) (tsc_mode 0) (nomigrate 0))"
   ostype = "linux"
   kernel = ""
   cmdline = ""
   ramdisk = ""
  on_poweroff = "destroy"
  bootloader_args = ""
  on_xend_start = "ignore"
  on_crash = "restart"
  xend = ""
   restart_count = "0"
  vcpus = "1"
  vcpu_avail = "1"
  bootloader = ""
  name = "Domain-0"
 2085991b-e8f7-8cbf-4fe0-72ce3de73abe = ""
  image = "(hvm (kernel '') (superpages 0) (tsc_mode 0) (videoram 4) (hpet 0) 
(bo\..."
   ostype = "hvm"
   kernel = ""
   cmdline = ""
   ramdisk = ""
   dmargs = "-domain-name HVM64-NetBSD -videoram 4 -vnc 
0.0.0.0:0 -vncunused -vcp\..."
   device-model = "/usr/local.22573.netbsd/libexec/qemu-dm"
   display = ""
  rtc = ""
   timeoffset = "0"
  device = ""
   vfb = ""
    0 = ""
     frontend = "/local/domain/1/device/vfb/0"
     frontend-id = "1"
     backend-id = "0"
     backend = "/local/domain/0/backend/vfb/1/0"
   vbd = ""
    768 = ""
     frontend = "/local/domain/1/device/vbd/768"
     frontend-id = "1"
     backend-id = "0"
     backend = "/local/domain/0/backend/vbd/1/768"
    832 = ""
     frontend = "/local/domain/1/device/vbd/832"
     frontend-id = "1"
     backend-id = "0"
     backend = "/local/domain/0/backend/vbd/1/832"
   vif = ""
    0 = ""
     frontend = "/local/domain/1/device/vif/0"
     frontend-id = "1"
     backend-id = "0"
     backend = "/local/domain/0/backend/vif/1/0"
   console = ""
    0 = ""
     frontend = "/local/domain/1/device/console/0"
     frontend-id = "1"
     backend-id = "0"
     backend = "/local/domain/0/backend/console/1/0"
  on_xend_stop = "ignore"
  pool_name = "Pool-0"
  shadow_memory = "28"
  uuid = "2085991b-e8f7-8cbf-4fe0-72ce3de73abe"
  on_reboot = "restart"
  start_time = "1293031772.55"
  on_poweroff = "destroy"
  bootloader_args = ""
  on_xend_start = "ignore"
  on_crash = "destroy"
  xend = ""
   restart_count = "0"
  vcpus = "4"
  vcpu_avail = "15"
  bootloader = ""
  name = "HVM64-NetBSD"


Christoph


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: qemu and xl semantics

[Ian Campbell]

On Wed, 2010-12-22 at 16:47 +0000, Christoph Egger wrote: 
> On Wednesday 22 December 2010 17:08:38 Ian Campbell wrote:
> > On Wed, 2010-12-22 at 15:42 +0000, Christoph Egger wrote:
> > > On Monday 20 December 2010 11:23:59 Ian Campbell wrote:
> > > > On Fri, 2010-12-17 at 17:21 +0000, Christoph Egger wrote:
> > > > > On Friday 17 December 2010 11:32:41 Ian Campbell wrote:
> > > > > > On Fri, 2010-12-17 at 09:49 +0000, Christoph Egger wrote:
> > > > > > > On Friday 17 December 2010 10:15:14 Ian Campbell wrote:
> > > > > > > > On Fri, 2010-12-17 at 09:00 +0000, Christoph Egger wrote:
> > > >
> > > > The only recent qemu change which I am aware of in this area is the
> > > > backport of the blkback functionality from upstream Qemu. However this
> > > > should only be enabled in cases where blkback or blktap are not
> > > > available and furthermore is tied to libxl/xl so shouldn't have done
> > > > anything on xend (although shouldn't != doesn't).
> > >
> > > Is this supposed to interact with the Dom0 PV disk backend driver ?
> >
> > The qemu disk backend is intended to be used when there is no dom0 PV
> > disk backend driver in the kernel, if there is a driver in the kernel
> > then it is intended that the kernel driver should be used.
> >
> > The background to this is that we have just gotten the basic dom0
> > support into the upstream Linux kernel and are about to start on
> > upstreaming the backend drivers. However we observed that there may not
> > be any need to upstream a block backend since a userspace implementation
> > may well suffice. It's not a done decision but the early signs are good,
> > especially compared with blktap which hits userspace anyway.
> >
> > Even if we do eventually decide to upstream the Linux blkback using the
> > disk backend in qemu tides us over for the time being.
> 
> NetBSD has a kernel backend driver since Xen 1.x days ... upstream.
> The last major change happened to it when it got updated to fit for Xen 2
> and still fits well for Xen 3/4, too.

That's cool, I didn't realise the block protocol had been so consistent
over the releases.

> > > > > xbdback backend/vbd/1/832: can't VOP_OPEN device 0xe13: 16
> > > > > 16 means EBUSY.
> > > >
> > > > But it works other than this message?
> > >
> > > Well, that means that there are now two processes which want to open the
> > > vbd simultaneously. The first one wins. And on guest shutdown the
> > > VOP_CLOSE fails.
> >
> > Right. This suggests that the qemu disk backend is being erroneously
> > activated on NetBSD when what you actually want is to use the xbdback
> > process. I think you likely need to patch libxl to do the right thing
> > for NetBSD, currently the decision is based on the presence or absence
> > of blktap, I guess it needs extending to detect NetBSD's backend too.
> 
> xbdback is actually the kernel backend driver. The conflicting processes
> are qemu-dm and the script that assigns the disk through the loopback device.
> 
> It seems to work in either way depending on which process wins...

Hrm, you shouldn't be getting both in the first place...

> 
> > Alternatively perhaps NetBSD also wants to transition to using the qemu
> > based block backend, I suppose there is benefit to be had from sharing
> > this code between Linux and NetBSD dom0.
> 
> Yes, I think so as long as it doesn't start requiring a kernel driver
> other than xbdback.

If you choose this route then there should be no kernel driver at all,
that's the point. 

> > > I did some more debugging and figured out xenbackendd runs the vif-bridge
> > > script so network with a PV driver works.
> >
> > > The scripts not running is the one associated with 'vbd' and qemu-ifup.
> > > AFAIK  qemu-dm always run qemu-ifup which is no longer the case.
> > >
> > > qemu-ifup adds tap interface to the bridge.
> >
> > This changed (on Linux) quite a while ago so I didn't think of it.
> > qemu-ifup is not used there any more and instead the vif-bridge script
> > is called for both PV and TAP devices. See xen-unstable.hg
> > 21944:0232bc7c9544, I guess either some equivalent change is needed to
> > tools/hotplug/NetBSD or the libxl bit needs to be conditional on NetBSD.
> >
> 
> hmm... so when I start the guest with 'xm' then qemu runs qemu-ifup.
> when I start the guest with 'xl' then qemu does not.
> 
> So when I adjust the script then I fix one and break the other.
> So to have it working on both the best way is to have the libxl bit
> conditional on NetBSD.

Or add the "script=no" stuff to the qemu command line in xend too so
that it behaves like libxl. I think this would be preferred (assuming it
works).

Ian.

Re: qemu and xl semantics

[Christoph Egger]

On Wednesday 22 December 2010 18:10:55 Ian Campbell wrote:
> On Wed, 2010-12-22 at 16:47 +0000, Christoph Egger wrote:
> > On Wednesday 22 December 2010 17:08:38 Ian Campbell wrote:
> > > On Wed, 2010-12-22 at 15:42 +0000, Christoph Egger wrote:
> > > > On Monday 20 December 2010 11:23:59 Ian Campbell wrote:
> > > > > On Fri, 2010-12-17 at 17:21 +0000, Christoph Egger wrote:
> > > > > > On Friday 17 December 2010 11:32:41 Ian Campbell wrote:
> > > > > > > On Fri, 2010-12-17 at 09:49 +0000, Christoph Egger wrote:
> > > > > > > > On Friday 17 December 2010 10:15:14 Ian Campbell wrote:
> > > > > > > > > On Fri, 2010-12-17 at 09:00 +0000, Christoph Egger wrote:
> > > > >
> > > > > The only recent qemu change which I am aware of in this area is the
> > > > > backport of the blkback functionality from upstream Qemu. However
> > > > > this should only be enabled in cases where blkback or blktap are
> > > > > not available and furthermore is tied to libxl/xl so shouldn't have
> > > > > done anything on xend (although shouldn't != doesn't).
> > > >
> > > > Is this supposed to interact with the Dom0 PV disk backend driver ?
> > >
> > > The qemu disk backend is intended to be used when there is no dom0 PV
> > > disk backend driver in the kernel, if there is a driver in the kernel
> > > then it is intended that the kernel driver should be used.
> > >
> > > The background to this is that we have just gotten the basic dom0
> > > support into the upstream Linux kernel and are about to start on
> > > upstreaming the backend drivers. However we observed that there may not
> > > be any need to upstream a block backend since a userspace
> > > implementation may well suffice. It's not a done decision but the early
> > > signs are good, especially compared with blktap which hits userspace
> > > anyway.
> > >
> > > Even if we do eventually decide to upstream the Linux blkback using the
> > > disk backend in qemu tides us over for the time being.
> >
> > NetBSD has a kernel backend driver since Xen 1.x days ... upstream.
> > The last major change happened to it when it got updated to fit for Xen 2
> > and still fits well for Xen 3/4, too.
>
> That's cool, I didn't realise the block protocol had been so consistent
> over the releases.

Yes. The only one pitfall of the block protocol is the limitation to 32kb 
blocksize. NetBSD uses 64kb blocksize. The NetBSD domU has to split it into
two requests and the backend driver has to merge it again.


> > > > > > xbdback backend/vbd/1/832: can't VOP_OPEN device 0xe13: 16
> > > > > > 16 means EBUSY.
> > > > >
> > > > > But it works other than this message?
> > > >
> > > > Well, that means that there are now two processes which want to open
> > > > the vbd simultaneously. The first one wins. And on guest shutdown the
> > > > VOP_CLOSE fails.
> > >
> > > Right. This suggests that the qemu disk backend is being erroneously
> > > activated on NetBSD when what you actually want is to use the xbdback
> > > process. I think you likely need to patch libxl to do the right thing
> > > for NetBSD, currently the decision is based on the presence or absence
> > > of blktap, I guess it needs extending to detect NetBSD's backend too.
> >
> > xbdback is actually the kernel backend driver. The conflicting processes
> > are qemu-dm and the script that assigns the disk through the loopback
> > device.
> >
> > It seems to work in either way depending on which process wins...
>
> Hrm, you shouldn't be getting both in the first place...

Right.

>
> > > Alternatively perhaps NetBSD also wants to transition to using the qemu
> > > based block backend, I suppose there is benefit to be had from sharing
> > > this code between Linux and NetBSD dom0.
> >
> > Yes, I think so as long as it doesn't start requiring a kernel driver
> > other than xbdback.
>
> If you choose this route then there should be no kernel driver at all,
> that's the point.

Ok. Can the qemu based block backend handle 64kb blocksize?

> > > > I did some more debugging and figured out xenbackendd runs the
> > > > vif-bridge script so network with a PV driver works.
> > > >
> > > > The scripts not running is the one associated with 'vbd' and
> > > > qemu-ifup. AFAIK  qemu-dm always run qemu-ifup which is no longer the
> > > > case.
> > > >
> > > > qemu-ifup adds tap interface to the bridge.
> > >
> > > This changed (on Linux) quite a while ago so I didn't think of it.
> > > qemu-ifup is not used there any more and instead the vif-bridge script
> > > is called for both PV and TAP devices. See xen-unstable.hg
> > > 21944:0232bc7c9544, I guess either some equivalent change is needed to
> > > tools/hotplug/NetBSD or the libxl bit needs to be conditional on
> > > NetBSD.
> >
> > hmm... so when I start the guest with 'xm' then qemu runs qemu-ifup.
> > when I start the guest with 'xl' then qemu does not.
> >
> > So when I adjust the script then I fix one and break the other.
> > So to have it working on both the best way is to have the libxl bit
> > conditional on NetBSD.
>
> Or add the "script=no" stuff to the qemu command line in xend too so
> that it behaves like libxl. I think this would be preferred (assuming it
> works).

I will give that a try and submit a patch if that works.

Christoph


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Re: qemu and xl semantics

[Ian Campbell]

On Mon, 2011-01-03 at 09:57 +0000, Christoph Egger wrote:

> 
> Yes. The only one pitfall of the block protocol is the limitation to
> 32kb blocksize. NetBSD uses 64kb blocksize. The NetBSD domU has to
> split it into two requests and the backend driver has to merge it
> again.
[...]
> Ok. Can the qemu based block backend handle 64kb blocksize? 

I wasn't aware of this limitation. Do you mean the "sector-size" value
passed from backend to frontend via xenstore?

That appears to be 512 for the qemu backend.

Ian.