Re: [libvirt] [RFC] libxenlight driver

Re: [libvirt] [RFC] libxenlight driver

[Jim Fehlig]

Daniel P. Berrange wrote:
> On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote:
>   
>> I'm looking into creating a driver for the new Xen xl/libxl toolstack
>> (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0
>> release.
>>
>> My first hurdle is deciding whether this should be a new driver or
>> integrated with existing xen-unified driver.  Initially I thought a new
>> driver would be a better approach - a clean break from the old code,
>> similar to the xenapi driver.  libxenlight is also stateless (no managed
>> domains), which seems like another good argument for a new driver.  But
>> libxenlight is really just another interface into the same hypervisor,
>> so in that regard it should be a xen-unified subdriver.
>>     
>
> Something on the system must be stateful, continually monitoring
> guests & taking neccessary actions ? eg If XenD isn't used, then
> what is responsible for restarting guests which crash, or performing
> core dumps on crashed guests, etc, etc ?
>   

Good questions.  I have just started looking at the new toolstack, and
frankly don't yet know how this is handled.  Adding xen-devel for
comment ...

> This would have a bearing on how best to design a libvirt driver
>
>   
>> There are certainly benefits to the xen-unified subdriver approach, e.g.
>> the existing hypervisor and xenstore subdrivers can be leveraged, the
>> former providing all the capabilities code.  But AFAIK, libxenlight and
>> xend should not be used together, so I don't think we would want the
>> xend subdriver activated if libxenlight is detected.  Supposedly xl can
>> be used as a direct replacement for xm, allowing unconditional use of
>> that subdriver.
>>
>> BTW, Ian Jackson responded [2] to some of my questions regarding
>> compatibility between old and new toolstack if you are interested.
>>
>> I'd like to hear other's opinions on a new driver vs. a xen-unified
>> subdriver.
>>     
>
> Due to the number of revisions of Xen userspace stack, and the
> need to talk to so many pieces to get an efficient driver, the
> current Xen unified driver is rather hairy. Particularly if
> XenD itself is deprecated as a control mechanism, then I'd
> go for a new standalone driver, that runs from libvirtd context
> and leverages the standard libvirt storage/network/inteface
> drivers for non-HV stuff (which I assume libxenlight doesn't
> cover).
>   

Correct.  AFAICT, libxenlight does not cover any host storage or network
management.  A libxenlight driver will be a hypervsisor driver only.

Regards,
Jim

Re: Re: [libvirt] [RFC] libxenlight driver

[Gianni Tedesco]

On Fri, 2011-01-21 at 15:48 +0000, Jim Fehlig wrote:
> Daniel P. Berrange wrote:
> > On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote:
> >   
> >> I'm looking into creating a driver for the new Xen xl/libxl toolstack
> >> (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0
> >> release.
> >>
> >> My first hurdle is deciding whether this should be a new driver or
> >> integrated with existing xen-unified driver.  Initially I thought a new
> >> driver would be a better approach - a clean break from the old code,
> >> similar to the xenapi driver.  libxenlight is also stateless (no managed
> >> domains), which seems like another good argument for a new driver.  But
> >> libxenlight is really just another interface into the same hypervisor,
> >> so in that regard it should be a xen-unified subdriver.
> >>     
> >
> > Something on the system must be stateful, continually monitoring
> > guests & taking neccessary actions ? eg If XenD isn't used, then
> > what is responsible for restarting guests which crash, or performing
> > core dumps on crashed guests, etc, etc ?
> >   
> 
> Good questions.  I have just started looking at the new toolstack, and
> frankly don't yet know how this is handled.  Adding xen-devel for
> comment ...

Yes and no, in xen, the hypervisor internal data structures combined
with xenstore is the state (pretty much anyway). The purpose of
libxenlight is to abstract away these details.

However, when a domain is created with libxenlight one must wait for a
few events to ensure the proper clean-up of the domain. The events are
basically 'domain death' or 'device ejected'. These exist so the
toolstack can handle rebooting, core dumping the domain, inserting the
next CD image, or whatever. I am not sure how libvirt handles such
events normally but you may either integrate with a poll/select based
mainloop or have threads or processes with self-pipes handling these.

So in summary, state is maintained by the xen system outside of any
libvirt driver, but handling the kind of events you mention will require
some special work to be done for domains created by libvirt.

> > This would have a bearing on how best to design a libvirt driver
> >
> >   
> >> There are certainly benefits to the xen-unified subdriver approach, e.g.
> >> the existing hypervisor and xenstore subdrivers can be leveraged, the
> >> former providing all the capabilities code.  But AFAIK, libxenlight and
> >> xend should not be used together, so I don't think we would want the
> >> xend subdriver activated if libxenlight is detected.  Supposedly xl can
> >> be used as a direct replacement for xm, allowing unconditional use of
> >> that subdriver.
> >>
> >> BTW, Ian Jackson responded [2] to some of my questions regarding
> >> compatibility between old and new toolstack if you are interested.
> >>
> >> I'd like to hear other's opinions on a new driver vs. a xen-unified
> >> subdriver.
> >>     
> >
> > Due to the number of revisions of Xen userspace stack, and the
> > need to talk to so many pieces to get an efficient driver, the
> > current Xen unified driver is rather hairy. Particularly if
> > XenD itself is deprecated as a control mechanism, then I'd
> > go for a new standalone driver, that runs from libvirtd context
> > and leverages the standard libvirt storage/network/inteface
> > drivers for non-HV stuff (which I assume libxenlight doesn't
> > cover).
> >   
> 
> Correct.  AFAICT, libxenlight does not cover any host storage or network
> management.  A libxenlight driver will be a hypervsisor driver only.

Yes definitely. We handle attaching and detaching NIC's and block
devices. Any other management is orthogonal to libxenlight (as it is for
xend).

BTW. I would also add my vote for a separate code-base for
xend/traditional vs. libxenlight drivers as there is no similarity of
interface. The xend HTTP RPC calls are unrelated to the libxl API.

In any case, please keep xen-devel in the distribution list :)

Gianni

Re: Re: [libvirt] [RFC] libxenlight driver

[Daniel P. Berrange]

On Fri, Jan 21, 2011 at 04:33:55PM +0000, Gianni Tedesco wrote:
> On Fri, 2011-01-21 at 15:48 +0000, Jim Fehlig wrote:
> > Daniel P. Berrange wrote:
> > > On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote:
> > >   
> > >> I'm looking into creating a driver for the new Xen xl/libxl toolstack
> > >> (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0
> > >> release.
> > >>
> > >> My first hurdle is deciding whether this should be a new driver or
> > >> integrated with existing xen-unified driver.  Initially I thought a new
> > >> driver would be a better approach - a clean break from the old code,
> > >> similar to the xenapi driver.  libxenlight is also stateless (no managed
> > >> domains), which seems like another good argument for a new driver.  But
> > >> libxenlight is really just another interface into the same hypervisor,
> > >> so in that regard it should be a xen-unified subdriver.
> > >>     
> > >
> > > Something on the system must be stateful, continually monitoring
> > > guests & taking neccessary actions ? eg If XenD isn't used, then
> > > what is responsible for restarting guests which crash, or performing
> > > core dumps on crashed guests, etc, etc ?
> > >   
> > 
> > Good questions.  I have just started looking at the new toolstack, and
> > frankly don't yet know how this is handled.  Adding xen-devel for
> > comment ...
> 
> Yes and no, in xen, the hypervisor internal data structures combined
> with xenstore is the state (pretty much anyway). The purpose of
> libxenlight is to abstract away these details.
> 
> However, when a domain is created with libxenlight one must wait for a
> few events to ensure the proper clean-up of the domain. The events are
> basically 'domain death' or 'device ejected'. These exist so the
> toolstack can handle rebooting, core dumping the domain, inserting the
> next CD image, or whatever. I am not sure how libvirt handles such
> events normally but you may either integrate with a poll/select based
> mainloop or have threads or processes with self-pipes handling these.
> 
> So in summary, state is maintained by the xen system outside of any
> libvirt driver, but handling the kind of events you mention will require
> some special work to be done for domains created by libvirt.

Ok, so basically the mgmt app that is using the libxenlight API
is taking the place of XenD. Thus in the new scheme libvirtd
would have to handle those kind of events. This is fine because
we already do that kind of thing with our other HV drivers for
QEMU/KVM, UserModeLinux and LXC.

> 
> > > This would have a bearing on how best to design a libvirt driver
> > >
> > >   
> > >> There are certainly benefits to the xen-unified subdriver approach, e.g.
> > >> the existing hypervisor and xenstore subdrivers can be leveraged, the
> > >> former providing all the capabilities code.  But AFAIK, libxenlight and
> > >> xend should not be used together, so I don't think we would want the
> > >> xend subdriver activated if libxenlight is detected.  Supposedly xl can
> > >> be used as a direct replacement for xm, allowing unconditional use of
> > >> that subdriver.
> > >>
> > >> BTW, Ian Jackson responded [2] to some of my questions regarding
> > >> compatibility between old and new toolstack if you are interested.
> > >>
> > >> I'd like to hear other's opinions on a new driver vs. a xen-unified
> > >> subdriver.
> > >>     
> > >
> > > Due to the number of revisions of Xen userspace stack, and the
> > > need to talk to so many pieces to get an efficient driver, the
> > > current Xen unified driver is rather hairy. Particularly if
> > > XenD itself is deprecated as a control mechanism, then I'd
> > > go for a new standalone driver, that runs from libvirtd context
> > > and leverages the standard libvirt storage/network/inteface
> > > drivers for non-HV stuff (which I assume libxenlight doesn't
> > > cover).
> > >   
> > 
> > Correct.  AFAICT, libxenlight does not cover any host storage or network
> > management.  A libxenlight driver will be a hypervsisor driver only.
> 
> Yes definitely. We handle attaching and detaching NIC's and block
> devices. Any other management is orthogonal to libxenlight (as it is for
> xend).
> 
> BTW. I would also add my vote for a separate code-base for
> xend/traditional vs. libxenlight drivers as there is no similarity of
> interface. The xend HTTP RPC calls are unrelated to the libxl API.

Agreed, now you clarified my question(s) above, I think a dedicated
libxenlight driver in libvirtd is the only sensible option.


On a related note, where does  XenAPI fit into this picture ?
Previously XenD would provide (at least some of) the XenAPI
functionality in the open source stack, while I understand
that the XenEnterprise had a separate impl of XenAPI. It
sounds like XenAPI is likely only relevant for XenEnterprise
connectivity in the future.

Regards,
Daniel

Re: Re: [libvirt] [RFC] libxenlight driver

[Ian Campbell]

On Fri, 2011-01-21 at 16:44 +0000, Daniel P. Berrange wrote:
> 
> On a related note, where does  XenAPI fit into this picture ?
> Previously XenD would provide (at least some of) the XenAPI
> functionality in the open source stack, while I understand
> that the XenEnterprise had a separate impl of XenAPI. It
> sounds like XenAPI is likely only relevant for XenEnterprise
> connectivity in the future. 

The xapi toolstack which implements XenAPI is now open source as well
and is used in XCP (as well as XenServer).

Ian.

Re: Re: [libvirt] [RFC] libxenlight driver

[Jim Fehlig]

Ian Campbell wrote:
> On Fri, 2011-01-21 at 16:44 +0000, Daniel P. Berrange wrote:
>   
>> On a related note, where does  XenAPI fit into this picture ?
>> Previously XenD would provide (at least some of) the XenAPI
>> functionality in the open source stack, while I understand
>> that the XenEnterprise had a separate impl of XenAPI. It
>> sounds like XenAPI is likely only relevant for XenEnterprise
>> connectivity in the future. 
>>     
>
> The xapi toolstack which implements XenAPI is now open source as well
> and is used in XCP (as well as XenServer).
>   

But there is no intent on putting this toolstack in the traditional Xen
releases from xen.org correct?

Jim

Re: Re: [libvirt] [RFC] libxenlight driver

[Ian Campbell]

On Fri, 2011-01-21 at 17:02 +0000, Jim Fehlig wrote:
> Ian Campbell wrote:
> > On Fri, 2011-01-21 at 16:44 +0000, Daniel P. Berrange wrote:
> >   
> >> On a related note, where does  XenAPI fit into this picture ?
> >> Previously XenD would provide (at least some of) the XenAPI
> >> functionality in the open source stack, while I understand
> >> that the XenEnterprise had a separate impl of XenAPI. It
> >> sounds like XenAPI is likely only relevant for XenEnterprise
> >> connectivity in the future. 
> >>     
> >
> > The xapi toolstack which implements XenAPI is now open source as well
> > and is used in XCP (as well as XenServer).
> >   
> 
> But there is no intent on putting this toolstack in the traditional Xen
> releases from xen.org correct?

Although XCP and xapi are themselves xen.org projects they are
maintained and released separately.

The intention is that in the future xapi will use libxl too, but still
as an external consumer not part of the main xen hg repository.

The xcp and xapi folks hang out on the xen-api@lists.xensource.com list.

Ian.

Re: Re: [libvirt] [RFC] libxenlight driver

[Ian Jackson]

Jim Fehlig writes ("Re: [Xen-devel] Re: [libvirt] [RFC] libxenlight driver"):
> Ian Campbell wrote:
> > The xapi toolstack which implements XenAPI is now open source as well
> > and is used in XCP (as well as XenServer).
> 
> But there is no intent on putting this toolstack in the traditional Xen
> releases from xen.org correct?

That's right, there isn't.

Ian.

Re: Re: [libvirt] [RFC] libxenlight driver

[Stefano Stabellini]

On Fri, 21 Jan 2011, Daniel P. Berrange wrote:
> Ok, so basically the mgmt app that is using the libxenlight API
> is taking the place of XenD. Thus in the new scheme libvirtd
> would have to handle those kind of events.

Right.
Libxenlight provides all the functions needed to do that and you can
find a reference implementation in xl (tools/libxl/xl_cmdimpl.c).


> This is fine because
> we already do that kind of thing with our other HV drivers for
> QEMU/KVM, UserModeLinux and LXC.

Good to hear.


> On a related note, where does  XenAPI fit into this picture ?
> Previously XenD would provide (at least some of) the XenAPI
> functionality in the open source stack, while I understand
> that the XenEnterprise had a separate impl of XenAPI. It
> sounds like XenAPI is likely only relevant for XenEnterprise
> connectivity in the future.
 
Nowadays XAPI (the XenServer toolstack) is Open Source and provides a
full feature implementation of the API. I think that users that need
XenAPI functionalities are probably best served by using XAPI rather
than XenD.

Re: Re: [libvirt] [RFC] libxenlight driver

[Gianni Tedesco]

On Fri, 2011-01-21 at 17:01 +0000, Stefano Stabellini wrote:
> On Fri, 21 Jan 2011, Daniel P. Berrange wrote:
> > Ok, so basically the mgmt app that is using the libxenlight API
> > is taking the place of XenD. Thus in the new scheme libvirtd
> > would have to handle those kind of events.
> 
> Right.
> Libxenlight provides all the functions needed to do that and you can
> find a reference implementation in xl (tools/libxl/xl_cmdimpl.c).
> 
> 
> > This is fine because
> > we already do that kind of thing with our other HV drivers for
> > QEMU/KVM, UserModeLinux and LXC.
> 
> Good to hear.

+1

> > On a related note, where does  XenAPI fit into this picture ?
> > Previously XenD would provide (at least some of) the XenAPI
> > functionality in the open source stack, while I understand
> > that the XenEnterprise had a separate impl of XenAPI. It
> > sounds like XenAPI is likely only relevant for XenEnterprise
> > connectivity in the future.
>  
> Nowadays XAPI (the XenServer toolstack) is Open Source and provides a
> full feature implementation of the API. I think that users that need
> XenAPI functionalities are probably best served by using XAPI rather
> than XenD.

Yes agreed, in my personal opinion, XenAPI in xend (and perhaps xend
itself) has a perilous future.

Gianni

Re: Re: [libvirt] [RFC] libxenlight driver

[Stefano Stabellini]

On Fri, 21 Jan 2011, Jim Fehlig wrote:
> Daniel P. Berrange wrote:
> > On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote:
> >   
> >> I'm looking into creating a driver for the new Xen xl/libxl toolstack
> >> (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0
> >> release.
> >>
> >> My first hurdle is deciding whether this should be a new driver or
> >> integrated with existing xen-unified driver.  Initially I thought a new
> >> driver would be a better approach - a clean break from the old code,
> >> similar to the xenapi driver.  libxenlight is also stateless (no managed
> >> domains), which seems like another good argument for a new driver.  But
> >> libxenlight is really just another interface into the same hypervisor,
> >> so in that regard it should be a xen-unified subdriver.
> >>     
> >
> > Something on the system must be stateful, continually monitoring
> > guests & taking neccessary actions ? eg If XenD isn't used, then
> > what is responsible for restarting guests which crash, or performing
> > core dumps on crashed guests, etc, etc ?
> >   
> 
> Good questions.  I have just started looking at the new toolstack, and
> frankly don't yet know how this is handled.  Adding xen-devel for
> comment ...

xenstore is stateful and the hypervisor keeps a state of the system so
there is no need for libxenlight to maintain yet another additional
state of the system on top of them: it just relies on the informations
already available through them.
xl (the minimal C toolstack on top of libxenlight) spawns a daemon per
domain to handle domain death events and cdrom events.
I should also mention that one of the design goals of libxenlight is
that no libxenlight users (toolstacks like libvirt) should need to use
anything else but libxenlight to perform Xen operations. So you
shouldn't need to call any xenctrl functions or deal with xenstore
directly.


> > This would have a bearing on how best to design a libvirt driver

I don't know much about libvirt but I would write a brand new
libxenlight driver.


> >> There are certainly benefits to the xen-unified subdriver approach, e.g.
> >> the existing hypervisor and xenstore subdrivers can be leveraged, the
> >> former providing all the capabilities code.  But AFAIK, libxenlight and
> >> xend should not be used together, so I don't think we would want the
> >> xend subdriver activated if libxenlight is detected.  Supposedly xl can
> >> be used as a direct replacement for xm, allowing unconditional use of
> >> that subdriver.
> >>
> >> BTW, Ian Jackson responded [2] to some of my questions regarding
> >> compatibility between old and new toolstack if you are interested.
> >>
> >> I'd like to hear other's opinions on a new driver vs. a xen-unified
> >> subdriver.
> >>     
> >
> > Due to the number of revisions of Xen userspace stack, and the
> > need to talk to so many pieces to get an efficient driver, the
> > current Xen unified driver is rather hairy. Particularly if
> > XenD itself is deprecated as a control mechanism, then I'd
> > go for a new standalone driver, that runs from libvirtd context
> > and leverages the standard libvirt storage/network/inteface
> > drivers for non-HV stuff (which I assume libxenlight doesn't
> > cover).
> >   
> 
> Correct.  AFAICT, libxenlight does not cover any host storage or network
> management.  A libxenlight driver will be a hypervsisor driver only.
 
Yep.
Feel free to ask any questions you might have on libxenlight to xen-devel!

Re: Re: [libvirt] [RFC] libxenlight driver

[Ian Jackson]

Stefano Stabellini writes ("Re: [Xen-devel] Re: [libvirt] [RFC] libxenlight driver"):
> I should also mention that one of the design goals of libxenlight is
> that no libxenlight users (toolstacks like libvirt) should need to use
> anything else but libxenlight to perform Xen operations. So you
> shouldn't need to call any xenctrl functions or deal with xenstore
> directly.

This is quite important.  We have tried to do this and we have
succeeded for the "xl" command's requirements, at least.

However since we currently have only one user of libxl, "xl" itself,
it is very likely that you will find that there are deficiencies in
the libxl API which make it difficult to use properly in libvirt.

We would very much like to fix these problems so please report them
all to us.  We will work with you to make libxl have the API you need
so that you can write libvirt properly.

Inevitably this will mean that the libvirt adaptation will need the
libxl from Xen 4.2, not the Xen 4.1 version which is currently in
freeze.  However in Xen 4.1 xend is still supported.

Please also send your libvirt patches to xen-devel for review, etc.

> I don't know much about libvirt but I would write a brand new
> libxenlight driver.

Yes, absolutely.  Having taken a look at the existing code in libvirt
for driving xenstore/xen/etc., none of it looks sensibly reuseable.

Ian.