Control tools work

Control tools work

[Charles Coffing]

I've been looking at the next-generation of control tools, and see some areas that I think still need some work.  I'm working on some code that will be ready "soon", but I wanted to post and get feedback on my ideas now rather than later.  Here are the things I'm working on:

1.  Providing a higher-level consistent API for domain actions (create, save, migrate, restore, pause, etc).

2.  Making the tools/libraries agnostic w.r.t. the guest OS type.

3.  Pushing rate-limiting lower into the IO streams.

4.  Making xfrd's functionality available as a library.


I'll give you some of my motivation, for each of these items:

1.  Higher-level API:  This is listed as a task in Andrew/Keir/Christian's Xen Control Tools Design Plan.  This is also becoming important to support items #2 and #4.

2.  OS agnostic:  I can give some examples of where this is not the case (migrations assume Linux; rate limiting is only Linux, etc).  But beyond specific examples, making things OS agnostic forces some concepts to be clarified in the code.  High-level commands (such as "create") logically have a fixed sequence of steps, regardless of the OS type.  Those high-level logical steps should be separate from OS-specific implementation details.  Mixing the two creates maintenance and portability headaches.

3.  Rate limiting:  Rate limiting should be available to all IO streams, regardless of OS and regardless of the current action.  Currently it is only available for a Linux domain.  I'm pushing this down the stack, so that it is more transparent (but optional still).

4.  libxfrd:  Xend and xfrd are tied pretty tightly together via an SXP-based protocol.  Once xend is gone, either xfrd will need to be reworked or the xend replacement will need to adhere to this protocol.  Since we're moving towards multiple single-purpose tools (e.g., VM-tools), continuing to maintain this protocol would be awkward.  I would prefer to have xfrd's functionality available as a library, that any tool can link against.  I still see the receiving end of xfrd running as a daemon (a small daemon using the xfrd library, that is).  But commands that want to initiate xfrd-style sends can call the library directly.  Other advantages of this are that progress reports can be done via a simple C callback, and you can get finer-grained error reporting (rather than the current success/failure code).  Once you start looking at this within a larger management framework, these things become important.


I'm refactoring / rewriting / writing code in xutil, libxc, and xfrd (although it wouldn't be hard to slip libxen in there instead of libxc if that is the ultimate direction).  I hope to have something to show in a week or two.

Cheers,
Charles

Re: Control tools work

[Christian Limpach]

On 5/31/05, Charles Coffing <ccoffing@novell.com> wrote:

> 1.  Providing a higher-level consistent API for domain actions
> (create, save, migrate, restore, pause, etc).

I'm no longer convinced there is a much higher API to have.  Except
for create and migrate, all the actions you list are already single
functions in libxc, migrate is save | restore.  It might be useful to
group several functions together for create, but it's not quite clear
how/where device configuration fits in there.

> I'm refactoring / rewriting / writing code in xutil, libxc, and xfrd
> (although it wouldn't be hard to slip libxen in there instead of libxc
> if that is the ultimate direction).  I hope to have something to show
> in a week or two.

Please note that xfrd does no longer exist in -unstable.  Also we
don't use libxutil anymore.
Xend handles the first half of a relocation itself and then runs the
xc_save or xc_restore helper programs to do the second part. During
the first part of a relocation, the xend domain configuration is
exchanged and the format of this part is specific to xend.  The
xc_save and xc_restore helpers are merely wrappers for the
xc_linux_save and xc_linux_restore functions and use pipes to
communicate with xend and write/read the virtual machine image to/from
a file handle or socket.

    christian

Re: Control tools work

[Jacob Gorm Hansen]

Charles Coffing wrote:

> 2.  Making the tools/libraries agnostic w.r.t. the guest OS type.

I recently created a small Xen guest that is able to build a guest OS
from a ramdisk containing a stage2 loader and a kernel image. My goal is
to have it load from a block device or the network as well as from the
ram disk. Starting my guest instead of vmlinuz directly would allow
unification of domain builders from the dom0 perspective, and would
prevent ELF-parsing attacks on dom0. The same guest would be used for
reviving checkpoints, so there would not need to be a special linux_load
(linux_save can be handled using self-checkpointing to a raw device btw).

If there is interest I can post the source.

Jacob

Re: Control tools work

[Grzegorz Milos]

I am interested in your code. I don't know if you will be posting the source 
to the list, but if you decide not to could you at least send it to me? I am 
currently working on trusted boot (ie TPM support, measuring new domains 
etc., we were also considering moving the domain builder out of domain0, to a 
separate domain maybe, I am not quite certain how your builder works, but it 
looks to be quite a reasonable approach).

Cheers
Gregor

> Charles Coffing wrote:
> > 2.  Making the tools/libraries agnostic w.r.t. the guest OS type.
>
> I recently created a small Xen guest that is able to build a guest OS
> from a ramdisk containing a stage2 loader and a kernel image. My goal is
> to have it load from a block device or the network as well as from the
> ram disk. Starting my guest instead of vmlinuz directly would allow
> unification of domain builders from the dom0 perspective, and would
> prevent ELF-parsing attacks on dom0. The same guest would be used for
> reviving checkpoints, so there would not need to be a special linux_load
> (linux_save can be handled using self-checkpointing to a raw device btw).
>
> If there is interest I can post the source.
>
> Jacob
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Quidquid latine dictum sit, altum viditur --- Anon

Re: Control tools work

[Charles Coffing]

Christian,

I've done a pull and now see the functionality in XendCheckpoint.py; thanks for the heads-up.

But how do you see things outside of xm & xend accessing this functionality?  In particular, I'm thinking of a CIMOM provider written in C++.  Forking/exec-ing an "xm migrate" command is less than ideal, for several reasons (progress reporting, error reporting without having to grok text, overhead on a busy server, ...)  In my ideal world, this level of functionality would be in C or C++ libraries, so you can put whatever you want on top of it, be it Python commands or C++ CIMOM code or anything else.

I don't necessarily want to revive the old Python debate, but this does complicate things.

Thanks,
Charles


>>>christian.limpach@gmail.com 05/31/05 3:07 pm >>> 
On 5/31/05, Charles Coffing <ccoffing@novell.com> wrote: 
 
>1.  Providing a higher-level consistent API for domain actions 
>(create, save, migrate, restore, pause, etc). 
 
I'm no longer convinced there is a much higher API to have.  Except 
for create and migrate, all the actions you list are already single 
functions in libxc, migrate is save | restore.  It might be useful to 
group several functions together for create, but it's not quite clear 
how/where device configuration fits in there. 
 
>I'm refactoring / rewriting / writing code in xutil, libxc, and xfrd 
>(although it wouldn't be hard to slip libxen in there instead of libxc 
>if that is the ultimate direction).  I hope to have something to show 
>in a week or two. 
 
Please note that xfrd does no longer exist in -unstable.  Also we 
don't use libxutil anymore. 
Xend handles the first half of a relocation itself and then runs the 
xc_save or xc_restore helper programs to do the second part. During 
the first part of a relocation, the xend domain configuration is 
exchanged and the format of this part is specific to xend.  The 
xc_save and xc_restore helpers are merely wrappers for the 
xc_linux_save and xc_linux_restore functions and use pipes to 
communicate with xend and write/read the virtual machine image to/from 
a file handle or socket. 
 
   christian 
 
 
Xen-devel mailing list 
Xen-devel@lists.xensource.com 
http://lists.xensource.com/xen-devel 

Re: Control tools work

[Daniel Stekloff]

On Tuesday 31 May 2005 14:29, Charles Coffing wrote:
> Christian,
>
> I've done a pull and now see the functionality in XendCheckpoint.py; thanks
> for the heads-up.
>
> But how do you see things outside of xm & xend accessing this
> functionality?  In particular, I'm thinking of a CIMOM provider written in
> C++.  Forking/exec-ing an "xm migrate" command is less than ideal, for
> several reasons (progress reporting, error reporting without having to grok
> text, overhead on a busy server, ...)  In my ideal world, this level of
> functionality would be in C or C++ libraries, so you can put whatever you
> want on top of it, be it Python commands or C++ CIMOM code or anything
> else.
>
> I don't necessarily want to revive the old Python debate, but this does
> complicate things.


Hi Charles,

Please look at vm-tools, a set of C tools for managing Xen systems. It's a 
work in progress, but it's coming along. We have single purpose C tools that 
can easily be used to do what you require, whether wrapping them with scripts 
or accessing them directly. 

Here's Anthony's post mentioned the latest version:

http://lists.xensource.com/archives/html/xen-devel/2005-05/msg01080.html

Here's where to get it:

http://www.cs.utexas.edu/users/aliguori/vm-tools/

Thanks,

Dan

Re: Control tools work

[Steven Hand]

>But how do you see things outside of xm & xend accessing this functionality?
>In particular, I'm thinking of a CIMOM provider written in C++.  
>Forking/exec-ing an "xm migrate" command is less than ideal, for several 
>reasons (progress reporting, error reporting without having to grok text, 
>overhead on a busy server, ...)  In my ideal world, this level of 
>functionality would be in C or C++ libraries, so you can put whatever you 
>want on top of it, be it Python commands or C++ CIMOM code or anything 
>else.

The basic save/restore stuff is now simplified (in particular has no 
dependencies on a callback mechanism out of the library). To do e.g.
a non-live migrate you then effectively need to: 

   * work out what the config of the domain is
   - do a save
   - use the config to create an 'empty' domain
   - restore into this, and 
   * perform final configuration tweaks

The key thing is that the steps with '*' (at least) require config 
information which xen doesn't have. You can maintain this yourself
in your C++ app or, in the new tools world (in progress), get this
from 'xenstore'. 

There's no real short-cut around this, and it has very little to 
do with python. There is a real + desired separation of concerns
between config info that xen needs to know about (domain instance 
state) and config info that a 'user' might care about (VM state). 
This latter info is currently managed by xend (one of its primary 
functions) but could equally be managed by the vmtools daemon or
an arbitrary user app. However since in most scenarios we believe 
it's sane to keep all this consistent, having a single locus for 
such config info (viz. xenostore) makes a lot of sense. 

Hope this addresses your concerns, 

cheers,

S.

Re: Control tools work

[Christian Limpach]

On 5/31/05, Steven Hand <Steven.Hand@cl.cam.ac.uk> wrote:
> The basic save/restore stuff is now simplified (in particular has no
> dependencies on a callback mechanism out of the library). To do e.g.
> a non-live migrate you then effectively need to:
> 
>    * work out what the config of the domain is
>    - do a save
>    - use the config to create an 'empty' domain
>    - restore into this, and
>    * perform final configuration tweaks

There's actually still a callback during save, when we need to tell
the domain to suspend itself.  This uses the control channel between
xend and the domain to send a control message to the domain.  We could
maybe make xc_linux_save talk directly to the domain through xcs or we
could use xenstore for suspend requests...

    christian

Re: Control tools work

[Anthony Liguori]

Charles Coffing wrote:

>Christian,
>
>I've done a pull and now see the functionality in XendCheckpoint.py; thanks for the heads-up.
>
>But how do you see things outside of xm & xend accessing this functionality?  In particular, I'm thinking of a CIMOM provider written in C++.  Forking/exec-ing an "xm migrate" command is less than ideal, for several reasons (progress reporting, error reporting without having to grok text, overhead on a busy server, ...)  In my ideal world, this level of functionality would be in C or C++ libraries, so you can put whatever you want on top of it, be it Python commands or C++ CIMOM code or anything else.
>  
>
I'm quite happy with the direction the lower level infrastructure is 
going in.  Once the store gets straightened out, the control channel 
multiplexing daemon becomes less interesting (which is the biggest 
barrier to tools entry today).

Christian's latest modifications to the xc_save/restore code were 
fantastic and we were able to start using it quickly.  I'd still like to 
see some of the libxc interfaces refactored a bit (for instance, I'd 
like xc_domain_create to do a bit less so better error conditions can be 
obtained) but I think we're going to try to start helping to improve the 
existing error paths first.

I've always thought of VM-Tools as a mechanism to determine what aspects 
of Xen needs more work to support additional management tools.

Your point about OS-agnostic interfaces is a good one--there's a bit of 
a combinatorial domain creation problem as we support more types of 
domains on more types of architecture (and even if we ever support 
things other than Linux for dom0).

Regards,

Anthony Liguori

>I don't necessarily want to revive the old Python debate, but this does complicate things.
>
>Thanks,
>Charles
>
>
>  
>
>>>>christian.limpach@gmail.com 05/31/05 3:07 pm >>> 
>>>>        
>>>>
>On 5/31/05, Charles Coffing <ccoffing@novell.com> wrote: 
> 
>  
>
>>1.  Providing a higher-level consistent API for domain actions 
>>(create, save, migrate, restore, pause, etc). 
>>    
>>
> 
>I'm no longer convinced there is a much higher API to have.  Except 
>for create and migrate, all the actions you list are already single 
>functions in libxc, migrate is save | restore.  It might be useful to 
>group several functions together for create, but it's not quite clear 
>how/where device configuration fits in there. 
> 
>  
>
>>I'm refactoring / rewriting / writing code in xutil, libxc, and xfrd 
>>(although it wouldn't be hard to slip libxen in there instead of libxc 
>>if that is the ultimate direction).  I hope to have something to show 
>>in a week or two. 
>>    
>>
> 
>Please note that xfrd does no longer exist in -unstable.  Also we 
>don't use libxutil anymore. 
>Xend handles the first half of a relocation itself and then runs the 
>xc_save or xc_restore helper programs to do the second part. During 
>the first part of a relocation, the xend domain configuration is 
>exchanged and the format of this part is specific to xend.  The 
>xc_save and xc_restore helpers are merely wrappers for the 
>xc_linux_save and xc_linux_restore functions and use pipes to 
>communicate with xend and write/read the virtual machine image to/from 
>a file handle or socket. 
> 
>   christian 
> 
> 
>Xen-devel mailing list 
>Xen-devel@lists.xensource.com 
>http://lists.xensource.com/xen-devel 
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>  
>

Re: Control tools work

[Christian Limpach]

On 5/31/05, Charles Coffing <ccoffing@novell.com> wrote:
> But how do you see things outside of xm & xend accessing
> this functionality?  In particular, I'm thinking of a CIMOM
> provider written in C++.  Forking/exec-ing an "xm migrate"
> command is less than ideal, for several reasons (progress
> reporting, error reporting without having to grok text, overhead
> on a busy server, ...)  In my ideal world, this level of
> functionality would be in C or C++ libraries, so you can put
> whatever you want on top of it, be it Python commands or
> C++ CIMOM code or anything else.

Well, the non-xend specific code needed to do a relocation is in
libxc[1] and I believe IBM's vm-tools also use this code.  This code
only saves/restores the low-level virtual machine image, it doesn't
even know about checkpointing/relocation, i.e. how the data gets on or
off the disk or to a different machine or how to configure the devices
needed by the virtual machine on the machine where the virtual machine
is restored.  All this configuration management and setup work needs
to be done by a tool like xend or a toolset like vm-tools.

xend currently handles storage and management of configuration
information and configuration of devices according to this information
and on top provides a client/server interface to control it all[2]. 
In a next step, we're going to move the storage and configuration
parts into what we call 'xenstore'.  With xenstore it will be even
easier to write simple command line tools or interface from C or C++
libraries, all the gory details of device setup will be hidden.  We
hope that with this seperation xend will be reduced to an even more
reasonable size and that there won't be a need for alternate toolsets,
or that these would at least use xenstore.

    christian

[1] I think I'll make the control pipe and error/progress reporting
filehandles function call arguments to make it easier to use
xc_linux_{save,restore} when you link libxc into your application. 
Although I really think you want to run these in a seperate process to
isolate them from your main application.  If there's real demand we
could bring back something like ioctxt to give you callbacks for
control/error/progress.
[2] and juggling around console data.

Re: Control tools work

[Charles Coffing]

Christian,

> [1] I think I'll make the control pipe and error/progress reporting 
> filehandles function call arguments to make it easier to use 
> {save,restore} when you link libxc into your application. 
> Although I really think you want to run these in a seperate process to 
> isolate them from your main application.  If there's real demand we 
> could bring back something like ioctxt to give you callbacks for 
> control/error/progress. 

Is there a specific reason you suggest running these in a separate process?  I'm wondering if this is just a general preference because isolation tends to be a good thing, or if you suspect, say, a maliciously crafted domain could somehow crash the domain builder function.

Thanks for your explanations, by the way.

Charles

Re: Control tools work

[Christian Limpach]

On Wed, Jun 01, 2005 at 07:56:54AM -0600, Charles Coffing wrote:
> > [1] I think I'll make the control pipe and error/progress reporting 
> > filehandles function call arguments to make it easier to use 
> > {save,restore} when you link libxc into your application. 
> > Although I really think you want to run these in a seperate process to 
> > isolate them from your main application.  If there's real demand we 
> > could bring back something like ioctxt to give you callbacks for 
> > control/error/progress. 
> 
> Is there a specific reason you suggest running these in a separate
> process?  I'm wondering if this is just a general preference because
> isolation tends to be a good thing, or if you suspect, say, a
> maliciously crafted domain could somehow crash the domain builder
> function.

I suspect not all error paths to be well tested.

    christian