Really really small xen0

Really really small xen0

[Barry Silverman]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

I have been away from the xen list for quite a while so bear with my
current level of ignorance.
 
 I was wondering if anyone has made a really minimal xen0 image. By this
I mean an image that doesn't have much more than the kernel (f/e and b/e
drivers linked in), and run from a crom or squashfs filesystem, and a
minimal set of tools running in a busybox-like init process. 
 
If necessary, maybe even have a runtime xen0 that just does I/O, and a
management privileged domain to run the mgmt tools. 

[-- Attachment #2: Type: text/html, Size: 3969 bytes --]

Re: Really really small xen0

[Mark A. Williamson]

>  I was wondering if anyone has made a really minimal xen0 image. By this
> I mean an image that doesn't have much more than the kernel (f/e and b/e
> drivers linked in), and run from a crom or squashfs filesystem, and a
> minimal set of tools running in a busybox-like init process.

I did something similar for the restartable device drivers work in the OASIS 
paper.  I cut out everything I could from the kernel config and booted with a 
tiny ramdisk, containing only a very small init (which Keir wrote to be a few 
lines of assembler!).  This was a bit more minimal than you describe - 
basically no userspace at all ;-)

To run device drivers, the domain shouldn't need any user space at all.  Its 
runtime footprint was about 3megabytes.  There's currently no way to 
configure bridging / routing this way tho - we'd need to add some control 
interface messages.

HTH,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

RE: Really really small xen0

[Barry Silverman]

Does it even need an init.exe? or a mounted file system? Couldn't the
kernel just jump into a kernel thread xen idle loop? Would it then be
possible to have this minimal xen0 linked in to xen?

I guess there needs to be some way of communicating the parameters to
the xen kernel. Is the ether bridging done in xen? or in domain 0? Maybe
it can be driven by an initrd in xen0.

Could you then still have another privileged domain do status
monitoring, and user domain startup/shutdown?


-----Original Message-----
From: maw48@hermes.cam.ac.uk [mailto:maw48@hermes.cam.ac.uk] On Behalf
Of Mark A. Williamson
Sent: Monday, November 08, 2004 10:21 AM
To: xen-devel@lists.sourceforge.net
Cc: Barry Silverman
Subject: Re: [Xen-devel] Really really small xen0

>  I was wondering if anyone has made a really minimal xen0 image. By
this
> I mean an image that doesn't have much more than the kernel (f/e and
b/e
> drivers linked in), and run from a crom or squashfs filesystem, and a
> minimal set of tools running in a busybox-like init process.

I did something similar for the restartable device drivers work in the
OASIS 
paper.  I cut out everything I could from the kernel config and booted
with a 
tiny ramdisk, containing only a very small init (which Keir wrote to be
a few 
lines of assembler!).  This was a bit more minimal than you describe - 
basically no userspace at all ;-)

To run device drivers, the domain shouldn't need any user space at all.
Its 
runtime footprint was about 3megabytes.  There's currently no way to 
configure bridging / routing this way tho - we'd need to add some
control 
interface messages.

HTH,
Mark



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Really really small xen0

[Mark A. Williamson]

> Does it even need an init.exe? or a mounted file system? 

These are necessary if you're not going to try hacking in the kernel, purely 
to stop Linux getting upset.  The init doesn't have to do anything and the 
filesystem can be empty (apart from the init).  Using a suitable filesystem, 
you can make the initrd really small.

> Couldn't the 
> kernel just jump into a kernel thread xen idle loop? Would it then be
> possible to have this minimal xen0 linked in to xen?

You might be able to hack the init kernel thread to just block but it's not 
clear there's much advantage over having a userspace init that just blocks.  
You still might find Linux wants to mount *something* at boot, though.

I can check the mini init into the unstable tree as it might be useful to some 
people someday.

> I guess there needs to be some way of communicating the parameters to
> the xen kernel. 

Yeah.  For the minimal driver kernels, we had the backend driver in the domain 
automatically set up bridging, etc.  For flexibility and cleanness, we now 
set this up all from userspace.

We already have a generic control message framework for passing messages to 
other domains from domain 0.  We'd need to add some more message types in 
order to issue bridge setup commands, etc.

> Is the ether bridging done in xen? or in domain 0?

Xen itself doesn't know / care about devices, the bridging is done in domain 
0.  The "backend" network driver runs in dom0 and creates a virtual ethernet 
device to talk to each "frontend" network interface in an unprivileged 
domain.  The standard Linux bridging / routing code is then used to bridge / 
route those virtual interfaces.  So it all happens in dom0.

> Maybe  
> it can be driven by an initrd in xen0.

You could put some extra tools in the initrd in the driver domain but you'd 
still to figure out some way of telling them how to bridge / route new VIFs 
as domains are started.  I suppose you could hack Xend to issue commands to 
the console interface, although that seems a bit skanky to me ;-)

> Could you then still have another privileged domain do status
> monitoring, and user domain startup/shutdown?

In principal you could get Xen (with some hacking to add the required 
functionality to XenLinux / Xend) do something like this.  This done, you 
could be able to run a driver for each block / net device in a separate 
domain and admin stuff in yet another.

Many of the pieces required for this are in place now.  There is a bit of a 
chicken and egg problem regarding how these domains get built in the first 
place (how does the driver domain get built without the admin domain / how 
does the admin domain load without the driver domain) but it's surmountable 
(e.g. use some large initrds!).

This is quite interesting stuff, it's just not been as high priority as some 
of the other features in the release.

HTH,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

RE: Really really small xen0

[Barry Silverman]

>>Yeah.  For the minimal driver kernels, we had the backend driver in
the domain 
>>automatically set up bridging, etc.  For flexibility and cleanness, we
now 
>>set this up all from userspace.

>>We already have a generic control message framework for passing
messages to 
>>other domains from domain 0.  We'd need to add some more message types
in 
>>order to issue bridge setup commands, etc.

Rather than have to carry python and all the rest of the toolset, would
it be possible to:
1) Save a pre-built configuration database to send to xen
and/or
2) Save a recording of the control message traffic between dom0 and the
other domains using a BIG xen0, store the recording in an initrd and
then replay it back at startup time? (and log any errors somewhere!!)

It would be nice not to have to carry any shared libraries...



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Really really small xen0

[Jacob Gorm Hansen]

Barry Silverman wrote:

> Rather than have to carry python and all the rest of the toolset, would
> it be possible to:
> 1) Save a pre-built configuration database to send to xen
> and/or
> 2) Save a recording of the control message traffic between dom0 and the
> other domains using a BIG xen0, store the recording in an initrd and
> then replay it back at startup time? (and log any errors somewhere!!)
> 
> It would be nice not to have to carry any shared libraries...

I am doing the following for my self-migration stuff:

- Building my own domain creation etc tools using straight libxc. All 
statically linked. Now I can instantiate a new domain in 0.017s.

- Sticking with Xen 1.3 which has all the drivers in ring 0, until xend 
is either massively simplified or rewritten in C.  The old way of 
handling networking was much easier for what I am doing.

- Writing my own unprived mini-os, running the tiny UIP TCP-stack, to 
receive incoming migrations, so that ultimately I can remove dom0 
completely. Almost there now, at least I can ping the thing now.

:-)

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Really really small xen0

[Nathan Lutchansky]

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]

On Mon, Nov 08, 2004 at 09:51:12AM -0500, Barry Silverman wrote:
> I was wondering if anyone has made a really minimal xen0 image. By this
> I mean an image that doesn't have much more than the kernel (f/e and b/e
> drivers linked in), and run from a crom or squashfs filesystem, and a
> minimal set of tools running in a busybox-like init process. 

I've had good luck with similar projects using the uClibc buildroot kit,
which is intended for making tiny root filesystems for embedded systems
but also works with x86 PC systems.  There's not much documentation for it
but see the CVSweb at <http://www.uclibc.org/cgi-bin/cvsweb/buildroot/> to
see what it comes with.  It can make an initrd that will boot with exactly
the software you want to run and a tmpfs for /tmp, /var and so on, but no
changes to the filesystem can be saved.  It comes with build scripts for
Python and bridge-utils, but you'd have to add Twisted and the XEN tools.

I secure my dom0 by only making it accessible over the console/serial port
and not even giving it an IP address (except on the loopback IF).  It acts
as a layer-2 bridge only.  This is still vulnerable to security bugs in
the hypervisor and VBD/VIF data paths, of course, but it's much better
than the typical config.  -Nathan

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]