[ANNOUNCE] Xen high-performance x86 virtualization

[ANNOUNCE] Xen high-performance x86 virtualization

[Ian Pratt]

We are pleased to announce the first stable release of the Xen
virtual machine monitor for x86, and port of Linux 2.4.22 as a
guest OS.

Xen lets you run multiple operating system images at the same
time on the same PC hardware, with unprecedented levels of
performance and resource isolation. Even under the most demanding
workloads the performance overhead is just a few percent:
considerably less than alternatives such as VMware Workstation
and User Mode Linux. This makes Xen ideal for use in providing
secure virtual hosting, or even just for running multiple OSes on
a desktop machine. 

Xen requires guest operating systems to be ported to run over
it. Crucially, only the kernel needs to be ported, and all
user-level application binaries and libraries can run
unmodified. We have a fully functional port of Linux 2.4.22
running over Xen, and regularly use it for running demanding
applications like Apache, PostgreSQL and Mozilla. Any Linux
distribution should run unmodified over the ported kernel. With
assistance from Microsoft Research, we have a port of Windows XP
to Xen nearly complete, and are planning a FreeBSD 4.8 port in
the near future.

Xen is brought to you by the University of Cambridge Computer
Laboratory Systems Research Group.  Visit the project homepage to
find out more, and download the project source code or the
XenDemoCD, a bootable `live iso' image that enables you to play
with Xen/Linux 2.4 without needing to install it on your hard
drive. The CD also contains full source code, build tools, and
benchmarks. Our SOSP paper gives an overview of the design of
Xen, and evaluates the performance against other virtualization
techniques.

Work on Xen is supported by UK EPSRC grant GR/S01894, Intel
Research Cambridge, and Microsoft Research Cambridge via an
Embedded XP IFP award.

Home page : http://www.cl.cam.ac.uk/netos/xen
SOSP paper : http://www.cl.cam.ac.uk/netos/papers/2003-xensosp.pdf

Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Lars Marowsky-Bree]

On 2003-10-02T10:17:18,
   Ian Pratt <Ian.Pratt@cl.cam.ac.uk> said:

> Work on Xen is supported by UK EPSRC grant GR/S01894, Intel
> Research Cambridge, and Microsoft Research Cambridge via an
> Embedded XP IFP award.
> 
> Home page : http://www.cl.cam.ac.uk/netos/xen
> SOSP paper : http://www.cl.cam.ac.uk/netos/papers/2003-xensosp.pdf

Hi Ian, this does sound very interesting.

Could you please elaborate on the licensing? I did not see a prominent
notice; of course the Linux kernel code will be GPL, but how about the
Xen host?


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering		ever tried. ever failed. no matter.
SuSE Labs				try again. fail again. fail better.
Research & Development, SUSE LINUX AG		-- Samuel Beckett

Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Oliver M. Bolzer]

On Thu, Oct 02, 2003 at 12:12:15PM +0200, Lars Marowsky-Bree <lmb@suse.de> wrote...
 
> Could you please elaborate on the licensing? I did not see a prominent
> notice; of course the Linux kernel code will be GPL, but how about the
> Xen host?

>From the top Xen webpage

> Xen is Open Source software, released under the terms of the GNU General
> Public License.
-- 
	Oliver M. Bolzer
	oliver@gol.com

GPG (PGP) Fingerprint = 621B 52F6 2AC1 36DB 8761  018F 8786 87AD EF50 D1FF

Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Karim Yaghmour]

Ian Pratt wrote:
> We are pleased to announce the first stable release of the Xen
> virtual machine monitor for x86, and port of Linux 2.4.22 as a
> guest OS.

Any plans for porting Xen to other architectures?

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 514-812-4145

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Keir Fraser]

> > We are pleased to announce the first stable release of the Xen
> > virtual machine monitor for x86, and port of Linux 2.4.22 as a
> > guest OS.
> 
> Any plans for porting Xen to other architectures?

Our aim was to implement an efficient VMM for commodity hardware, and
that really means x86. We're considering a port to x86-64, but so far
we're limited in man power (this is why *BSD is not yet available, for
example). 

 -- Keir

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Karim Yaghmour]

Keir Fraser wrote:
> Our aim was to implement an efficient VMM for commodity hardware, and
> that really means x86. We're considering a port to x86-64, but so far
> we're limited in man power (this is why *BSD is not yet available, for
> example). 

I understand. Obviously infinite resources are everyone's wish ;)

What type of an effort would it be to port Xen to a new architecture?
I haven't looked at the code, so I can't say, but I'm really looking for
a rough approximation: 1 man-month, 10 man-months, 100 man-months?

Thanks,

Karim
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 514-812-4145

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Keir Fraser]

> > Our aim was to implement an efficient VMM for commodity hardware, and
> > that really means x86. We're considering a port to x86-64, but so far
> > we're limited in man power (this is why *BSD is not yet available, for
> > example). 
> 
> I understand. Obviously infinite resources are everyone's wish ;)
> 
> What type of an effort would it be to port Xen to a new architecture?
> I haven't looked at the code, so I can't say, but I'm really looking for
> a rough approximation: 1 man-month, 10 man-months, 100 man-months?

I did most of the arch-dependent Xen implementation myself, and that
took around 2-3 man-months. But x86-64 is not a million miles from x86,
so it wouldn't be a complete port.

My guess is it would take one or two competent-hacker-months. :-)

 -- Keir

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[John Bradford]

Quote from Keir Fraser <Keir.Fraser@cl.cam.ac.uk>:
> 
> > > We are pleased to announce the first stable release of the Xen
> > > virtual machine monitor for x86, and port of Linux 2.4.22 as a
> > > guest OS.
> > 
> > Any plans for porting Xen to other architectures?
> 
> Our aim was to implement an efficient VMM for commodity hardware, and
> that really means x86. We're considering a port to x86-64, but so far
> we're limited in man power (this is why *BSD is not yet available, for
> example). 

Does it run recursively?  I.E. can you can Xen within a Xen virtual
machine for development and testing purposes?

John.

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Keir Fraser]

> > Our aim was to implement an efficient VMM for commodity hardware, and
> > that really means x86. We're considering a port to x86-64, but so far
> > we're limited in man power (this is why *BSD is not yet available, for
> > example). 
> 
> Does it run recursively?  I.E. can you can Xen within a Xen virtual
> machine for development and testing purposes?

No --- Xen runs on x86 but exports a different 'x86-xeno' virtual
architecture that OSes must be ported to (basically, privileged ops
must go through Xen for validation).

x86 != x86-xeno, so Xen will not run on Xen.

 -- Keir

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Theodore Ts'o]

On Thu, Oct 02, 2003 at 04:30:04PM +0100, Keir Fraser wrote:
> No --- Xen runs on x86 but exports a different 'x86-xeno' virtual
> architecture that OSes must be ported to (basically, privileged ops
> must go through Xen for validation).
> 
> x86 != x86-xeno, so Xen will not run on Xen.

And the amount of work to port the architecture-specific porions of
Xen to x86-xeno would be.....?   :-)

							- Ted

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Keir Fraser]

> And the amount of work to port the architecture-specific porions of
> Xen to x86-xeno would be.....?   :-)

To allow efficient switching in and out of Xen we take a small amount
of every virtual address space, and also grab ring 0. Since we don't
hide that from overlying OSes, we couldn't do a full recursive
implementation of Xen -- we'd run out of rings (quickly) and address
space (eventually) :-) 

Full recursion needs full virtualization. Our approach offers much
better performance in the situations where full virtualization isn't
required -- i.e., where it's feasible to distribute a ported OS.

 -- Keir

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Karim Yaghmour]

Keir Fraser wrote:
> Full recursion needs full virtualization. Our approach offers much
> better performance in the situations where full virtualization isn't
> required -- i.e., where it's feasible to distribute a ported OS.

I noticed that the SOSP Xen paper briefly mentions Jacques Gelinas' work
on VServers (http://www.solucorp.qc.ca/miscprj/s_context.hc). While
Jacques' work hasn't attracted as much public attention as other Linux
virtualization efforts, I've personally found the approach and concepts
quite fascinating. Among other things, most of the code implementing the
contexts is architecture-independent (save for a few syscalls added to
arch/*/kernel/entry.S). So, thinking aloud here, I'm wondering in what
circumstances I'd prefer using something as architecture specific as
Xen over something as architecture independent as Jacques' VServers?
(Granted VServers can't run Windows, but I'm asking this from the angle
of people looking for resource isolation in the Linux context.) Among
other things, VServers are already in use by many ISPs to provide
simultaneous hosting of many "virtual machines" on the same box while
maintaining strict separation between machines and still providing a
secure environment.

Karim

P.S.:
For those who aren't familiar with Jacques' stuff, have a look at this
document here:
http://www.solucorp.qc.ca/miscprj/s_context.hc?prjstate=1&nodoc=0
The actual concepts implemented in VServers are here:
http://www.solucorp.qc.ca/miscprj/s_context.hc?s1=2&s2=0&s3=0&s4=0&full=0&prjstate=1&nodoc=0
-- 
Author, Speaker, Developer, Consultant
Pushing Embedded and Real-Time Linux Systems Beyond the Limits
http://www.opersys.com || karim@opersys.com || 514-812-4145

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Keir Fraser]

> 
> Keir Fraser wrote:
> > Full recursion needs full virtualization. Our approach offers much
> > better performance in the situations where full virtualization isn't
> > required -- i.e., where it's feasible to distribute a ported OS.
> 
> I noticed that the SOSP Xen paper briefly mentions Jacques Gelinas' work
> on VServers (http://www.solucorp.qc.ca/miscprj/s_context.hc). While
> Jacques' work hasn't attracted as much public attention as other Linux
> virtualization efforts, I've personally found the approach and concepts
> quite fascinating. Among other things, most of the code implementing the
> contexts is architecture-independent (save for a few syscalls added to
> arch/*/kernel/entry.S). So, thinking aloud here, I'm wondering in what
> circumstances I'd prefer using something as architecture specific as
> Xen over something as architecture independent as Jacques' VServers?
> (Granted VServers can't run Windows, but I'm asking this from the angle
> of people looking for resource isolation in the Linux context.) Among
> other things, VServers are already in use by many ISPs to provide
> simultaneous hosting of many "virtual machines" on the same box while
> maintaining strict separation between machines and still providing a
> secure environment.

One of the main differences is that we provide resource isolation, so
that each virtual machine only gets the resources that its sponsor
paid for. This allows companies providing virtual servers to
provide differentiated service according to the amount paid.

 -- Keir

RE: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Paul Brett]

|    -----Original Message-----
|    From: xen-devel-admin@lists.sourceforge.net 
|    [mailto:xen-devel-admin@lists.sourceforge.net] On Behalf 
|    Of Karim Yaghmour
|    Sent: Thursday, October 02, 2003 11:42 AM
|    To: Keir Fraser
|    Cc: Theodore Ts'o; xen-devel@lists.sourceforge.net; 
|    linux-kernel@vger.kernel.org; Jacques Gelinas
|    Subject: Re: [Xen-devel] Re: [ANNOUNCE] Xen 
|    high-performance x86 virtualization
|    
|    
|    
|    Keir Fraser wrote:
|    > Full recursion needs full virtualization. Our approach 
|    offers much
|    > better performance in the situations where full 
|    virtualization isn't
|    > required -- i.e., where it's feasible to distribute a ported OS.
|    
|    ... So, thinking aloud here, I'm wondering in what
|    circumstances I'd prefer using something as architecture 
|    specific as Xen over something as architecture independent
|    as Jacques' 

And of course, you can always run many vservers inside a single Xen
domain to get the best of both worlds.

Paul Brett
PlanetLab Support
Email: paul.brett@planet-lab.org
Tel No: +1 503 712 4520

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Herbert Poetzl]

On Thu, Oct 02, 2003 at 07:53:51PM +0100, Keir Fraser wrote:
> > 
> > Keir Fraser wrote:
> > > Full recursion needs full virtualization. Our approach offers much
> > > better performance in the situations where full virtualization isn't
> > > required -- i.e., where it's feasible to distribute a ported OS.
> > 
> > I noticed that the SOSP Xen paper briefly mentions Jacques Gelinas' work
> > on VServers (http://www.solucorp.qc.ca/miscprj/s_context.hc). While
> > Jacques' work hasn't attracted as much public attention as other Linux
> > virtualization efforts, I've personally found the approach and concepts
> > quite fascinating. Among other things, most of the code implementing the
> > contexts is architecture-independent (save for a few syscalls added to
> > arch/*/kernel/entry.S). So, thinking aloud here, I'm wondering in what
> > circumstances I'd prefer using something as architecture specific as
> > Xen over something as architecture independent as Jacques' VServers?
> > (Granted VServers can't run Windows, but I'm asking this from the angle
> > of people looking for resource isolation in the Linux context.) Among
> > other things, VServers are already in use by many ISPs to provide
> > simultaneous hosting of many "virtual machines" on the same box while
> > maintaining strict separation between machines and still providing a
> > secure environment.
> 
> One of the main differences is that we provide resource isolation, so
> that each virtual machine only gets the resources that its sponsor
> paid for. This allows companies providing virtual servers to
> provide differentiated service according to the amount paid.

although the resources are usually shared in vserver environments
(this _is_ considered an advantage) Jacques' VServers allow the
administrator to limit the resources available to each virtual
server (like memory, file handles, processes, cpu power and disk
space), which should provide similar functionality ...

best,
Herbert

>  -- Keir

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Ian Pratt]

> > One of the main differences is that we provide resource isolation, so
> > that each virtual machine only gets the resources that its sponsor
> > paid for. This allows companies providing virtual servers to
> > provide differentiated service according to the amount paid.
>
> although the resources are usually shared in vserver environments
> (this _is_ considered an advantage) Jacques' VServers allow the
> administrator to limit the resources available to each virtual
> server (like memory, file handles, processes, cpu power and disk
> space), which should provide similar functionality ...

Jacques' VServers patch is really nice work, and we've got quite
a bit of experience using it. However, the level of isolation it
provides between different users is actually pretty low. It might
be OK for a machine with broadly co-operative users, but
its pretty easy for a malicious user to hose the machine, e.g. by
causing the OS to do huge amounts of work on its behalf.

It's very tough to do real isolation within a 'nix kernel. There
are just so many interactions between processes that can cause
"QoS crosstalk" e.g. competition for locks and shared resources such
as the buffer cache.

Ensim's 'private virtual server' product is the most complete
attempt at doing this that I've heard of. They've made a real
effort to account memory usage, even in the buffer cache. I'm
still sceptical as to whether it could provide isolation in the
presence of malicious user. 

We built Xen for use in the XenoServers project, which aims to
create an 'Open Infrastructure for Global Distributed Computing'.
We envisage Xenoserver execution platforms scattered across the
globe and available for any member of the public to execute code
on.  The sponsor of the code will be billed for all the resources
used or reserved during the course of execution. You'd be able to
create on-demand 'dedicated servers' with tailored amounts of
RAM, CPU, net b/w, disk b/w and disk space, and run the OS of
your choice. For example, you could buy a slice of a machine to
run a counterstrike server for a few minutes while you play a
game with a couple of friends. You'd pick the server location
such as to minimize the maximum RTT between the server and the
players.

In this kind of environment, we need to ensure people get what
they pay for, and hence we really care about isolation.  Our
approach with Xen is to provide guest OSes with strict minimum
resource guarantees, but enable them to make use of resources
that are otherwise idle on a proportional best-effort basis. For
example, we have plans to allow currently unallocated memory to
be used by domains as a 'last chance' swap cache, or as a shared
buffer cache for files accessed from a special network file
filesystem (based on OpenAFS). This way, we get most of the
benefits of sharing back, but in a controlled fashion.

The actual CPU overhead of having multiple operating systems is
actually very small. In the SOSP paper we show results for an
application running on 128 guests OSes vs. running 128 copies on
the same OS. There's a memory overhead of running multiple guest
OSes, but it's only ~20KB per-guest in Xen, and a couple of MB
for a Linux kernel and its non-pageable data structures.

As Keir says, the vservers patch applies pretty cleanly over the
xen patched linux, so you can always do a combination of both
techniques.  In fact, I know of at least one heavily used
Xen/Linux installation that has a four month uptime that has
been doing exactly this. None of the users seem to have noticed
they're running over Xen. ;-)


Ian

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[John Bradford]

Quote from Keir Fraser <Keir.Fraser@cl.cam.ac.uk>:
> One of the main differences is that we provide resource isolation, so
> that each virtual machine only gets the resources that its sponsor
> paid for. This allows companies providing virtual servers to
> provide differentiated service according to the amount paid.

You might find that that's a dis-advantage as you scale up.  Rather
than having CPU sit idle waiting for whoever paid for it to put it to
use, companies offering virtual servers would probably prefer to
oversubscribe the resources they've got, much like bandwidth is
contended on a DSL line.

For example, say you have an 8-way box running virtual servers.
Rather than sell each customer 1 cpu, let them burst to all 8 when
they need it.  Only when the load would exceed 100% do yo need to give
precedence to those who paid more.  You can sell a virtual 8 way
machine to 8 customers, as long as they realise that it is contended.
Web servers are often idle 90% of the time, but you want the best
performance when they're not.

Sharing resources such as network cards is also likely to be vital for
this to be scalable to any degree.

Both of those points, as well as virtual local networking, are what
make Z/Series boxes attractive for a lot of applications.

What is the performance penalty of running an X86-Xeno port of an OS
natively on the hardware?  Some distributions may not be prepared to
support it in addition to native X86, but if they can make X86-Xeno
their main architecture...

John.

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Keir Fraser]

> You might find that that's a dis-advantage as you scale up.  Rather
> than having CPU sit idle waiting for whoever paid for it to put it to
> use, companies offering virtual servers would probably prefer to
> oversubscribe the resources they've got, much like bandwidth is
> contended on a DSL line.
> 
> For example, say you have an 8-way box running virtual servers.
> Rather than sell each customer 1 cpu, let them burst to all 8 when
> they need it.  Only when the load would exceed 100% do yo need to give
> precedence to those who paid more.  You can sell a virtual 8 way
> machine to 8 customers, as long as they realise that it is contended.
> Web servers are often idle 90% of the time, but you want the best
> performance when they're not.
> 
> Sharing resources such as network cards is also likely to be vital for
> this to be scalable to any degree.

I should probably make it clear exactly how resources are muxed in the
current version of Xen. 

 CPU: We use a proportional fair-share scheduler (Borrowed Virtual
 Time). Guests can be be given different scheduler weights ---
 if reservations are being handed out then admission control is
 required to ensure that a given guest's fair share doesn't fall below
 its guaranteed minimum.

 Memory: Each guest is allocated a share of memory when it is started
 up. This is configurable per guest. Furthermore, a guest can request
 more memory, or give some memory back, according to current load (we
 implement a "balloon driver" as described by Waldspurger in a paper
 he presented at OSDI 2002).  

 Network + disk: Currently each domain gets a fair share (we use a
 round-robin scheduler). We plan to implement weighted share r.s.n.

We're interested in exploring reservation-based schedulers but, apart
from memory, resources currently aren't statically segregated.

Xen provides an excellent base for trying out new strategies for
resource sharing. Deploying new schedulers is not that hard --- it
just requires more man power than we have right now!

> Both of those points, as well as virtual local networking, are what
> make Z/Series boxes attractive for a lot of applications.

In fact, one of our main aims is to provide zseries-style
virtualization on x86 hardware! We've got a fair way towards this, but
we're not fully there yet :-)

> What is the performance penalty of running an X86-Xeno port of an OS
> natively on the hardware?  Some distributions may not be prepared to
> support it in addition to native X86, but if they can make X86-Xeno
> their main architecture...

Right, another good point. The performance penalty on a range of
system benchmarks (including SPEC WEB99) shows that there's up to
around 5% overhead for running x86-xen. This is far far less than any
other virtualization of x86 that is capable of running full OSes.

So yes: we envisage Xen being a next-gen replacement BIOS, in that it
could be installed on a machine and would then be responsible for
booting and supervising all OSes running on that machine. Device
drivers would be written according to a well-defined interface,
implemented within Xen, or within isolated "domains" running atop of
Xen. This kind of fits with the previous observation -- we want
zseries for x86 :-)

 -- Keir

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Andreas Hauser]

On Fri, Oct 03, 2003 at 09:13:10AM +0100, Ian Pratt wrote:
[...]
> We built Xen for use in the XenoServers project, which aims to
> create an 'Open Infrastructure for Global Distributed Computing'.
> We envisage Xenoserver execution platforms scattered across the
> globe and available for any member of the public to execute code
> on.  The sponsor of the code will be billed for all the resources
> used or reserved during the course of execution. You'd be able to
> create on-demand 'dedicated servers' with tailored amounts of
> RAM, CPU, net b/w, disk b/w and disk space, and run the OS of
> your choice. For example, you could buy a slice of a machine to
> run a counterstrike server for a few minutes while you play a
> game with a couple of friends. You'd pick the server location
> such as to minimize the maximum RTT between the server and the
> players.

So does this run over openmosix ?

aha

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Ian Pratt]

> On Fri, Oct 03, 2003 at 09:13:10AM +0100, Ian Pratt wrote:
> [...]
> > We built Xen for use in the XenoServers project, which aims to
> > create an 'Open Infrastructure for Global Distributed Computing'.
> > We envisage Xenoserver execution platforms scattered across the
> > globe and available for any member of the public to execute code
> > on.  The sponsor of the code will be billed for all the resources
> > used or reserved during the course of execution. You'd be able to
> > create on-demand 'dedicated servers' with tailored amounts of
> > RAM, CPU, net b/w, disk b/w and disk space, and run the OS of
> > your choice. For example, you could buy a slice of a machine to
> > run a counterstrike server for a few minutes while you play a
> > game with a couple of friends. You'd pick the server location
> > such as to minimize the maximum RTT between the server and the
> > players.
> 
> So does this run over openmosix ?

XenoServers is more about global-scale distributed computing
rather than clusters.

However, there's no reason why you can't apply the OpenMosix
patches to a xen -patched linux, and hence emulate a cluster on a
multi-processor machine. This isn't as daft as it sounds, as if
you have a highly parallel ccNUMA machine you might actually be
better off in terms of performance by carving it up into a set of
smaller multi processor virtual machines that you then glue back
together with something like OpenMosix.

However, although Xen is itself SMP capable, it currently doesn't
support SMP guests, though we're actively working on it.

Best,
Ian

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[John Bradford]

> > You might find that that's a dis-advantage as you scale up.
> 
> I should probably make it clear exactly how resources are muxed in the
> current version of Xen. 

[snip]

Ah, OK, I wasn't aware how far advanced you were on this.  I should
have read up a bit more :-).

> > What is the performance penalty of running an X86-Xeno port of an OS
> > natively on the hardware?  Some distributions may not be prepared to
> > support it in addition to native X86, but if they can make X86-Xeno
> > their main architecture...
> 
> Right, another good point. The performance penalty on a range of
> system benchmarks (including SPEC WEB99) shows that there's up to
> around 5% overhead for running x86-xen. This is far far less than any
> other virtualization of x86 that is capable of running full OSes.

I think we might be talking about different things -  what I meant was
if you run a kernel compiled to support Xen on X86 natively without
Xen, is there a big performance penalty, not if you run a single VM in
Xen?  I'm thinking that if $BigDistro's installation CD will install
just the same in a virtual machine as anywhere else it'll help to gain
acceptance, and if the performance penalty is low enough, there won't
be a problem there.  Or, to look at it another way, you're effectively
sidestepping the need to distribute a patched OS, because the patched
OS *is* the distributions normal OS.  On the other hand, we don't want
to introduce anything to mainline that is going to hurt embedded
applications, so I think it would be likely to be a distribution
thing.

> Device
> drivers would be written according to a well-defined interface,
> implemented within Xen, or within isolated "domains" running atop of
> Xen. This kind of fits with the previous observation -- we want
> zseries for x86 :-)

Z/Series represents a lot more than just virtualisation :-).

On the other hand, I'm sure there are installations where the
virtualisation is the only aspect that they couldn't live without.

John.

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Andreas Hauser]

On Fri, Oct 03, 2003 at 11:29:24AM +0100, Ian Pratt wrote:
> > On Fri, Oct 03, 2003 at 09:13:10AM +0100, Ian Pratt wrote:
> > [...]
> > > We built Xen for use in the XenoServers project, which aims to
> > > create an 'Open Infrastructure for Global Distributed Computing'.
> > > We envisage Xenoserver execution platforms scattered across the
> > > globe and available for any member of the public to execute code
> > > on.  The sponsor of the code will be billed for all the resources
> > > used or reserved during the course of execution. You'd be able to
> > > create on-demand 'dedicated servers' with tailored amounts of
> > > RAM, CPU, net b/w, disk b/w and disk space, and run the OS of
> > > your choice. For example, you could buy a slice of a machine to
> > > run a counterstrike server for a few minutes while you play a
> > > game with a couple of friends. You'd pick the server location
> > > such as to minimize the maximum RTT between the server and the
> > > players.
> > 
> > So does this run over openmosix ?
> 
> XenoServers is more about global-scale distributed computing
> rather than clusters.
> 
> However, there's no reason why you can't apply the OpenMosix
> patches to a xen -patched linux, and hence emulate a cluster on a
> multi-processor machine. This isn't as daft as it sounds, as if
> you have a highly parallel ccNUMA machine you might actually be
> better off in terms of performance by carving it up into a set of
> smaller multi processor virtual machines that you then glue back
> together with something like OpenMosix.

I was more thinking of running Xen on one node of a cluster,
using it as the management tool.
Each user gets his own VM on the master node,
and can only access that.
A real value would be if one could use the
cluster resources from within a e.g. freebsd guest.

So the question would be, can the processes of the
guest OS be migrated to to other nodes?
(openmosix can only migrate the userspace part of a process)

Anyways i will try it next week.

aha

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Ian Pratt]

> I was more thinking of running Xen on one node of a cluster,
> using it as the management tool.
> Each user gets his own VM on the master node,
> and can only access that.
> A real value would be if one could use the
> cluster resources from within a e.g. freebsd guest.
> 
> So the question would be, can the processes of the
> guest OS be migrated to to other nodes?
> (openmosix can only migrate the userspace part of a process)

Suspend/resume migration of guest OSes is on the "todo" list. Its
not hard, but we haven't gotten around to it.

See below for my reply to Jacob Gorm Hansen as to what needs
doing. Jacob did the NomadBIOS work, which is cool stuff, and has
many similar issues to suspend/resume in Xen.

Cheers,
Ian

---

> last year myself and a friend implemented the NomadBIOS system, a
> microkernel-based project similar in ambition to Xen, but with the
> ability to migrate guests quickly between host-machines. 
> 
> Do you plan to implement guest migration in Xen, or do you see any
> reasons why it might be difficult to do?

It's on our list of things to do, but haven't quiet got around to.

Our plan is to have the code that does the suspend/resume run in
domain 0 (the privileged domain), where it can memory map the
physical memory of other domains to read/write, and has easy
access to the disk/network etc.

The one complication with doing this on Xen is that since we
deliberately expose real resources, it's likely that the set of
physical 'machine' pages available to the domain when it is
resumed will be different from when it was suspended.

Hence, we need to re-write the OS's page tables to reflect the
change. This is actually quite straight forward since Xen already
needs to track which pages contain page directories/tables.

However, it's also necessary to update the OS's "mem_map"
structure (or equivalent) to reflect the new 'physical' to
'machine' page mapping.  This is obviously highly OS specific. We
could do this with a simple stub function in the OS, or we could
make the domain0 'resume' program OS specific. We haven't decided
which approach we prefer.  Having some some limited co-operation
from the guest OS in suspend/resume is quite sensible anyway, as
you need it to rerun DHCP etc anyway.

We'd probably modify the guest OS to cause it to evict the entire
buffer cache before asking for a suspend, to minimize the state.

Volunteers welcome ;-)

Cheers,
Ian

Re: [Xen-devel] Re: [ANNOUNCE] Xen high-performance x86 virtualization

[Doug McNaught]

John Bradford <john@grabjohn.com> writes:

> I think we might be talking about different things -  what I meant was
> if you run a kernel compiled to support Xen on X86 natively without
> Xen, is there a big performance penalty, not if you run a single VM in
> Xen? 

My understanding from reading the paper is that you can't do
this--'x86-xen' is a separate "architecture" and won't boot on the
bare metal.

-Doug