Xen as a kernel module

Xen as a kernel module

[Jacob Gorm Hansen]

hi,

with Xen increasingly depending on Linux for bootstrap, drivers, packet 
filtering etc., would it make sense to have the option of compiling Xen 
as a Linux kernel module, like in VMWare or coLinux?

It seems this would give similar performance to Xen 1.2, while retaining 
most of the benefits of the NGIO model (i.e. not having to port 
drivers). The only downside would be the lack of driver isolation, but 
most people would be willing to live with that is my guess (plus as long 
as there is no IO-MMU a bad driver is still able to take down the 
complete system anyhow).

I imagine this could be done in a way that would also work under other 
host-OSes, like *BSD or Windows.

Any comments?

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Kip Macy]

I'm obviously partisan, but I think that would largely defeat the
privilege de-coupling that they're trying to achieve. Not to mention
nullifying the notion of a driver domain.

			-Kip



On Tue, 25 Jan 2005, Jacob Gorm Hansen wrote:

> hi,
>
> with Xen increasingly depending on Linux for bootstrap, drivers, packet
> filtering etc., would it make sense to have the option of compiling Xen
> as a Linux kernel module, like in VMWare or coLinux?
>
> It seems this would give similar performance to Xen 1.2, while retaining
> most of the benefits of the NGIO model (i.e. not having to port
> drivers). The only downside would be the lack of driver isolation, but
> most people would be willing to live with that is my guess (plus as long
> as there is no IO-MMU a bad driver is still able to take down the
> complete system anyhow).
>
> I imagine this could be done in a way that would also work under other
> host-OSes, like *BSD or Windows.
>
> Any comments?
>
> Jacob
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Jacob Gorm Hansen]

Kip Macy wrote:
> I'm obviously partisan, but I think that would largely defeat the
> privilege de-coupling that they're trying to achieve. Not to mention
> nullifying the notion of a driver domain.

Is anyone actually using driver domains, other than dom0? It would be 
really cool if hardware vendors started shipping drivers in Xen-VM 
format, but until that happens I do not see much motivation for having 
multiple full Linuxes around just to achieve a limited amount of isolation.

It is no secret that I much preferred the old and simpler model, but I 
understand that noone likes having to port drivers, and I guess this is 
one reason for the current state of things. With Xen inside the host OS 
kernel, you would get the performance of the old model, and the ease of 
development of the new model, plus potentially the option of running Xen 
on millions of Windows-desktops (I am sure the Silicon Valley VCs would 
love to hear that ;-)).

With all the OSes currently ported to Xen, the Xen hypercall interface 
could become a de-facto standard for paravirtualized OSes, especially if 
Xen were able to run on top of Windows as well as on top of Linux.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Ronald G. Minnich]

On Tue, 25 Jan 2005, Jacob Gorm Hansen wrote:

> with Xen increasingly depending on Linux for bootstrap, drivers, packet
> filtering etc., would it make sense to have the option of compiling Xen as a
> Linux kernel module, like in VMWare or coLinux?

that's weird, I just got this message a second time, but I still don't 
think it is a good idea :-)

ah, sourceforge.

Anyway, I think the hypervisor should stand alone. Just my feeling about 
it.


ron



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Steven Hand]

>with Xen increasingly depending on Linux for bootstrap, drivers, packet 
>filtering etc., would it make sense to have the option of compiling Xen 
>as a Linux kernel module, like in VMWare or coLinux?

Huh? Which kind of vmware? Afaik the hosted (type II) versions of 
vmware when running on a linux host have some modules which get installed 
but the VMM itself is not a module. And coLinux is basically a windows
device driver which does task switching - a very clever and useful piece
of software but not really a Linux kernel module. 

Perhaps you mean: would it make sense to have a type II version of 
Xen? I certainly see no particular reason to do this, but no particular
reason not to either. I'm not sure how much code similarity there 
would be though...

>It seems this would give similar performance to Xen 1.2, while retaining 
>most of the benefits of the NGIO model (i.e. not having to port 
>drivers). 

Maybe - I guess it depends on what you mean. If you have: 

     [ VM1 ]   [ VM2 ] ....   [ VMN ] 

      [ new type II version of Xen ] 

            [ linux kernel ] 

              [ hardware ]

then you require a way for VMx to communicate the new Xen thing, 
which then needs to syscall into the linux kernel. I'm not sure 
what VMx<->Xen comms would look like, or how it would perform. If
you retain safety it seems like you might end up with the performance
of UML, which if you go for 'high performance' then you may need to 
turn off the safety catch. 

How did you see this working? 

What aspects of performance under Xen are you finding unacceptable? 

There will always be an overhead involved in running N operating 
systems and apps on a machine compared with just 1. Indeed, if 
you really want 'blistering performance' you may find that even
the overhead of a general purpose OS is too much. Application-specific
OSes can increase performance (as can dynamic application-specific code
path optimization, see e.g. synthesis, wiggin-redstone, etc). 

>The only downside would be the lack of driver isolation, but 
>most people would be willing to live with that is my guess (plus as long 
>as there is no IO-MMU a bad driver is still able to take down the 
>complete system anyhow).

Well isolation (both security and performance) are two explicit 
design goals of Xen. If you want to have the illusion of multiple
kernels without these properties, you can use linux vservers or 
BSD jail. 

We're quite keen to evolve Xen in a way which makes it easier to 
run multiple configurations, but mainly in the space that /increases/ 
isolation (e.g. driver domains) rather than the other way. 

>I imagine this could be done in a way that would also work under other 
>host-OSes, like *BSD or Windows.

Again, I'm not sure how much code-base similarity there would be with
either current Xen or the type-II variant that you propose above. 

One nice thing about a type-I (unhosted) hypervisor is that you 
are - in principle at least - independent of the OSes you host. 
This means one can have a dom0 running Linux, BSD, Plan9 or even 
Windows. With type-II hypervisors you effectively need a new 
hypervisor for every hosted OS type (e.g. VMware workstation). 


>Any comments?

It might be useful if you were to state what precisely the problem 
is that you wish to solve, why existing solutions are insufficient, 
and how your proposed solution would solve the problem.  I'm not sure 
I really understand the answers to any of these at the moment. 


cheers, 

S.



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Jacob Gorm Hansen]

Steven Hand wrote:

> Maybe - I guess it depends on what you mean. If you have: 
> 
>      [ VM1 ]   [ VM2 ] ....   [ VMN ] 
> 
>       [ new type II version of Xen ] 
> 
>             [ linux kernel ] 
> 
>               [ hardware ]

Sorry if this came out sounding as a bit of a troll, anyway, my 
suggested setup would look like this:

       [ VM1 ]   [ VM2 ] ....   [ VMN ]

           [ Xen + linux kernel ]

                [ hardware ]


> then you require a way for VMx to communicate the new Xen thing, 
> which then needs to syscall into the linux kernel. I'm not sure 
> what VMx<->Xen comms would look like, or how it would perform. If
> you retain safety it seems like you might end up with the performance
> of UML, which if you go for 'high performance' then you may need to 
> turn off the safety catch. 

Right now Xen is mapped somewhere in top of memory, I am not sure how 
domains are kept out of there, but I suppose it has to do with segments. 
The good thing about that is that hypercalls are cheap, and in Xen1.x 
I/O was cheap as well.

My suggestion/question was a) why don't we just put a full Linux up 
there, including drivers, and b) can we provide the Xen hypercall 
interface on top of other OSes as well?

> How did you see this working? 

For Linux, I would relocate it to the top X megs of memory, and I would 
merge the Xen and Linux syscall handlers, essentially supporting two 
process models under the same OS. I would not map all of memory to 
Linux, just the pages it needs for its own stuff. For a driver OS, this 
would be fine, if you want to run applications as well there would be a 
tradeoff between how much you map to Linux and how much to Xen domains.

> What aspects of performance under Xen are you finding unacceptable? 

I generally find performance acceptable, but as I said there are cases 
where there appears to be some friction against the goals of Xen (driver 
isolation) and the goals of the application (throughput, low latency).

> Well isolation (both security and performance) are two explicit 
> design goals of Xen. If you want to have the illusion of multiple
> kernels without these properties, you can use linux vservers or 
> BSD jail. 

I would argue that you could get the same level of isolation (except 
from driver isolation) if you merge the two, while achieving the same IO 
performance as the monolithic model, and still be able to reuse existing 
driver code.

>>I imagine this could be done in a way that would also work under other 
>>host-OSes, like *BSD or Windows.
> 
> 
> Again, I'm not sure how much code-base similarity there would be with
> either current Xen or the type-II variant that you propose above. 

It would still be interesting to reuse existing Xen guestOS ports on top 
of different hypervisor implementations.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Tobias Hunger]

[-- Attachment #1: Type: text/plain, Size: 4245 bytes --]

On Thursday 27 January 2005 00:19, Jacob Gorm Hansen wrote:
> Sorry if this came out sounding as a bit of a troll, anyway, my
> suggested setup would look like this:
>
>        [ VM1 ]   [ VM2 ] ....   [ VMN ]
>
>            [ Xen + linux kernel ]
>
>                 [ hardware ]

I am interested in fast VMs. Assuming that you want dom0 to be '[ Xen + linux 
kernel ]' I fail to see how your proposed architecture helps there.

> Right now Xen is mapped somewhere in top of memory, I am not sure how
> domains are kept out of there, but I suppose it has to do with segments.

As I understand this xen runs in ring0 and pushes the guests kernels one ring 
up into ring1 and then uses traps to allow the guest OSes to trap into the 
hypervisor as necessary.

> The good thing about that is that hypercalls are cheap, and in Xen1.x
> I/O was cheap as well.

Cheap where? In dom0 or the VMs?

> My suggestion/question was a) why don't we just put a full Linux up
> there, including drivers, and b) can we provide the Xen hypercall
> interface on top of other OSes as well?

a) Because that  impacts security and robustness a lot. Security and 
robustness are the two attributes I want in software, especially a kernel. 
This is even more true for a hypervisor.

b) I don't understand what you are going at in b). 

> > What aspects of performance under Xen are you finding unacceptable?
>
> I generally find performance acceptable, but as I said there are cases
> where there appears to be some friction against the goals of Xen (driver
> isolation) and the goals of the application (throughput, low latency).

Hmmm.... adding a layer of abstraction rarely improves throughput/latency. You 
do it anyway to gain flexibility. I do not like you proposal as it sacrifices 
flexibility I want for throughput/latency in a place I don't care about.

> > Well isolation (both security and performance) are two explicit
> > design goals of Xen. If you want to have the illusion of multiple
> > kernels without these properties, you can use linux vservers or
> > BSD jail.

Please keep those goals!

> I would argue that you could get the same level of isolation (except
> from driver isolation) if you merge the two, while achieving the same IO
> performance as the monolithic model, and still be able to reuse existing
> driver code.

I fail to see where monolithic kernel comes into this... I assume you are 
referring to kernel running on a real machine instead of a virtual one.

Your proposal would force me to have all network traffic pass through dom0, 
the system able to halt all VMs on the machine! I'd feel extremely nervous 
with such a setup (OK, I am paranoid;-).

You do that to improve IO performance in dom0, which is the one virtual 
machine that I do not need IO performance: dom0 is meant on my systems to be 
able to setup VMs and nothing more (currently I use 16MiB of RAM for that 
domain). All work is done in other domains. Those can no longer access 
hardware directly with your proposal, thus it would even hurt IO performance 
for me.

> It would still be interesting to reuse existing Xen guestOS ports on top
> of different hypervisor implementations.

The critical part of the hypervisor (from the perspective of a guest OS) is 
the interface. That seems well defined and wouldn't need to change with your 
proposal. So you should not need to modify domU OSes.

Your needs seem to be to have a fast dom0 and only spin of small sessions 
occasionally, doing most of the work in dom0. Mine is to have several servers 
sharing a piece of hardware. Dom0 is mostly idling while the other domains do 
the heavy lifting. My hope is that I will see more proposals about reducing 
the coupling of xen and dom0 (like being able to reboot dom0 without 
effecting the other domains). I have no relation with the xen project apart 
from using it, so I may only give my feelings.

-- 
Gruss,
Tobias

------------------------------------------------------------
Tobias Hunger           The box said: 'Windows 95 or better'
tobias@aquazul.com                     So I installed Linux.
------------------------------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Xen as a kernel module

[Mark Williamson]

> > Right now Xen is mapped somewhere in top of memory, I am not sure how
> > domains are kept out of there, but I suppose it has to do with segments.
>
> As I understand this xen runs in ring0 and pushes the guests kernels one
> ring up into ring1 and then uses traps to allow the guest OSes to trap into
> the hypervisor as necessary.

Yup.  Xen is mapped into the top 64MB of virtual address space of every 
process (just above the Linux kernel).  Segmentation is indeed used to ensure 
that:
* user processes can only see their own memory
* guest kernels can see their own memory and user process memory
* Xen can see guest kernel memory and user process memory within the guest

This is a straightforward extension of the way segments are used in vanilla 
Linux on x86.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Jacob Gorm Hansen]

Tobias Hunger wrote:
> 
>>Right now Xen is mapped somewhere in top of memory, I am not sure how
>>domains are kept out of there, but I suppose it has to do with segments.
> 
> 
> As I understand this xen runs in ring0 and pushes the guests kernels one ring 
> up into ring1 and then uses traps to allow the guest OSes to trap into the 
> hypervisor as necessary.
> 
> 
>>The good thing about that is that hypercalls are cheap, and in Xen1.x
>>I/O was cheap as well.
> 
> 
> Cheap where? In dom0 or the VMs?

Both. Assuming that context switching is an expensive operation, and 
that most of the cost comes from flushing the TLBs, the current model 
has more overhead than a model where the drivers are mapped into the 
same address space (though still protected from domU access other that 
through hypercalls) as the currently running domain. Xen already 
contains magic that allows it to be permanently mapped into running 
domains, while still being protected from accidential/malicious access.

I am calling that monolithic, because that is what standard linux does, 
and what the old Xen 1.x did. The current model is more like a 
microkernel, such as L4 or Mach (though of course a lot more clever, no 
offense people ;-))

Apart from perhaps increasing performance, I was saying it would be cool 
if the Xen-interface could be provided on platforms such as Windows or 
native Linux. People have done similar stuff before (e.g. VMWare vmmon, 
the Adeos nanokernel, coLinux), so it should be possible. It seems Xen 
would have better chances of world domination that way, if that is a 
goal. It would also save some duplication of effort in porting guestOSes 
to the different models out there.

Anyway, it was just an idea, and it is likely that with future hardware 
the Xen NGIO/microkernel model will finally be overhead-free, vendors 
will start writing Xen-VM drivers, and we will all live happily ever 
after :-).

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Mark Williamson]

On Wednesday 26 January 2005 08:41, Steven Hand wrote:
> >with Xen increasingly depending on Linux for bootstrap, drivers, packet
> >filtering etc., would it make sense to have the option of compiling Xen
> >as a Linux kernel module, like in VMWare or coLinux?
>
> Huh? Which kind of vmware? Afaik the hosted (type II) versions of
> vmware when running on a linux host have some modules which get installed
> but the VMM itself is not a module. And coLinux is basically a windows
> device driver which does task switching - a very clever and useful piece
> of software but not really a Linux kernel module.

IIRC, the coLinux device driver is available both for Windows and as a Linux 
kernel module.  The same Linux kernel can then run in either environment.

> Maybe - I guess it depends on what you mean. If you have:
>
>      [ VM1 ]   [ VM2 ] ....   [ VMN ]
>
>       [ new type II version of Xen ]
>
>             [ linux kernel ]
>
>               [ hardware ]
>
> then you require a way for VMx to communicate the new Xen thing,
> which then needs to syscall into the linux kernel. I'm not sure
> what VMx<->Xen comms would look like, or how it would perform. If
> you retain safety it seems like you might end up with the performance
> of UML, which if you go for 'high performance' then you may need to
> turn off the safety catch.

Couldn't we have the "Xen module" hijack the interrupt-handling of the host 
kernel (like the VMWare and coLinux modules themselves do AFAIK) in order to 
handle hypercalls directly?

Device communications could be handled by backend / frontend drivers 
essentially the same as the ones we use for vanilla Xen.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: Xen as a kernel module

[Neugebauer, Rolf]

> -----Original Message-----
> From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> admin@lists.sourceforge.net] On Behalf Of Jacob Gorm Hansen
> Sent: 26 January 2005 02:00
> To: Kip Macy
> Cc: Xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] Xen as a kernel module
> 
> Kip Macy wrote:
> > I'm obviously partisan, but I think that would largely defeat the
> > privilege de-coupling that they're trying to achieve. Not to mention
> > nullifying the notion of a driver domain.
> 
> Is anyone actually using driver domains, other than dom0? It would be
> really cool if hardware vendors started shipping drivers in Xen-VM
> format, but until that happens I do not see much motivation for having
> multiple full Linuxes around just to achieve a limited amount of
> isolation.

I don't think many people currently use separate driver domains and most
people run all their device drivers in dom0 and export them from there.

However, I think it would be foolish to remove the ability to run
separate driver domains from Xen as it doesn't seem to add significant
overhead (over running them in Dom0) and it has the potential of being
extremely useful. You already mentioned the shipping of device drivers,
there might also some interesting security applications. We can
currently already provide some degree of isolation against device driver
bugs and for the DMA issue there are a number of known and existing
solutions (see eg the L4 OSDI device driver paper). The grant-table
design (unfortunately not fully implemented yet) with some form of
IO-MMU should provide the rest of the isolation.

The interfaces exported by Xen (for PCI config space, IO register
access, and interrupts) should be generic enough to support other OSes
as driver domains and the virtual device interfaces should be easily
implementable in other OSes too. Again, I think it would be foolish to
limit ourselves to Linux in this regard as your original post seems to
suggest. I think it would be great to see other OSes being used as
driver domains.

> It is no secret that I much preferred the old and simpler model, but I
> understand that noone likes having to port drivers, and I guess this
is
> one reason for the current state of things. With Xen inside the host
OS
> kernel, you would get the performance of the old model, and the ease
of
> development of the new model, plus potentially the option of running
Xen
> on millions of Windows-desktops (I am sure the Silicon Valley VCs
would
> love to hear that ;-)).
> 
> With all the OSes currently ported to Xen, the Xen hypercall interface
> could become a de-facto standard for paravirtualized OSes, especially
if
> Xen were able to run on top of Windows as well as on top of Linux.

I think it would be harder to come up with OS agnostic/generic
interfaces for Xen to be used inside a host OS (essentially sliding Xen
underneath a OS) than with generic abstractions which a OS can
use...plus there is the privilege decoupling issue Kip already raised. 

The current plan to move some of the platform init code out of Xen into
a VM will certainly make this slightly more complicated (eg having the
IOAPIC initialized by a VM and then handing control back to Xen) but I
hope we will come up with a generic interface for Xen to deal with
possibly different OSes to perform this task. At least that's something
I am advocating.

Rolf



> Jacob
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive
Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Jacob Gorm Hansen]

Neugebauer, Rolf wrote:

> However, I think it would be foolish to remove the ability to run
> separate driver domains from Xen as it doesn't seem to add significant
> overhead (over running them in Dom0) and it has the potential of being
> extremely useful. You already mentioned the shipping of device drivers,
> there might also some interesting security applications. We can
> currently already provide some degree of isolation against device driver
> bugs and for the DMA issue there are a number of known and existing
> solutions (see eg the L4 OSDI device driver paper). The grant-table
> design (unfortunately not fully implemented yet) with some form of
> IO-MMU should provide the rest of the isolation.

First of all, my suggestion was to have this as a configuration option, 
or even a separate implementation, providing just the ability to host 
domUs. I am sure this is doable, as it seems to be what coLinux is doing 
with some success. I was not suggestion throwing away existing 
functionality, but to optimize for the common case, i.e. a single Linux 
providing all drivers.

Secondly, there have been repeated reports on this list of people having 
problems with lower performance in domU than in dom0, perhaps due to 
cheap hardware, perhaps just due to misconfiguration, and the figures on 
the Xen website have not been updated to reflect what the actual 
situation is, so I guess nobody knows what the overhead will look like 
for a specific type of application. I would worry about MPI-style jobs, 
where you need both low latency and high bandwidth networking, and where 
you are likely to fully utilize the TLBs as well. I cannot see how 
performance does not get hurt in this situation, when Xen needs to flush 
the TLB for every interrupt, or alternatively needs to bundle 
interrupts, thus increasing latency.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Keir Fraser]

> I would worry about MPI-style jobs, 
> where you need both low latency and high bandwidth networking, and where 
> you are likely to fully utilize the TLBs as well. I cannot see how 
> performance does not get hurt in this situation, when Xen needs to flush 
> the TLB for every interrupt, or alternatively needs to bundle 
> interrupts, thus increasing latency.

This kind of workload needs measuring to find out just how bad things
are -- it's interesting if only because it will certainly represent a
worst case scenario for us. Ultimately this should be fixed with
smarter NICs. I am optimistic suitable support may end up in high-end
NICs: manufacturers are always looking at how to differentiate their
product, and we have a few contacts we would like to follow up when we
have more time. :-)

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Mark Williamson]

> for a specific type of application. I would worry about MPI-style jobs,
> where you need both low latency and high bandwidth networking, and where

Won't many people running this sort of workload be using specialised network 
hardware anyhow?  A modern cluster interconnect (Infiniband, for instance) 
will likely support direct IO to user level applications - this could still 
be done under Xen.

Cheers,
Mark

> you are likely to fully utilize the TLBs as well. I cannot see how
> performance does not get hurt in this situation, when Xen needs to flush
> the TLB for every interrupt, or alternatively needs to bundle
> interrupts, thus increasing latency.
>
> Jacob
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Ronald G. Minnich]

On Thu, 27 Jan 2005, Mark Williamson wrote:

> Won't many people running this sort of workload be using specialised
> network hardware anyhow?  A modern cluster interconnect (Infiniband, for
> instance)  will likely support direct IO to user level applications -
> this could still be done under Xen.

that's been our assumption.

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Jacob Gorm Hansen]

Ronald G. Minnich wrote:
> 
> On Thu, 27 Jan 2005, Mark Williamson wrote:
> 
> 
>>Won't many people running this sort of workload be using specialised
>>network hardware anyhow?  A modern cluster interconnect (Infiniband, for
>>instance)  will likely support direct IO to user level applications -
>>this could still be done under Xen.
> 
> 
> that's been our assumption.

I guess so, though the clusters I have seen have all been Ethernet-only, 
due to the lower cost. Has anyone had any experience using Xen with this 
kind of hardware?

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Ronald G. Minnich]

On Fri, 28 Jan 2005, Jacob Gorm Hansen wrote:

> I guess so, though the clusters I have seen have all been Ethernet-only, due
> to the lower cost. Has anyone had any experience using Xen with this kind of
> hardware?

Adam sulmicki is doing testing of that type of thing now, and we should 
have some results in a few weeks on a dual-xeon cluster of about 128 
nodes.Part of that will involve a virtual Plan 9 cluster as well on the 
same hardware. You just gotta love Xen.

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Jacob Gorm Hansen]

Ronald G. Minnich wrote:
> 
> Adam sulmicki is doing testing of that type of thing now, and we should 
> have some results in a few weeks on a dual-xeon cluster of about 128 
> nodes.Part of that will involve a virtual Plan 9 cluster as well on the 
> same hardware. You just gotta love Xen.

I do love Xen. Not to worry ;-)

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: Xen as a kernel module

[Adam Sulmicki]

>> I guess so, though the clusters I have seen have all been Ethernet-only, due
>> to the lower cost. Has anyone had any experience using Xen with this kind of
>> hardware?
>
> Adam sulmicki is doing testing of that type of thing now, and we should
> have some results in a few weeks on a dual-xeon cluster of about 128
> nodes.


> Part of that will involve a virtual Plan 9 cluster as well

ah, learn new stuff from the list :-)


>  on the same hardware. You just gotta love Xen.


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: Xen as a kernel module

[Ronald G. Minnich]

well, I vote for Xen 2.0, i.e. keep dom0 and xen separate. My laptop runs
xen as the default and I don't see a huge hit performance wise.

Plus, I get to dream that someday plan9 will be dom0.

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: Xen as a kernel module

[Ian Pratt]

 
> Secondly, there have been repeated reports on this list of 
> people having 
> problems with lower performance in domU than in dom0, perhaps due to 
> cheap hardware, perhaps just due to misconfiguration, 

I think most of these problems were down to debugging accidently getting
enabled in the stable build. We seem to be back to normal performance
now.

I think there are still some issues with particular ioapic's, but this
code is about to get rewritten and moved into dom0 anyhow. 

> and the 
> figures on 
> the Xen website have not been updated to reflect what the actual 
> situation is, so I guess nobody knows what the overhead will 
> look like 
> for a specific type of application. 

The performance figures for SPECCPU, OSDB/PostgreSQl, SPECWeb99/Apache,
Postmark etc are pretty much identical between 2.0 and 1.2 --- see the
"Safe Hardware Access with the Xen Virtual Machine Monitor" paper. The
new IO model tends to burn more CPU for a given IO rate, but under high
load things pipeline reasonably nicely and the privilege transitions are
amortized. Latency does suffer, but it was never that great under 1.2
anyhow. Smarter NICs will help this a lot (e.g. see the Arsenic paper by
myself and Keir).

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl