mini-guest io emulation

mini-guest io emulation

[Ian Pratt]

Folks,

At the last summit I presented a proposal for rearchitecting the way we
do io emulation for fully-virtualized (hvm) guests. I'd really like to
try and get the work to implement this underway, as it cleans up a bunch
of mess, is a prerequisite for save/restore/relocation of hvm guests,
and is a precursor to some significant performance improvements. It
involves a fair chunk of work, so we really want to try and get multiple
folk working on it.

The plan is to move the io emulation code (qemu-dm) from running as a
user-space app in domain 0 into a 'mini guest' that is effectively a
small paravirtualized guest in the root hardware context associated with
each hvm domain. 

I guess a very high-level work plan would look something like this:

* get minios running well on x86_64; add a few simple infrastructure
functions e.g. simple memory allocator. No need for any 'user space' mmu
support
* port (simplified)xenbus/netfront/blkfront to minios; test simple
net/disk IO
* implement enough infrastructure to allow qemu-dm to be compiled into
minios, calling into net/blkfront for IO.
* plumb the vmexit entry points from MMIO and in/out into minios and
hence qemu-dm

Once the above is working we'll be in good shape. We can remove all the
skany qemu-dm support from the tools as from their POV paravirt and hvm
guests will look identical. It should also be easy to implement
save/restore of hvm guests -- just save the miniguest as part of the hvm
guests', memory image. The next stage would then be to improve
performance by enhancing the device models, e.g. adding a network card
that suports jumbo frames and csum offload, and requires fewer vmexits
in operation.

How best to move forward on this? Any volunteers?

Thanks,
Ian

RE: mini-guest io emulation

[Puthiyaparambil, Aravindh]

Ian,

Gregor and I have been working on getting Mini-OS to run on x86_64. Here
is the status:

- The hypercall interface has been updated to reflect XenLinux. 
- x86_64 pagetable builder was added. 
- I am currently working on getting the scheduler to work in x86_64 mode
(I will check in something in a couple of days)
- I think Gregor is working on console support. 

Some of the other things that need to be done are:
- Fix traps.c
	a) To show more useful information.
	b) Fix pagetable walk.

- x86_32 PAE

- Look into the stack overrun issue.

We will shortly also have a couple of more people from Unisys investing
time in this effort. 

Thanks,
Aravindh

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Ian Pratt
> Sent: Sunday, March 12, 2006 4:26 PM
> To: xen-devel
> Subject: [Xen-devel] mini-guest io emulation
> 
> 
> Folks,
> 
> At the last summit I presented a proposal for rearchitecting the way
we
> do io emulation for fully-virtualized (hvm) guests. I'd really like to
> try and get the work to implement this underway, as it cleans up a
bunch
> of mess, is a prerequisite for save/restore/relocation of hvm guests,
> and is a precursor to some significant performance improvements. It
> involves a fair chunk of work, so we really want to try and get
multiple
> folk working on it.
> 
> The plan is to move the io emulation code (qemu-dm) from running as a
> user-space app in domain 0 into a 'mini guest' that is effectively a
> small paravirtualized guest in the root hardware context associated
with
> each hvm domain.
> 
> I guess a very high-level work plan would look something like this:
> 
> * get minios running well on x86_64; add a few simple infrastructure
> functions e.g. simple memory allocator. No need for any 'user space'
mmu
> support
> * port (simplified)xenbus/netfront/blkfront to minios; test simple
> net/disk IO
> * implement enough infrastructure to allow qemu-dm to be compiled into
> minios, calling into net/blkfront for IO.
> * plumb the vmexit entry points from MMIO and in/out into minios and
> hence qemu-dm
> 
> Once the above is working we'll be in good shape. We can remove all
the
> skany qemu-dm support from the tools as from their POV paravirt and
hvm
> guests will look identical. It should also be easy to implement
> save/restore of hvm guests -- just save the miniguest as part of the
hvm
> guests', memory image. The next stage would then be to improve
> performance by enhancing the device models, e.g. adding a network card
> that suports jumbo frames and csum offload, and requires fewer vmexits
> in operation.
> 
> How best to move forward on this? Any volunteers?
> 
> Thanks,
> Ian
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: mini-guest io emulation

[Nakajima, Jun]

Ian Pratt wrote:
> Folks,
> 
> At the last summit I presented a proposal for rearchitecting the way
> we do io emulation for fully-virtualized (hvm) guests. I'd really
> like to try and get the work to implement this underway, as it cleans
> up a bunch of mess, is a prerequisite for save/restore/relocation of
> hvm guests, and is a precursor to some significant performance
> improvements. It involves a fair chunk of work, so we really want to
> try and get multiple folk working on it.
> 
> The plan is to move the io emulation code (qemu-dm) from running as a
> user-space app in domain 0 into a 'mini guest' that is effectively a
> small paravirtualized guest in the root hardware context associated
> with each hvm domain.
> 
> I guess a very high-level work plan would look something like this:
> 
> * get minios running well on x86_64; add a few simple infrastructure
> functions e.g. simple memory allocator. No need for any 'user space'
> mmu support
> * port (simplified)xenbus/netfront/blkfront to minios; test simple
> net/disk IO
> * implement enough infrastructure to allow qemu-dm to be compiled into
> minios, calling into net/blkfront for IO.
> * plumb the vmexit entry points from MMIO and in/out into minios and
> hence qemu-dm

For the "mini guest", I think it could be much easier if we
substantially strip down xenlinux rather than adding (eventually) a lot
of stuff to the current mini-os, mainly because we need probably a
multi-threaded run-time environment, scheduler, memory allocator, event
handling, drivers such as xenbus/netfront/blkfront, etc. At least, I
think we can use xenlinux as the development platform. For example,
implement the qemu-dm as a driver adding the infrastructure required
(e.g. small in-kernel glibc).

> 
> Once the above is working we'll be in good shape. We can remove all
> the skany qemu-dm support from the tools as from their POV paravirt
> and hvm guests will look identical. It should also be easy to
> implement save/restore of hvm guests -- just save the miniguest as
> part of the hvm guests', memory image. The next stage would then be
> to improve performance by enhancing the device models, e.g. adding a
> network card that suports jumbo frames and csum offload, and requires
> fewer vmexits in operation.
> 
> How best to move forward on this? Any volunteers?
> 
> Thanks,
> Ian
> 

Jun
---
Intel Open Source Technology Center

Re: mini-guest io emulation

[Keir Fraser]

On 13 Mar 2006, at 05:04, Nakajima, Jun wrote:

> For the "mini guest", I think it could be much easier if we
> substantially strip down xenlinux rather than adding (eventually) a lot
> of stuff to the current mini-os, mainly because we need probably a
> multi-threaded run-time environment, scheduler, memory allocator, event
> handling, drivers such as xenbus/netfront/blkfront, etc. At least, I
> think we can use xenlinux as the development platform. For example,
> implement the qemu-dm as a driver adding the infrastructure required
> (e.g. small in-kernel glibc).

This sounds sensible. At least it allows ioemu development in parallel 
with mini-os development.

  -- Keir

Re: mini-guest io emulation

[Mark Ryden]

Hello all,

 In this occasion, may I ask : what is the puprpse of the mini-os (under extras
in the Xen tree) ? What is its usages ?
Can soembody say 3-4 sentences about xen mini-os?
Mark

On 3/13/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
>
> On 13 Mar 2006, at 05:04, Nakajima, Jun wrote:
>
> > For the "mini guest", I think it could be much easier if we
> > substantially strip down xenlinux rather than adding (eventually) a lot
> > of stuff to the current mini-os, mainly because we need probably a
> > multi-threaded run-time environment, scheduler, memory allocator, event
> > handling, drivers such as xenbus/netfront/blkfront, etc. At least, I
> > think we can use xenlinux as the development platform. For example,
> > implement the qemu-dm as a driver adding the infrastructure required
> > (e.g. small in-kernel glibc).
>
> This sounds sensible. At least it allows ioemu development in parallel
> with mini-os development.
>
>   -- Keir
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: mini-guest io emulation

[Jacob Gorm Hansen]

On 3/13/06, Nakajima, Jun <jun.nakajima@intel.com> wrote:

> For the "mini guest", I think it could be much easier if we
> substantially strip down xenlinux rather than adding (eventually) a lot
> of stuff to the current mini-os, mainly because we need probably a
> multi-threaded run-time environment, scheduler, memory allocator, event
> handling, drivers such as xenbus/netfront/blkfront, etc. At least, I
> think we can use xenlinux as the development platform. For example,
> implement the qemu-dm as a driver adding the infrastructure required
> (e.g. small in-kernel glibc).

It seems to me the main reason we needs threads and scheduling is to
interact with Xenstore. A page allocator can't be that hard to
implement. I wonder if the Xenstore API could be simplified in a way
that does not require threading, thus making the job of implementing
drivers in a min-os a bit simpler?

My feeling is that even a stripped down Linux would take some work to
maintain, at least if we wish to remove the need for hotplug scripts
for driver backends and the like from the miniLinux.

I have little interest in hvm guests, but having a functional mini-os
with an extensible, perhaps oskit- or TinyOS-like, structure would be
a huge win in a number of other situations as well. If I can I would
like to help. If the mini-os ever gets functional, I suppose it would
help to include it in the regression tests, to prevent the bit-rot it
is currently suffering from.

Jacob

Re: mini-guest io emulation

[Jacob Gorm Hansen]

On 3/13/06, Mark Ryden <markryde@gmail.com> wrote:
> Hello all,
>
>  In this occasion, may I ask : what is the puprpse of the mini-os (under extras
> in the Xen tree) ? What is its usages ?
> Can soembody say 3-4 sentences about xen mini-os?

The mini-os (or similar small Xen guests, like my own cstrap), are
useful for stuff like:

1) testing the Xen APIs.
2) as a base on which to implement or port a new Xen guest OS.
3) as a boot loader which can do 'dangerous' stuff such as ELF parsing
from guest rather than dom0 content.
4) for writing small and quickly booting Xen applications. For
instance, I once implemented a slide-show viewer in a Xen VM.

The problem with mini-os-like projects is that they need constant
maintenance to keep up with changes in the Xen API, and that the code
of such projects often becomes a bit unwieldy because a lot of bloat
is often 'faulted in' from Linux in order to make copy-pasted code
such as frontend drivers compile out of their native environment.

Jacob

Re: mini-guest io emulation

[Grzegorz Milos]

As Ian mentioned some work needs to be done to make mini-os work on 64 
bit platform but the current 32 bit version of mini-os already has:
- simple scheduler (RR like)
- memory allocator (buddy allocator, and ported xmalloc build on top of 
buddy)
- xenbus implementation
I've implemented all of it last year. Obviously if we start using it for 
I/O emulation it will have to be tested more extensively, but it seemed 
to work just when I used it for my trusted domain builder.

As Aravindh mentioned it his email, we've recently added 64-bit page 
table builder, which means that the memory allocator should now work on 
64bit platform as well.

There are still a few things on TODO list we've created with Aravindh 
but I think it should be much easier to resolve those then strip Linux 
down. And I do agree with Jacob, mini-os could be useful for many other 
things then just I/O emulation.

Cheers
Gregor

>> For the "mini guest", I think it could be much easier if we
>> substantially strip down xenlinux rather than adding (eventually) a lot
>> of stuff to the current mini-os, mainly because we need probably a
>> multi-threaded run-time environment, scheduler, memory allocator, event
>> handling, drivers such as xenbus/netfront/blkfront, etc. At least, I
>> think we can use xenlinux as the development platform. For example,
>> implement the qemu-dm as a driver adding the infrastructure required
>> (e.g. small in-kernel glibc).
> 
> It seems to me the main reason we needs threads and scheduling is to
> interact with Xenstore. A page allocator can't be that hard to
> implement. I wonder if the Xenstore API could be simplified in a way
> that does not require threading, thus making the job of implementing
> drivers in a min-os a bit simpler?
> 
> My feeling is that even a stripped down Linux would take some work to
> maintain, at least if we wish to remove the need for hotplug scripts
> for driver backends and the like from the miniLinux.
> 
> I have little interest in hvm guests, but having a functional mini-os
> with an extensible, perhaps oskit- or TinyOS-like, structure would be
> a huge win in a number of other situations as well. If I can I would
> like to help. If the mini-os ever gets functional, I suppose it would
> help to include it in the regression tests, to prevent the bit-rot it
> is currently suffering from.
> 
> Jacob
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: mini-guest io emulation

[Anthony Liguori]

Nakajima, Jun wrote:
> For the "mini guest", I think it could be much easier if we
> substantially strip down xenlinux rather than adding (eventually) a lot
> of stuff to the current mini-os, mainly because we need probably a
> multi-threaded run-time environment, scheduler, memory allocator, event
> handling, drivers such as xenbus/netfront/blkfront, etc. At least, I
> think we can use xenlinux as the development platform. For example,
> implement the qemu-dm as a driver adding the infrastructure required
> (e.g. small in-kernel glibc).
>   
Once you get past vl.c, qemu-dm has very little reliance on glibc 
functions.  Since we're only trying to do hardware emulation here, I'd 
expect that vl.c would not be included.

I suspect stripping down Linux is going to prove harder in the long 
run.  As Jacob mentioned, you only really need a simple page allocator.  
The only reasons I can think of to use threads is XenBus (threads 
shouldn't be required to implement it) and asynchronous IO.

I think an interesting alternative for AIO would actually be to create 
another VCPU specifically for the mini-os code to run in.  The 
physically analogy is sane and if you truly do need more parallelism you 
can always just use two VCPUs.

Regards,

Anthony Liguori
>> Once the above is working we'll be in good shape. We can remove all
>> the skany qemu-dm support from the tools as from their POV paravirt
>> and hvm guests will look identical. It should also be easy to
>> implement save/restore of hvm guests -- just save the miniguest as
>> part of the hvm guests', memory image. The next stage would then be
>> to improve performance by enhancing the device models, e.g. adding a
>> network card that suports jumbo frames and csum offload, and requires
>> fewer vmexits in operation.
>>
>> How best to move forward on this? Any volunteers?
>>
>> Thanks,
>> Ian
>>
>>     
>
> Jun
> ---
> Intel Open Source Technology Center
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   

RE: mini-guest io emulation

[Nakajima, Jun]

Anthony Liguori wrote:
> Nakajima, Jun wrote:
>> For the "mini guest", I think it could be much easier if we
>> substantially strip down xenlinux rather than adding (eventually) a
>> lot of stuff to the current mini-os, mainly because we need probably
>> a multi-threaded run-time environment, scheduler, memory allocator,
>> event handling, drivers such as xenbus/netfront/blkfront, etc. At
>> least, I think we can use xenlinux as the development platform. For
>> example, implement the qemu-dm as a driver adding the infrastructure
>> required (e.g. small in-kernel glibc). 
>> 
> Once you get past vl.c, qemu-dm has very little reliance on glibc
> functions.  Since we're only trying to do hardware emulation here, I'd
> expect that vl.c would not be included.

I think it depends on how much code we borrow from Linux to get the
netfront/blkfront working. We don't need the plumbing code for
user-level apps, but we need to identify which networking/block layer
code we need for hardware emulation (and virutal device drivers, such as
VBD/VIF for HVM guests). The dependencies in the code often cause the
domino effect, ending up with a lot of codes.

> 
> I suspect stripping down Linux is going to prove harder in the long
> run.  As Jacob mentioned, you only really need a simple page
> allocator. The only reasons I can think of to use threads is XenBus
> (threads shouldn't be required to implement it) and asynchronous IO.
I agree that in the long run Linux-based mini-OS will be harder, and it
depends on how soon we need it. If it's a couple of months of timeframe
(which was in my mind), I would choose xenlinux to start with -- cut the
code as much as possible so that we only use the kernel threads and
drivers. You can easily tell if you broke it because you start from
something working fine. 

The reason we need threads and scheduling is to handle simultaneous I/O
requests from SMP HVM guests. Getting reliable SMP environment is not a
trivial thing.

Jun
---
Intel Open Source Technology Center

Re: mini-guest io emulation

[Tristan Gingold]

Le Dimanche 12 Mars 2006 22:26, Ian Pratt a écrit :
> Folks,
>
> At the last summit I presented a proposal for rearchitecting the way we
> do io emulation for fully-virtualized (hvm) guests. 
[...]
>
> How best to move forward on this? Any volunteers?
Just a comment from an xen/ia64 member.
xen/ia64 has a vmx-like feature named VTi.  The implementation of this feature 
relies heavily on the current vmx/hvm implementation: it uses qemu-dm.

If VMX switches to mini-os, this also means a lot of works for ia64.  And the 
Xen/ia64 team is very small compared to Xen/x86.

So either we should have two ways: current qemu-dm and mini-os, or we will 
have to write our own mini-os.

[Joke: As a consequence AMD may like mini-os while Intel may prefer current 
qemu :-)]

Tristan.

Re: mini-guest io emulation

[Edwin Zhai]

Ian,
have some questions for mini-os io emulation
1. seems dom0 still need do the user interaction( screen painting, kbd/mouse
detection), so how much performance gain with extra dom switch(dom0<->mini
os<->VMX guest) ?

2. even with mini-os io emulation, pit/pic/iopaic/lapic still sit in Hypervisior,
right?

3. we are currently doing the save/restore for VMX guest with the expection that
mini-os emulation only affect device model save/resotore. we can continue
current work (just save the qemu-dm state) and sync with mini os in future.


On Sun, Mar 12, 2006 at 09:26:26PM -0000, Ian Pratt wrote:
> 
> Folks,
> 
> At the last summit I presented a proposal for rearchitecting the way we
> do io emulation for fully-virtualized (hvm) guests. I'd really like to
> try and get the work to implement this underway, as it cleans up a bunch
> of mess, is a prerequisite for save/restore/relocation of hvm guests,
> and is a precursor to some significant performance improvements. It
> involves a fair chunk of work, so we really want to try and get multiple
> folk working on it.
> 
> The plan is to move the io emulation code (qemu-dm) from running as a
> user-space app in domain 0 into a 'mini guest' that is effectively a
> small paravirtualized guest in the root hardware context associated with
> each hvm domain. 
> 
> I guess a very high-level work plan would look something like this:
> 
> * get minios running well on x86_64; add a few simple infrastructure
> functions e.g. simple memory allocator. No need for any 'user space' mmu
> support
> * port (simplified)xenbus/netfront/blkfront to minios; test simple
> net/disk IO
> * implement enough infrastructure to allow qemu-dm to be compiled into
> minios, calling into net/blkfront for IO.
> * plumb the vmexit entry points from MMIO and in/out into minios and
> hence qemu-dm
> 
> Once the above is working we'll be in good shape. We can remove all the
> skany qemu-dm support from the tools as from their POV paravirt and hvm
> guests will look identical. It should also be easy to implement
> save/restore of hvm guests -- just save the miniguest as part of the hvm
> guests', memory image. The next stage would then be to improve
> performance by enhancing the device models, e.g. adding a network card
> that suports jumbo frames and csum offload, and requires fewer vmexits
> in operation.
> 
> How best to move forward on this? Any volunteers?
> 
> Thanks,
> Ian
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
thanks,
edwin

Re: mini-guest io emulation

[Keir Fraser]

On 15 Mar 2006, at 03:03, Edwin Zhai wrote:

> 1. seems dom0 still need do the user interaction( screen painting, 
> kbd/mouse
> detection), so how much performance gain with extra dom 
> switch(dom0<->mini
> os<->VMX guest) ?

Some of those will not be critical path operations, and are moving to 
mini-os only to clean up the control interfaces in dom0 (consistent 
device interfaces for pv and hvm guests). In other cases we will still 
get better performance because we get to define the minios<->dom0 
interface and make it amenable to e.g., batching.

> 2. even with mini-os io emulation, pit/pic/iopaic/lapic still sit in 
> Hypervisior,
> right?

Yes. The only reason not to put something in Xen is because the backend 
cannot be implemented in Xen. Xen doesn't know about displays, so 
display emulation does not go in Xen, for example.

> 3. we are currently doing the save/restore for VMX guest with the 
> expection that
> mini-os emulation only affect device model save/resotore. we can 
> continue
> current work (just save the qemu-dm state) and sync with mini os in 
> future.

yes.

  -- Keir