RE: RE: Rearchitecting IO Emulation for HVM Guests

RE: RE: Rearchitecting IO Emulation for HVM Guests

[Nakajima, Jun]

We've been working on re-architecting the virtual device drivers (such
as netfront, blkfront) to minimize their Xen or Linux dependencies so
that we can load these drivers on broader ranges of HVM guests. The new
architecture has Xen and non-driver Linux code in the stub domain. 

Our scope at this point is not the IO emulation as in qemu-dm, and I
believe our project and the one below will be complementary to each
other.

> port xenbus/netfront/blkfront to minios

We've ported/moved part of the code above to minios (stub domain) to
make the cleaner netfront and blkfront drivers working, and the
prototype drivers have started working with the stub domain. 

Jun
---
Intel Open Source Technology Center 

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Ian Pratt
Sent: Thursday, June 08, 2006 5:07 AM
To: Daniel Stekloff
Cc: Natasha Jarymowycz; xen-devel
Subject: [Xen-devel] RE: Rearchitecting IO Emulation for HVM Guests

> Natasha and I are interested in helping with the io emulation work for
> fully virtualized HVM guests you mentioned in your March 12th email:

Great!

I think the updated task list probably looks something like:
 1. update qemu-dm to use latest QEMU version (Christian is working on
this)
 2. modify the xen/qemu interface to use the 'v2e' approach (Leendert is
looking at this)
 3. implement 'stub domains' -- rings 1-3 in the root VMCS (normally
used by PV guests) are free for use in HVM domains (we need to have some
discussion on the best way of doing this [*])
 4. run linux in the stub domain with qemu running from an initrd
 5a. link qemu directly against the linux kernel to avoid system calls
 5b. or, link qemu against minios if we have IO support in minios.

Thanks,
Ian

[*] do we have a separate domain struct that the scheduler knows is
actually the same as another domain for scheduling/accounting purposes,
or do we modify the domain struct so hvm and pv guests don't share the
same fields? Probably the latter. 


>
http://lists.xensource.com/archives/html/xen-devel/2006-03/msg00394.html
 
> In the post, you mentioned the following tasks:
> 
> - get minios running on x86_64
> - port xenbus/netfront/blkfront to minios
> - compile qemu-dm into minios
> - redirect IO to qemu-dm in minios
> 
> Since this work started a couple months ago and some of it has been
> accomplished, we'd like to know what the current status is, what
current
> tasks there are, who's working on those tasks, and if there's anything
> you'd like us to look into.
> 
> Thanks,
> 
> Dan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: RE: Rearchitecting IO Emulation for HVM Guests

[Ronald G Minnich]

The plan 9 on xen 3 port has restarted (not by me, but I know the person 
doing it); I'd be interested in what you guys at Intel are doing with 
the re-architected drivers.

thanks

ron

RE: RE: Rearchitecting IO Emulation for HVM Guests

[Daniel Stekloff]

On Thu, 2006-06-08 at 13:02 -0700, Nakajima, Jun wrote:
> We've been working on re-architecting the virtual device drivers (such
> as netfront, blkfront) to minimize their Xen or Linux dependencies so
> that we can load these drivers on broader ranges of HVM guests. The new
> architecture has Xen and non-driver Linux code in the stub domain. 
> 
> Our scope at this point is not the IO emulation as in qemu-dm, and I
> believe our project and the one below will be complementary to each
> other.
> 
> > port xenbus/netfront/blkfront to minios
> 
> We've ported/moved part of the code above to minios (stub domain) to
> make the cleaner netfront and blkfront drivers working, and the
> prototype drivers have started working with the stub domain. 


Will the qemu io emulation that's planned for the stub domain use the
stub domain's frontends to communicate with the backends in dom0? 

Is there a public tree that you're using? Or a public tree for this
overall work, since adding changes to xen-unstable would be disruptive?

Thanks,

Dan

RE: RE: Rearchitecting IO Emulation for HVM Guests

[Nakajima, Jun]

Daniel Stekloff wrote:
> On Thu, 2006-06-08 at 13:02 -0700, Nakajima, Jun wrote:
>> We've been working on re-architecting the virtual device drivers
>> (such as netfront, blkfront) to minimize their Xen or Linux
>> dependencies so that we can load these drivers on broader ranges of
>> HVM guests. The new architecture has Xen and non-driver Linux code
>> in the stub domain. 
>> 
>> Our scope at this point is not the IO emulation as in qemu-dm, and I
>> believe our project and the one below will be complementary to each
>> other. 
>> 
>>> port xenbus/netfront/blkfront to minios
>> 
>> We've ported/moved part of the code above to minios (stub domain) to
>> make the cleaner netfront and blkfront drivers working, and the
>> prototype drivers have started working with the stub domain.
> 
> 
> Will the qemu io emulation that's planned for the stub domain use the
> stub domain's frontends to communicate with the backends in dom0?

I think that's a sensible design. The stub domain eventually needs to
talk to dom0 to read/write data, and we want to reuse the current
frontend and backend model. 

If you look at the current frontends in xenlinux, they are of course a
Linux driver. If you move them as they are to the stub domain, many
things are redundant when they work with the unmodified drivers in HVM
guests. For example, such drivers in HVM guests move data in buffers ->
the device (in the guest), then -> Linux data structure -> the Xen ring
buffer (in the stub domain). So what we need is to remove "-> Linux data
structure". 

> Is there a public tree that you're using? Or a public tree for this
> overall work, since adding changes to xen-unstable would be
> disruptive?

No, the tree requires cleanups for public consumption. At this point, we
are gathering data (not just performance) to convince ourselves that the
new architecture is the right one. 
 
> 
> Thanks,
> 
> Dan

Jun
---
Intel Open Source Technology Center

Re: RE: Rearchitecting IO Emulation for HVM Guests

[Khoa Huynh]

"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> wrote on 06/08/2006 07:07:15 AM:

> > Natasha and I are interested in helping with the io emulation work for
> > fully virtualized HVM guests you mentioned in your March 12th email:
>
> Great!
>
> I think the updated task list probably looks something like:
>  1. update qemu-dm to use latest QEMU version (Christian is working on
> this)

I've heard that Christian had successfully updated qemu-dm to 0.7.0
and is working to update it to 0.8.1.  Is there anyway we can get
the (interim) patches from Christian ?  Is there a tree somewhere with
these patches ?

>  2. modify the xen/qemu interface to use the 'v2e' approach (Leendert is
> looking at this)

Leendert has asked me to look at the xen/qemu interface (e.g. a shared,
per-vcpu buffer that can be extended to contain hvm state) for this
'v2e' approach.  Leendert has given me details on his thinking, and I
am working on them right now.  However, this stuff is also dependent
on the updated qemu-dm from Christian (hence, my inquiry above).
Thanks.

Regards,

Khoa

Re: RE: Rearchitecting IO Emulation for HVM Guests

[Natasha Jarymowycz]

On Thu, Jun 08, 2006 at 01:07:15PM +0100, Ian Pratt wrote:
> Great!
> 
> I think the updated task list probably looks something like:
>  1. update qemu-dm to use latest QEMU version (Christian is working on
> this)
>  2. modify the xen/qemu interface to use the 'v2e' approach (Leendert is
> looking at this)
>  3. implement 'stub domains' -- rings 1-3 in the root VMCS (normally
> used by PV guests) are free for use in HVM domains (we need to have some
> discussion on the best way of doing this [*])

Is anyone currently working on this?  Have any discussions been had
previously about the best way to hook the stub domain into the HVM
domain?

It looks like this work may have to be done outside of the xen-unstable
tree, especially if we're modifying xen data structures.  Will there 
be/is there a public tree available for this effort?  

Thanks,
Natasha

>  4. run linux in the stub domain with qemu running from an initrd
>  5a. link qemu directly against the linux kernel to avoid system calls
>  5b. or, link qemu against minios if we have IO support in minios.
> 
> Thanks,
> Ian
> 
> [*] do we have a separate domain struct that the scheduler knows is
> actually the same as another domain for scheduling/accounting purposes,
> or do we modify the domain struct so hvm and pv guests don't share the
> same fields? Probably the latter. 
> 

RE: RE: Rearchitecting IO Emulation for HVM Guests

[Ian Pratt]

> >  3. implement 'stub domains' -- rings 1-3 in the root VMCS (normally
> > used by PV guests) are free for use in HVM domains (we need to have
some
> > discussion on the best way of doing this [*])
> 
> Is anyone currently working on this?  Have any discussions been had
> previously about the best way to hook the stub domain into the HVM
> domain?

No one is currently working on it. It requires a bit more thought before
action.

There are two basic possibilities;

 * stub domain and the HVM domain each have their own domain structures.
The scheduler knows that they're actually the same real domain, and
hacking is required to let the stub domain map the HVM domain's memory.

 * Implement them within the same domain structure, duplicating fields
as required to make it work (effectively having PV guest and HVM areas
to the domain struct).

The 2nd is probably preferable, it needs more work though.

When scheduling the domain, the PV guest 'stub domain' would always be
run if it is runable, otherwise xen will run the hvm guest.

Ian
 
> It looks like this work may have to be done outside of the
xen-unstable
> tree, especially if we're modifying xen data structures.  Will there
> be/is there a public tree available for this effort?
> 
> Thanks,
> Natasha
> 
> >  4. run linux in the stub domain with qemu running from an initrd
> >  5a. link qemu directly against the linux kernel to avoid system
calls
> >  5b. or, link qemu against minios if we have IO support in minios.
> >
> > Thanks,
> > Ian
> >
> > [*] do we have a separate domain struct that the scheduler knows is
> > actually the same as another domain for scheduling/accounting
purposes,
> > or do we modify the domain struct so hvm and pv guests don't share
the
> > same fields? Probably the latter.
> >

RE: RE: Rearchitecting IO Emulation for HVM Guests

[Daniel Stekloff]

On Thu, 2006-06-08 at 13:02 -0700, Nakajima, Jun wrote:
> We've been working on re-architecting the virtual device drivers (such
> as netfront, blkfront) to minimize their Xen or Linux dependencies so
> that we can load these drivers on broader ranges of HVM guests. The new
> architecture has Xen and non-driver Linux code in the stub domain. 
> 
> Our scope at this point is not the IO emulation as in qemu-dm, and I
> believe our project and the one below will be complementary to each
> other.
> 
> > port xenbus/netfront/blkfront to minios
> 
> We've ported/moved part of the code above to minios (stub domain) to
> make the cleaner netfront and blkfront drivers working, and the
> prototype drivers have started working with the stub domain. 

Hi Jun,

I'm a bit confused by your response, are you working on creating a
"stub" domain along with adding the drivers to the minios or both? They
aren't the same. If you've already started on the adding a PV "stub" to
an HVM domain, how'd you define the relationship between the PV stub and
the HVM domain? What stage are you at with the effort?

Thanks,

Dan