Re: [Qemu-devel] [Xen-devel] Qemu disaggregation in Xen environment

[Ian Campbell <Ian.Campbell@citrix.com>]

On Tue, 2012-02-28 at 06:46 -0500, Julien Grall wrote:
> Hello,
> 
> In the current model, only one instance of qemu is running for each running HVM domain.
> 
> We are looking at disaggregating qemu to have, for example, an instance to emulate only
> network controllers, another to emulate block devices, etc...
> 
> Multiple instances of qemu would run for a single Xen domain. Each one would handle
> a subset of the hardware.
> 
> Has someone already looked at it and potentially already submitted code for qemu ?

I'm not aware of any code existing to do this.

There's a bunch of interesting stuff to do on the Xen side to make this
stuff work.

Firstly you would need to add support to the hypervisor for dispatching
I/O requests to multiple qemu instances (via multiple io req rings). I
think at the moment there is only support for a single ring (or maybe
it's one sync and one buffered I/O ring).

You'd also need to make sure that qemu explicitly requests all the MMIO
regions it is interested in. Currently the hypervisor forwards any
unknown MMIO to qemu so the explicit registration is probably not done
as consistently as it could be. If you want to have N qemus then you
need to make sure that at least N-1 of register for everything they are
interested in.

Currently the PCI config space decode is done within qemu which is a bit
tricky if you are wanting to have different emulated PCI devices in
different qemu processes. We think it would independently be an
architectural improvement to have the hypervisor do the PCI config space
decode anyway. This would allow it to forward the I/O to the correct
qemu (there are other benefits to this change, e.g. relating to PCI
passthrough and the handling of MSI configuration etc)

Then you'd need to do a bunch of toolstack level work to start and
manage the multiple qemu processes instead of the existing single
process.

So, a bunch of stuff but I think it is all reasonable to do individually
and each brings its own advantages to the architecture outside of this
project too.

> The purpose of this e-mail is to start a discussion and gather opinions on how the
> qemu developers community would like to see it implemented.
> 
> A couple of questions comes to mind:

I guess these are mostly qemu side questions. I'm not familiar enough
with the internals on that side to answer really.

> 1) How hard would it be to untangle "machine" specific (PC hardware) emulation
> from "device" specific emulation (PCI devices) ?
> 
> 2) How can we achieve disaggregation from a configuration point of view. Currently,
> Xen toolstack starts qemu, and tells qemu which device to emulate using the command
> line. I've heard about a project for creating machine description configuration files
> for QEMU which could help greatly in dividing up which hardware to emulate in which
> instance of qemu.

I've only vaguely heard about this but it certainly seems like
functionality which would benefit more than just Xen.

Ian.

>  What is the status of this project ?
> 
> Thank you for your answers,
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel