[Qemu-devel] [XEN][RFC PATCH V2 00/17] QEMU disaggregation in Xen environment

[Julien Grall <julien.grall@citrix.com>]

Hello,

This patch series only concerns Xen. Another serie will come for QEMU.

I'm currently working on QEMU disaggregation in Xen environment. The
goal is to be able to running multiple QEMU for a same domain
(http://lists.xen.org/archives/html/xen-devel/2012-03/msg00299.html).

I have already sent a version of patch series few months ago:
    - QEMU: https://lists.gnu.org/archive/html/qemu-devel/2012-03/msg04401.html
    - Xen: http://lists.xen.org/archives/html/xen-devel/2012-03/msg01947.html
With the different feedbacks, I have improved both QEMU and Xen modifications.
As before, I will sent two patch series, one for QEMU the other for Xen.

Full disaggregation is not possible (one device = one QEMU) because lots
of device depends on each others. With the help of Stefano, I have defined
as possible disaggregation:
    - ui: emulate default devices (root bridge, south bridge), VGA,
    keyboard, mouse and USB
    - audio: emulate audio
    - ide: emulate disks
    - serial: emulate serial port
    - net: it possible to have multiple QEMU that emulates one or more
    network card

Of course, a same QEMU can emulate both ui and audio. Old configuration
file with qemu-xen still works.
The patch series adds an option "device_models".

Example:

builder='hvm'
memory = 1024
name = "Debian"
vcpus=1
vif = [ 'type=ioemu, bridge=eth0, mac=00:16:3e:0e:f5:ef, id=nic1' ]
disk = [ 'tap:tapdisk:qcow2:/home/xentest/works/vms/debian.img,xvda,w' ]
device_model_override = '/home/xentest/works/qemu-devel/qemu-wrapper'
device_model_version = 'qemu-xen'
device_models = [ 'name=qnet,vifs=nic1',
                  'name=qall,ui,ide' ]

It possible to override device model path for each device model. It could be
useful for debugging. For instance, 'name=qnet,vifs=nic1,path=/my/path/wrapper'.
The option "name" is used for logging filename or debugging, if it's not
specify, a number is used.

Modifications between V1 and V2:
    - rewrite libxl patch according to the new API
    - improve user experience with configuration file (avoid to specify
      bdf)
    - improve PCI hypercall: use bus, domain, device, function instead of
    bdf.
    - fix PCI config space handler
    - remove unused HVM paramaters
    - handle save/restore

Drawbacks:
    - PCI hotplug doesn't works
    - stubdomain doesn't works because old QEMU is not modify for
    disaggregation. By the way it's works on XenClient stubdomain
    - Which QEMU need to emulate Xen Platform ? It's mainly used
    to unplug network cards and disks

Possible improvements:
    - Like hvm get parameters, introduce an hypercall to retrieve shared
    pages. For the moment the server id is used
    - Specify if we want buffered I/O shared page or not (It was an idea
    of Christian Limpach)

I don't test all configurations. Comments, bug reports, ... are welcome.

Julien Grall (17):
  hvm: Modify interface to support multiple ioreq server
  hvm: Add functions to handle ioreq servers
  hvm-pci: Handle PCI config space in Xen
  hvm: Change initialization/destruction of an hvm
  hvm: Modify hvm_op
  hvm-io: IO refactoring with ioreq server
  hvm-io: send invalidate map cache to each registered servers
  hvm-io: Handle server in buffered IO
  xc: Add the hypercall for multiple servers
  xc: Add argument to allocate more special pages
  xc: modify save/restore to support multiple device models
  xl: Add interface to handle qemu disaggregation
  xl: add device model id to qmp functions
  xl-parsing: Parse new device_models option
  xl: support spawn/destroy on multiple device model
  xl: Fix PCI library
  xl: implement save/restore for multiple device models

 tools/libxc/xc_domain.c           |  155 ++++++++++
 tools/libxc/xc_domain_restore.c   |  150 ++++++++---
 tools/libxc/xc_domain_save.c      |    6 +-
 tools/libxc/xc_hvm_build_x86.c    |   59 ++--
 tools/libxc/xenctrl.h             |   21 ++
 tools/libxc/xenguest.h            |    4 +-
 tools/libxl/Makefile              |    2 +-
 tools/libxl/libxl.c               |   21 +-
 tools/libxl/libxl.h               |    3 +
 tools/libxl/libxl_create.c        |  150 ++++++++---
 tools/libxl/libxl_device.c        |    7 +-
 tools/libxl/libxl_dm.c            |  369 +++++++++++++++++-------
 tools/libxl/libxl_dom.c           |  147 ++++++++--
 tools/libxl/libxl_internal.h      |   76 ++++--
 tools/libxl/libxl_pci.c           |   19 +-
 tools/libxl/libxl_qmp.c           |   49 ++--
 tools/libxl/libxl_types.idl       |   15 +
 tools/libxl/libxlu_dm.c           |   96 +++++++
 tools/libxl/libxlutil.h           |    5 +
 tools/libxl/xl_cmdimpl.c          |   29 ++-
 tools/python/xen/lowlevel/xc/xc.c |    3 +-
 xen/arch/x86/hvm/Makefile         |    1 +
 xen/arch/x86/hvm/emulate.c        |   56 ++++
 xen/arch/x86/hvm/hvm.c            |  567 +++++++++++++++++++++++++++++++------
 xen/arch/x86/hvm/io.c             |   90 +++++--
 xen/arch/x86/hvm/pci_emul.c       |  168 +++++++++++
 xen/include/asm-x86/hvm/domain.h  |   25 ++-
 xen/include/asm-x86/hvm/support.h |   26 ++-
 xen/include/asm-x86/hvm/vcpu.h    |    4 +-
 xen/include/public/hvm/hvm_op.h   |   51 ++++
 xen/include/public/hvm/ioreq.h    |    1 +
 xen/include/public/hvm/params.h   |   11 +-
 xen/include/public/xen.h          |    1 +
 xen/include/xen/hvm/pci_emul.h    |   29 ++
 34 files changed, 1986 insertions(+), 430 deletions(-)
 create mode 100644 tools/libxl/libxlu_dm.c
 create mode 100644 xen/arch/x86/hvm/pci_emul.c
 create mode 100644 xen/include/xen/hvm/pci_emul.h

-- 
Julien Grall