Re: [XEN][RFC PATCH V2 01/17] hvm: Modify interface to support multiple ioreq server

Re: [XEN][RFC PATCH V2 01/17] hvm: Modify interface to support multiple ioreq server

[Keir Fraser]

On 23/08/2012 14:18, "Ian Campbell" <Ian.Campbell@citrix.com> wrote:

>> diff --git a/xen/include/public/hvm/ioreq.h b/xen/include/public/hvm/ioreq.h
>> index 4022a1d..87aacd3 100644
>> --- a/xen/include/public/hvm/ioreq.h
>> +++ b/xen/include/public/hvm/ioreq.h
>> @@ -34,6 +34,7 @@
>>  
>>  #define IOREQ_TYPE_PIO          0 /* pio */
>>  #define IOREQ_TYPE_COPY         1 /* mmio ops */
>> +#define IOREQ_TYPE_PCI_CONFIG   2 /* pci config space ops */
>>  #define IOREQ_TYPE_TIMEOFFSET   7
>>  #define IOREQ_TYPE_INVALIDATE   8 /* mapcache */
> 
> I wonder why we skip 2-6 now -- perhaps they used to be something else
> and we are avoiding them to avoid strange errors? In which case adding
> the new on as 9 might be a good idea.

They were almost certainly used for representing R-M-W ALU operations back
in the days of the old IO emulator, very long ago. Still, there's no harm in
leaving them unused.

 -- Keir

Re: [XEN][RFC PATCH V2 01/17] hvm: Modify interface to support multiple ioreq server

[Julien Grall]

On 08/23/2012 02:26 PM, Keir Fraser wrote:
> On 23/08/2012 14:18, "Ian Campbell"<Ian.Campbell@citrix.com>  wrote:
>
>    
>>> diff --git a/xen/include/public/hvm/ioreq.h b/xen/include/public/hvm/ioreq.h
>>> index 4022a1d..87aacd3 100644
>>> --- a/xen/include/public/hvm/ioreq.h
>>> +++ b/xen/include/public/hvm/ioreq.h
>>> @@ -34,6 +34,7 @@
>>>
>>>   #define IOREQ_TYPE_PIO          0 /* pio */
>>>   #define IOREQ_TYPE_COPY         1 /* mmio ops */
>>> +#define IOREQ_TYPE_PCI_CONFIG   2 /* pci config space ops */
>>>   #define IOREQ_TYPE_TIMEOFFSET   7
>>>   #define IOREQ_TYPE_INVALIDATE   8 /* mapcache */
>>>        
>> I wonder why we skip 2-6 now -- perhaps they used to be something else
>> and we are avoiding them to avoid strange errors? In which case adding
>> the new on as 9 might be a good idea.
>>      
> They were almost certainly used for representing R-M-W ALU operations back
> in the days of the old IO emulator, very long ago. Still, there's no harm in
> leaving them unused.
>    

Ok. So I will use number 9 for IOREQ_TYPE_PCI_CONFIG.

[XEN][RFC PATCH V2 00/17] QEMU disaggregation in Xen environment

[Julien Grall]

Hello,

This patch series only concerns Xen. Another serie will come for QEMU.

I'm currently working on QEMU disaggregation in Xen environment. The
goal is to be able to running multiple QEMU for a same domain
(http://lists.xen.org/archives/html/xen-devel/2012-03/msg00299.html).

I have already sent a version of patch series few months ago:
    - QEMU: https://lists.gnu.org/archive/html/qemu-devel/2012-03/msg04401.html
    - Xen: http://lists.xen.org/archives/html/xen-devel/2012-03/msg01947.html
With the different feedbacks, I have improved both QEMU and Xen modifications.
As before, I will sent two patch series, one for QEMU the other for Xen.

Full disaggregation is not possible (one device = one QEMU) because lots
of device depends on each others. With the help of Stefano, I have defined
as possible disaggregation:
    - ui: emulate default devices (root bridge, south bridge), VGA,
    keyboard, mouse and USB
    - audio: emulate audio
    - ide: emulate disks
    - serial: emulate serial port
    - net: it possible to have multiple QEMU that emulates one or more
    network card

Of course, a same QEMU can emulate both ui and audio. Old configuration
file with qemu-xen still works.
The patch series adds an option "device_models".

Example:

builder='hvm'
memory = 1024
name = "Debian"
vcpus=1
vif = [ 'type=ioemu, bridge=eth0, mac=00:16:3e:0e:f5:ef, id=nic1' ]
disk = [ 'tap:tapdisk:qcow2:/home/xentest/works/vms/debian.img,xvda,w' ]
device_model_override = '/home/xentest/works/qemu-devel/qemu-wrapper'
device_model_version = 'qemu-xen'
device_models = [ 'name=qnet,vifs=nic1',
                  'name=qall,ui,ide' ]

It possible to override device model path for each device model. It could be
useful for debugging. For instance, 'name=qnet,vifs=nic1,path=/my/path/wrapper'.
The option "name" is used for logging filename or debugging, if it's not
specify, a number is used.

Modifications between V1 and V2:
    - rewrite libxl patch according to the new API
    - improve user experience with configuration file (avoid to specify
      bdf)
    - improve PCI hypercall: use bus, domain, device, function instead of
    bdf.
    - fix PCI config space handler
    - remove unused HVM paramaters
    - handle save/restore

Drawbacks:
    - PCI hotplug doesn't works
    - stubdomain doesn't works because old QEMU is not modify for
    disaggregation. By the way it's works on XenClient stubdomain
    - Which QEMU need to emulate Xen Platform ? It's mainly used
    to unplug network cards and disks

Possible improvements:
    - Like hvm get parameters, introduce an hypercall to retrieve shared
    pages. For the moment the server id is used
    - Specify if we want buffered I/O shared page or not (It was an idea
    of Christian Limpach)

I don't test all configurations. Comments, bug reports, ... are welcome.

Julien Grall (17):
  hvm: Modify interface to support multiple ioreq server
  hvm: Add functions to handle ioreq servers
  hvm-pci: Handle PCI config space in Xen
  hvm: Change initialization/destruction of an hvm
  hvm: Modify hvm_op
  hvm-io: IO refactoring with ioreq server
  hvm-io: send invalidate map cache to each registered servers
  hvm-io: Handle server in buffered IO
  xc: Add the hypercall for multiple servers
  xc: Add argument to allocate more special pages
  xc: modify save/restore to support multiple device models
  xl: Add interface to handle qemu disaggregation
  xl: add device model id to qmp functions
  xl-parsing: Parse new device_models option
  xl: support spawn/destroy on multiple device model
  xl: Fix PCI library
  xl: implement save/restore for multiple device models

 tools/libxc/xc_domain.c           |  155 ++++++++++
 tools/libxc/xc_domain_restore.c   |  150 ++++++++---
 tools/libxc/xc_domain_save.c      |    6 +-
 tools/libxc/xc_hvm_build_x86.c    |   59 ++--
 tools/libxc/xenctrl.h             |   21 ++
 tools/libxc/xenguest.h            |    4 +-
 tools/libxl/Makefile              |    2 +-
 tools/libxl/libxl.c               |   21 +-
 tools/libxl/libxl.h               |    3 +
 tools/libxl/libxl_create.c        |  150 ++++++++---
 tools/libxl/libxl_device.c        |    7 +-
 tools/libxl/libxl_dm.c            |  369 +++++++++++++++++-------
 tools/libxl/libxl_dom.c           |  147 ++++++++--
 tools/libxl/libxl_internal.h      |   76 ++++--
 tools/libxl/libxl_pci.c           |   19 +-
 tools/libxl/libxl_qmp.c           |   49 ++--
 tools/libxl/libxl_types.idl       |   15 +
 tools/libxl/libxlu_dm.c           |   96 +++++++
 tools/libxl/libxlutil.h           |    5 +
 tools/libxl/xl_cmdimpl.c          |   29 ++-
 tools/python/xen/lowlevel/xc/xc.c |    3 +-
 xen/arch/x86/hvm/Makefile         |    1 +
 xen/arch/x86/hvm/emulate.c        |   56 ++++
 xen/arch/x86/hvm/hvm.c            |  567 +++++++++++++++++++++++++++++++------
 xen/arch/x86/hvm/io.c             |   90 +++++--
 xen/arch/x86/hvm/pci_emul.c       |  168 +++++++++++
 xen/include/asm-x86/hvm/domain.h  |   25 ++-
 xen/include/asm-x86/hvm/support.h |   26 ++-
 xen/include/asm-x86/hvm/vcpu.h    |    4 +-
 xen/include/public/hvm/hvm_op.h   |   51 ++++
 xen/include/public/hvm/ioreq.h    |    1 +
 xen/include/public/hvm/params.h   |   11 +-
 xen/include/public/xen.h          |    1 +
 xen/include/xen/hvm/pci_emul.h    |   29 ++
 34 files changed, 1986 insertions(+), 430 deletions(-)
 create mode 100644 tools/libxl/libxlu_dm.c
 create mode 100644 xen/arch/x86/hvm/pci_emul.c
 create mode 100644 xen/include/xen/hvm/pci_emul.h

-- 
Julien Grall

[XEN][RFC PATCH V2 01/17] hvm: Modify interface to support multiple ioreq server

[Julien Grall]

Add structure to handle ioreq server. It's a server which can
handle a range of IO (MMIO and/or PIO) and emulate a PCI device.
Each server has its own shared page to receive ioreq. So
we have introduced to HVM PARAM to set/get the first and
the last shared page used for ioreq. With this id, the server
is able to retrieve its page.

Introduce a new kind of ioreq type IOREQ_TYPE_PCICONFIG
which permits to forward easily PCI config space access.

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---
 xen/include/asm-x86/hvm/domain.h |   25 ++++++++++++++++++-
 xen/include/asm-x86/hvm/vcpu.h   |    4 ++-
 xen/include/public/hvm/hvm_op.h  |   51 ++++++++++++++++++++++++++++++++++++++
 xen/include/public/hvm/ioreq.h   |    1 +
 xen/include/public/hvm/params.h  |    6 +++-
 xen/include/public/xen.h         |    1 +
 xen/include/xen/hvm/pci_emul.h   |   29 +++++++++++++++++++++
 7 files changed, 114 insertions(+), 3 deletions(-)
 create mode 100644 xen/include/xen/hvm/pci_emul.h

diff --git a/xen/include/asm-x86/hvm/domain.h b/xen/include/asm-x86/hvm/domain.h
index 27b3de5..49d1ca0 100644
--- a/xen/include/asm-x86/hvm/domain.h
+++ b/xen/include/asm-x86/hvm/domain.h
@@ -28,6 +28,7 @@
 #include <asm/hvm/vioapic.h>
 #include <asm/hvm/io.h>
 #include <xen/hvm/iommu.h>
+#include <xen/hvm/pci_emul.h>
 #include <asm/hvm/viridian.h>
 #include <asm/hvm/vmx/vmcs.h>
 #include <asm/hvm/svm/vmcb.h>
@@ -41,14 +42,36 @@ struct hvm_ioreq_page {
     void *va;
 };
 
+struct hvm_io_range {
+    uint64_t s, e;
+    struct hvm_io_range *next;
+};
+
+struct hvm_ioreq_server {
+    unsigned int id;
+    domid_t domid;
+    struct hvm_io_range *mmio_range_list;
+    struct hvm_io_range *portio_range_list;
+    struct hvm_ioreq_server *next;
+    struct hvm_ioreq_page ioreq;
+    struct hvm_ioreq_page buf_ioreq;
+    unsigned int buf_ioreq_evtchn;
+};
+
 struct hvm_domain {
+    /* Use for the IO handles by Xen */
     struct hvm_ioreq_page  ioreq;
-    struct hvm_ioreq_page  buf_ioreq;
+    struct hvm_ioreq_server *ioreq_server_list;
+    uint32_t		     nr_ioreq_server;
+    spinlock_t               ioreq_server_lock;
 
     struct pl_time         pl_time;
 
     struct hvm_io_handler *io_handler;
 
+    /* PCI Information */
+    struct pci_root_emul pci_root;
+
     /* Lock protects access to irq, vpic and vioapic. */
     spinlock_t             irq_lock;
     struct hvm_irq         irq;
diff --git a/xen/include/asm-x86/hvm/vcpu.h b/xen/include/asm-x86/hvm/vcpu.h
index 9d68ed2..812b16e 100644
--- a/xen/include/asm-x86/hvm/vcpu.h
+++ b/xen/include/asm-x86/hvm/vcpu.h
@@ -125,7 +125,9 @@ struct hvm_vcpu {
     spinlock_t          tm_lock;
     struct list_head    tm_list;
 
-    int                 xen_port;
+    struct hvm_ioreq_page	*ioreq;
+    /* PCI Information */
+    uint32_t		pci_cf8;
 
     bool_t              flag_dr_dirty;
     bool_t              debug_state_latch;
diff --git a/xen/include/public/hvm/hvm_op.h b/xen/include/public/hvm/hvm_op.h
index a9aab4b..6b17c5f 100644
--- a/xen/include/public/hvm/hvm_op.h
+++ b/xen/include/public/hvm/hvm_op.h
@@ -23,6 +23,9 @@
 
 #include "../xen.h"
 #include "../trace.h"
+#include "../event_channel.h"
+
+#include "hvm_info_table.h" /* HVM_MAX_VCPUS */
 
 /* Get/set subcommands: extra argument == pointer to xen_hvm_param struct. */
 #define HVMOP_set_param           0
@@ -238,6 +241,54 @@ struct xen_hvm_inject_trap {
 typedef struct xen_hvm_inject_trap xen_hvm_inject_trap_t;
 DEFINE_XEN_GUEST_HANDLE(xen_hvm_inject_trap_t);
 
+#define HVMOP_register_ioreq_server 20
+struct xen_hvm_register_ioreq_server {
+    domid_t domid;  /* IN - domain to be serviced */
+    ioservid_t id;  /* OUT - handle for identifying this server */
+};
+typedef struct xen_hvm_register_ioreq_server xen_hvm_register_ioreq_server_t;
+DEFINE_XEN_GUEST_HANDLE(xen_hvm_register_ioreq_server_t);
+
+#define HVMOP_get_ioreq_server_buf_channel 21
+struct xen_hvm_get_ioreq_server_buf_channel {
+    domid_t domid;	        /* IN - domain to be serviced */
+    ioservid_t id;	        /* IN - handle from HVMOP_register_ioreq_server */
+    evtchn_port_t channel;  /* OUT - buf ioreq channel */
+};
+typedef struct xen_hvm_get_ioreq_server_buf_channel xen_hvm_get_ioreq_server_buf_channel_t;
+DEFINE_XEN_GUEST_HANDLE(xen_hvm_get_ioreq_server_buf_channel_t);
+
+#define HVMOP_map_io_range_to_ioreq_server 22
+struct xen_hvm_map_io_range_to_ioreq_server {
+    domid_t domid;          /* IN - domain to be serviced */
+    int is_mmio;         /* IN - MMIO or port IO? */
+    ioservid_t id;          /* IN - handle from HVMOP_register_ioreq_server */
+    uint64_aligned_t s, e;  /* IN - inclusive start and end of range */
+};
+typedef struct xen_hvm_map_io_range_to_ioreq_server xen_hvm_map_io_range_to_ioreq_server_t;
+DEFINE_XEN_GUEST_HANDLE(xen_hvm_map_io_range_to_ioreq_server_t);
+
+#define HVMOP_unmap_io_range_from_ioreq_server 23
+struct xen_hvm_unmap_io_range_from_ioreq_server {
+    domid_t domid;          /* IN - domain to be serviced */
+    uint8_t is_mmio;        /* IN - MMIO or port IO? */
+    ioservid_t id;          /* IN - handle from HVMOP_register_ioreq_server */
+    uint64_aligned_t addr;  /* IN - address inside the range to remove */
+};
+typedef struct xen_hvm_unmap_io_range_from_ioreq_server xen_hvm_unmap_io_range_from_ioreq_server_t;
+DEFINE_XEN_GUEST_HANDLE(xen_hvm_unmap_io_range_from_ioreq_server_t);
+
+#define HVMOP_register_pcidev 24
+struct xen_hvm_register_pcidev {
+    domid_t domid;	   /* IN - domain to be serviced */
+    ioservid_t id;	   /* IN - handle from HVMOP_register_ioreq_server */
+    /* IN - PCI identification in PCI topology (domain:bus:device:function) */
+    uint8_t domain, bus, device, function;
+};
+typedef struct xen_hvm_register_pcidev xen_hvm_register_pcidev_t;
+DEFINE_XEN_GUEST_HANDLE(xen_hvm_register_pcidev_t);
+
+
 #endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
 
 #define HVMOP_get_mem_type    15
diff --git a/xen/include/public/hvm/ioreq.h b/xen/include/public/hvm/ioreq.h
index 4022a1d..87aacd3 100644
--- a/xen/include/public/hvm/ioreq.h
+++ b/xen/include/public/hvm/ioreq.h
@@ -34,6 +34,7 @@
 
 #define IOREQ_TYPE_PIO          0 /* pio */
 #define IOREQ_TYPE_COPY         1 /* mmio ops */
+#define IOREQ_TYPE_PCI_CONFIG   2 /* pci config space ops */
 #define IOREQ_TYPE_TIMEOFFSET   7
 #define IOREQ_TYPE_INVALIDATE   8 /* mapcache */
 
diff --git a/xen/include/public/hvm/params.h b/xen/include/public/hvm/params.h
index 55c1b57..309ac1b 100644
--- a/xen/include/public/hvm/params.h
+++ b/xen/include/public/hvm/params.h
@@ -147,6 +147,10 @@
 #define HVM_PARAM_ACCESS_RING_PFN   28
 #define HVM_PARAM_SHARING_RING_PFN  29
 
-#define HVM_NR_PARAMS          30
+/* Param for ioreq servers */
+#define HVM_PARAM_IO_PFN_FIRST	30
+#define HVM_PARAM_IO_PFN_LAST	31
+
+#define HVM_NR_PARAMS          32
 
 #endif /* __XEN_PUBLIC_HVM_PARAMS_H__ */
diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
index b2f6c50..0de17b2 100644
--- a/xen/include/public/xen.h
+++ b/xen/include/public/xen.h
@@ -466,6 +466,7 @@ DEFINE_XEN_GUEST_HANDLE(mmuext_op_t);
 #ifndef __ASSEMBLY__
 
 typedef uint16_t domid_t;
+typedef uint32_t ioservid_t;
 
 /* Domain ids >= DOMID_FIRST_RESERVED cannot be used for ordinary domains. */
 #define DOMID_FIRST_RESERVED (0x7FF0U)
diff --git a/xen/include/xen/hvm/pci_emul.h b/xen/include/xen/hvm/pci_emul.h
new file mode 100644
index 0000000..4dfb577
--- /dev/null
+++ b/xen/include/xen/hvm/pci_emul.h
@@ -0,0 +1,29 @@
+#ifndef PCI_EMUL_H_
+# define PCI_EMUL_H_
+
+# include <xen/radix-tree.h>
+# include <xen/spinlock.h>
+# include <xen/types.h>
+
+void hvm_init_pci_emul(struct domain *d);
+void hvm_destroy_pci_emul(struct domain *d);
+int hvm_register_pcidev(domid_t domid, ioservid_t id,
+                        uint8_t domain, uint8_t bus,
+                        uint8_t device, uint8_t function);
+
+struct pci_root_emul {
+    spinlock_t pci_lock;
+    struct radix_tree_root pci_list;
+};
+
+#endif /* !PCI_EMUL_H_ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-set-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
Julien Grall

Re: [XEN][RFC PATCH V2 01/17] hvm: Modify interface to support multiple ioreq server

[Ian Campbell]

> diff --git a/xen/include/asm-x86/hvm/domain.h b/xen/include/asm-x86/hvm/domain.h
> index 27b3de5..49d1ca0 100644
> --- a/xen/include/asm-x86/hvm/domain.h
> +++ b/xen/include/asm-x86/hvm/domain.h
[...]
>  struct hvm_domain {
> +    /* Use for the IO handles by Xen */
>      struct hvm_ioreq_page  ioreq;
> -    struct hvm_ioreq_page  buf_ioreq;
> +    struct hvm_ioreq_server *ioreq_server_list;
> +    uint32_t		     nr_ioreq_server;
> +    spinlock_t               ioreq_server_lock;

There's some whitespace weirdness here plus some in
xen/include/asm-x86/hvm/vcpu.h and xen/include/public/hvm/hvm_op.h.

> diff --git a/xen/include/public/hvm/ioreq.h b/xen/include/public/hvm/ioreq.h
> index 4022a1d..87aacd3 100644
> --- a/xen/include/public/hvm/ioreq.h
> +++ b/xen/include/public/hvm/ioreq.h
> @@ -34,6 +34,7 @@
>  
>  #define IOREQ_TYPE_PIO          0 /* pio */
>  #define IOREQ_TYPE_COPY         1 /* mmio ops */
> +#define IOREQ_TYPE_PCI_CONFIG   2 /* pci config space ops */
>  #define IOREQ_TYPE_TIMEOFFSET   7
>  #define IOREQ_TYPE_INVALIDATE   8 /* mapcache */

I wonder why we skip 2-6 now -- perhaps they used to be something else
and we are avoiding them to avoid strange errors? In which case adding
the new on as 9 might be a good idea.

Ian.