[PATCH for 4.5 v8 0/1] Add mmio_hole (was Add mmio_hole_size, Add pci_hole_min_size)

[PATCH for 4.5 v8 0/1] Add mmio_hole (was Add mmio_hole_size, Add pci_hole_min_size)

[Don Slutz]

Changes from v7 to v8:
  Ian Campbell
    Use MBYTES in xl.cfg
      Done.
    Use LOG().
      Done.
    Please make this a MemKB at the libxl interface level.
      Done -- s/mmio_hole_size/mmio_hole_memkb/ in tools/libxl/*
        s/mmio_hole_size/mmio_hole/ in docs/man/xl.cfg.pod.5 and
        tools/libxl/xl_cmdimpl.c
    Please drop Konrad's comment.
      Done.

  George Dunlap
    Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
      Did not add do changes to many parts.
      No change to tools/firmware/* so I think it still
        applies there.

  Konrad Rzeszutek Wilk
    I am OK with this patch going in Xen 4.5 (as a release-manager).
      I think this is still applies.


Changes from v6 to v7:
  Konrad Rzeszutek Wilk:
    QEMU 2.1.0 reference confusing.
      Dropped.
    Why have assert(i < 9) & assert(i < 11)
      Dropped.
    This check for 256MB is surely obfuscated by the macros.
      Added comment to help.


Changes from v5 to v6:
  rebased.

  George Dunlap:
    I think we should get rid of xc_hvm_build_with_hole() entirely.
      Done.

  Boris Ostrovsky:
    Add text describing restrictions on hole size
      Done.
    "<=", to be consistent with the check in
        libxl__domain_create_info_setdefault ()
      Done

Changes from v4 to v5:
  Add LIBXL_HAVE_BUILDINFO_HVM_MMIO_HOLE_SIZE.

Changes from v3 to v4:

  Jan Beulich:
    s/pci_hole_min_size/mmio_hole_size/
    s/HVM_BELOW_4G_RAM_END/HVM_BELOW_4G_MMIO_START/
    Add ignore to hvmloader message
    Adjust commit message    
    Dropped min; stopped mmio_hole_size changing in hvmloader

Open question: Should I switch this to max_ram_below_4g or stay with
pci_hole_min_size?

This may help or fix http://bugs.xenproject.org/xen/bug/28

Changes from v2 to v3:
  Dropped RFC since QEMU change will be in QEMU 2.1.0 release.

  Dropped "if (dom_info.debug)" in parse_config_data()
  since dom_info no long passed in.

  Lots more bounds checking.  Since QEMU is a min(qemu_limit,
  max_ram_below_4g) adjust code the do the same.

  Boris Ostrovsky:
    More in commit message.


Changes from v1 to v2:
  Boris Ostrovsky:
    Need to init pci_hole_min_size

  Changed the qemu name from pci_hole_min_size to
  pc-memory-layout.max-ram-below-4g.

  Did not change the Xen version for now.

  Added quick note to xl.cfg.pod.5

  Added a check for a too big value. (Most likely in the wrong
  place.)

If you add enough PCI devices then all mmio may not fit below 4G
which may not be the layout the user wanted. This allows you to
increase the below 4G address space that PCI devices can use and
therefore in more cases not have any mmio that is above 4G.

There are real PCI cards that do not support mmio over 4G, so if you
want to emulate them precisely, you may also need to increase the
space below 4G for them. There are drivers for these cards that also
do not work if they have their mmio space mapped above 4G.

This is posted as an RFC because you need the upstream version of
qemu with all 4 patches:

http://marc.info/?l=qemu-devel&m=139455360016654&w=2

This allows growing the pci_hole to the size needed.

This may help with using pci passthru and HVM. 


Don Slutz (1):
  Add mmio_hole

 docs/man/xl.cfg.pod.5             | 15 +++++++++
 tools/firmware/hvmloader/config.h |  1 -
 tools/firmware/hvmloader/pci.c    | 71 +++++++++++++++++++++++++++------------
 tools/libxl/libxl.h               |  5 +++
 tools/libxl/libxl_create.c        | 29 ++++++++++++----
 tools/libxl/libxl_dm.c            | 20 +++++++++--
 tools/libxl/libxl_dom.c           |  8 +++++
 tools/libxl/libxl_types.idl       |  1 +
 tools/libxl/xl_cmdimpl.c          | 13 +++++++
 9 files changed, 132 insertions(+), 31 deletions(-)

-- 
1.8.4

[PATCH for 4.5 v8 1/1] Add mmio_hole

[Don Slutz]

If you add enough PCI devices then all mmio may not fit below 4G
which may not be the layout the user wanted. This allows you to
increase the below 4G address space that PCI devices can use and
therefore in more cases not have any mmio that is above 4G.

There are real PCI cards that do not support mmio over 4G, so if you
want to emulate them precisely, you may also need to increase the
space below 4G for them. There are drivers for these cards that also
do not work if they have their mmio space mapped above 4G.

This allows growing the MMIO hole to the size needed.

This may help with using pci passthru and HVM.

In the tools this is named mmio_hole_memkb.

Signed-off-by: Don Slutz <dslutz@verizon.com>
---
v8:
    Use MBYTES in xl.cfg
      Done.
    Use LOG().
      Done.
    Please make this a MemKB at the libxl interface level.
      Done -- s/mmio_hole_size/mmio_hole_memkb/ in tools/libxl/*
        s/mmio_hole_size/mmio_hole/ in docs/man/xl.cfg.pod.5 and
        tools/libxl/xl_cmdimpl.c
    Please drop Konrad's comment.
      Done.
    Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
      Did not add do changes to many parts.
      No change to tools/firmware/* so I think it still
        applies there.
    I am OK with this patch going in Xen 4.5 (as a release-manager).
      I think this is still applies.


v7:
    QEMU 2.1.0 reference confusing.
      Dropped.
    Why have assert(i < 9) & assert(i < 11)
      Dropped.
    This check for 256MB is surely obfuscated by the macros.
      Added comment to help.


v6:
  rebased.
    I think we should get rid of xc_hvm_build_with_hole() entirely.
      Done.
    Add text describing restrictions on hole size
      Done.
    "<=", to be consistent with the check in
        libxl__domain_create_info_setdefault ()
      Done

v5:
  Add LIBXL_HAVE_BUILDINFO_HVM_MMIO_HOLE_SIZE.

v4:
    s/pci_hole_min_size/mmio_hole_size/
    s/HVM_BELOW_4G_RAM_END/HVM_BELOW_4G_MMIO_START/
    Add ignore to hvmloader message
    Adjust commit message    
    Dropped min; stopped mmio_hole_size changing in hvmloader

 docs/man/xl.cfg.pod.5             | 15 +++++++++
 tools/firmware/hvmloader/config.h |  1 -
 tools/firmware/hvmloader/pci.c    | 71 +++++++++++++++++++++++++++------------
 tools/libxl/libxl.h               |  5 +++
 tools/libxl/libxl_create.c        | 29 ++++++++++++----
 tools/libxl/libxl_dm.c            | 20 +++++++++--
 tools/libxl/libxl_dom.c           |  8 +++++
 tools/libxl/libxl_types.idl       |  1 +
 tools/libxl/xl_cmdimpl.c          | 13 +++++++
 9 files changed, 132 insertions(+), 31 deletions(-)

diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
index 5830e09..622ea53 100644
--- a/docs/man/xl.cfg.pod.5
+++ b/docs/man/xl.cfg.pod.5
@@ -1164,6 +1164,21 @@ wallclock (i.e., real) time.
 
 =back
 
+=head3 Memory layout
+
+=over 4
+
+=item B<mmio_hole=MBYTES>
+
+Specifies the size the MMIO hole below 4GiB will be.  Only valid for
+device_model_version = "qemu-xen".
+
+Cannot be smaller than 256. Cannot be larger than 3840.
+
+Known good large value is 3072.
+
+=back
+
 =head3 Support for Paravirtualisation of HVM Guests
 
 The following options allow Paravirtualised features (such as devices)
diff --git a/tools/firmware/hvmloader/config.h b/tools/firmware/hvmloader/config.h
index 80bea46..b838cf9 100644
--- a/tools/firmware/hvmloader/config.h
+++ b/tools/firmware/hvmloader/config.h
@@ -53,7 +53,6 @@ extern struct bios_config ovmf_config;
 #define PCI_ISA_IRQ_MASK    0x0c20U /* ISA IRQs 5,10,11 are PCI connected */
 
 /* MMIO hole: Hardcoded defaults, which can be dynamically expanded. */
-#define PCI_MEM_START       0xf0000000
 #define PCI_MEM_END         0xfc000000
 
 extern unsigned long pci_mem_start, pci_mem_end;
diff --git a/tools/firmware/hvmloader/pci.c b/tools/firmware/hvmloader/pci.c
index 3712988..4e8d803 100644
--- a/tools/firmware/hvmloader/pci.c
+++ b/tools/firmware/hvmloader/pci.c
@@ -28,9 +28,10 @@
 #include <xen/memory.h>
 #include <xen/hvm/ioreq.h>
 #include <xen/hvm/hvm_xs_strings.h>
+#include <xen/hvm/e820.h>
 #include <stdbool.h>
 
-unsigned long pci_mem_start = PCI_MEM_START;
+unsigned long pci_mem_start = HVM_BELOW_4G_MMIO_START;
 unsigned long pci_mem_end = PCI_MEM_END;
 uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0;
 
@@ -59,6 +60,7 @@ void pci_setup(void)
         uint64_t bar_sz;
     } *bars = (struct bars *)scratch_start;
     unsigned int i, nr_bars = 0;
+    uint64_t mmio_hole_size = 0;
 
     const char *s;
     /*
@@ -86,6 +88,10 @@ void pci_setup(void)
     printf("Relocating guest memory for lowmem MMIO space %s\n",
            allow_memory_relocate?"enabled":"disabled");
 
+    s = xenstore_read("platform/mmio_hole_size", NULL);
+    if ( s )
+        mmio_hole_size = strtoll(s, NULL, 0);
+
     /* Program PCI-ISA bridge with appropriate link routes. */
     isa_irq = 0;
     for ( link = 0; link < 4; link++ )
@@ -237,26 +243,49 @@ void pci_setup(void)
         pci_writew(devfn, PCI_COMMAND, cmd);
     }
 
-    /*
-     * At the moment qemu-xen can't deal with relocated memory regions.
-     * It's too close to the release to make a proper fix; for now,
-     * only allow the MMIO hole to grow large enough to move guest memory
-     * if we're running qemu-traditional.  Items that don't fit will be
-     * relocated into the 64-bit address space.
-     *
-     * This loop now does the following:
-     * - If allow_memory_relocate, increase the MMIO hole until it's
-     *   big enough, or until it's 2GiB
-     * - If !allow_memory_relocate, increase the MMIO hole until it's
-     *   big enough, or until it's 2GiB, or until it overlaps guest
-     *   memory
-     */
-    while ( (mmio_total > (pci_mem_end - pci_mem_start)) 
-            && ((pci_mem_start << 1) != 0)
-            && (allow_memory_relocate
-                || (((pci_mem_start << 1) >> PAGE_SHIFT)
-                    >= hvm_info->low_mem_pgend)) )
-        pci_mem_start <<= 1;
+    if ( mmio_hole_size )
+    {
+        uint64_t max_ram_below_4g = (1ULL << 32) - mmio_hole_size;
+
+        if ( max_ram_below_4g > HVM_BELOW_4G_MMIO_START )
+        {
+            printf("max_ram_below_4g=0x"PRIllx
+                   " too big for mmio_hole_size=0x"PRIllx
+                   " has been ignored.\n",
+                   PRIllx_arg(max_ram_below_4g),
+                   PRIllx_arg(mmio_hole_size));
+        }
+        else
+        {
+            pci_mem_start = max_ram_below_4g;
+            printf("pci_mem_start=0x%lx (was 0x%x) for mmio_hole_size=%lu\n",
+                   pci_mem_start, HVM_BELOW_4G_MMIO_START,
+                   (long)mmio_hole_size);
+        }
+    }
+    else
+    {
+        /*
+         * At the moment qemu-xen can't deal with relocated memory regions.
+         * It's too close to the release to make a proper fix; for now,
+         * only allow the MMIO hole to grow large enough to move guest memory
+         * if we're running qemu-traditional.  Items that don't fit will be
+         * relocated into the 64-bit address space.
+         *
+         * This loop now does the following:
+         * - If allow_memory_relocate, increase the MMIO hole until it's
+         *   big enough, or until it's 2GiB
+         * - If !allow_memory_relocate, increase the MMIO hole until it's
+         *   big enough, or until it's 2GiB, or until it overlaps guest
+         *   memory
+         */
+        while ( (mmio_total > (pci_mem_end - pci_mem_start))
+                && ((pci_mem_start << 1) != 0)
+                && (allow_memory_relocate
+                    || (((pci_mem_start << 1) >> PAGE_SHIFT)
+                        >= hvm_info->low_mem_pgend)) )
+            pci_mem_start <<= 1;
+    }
 
     if ( mmio_total > (pci_mem_end - pci_mem_start) )
     {
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 28b614f..c3451bd 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -158,6 +158,11 @@
 #define LIBXL_BUILDINFO_HVM_VIRIDIAN_ENABLE_DISABLE_WIDTH 64
 
 /*
+ * libxl_domain_build_info has the u.hvm.mmio_hole_memkb field.
+ */
+#define LIBXL_HAVE_BUILDINFO_HVM_MMIO_HOLE_MEMKB 1
+
+/*
  * libxl ABI compatibility
  *
  * The only guarantee which libxl makes regarding ABI compatibility
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 4b33d07..b1ff5ae 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -23,6 +23,7 @@
 #include <xc_dom.h>
 #include <xenguest.h>
 #include <xen/hvm/hvm_info_table.h>
+#include <xen/hvm/e820.h>
 
 int libxl__domain_create_info_setdefault(libxl__gc *gc,
                                          libxl_domain_create_info *c_info)
@@ -226,6 +227,8 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
     case LIBXL_DOMAIN_TYPE_HVM:
         if (b_info->shadow_memkb == LIBXL_MEMKB_DEFAULT)
             b_info->shadow_memkb = 0;
+        if (b_info->u.hvm.mmio_hole_memkb == LIBXL_MEMKB_DEFAULT)
+            b_info->u.hvm.mmio_hole_memkb = 0;
 
         if (!b_info->u.hvm.vga.kind)
             b_info->u.hvm.vga.kind = LIBXL_VGA_INTERFACE_TYPE_CIRRUS;
@@ -429,13 +432,25 @@ int libxl__domain_build(libxl__gc *gc,
         vments[4] = "start_time";
         vments[5] = libxl__sprintf(gc, "%lu.%02d", start_time.tv_sec,(int)start_time.tv_usec/10000);
 
-        localents = libxl__calloc(gc, 7, sizeof(char *));
-        localents[0] = "platform/acpi";
-        localents[1] = libxl_defbool_val(info->u.hvm.acpi) ? "1" : "0";
-        localents[2] = "platform/acpi_s3";
-        localents[3] = libxl_defbool_val(info->u.hvm.acpi_s3) ? "1" : "0";
-        localents[4] = "platform/acpi_s4";
-        localents[5] = libxl_defbool_val(info->u.hvm.acpi_s4) ? "1" : "0";
+        localents = libxl__calloc(gc, 9, sizeof(char *));
+        i = 0;
+        localents[i++] = "platform/acpi";
+        localents[i++] = libxl_defbool_val(info->u.hvm.acpi) ? "1" : "0";
+        localents[i++] = "platform/acpi_s3";
+        localents[i++] = libxl_defbool_val(info->u.hvm.acpi_s3) ? "1" : "0";
+        localents[i++] = "platform/acpi_s4";
+        localents[i++] = libxl_defbool_val(info->u.hvm.acpi_s4) ? "1" : "0";
+        if (info->u.hvm.mmio_hole_memkb) {
+            uint64_t max_ram_below_4g =
+                (1ULL << 32) - (info->u.hvm.mmio_hole_memkb << 10);
+
+            if (max_ram_below_4g <= HVM_BELOW_4G_MMIO_START) {
+                localents[i++] = "platform/mmio_hole_size";
+                localents[i++] =
+                    libxl__sprintf(gc, "%"PRIu64,
+                                   info->u.hvm.mmio_hole_memkb << 10);
+            }
+        }
 
         break;
     case LIBXL_DOMAIN_TYPE_PV:
diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index 14dc9fa..3e191c3 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -18,6 +18,7 @@
 #include "libxl_osdeps.h" /* must come before any other headers */
 
 #include "libxl_internal.h"
+#include <xen/hvm/e820.h>
 
 static const char *libxl_tapif_script(libxl__gc *gc)
 {
@@ -416,6 +417,7 @@ static char ** libxl__build_device_model_args_new(libxl__gc *gc,
     const libxl_vnc_info *vnc = libxl__dm_vnc(guest_config);
     const libxl_sdl_info *sdl = dm_sdl(guest_config);
     const char *keymap = dm_keymap(guest_config);
+    char *machinearg;
     flexarray_t *dm_args;
     int i, connection, devid;
     uint64_t ram_size;
@@ -720,10 +722,24 @@ static char ** libxl__build_device_model_args_new(libxl__gc *gc,
             /* Switching here to the machine "pc" which does not add
              * the xen-platform device instead of the default "xenfv" machine.
              */
-            flexarray_append(dm_args, "pc,accel=xen");
+            machinearg = libxl__sprintf(gc, "pc,accel=xen");
         } else {
-            flexarray_append(dm_args, "xenfv");
+            machinearg = libxl__sprintf(gc, "xenfv");
         }
+        if (b_info->u.hvm.mmio_hole_memkb) {
+            uint64_t max_ram_below_4g = (1ULL << 32) -
+                (b_info->u.hvm.mmio_hole_memkb << 10);
+
+            if (max_ram_below_4g > HVM_BELOW_4G_MMIO_START) {
+                LOG(WARN, "mmio_hole_memkb=%"PRIu64
+                    " invalid ignored.\n",
+                    b_info->u.hvm.mmio_hole_memkb);
+            } else {
+                machinearg = libxl__sprintf(gc, "%s,max-ram-below-4g=%"PRIu64,
+                                            machinearg, max_ram_below_4g);
+            }
+        }
+        flexarray_append(dm_args, machinearg);
         for (i = 0; b_info->extra_hvm && b_info->extra_hvm[i] != NULL; i++)
             flexarray_append(dm_args, b_info->extra_hvm[i]);
         break;
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 9d830e1..74ea84b 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -23,6 +23,7 @@
 #include <xc_dom.h>
 #include <xen/hvm/hvm_info_table.h>
 #include <xen/hvm/hvm_xs_strings.h>
+#include <xen/hvm/e820.h>
 
 libxl_domain_type libxl__domain_type(libxl__gc *gc, uint32_t domid)
 {
@@ -800,6 +801,13 @@ int libxl__build_hvm(libxl__gc *gc, uint32_t domid,
     args.mem_size = (uint64_t)(info->max_memkb - info->video_memkb) << 10;
     args.mem_target = (uint64_t)(info->target_memkb - info->video_memkb) << 10;
     args.claim_enabled = libxl_defbool_val(info->claim_mode);
+    if (info->u.hvm.mmio_hole_memkb) {
+        uint64_t max_ram_below_4g = (1ULL << 32) -
+            (info->u.hvm.mmio_hole_memkb << 10);
+
+        if (max_ram_below_4g < HVM_BELOW_4G_MMIO_START)
+            args.mmio_size = info->u.hvm.mmio_hole_memkb << 10;
+    }
     if (libxl__domain_firmware(gc, info, &args)) {
         LOG(ERROR, "initializing domain firmware failed");
         goto out;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index fdda761..ca3f724 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -413,6 +413,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
                                        ("timeoffset",       string),
                                        ("hpet",             libxl_defbool),
                                        ("vpt_align",        libxl_defbool),
+                                       ("mmio_hole_memkb",  MemKB),
                                        ("timer_mode",       libxl_timer_mode),
                                        ("nested_hvm",       libxl_defbool),
                                        ("smbios_firmware",  string),
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index ea43761..9a88750 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -34,6 +34,7 @@
 #include <ctype.h>
 #include <inttypes.h>
 #include <limits.h>
+#include <xen/hvm/e820.h>
 
 #include "libxl.h"
 #include "libxl_utils.h"
@@ -1191,6 +1192,18 @@ static void parse_config_data(const char *config_source,
             exit(-ERROR_FAIL);
         }
 
+        if (!xlu_cfg_get_long(config, "mmio_hole", &l, 0)) {
+            uint64_t mmio_hole_size;
+
+            b_info->u.hvm.mmio_hole_memkb = l * 1024;
+            mmio_hole_size = b_info->u.hvm.mmio_hole_memkb * 1024;
+            if (mmio_hole_size < HVM_BELOW_4G_MMIO_LENGTH ||
+                mmio_hole_size > HVM_BELOW_4G_MMIO_START) {
+                fprintf(stderr, "ERROR: invalid value %"PRIu64
+                        " for \"mmio_hole\"\n", l);
+                exit (1);
+            }
+        }
         if (!xlu_cfg_get_long(config, "timer_mode", &l, 1)) {
             const char *s = libxl_timer_mode_to_string(l);
             fprintf(stderr, "WARNING: specifying \"timer_mode\" as an integer is deprecated. "
-- 
1.8.4

Re: [PATCH for 4.5 v8 1/1] Add mmio_hole

[Ian Campbell]

On Tue, 2014-10-21 at 08:42 -0400, Don Slutz wrote:
> If you add enough PCI devices then all mmio may not fit below 4G
> which may not be the layout the user wanted. This allows you to
> increase the below 4G address space that PCI devices can use and
> therefore in more cases not have any mmio that is above 4G.
> 
> There are real PCI cards that do not support mmio over 4G, so if you
> want to emulate them precisely, you may also need to increase the
> space below 4G for them. There are drivers for these cards that also
> do not work if they have their mmio space mapped above 4G.
> 
> This allows growing the MMIO hole to the size needed.
> 
> This may help with using pci passthru and HVM.
> 
> In the tools this is named mmio_hole_memkb.
> 
> Signed-off-by: Don Slutz <dslutz@verizon.com>

For the tools side: Acked-by: Ian Campbell <ian.campbell@citrix.com>

Has this had a release exception? Or acks for the hvmloader side of
things?


> +        /*
> +         * At the moment qemu-xen can't deal with relocated memory regions.
> +         * It's too close to the release to make a proper fix; for now,

This isn't really accurate any more.

No need for a v9 for this, and it would be inappropriate to change it in
what is otherwise a code motion part of the patch. 

> +         * only allow the MMIO hole to grow large enough to move guest memory
> +         * if we're running qemu-traditional.  Items that don't fit will be
> +         * relocated into the 64-bit address space.

Ian.

Re: [PATCH for 4.5 v8 1/1] Add mmio_hole

[Don Slutz]

On 10/22/14 07:21, Ian Campbell wrote:
> On Tue, 2014-10-21 at 08:42 -0400, Don Slutz wrote:
>> If you add enough PCI devices then all mmio may not fit below 4G
>> which may not be the layout the user wanted. This allows you to
>> increase the below 4G address space that PCI devices can use and
>> therefore in more cases not have any mmio that is above 4G.
>>
>> There are real PCI cards that do not support mmio over 4G, so if you
>> want to emulate them precisely, you may also need to increase the
>> space below 4G for them. There are drivers for these cards that also
>> do not work if they have their mmio space mapped above 4G.
>>
>> This allows growing the MMIO hole to the size needed.
>>
>> This may help with using pci passthru and HVM.
>>
>> In the tools this is named mmio_hole_memkb.
>>
>> Signed-off-by: Don Slutz <dslutz@verizon.com>
> For the tools side: Acked-by: Ian Campbell <ian.campbell@citrix.com>
>
> Has this had a release exception? Or acks for the hvmloader side of
> things?
>

On Tue, 14 Oct 2014 11:56:22 -0400, Konrad Rzeszutek Wilk said:

> I am OK with this patch going in Xen 4.5 (as a release-manager)
> but it can't go anywhere unless there is an Ack/Review.


On Mon, 20 Oct 2014 15:10:25 +0100, George Dunlap said:

> I haven't done a full review, but all of my previous concerns have
> been addressed:

> Acked-by: George Dunlap <george.dunlap@eu.citrix.com>


This was on v7.  There is no change to tools/firmware/* from v7 to v8.  
So as I said:


Changes from v7 to v8:
...

   George Dunlap
     Acked-by: George Dunlap<george.dunlap@eu.citrix.com>
       Did not add do changes to many parts.
       No change to tools/firmware/* so I think it still
         applies there.


Hope this is enough for you.

>> +        /*
>> +         * At the moment qemu-xen can't deal with relocated memory regions.
>> +         * It's too close to the release to make a proper fix; for now,
> This isn't really accurate any more.

Yes.

> No need for a v9 for this, and it would be inappropriate to change it in
> what is otherwise a code motion part of the patch.

Great.

    -Don Slutz

>> +         * only allow the MMIO hole to grow large enough to move guest memory
>> +         * if we're running qemu-traditional.  Items that don't fit will be
>> +         * relocated into the 64-bit address space.
> Ian.
>

Re: [PATCH for 4.5 v8 1/1] Add mmio_hole

[Ian Campbell]

On Wed, 2014-10-22 at 09:08 -0400, Don Slutz wrote:
> On 10/22/14 07:21, Ian Campbell wrote:
> > On Tue, 2014-10-21 at 08:42 -0400, Don Slutz wrote:
> >> If you add enough PCI devices then all mmio may not fit below 4G
> >> which may not be the layout the user wanted. This allows you to
> >> increase the below 4G address space that PCI devices can use and
> >> therefore in more cases not have any mmio that is above 4G.
> >>
> >> There are real PCI cards that do not support mmio over 4G, so if you
> >> want to emulate them precisely, you may also need to increase the
> >> space below 4G for them. There are drivers for these cards that also
> >> do not work if they have their mmio space mapped above 4G.
> >>
> >> This allows growing the MMIO hole to the size needed.
> >>
> >> This may help with using pci passthru and HVM.
> >>
> >> In the tools this is named mmio_hole_memkb.
> >>
> >> Signed-off-by: Don Slutz <dslutz@verizon.com>
> > For the tools side: Acked-by: Ian Campbell <ian.campbell@citrix.com>
> >
> > Has this had a release exception? Or acks for the hvmloader side of
> > things?
> >
> 
> On Tue, 14 Oct 2014 11:56:22 -0400, Konrad Rzeszutek Wilk said:
> 
> > I am OK with this patch going in Xen 4.5 (as a release-manager)
> > but it can't go anywhere unless there is an Ack/Review.
> 
> 
> On Mon, 20 Oct 2014 15:10:25 +0100, George Dunlap said:
> 
> > I haven't done a full review, but all of my previous concerns have
> > been addressed:
> 
> > Acked-by: George Dunlap <george.dunlap@eu.citrix.com>

Thanks, applied. I had to fix a warning due to the use of PRIu64 to
print an unsigned long (printf format error on x86_64 and arm32).

I changed the title to "x86: hvm: Allow configuration of the size of the
mmio_hole." since "Add mmio_hole".


Ian.