[Qemu-devel] [PATCH v4 0/2] spapr-pci: Enable huge BARs

[Qemu-devel] [PATCH v4 0/2] spapr-pci: Enable huge BARs

[Alexey Kardashevskiy]

This defines a new pseries machine and adds huge bars support which is
disabled by default on older pseries machines.

Please comment. Thanks!

Changes:
v4:
* added pseries-2.3
* now it is 2 patches instead of one :)


Alexey Kardashevskiy (2):
  spapr: Add pseries-2.3 machine
  spapr-pci: Enable huge BARs

 hw/ppc/spapr.c              | 35 +++++++++++++++++++++++++++++++++--
 hw/ppc/spapr_pci.c          | 14 ++++++++++++--
 include/hw/pci-host/spapr.h |  7 ++++---
 3 files changed, 49 insertions(+), 7 deletions(-)

-- 
2.0.0

[Qemu-devel] [PATCH v4 1/2] spapr: Add pseries-2.3 machine

[Alexey Kardashevskiy]

The next patch will make MMIO space bigger and keep the old value for
older pseries machines.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 hw/ppc/spapr.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index b560459..65433fd 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1780,8 +1780,6 @@ static void spapr_machine_2_2_class_init(ObjectClass *oc, void *data)
 
     mc->name = "pseries-2.2";
     mc->desc = "pSeries Logical Partition (PAPR compliant) v2.2";
-    mc->alias = "pseries";
-    mc->is_default = 1;
 }
 
 static const TypeInfo spapr_machine_2_2_info = {
@@ -1790,11 +1788,28 @@ static const TypeInfo spapr_machine_2_2_info = {
     .class_init    = spapr_machine_2_2_class_init,
 };
 
+static void spapr_machine_2_3_class_init(ObjectClass *oc, void *data)
+{
+    MachineClass *mc = MACHINE_CLASS(oc);
+
+    mc->name = "pseries-2.3";
+    mc->desc = "pSeries Logical Partition (PAPR compliant) v2.3";
+    mc->alias = "pseries";
+    mc->is_default = 1;
+}
+
+static const TypeInfo spapr_machine_2_3_info = {
+    .name          = TYPE_SPAPR_MACHINE "2.3",
+    .parent        = TYPE_SPAPR_MACHINE,
+    .class_init    = spapr_machine_2_3_class_init,
+};
+
 static void spapr_machine_register_types(void)
 {
     type_register_static(&spapr_machine_info);
     type_register_static(&spapr_machine_2_1_info);
     type_register_static(&spapr_machine_2_2_info);
+    type_register_static(&spapr_machine_2_3_info);
 }
 
 type_init(spapr_machine_register_types)
-- 
2.0.0

Re: [Qemu-devel] [PATCH v4 1/2] spapr: Add pseries-2.3 machine

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

On Fri, Jan 30, 2015 at 12:53:18PM +1100, Alexey Kardashevskiy wrote:
> The next patch will make MMIO space bigger and keep the old value for
> older pseries machines.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

[Qemu-devel] [PATCH v4 2/2] spapr-pci: Enable huge BARs

[Alexey Kardashevskiy]

At the moment sPAPR only supports 512MB window for MMIO BARs. However
modern devices might want bigger 64bit BARs.

This extends MMIO window from 512MB to 62GB (aligned to
SPAPR_PCI_WINDOW_SPACING) and advertises it in 2 records in
the PHB "ranges" property. 32bit gets the space from
SPAPR_PCI_MEM_WIN_BUS_OFFSET till the end of 4GB, 64bit gets the rest
of the space. If no space is left, 64bit range is not advertised.

The MMIO space size is set to old value of 0x20000000 by default
for pseries machines older than 2.3.

The approach changes the device tree which is a guest visible change, however
it won't break migration as:
1. we do not support migration to older QEMU versions
2. migration to newer QEMU will migrate the device tree as well and since
the new layout only extends the old one and does not change address mappigns,
no breakage is expected here too.

SLOF change is required to utilize this extension.

Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
Changes:
v5:
* disable huge BARs on pseries < 2.3

v4:
* do not add 64bit range if mmio size is less than 2GB and fits 32bit space

v3:
* used SPAPR_PCI_WINDOW_SPACING in SPAPR_PCI_MMIO_WIN_SIZE

v2:
* do not change existing memory layout
* do not create another mmio window
---
 hw/ppc/spapr.c              | 16 ++++++++++++++++
 hw/ppc/spapr_pci.c          | 14 ++++++++++++--
 include/hw/pci-host/spapr.h |  7 ++++---
 3 files changed, 32 insertions(+), 5 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 65433fd..910bd31 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1755,11 +1755,22 @@ static const TypeInfo spapr_machine_info = {
     },
 };
 
+#define SPAPR_COMPAT_2_2 \
+        {\
+            .driver   = TYPE_SPAPR_PCI_HOST_BRIDGE,\
+            .property = "mem_win_size",\
+            .value    = "0x20000000",\
+        }
+
+#define SPAPR_COMPAT_2_1 \
+        SPAPR_COMPAT_2_2
+
 static void spapr_machine_2_1_class_init(ObjectClass *oc, void *data)
 {
     MachineClass *mc = MACHINE_CLASS(oc);
     static GlobalProperty compat_props[] = {
         HW_COMPAT_2_1,
+        SPAPR_COMPAT_2_1,
         { /* end of list */ }
     };
 
@@ -1776,10 +1787,15 @@ static const TypeInfo spapr_machine_2_1_info = {
 
 static void spapr_machine_2_2_class_init(ObjectClass *oc, void *data)
 {
+    static GlobalProperty compat_props[] = {
+        SPAPR_COMPAT_2_2,
+        { /* end of list */ }
+    };
     MachineClass *mc = MACHINE_CLASS(oc);
 
     mc->name = "pseries-2.2";
     mc->desc = "pSeries Logical Partition (PAPR compliant) v2.2";
+    mc->compat_props = compat_props;
 }
 
 static const TypeInfo spapr_machine_2_2_info = {
diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index c0d8c1c..4500849 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -862,6 +862,10 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
     int bus_off, i, j;
     char nodename[256];
     uint32_t bus_range[] = { cpu_to_be32(0), cpu_to_be32(0xff) };
+    const uint64_t mmiosize = memory_region_size(&phb->memwindow);
+    const uint64_t w32max = (1ULL << 32) - SPAPR_PCI_MEM_WIN_BUS_OFFSET;
+    const uint64_t w32size = MIN(w32max, mmiosize);
+    const uint64_t w64size = (mmiosize > w32size) ? (mmiosize - w32size) : 0;
     struct {
         uint32_t hi;
         uint64_t child;
@@ -876,9 +880,15 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
         {
             cpu_to_be32(b_ss(2)), cpu_to_be64(SPAPR_PCI_MEM_WIN_BUS_OFFSET),
             cpu_to_be64(phb->mem_win_addr),
-            cpu_to_be64(memory_region_size(&phb->memwindow)),
+            cpu_to_be64(w32size),
+        },
+        {
+            cpu_to_be32(b_ss(3)), cpu_to_be64(1ULL << 32),
+            cpu_to_be64(phb->mem_win_addr + w32size),
+            cpu_to_be64(w64size)
         },
     };
+    const unsigned sizeof_ranges = (w64size ? 3 : 2) * sizeof(ranges[0]);
     uint64_t bus_reg[] = { cpu_to_be64(phb->buid), 0 };
     uint32_t interrupt_map_mask[] = {
         cpu_to_be32(b_ddddd(-1)|b_fff(0)), 0x0, 0x0, cpu_to_be32(-1)};
@@ -907,7 +917,7 @@ int spapr_populate_pci_dt(sPAPRPHBState *phb,
     _FDT(fdt_setprop_cell(fdt, bus_off, "#interrupt-cells", 0x1));
     _FDT(fdt_setprop(fdt, bus_off, "used-by-rtas", NULL, 0));
     _FDT(fdt_setprop(fdt, bus_off, "bus-range", &bus_range, sizeof(bus_range)));
-    _FDT(fdt_setprop(fdt, bus_off, "ranges", &ranges, sizeof(ranges)));
+    _FDT(fdt_setprop(fdt, bus_off, "ranges", &ranges, sizeof_ranges));
     _FDT(fdt_setprop(fdt, bus_off, "reg", &bus_reg, sizeof(bus_reg)));
     _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pci-config-space-type", 0x1));
     _FDT(fdt_setprop_cell(fdt, bus_off, "ibm,pe-total-#msi", XICS_IRQS));
diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h
index 4ea2a0d..92695b6 100644
--- a/include/hw/pci-host/spapr.h
+++ b/include/hw/pci-host/spapr.h
@@ -96,17 +96,18 @@ struct sPAPRPHBVFIOState {
 
 #define SPAPR_PCI_BASE_BUID          0x800000020000000ULL
 
+#define SPAPR_PCI_MEM_WIN_BUS_OFFSET 0x80000000ULL
+
 #define SPAPR_PCI_WINDOW_BASE        0x10000000000ULL
 #define SPAPR_PCI_WINDOW_SPACING     0x1000000000ULL
 #define SPAPR_PCI_MMIO_WIN_OFF       0xA0000000
-#define SPAPR_PCI_MMIO_WIN_SIZE      0x20000000
+#define SPAPR_PCI_MMIO_WIN_SIZE      (SPAPR_PCI_WINDOW_SPACING - \
+                                     SPAPR_PCI_MEM_WIN_BUS_OFFSET)
 #define SPAPR_PCI_IO_WIN_OFF         0x80000000
 #define SPAPR_PCI_IO_WIN_SIZE        0x10000
 
 #define SPAPR_PCI_MSI_WINDOW         0x40000000000ULL
 
-#define SPAPR_PCI_MEM_WIN_BUS_OFFSET 0x80000000ULL
-
 static inline qemu_irq spapr_phb_lsi_qirq(struct sPAPRPHBState *phb, int pin)
 {
     return xics_get_qirq(spapr->icp, phb->lsi_table[pin].irq);
-- 
2.0.0

Re: [Qemu-devel] [PATCH v4 2/2] spapr-pci: Enable huge BARs

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1436 bytes --]

On Fri, Jan 30, 2015 at 12:53:19PM +1100, Alexey Kardashevskiy wrote:
> At the moment sPAPR only supports 512MB window for MMIO BARs. However
> modern devices might want bigger 64bit BARs.
> 
> This extends MMIO window from 512MB to 62GB (aligned to
> SPAPR_PCI_WINDOW_SPACING) and advertises it in 2 records in
> the PHB "ranges" property. 32bit gets the space from
> SPAPR_PCI_MEM_WIN_BUS_OFFSET till the end of 4GB, 64bit gets the rest
> of the space. If no space is left, 64bit range is not advertised.
> 
> The MMIO space size is set to old value of 0x20000000 by default
> for pseries machines older than 2.3.
> 
> The approach changes the device tree which is a guest visible change, however
> it won't break migration as:
> 1. we do not support migration to older QEMU versions
> 2. migration to newer QEMU will migrate the device tree as well and since
> the new layout only extends the old one and does not change address mappigns,
> no breakage is expected here too.
> 
> SLOF change is required to utilize this extension.
> 
> Suggested-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [PATCH v4 0/2] spapr-pci: Enable huge BARs

[Alexander Graf]

On 30.01.15 02:53, Alexey Kardashevskiy wrote:
> This defines a new pseries machine and adds huge bars support which is
> disabled by default on older pseries machines.
> 
> Please comment. Thanks!

Thanks, applied to ppc-next.


Alex