[Qemu-devel] [PATCH V8 00/17] hw/pc: implement multiple primary busses for pc machines

[Qemu-devel] [PATCH V8 00/17] hw/pc: implement multiple primary busses for pc machines

[Marcel Apfelbaum]

The series is fully functional.
 - Limitations:
   - Pxb's bus does not support hotplug. It will be addressed on top of this series
     because is already getting to big.
   - Pxb devices work only for i440fx and can be attached only to bus 0.
 - You are more than welcome to try using:
       -device pxb,id=pxb,bus_nr=4,numa_node=1 -device e1000,bus=pxb,addr=0x1

v7->v8:
 - rebased on latest pci branch
 - aml patches already taken
 - added a fix for one of the aml patches

v6->v7:
 - This version includes some refactoring requested by Michael S. Tsirking,
   but no new functionality:
   - Removed TYPE_PCI_MAIN_HOST_BRIDGE interface and scan only pc/q35 host-bridges when
     needed, see patch 11/24.
   - Removed TYPE_PCI_HOST_BRIDGE_SNOOPED interface and added PXB buses as child buses
     of i440fx. The pci configuration is changed to support PXBs, see patch 12/24.
   - Removed patch "hw/pci: move pci bus related code to separate files" and refactor
     all patches that touched the new file.
 - Addressed Paolo's review:
   - Changed documentation to always use numa policy "bind".

v5->v6:
 - This version includes a lot of refactoring requested by Michael S. Tsirking,
   but no new/different functionality:
   - Removed the HOST_BRIDGE_FOR_EACH loop because it too generic
   - Reduced the generic "extra pci roots" aproach to a more "non-generic"
     root bus having snooping buses listening to its configuration space.
     Instead of going over all host bridges, we go only over the snooping
     host bridges associated with the main host bridge.
   - The current implementation made i440fx the only a "snooped" host bridge
 - Addressed Michael S. Tsirkin's review:
   - Replaced qmp queries with native pci ones.
   - Squashed later patches into their places
   - Tweaked a few comments
 - Addressed Shannon Zhao's review:
   - Used build_append_byte instead of build_append_int
 - Addressed Gerd's review:
   - Reduced the "Line over 80" warnings.
 - Rebased on pci branch, tree: git://git.kernel.org/pub/scm/virt/kvm/mst/qemu.git
   - a few days before, I hope is enough
 - Changed some patches order

v4->v5:
 - Rebased on pci branch, tree: git://git.kernel.org/pub/scm/virt/kvm/mst/qemu.git
 - Added PXB documentation (patch 28/28)
 - Addressed Gerd Hoffmann's review:
   - fix PXB behaviour if used with unsupported BIOS (patch 27/28)
 - Addressed Michael S. Tsirkin's review:
   - Removed assert in aml_index (patch 5/28)
   - Renamed pci_ functions to crs_ (patch 12/28)
   - used uint64_t variables instead of signed ones (patch 12/28)
   - Emit MEM/IO AML only for PXBs and i440fx (patch 26/28)
 - Addressed Shannon Zhao's review:
   - Changed build_append_int to  build_append_byte in aml_or (patch 2/25)
 - Thanks to Igor and Kevin for reviews

v3->v4:
 - Addressed Michael S. Tsirkin's review:
   - refactored build_prt method (patch 11/25) 
     hw/apci: add _PRT method for extra PCI root busses
 - Addressed Igor Mammedov's reiew
   - add assert to aml_index (patch 5/25)
 - Fixed aml_equal implementation (patch 1/25)

v2->v3:
 - Rebased on Michael S. Tsirkin's pci branch (that includes now all the dependencies)
 - Refactored acpi terms patch into multiple patches to match Igor's design.

v1->v2:
 - Add support for multiple pxb devices.
 - Attach pxb's bus to specific NUMA node.
 - Got rid of the hacks from prev version.
 - Tested also for Win7 and Fedora 20, and for virtio blk devices.
 - Several bug-fixes resulting in a stable version ready for submission.

Reasoning:
We need multiple primary busess for a few reasons, the most important one
is to be able to associate a pass-trough device with a guest NUMA node.
The OS-es are able to associate a NUMA node only to a primary bus, not to
a specific PCI device or a pci-2-pci bridge.
PC machines support multiple NUMA nodes for CPUs and memory, however the IO
was not yet supported.


Marcel Apfelbaum (17):
  acpi: add implementation of aml_while() term
  hw/pci: made pci_bus_is_root a PCIBusClass method
  hw/pci: made pci_bus_num a PCIBusClass method
  hw/i386: query only for q35/pc when looking for pci host bridge
  hw/pci: extend PCI config access to support devices behind PXB
  hw/acpi: add support for i440fx 'snooping' root busses
  hw/apci: add _PRT method for extra PCI root busses
  hw/acpi: add _CRS method for extra root busses
  hw/acpi: remove from root bus 0 the crs resources used by other buses.
  hw/pci: removed 'rootbus nr is 0' assumption from qmp_pci_query
  hw/pci: introduce PCI Expander Bridge (PXB)
  hw/pci: inform bios if the system has extra pci root buses
  hw/pxb: add map_irq func
  hw/pci: add support for NUMA nodes
  hw/pxb: add numa_node parameter
  apci: fix PXB behaviour if used with unsupported BIOS
  docs: Add PXB documentation

 docs/pci_expander_bridge.txt        |  58 ++++++
 hw/acpi/aml-build.c                 |   8 +
 hw/i386/acpi-build.c                | 396 ++++++++++++++++++++++++++++++++++--
 hw/i386/pc.c                        |  20 ++
 hw/pci-bridge/Makefile.objs         |   1 +
 hw/pci-bridge/pci_expander_bridge.c | 231 +++++++++++++++++++++
 hw/pci/pci.c                        |  78 +++++--
 include/hw/pci/pci.h                |   4 +
 include/hw/pci/pci_bus.h            |  10 +
 include/sysemu/sysemu.h             |   1 +
 10 files changed, 776 insertions(+), 31 deletions(-)
 create mode 100644 docs/pci_expander_bridge.txt
 create mode 100644 hw/pci-bridge/pci_expander_bridge.c

-- 
2.1.0

[Qemu-devel] [PATCH V8 01/17] acpi: add implementation of aml_while() term

[Marcel Apfelbaum]

Commit 68e6b0af7 (acpi: add aml_while() term) added
the definition of aml_while without the actual implementation.
Implement the term.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/acpi/aml-build.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/hw/acpi/aml-build.c b/hw/acpi/aml-build.c
index 2bebf23..0d4b324 100644
--- a/hw/acpi/aml-build.c
+++ b/hw/acpi/aml-build.c
@@ -687,6 +687,14 @@ Aml *aml_else(void)
     return var;
 }
 
+/* ACPI 1.0b: 16.2.5.3 Type 1 Opcodes Encoding: DefWhile */
+Aml *aml_while(Aml *predicate)
+{
+    Aml *var = aml_bundle(0xA2 /* WhileOp */, AML_PACKAGE);
+    aml_append(var, predicate);
+    return var;
+}
+
 /* ACPI 1.0b: 16.2.5.2 Named Objects Encoding: DefMethod */
 Aml *aml_method(const char *name, int arg_count)
 {
-- 
2.1.0

[Qemu-devel] [PATCH V8 02/17] hw/pci: made pci_bus_is_root a PCIBusClass method

[Marcel Apfelbaum]

From: Marcel Apfelbaum <marcel.a@redhat.com>

Refactoring it as a method of PCIBusClass will allow
different implementations for subclasses.

Removed the assumption that the root bus does not
have a parent device because is specific only
to the default class implementation.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/pci/pci.c             | 17 ++++++++++++++---
 include/hw/pci/pci.h     |  2 ++
 include/hw/pci/pci_bus.h |  8 ++++++++
 3 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 48f19a3..132d19e 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -88,9 +88,15 @@ static void pci_bus_unrealize(BusState *qbus, Error **errp)
     vmstate_unregister(NULL, &vmstate_pcibus, bus);
 }
 
+static bool pcibus_is_root(PCIBus *bus)
+{
+    return !bus->parent_dev;
+}
+
 static void pci_bus_class_init(ObjectClass *klass, void *data)
 {
     BusClass *k = BUS_CLASS(klass);
+    PCIBusClass *pbc = PCI_BUS_CLASS(klass);
 
     k->print_dev = pcibus_dev_print;
     k->get_dev_path = pcibus_get_dev_path;
@@ -98,12 +104,15 @@ static void pci_bus_class_init(ObjectClass *klass, void *data)
     k->realize = pci_bus_realize;
     k->unrealize = pci_bus_unrealize;
     k->reset = pcibus_reset;
+
+    pbc->is_root = pcibus_is_root;
 }
 
 static const TypeInfo pci_bus_info = {
     .name = TYPE_PCI_BUS,
     .parent = TYPE_BUS,
     .instance_size = sizeof(PCIBus),
+    .class_size = sizeof(PCIBusClass),
     .class_init = pci_bus_class_init,
 };
 
@@ -278,7 +287,10 @@ PCIBus *pci_device_root_bus(const PCIDevice *d)
 {
     PCIBus *bus = d->bus;
 
-    while ((d = bus->parent_dev) != NULL) {
+    while (!pci_bus_is_root(bus)) {
+        d = bus->parent_dev;
+        assert(d != NULL);
+
         bus = d->bus;
     }
 
@@ -291,7 +303,6 @@ const char *pci_root_bus_path(PCIDevice *dev)
     PCIHostState *host_bridge = PCI_HOST_BRIDGE(rootbus->qbus.parent);
     PCIHostBridgeClass *hc = PCI_HOST_BRIDGE_GET_CLASS(host_bridge);
 
-    assert(!rootbus->parent_dev);
     assert(host_bridge->bus == rootbus);
 
     if (hc->root_bus_path) {
@@ -325,7 +336,7 @@ bool pci_bus_is_express(PCIBus *bus)
 
 bool pci_bus_is_root(PCIBus *bus)
 {
-    return !bus->parent_dev;
+    return PCI_BUS_GET_CLASS(bus)->is_root(bus);
 }
 
 void pci_bus_new_inplace(PCIBus *bus, size_t bus_size, DeviceState *parent,
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index 5d050c8..df05c96 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -340,6 +340,8 @@ typedef PCIINTxRoute (*pci_route_irq_fn)(void *opaque, int pin);
 
 #define TYPE_PCI_BUS "PCI"
 #define PCI_BUS(obj) OBJECT_CHECK(PCIBus, (obj), TYPE_PCI_BUS)
+#define PCI_BUS_CLASS(klass) OBJECT_CLASS_CHECK(PCIBusClass, (klass), TYPE_PCI_BUS)
+#define PCI_BUS_GET_CLASS(obj) OBJECT_GET_CLASS(PCIBusClass, (obj), TYPE_PCI_BUS)
 #define TYPE_PCIE_BUS "PCIE"
 
 bool pci_bus_is_express(PCIBus *bus);
diff --git a/include/hw/pci/pci_bus.h b/include/hw/pci/pci_bus.h
index fabaeee..b5ba9c4 100644
--- a/include/hw/pci/pci_bus.h
+++ b/include/hw/pci/pci_bus.h
@@ -8,6 +8,14 @@
  * use accessor functions in pci.h, pci_bridge.h
  */
 
+typedef struct PCIBusClass {
+    /*< private >*/
+    BusClass parent_class;
+    /*< public >*/
+
+    bool (*is_root)(PCIBus *bus);
+} PCIBusClass;
+
 struct PCIBus {
     BusState qbus;
     PCIIOMMUFunc iommu_fn;
-- 
2.1.0

[Qemu-devel] [PATCH V8 03/17] hw/pci: made pci_bus_num a PCIBusClass method

[Marcel Apfelbaum]

From: Marcel Apfelbaum <marcel.a@redhat.com>

Refactoring it as a method of PCIBusClass will allow
different implementations for subclasses.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/pci/pci.c             | 13 ++++++++++---
 include/hw/pci/pci_bus.h |  1 +
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 132d19e..2f24f74 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -93,6 +93,14 @@ static bool pcibus_is_root(PCIBus *bus)
     return !bus->parent_dev;
 }
 
+static int pcibus_num(PCIBus *bus)
+{
+    if (pcibus_is_root(bus)) {
+        return 0; /* pci host bridge */
+    }
+    return bus->parent_dev->config[PCI_SECONDARY_BUS];
+}
+
 static void pci_bus_class_init(ObjectClass *klass, void *data)
 {
     BusClass *k = BUS_CLASS(klass);
@@ -106,6 +114,7 @@ static void pci_bus_class_init(ObjectClass *klass, void *data)
     k->reset = pcibus_reset;
 
     pbc->is_root = pcibus_is_root;
+    pbc->bus_num = pcibus_num;
 }
 
 static const TypeInfo pci_bus_info = {
@@ -390,9 +399,7 @@ PCIBus *pci_register_bus(DeviceState *parent, const char *name,
 
 int pci_bus_num(PCIBus *s)
 {
-    if (pci_bus_is_root(s))
-        return 0;       /* pci host bridge */
-    return s->parent_dev->config[PCI_SECONDARY_BUS];
+    return PCI_BUS_GET_CLASS(s)->bus_num(s);
 }
 
 static int get_pci_config_device(QEMUFile *f, void *pv, size_t size)
diff --git a/include/hw/pci/pci_bus.h b/include/hw/pci/pci_bus.h
index b5ba9c4..7b9939e 100644
--- a/include/hw/pci/pci_bus.h
+++ b/include/hw/pci/pci_bus.h
@@ -14,6 +14,7 @@ typedef struct PCIBusClass {
     /*< public >*/
 
     bool (*is_root)(PCIBus *bus);
+    int (*bus_num)(PCIBus *bus);
 } PCIBusClass;
 
 struct PCIBus {
-- 
2.1.0

[Qemu-devel] [PATCH V8 04/17] hw/i386: query only for q35/pc when looking for pci host bridge

[Marcel Apfelbaum]

Because of the PXB hosts we cannot simply query TYPE_PCI_HOST_BRIDGE anymore.
On i386 arch we only have two pci hosts, so we can look only for them.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/i386/acpi-build.c | 34 +++++++++++++++++++++++++---------
 1 file changed, 25 insertions(+), 9 deletions(-)

diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 2c7399b..50b93bd 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -240,13 +240,32 @@ static void acpi_get_misc_info(AcpiMiscInfo *info)
     info->applesmc_io_base = applesmc_port();
 }
 
+/*
+ * Because of the PXB hosts we cannot simply query TYPE_PCI_HOST_BRIDGE.
+ * On i386 arch we only have two pci hosts, so we can look only for them.
+ */
+static Object *acpi_get_i386_pci_host(void)
+{
+    PCIHostState *host;
+
+    host = OBJECT_CHECK(PCIHostState,
+                        object_resolve_path("/machine/i440fx", NULL),
+                        TYPE_PCI_HOST_BRIDGE);
+    if (!host) {
+        host = OBJECT_CHECK(PCIHostState,
+                            object_resolve_path("/machine/q35", NULL),
+                            TYPE_PCI_HOST_BRIDGE);
+    }
+
+    return OBJECT(host);
+}
+
 static void acpi_get_pci_info(PcPciInfo *info)
 {
     Object *pci_host;
-    bool ambiguous;
 
-    pci_host = object_resolve_path_type("", TYPE_PCI_HOST_BRIDGE, &ambiguous);
-    g_assert(!ambiguous);
+
+    pci_host = acpi_get_i386_pci_host();
     g_assert(pci_host);
 
     info->w32.begin = object_property_get_int(pci_host,
@@ -957,10 +976,9 @@ build_ssdt(GArray *table_data, GArray *linker,
         {
             Object *pci_host;
             PCIBus *bus = NULL;
-            bool ambiguous;
 
-            pci_host = object_resolve_path_type("", TYPE_PCI_HOST_BRIDGE, &ambiguous);
-            if (!ambiguous && pci_host) {
+            pci_host = acpi_get_i386_pci_host();
+            if (pci_host) {
                 bus = PCI_HOST_BRIDGE(pci_host)->bus;
             }
 
@@ -1272,10 +1290,8 @@ static bool acpi_get_mcfg(AcpiMcfgInfo *mcfg)
 {
     Object *pci_host;
     QObject *o;
-    bool ambiguous;
 
-    pci_host = object_resolve_path_type("", TYPE_PCI_HOST_BRIDGE, &ambiguous);
-    g_assert(!ambiguous);
+    pci_host = acpi_get_i386_pci_host();
     g_assert(pci_host);
 
     o = object_property_get_qobject(pci_host, PCIE_HOST_MCFG_BASE, NULL);
-- 
2.1.0

[Qemu-devel] [PATCH V8 05/17] hw/pci: extend PCI config access to support devices behind PXB

[Marcel Apfelbaum]

PXB buses are assumed to be children of bus 0. Look for them
while scanning the buses.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/pci/pci.c | 34 +++++++++++++++++++++++++++++-----
 1 file changed, 29 insertions(+), 5 deletions(-)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 2f24f74..3361d85 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -1699,10 +1699,28 @@ static bool pci_secondary_bus_in_range(PCIDevice *dev, int bus_num)
 {
     return !(pci_get_word(dev->config + PCI_BRIDGE_CONTROL) &
              PCI_BRIDGE_CTL_BUS_RESET) /* Don't walk the bus if it's reset. */ &&
-        dev->config[PCI_SECONDARY_BUS] < bus_num &&
+        dev->config[PCI_SECONDARY_BUS] <= bus_num &&
         bus_num <= dev->config[PCI_SUBORDINATE_BUS];
 }
 
+/* Whether a given bus number is in a range of a root bus */
+static bool pci_root_bus_in_range(PCIBus *bus, int bus_num)
+{
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(bus->devices); ++i) {
+        PCIDevice *dev = bus->devices[i];
+
+        if (dev && PCI_DEVICE_GET_CLASS(dev)->is_bridge) {
+            if (pci_secondary_bus_in_range(dev, bus_num)) {
+                return true;
+            }
+        }
+    }
+
+    return false;
+}
+
 static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num)
 {
     PCIBus *sec;
@@ -1724,12 +1742,18 @@ static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num)
     /* try child bus */
     for (; bus; bus = sec) {
         QLIST_FOREACH(sec, &bus->child, sibling) {
-            assert(!pci_bus_is_root(sec));
-            if (sec->parent_dev->config[PCI_SECONDARY_BUS] == bus_num) {
+            if (pci_bus_num(sec) == bus_num) {
                 return sec;
             }
-            if (pci_secondary_bus_in_range(sec->parent_dev, bus_num)) {
-                break;
+            /* PXB buses assumed to be children of bus 0 */
+            if (pci_bus_is_root(sec)) {
+                if (pci_root_bus_in_range(sec, bus_num)) {
+                    break;
+                }
+            } else {
+                if (pci_secondary_bus_in_range(sec->parent_dev, bus_num)) {
+                    break;
+                }
             }
         }
     }
-- 
2.1.0

[Qemu-devel] [PATCH V8 06/17] hw/acpi: add support for i440fx 'snooping' root busses

[Marcel Apfelbaum]

If the machine has extra root busses that are snooping to
the i440fx host bridge, we need to add them to
acpi in order to be properly detected by guests.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/i386/acpi-build.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 50b93bd..a7e248d 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -626,6 +626,7 @@ build_ssdt(GArray *table_data, GArray *linker,
     uint32_t nr_mem = machine->ram_slots;
     unsigned acpi_cpus = guest_info->apic_id_limit;
     Aml *ssdt, *sb_scope, *scope, *pkg, *dev, *method, *crs, *field, *ifctx;
+    PCIBus *bus = NULL;
     int i;
 
     ssdt = init_aml_allocator();
@@ -637,6 +638,28 @@ build_ssdt(GArray *table_data, GArray *linker,
     /* Reserve space for header */
     acpi_data_push(ssdt->buf, sizeof(AcpiTableHeader));
 
+    /* Extra PCI root buses are implemented  only for i440fx */
+    bus = find_i440fx();
+    if (bus) {
+        QLIST_FOREACH(bus, &bus->child, sibling) {
+            uint8_t bus_num = pci_bus_num(bus);
+
+            /* look only for expander root buses */
+            if (!pci_bus_is_root(bus)) {
+                continue;
+            }
+
+            scope = aml_scope("\\_SB");
+            dev = aml_device("PC%.02X", bus_num);
+            aml_append(dev,
+                       aml_name_decl("_UID", aml_string("PC%.02X", bus_num)));
+            aml_append(dev, aml_name_decl("_HID", aml_string("PNP0A03")));
+            aml_append(dev, aml_name_decl("_BBN", aml_int(bus_num)));
+            aml_append(scope, dev);
+            aml_append(ssdt, scope);
+        }
+    }
+
     scope = aml_scope("\\_SB.PCI0");
     /* build PCI0._CRS */
     crs = aml_resource_template();
-- 
2.1.0

[Qemu-devel] [PATCH V8 07/17] hw/apci: add _PRT method for extra PCI root busses

[Marcel Apfelbaum]

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/i386/acpi-build.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 81 insertions(+)

diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index a7e248d..451566f 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -617,6 +617,86 @@ static void build_append_pci_bus_devices(Aml *parent_scope, PCIBus *bus,
     aml_append(parent_scope, method);
 }
 
+/*
+ * initialize_route - Initialize the interrupt routing rule
+ * through a specific LINK:
+ *  if (lnk_idx == idx)
+ *      route using link 'link_name'
+ */
+static Aml *initialize_route(Aml *route, const char *link_name,
+                             Aml *lnk_idx, int idx)
+{
+    Aml *if_ctx = aml_if(aml_equal(lnk_idx, aml_int(idx)));
+    Aml *pkg = aml_package(4);
+
+    aml_append(pkg, aml_int(0));
+    aml_append(pkg, aml_int(0));
+    aml_append(pkg, aml_name("%s", link_name));
+    aml_append(pkg, aml_int(0));
+    aml_append(if_ctx, aml_store(pkg, route));
+
+    return if_ctx;
+}
+
+/*
+ * build_prt - Define interrupt rounting rules
+ *
+ * Returns an array of 128 routes, one for each device,
+ * based on device location.
+ * The main goal is to equaly distribute the interrupts
+ * over the 4 existing ACPI links (works only for i440fx).
+ * The hash function is  (slot + pin) & 3 -> "LNK[D|A|B|C]".
+ *
+ */
+static Aml *build_prt(void)
+{
+    Aml *method, *while_ctx, *pin, *res;
+
+    method = aml_method("_PRT", 0);
+    res = aml_local(0);
+    pin = aml_local(1);
+    aml_append(method, aml_store(aml_package(128), res));
+    aml_append(method, aml_store(aml_int(0), pin));
+
+    /* while (pin < 128) */
+    while_ctx = aml_while(aml_lless(pin, aml_int(128)));
+    {
+        Aml *slot = aml_local(2);
+        Aml *lnk_idx = aml_local(3);
+        Aml *route = aml_local(4);
+
+        /* slot = pin >> 2 */
+        aml_append(while_ctx,
+                   aml_store(aml_shiftright(pin, aml_int(2)), slot));
+        /* lnk_idx = (slot + pin) & 3 */
+        aml_append(while_ctx,
+                   aml_store(aml_and(aml_add(pin, slot), aml_int(3)), lnk_idx));
+
+        /* route[2] = "LNK[D|A|B|C]", selection based on pin % 3  */
+        aml_append(while_ctx, initialize_route(route, "LNKD", lnk_idx, 0));
+        aml_append(while_ctx, initialize_route(route, "LNKA", lnk_idx, 1));
+        aml_append(while_ctx, initialize_route(route, "LNKB", lnk_idx, 2));
+        aml_append(while_ctx, initialize_route(route, "LNKC", lnk_idx, 3));
+
+        /* route[0] = 0x[slot]FFFF */
+        aml_append(while_ctx,
+            aml_store(aml_or(aml_shiftleft(slot, aml_int(16)), aml_int(0xFFFF)),
+                      aml_index(route, aml_int(0))));
+        /* route[1] = pin & 3 */
+        aml_append(while_ctx,
+            aml_store(aml_and(pin, aml_int(3)), aml_index(route, aml_int(1))));
+        /* res[pin] = route */
+        aml_append(while_ctx, aml_store(route, aml_index(res, pin)));
+        /* pin++ */
+        aml_append(while_ctx, aml_increment(pin));
+    }
+    aml_append(method, while_ctx);
+    /* return res*/
+    aml_append(method, aml_return(res));
+
+    return method;
+}
+
 static void
 build_ssdt(GArray *table_data, GArray *linker,
            AcpiCpuInfo *cpu, AcpiPmInfo *pm, AcpiMiscInfo *misc,
@@ -655,6 +735,7 @@ build_ssdt(GArray *table_data, GArray *linker,
                        aml_name_decl("_UID", aml_string("PC%.02X", bus_num)));
             aml_append(dev, aml_name_decl("_HID", aml_string("PNP0A03")));
             aml_append(dev, aml_name_decl("_BBN", aml_int(bus_num)));
+            aml_append(dev, build_prt());
             aml_append(scope, dev);
             aml_append(ssdt, scope);
         }
-- 
2.1.0

[Qemu-devel] [PATCH V8 08/17] hw/acpi: add _CRS method for extra root busses

[Marcel Apfelbaum]

Save the IO/mem/bus numbers ranges assigned to the extra root busses
to be removed from the root bus 0 range.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/i386/acpi-build.c | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 139 insertions(+)

diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 451566f..8b1e6b1 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -697,6 +697,137 @@ static Aml *build_prt(void)
     return method;
 }
 
+typedef struct CrsRangeEntry {
+    uint64_t base;
+    uint64_t limit;
+} CrsRangeEntry;
+
+static void crs_range_insert(GPtrArray *ranges, uint64_t base, uint64_t limit)
+{
+    CrsRangeEntry *entry;
+
+    entry = g_malloc(sizeof(*entry));
+    entry->base = base;
+    entry->limit = limit;
+
+    g_ptr_array_add(ranges, entry);
+}
+
+static void crs_range_free(gpointer data)
+{
+    CrsRangeEntry *entry = (CrsRangeEntry *)data;
+    g_free(entry);
+}
+
+static Aml *build_crs(PCIHostState *host,
+                      GPtrArray *io_ranges, GPtrArray *mem_ranges)
+{
+    Aml *crs = aml_resource_template();
+    uint8_t max_bus = pci_bus_num(host->bus);
+    uint8_t type;
+    int devfn;
+
+    for (devfn = 0; devfn < ARRAY_SIZE(host->bus->devices); devfn++) {
+        int i;
+        uint64_t range_base, range_limit;
+        PCIDevice *dev = host->bus->devices[devfn];
+
+        if (!dev) {
+            continue;
+        }
+
+        for (i = 0; i < PCI_NUM_REGIONS; i++) {
+            PCIIORegion *r = &dev->io_regions[i];
+
+            range_base = r->addr;
+            range_limit = r->addr + r->size - 1;
+
+            if (r->type & PCI_BASE_ADDRESS_SPACE_IO) {
+                aml_append(crs,
+                    aml_word_io(aml_min_fixed, aml_max_fixed,
+                                aml_pos_decode, aml_entire_range,
+                                0,
+                                range_base,
+                                range_limit,
+                                0,
+                                range_limit - range_base + 1));
+                crs_range_insert(io_ranges, range_base, range_limit);
+            } else { /* "memory" */
+                aml_append(crs,
+                    aml_dword_memory(aml_pos_decode, aml_min_fixed,
+                                     aml_max_fixed, aml_non_cacheable,
+                                     aml_ReadWrite,
+                                     0,
+                                     range_base,
+                                     range_limit,
+                                     0,
+                                     range_limit - range_base + 1));
+                crs_range_insert(mem_ranges, range_base, range_limit);
+            }
+        }
+
+        type = dev->config[PCI_HEADER_TYPE] & ~PCI_HEADER_TYPE_MULTI_FUNCTION;
+        if (type == PCI_HEADER_TYPE_BRIDGE) {
+            uint8_t subordinate = dev->config[PCI_SUBORDINATE_BUS];
+            if (subordinate > max_bus) {
+                max_bus = subordinate;
+            }
+
+            range_base = pci_bridge_get_base(dev, PCI_BASE_ADDRESS_SPACE_IO);
+            range_limit = pci_bridge_get_limit(dev, PCI_BASE_ADDRESS_SPACE_IO);
+            aml_append(crs,
+                aml_word_io(aml_min_fixed, aml_max_fixed,
+                            aml_pos_decode, aml_entire_range,
+                            0,
+                            range_base,
+                            range_limit,
+                            0,
+                            range_limit - range_base + 1));
+            crs_range_insert(io_ranges, range_base, range_limit);
+
+            range_base =
+                pci_bridge_get_base(dev, PCI_BASE_ADDRESS_SPACE_MEMORY);
+            range_limit =
+                pci_bridge_get_limit(dev, PCI_BASE_ADDRESS_SPACE_MEMORY);
+            aml_append(crs,
+                aml_dword_memory(aml_pos_decode, aml_min_fixed,
+                                 aml_max_fixed, aml_non_cacheable,
+                                 aml_ReadWrite,
+                                 0,
+                                 range_base,
+                                 range_limit,
+                                 0,
+                                 range_limit - range_base + 1));
+            crs_range_insert(mem_ranges, range_base, range_limit);
+
+            range_base =
+                pci_bridge_get_base(dev, PCI_BASE_ADDRESS_MEM_PREFETCH);
+            range_limit =
+                pci_bridge_get_limit(dev, PCI_BASE_ADDRESS_MEM_PREFETCH);
+            aml_append(crs,
+                aml_dword_memory(aml_pos_decode, aml_min_fixed,
+                                 aml_max_fixed, aml_non_cacheable,
+                                 aml_ReadWrite,
+                                 0,
+                                 range_base,
+                                 range_limit,
+                                 0,
+                                 range_limit - range_base + 1));
+            crs_range_insert(mem_ranges, range_base, range_limit);
+        }
+    }
+
+    aml_append(crs,
+        aml_word_bus_number(aml_min_fixed, aml_max_fixed, aml_pos_decode,
+                            0,
+                            pci_bus_num(host->bus),
+                            max_bus,
+                            0,
+                            max_bus - pci_bus_num(host->bus) + 1));
+
+    return crs;
+}
+
 static void
 build_ssdt(GArray *table_data, GArray *linker,
            AcpiCpuInfo *cpu, AcpiPmInfo *pm, AcpiMiscInfo *misc,
@@ -707,6 +838,8 @@ build_ssdt(GArray *table_data, GArray *linker,
     unsigned acpi_cpus = guest_info->apic_id_limit;
     Aml *ssdt, *sb_scope, *scope, *pkg, *dev, *method, *crs, *field, *ifctx;
     PCIBus *bus = NULL;
+    GPtrArray *io_ranges = g_ptr_array_new_with_free_func(crs_range_free);
+    GPtrArray *mem_ranges = g_ptr_array_new_with_free_func(crs_range_free);
     int i;
 
     ssdt = init_aml_allocator();
@@ -736,9 +869,15 @@ build_ssdt(GArray *table_data, GArray *linker,
             aml_append(dev, aml_name_decl("_HID", aml_string("PNP0A03")));
             aml_append(dev, aml_name_decl("_BBN", aml_int(bus_num)));
             aml_append(dev, build_prt());
+            crs = build_crs(PCI_HOST_BRIDGE(BUS(bus)->parent),
+                            io_ranges, mem_ranges);
+            aml_append(dev, aml_name_decl("_CRS", crs));
             aml_append(scope, dev);
             aml_append(ssdt, scope);
         }
+
+        g_ptr_array_free(io_ranges, true);
+        g_ptr_array_free(mem_ranges, true);
     }
 
     scope = aml_scope("\\_SB.PCI0");
-- 
2.1.0

[Qemu-devel] [PATCH V8 09/17] hw/acpi: remove from root bus 0 the crs resources used by other buses.

[Marcel Apfelbaum]

If multiple root buses are used, root bus 0 cannot use all the
pci holes ranges. Remove the IO/mem ranges used by the other
primary buses.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/i386/acpi-build.c | 118 +++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 91 insertions(+), 27 deletions(-)

diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 8b1e6b1..1290983 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -719,6 +719,50 @@ static void crs_range_free(gpointer data)
     g_free(entry);
 }
 
+static gint crs_range_compare(gconstpointer a, gconstpointer b)
+{
+     CrsRangeEntry *entry_a = *(CrsRangeEntry **)a;
+     CrsRangeEntry *entry_b = *(CrsRangeEntry **)b;
+
+     return (int64_t)entry_a->base - (int64_t)entry_b->base;
+}
+
+/*
+ * crs_replace_with_free_ranges - given the 'used' ranges within [start - end]
+ * interval, computes the 'free' ranges from the same interval.
+ * Example: If the input array is { [a1 - a2],[b1 - b2] }, the function
+ * will return { [base - a1], [a2 - b1], [b2 - limit] }.
+ */
+static void crs_replace_with_free_ranges(GPtrArray *ranges,
+                                         uint64_t start, uint64_t end)
+{
+    GPtrArray *free_ranges = g_ptr_array_new_with_free_func(crs_range_free);
+    uint64_t free_base = start;
+    int i;
+
+    g_ptr_array_sort(ranges, crs_range_compare);
+    for (i = 0; i < ranges->len; i++) {
+        CrsRangeEntry *used = g_ptr_array_index(ranges, i);
+
+        if (free_base < used->base) {
+            crs_range_insert(free_ranges, free_base, used->base - 1);
+        }
+
+        free_base = used->limit + 1;
+    }
+
+    if (free_base < end) {
+        crs_range_insert(free_ranges, free_base, end);
+    }
+
+    g_ptr_array_set_size(ranges, 0);
+    for (i = 0; i < free_ranges->len; i++) {
+        g_ptr_array_add(ranges, g_ptr_array_index(free_ranges, i));
+    }
+
+    g_ptr_array_free(free_ranges, false);
+}
+
 static Aml *build_crs(PCIHostState *host,
                       GPtrArray *io_ranges, GPtrArray *mem_ranges)
 {
@@ -744,8 +788,8 @@ static Aml *build_crs(PCIHostState *host,
 
             if (r->type & PCI_BASE_ADDRESS_SPACE_IO) {
                 aml_append(crs,
-                    aml_word_io(aml_min_fixed, aml_max_fixed,
-                                aml_pos_decode, aml_entire_range,
+                    aml_word_io(AML_MIN_FIXED, AML_MAX_FIXED,
+                                AML_POS_DECODE, AML_ENTIRE_RANGE,
                                 0,
                                 range_base,
                                 range_limit,
@@ -754,9 +798,9 @@ static Aml *build_crs(PCIHostState *host,
                 crs_range_insert(io_ranges, range_base, range_limit);
             } else { /* "memory" */
                 aml_append(crs,
-                    aml_dword_memory(aml_pos_decode, aml_min_fixed,
-                                     aml_max_fixed, aml_non_cacheable,
-                                     aml_ReadWrite,
+                    aml_dword_memory(AML_POS_DECODE, AML_MIN_FIXED,
+                                     AML_MAX_FIXED, AML_NON_CACHEABLE,
+                                     AML_READ_WRITE,
                                      0,
                                      range_base,
                                      range_limit,
@@ -776,8 +820,8 @@ static Aml *build_crs(PCIHostState *host,
             range_base = pci_bridge_get_base(dev, PCI_BASE_ADDRESS_SPACE_IO);
             range_limit = pci_bridge_get_limit(dev, PCI_BASE_ADDRESS_SPACE_IO);
             aml_append(crs,
-                aml_word_io(aml_min_fixed, aml_max_fixed,
-                            aml_pos_decode, aml_entire_range,
+                aml_word_io(AML_MIN_FIXED, AML_MAX_FIXED,
+                            AML_POS_DECODE, AML_ENTIRE_RANGE,
                             0,
                             range_base,
                             range_limit,
@@ -790,9 +834,9 @@ static Aml *build_crs(PCIHostState *host,
             range_limit =
                 pci_bridge_get_limit(dev, PCI_BASE_ADDRESS_SPACE_MEMORY);
             aml_append(crs,
-                aml_dword_memory(aml_pos_decode, aml_min_fixed,
-                                 aml_max_fixed, aml_non_cacheable,
-                                 aml_ReadWrite,
+                aml_dword_memory(AML_POS_DECODE, AML_MIN_FIXED,
+                                 AML_MAX_FIXED, AML_NON_CACHEABLE,
+                                 AML_READ_WRITE,
                                  0,
                                  range_base,
                                  range_limit,
@@ -805,9 +849,9 @@ static Aml *build_crs(PCIHostState *host,
             range_limit =
                 pci_bridge_get_limit(dev, PCI_BASE_ADDRESS_MEM_PREFETCH);
             aml_append(crs,
-                aml_dword_memory(aml_pos_decode, aml_min_fixed,
-                                 aml_max_fixed, aml_non_cacheable,
-                                 aml_ReadWrite,
+                aml_dword_memory(AML_POS_DECODE, AML_MIN_FIXED,
+                                 AML_MAX_FIXED, AML_NON_CACHEABLE,
+                                 AML_READ_WRITE,
                                  0,
                                  range_base,
                                  range_limit,
@@ -818,7 +862,7 @@ static Aml *build_crs(PCIHostState *host,
     }
 
     aml_append(crs,
-        aml_word_bus_number(aml_min_fixed, aml_max_fixed, aml_pos_decode,
+        aml_word_bus_number(AML_MIN_FIXED, AML_MAX_FIXED, AML_POS_DECODE,
                             0,
                             pci_bus_num(host->bus),
                             max_bus,
@@ -840,6 +884,8 @@ build_ssdt(GArray *table_data, GArray *linker,
     PCIBus *bus = NULL;
     GPtrArray *io_ranges = g_ptr_array_new_with_free_func(crs_range_free);
     GPtrArray *mem_ranges = g_ptr_array_new_with_free_func(crs_range_free);
+    CrsRangeEntry *entry;
+    int root_bus_limit = 0xFF;
     int i;
 
     ssdt = init_aml_allocator();
@@ -862,6 +908,10 @@ build_ssdt(GArray *table_data, GArray *linker,
                 continue;
             }
 
+            if (bus_num < root_bus_limit) {
+                root_bus_limit = bus_num - 1;
+            }
+
             scope = aml_scope("\\_SB");
             dev = aml_device("PC%.02X", bus_num);
             aml_append(dev,
@@ -875,9 +925,6 @@ build_ssdt(GArray *table_data, GArray *linker,
             aml_append(scope, dev);
             aml_append(ssdt, scope);
         }
-
-        g_ptr_array_free(io_ranges, true);
-        g_ptr_array_free(mem_ranges, true);
     }
 
     scope = aml_scope("\\_SB.PCI0");
@@ -885,26 +932,40 @@ build_ssdt(GArray *table_data, GArray *linker,
     crs = aml_resource_template();
     aml_append(crs,
         aml_word_bus_number(AML_MIN_FIXED, AML_MAX_FIXED, AML_POS_DECODE,
-                            0x0000, 0x0000, 0x00FF, 0x0000, 0x0100));
+                            0x0000, 0x0, root_bus_limit,
+                            0x0000, root_bus_limit + 1));
     aml_append(crs, aml_io(AML_DECODE16, 0x0CF8, 0x0CF8, 0x01, 0x08));
 
     aml_append(crs,
         aml_word_io(AML_MIN_FIXED, AML_MAX_FIXED,
                     AML_POS_DECODE, AML_ENTIRE_RANGE,
                     0x0000, 0x0000, 0x0CF7, 0x0000, 0x0CF8));
-    aml_append(crs,
-        aml_word_io(AML_MIN_FIXED, AML_MAX_FIXED,
-                    AML_POS_DECODE, AML_ENTIRE_RANGE,
-                    0x0000, 0x0D00, 0xFFFF, 0x0000, 0xF300));
+
+    crs_replace_with_free_ranges(io_ranges, 0x0D00, 0xFFFF);
+    for (i = 0; i < io_ranges->len; i++) {
+        entry = g_ptr_array_index(io_ranges, i);
+        aml_append(crs,
+            aml_word_io(AML_MIN_FIXED, AML_MAX_FIXED,
+                        AML_POS_DECODE, AML_ENTIRE_RANGE,
+                        0x0000, entry->base, entry->limit,
+                        0x0000, entry->limit - entry->base + 1));
+    }
+
     aml_append(crs,
         aml_dword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
                          AML_CACHEABLE, AML_READ_WRITE,
                          0, 0x000A0000, 0x000BFFFF, 0, 0x00020000));
-    aml_append(crs,
-        aml_dword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
-                         AML_NON_CACHEABLE, AML_READ_WRITE,
-                         0, pci->w32.begin, pci->w32.end - 1, 0,
-                         pci->w32.end - pci->w32.begin));
+
+    crs_replace_with_free_ranges(mem_ranges, pci->w32.begin, pci->w32.end - 1);
+    for (i = 0; i < mem_ranges->len; i++) {
+        entry = g_ptr_array_index(mem_ranges, i);
+        aml_append(crs,
+            aml_dword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
+                             AML_NON_CACHEABLE, AML_READ_WRITE,
+                             0, entry->base, entry->limit,
+                             0, entry->limit - entry->base + 1));
+    }
+
     if (pci->w64.begin) {
         aml_append(crs,
             aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED, AML_MAX_FIXED,
@@ -927,6 +988,9 @@ build_ssdt(GArray *table_data, GArray *linker,
     aml_append(dev, aml_name_decl("_CRS", crs));
     aml_append(scope, dev);
 
+    g_ptr_array_free(io_ranges, true);
+    g_ptr_array_free(mem_ranges, true);
+
     /* reserve PCIHP resources */
     if (pm->pcihp_io_len) {
         dev = aml_device("PHPR");
-- 
2.1.0

[Qemu-devel] [PATCH V8 10/17] hw/pci: removed 'rootbus nr is 0' assumption from qmp_pci_query

[Marcel Apfelbaum]

From: Marcel Apfelbaum <marcel.a@redhat.com>

Use the newer pci_bus_num to correctly get the root bus number.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/pci/pci.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 3361d85..a956640 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -1594,7 +1594,8 @@ PciInfoList *qmp_query_pci(Error **errp)
 
     QLIST_FOREACH(host_bridge, &pci_host_bridges, next) {
         info = g_malloc0(sizeof(*info));
-        info->value = qmp_query_pci_bus(host_bridge->bus, 0);
+        info->value = qmp_query_pci_bus(host_bridge->bus,
+                                        pci_bus_num(host_bridge->bus));
 
         /* XXX: waiting for the qapi to support GSList */
         if (!cur_item) {
-- 
2.1.0

[Qemu-devel] [PATCH V8 11/17] hw/pci: introduce PCI Expander Bridge (PXB)

[Marcel Apfelbaum]

From: Marcel Apfelbaum <marcel.a@redhat.com>

PXB is a "light-weight" host bridge whose purpose is to enable
the main host bridge to support multiple PCI root buses
for pc machines.

As oposed to PCI-2-PCI bridge's secondary bus, PXB's bus
is a primary bus and can be associated with a NUMA node
(different from the main host bridge) allowing the guest OS
to recognize the proximity of a pass-through device to
other resources as RAM and CPUs.

The PXB is composed from:
 - A primary PCI bus (can be associated with a NUMA node)
   Acts like a normal pci bus and from the functionality point
   of view is an "expansion" of the bus behind the
   main host bridge.
 - A pci-2-pci bridge behind the primary PCI bus where the actual
   devices will be attached.
 - A host-bridge PCI device
   Situated on the bus behind the main host bridge, allows
   the BIOS to configure the bus number and IO/mem resources.
   It does not have its own config/data register for configuration
   cycles, this being handled by the main host bridge.
-  A host-bridge sysbus to comply with QEMU current design.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/pci-bridge/Makefile.objs         |   1 +
 hw/pci-bridge/pci_expander_bridge.c | 196 ++++++++++++++++++++++++++++++++++++
 include/hw/pci/pci.h                |   1 +
 3 files changed, 198 insertions(+)
 create mode 100644 hw/pci-bridge/pci_expander_bridge.c

diff --git a/hw/pci-bridge/Makefile.objs b/hw/pci-bridge/Makefile.objs
index 96c596e..f2adfe3 100644
--- a/hw/pci-bridge/Makefile.objs
+++ b/hw/pci-bridge/Makefile.objs
@@ -1,4 +1,5 @@
 common-obj-y += pci_bridge_dev.o
+common-obj-y += pci_expander_bridge.o
 common-obj-$(CONFIG_XIO3130) += xio3130_upstream.o xio3130_downstream.o
 common-obj-$(CONFIG_IOH3420) += ioh3420.o
 common-obj-$(CONFIG_I82801B11) += i82801b11.o
diff --git a/hw/pci-bridge/pci_expander_bridge.c b/hw/pci-bridge/pci_expander_bridge.c
new file mode 100644
index 0000000..88e85c1
--- /dev/null
+++ b/hw/pci-bridge/pci_expander_bridge.c
@@ -0,0 +1,196 @@
+/*
+ * PCI Expander Bridge Device Emulation
+ *
+ * Copyright (C) 2015 Red Hat Inc
+ *
+ * Authors:
+ *   Marcel Apfelbaum <marcel@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "hw/pci/pci.h"
+#include "hw/pci/pci_bus.h"
+#include "hw/pci/pci_host.h"
+#include "hw/pci/pci_bus.h"
+#include "hw/i386/pc.h"
+#include "qemu/range.h"
+#include "qemu/error-report.h"
+
+#define TYPE_PXB_BUS "pxb-bus"
+#define PXB_BUS(obj) OBJECT_CHECK(PXBBus, (obj), TYPE_PXB_BUS)
+
+typedef struct PXBBus {
+    /*< private >*/
+    PCIBus parent_obj;
+    /*< public >*/
+
+    char bus_path[8];
+} PXBBus;
+
+#define TYPE_PXB_DEVICE "pxb"
+#define PXB_DEV(obj) OBJECT_CHECK(PXBDev, (obj), TYPE_PXB_DEVICE)
+
+typedef struct PXBDev {
+    /*< private >*/
+    PCIDevice parent_obj;
+    /*< public >*/
+
+    uint8_t bus_nr;
+} PXBDev;
+
+#define TYPE_PXB_HOST "pxb-host"
+
+static int pxb_bus_num(PCIBus *bus)
+{
+    PXBDev *pxb = PXB_DEV(bus->parent_dev);
+
+    return pxb->bus_nr;
+}
+
+static bool pxb_is_root(PCIBus *bus)
+{
+    return true; /* by definition */
+}
+
+static void pxb_bus_class_init(ObjectClass *class, void *data)
+{
+    PCIBusClass *pbc = PCI_BUS_CLASS(class);
+
+    pbc->bus_num = pxb_bus_num;
+    pbc->is_root = pxb_is_root;
+}
+
+static const TypeInfo pxb_bus_info = {
+    .name          = TYPE_PXB_BUS,
+    .parent        = TYPE_PCI_BUS,
+    .instance_size = sizeof(PXBBus),
+    .class_init    = pxb_bus_class_init,
+};
+
+static const char *pxb_host_root_bus_path(PCIHostState *host_bridge,
+                                          PCIBus *rootbus)
+{
+    PXBBus *bus = PXB_BUS(rootbus);
+
+    snprintf(bus->bus_path, 8, "0000:%02x", pxb_bus_num(rootbus));
+    return bus->bus_path;
+}
+
+static void pxb_host_class_init(ObjectClass *class, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(class);
+    PCIHostBridgeClass *hc = PCI_HOST_BRIDGE_CLASS(class);
+
+    dc->fw_name = "pci";
+    hc->root_bus_path = pxb_host_root_bus_path;
+}
+
+static const TypeInfo pxb_host_info = {
+    .name          = TYPE_PXB_HOST,
+    .parent        = TYPE_PCI_HOST_BRIDGE,
+    .class_init    = pxb_host_class_init,
+};
+
+/*
+ * Registers the PXB bus as a child of the i440fx root bus.
+ *
+ * Returns 0 on successs, -1 if i440fx host was not
+ * found or the bus number is already in use.
+ */
+static int pxb_register_bus(PCIDevice *dev, PCIBus *pxb_bus)
+{
+    PCIBus *bus = dev->bus;
+    int pxb_bus_num = pci_bus_num(pxb_bus);
+
+    if (bus->parent_dev) {
+        error_report("PXB devices can be attached only to root bus.");
+        return -1;
+    }
+
+    QLIST_FOREACH(bus, &bus->child, sibling) {
+        if (pci_bus_num(bus) == pxb_bus_num) {
+            error_report("Bus %d is already in use.", pxb_bus_num);
+            return -1;
+        }
+    }
+    QLIST_INSERT_HEAD(&dev->bus->child, pxb_bus, sibling);
+
+    return 0;
+}
+
+static int pxb_dev_initfn(PCIDevice *dev)
+{
+    PXBDev *pxb = PXB_DEV(dev);
+    DeviceState *ds, *bds;
+    PCIBus *bus;
+    const char *dev_name = NULL;
+
+    if (dev->qdev.id && *dev->qdev.id) {
+        dev_name = dev->qdev.id;
+    }
+
+    ds = qdev_create(NULL, TYPE_PXB_HOST);
+    bus = pci_bus_new(ds, "pxb-internal", NULL, NULL, 0, TYPE_PXB_BUS);
+
+    bus->parent_dev = dev;
+    bus->address_space_mem = dev->bus->address_space_mem;
+    bus->address_space_io = dev->bus->address_space_io;
+    bus->map_irq = pci_swizzle_map_irq_fn;
+
+    bds = qdev_create(BUS(bus), "pci-bridge");
+    bds->id = dev_name;
+    qdev_prop_set_uint8(bds, "chassis_nr", pxb->bus_nr);
+
+    PCI_HOST_BRIDGE(ds)->bus = bus;
+
+    if (pxb_register_bus(dev, bus)) {
+        return -EINVAL;
+    }
+
+    qdev_init_nofail(ds);
+    qdev_init_nofail(bds);
+
+    pci_word_test_and_set_mask(dev->config + PCI_STATUS,
+                               PCI_STATUS_66MHZ | PCI_STATUS_FAST_BACK);
+    pci_config_set_class(dev->config, PCI_CLASS_BRIDGE_HOST);
+
+    return 0;
+}
+
+static Property pxb_dev_properties[] = {
+    /* Note: 0 is not a legal a PXB bus number. */
+    DEFINE_PROP_UINT8("bus_nr", PXBDev, bus_nr, 0),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void pxb_dev_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
+
+    k->init = pxb_dev_initfn;
+    k->vendor_id = PCI_VENDOR_ID_REDHAT;
+    k->device_id = PCI_DEVICE_ID_REDHAT_PXB;
+    k->class_id = PCI_CLASS_BRIDGE_HOST;
+
+    dc->desc = "PCI Expander Bridge";
+    dc->props = pxb_dev_properties;
+}
+
+static const TypeInfo pxb_dev_info = {
+    .name          = TYPE_PXB_DEVICE,
+    .parent        = TYPE_PCI_DEVICE,
+    .instance_size = sizeof(PXBDev),
+    .class_init    = pxb_dev_class_init,
+};
+
+static void pxb_register_types(void)
+{
+    type_register_static(&pxb_bus_info);
+    type_register_static(&pxb_host_info);
+    type_register_static(&pxb_dev_info);
+}
+
+type_init(pxb_register_types)
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index df05c96..7940700 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -91,6 +91,7 @@
 #define PCI_DEVICE_ID_REDHAT_ROCKER      0x0006
 #define PCI_DEVICE_ID_REDHAT_SDHCI       0x0007
 #define PCI_DEVICE_ID_REDHAT_PCIE_HOST   0x0008
+#define PCI_DEVICE_ID_REDHAT_PXB         0x0009
 #define PCI_DEVICE_ID_REDHAT_QXL         0x0100
 
 #define FMT_PCIBUS                      PRIx64
-- 
2.1.0

[Qemu-devel] [PATCH V8 12/17] hw/pci: inform bios if the system has extra pci root buses

[Marcel Apfelbaum]

From: Marcel Apfelbaum <marcel.a@redhat.com>

The bios looks for 'etc/extra-pci-roots' to decide if
is going to scan further buses after bus 0 tree.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/i386/pc.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 1eb1db0..a93972f 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -30,6 +30,7 @@
 #include "hw/block/fdc.h"
 #include "hw/ide.h"
 #include "hw/pci/pci.h"
+#include "hw/pci/pci_bus.h"
 #include "monitor/monitor.h"
 #include "hw/nvram/fw_cfg.h"
 #include "hw/timer/hpet.h"
@@ -1119,6 +1120,25 @@ void pc_guest_info_machine_done(Notifier *notifier, void *data)
     PcGuestInfoState *guest_info_state = container_of(notifier,
                                                       PcGuestInfoState,
                                                       machine_done);
+    PCIBus *bus = find_i440fx();
+
+    if (bus) {
+        int extra_hosts = 0;
+
+        QLIST_FOREACH(bus, &bus->child, sibling) {
+            /* look for expander root buses */
+            if (pci_bus_is_root(bus)) {
+                extra_hosts++;
+            }
+        }
+        if (extra_hosts && guest_info_state->info.fw_cfg) {
+            uint64_t *val = g_malloc(sizeof(*val));
+            *val = cpu_to_le64(extra_hosts);
+            fw_cfg_add_file(guest_info_state->info.fw_cfg,
+                    "etc/extra-pci-roots", val, sizeof(*val));
+        }
+    }
+
     acpi_setup(&guest_info_state->info);
 }
 
-- 
2.1.0

[Qemu-devel] [PATCH V8 13/17] hw/pxb: add map_irq func

[Marcel Apfelbaum]

The bios does not index the pxb slot number when
it computes the IRQ because it resides on bus 0
and not on the current bus.
However Qemu routes the irq through bus 0 and adds
the pxb slot to the IRQ computation of the PXB device.

Synchronize between bios and Qemu by canceling
pxb's effect.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/pci-bridge/pci_expander_bridge.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/hw/pci-bridge/pci_expander_bridge.c b/hw/pci-bridge/pci_expander_bridge.c
index 88e85c1..8660a00 100644
--- a/hw/pci-bridge/pci_expander_bridge.c
+++ b/hw/pci-bridge/pci_expander_bridge.c
@@ -120,6 +120,24 @@ static int pxb_register_bus(PCIDevice *dev, PCIBus *pxb_bus)
     return 0;
 }
 
+static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin)
+{
+    PCIDevice *pxb = pci_dev->bus->parent_dev;
+
+    /*
+     * The bios does not index the pxb slot number when
+     * it computes the IRQ because it resides on bus 0
+     * and not on the current bus.
+     * However QEMU routes the irq through bus 0 and adds
+     * the pxb slot to the IRQ computation of the PXB
+     * device.
+     *
+     * Synchronize between bios and QEMU by canceling
+     * pxb's effect.
+     */
+    return pin - PCI_SLOT(pxb->devfn);
+}
+
 static int pxb_dev_initfn(PCIDevice *dev)
 {
     PXBDev *pxb = PXB_DEV(dev);
@@ -137,7 +155,7 @@ static int pxb_dev_initfn(PCIDevice *dev)
     bus->parent_dev = dev;
     bus->address_space_mem = dev->bus->address_space_mem;
     bus->address_space_io = dev->bus->address_space_io;
-    bus->map_irq = pci_swizzle_map_irq_fn;
+    bus->map_irq = pxb_map_irq_fn;
 
     bds = qdev_create(BUS(bus), "pci-bridge");
     bds->id = dev_name;
-- 
2.1.0

[Qemu-devel] [PATCH V8 14/17] hw/pci: add support for NUMA nodes

[Marcel Apfelbaum]

PCI root buses can be attached to a specific NUMA node.
PCI buses are not attached by default to a NUMA node.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/pci/pci.c             | 11 +++++++++++
 include/hw/pci/pci.h     |  1 +
 include/hw/pci/pci_bus.h |  1 +
 include/sysemu/sysemu.h  |  1 +
 4 files changed, 14 insertions(+)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index a956640..4989408 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -101,6 +101,11 @@ static int pcibus_num(PCIBus *bus)
     return bus->parent_dev->config[PCI_SECONDARY_BUS];
 }
 
+static uint16_t pcibus_numa_node(PCIBus *bus)
+{
+    return NUMA_NODE_UNASSIGNED;
+}
+
 static void pci_bus_class_init(ObjectClass *klass, void *data)
 {
     BusClass *k = BUS_CLASS(klass);
@@ -115,6 +120,7 @@ static void pci_bus_class_init(ObjectClass *klass, void *data)
 
     pbc->is_root = pcibus_is_root;
     pbc->bus_num = pcibus_num;
+    pbc->numa_node = pcibus_numa_node;
 }
 
 static const TypeInfo pci_bus_info = {
@@ -402,6 +408,11 @@ int pci_bus_num(PCIBus *s)
     return PCI_BUS_GET_CLASS(s)->bus_num(s);
 }
 
+int pci_bus_numa_node(PCIBus *bus)
+{
+    return PCI_BUS_GET_CLASS(bus)->numa_node(bus);
+}
+
 static int get_pci_config_device(QEMUFile *f, void *pv, size_t size)
 {
     PCIDevice *s = container_of(pv, PCIDevice, config);
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index 7940700..c2a427f 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -382,6 +382,7 @@ PCIDevice *pci_nic_init_nofail(NICInfo *nd, PCIBus *rootbus,
 PCIDevice *pci_vga_init(PCIBus *bus);
 
 int pci_bus_num(PCIBus *s);
+int pci_bus_numa_node(PCIBus *bus);
 void pci_for_each_device(PCIBus *bus, int bus_num,
                          void (*fn)(PCIBus *bus, PCIDevice *d, void *opaque),
                          void *opaque);
diff --git a/include/hw/pci/pci_bus.h b/include/hw/pci/pci_bus.h
index 7b9939e..403fec6 100644
--- a/include/hw/pci/pci_bus.h
+++ b/include/hw/pci/pci_bus.h
@@ -15,6 +15,7 @@ typedef struct PCIBusClass {
 
     bool (*is_root)(PCIBus *bus);
     int (*bus_num)(PCIBus *bus);
+    uint16_t (*numa_node)(PCIBus *bus);
 } PCIBusClass;
 
 struct PCIBus {
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 8a52934..4fcc20e 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -137,6 +137,7 @@ extern const char *mem_path;
 extern int mem_prealloc;
 
 #define MAX_NODES 128
+#define NUMA_NODE_UNASSIGNED MAX_NODES
 
 /* The following shall be true for all CPUs:
  *   cpu->cpu_index < max_cpus <= MAX_CPUMASK_BITS
-- 
2.1.0

[Qemu-devel] [PATCH V8 15/17] hw/pxb: add numa_node parameter

[Marcel Apfelbaum]

The pxb can be attach to and existing numa node by specifying
numa_node option that equals the desired numa nodeid.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/i386/acpi-build.c                |  6 ++++++
 hw/pci-bridge/pci_expander_bridge.c | 17 +++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 1290983..45c36a8 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -902,6 +902,7 @@ build_ssdt(GArray *table_data, GArray *linker,
     if (bus) {
         QLIST_FOREACH(bus, &bus->child, sibling) {
             uint8_t bus_num = pci_bus_num(bus);
+            uint8_t numa_node = pci_bus_numa_node(bus);
 
             /* look only for expander root buses */
             if (!pci_bus_is_root(bus)) {
@@ -918,6 +919,11 @@ build_ssdt(GArray *table_data, GArray *linker,
                        aml_name_decl("_UID", aml_string("PC%.02X", bus_num)));
             aml_append(dev, aml_name_decl("_HID", aml_string("PNP0A03")));
             aml_append(dev, aml_name_decl("_BBN", aml_int(bus_num)));
+
+            if (numa_node != NUMA_NODE_UNASSIGNED) {
+                aml_append(dev, aml_name_decl("_PXM", aml_int(numa_node)));
+            }
+
             aml_append(dev, build_prt());
             crs = build_crs(PCI_HOST_BRIDGE(BUS(bus)->parent),
                             io_ranges, mem_ranges);
diff --git a/hw/pci-bridge/pci_expander_bridge.c b/hw/pci-bridge/pci_expander_bridge.c
index 8660a00..ec2bb45 100644
--- a/hw/pci-bridge/pci_expander_bridge.c
+++ b/hw/pci-bridge/pci_expander_bridge.c
@@ -17,6 +17,7 @@
 #include "hw/i386/pc.h"
 #include "qemu/range.h"
 #include "qemu/error-report.h"
+#include "sysemu/numa.h"
 
 #define TYPE_PXB_BUS "pxb-bus"
 #define PXB_BUS(obj) OBJECT_CHECK(PXBBus, (obj), TYPE_PXB_BUS)
@@ -38,6 +39,7 @@ typedef struct PXBDev {
     /*< public >*/
 
     uint8_t bus_nr;
+    uint16_t numa_node;
 } PXBDev;
 
 #define TYPE_PXB_HOST "pxb-host"
@@ -54,12 +56,20 @@ static bool pxb_is_root(PCIBus *bus)
     return true; /* by definition */
 }
 
+static uint16_t pxb_bus_numa_node(PCIBus *bus)
+{
+    PXBDev *pxb = PXB_DEV(bus->parent_dev);
+
+    return pxb->numa_node;
+}
+
 static void pxb_bus_class_init(ObjectClass *class, void *data)
 {
     PCIBusClass *pbc = PCI_BUS_CLASS(class);
 
     pbc->bus_num = pxb_bus_num;
     pbc->is_root = pxb_is_root;
+    pbc->numa_node = pxb_bus_numa_node;
 }
 
 static const TypeInfo pxb_bus_info = {
@@ -145,6 +155,12 @@ static int pxb_dev_initfn(PCIDevice *dev)
     PCIBus *bus;
     const char *dev_name = NULL;
 
+    if (pxb->numa_node != NUMA_NODE_UNASSIGNED &&
+        pxb->numa_node >= nb_numa_nodes) {
+        error_report("Illegal numa node %d.", pxb->numa_node);
+        return -EINVAL;
+    }
+
     if (dev->qdev.id && *dev->qdev.id) {
         dev_name = dev->qdev.id;
     }
@@ -180,6 +196,7 @@ static int pxb_dev_initfn(PCIDevice *dev)
 static Property pxb_dev_properties[] = {
     /* Note: 0 is not a legal a PXB bus number. */
     DEFINE_PROP_UINT8("bus_nr", PXBDev, bus_nr, 0),
+    DEFINE_PROP_UINT16("numa_node", PXBDev, numa_node, NUMA_NODE_UNASSIGNED),
     DEFINE_PROP_END_OF_LIST(),
 };
 
-- 
2.1.0

[Qemu-devel] [PATCH V8 16/17] apci: fix PXB behaviour if used with unsupported BIOS

[Marcel Apfelbaum]

PXB does not work with unsupported bioses, but should
not interfere with normal OS operation.
We don't ship them anymore, but it's reasonable
to keep the work-around until we update the bios in qemu.

Fix this by not adding PXB mem/IO chunks to _CRS
if they weren't configured by BIOS.

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 hw/i386/acpi-build.c | 87 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 58 insertions(+), 29 deletions(-)

diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 45c36a8..db32fd1 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -786,6 +786,14 @@ static Aml *build_crs(PCIHostState *host,
             range_base = r->addr;
             range_limit = r->addr + r->size - 1;
 
+            /*
+             * Work-around for old bioses
+             * that do not support multiple root buses
+             */
+            if (!range_base || range_base > range_limit) {
+                continue;
+            }
+
             if (r->type & PCI_BASE_ADDRESS_SPACE_IO) {
                 aml_append(crs,
                     aml_word_io(AML_MIN_FIXED, AML_MAX_FIXED,
@@ -819,45 +827,66 @@ static Aml *build_crs(PCIHostState *host,
 
             range_base = pci_bridge_get_base(dev, PCI_BASE_ADDRESS_SPACE_IO);
             range_limit = pci_bridge_get_limit(dev, PCI_BASE_ADDRESS_SPACE_IO);
-            aml_append(crs,
-                aml_word_io(AML_MIN_FIXED, AML_MAX_FIXED,
-                            AML_POS_DECODE, AML_ENTIRE_RANGE,
-                            0,
-                            range_base,
-                            range_limit,
-                            0,
-                            range_limit - range_base + 1));
-            crs_range_insert(io_ranges, range_base, range_limit);
+
+            /*
+             * Work-around for old bioses
+             * that do not support multiple root buses
+             */
+            if (range_base || range_base > range_limit) {
+                aml_append(crs,
+                           aml_word_io(AML_MIN_FIXED, AML_MAX_FIXED,
+                                       AML_POS_DECODE, AML_ENTIRE_RANGE,
+                                       0,
+                                       range_base,
+                                       range_limit,
+                                       0,
+                                       range_limit - range_base + 1));
+                crs_range_insert(io_ranges, range_base, range_limit);
+            }
 
             range_base =
                 pci_bridge_get_base(dev, PCI_BASE_ADDRESS_SPACE_MEMORY);
             range_limit =
                 pci_bridge_get_limit(dev, PCI_BASE_ADDRESS_SPACE_MEMORY);
-            aml_append(crs,
-                aml_dword_memory(AML_POS_DECODE, AML_MIN_FIXED,
-                                 AML_MAX_FIXED, AML_NON_CACHEABLE,
-                                 AML_READ_WRITE,
-                                 0,
-                                 range_base,
-                                 range_limit,
-                                 0,
-                                 range_limit - range_base + 1));
-            crs_range_insert(mem_ranges, range_base, range_limit);
+
+            /*
+             * Work-around for old bioses
+             * that do not support multiple root buses
+             */
+            if (range_base || range_base > range_limit) {
+                aml_append(crs,
+                           aml_dword_memory(AML_POS_DECODE, AML_MIN_FIXED,
+                                            AML_MAX_FIXED, AML_NON_CACHEABLE,
+                                            AML_READ_WRITE,
+                                            0,
+                                            range_base,
+                                            range_limit,
+                                            0,
+                                            range_limit - range_base + 1));
+                crs_range_insert(mem_ranges, range_base, range_limit);
+          }
 
             range_base =
                 pci_bridge_get_base(dev, PCI_BASE_ADDRESS_MEM_PREFETCH);
             range_limit =
                 pci_bridge_get_limit(dev, PCI_BASE_ADDRESS_MEM_PREFETCH);
-            aml_append(crs,
-                aml_dword_memory(AML_POS_DECODE, AML_MIN_FIXED,
-                                 AML_MAX_FIXED, AML_NON_CACHEABLE,
-                                 AML_READ_WRITE,
-                                 0,
-                                 range_base,
-                                 range_limit,
-                                 0,
-                                 range_limit - range_base + 1));
-            crs_range_insert(mem_ranges, range_base, range_limit);
+
+            /*
+             * Work-around for old bioses
+             * that do not support multiple root buses
+             */
+            if (range_base || range_base > range_limit) {
+                aml_append(crs,
+                           aml_dword_memory(AML_POS_DECODE, AML_MIN_FIXED,
+                                            AML_MAX_FIXED, AML_NON_CACHEABLE,
+                                            AML_READ_WRITE,
+                                            0,
+                                            range_base,
+                                            range_limit,
+                                            0,
+                                            range_limit - range_base + 1));
+                crs_range_insert(mem_ranges, range_base, range_limit);
+            }
         }
     }
 
-- 
2.1.0

[Qemu-devel] [PATCH V8 17/17] docs: Add PXB documentation

[Marcel Apfelbaum]

Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
---
 docs/pci_expander_bridge.txt | 58 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)
 create mode 100644 docs/pci_expander_bridge.txt

diff --git a/docs/pci_expander_bridge.txt b/docs/pci_expander_bridge.txt
new file mode 100644
index 0000000..d7913fb
--- /dev/null
+++ b/docs/pci_expander_bridge.txt
@@ -0,0 +1,58 @@
+PCI EXPANDER BRIDGE (PXB)
+=========================
+
+Description
+===========
+PXB is a "light-weight" host bridge in the same PCI domain
+as the main host bridge whose purpose is to enable
+the main host bridge to support multiple PCI root buses.
+It is implemented only for i440fx and can be placed only
+on bus 0 (pci.0).
+
+As opposed to PCI-2-PCI bridge's secondary bus, PXB's bus
+is a primary bus and can be associated with a NUMA node
+(different from the main host bridge) allowing the guest OS
+to recognize the proximity of a pass-through device to
+other resources as RAM and CPUs.
+
+Usage
+=====
+A detailed command line would be:
+
+[qemu-bin + storage options]
+-m 2G
+-object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 -numa node,nodeid=0,cpus=0,memdev=ram-node0
+-object memory-backend-ram,size=1024M,policy=bind,host-nodes=1,id=ram-node1 -numa node,nodeid=1,cpus=1,memdev=ram-node1
+-device pxb,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 -netdev user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd
+-device pxb,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8,bus=pci.0 -device e1000,bus=bridge2,addr=0x3
+-device pxb,id=bridge3,bus=pci.0,bus_nr=40,bus=pci.0 -drive if=none,id=drive0,file=[img] -device virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1
+
+Here you have:
+ - 2 NUMA nodes for the guest, 0 and 1. (both mapped to the same NUMA node in host, but you can and should put it in different host NUMA nodes)
+ - a pxb host bridge attached to NUMA 1 with an e1000 behind it
+ - a pxb host bridge attached to NUMA 0 with an e1000 behind it
+ - a pxb host bridge not attached to any NUMA with a hard drive behind it.
+
+Limitations
+===========
+Please observe that we specified the bus "pci.0" for the second and third pxb.
+This is because when no bus is given, another pxb can be selected by QEMU as default bus,
+however, PXBs can be placed only under the root bus.
+
+Implementation
+==============
+The PXB is composed by:
+- HostBridge (TYPE_PXB_HOST)
+  The host bridge allows to register and query the PXB's rPCI root bus in QEMU.
+- PXBDev(TYPE_PXB_DEVICE)
+  It is a regular PCI Device that resides on the piix host-bridge bus and its bus uses the same PCI domain.
+  However, the bus behind is exposed through ACPI as a primary PCI bus and starts a new PCI hierarchy.
+  The interrupts from devices behind the PXB are routed through this device the same as if it were a
+  PCI-2-PCI bridge. The _PRT follows the i440fx model.
+- PCIBridgeDev(TYPE_PCI_BRIDGE_DEV)
+  Created automatically as part of init sequence.
+  When adding a device to PXB it is attached to the bridge for two reasons:
+  - Using the bridge will enable hotplug support
+  - All the devices behind the bridge will use bridge's IO/MEM windows compacting
+    the PCI address space.
+
-- 
2.1.0

Re: [Qemu-devel] [PATCH V8 00/17] hw/pc: implement multiple primary busses for pc machines

[Laszlo Ersek]

On 06/02/15 13:22, Marcel Apfelbaum wrote:
> The series is fully functional.
>  - Limitations:
>    - Pxb's bus does not support hotplug. It will be addressed on top of this series
>      because is already getting to big.
>    - Pxb devices work only for i440fx and can be attached only to bus 0.
>  - You are more than welcome to try using:
>        -device pxb,id=pxb,bus_nr=4,numa_node=1 -device e1000,bus=pxb,addr=0x1
> 
> v7->v8:
>  - rebased on latest pci branch
>  - aml patches already taken
>  - added a fix for one of the aml patches
> 
> v6->v7:
>  - This version includes some refactoring requested by Michael S. Tsirking,
>    but no new functionality:
>    - Removed TYPE_PCI_MAIN_HOST_BRIDGE interface and scan only pc/q35 host-bridges when
>      needed, see patch 11/24.
>    - Removed TYPE_PCI_HOST_BRIDGE_SNOOPED interface and added PXB buses as child buses
>      of i440fx. The pci configuration is changed to support PXBs, see patch 12/24.
>    - Removed patch "hw/pci: move pci bus related code to separate files" and refactor
>      all patches that touched the new file.
>  - Addressed Paolo's review:
>    - Changed documentation to always use numa policy "bind".
> 
> v5->v6:
>  - This version includes a lot of refactoring requested by Michael S. Tsirking,
>    but no new/different functionality:
>    - Removed the HOST_BRIDGE_FOR_EACH loop because it too generic
>    - Reduced the generic "extra pci roots" aproach to a more "non-generic"
>      root bus having snooping buses listening to its configuration space.
>      Instead of going over all host bridges, we go only over the snooping
>      host bridges associated with the main host bridge.
>    - The current implementation made i440fx the only a "snooped" host bridge
>  - Addressed Michael S. Tsirkin's review:
>    - Replaced qmp queries with native pci ones.
>    - Squashed later patches into their places
>    - Tweaked a few comments
>  - Addressed Shannon Zhao's review:
>    - Used build_append_byte instead of build_append_int
>  - Addressed Gerd's review:
>    - Reduced the "Line over 80" warnings.
>  - Rebased on pci branch, tree: git://git.kernel.org/pub/scm/virt/kvm/mst/qemu.git
>    - a few days before, I hope is enough
>  - Changed some patches order
> 
> v4->v5:
>  - Rebased on pci branch, tree: git://git.kernel.org/pub/scm/virt/kvm/mst/qemu.git
>  - Added PXB documentation (patch 28/28)
>  - Addressed Gerd Hoffmann's review:
>    - fix PXB behaviour if used with unsupported BIOS (patch 27/28)
>  - Addressed Michael S. Tsirkin's review:
>    - Removed assert in aml_index (patch 5/28)
>    - Renamed pci_ functions to crs_ (patch 12/28)
>    - used uint64_t variables instead of signed ones (patch 12/28)
>    - Emit MEM/IO AML only for PXBs and i440fx (patch 26/28)
>  - Addressed Shannon Zhao's review:
>    - Changed build_append_int to  build_append_byte in aml_or (patch 2/25)
>  - Thanks to Igor and Kevin for reviews
> 
> v3->v4:
>  - Addressed Michael S. Tsirkin's review:
>    - refactored build_prt method (patch 11/25) 
>      hw/apci: add _PRT method for extra PCI root busses
>  - Addressed Igor Mammedov's reiew
>    - add assert to aml_index (patch 5/25)
>  - Fixed aml_equal implementation (patch 1/25)
> 
> v2->v3:
>  - Rebased on Michael S. Tsirkin's pci branch (that includes now all the dependencies)
>  - Refactored acpi terms patch into multiple patches to match Igor's design.
> 
> v1->v2:
>  - Add support for multiple pxb devices.
>  - Attach pxb's bus to specific NUMA node.
>  - Got rid of the hacks from prev version.
>  - Tested also for Win7 and Fedora 20, and for virtio blk devices.
>  - Several bug-fixes resulting in a stable version ready for submission.
> 
> Reasoning:
> We need multiple primary busess for a few reasons, the most important one
> is to be able to associate a pass-trough device with a guest NUMA node.
> The OS-es are able to associate a NUMA node only to a primary bus, not to
> a specific PCI device or a pci-2-pci bridge.
> PC machines support multiple NUMA nodes for CPUs and memory, however the IO
> was not yet supported.
> 
> 
> Marcel Apfelbaum (17):
>   acpi: add implementation of aml_while() term
>   hw/pci: made pci_bus_is_root a PCIBusClass method
>   hw/pci: made pci_bus_num a PCIBusClass method
>   hw/i386: query only for q35/pc when looking for pci host bridge
>   hw/pci: extend PCI config access to support devices behind PXB
>   hw/acpi: add support for i440fx 'snooping' root busses
>   hw/apci: add _PRT method for extra PCI root busses
>   hw/acpi: add _CRS method for extra root busses
>   hw/acpi: remove from root bus 0 the crs resources used by other buses.
>   hw/pci: removed 'rootbus nr is 0' assumption from qmp_pci_query
>   hw/pci: introduce PCI Expander Bridge (PXB)
>   hw/pci: inform bios if the system has extra pci root buses
>   hw/pxb: add map_irq func
>   hw/pci: add support for NUMA nodes
>   hw/pxb: add numa_node parameter
>   apci: fix PXB behaviour if used with unsupported BIOS
>   docs: Add PXB documentation
> 
>  docs/pci_expander_bridge.txt        |  58 ++++++
>  hw/acpi/aml-build.c                 |   8 +
>  hw/i386/acpi-build.c                | 396 ++++++++++++++++++++++++++++++++++--
>  hw/i386/pc.c                        |  20 ++
>  hw/pci-bridge/Makefile.objs         |   1 +
>  hw/pci-bridge/pci_expander_bridge.c | 231 +++++++++++++++++++++
>  hw/pci/pci.c                        |  78 +++++--
>  include/hw/pci/pci.h                |   4 +
>  include/hw/pci/pci_bus.h            |  10 +
>  include/sysemu/sysemu.h             |   1 +
>  10 files changed, 776 insertions(+), 31 deletions(-)
>  create mode 100644 docs/pci_expander_bridge.txt
>  create mode 100644 hw/pci-bridge/pci_expander_bridge.c
> 

series
Acked-by: Laszlo Ersek <lersek@redhat.com>