[PATCH v3 0/6] hw/arm/virt: Add support for user creatable SMMUv3 device

[PATCH v3 0/6] hw/arm/virt: Add support for user creatable SMMUv3 device

[Shameer Kolothum via]

Hi All,

Changes from v2:
https://lore.kernel.org/qemu-devel/20250502102707.110516-1-shameerali.kolothum.thodi@huawei.com/

Addressed feedback on v2. Thanks to all.
Major changes:
1. Not use the default "bus" property for associated PCIe RC.
   Instead use "primary-bus".
2. Split of IORT ACPI changes into two different patches(Patches 2 & 3).
3. Picked some R-by tags. (I haven't included Nicolin's R-by for the v2
   ACPI IORT patch as that is now splitted into patches 2 &3.)

Cover letter:

This patch series introduces support for a user-creatable SMMUv3 device
(-device arm-smmuv3) in QEMU.

The implementation is based on feedback received from the RFCv2[0]:
"hw/arm/virt: Add support for user-creatable accelerated SMMUv3"

Currently, QEMU's SMMUv3 emulation (iommu=smmuv3) is tied to the machine
and does not support instantiating multiple SMMUv3 devices—each associated
with a separate PCIe root complex. In contrast, real-world ARM systems
often include multiple SMMUv3 instances, each bound to a different PCIe
root complex.

This series allows to specify multiple SMMUv3 instances as below,

 -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0
  ...
 -device arm-smmuv3,primary-bus=pcie.1,,id=smmuv3.1

The multiple SMMUv3 instance support lays the groundwork for supporting
accelerated SMMUv3, as proposed in the aforementioned RFCv2[0]. The
proposed accelerated support will be an optional property like below,
-device arm-smmuv3,primary-bus=pcie.1,accel=on,..

Please note, the accelerated SMMUv3 support is not part of this series
and will be sent out as a separate series later on top of this one.

This series also,

-Supports either the legacy iommu=smmuv3 option or the new
  "-device arm-smmuv3" model.
  -Adds device tree bindings for the new SMMUv3 device on the arm/virt
   machine only, and only for the default pcie.0 root complex.
   (Note: pxb-pcie root complexes are currently not supported with the
    device tree due to known limitations[1].)

Example usage:
  -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0
  -device virtio-net-pci,bus=pcie.0
  -device pxb-pcie,id=pcie.1,bus_nr=2
  -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1
  -device pcie-root-port,id=pcie.port1,bus=pcie.1
  -device virtio-net-pci,bus=pcie.port1

Please take a look and let me know your feedback.

Thanks,
Shameer
[0]:https://lore.kernel.org/qemu-devel/20250311141045.66620-1-shameerali.kolothum.thodi@huawei.com/
[1]:https://lore.kernel.org/qemu-devel/20230421165037.2506-1-Jonathan.Cameron@huawei.com/

Nicolin Chen (1):
  hw/arm/virt: Add an SMMU_IO_LEN macro

Shameer Kolothum (5):
  hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association
  hw/arm/virt-acpi-build: Re-arrange SMMUv3 IORT build
  hw/arm/virt-acpi-build: Update IORT for multiple smmuv3 devices
  hw/arm/virt: Factor out common SMMUV3 dt bindings code
  hw/arm/virt: Allow user-creatable SMMUv3 dev instantiation

 hw/arm/smmuv3.c          |  10 +++
 hw/arm/virt-acpi-build.c | 164 +++++++++++++++++++++++++++++++--------
 hw/arm/virt.c            | 111 +++++++++++++++++++-------
 hw/core/sysbus-fdt.c     |   3 +
 include/hw/arm/virt.h    |   1 +
 5 files changed, 229 insertions(+), 60 deletions(-)

-- 
2.34.1

[PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association

[Shameer Kolothum via]

Although this change does not affect functionality at present, it is
required when we add support for user-creatable SMMUv3 devices in
future patches.

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 hw/arm/smmuv3.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index ab67972353..7e934336c2 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -24,6 +24,7 @@
 #include "hw/qdev-properties.h"
 #include "hw/qdev-core.h"
 #include "hw/pci/pci.h"
+#include "hw/pci/pci_bridge.h"
 #include "cpu.h"
 #include "exec/target_page.h"
 #include "trace.h"
@@ -1881,6 +1882,13 @@ static void smmu_realize(DeviceState *d, Error **errp)
     SMMUv3Class *c = ARM_SMMUV3_GET_CLASS(s);
     SysBusDevice *dev = SYS_BUS_DEVICE(d);
     Error *local_err = NULL;
+    Object *bus;
+
+    bus = object_property_get_link(OBJECT(d), "primary-bus", &error_abort);
+    if (!bus || !object_dynamic_cast(bus->parent, TYPE_PCI_HOST_BRIDGE)) {
+        error_setg(errp, "SMMUv3 is not attached to any PCIe Root Complex!");
+        return;
+    }
 
     c->parent_realize(d, &local_err);
     if (local_err) {
-- 
2.34.1

[PATCH v3 2/6] hw/arm/virt-acpi-build: Re-arrange SMMUv3 IORT build

[Shameer Kolothum via]

Introduces a new struct AcpiIortSMMUv3Dev to hold all the information
required for SMMUv3 IORT node and use that for populating the node.

The current machine wide SMMUv3 is named as legacy SMMUv3 as we will
soon add support for user-creatable SMMUv3 devices. These changes will
be useful to have common code paths when we add that support.

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 hw/arm/virt-acpi-build.c | 112 +++++++++++++++++++++++++++------------
 hw/arm/virt.c            |   1 +
 include/hw/arm/virt.h    |   1 +
 3 files changed, 80 insertions(+), 34 deletions(-)

diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
index 7e8e0f0298..bd26853ef6 100644
--- a/hw/arm/virt-acpi-build.c
+++ b/hw/arm/virt-acpi-build.c
@@ -266,6 +266,28 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b)
     return idmap_a->input_base - idmap_b->input_base;
 }
 
+struct AcpiIortSMMUv3Dev {
+    int irq;
+    hwaddr base;
+    GArray *idmaps;
+    size_t offset;
+};
+typedef struct AcpiIortSMMUv3Dev AcpiIortSMMUv3Dev;
+
+static void
+get_smmuv3_legacy_dev(VirtMachineState *vms, void *opaque)
+{
+    GArray *sdev_blob = opaque;
+    AcpiIortSMMUv3Dev sdev;
+
+    sdev.idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
+    object_child_foreach_recursive(object_get_root(),
+                                   iort_host_bridges, sdev.idmaps);
+    sdev.base = vms->memmap[VIRT_SMMU].base;
+    sdev.irq = vms->irqmap[VIRT_SMMU] + ARM_SPI_BASE;
+    g_array_append_val(sdev_blob, sdev);
+}
+
 /*
  * Input Output Remapping Table (IORT)
  * Conforms to "IO Remapping Table System Software on ARM Platforms",
@@ -274,11 +296,12 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b)
 static void
 build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
 {
-    int i, nb_nodes, rc_mapping_count;
-    size_t node_size, smmu_offset = 0;
-    AcpiIortIdMapping *idmap;
+    int i, j, nb_nodes, rc_mapping_count;
+    AcpiIortSMMUv3Dev *sdev;
+    size_t node_size;
+    int num_smmus = 0;
     uint32_t id = 0;
-    GArray *smmu_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
+    GArray *smmuv3_devs = g_array_new(false, true, sizeof(AcpiIortSMMUv3Dev));
     GArray *its_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
 
     AcpiTable table = { .sig = "IORT", .rev = 3, .oem_id = vms->oem_id,
@@ -286,28 +309,41 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     /* Table 2 The IORT */
     acpi_table_begin(&table, table_data);
 
-    if (vms->iommu == VIRT_IOMMU_SMMUV3) {
-        AcpiIortIdMapping next_range = {0};
-
-        object_child_foreach_recursive(object_get_root(),
-                                       iort_host_bridges, smmu_idmaps);
-
-        /* Sort the smmu idmap by input_base */
-        g_array_sort(smmu_idmaps, iort_idmap_compare);
+    nb_nodes = 2; /* RC, ITS */
+    if (vms->legacy_smmuv3_present) {
+        get_smmuv3_legacy_dev(vms, smmuv3_devs);
+        /*
+         * There will be only one legacy SMMUv3 as it is a machine wide one.
+         * And since it covers all the PCIe RCs in the machine, may have
+         * multiple SMMUv3 idmaps. Sort it by input_base.
+         */
+        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, 0);
+        g_array_sort(sdev->idmaps, iort_idmap_compare);
+    }
 
+    num_smmus = smmuv3_devs->len;
+    if (num_smmus) {
+        AcpiIortIdMapping next_range = {0};
+        int smmu_map_cnt = 0;
         /*
          * Split the whole RIDs by mapping from RC to SMMU,
          * build the ID mapping from RC to ITS directly.
          */
-        for (i = 0; i < smmu_idmaps->len; i++) {
-            idmap = &g_array_index(smmu_idmaps, AcpiIortIdMapping, i);
-
-            if (next_range.input_base < idmap->input_base) {
-                next_range.id_count = idmap->input_base - next_range.input_base;
-                g_array_append_val(its_idmaps, next_range);
+        for (i = 0; i < num_smmus; i++) {
+            AcpiIortIdMapping *idmap;
+            sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+
+            for (j = 0; j < sdev->idmaps->len; j++) {
+                idmap = &g_array_index(sdev->idmaps, AcpiIortIdMapping, j);
+
+                if (next_range.input_base < idmap->input_base) {
+                    next_range.id_count = idmap->input_base -
+                                          next_range.input_base;
+                    g_array_append_val(its_idmaps, next_range);
+                }
+                next_range.input_base = idmap->input_base + idmap->id_count;
+                smmu_map_cnt++;
             }
-
-            next_range.input_base = idmap->input_base + idmap->id_count;
         }
 
         /* Append the last RC -> ITS ID mapping */
@@ -316,10 +352,9 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
             g_array_append_val(its_idmaps, next_range);
         }
 
-        nb_nodes = 3; /* RC, ITS, SMMUv3 */
-        rc_mapping_count = smmu_idmaps->len + its_idmaps->len;
+        nb_nodes += num_smmus;
+        rc_mapping_count = smmu_map_cnt + its_idmaps->len;
     } else {
-        nb_nodes = 2; /* RC, ITS */
         rc_mapping_count = 1;
     }
     /* Number of IORT Nodes */
@@ -341,10 +376,11 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     /* GIC ITS Identifier Array */
     build_append_int_noprefix(table_data, 0 /* MADT translation_id */, 4);
 
-    if (vms->iommu == VIRT_IOMMU_SMMUV3) {
-        int irq =  vms->irqmap[VIRT_SMMU] + ARM_SPI_BASE;
+    for (i = 0; i < num_smmus; i++) {
+        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+        int irq = sdev->irq;
 
-        smmu_offset = table_data->len - table.table_offset;
+        sdev->offset = table_data->len - table.table_offset;
         /* Table 9 SMMUv3 Format */
         build_append_int_noprefix(table_data, 4 /* SMMUv3 */, 1); /* Type */
         node_size =  SMMU_V3_ENTRY_SIZE + ID_MAPPING_ENTRY_SIZE;
@@ -355,7 +391,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
         /* Reference to ID Array */
         build_append_int_noprefix(table_data, SMMU_V3_ENTRY_SIZE, 4);
         /* Base address */
-        build_append_int_noprefix(table_data, vms->memmap[VIRT_SMMU].base, 8);
+        build_append_int_noprefix(table_data, sdev->base, 8);
         /* Flags */
         build_append_int_noprefix(table_data, 1 /* COHACC Override */, 4);
         build_append_int_noprefix(table_data, 0, 4); /* Reserved */
@@ -404,15 +440,19 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     build_append_int_noprefix(table_data, 0, 3); /* Reserved */
 
     /* Output Reference */
-    if (vms->iommu == VIRT_IOMMU_SMMUV3) {
+    if (num_smmus) {
         AcpiIortIdMapping *range;
 
         /* translated RIDs connect to SMMUv3 node: RC -> SMMUv3 -> ITS */
-        for (i = 0; i < smmu_idmaps->len; i++) {
-            range = &g_array_index(smmu_idmaps, AcpiIortIdMapping, i);
-            /* output IORT node is the smmuv3 node */
-            build_iort_id_mapping(table_data, range->input_base,
-                                  range->id_count, smmu_offset);
+        for (i = 0; i < num_smmus; i++) {
+            sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+
+            for (j = 0; j < sdev->idmaps->len; j++) {
+                range = &g_array_index(sdev->idmaps, AcpiIortIdMapping, j);
+                /* output IORT node is the smmuv3 node */
+                build_iort_id_mapping(table_data, range->input_base,
+                                      range->id_count, sdev->offset);
+            }
         }
 
         /* bypassed RIDs connect to ITS group node directly: RC -> ITS */
@@ -428,8 +468,12 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
     }
 
     acpi_table_end(linker, &table);
-    g_array_free(smmu_idmaps, true);
     g_array_free(its_idmaps, true);
+    for (i = 0; i < num_smmus; i++) {
+        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
+        g_array_free(sdev->idmaps, true);
+    }
+    g_array_free(smmuv3_devs, true);
 }
 
 /*
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 9a6cd085a3..73bd2bd5f2 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1614,6 +1614,7 @@ static void create_pcie(VirtMachineState *vms)
             create_smmu(vms, vms->bus);
             qemu_fdt_setprop_cells(ms->fdt, nodename, "iommu-map",
                                    0x0, vms->iommu_phandle, 0x0, 0x10000);
+            vms->legacy_smmuv3_present = true;
             break;
         default:
             g_assert_not_reached();
diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h
index 9a1b0f53d2..8b1404b5f6 100644
--- a/include/hw/arm/virt.h
+++ b/include/hw/arm/virt.h
@@ -174,6 +174,7 @@ struct VirtMachineState {
     char *oem_id;
     char *oem_table_id;
     bool ns_el2_virt_timer_irq;
+    bool legacy_smmuv3_present;
 };
 
 #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)
-- 
2.34.1

[PATCH v3 3/6] hw/arm/virt-acpi-build: Update IORT for multiple smmuv3 devices

[Shameer Kolothum via]

With the soon to be introduced user-creatable SMMUv3 devices for
virt, it is possible to have multiple SMMUv3 devices associated
with different PCIe root complexes.

Update IORT nodes accordingly.

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 hw/arm/virt-acpi-build.c | 54 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
index bd26853ef6..b4ff71b8b7 100644
--- a/hw/arm/virt-acpi-build.c
+++ b/hw/arm/virt-acpi-build.c
@@ -43,6 +43,7 @@
 #include "hw/acpi/generic_event_device.h"
 #include "hw/acpi/tpm.h"
 #include "hw/acpi/hmat.h"
+#include "hw/arm/smmuv3.h"
 #include "hw/pci/pcie_host.h"
 #include "hw/pci/pci.h"
 #include "hw/pci/pci_bus.h"
@@ -288,6 +289,54 @@ get_smmuv3_legacy_dev(VirtMachineState *vms, void *opaque)
     g_array_append_val(sdev_blob, sdev);
 }
 
+static int smmuv3_dev_idmap_compare(gconstpointer a, gconstpointer b)
+{
+    AcpiIortSMMUv3Dev *sdev_a = (AcpiIortSMMUv3Dev *)a;
+    AcpiIortSMMUv3Dev *sdev_b = (AcpiIortSMMUv3Dev *)b;
+    AcpiIortIdMapping *map_a = &g_array_index(sdev_a->idmaps,
+                                              AcpiIortIdMapping, 0);
+    AcpiIortIdMapping *map_b = &g_array_index(sdev_b->idmaps,
+                                              AcpiIortIdMapping, 0);
+    return map_a->input_base - map_b->input_base;
+}
+
+static int get_smmuv3_devices(Object *obj, void *opaque)
+{
+    VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine());
+    GArray *sdev_blob = opaque;
+    AcpiIortIdMapping idmap;
+    PlatformBusDevice *pbus;
+    AcpiIortSMMUv3Dev sdev;
+    int min_bus, max_bus;
+    SysBusDevice *sbdev;
+    PCIBus *bus;
+
+    if (!object_dynamic_cast(obj, TYPE_ARM_SMMUV3)) {
+        return 0;
+    }
+
+    bus = PCI_BUS(object_property_get_link(obj, "primary-bus", &error_abort));
+    if (!bus) {
+        return 0;
+    }
+
+    pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
+    sbdev = SYS_BUS_DEVICE(obj);
+    sdev.base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
+    sdev.base += vms->memmap[VIRT_PLATFORM_BUS].base;
+    sdev.irq = platform_bus_get_irqn(pbus, sbdev, 0);
+    sdev.irq += vms->irqmap[VIRT_PLATFORM_BUS];
+    sdev.irq += ARM_SPI_BASE;
+
+    pci_bus_range(bus, &min_bus, &max_bus);
+    sdev.idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
+    idmap.input_base = min_bus << 8,
+    idmap.id_count = (max_bus - min_bus + 1) << 8,
+    g_array_append_val(sdev.idmaps, idmap);
+    g_array_append_val(sdev_blob, sdev);
+    return 0;
+}
+
 /*
  * Input Output Remapping Table (IORT)
  * Conforms to "IO Remapping Table System Software on ARM Platforms",
@@ -319,6 +368,11 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
          */
         sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, 0);
         g_array_sort(sdev->idmaps, iort_idmap_compare);
+    } else {
+        object_child_foreach_recursive(object_get_root(),
+                                       get_smmuv3_devices, smmuv3_devs);
+        /* Sort the smmuv3 devices(if any) by smmu idmap input_base */
+        g_array_sort(smmuv3_devs, smmuv3_dev_idmap_compare);
     }
 
     num_smmus = smmuv3_devs->len;
-- 
2.34.1

[PATCH v3 4/6] hw/arm/virt: Factor out common SMMUV3 dt bindings code

[Shameer Kolothum via]

No functional changes intended. This will be useful when we
add support for user-creatable smmuv3 device.

Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 hw/arm/virt.c | 54 +++++++++++++++++++++++++++------------------------
 1 file changed, 29 insertions(+), 25 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 73bd2bd5f2..71b923f786 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1412,19 +1412,43 @@ static void create_pcie_irq_map(const MachineState *ms,
                            0x7           /* PCI irq */);
 }
 
+static void create_smmuv3_dt_bindings(const VirtMachineState *vms, hwaddr base,
+                                      hwaddr size, int irq)
+{
+    char *node;
+    const char compat[] = "arm,smmu-v3";
+    const char irq_names[] = "eventq\0priq\0cmdq-sync\0gerror";
+    MachineState *ms = MACHINE(vms);
+
+    node = g_strdup_printf("/smmuv3@%" PRIx64, base);
+    qemu_fdt_add_subnode(ms->fdt, node);
+    qemu_fdt_setprop(ms->fdt, node, "compatible", compat, sizeof(compat));
+    qemu_fdt_setprop_sized_cells(ms->fdt, node, "reg", 2, base, 2, size);
+
+    qemu_fdt_setprop_cells(ms->fdt, node, "interrupts",
+            GIC_FDT_IRQ_TYPE_SPI, irq    , GIC_FDT_IRQ_FLAGS_EDGE_LO_HI,
+            GIC_FDT_IRQ_TYPE_SPI, irq + 1, GIC_FDT_IRQ_FLAGS_EDGE_LO_HI,
+            GIC_FDT_IRQ_TYPE_SPI, irq + 2, GIC_FDT_IRQ_FLAGS_EDGE_LO_HI,
+            GIC_FDT_IRQ_TYPE_SPI, irq + 3, GIC_FDT_IRQ_FLAGS_EDGE_LO_HI);
+
+    qemu_fdt_setprop(ms->fdt, node, "interrupt-names", irq_names,
+                     sizeof(irq_names));
+
+    qemu_fdt_setprop(ms->fdt, node, "dma-coherent", NULL, 0);
+    qemu_fdt_setprop_cell(ms->fdt, node, "#iommu-cells", 1);
+    qemu_fdt_setprop_cell(ms->fdt, node, "phandle", vms->iommu_phandle);
+    g_free(node);
+}
+
 static void create_smmu(const VirtMachineState *vms,
                         PCIBus *bus)
 {
     VirtMachineClass *vmc = VIRT_MACHINE_GET_CLASS(vms);
-    char *node;
-    const char compat[] = "arm,smmu-v3";
     int irq =  vms->irqmap[VIRT_SMMU];
     int i;
     hwaddr base = vms->memmap[VIRT_SMMU].base;
     hwaddr size = vms->memmap[VIRT_SMMU].size;
-    const char irq_names[] = "eventq\0priq\0cmdq-sync\0gerror";
     DeviceState *dev;
-    MachineState *ms = MACHINE(vms);
 
     if (vms->iommu != VIRT_IOMMU_SMMUV3 || !vms->iommu_phandle) {
         return;
@@ -1443,27 +1467,7 @@ static void create_smmu(const VirtMachineState *vms,
         sysbus_connect_irq(SYS_BUS_DEVICE(dev), i,
                            qdev_get_gpio_in(vms->gic, irq + i));
     }
-
-    node = g_strdup_printf("/smmuv3@%" PRIx64, base);
-    qemu_fdt_add_subnode(ms->fdt, node);
-    qemu_fdt_setprop(ms->fdt, node, "compatible", compat, sizeof(compat));
-    qemu_fdt_setprop_sized_cells(ms->fdt, node, "reg", 2, base, 2, size);
-
-    qemu_fdt_setprop_cells(ms->fdt, node, "interrupts",
-            GIC_FDT_IRQ_TYPE_SPI, irq    , GIC_FDT_IRQ_FLAGS_EDGE_LO_HI,
-            GIC_FDT_IRQ_TYPE_SPI, irq + 1, GIC_FDT_IRQ_FLAGS_EDGE_LO_HI,
-            GIC_FDT_IRQ_TYPE_SPI, irq + 2, GIC_FDT_IRQ_FLAGS_EDGE_LO_HI,
-            GIC_FDT_IRQ_TYPE_SPI, irq + 3, GIC_FDT_IRQ_FLAGS_EDGE_LO_HI);
-
-    qemu_fdt_setprop(ms->fdt, node, "interrupt-names", irq_names,
-                     sizeof(irq_names));
-
-    qemu_fdt_setprop(ms->fdt, node, "dma-coherent", NULL, 0);
-
-    qemu_fdt_setprop_cell(ms->fdt, node, "#iommu-cells", 1);
-
-    qemu_fdt_setprop_cell(ms->fdt, node, "phandle", vms->iommu_phandle);
-    g_free(node);
+    create_smmuv3_dt_bindings(vms, base, size, irq);
 }
 
 static void create_virtio_iommu_dt_bindings(VirtMachineState *vms)
-- 
2.34.1

[PATCH v3 5/6] hw/arm/virt: Add an SMMU_IO_LEN macro

[Shameer Kolothum via]

From: Nicolin Chen <nicolinc@nvidia.com>

This is useful as the subsequent support for new SMMUv3 dev will also
use the same.

Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Donald Dutile <ddutile@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 hw/arm/virt.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 71b923f786..eeace4754d 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -146,6 +146,9 @@ static void arm_virt_compat_set(MachineClass *mc)
 #define LEGACY_RAMLIMIT_GB 255
 #define LEGACY_RAMLIMIT_BYTES (LEGACY_RAMLIMIT_GB * GiB)
 
+/* MMIO region size for SMMUv3 */
+#define SMMU_IO_LEN 0x20000
+
 /* Addresses and sizes of our components.
  * 0..128MB is space for a flash device so we can run bootrom code such as UEFI.
  * 128MB..256MB is used for miscellaneous device I/O.
@@ -177,7 +180,7 @@ static const MemMapEntry base_memmap[] = {
     [VIRT_FW_CFG] =             { 0x09020000, 0x00000018 },
     [VIRT_GPIO] =               { 0x09030000, 0x00001000 },
     [VIRT_UART1] =              { 0x09040000, 0x00001000 },
-    [VIRT_SMMU] =               { 0x09050000, 0x00020000 },
+    [VIRT_SMMU] =               { 0x09050000, SMMU_IO_LEN },
     [VIRT_PCDIMM_ACPI] =        { 0x09070000, MEMORY_HOTPLUG_IO_LEN },
     [VIRT_ACPI_GED] =           { 0x09080000, ACPI_GED_EVT_SEL_LEN },
     [VIRT_NVDIMM_ACPI] =        { 0x09090000, NVDIMM_ACPI_IO_LEN},
@@ -1447,7 +1450,6 @@ static void create_smmu(const VirtMachineState *vms,
     int irq =  vms->irqmap[VIRT_SMMU];
     int i;
     hwaddr base = vms->memmap[VIRT_SMMU].base;
-    hwaddr size = vms->memmap[VIRT_SMMU].size;
     DeviceState *dev;
 
     if (vms->iommu != VIRT_IOMMU_SMMUV3 || !vms->iommu_phandle) {
@@ -1467,7 +1469,7 @@ static void create_smmu(const VirtMachineState *vms,
         sysbus_connect_irq(SYS_BUS_DEVICE(dev), i,
                            qdev_get_gpio_in(vms->gic, irq + i));
     }
-    create_smmuv3_dt_bindings(vms, base, size, irq);
+    create_smmuv3_dt_bindings(vms, base, SMMU_IO_LEN, irq);
 }
 
 static void create_virtio_iommu_dt_bindings(VirtMachineState *vms)
-- 
2.34.1

[PATCH v3 6/6] hw/arm/virt: Allow user-creatable SMMUv3 dev instantiation

[Shameer Kolothum via]

Allow cold-plug of smmuv3 device to virt if there is no machine
wide legacy smmuv3 or a virtio-iommu is specified.

Device tree support for new smmuv3 dev is limited to the case where
it is associated with the default pcie.0 RC.

Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
---
 hw/arm/smmuv3.c      |  2 ++
 hw/arm/virt.c        | 50 ++++++++++++++++++++++++++++++++++++++++++++
 hw/core/sysbus-fdt.c |  3 +++
 3 files changed, 55 insertions(+)

diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
index 7e934336c2..d1f66dcd8f 100644
--- a/hw/arm/smmuv3.c
+++ b/hw/arm/smmuv3.c
@@ -2004,6 +2004,8 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)
     device_class_set_parent_realize(dc, smmu_realize,
                                     &c->parent_realize);
     device_class_set_props(dc, smmuv3_properties);
+    dc->hotpluggable = false;
+    dc->user_creatable = true;
 }
 
 static int smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index eeace4754d..04bc940fc3 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -56,6 +56,7 @@
 #include "qemu/cutils.h"
 #include "qemu/error-report.h"
 #include "qemu/module.h"
+#include "hw/pci/pci_bus.h"
 #include "hw/pci-host/gpex.h"
 #include "hw/virtio/virtio-pci.h"
 #include "hw/core/sysbus-fdt.h"
@@ -1443,6 +1444,28 @@ static void create_smmuv3_dt_bindings(const VirtMachineState *vms, hwaddr base,
     g_free(node);
 }
 
+static void create_smmuv3_dev_dtb(VirtMachineState *vms,
+                                  DeviceState *dev, PCIBus *bus)
+{
+    PlatformBusDevice *pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
+    SysBusDevice *sbdev = SYS_BUS_DEVICE(dev);
+    int irq = platform_bus_get_irqn(pbus, sbdev, 0);
+    hwaddr base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
+    MachineState *ms = MACHINE(vms);
+
+    if (strcmp("pcie.0", bus->qbus.name)) {
+        warn_report("SMMUv3 device only supported with pcie.0 for DT");
+        return;
+    }
+    base += vms->memmap[VIRT_PLATFORM_BUS].base;
+    irq += vms->irqmap[VIRT_PLATFORM_BUS];
+
+    vms->iommu_phandle = qemu_fdt_alloc_phandle(ms->fdt);
+    create_smmuv3_dt_bindings(vms, base, SMMU_IO_LEN, irq);
+    qemu_fdt_setprop_cells(ms->fdt, vms->pciehb_nodename, "iommu-map",
+                           0x0, vms->iommu_phandle, 0x0, 0x10000);
+}
+
 static void create_smmu(const VirtMachineState *vms,
                         PCIBus *bus)
 {
@@ -2931,6 +2954,13 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
         qlist_append_str(reserved_regions, resv_prop_str);
         qdev_prop_set_array(dev, "reserved-regions", reserved_regions);
         g_free(resv_prop_str);
+    } else if (object_dynamic_cast(OBJECT(dev), TYPE_ARM_SMMUV3)) {
+        if (vms->legacy_smmuv3_present || vms->iommu == VIRT_IOMMU_VIRTIO) {
+            error_setg(errp, "virt machine already has %s set. "
+                       "Doesn't support incompatible iommus",
+                       (vms->legacy_smmuv3_present) ?
+                       "iommu=smmuv3" : "virtio-iommu");
+        }
     }
 }
 
@@ -2954,6 +2984,25 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
         virtio_md_pci_plug(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp);
     }
 
+    if (object_dynamic_cast(OBJECT(dev), TYPE_ARM_SMMUV3)) {
+        if (!vms->legacy_smmuv3_present && vms->platform_bus_dev) {
+            PCIBus *bus;
+
+            bus = PCI_BUS(object_property_get_link(OBJECT(dev), "primary-bus",
+                                                   &error_abort));
+            if (pci_bus_bypass_iommu(bus)) {
+                error_setg(errp, "Bypass option cannot be set for SMMUv3 "
+                           "associated PCIe RC");
+                return;
+            }
+
+            create_smmuv3_dev_dtb(vms, dev, bus);
+            if (vms->iommu != VIRT_IOMMU_SMMUV3) {
+                vms->iommu = VIRT_IOMMU_SMMUV3;
+            }
+        }
+    }
+
     if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
         PCIDevice *pdev = PCI_DEVICE(dev);
 
@@ -3156,6 +3205,7 @@ static void virt_machine_class_init(ObjectClass *oc, const void *data)
     machine_class_allow_dynamic_sysbus_dev(mc, TYPE_RAMFB_DEVICE);
     machine_class_allow_dynamic_sysbus_dev(mc, TYPE_VFIO_PLATFORM);
     machine_class_allow_dynamic_sysbus_dev(mc, TYPE_UEFI_VARS_SYSBUS);
+    machine_class_allow_dynamic_sysbus_dev(mc, TYPE_ARM_SMMUV3);
 #ifdef CONFIG_TPM
     machine_class_allow_dynamic_sysbus_dev(mc, TYPE_TPM_TIS_SYSBUS);
 #endif
diff --git a/hw/core/sysbus-fdt.c b/hw/core/sysbus-fdt.c
index c339a27875..e80776080b 100644
--- a/hw/core/sysbus-fdt.c
+++ b/hw/core/sysbus-fdt.c
@@ -31,6 +31,7 @@
 #include "qemu/error-report.h"
 #include "system/device_tree.h"
 #include "system/tpm.h"
+#include "hw/arm/smmuv3.h"
 #include "hw/platform-bus.h"
 #include "hw/vfio/vfio-platform.h"
 #include "hw/vfio/vfio-calxeda-xgmac.h"
@@ -518,6 +519,8 @@ static const BindingEntry bindings[] = {
 #ifdef CONFIG_TPM
     TYPE_BINDING(TYPE_TPM_TIS_SYSBUS, add_tpm_tis_fdt_node),
 #endif
+    /* No generic DT support for smmuv3 dev. Support added for arm virt only */
+    TYPE_BINDING(TYPE_ARM_SMMUV3, no_fdt_node),
     TYPE_BINDING(TYPE_RAMFB_DEVICE, no_fdt_node),
     TYPE_BINDING(TYPE_UEFI_VARS_SYSBUS, add_uefi_vars_node),
     TYPE_BINDING("", NULL), /* last element */
-- 
2.34.1

Re: [PATCH v3 0/6] hw/arm/virt: Add support for user creatable SMMUv3 device

[Nathan Chen]

On 6/2/2025 8:41 AM, Shameer Kolothum wrote:
> This patch series introduces support for a user-creatable SMMUv3 device
> (-device arm-smmuv3) in QEMU.

Tested-by: Nathan Chen <nathanc@nvidia.com>

I am able to create 16 SMMUv3 devices in a qemu VM with emulated devices 
properly associated with the guest SMMUs in guest sysfs - verified with 
some guest SMMUs having two or three emulated NICs assigned to them 
while other guest SMMUs have a minimum of one assigned.

Thanks,
Nathan

Re: [PATCH v3 0/6] hw/arm/virt: Add support for user creatable SMMUv3 device

[Donald Dutile]

On 6/4/25 10:02 PM, Nathan Chen wrote:
> On 6/2/2025 8:41 AM, Shameer Kolothum wrote:
>> This patch series introduces support for a user-creatable SMMUv3 device
>> (-device arm-smmuv3) in QEMU.
> 
> Tested-by: Nathan Chen <nathanc@nvidia.com>
> 
> I am able to create 16 SMMUv3 devices in a qemu VM with emulated devices properly associated with the guest SMMUs in guest sysfs - verified with some guest SMMUs having two or three emulated NICs assigned to them while other guest SMMUs have a minimum of one assigned.
> 
> Thanks,
> Nathan
> 
Nathan,
Great test!
Can you share the xml &/or qemu cmdline, to demonstrate this broad capability?
Also would be a good reference when (we know it isn't 'if') a new/different config
that wasn't tested shows a bug, and we'll have more info to work with.
Thanks,
  Don

Re: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association

[Eric Auger]

Hi Shameer,

On 6/2/25 5:41 PM, Shameer Kolothum wrote:
> Although this change does not affect functionality at present, it is
> required when we add support for user-creatable SMMUv3 devices in
> future patches.
>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric
> ---
>  hw/arm/smmuv3.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> index ab67972353..7e934336c2 100644
> --- a/hw/arm/smmuv3.c
> +++ b/hw/arm/smmuv3.c
> @@ -24,6 +24,7 @@
>  #include "hw/qdev-properties.h"
>  #include "hw/qdev-core.h"
>  #include "hw/pci/pci.h"
> +#include "hw/pci/pci_bridge.h"
>  #include "cpu.h"
>  #include "exec/target_page.h"
>  #include "trace.h"
> @@ -1881,6 +1882,13 @@ static void smmu_realize(DeviceState *d, Error **errp)
>      SMMUv3Class *c = ARM_SMMUV3_GET_CLASS(s);
>      SysBusDevice *dev = SYS_BUS_DEVICE(d);
>      Error *local_err = NULL;
> +    Object *bus;
> +
> +    bus = object_property_get_link(OBJECT(d), "primary-bus", &error_abort);
> +    if (!bus || !object_dynamic_cast(bus->parent, TYPE_PCI_HOST_BRIDGE)) {
> +        error_setg(errp, "SMMUv3 is not attached to any PCIe Root Complex!");
> +        return;
> +    }
>  
>      c->parent_realize(d, &local_err);
>      if (local_err) {

Re: [PATCH v3 2/6] hw/arm/virt-acpi-build: Re-arrange SMMUv3 IORT build

[Eric Auger]

Hi Shameer,

On 6/2/25 5:41 PM, Shameer Kolothum wrote:
> Introduces a new struct AcpiIortSMMUv3Dev to hold all the information
> required for SMMUv3 IORT node and use that for populating the node.
>
> The current machine wide SMMUv3 is named as legacy SMMUv3 as we will
> soon add support for user-creatable SMMUv3 devices. These changes will
> be useful to have common code paths when we add that support.
>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  hw/arm/virt-acpi-build.c | 112 +++++++++++++++++++++++++++------------
>  hw/arm/virt.c            |   1 +
>  include/hw/arm/virt.h    |   1 +
>  3 files changed, 80 insertions(+), 34 deletions(-)
>
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index 7e8e0f0298..bd26853ef6 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -266,6 +266,28 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b)
>      return idmap_a->input_base - idmap_b->input_base;
>  }
>  
> +struct AcpiIortSMMUv3Dev {
> +    int irq;
> +    hwaddr base;
> +    GArray *idmaps;
> +    size_t offset;
I would suggest to comment at least the offset field which is not obvious
> +};
> +typedef struct AcpiIortSMMUv3Dev AcpiIortSMMUv3Dev;
> +
> +static void
> +get_smmuv3_legacy_dev(VirtMachineState *vms, void *opaque)
> +{
> +    GArray *sdev_blob = opaque;
I don't get why we use an opaque pointer here.
> +    AcpiIortSMMUv3Dev sdev;
> +
> +    sdev.idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
> +    object_child_foreach_recursive(object_get_root(),
> +                                   iort_host_bridges, sdev.idmaps);
> +    sdev.base = vms->memmap[VIRT_SMMU].base;
> +    sdev.irq = vms->irqmap[VIRT_SMMU] + ARM_SPI_BASE;
> +    g_array_append_val(sdev_blob, sdev);
So if I understand correctly this helper populate an AcpiIortSMMUv3Dev
object with the legacy iommu info and add it in the array of
AcpiIortSMMUv3Dev. The name of the helper does not really reflect what
it does and generally get is associated with a put. I would suggest to
rename. I see that afterwards you call

sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, 0);
to retrieve the first element and then sort the ipmap array.

Why can't you do all that stuff in the helper?

> +}
> +
>  /*
>   * Input Output Remapping Table (IORT)
>   * Conforms to "IO Remapping Table System Software on ARM Platforms",
> @@ -274,11 +296,12 @@ static int iort_idmap_compare(gconstpointer a, gconstpointer b)
>  static void
>  build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
>  {
> -    int i, nb_nodes, rc_mapping_count;
> -    size_t node_size, smmu_offset = 0;
> -    AcpiIortIdMapping *idmap;
> +    int i, j, nb_nodes, rc_mapping_count;
> +    AcpiIortSMMUv3Dev *sdev;
> +    size_t node_size;
> +    int num_smmus = 0;
>      uint32_t id = 0;
> -    GArray *smmu_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
> +    GArray *smmuv3_devs = g_array_new(false, true, sizeof(AcpiIortSMMUv3Dev));
>      GArray *its_idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
>  
>      AcpiTable table = { .sig = "IORT", .rev = 3, .oem_id = vms->oem_id,
> @@ -286,28 +309,41 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
>      /* Table 2 The IORT */
>      acpi_table_begin(&table, table_data);
>  
> -    if (vms->iommu == VIRT_IOMMU_SMMUV3) {
> -        AcpiIortIdMapping next_range = {0};
> -
> -        object_child_foreach_recursive(object_get_root(),
> -                                       iort_host_bridges, smmu_idmaps);
> -
> -        /* Sort the smmu idmap by input_base */
> -        g_array_sort(smmu_idmaps, iort_idmap_compare);
> +    nb_nodes = 2; /* RC, ITS */
> +    if (vms->legacy_smmuv3_present) {
> +        get_smmuv3_legacy_dev(vms, smmuv3_devs);
> +        /*
> +         * There will be only one legacy SMMUv3 as it is a machine wide one.
> +         * And since it covers all the PCIe RCs in the machine, may have
> +         * multiple SMMUv3 idmaps. Sort it by input_base.
> +         */
> +        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, 0);
> +        g_array_sort(sdev->idmaps, iort_idmap_compare);
> +    }
>  
> +    num_smmus = smmuv3_devs->len;
> +    if (num_smmus) {
> +        AcpiIortIdMapping next_range = {0};
> +        int smmu_map_cnt = 0;
>          /*
>           * Split the whole RIDs by mapping from RC to SMMU,
>           * build the ID mapping from RC to ITS directly.
>           */
> -        for (i = 0; i < smmu_idmaps->len; i++) {
> -            idmap = &g_array_index(smmu_idmaps, AcpiIortIdMapping, i);
> -
> -            if (next_range.input_base < idmap->input_base) {
> -                next_range.id_count = idmap->input_base - next_range.input_base;
> -                g_array_append_val(its_idmaps, next_range);
> +        for (i = 0; i < num_smmus; i++) {
> +            AcpiIortIdMapping *idmap;
could belong to the inner block

Otherwise looks good to me

Cheers

Eric
> +            sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
> +
> +            for (j = 0; j < sdev->idmaps->len; j++) {
> +                idmap = &g_array_index(sdev->idmaps, AcpiIortIdMapping, j);
> +
> +                if (next_range.input_base < idmap->input_base) {
> +                    next_range.id_count = idmap->input_base -
> +                                          next_range.input_base;
> +                    g_array_append_val(its_idmaps, next_range);
> +                }
> +                next_range.input_base = idmap->input_base + idmap->id_count;
> +                smmu_map_cnt++;
>              }
> -
> -            next_range.input_base = idmap->input_base + idmap->id_count;
>          }
>  
>          /* Append the last RC -> ITS ID mapping */
> @@ -316,10 +352,9 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
>              g_array_append_val(its_idmaps, next_range);
>          }
>  
> -        nb_nodes = 3; /* RC, ITS, SMMUv3 */
> -        rc_mapping_count = smmu_idmaps->len + its_idmaps->len;
> +        nb_nodes += num_smmus;
> +        rc_mapping_count = smmu_map_cnt + its_idmaps->len;
>      } else {
> -        nb_nodes = 2; /* RC, ITS */
>          rc_mapping_count = 1;
>      }
>      /* Number of IORT Nodes */
> @@ -341,10 +376,11 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
>      /* GIC ITS Identifier Array */
>      build_append_int_noprefix(table_data, 0 /* MADT translation_id */, 4);
>  
> -    if (vms->iommu == VIRT_IOMMU_SMMUV3) {
> -        int irq =  vms->irqmap[VIRT_SMMU] + ARM_SPI_BASE;
> +    for (i = 0; i < num_smmus; i++) {
> +        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
> +        int irq = sdev->irq;
>  
> -        smmu_offset = table_data->len - table.table_offset;
> +        sdev->offset = table_data->len - table.table_offset;
>          /* Table 9 SMMUv3 Format */
>          build_append_int_noprefix(table_data, 4 /* SMMUv3 */, 1); /* Type */
>          node_size =  SMMU_V3_ENTRY_SIZE + ID_MAPPING_ENTRY_SIZE;
> @@ -355,7 +391,7 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
>          /* Reference to ID Array */
>          build_append_int_noprefix(table_data, SMMU_V3_ENTRY_SIZE, 4);
>          /* Base address */
> -        build_append_int_noprefix(table_data, vms->memmap[VIRT_SMMU].base, 8);
> +        build_append_int_noprefix(table_data, sdev->base, 8);
>          /* Flags */
>          build_append_int_noprefix(table_data, 1 /* COHACC Override */, 4);
>          build_append_int_noprefix(table_data, 0, 4); /* Reserved */
> @@ -404,15 +440,19 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
>      build_append_int_noprefix(table_data, 0, 3); /* Reserved */
>  
>      /* Output Reference */
> -    if (vms->iommu == VIRT_IOMMU_SMMUV3) {
> +    if (num_smmus) {
>          AcpiIortIdMapping *range;
>  
>          /* translated RIDs connect to SMMUv3 node: RC -> SMMUv3 -> ITS */
> -        for (i = 0; i < smmu_idmaps->len; i++) {
> -            range = &g_array_index(smmu_idmaps, AcpiIortIdMapping, i);
> -            /* output IORT node is the smmuv3 node */
> -            build_iort_id_mapping(table_data, range->input_base,
> -                                  range->id_count, smmu_offset);
> +        for (i = 0; i < num_smmus; i++) {
> +            sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
> +
> +            for (j = 0; j < sdev->idmaps->len; j++) {
> +                range = &g_array_index(sdev->idmaps, AcpiIortIdMapping, j);
> +                /* output IORT node is the smmuv3 node */
> +                build_iort_id_mapping(table_data, range->input_base,
> +                                      range->id_count, sdev->offset);
> +            }
>          }
>  
>          /* bypassed RIDs connect to ITS group node directly: RC -> ITS */
> @@ -428,8 +468,12 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
>      }
>  
>      acpi_table_end(linker, &table);
> -    g_array_free(smmu_idmaps, true);
>      g_array_free(its_idmaps, true);
> +    for (i = 0; i < num_smmus; i++) {
> +        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
> +        g_array_free(sdev->idmaps, true);
> +    }
> +    g_array_free(smmuv3_devs, true);
>  }
>  
>  /*
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 9a6cd085a3..73bd2bd5f2 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -1614,6 +1614,7 @@ static void create_pcie(VirtMachineState *vms)
>              create_smmu(vms, vms->bus);
>              qemu_fdt_setprop_cells(ms->fdt, nodename, "iommu-map",
>                                     0x0, vms->iommu_phandle, 0x0, 0x10000);
> +            vms->legacy_smmuv3_present = true;
>              break;
>          default:
>              g_assert_not_reached();
> diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h
> index 9a1b0f53d2..8b1404b5f6 100644
> --- a/include/hw/arm/virt.h
> +++ b/include/hw/arm/virt.h
> @@ -174,6 +174,7 @@ struct VirtMachineState {
>      char *oem_id;
>      char *oem_table_id;
>      bool ns_el2_virt_timer_irq;
> +    bool legacy_smmuv3_present;
>  };
>  
>  #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM : VIRT_PCIE_ECAM)

Re: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association

[Eric Auger]

On 6/2/25 5:41 PM, Shameer Kolothum wrote:
> Although this change does not affect functionality at present, it is
> required when we add support for user-creatable SMMUv3 devices in
> future patches.
>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  hw/arm/smmuv3.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> index ab67972353..7e934336c2 100644
> --- a/hw/arm/smmuv3.c
> +++ b/hw/arm/smmuv3.c
> @@ -24,6 +24,7 @@
>  #include "hw/qdev-properties.h"
>  #include "hw/qdev-core.h"
>  #include "hw/pci/pci.h"
> +#include "hw/pci/pci_bridge.h"
>  #include "cpu.h"
>  #include "exec/target_page.h"
>  #include "trace.h"
> @@ -1881,6 +1882,13 @@ static void smmu_realize(DeviceState *d, Error **errp)
>      SMMUv3Class *c = ARM_SMMUV3_GET_CLASS(s);
>      SysBusDevice *dev = SYS_BUS_DEVICE(d);
>      Error *local_err = NULL;
> +    Object *bus;
> +
> +    bus = object_property_get_link(OBJECT(d), "primary-bus", &error_abort);
> +    if (!bus || !object_dynamic_cast(bus->parent, TYPE_PCI_HOST_BRIDGE)) {
> +        error_setg(errp, "SMMUv3 is not attached to any PCIe Root Complex!");
> +        return;
> +    }
shouldn't you check as well that !pci_bus_bypass_iommu(bus)?

Eric
>  
>      c->parent_realize(d, &local_err);
>      if (local_err) {

Re: [PATCH v3 3/6] hw/arm/virt-acpi-build: Update IORT for multiple smmuv3 devices

[Eric Auger]

Hi Shameer,
On 6/2/25 5:41 PM, Shameer Kolothum wrote:
> With the soon to be introduced user-creatable SMMUv3 devices for
> virt, it is possible to have multiple SMMUv3 devices associated
> with different PCIe root complexes.
>
> Update IORT nodes accordingly.
>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  hw/arm/virt-acpi-build.c | 54 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 54 insertions(+)
>
> diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> index bd26853ef6..b4ff71b8b7 100644
> --- a/hw/arm/virt-acpi-build.c
> +++ b/hw/arm/virt-acpi-build.c
> @@ -43,6 +43,7 @@
>  #include "hw/acpi/generic_event_device.h"
>  #include "hw/acpi/tpm.h"
>  #include "hw/acpi/hmat.h"
> +#include "hw/arm/smmuv3.h"
>  #include "hw/pci/pcie_host.h"
>  #include "hw/pci/pci.h"
>  #include "hw/pci/pci_bus.h"
> @@ -288,6 +289,54 @@ get_smmuv3_legacy_dev(VirtMachineState *vms, void *opaque)
>      g_array_append_val(sdev_blob, sdev);
>  }
>  
> +static int smmuv3_dev_idmap_compare(gconstpointer a, gconstpointer b)
> +{
> +    AcpiIortSMMUv3Dev *sdev_a = (AcpiIortSMMUv3Dev *)a;
> +    AcpiIortSMMUv3Dev *sdev_b = (AcpiIortSMMUv3Dev *)b;
> +    AcpiIortIdMapping *map_a = &g_array_index(sdev_a->idmaps,
> +                                              AcpiIortIdMapping, 0);
> +    AcpiIortIdMapping *map_b = &g_array_index(sdev_b->idmaps,
> +                                              AcpiIortIdMapping, 0);
> +    return map_a->input_base - map_b->input_base;
> +}
> +
> +static int get_smmuv3_devices(Object *obj, void *opaque)
> +{
> +    VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine());
> +    GArray *sdev_blob = opaque;
> +    AcpiIortIdMapping idmap;
> +    PlatformBusDevice *pbus;
> +    AcpiIortSMMUv3Dev sdev;
> +    int min_bus, max_bus;
> +    SysBusDevice *sbdev;
> +    PCIBus *bus;
> +
> +    if (!object_dynamic_cast(obj, TYPE_ARM_SMMUV3)) {
> +        return 0;
> +    }
> +
> +    bus = PCI_BUS(object_property_get_link(obj, "primary-bus", &error_abort));
> +    if (!bus) {
> +        return 0;
> +    }
can it happen after patch 1?
> +
> +    pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
> +    sbdev = SYS_BUS_DEVICE(obj);
> +    sdev.base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
> +    sdev.base += vms->memmap[VIRT_PLATFORM_BUS].base;
> +    sdev.irq = platform_bus_get_irqn(pbus, sbdev, 0);
> +    sdev.irq += vms->irqmap[VIRT_PLATFORM_BUS];
> +    sdev.irq += ARM_SPI_BASE;
> +
> +    pci_bus_range(bus, &min_bus, &max_bus);
> +    sdev.idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
> +    idmap.input_base = min_bus << 8,
> +    idmap.id_count = (max_bus - min_bus + 1) << 8,
> +    g_array_append_val(sdev.idmaps, idmap);
> +    g_array_append_val(sdev_blob, sdev);
> +    return 0;
> +}
> +
>  /*
>   * Input Output Remapping Table (IORT)
>   * Conforms to "IO Remapping Table System Software on ARM Platforms",
> @@ -319,6 +368,11 @@ build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState *vms)
>           */
>          sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, 0);
>          g_array_sort(sdev->idmaps, iort_idmap_compare);
> +    } else {
> +        object_child_foreach_recursive(object_get_root(),
> +                                       get_smmuv3_devices, smmuv3_devs);
> +        /* Sort the smmuv3 devices(if any) by smmu idmap input_base */
> +        g_array_sort(smmuv3_devs, smmuv3_dev_idmap_compare);
Given the gorwing complexity of the IDMAP computation computation I
would suggest to add some IDMAP examples either in comment or commit msg
to illustrate how the idmaps look like (similar to what can be found in
the IORT spec).

Thanks

Eric
>      }
>  
>      num_smmus = smmuv3_devs->len;

Re: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association

[Eric Auger via]

On 6/5/25 11:53 AM, Eric Auger wrote:
> 
> 
> On 6/2/25 5:41 PM, Shameer Kolothum wrote:
>> Although this change does not affect functionality at present, it is
>> required when we add support for user-creatable SMMUv3 devices in
>> future patches.
>>
>> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
>> ---
>>  hw/arm/smmuv3.c | 8 ++++++++
>>  1 file changed, 8 insertions(+)
>>
>> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
>> index ab67972353..7e934336c2 100644
>> --- a/hw/arm/smmuv3.c
>> +++ b/hw/arm/smmuv3.c
>> @@ -24,6 +24,7 @@
>>  #include "hw/qdev-properties.h"
>>  #include "hw/qdev-core.h"
>>  #include "hw/pci/pci.h"
>> +#include "hw/pci/pci_bridge.h"
>>  #include "cpu.h"
>>  #include "exec/target_page.h"
>>  #include "trace.h"
>> @@ -1881,6 +1882,13 @@ static void smmu_realize(DeviceState *d, Error **errp)
>>      SMMUv3Class *c = ARM_SMMUV3_GET_CLASS(s);
>>      SysBusDevice *dev = SYS_BUS_DEVICE(d);
>>      Error *local_err = NULL;
>> +    Object *bus;
>> +
>> +    bus = object_property_get_link(OBJECT(d), "primary-bus", &error_abort);
>> +    if (!bus || !object_dynamic_cast(bus->parent, TYPE_PCI_HOST_BRIDGE)) {
>> +        error_setg(errp, "SMMUv3 is not attached to any PCIe Root Complex!");
>> +        return;
>> +    }
> shouldn't you check as well that !pci_bus_bypass_iommu(bus)?

I see you do the check in 6/6 and I think this is the correct way
because in case of legacy SMMU it is allowed to have
pci_bus_bypass_iommu set on the root bus to let the SMMU apply only on
pxb buses only. So please ignore this comment.

Eric
> 
> Eric
>>  
>>      c->parent_realize(d, &local_err);
>>      if (local_err) {
> 

Re: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association

[Igor Mammedov]

On Mon, 2 Jun 2025 16:41:05 +0100
Shameer Kolothum via <qemu-devel@nongnu.org> wrote:

> Although this change does not affect functionality at present, it is
> required when we add support for user-creatable SMMUv3 devices in
> future patches.
> 
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  hw/arm/smmuv3.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> index ab67972353..7e934336c2 100644
> --- a/hw/arm/smmuv3.c
> +++ b/hw/arm/smmuv3.c
> @@ -24,6 +24,7 @@
>  #include "hw/qdev-properties.h"
>  #include "hw/qdev-core.h"
>  #include "hw/pci/pci.h"
> +#include "hw/pci/pci_bridge.h"
>  #include "cpu.h"
>  #include "exec/target_page.h"
>  #include "trace.h"
> @@ -1881,6 +1882,13 @@ static void smmu_realize(DeviceState *d, Error **errp)
>      SMMUv3Class *c = ARM_SMMUV3_GET_CLASS(s);
>      SysBusDevice *dev = SYS_BUS_DEVICE(d);
>      Error *local_err = NULL;
> +    Object *bus;
> +
> +    bus = object_property_get_link(OBJECT(d), "primary-bus", &error_abort);
I'd replace this with direct field access like in smmu_base_realize

in QEMU with PCI, usually we specify bus to attach to with 'bus' property,
wouldn't it better to rename "primary-bus" to 'bus' to be consistent with
the rest of PCI code (and before "primary-bus" shows up as a CLI option,
so far (before this series) it looks like it's an internal property)?

> +    if (!bus || !object_dynamic_cast(bus->parent, TYPE_PCI_HOST_BRIDGE)) {
Also looking at smmu_base_realize, it has NULL pointer check already.
Which also rises question, shouldn't smmu_base_realize check for
TYPE_PCI_HOST_BRIDGE as well (aka can smmu be attached to anything else but a host bridge)?


> +        error_setg(errp, "SMMUv3 is not attached to any PCIe Root Complex!");
> +        return;
> +    }
>  
>      c->parent_realize(d, &local_err);
>      if (local_err) {

RE: [PATCH v3 2/6] hw/arm/virt-acpi-build: Re-arrange SMMUv3 IORT build

[Shameerali Kolothum Thodi via]

Hi Eric,

> -----Original Message-----
> From: Eric Auger <eric.auger@redhat.com>
> Sent: Thursday, June 5, 2025 10:40 AM
> To: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>; qemu-
> arm@nongnu.org; qemu-devel@nongnu.org
> Cc: peter.maydell@linaro.org; jgg@nvidia.com; nicolinc@nvidia.com;
> ddutile@redhat.com; berrange@redhat.com; nathanc@nvidia.com;
> mochs@nvidia.com; smostafa@google.com; linuxarm@huawei.com;
> wangzhou1@hisilicon.com; jiangkunkun@huawei.com;
> jonathan.cameron@huawei.com; zhangfei.gao@linaro.org
> Subject: Re: [PATCH v3 2/6] hw/arm/virt-acpi-build: Re-arrange SMMUv3
> IORT build
> 
> Hi Shameer,
> 
> On 6/2/25 5:41 PM, Shameer Kolothum wrote:
> > Introduces a new struct AcpiIortSMMUv3Dev to hold all the information
> > required for SMMUv3 IORT node and use that for populating the node.
> >
> > The current machine wide SMMUv3 is named as legacy SMMUv3 as we
> will
> > soon add support for user-creatable SMMUv3 devices. These changes will
> > be useful to have common code paths when we add that support.
> >
> > Signed-off-by: Shameer Kolothum
> <shameerali.kolothum.thodi@huawei.com>
> > ---
> >  hw/arm/virt-acpi-build.c | 112 +++++++++++++++++++++++++++------------
> >  hw/arm/virt.c            |   1 +
> >  include/hw/arm/virt.h    |   1 +
> >  3 files changed, 80 insertions(+), 34 deletions(-)
> >
> > diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
> > index 7e8e0f0298..bd26853ef6 100644
> > --- a/hw/arm/virt-acpi-build.c
> > +++ b/hw/arm/virt-acpi-build.c
> > @@ -266,6 +266,28 @@ static int iort_idmap_compare(gconstpointer a,
> gconstpointer b)
> >      return idmap_a->input_base - idmap_b->input_base;
> >  }
> >
> > +struct AcpiIortSMMUv3Dev {
> > +    int irq;
> > +    hwaddr base;
> > +    GArray *idmaps;
> > +    size_t offset;
> I would suggest to comment at least the offset field which is not obvious

Ok.

> > +};
> > +typedef struct AcpiIortSMMUv3Dev AcpiIortSMMUv3Dev;
> > +
> > +static void
> > +get_smmuv3_legacy_dev(VirtMachineState *vms, void *opaque)
> > +{
> > +    GArray *sdev_blob = opaque;
> I don't get why we use an opaque pointer here.

Yeah. That was based on a previous comment from Nicolin I guess to make
it similar to the new SMMUv3 dev handling code in next patch,
get_smmuv3_devices().

I will revisit and see if that still makes sense. Otherwise will get rid of it.

> > +    AcpiIortSMMUv3Dev sdev;
> > +
> > +    sdev.idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
> > +    object_child_foreach_recursive(object_get_root(),
> > +                                   iort_host_bridges, sdev.idmaps);
> > +    sdev.base = vms->memmap[VIRT_SMMU].base;
> > +    sdev.irq = vms->irqmap[VIRT_SMMU] + ARM_SPI_BASE;
> > +    g_array_append_val(sdev_blob, sdev);
> So if I understand correctly this helper populate an AcpiIortSMMUv3Dev
> object with the legacy iommu info and add it in the array of
> AcpiIortSMMUv3Dev. The name of the helper does not really reflect what
> it does and generally get is associated with a put. I would suggest to
> rename. I see that afterwards you call
> 
> sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, 0);
> to retrieve the first element and then sort the ipmap array.
> 
> Why can't you do all that stuff in the helper?

Ok. I will revisit this and may be good to fold everything into one function.

> 
> > +}
> > +
> >  /*
> >   * Input Output Remapping Table (IORT)
> >   * Conforms to "IO Remapping Table System Software on ARM
> Platforms",
> > @@ -274,11 +296,12 @@ static int iort_idmap_compare(gconstpointer a,
> gconstpointer b)
> >  static void
> >  build_iort(GArray *table_data, BIOSLinker *linker, VirtMachineState
> *vms)
> >  {
> > -    int i, nb_nodes, rc_mapping_count;
> > -    size_t node_size, smmu_offset = 0;
> > -    AcpiIortIdMapping *idmap;
> > +    int i, j, nb_nodes, rc_mapping_count;
> > +    AcpiIortSMMUv3Dev *sdev;
> > +    size_t node_size;
> > +    int num_smmus = 0;
> >      uint32_t id = 0;
> > -    GArray *smmu_idmaps = g_array_new(false, true,
> sizeof(AcpiIortIdMapping));
> > +    GArray *smmuv3_devs = g_array_new(false, true,
> sizeof(AcpiIortSMMUv3Dev));
> >      GArray *its_idmaps = g_array_new(false, true,
> sizeof(AcpiIortIdMapping));
> >
> >      AcpiTable table = { .sig = "IORT", .rev = 3, .oem_id = vms->oem_id,
> > @@ -286,28 +309,41 @@ build_iort(GArray *table_data, BIOSLinker
> *linker, VirtMachineState *vms)
> >      /* Table 2 The IORT */
> >      acpi_table_begin(&table, table_data);
> >
> > -    if (vms->iommu == VIRT_IOMMU_SMMUV3) {
> > -        AcpiIortIdMapping next_range = {0};
> > -
> > -        object_child_foreach_recursive(object_get_root(),
> > -                                       iort_host_bridges, smmu_idmaps);
> > -
> > -        /* Sort the smmu idmap by input_base */
> > -        g_array_sort(smmu_idmaps, iort_idmap_compare);
> > +    nb_nodes = 2; /* RC, ITS */
> > +    if (vms->legacy_smmuv3_present) {
> > +        get_smmuv3_legacy_dev(vms, smmuv3_devs);
> > +        /*
> > +         * There will be only one legacy SMMUv3 as it is a machine wide
> one.
> > +         * And since it covers all the PCIe RCs in the machine, may have
> > +         * multiple SMMUv3 idmaps. Sort it by input_base.
> > +         */
> > +        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, 0);
> > +        g_array_sort(sdev->idmaps, iort_idmap_compare);
> > +    }
> >
> > +    num_smmus = smmuv3_devs->len;
> > +    if (num_smmus) {
> > +        AcpiIortIdMapping next_range = {0};
> > +        int smmu_map_cnt = 0;
> >          /*
> >           * Split the whole RIDs by mapping from RC to SMMU,
> >           * build the ID mapping from RC to ITS directly.
> >           */
> > -        for (i = 0; i < smmu_idmaps->len; i++) {
> > -            idmap = &g_array_index(smmu_idmaps, AcpiIortIdMapping, i);
> > -
> > -            if (next_range.input_base < idmap->input_base) {
> > -                next_range.id_count = idmap->input_base -
> next_range.input_base;
> > -                g_array_append_val(its_idmaps, next_range);
> > +        for (i = 0; i < num_smmus; i++) {
> > +            AcpiIortIdMapping *idmap;
> could belong to the inner block
> 
> Otherwise looks good to me

Thanks,
Shameer

> Cheers
> 
> Eric
> > +            sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
> > +
> > +            for (j = 0; j < sdev->idmaps->len; j++) {
> > +                idmap = &g_array_index(sdev->idmaps, AcpiIortIdMapping, j);
> > +
> > +                if (next_range.input_base < idmap->input_base) {
> > +                    next_range.id_count = idmap->input_base -
> > +                                          next_range.input_base;
> > +                    g_array_append_val(its_idmaps, next_range);
> > +                }
> > +                next_range.input_base = idmap->input_base + idmap-
> >id_count;
> > +                smmu_map_cnt++;
> >              }
> > -
> > -            next_range.input_base = idmap->input_base + idmap->id_count;
> >          }
> >
> >          /* Append the last RC -> ITS ID mapping */
> > @@ -316,10 +352,9 @@ build_iort(GArray *table_data, BIOSLinker
> *linker, VirtMachineState *vms)
> >              g_array_append_val(its_idmaps, next_range);
> >          }
> >
> > -        nb_nodes = 3; /* RC, ITS, SMMUv3 */
> > -        rc_mapping_count = smmu_idmaps->len + its_idmaps->len;
> > +        nb_nodes += num_smmus;
> > +        rc_mapping_count = smmu_map_cnt + its_idmaps->len;
> >      } else {
> > -        nb_nodes = 2; /* RC, ITS */
> >          rc_mapping_count = 1;
> >      }
> >      /* Number of IORT Nodes */
> > @@ -341,10 +376,11 @@ build_iort(GArray *table_data, BIOSLinker
> *linker, VirtMachineState *vms)
> >      /* GIC ITS Identifier Array */
> >      build_append_int_noprefix(table_data, 0 /* MADT translation_id */,
> 4);
> >
> > -    if (vms->iommu == VIRT_IOMMU_SMMUV3) {
> > -        int irq =  vms->irqmap[VIRT_SMMU] + ARM_SPI_BASE;
> > +    for (i = 0; i < num_smmus; i++) {
> > +        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
> > +        int irq = sdev->irq;
> >
> > -        smmu_offset = table_data->len - table.table_offset;
> > +        sdev->offset = table_data->len - table.table_offset;
> >          /* Table 9 SMMUv3 Format */
> >          build_append_int_noprefix(table_data, 4 /* SMMUv3 */, 1); /* Type
> */
> >          node_size =  SMMU_V3_ENTRY_SIZE + ID_MAPPING_ENTRY_SIZE;
> > @@ -355,7 +391,7 @@ build_iort(GArray *table_data, BIOSLinker *linker,
> VirtMachineState *vms)
> >          /* Reference to ID Array */
> >          build_append_int_noprefix(table_data, SMMU_V3_ENTRY_SIZE, 4);
> >          /* Base address */
> > -        build_append_int_noprefix(table_data, vms-
> >memmap[VIRT_SMMU].base, 8);
> > +        build_append_int_noprefix(table_data, sdev->base, 8);
> >          /* Flags */
> >          build_append_int_noprefix(table_data, 1 /* COHACC Override */, 4);
> >          build_append_int_noprefix(table_data, 0, 4); /* Reserved */
> > @@ -404,15 +440,19 @@ build_iort(GArray *table_data, BIOSLinker
> *linker, VirtMachineState *vms)
> >      build_append_int_noprefix(table_data, 0, 3); /* Reserved */
> >
> >      /* Output Reference */
> > -    if (vms->iommu == VIRT_IOMMU_SMMUV3) {
> > +    if (num_smmus) {
> >          AcpiIortIdMapping *range;
> >
> >          /* translated RIDs connect to SMMUv3 node: RC -> SMMUv3 -> ITS
> */
> > -        for (i = 0; i < smmu_idmaps->len; i++) {
> > -            range = &g_array_index(smmu_idmaps, AcpiIortIdMapping, i);
> > -            /* output IORT node is the smmuv3 node */
> > -            build_iort_id_mapping(table_data, range->input_base,
> > -                                  range->id_count, smmu_offset);
> > +        for (i = 0; i < num_smmus; i++) {
> > +            sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
> > +
> > +            for (j = 0; j < sdev->idmaps->len; j++) {
> > +                range = &g_array_index(sdev->idmaps, AcpiIortIdMapping, j);
> > +                /* output IORT node is the smmuv3 node */
> > +                build_iort_id_mapping(table_data, range->input_base,
> > +                                      range->id_count, sdev->offset);
> > +            }
> >          }
> >
> >          /* bypassed RIDs connect to ITS group node directly: RC -> ITS */
> > @@ -428,8 +468,12 @@ build_iort(GArray *table_data, BIOSLinker
> *linker, VirtMachineState *vms)
> >      }
> >
> >      acpi_table_end(linker, &table);
> > -    g_array_free(smmu_idmaps, true);
> >      g_array_free(its_idmaps, true);
> > +    for (i = 0; i < num_smmus; i++) {
> > +        sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, i);
> > +        g_array_free(sdev->idmaps, true);
> > +    }
> > +    g_array_free(smmuv3_devs, true);
> >  }
> >
> >  /*
> > diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> > index 9a6cd085a3..73bd2bd5f2 100644
> > --- a/hw/arm/virt.c
> > +++ b/hw/arm/virt.c
> > @@ -1614,6 +1614,7 @@ static void create_pcie(VirtMachineState *vms)
> >              create_smmu(vms, vms->bus);
> >              qemu_fdt_setprop_cells(ms->fdt, nodename, "iommu-map",
> >                                     0x0, vms->iommu_phandle, 0x0, 0x10000);
> > +            vms->legacy_smmuv3_present = true;
> >              break;
> >          default:
> >              g_assert_not_reached();
> > diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h
> > index 9a1b0f53d2..8b1404b5f6 100644
> > --- a/include/hw/arm/virt.h
> > +++ b/include/hw/arm/virt.h
> > @@ -174,6 +174,7 @@ struct VirtMachineState {
> >      char *oem_id;
> >      char *oem_table_id;
> >      bool ns_el2_virt_timer_irq;
> > +    bool legacy_smmuv3_present;
> >  };
> >
> >  #define VIRT_ECAM_ID(high) (high ? VIRT_HIGH_PCIE_ECAM :
> VIRT_PCIE_ECAM)
> 

RE: [PATCH v3 3/6] hw/arm/virt-acpi-build: Update IORT for multiple smmuv3 devices

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Eric Auger <eric.auger@redhat.com>
> Sent: Thursday, June 5, 2025 10:58 AM
> To: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
> qemu-devel@nongnu.org
> Cc: peter.maydell@linaro.org; jgg@nvidia.com; nicolinc@nvidia.com;
> ddutile@redhat.com; berrange@redhat.com; nathanc@nvidia.com;
> mochs@nvidia.com; smostafa@google.com; Linuxarm
> <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
> Subject: Re: [PATCH v3 3/6] hw/arm/virt-acpi-build: Update IORT for
> multiple smmuv3 devices
> 
> Hi Shameer,
> On 6/2/25 5:41 PM, Shameer Kolothum wrote:
> > With the soon to be introduced user-creatable SMMUv3 devices for virt,
> > it is possible to have multiple SMMUv3 devices associated with
> > different PCIe root complexes.
> >
> > Update IORT nodes accordingly.
> >
> > Signed-off-by: Shameer Kolothum
> <shameerali.kolothum.thodi@huawei.com>
> > ---
> >  hw/arm/virt-acpi-build.c | 54
> > ++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 54 insertions(+)
> >
> > diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c index
> > bd26853ef6..b4ff71b8b7 100644
> > --- a/hw/arm/virt-acpi-build.c
> > +++ b/hw/arm/virt-acpi-build.c
> > @@ -43,6 +43,7 @@
> >  #include "hw/acpi/generic_event_device.h"
> >  #include "hw/acpi/tpm.h"
> >  #include "hw/acpi/hmat.h"
> > +#include "hw/arm/smmuv3.h"
> >  #include "hw/pci/pcie_host.h"
> >  #include "hw/pci/pci.h"
> >  #include "hw/pci/pci_bus.h"
> > @@ -288,6 +289,54 @@ get_smmuv3_legacy_dev(VirtMachineState *vms,
> void *opaque)
> >      g_array_append_val(sdev_blob, sdev);  }
> >
> > +static int smmuv3_dev_idmap_compare(gconstpointer a, gconstpointer
> b)
> > +{
> > +    AcpiIortSMMUv3Dev *sdev_a = (AcpiIortSMMUv3Dev *)a;
> > +    AcpiIortSMMUv3Dev *sdev_b = (AcpiIortSMMUv3Dev *)b;
> > +    AcpiIortIdMapping *map_a = &g_array_index(sdev_a->idmaps,
> > +                                              AcpiIortIdMapping, 0);
> > +    AcpiIortIdMapping *map_b = &g_array_index(sdev_b->idmaps,
> > +                                              AcpiIortIdMapping, 0);
> > +    return map_a->input_base - map_b->input_base; }
> > +
> > +static int get_smmuv3_devices(Object *obj, void *opaque) {
> > +    VirtMachineState *vms = VIRT_MACHINE(qdev_get_machine());
> > +    GArray *sdev_blob = opaque;
> > +    AcpiIortIdMapping idmap;
> > +    PlatformBusDevice *pbus;
> > +    AcpiIortSMMUv3Dev sdev;
> > +    int min_bus, max_bus;
> > +    SysBusDevice *sbdev;
> > +    PCIBus *bus;
> > +
> > +    if (!object_dynamic_cast(obj, TYPE_ARM_SMMUV3)) {
> > +        return 0;
> > +    }
> > +
> > +    bus = PCI_BUS(object_property_get_link(obj, "primary-bus",
> &error_abort));
> > +    if (!bus) {
> > +        return 0;
> > +    }
> can it happen after patch 1?

Not really. Will remove.

> > +
> > +    pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
> > +    sbdev = SYS_BUS_DEVICE(obj);
> > +    sdev.base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
> > +    sdev.base += vms->memmap[VIRT_PLATFORM_BUS].base;
> > +    sdev.irq = platform_bus_get_irqn(pbus, sbdev, 0);
> > +    sdev.irq += vms->irqmap[VIRT_PLATFORM_BUS];
> > +    sdev.irq += ARM_SPI_BASE;
> > +
> > +    pci_bus_range(bus, &min_bus, &max_bus);
> > +    sdev.idmaps = g_array_new(false, true, sizeof(AcpiIortIdMapping));
> > +    idmap.input_base = min_bus << 8,
> > +    idmap.id_count = (max_bus - min_bus + 1) << 8,
> > +    g_array_append_val(sdev.idmaps, idmap);
> > +    g_array_append_val(sdev_blob, sdev);
> > +    return 0;
> > +}
> > +
> >  /*
> >   * Input Output Remapping Table (IORT)
> >   * Conforms to "IO Remapping Table System Software on ARM
> Platforms",
> > @@ -319,6 +368,11 @@ build_iort(GArray *table_data, BIOSLinker
> *linker, VirtMachineState *vms)
> >           */
> >          sdev = &g_array_index(smmuv3_devs, AcpiIortSMMUv3Dev, 0);
> >          g_array_sort(sdev->idmaps, iort_idmap_compare);
> > +    } else {
> > +        object_child_foreach_recursive(object_get_root(),
> > +                                       get_smmuv3_devices, smmuv3_devs);
> > +        /* Sort the smmuv3 devices(if any) by smmu idmap input_base */
> > +        g_array_sort(smmuv3_devs, smmuv3_dev_idmap_compare);
> Given the gorwing complexity of the IDMAP computation computation I
> would suggest to add some IDMAP examples either in comment or commit
> msg to illustrate how the idmaps look like (similar to what can be found in
> the IORT spec).

Ok. I will add some comments with example to describe the mappings.

Thanks,
Shameer
 

RE: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Eric Auger <eric.auger@redhat.com>
> Sent: Thursday, June 5, 2025 11:02 AM
> To: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
> qemu-devel@nongnu.org
> Cc: peter.maydell@linaro.org; jgg@nvidia.com; nicolinc@nvidia.com;
> ddutile@redhat.com; berrange@redhat.com; nathanc@nvidia.com;
> mochs@nvidia.com; smostafa@google.com; Linuxarm
> <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
> Subject: Re: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root
> Complex association
> 
> > shouldn't you check as well that !pci_bus_bypass_iommu(bus)?
> 
> I see you do the check in 6/6 and I think this is the correct way because in
> case of legacy SMMU it is allowed to have pci_bus_bypass_iommu set on
> the root bus to let the SMMU apply only on pxb buses only. So please
> ignore this comment.

Yes. It is to keep the existing behavior for legacy SMMUv3.

Thanks,
Shameer

RE: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Igor Mammedov <imammedo@redhat.com>
> Sent: Thursday, June 5, 2025 11:55 AM
> To: Shameer Kolothum via <qemu-devel@nongnu.org>
> Cc: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
> eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
> nicolinc@nvidia.com; ddutile@redhat.com; berrange@redhat.com;
> nathanc@nvidia.com; mochs@nvidia.com; smostafa@google.com; Linuxarm
> <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
> Subject: Re: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root
> Complex association
> 
> On Mon, 2 Jun 2025 16:41:05 +0100
> Shameer Kolothum via <qemu-devel@nongnu.org> wrote:
> 
> > Although this change does not affect functionality at present, it is
> > required when we add support for user-creatable SMMUv3 devices in
> > future patches.
> >
> > Signed-off-by: Shameer Kolothum
> <shameerali.kolothum.thodi@huawei.com>
> > ---
> >  hw/arm/smmuv3.c | 8 ++++++++
> >  1 file changed, 8 insertions(+)
> >
> > diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> > index ab67972353..7e934336c2 100644
> > --- a/hw/arm/smmuv3.c
> > +++ b/hw/arm/smmuv3.c
> > @@ -24,6 +24,7 @@
> >  #include "hw/qdev-properties.h"
> >  #include "hw/qdev-core.h"
> >  #include "hw/pci/pci.h"
> > +#include "hw/pci/pci_bridge.h"
> >  #include "cpu.h"
> >  #include "exec/target_page.h"
> >  #include "trace.h"
> > @@ -1881,6 +1882,13 @@ static void smmu_realize(DeviceState *d, Error
> **errp)
> >      SMMUv3Class *c = ARM_SMMUV3_GET_CLASS(s);
> >      SysBusDevice *dev = SYS_BUS_DEVICE(d);
> >      Error *local_err = NULL;
> > +    Object *bus;
> > +
> > +    bus = object_property_get_link(OBJECT(d), "primary-bus",
> &error_abort);
> I'd replace this with direct field access like in smmu_base_realize

Ok.
 
> in QEMU with PCI, usually we specify bus to attach to with 'bus' property,
> wouldn't it better to rename "primary-bus" to 'bus' to be consistent with
> the rest of PCI code (and before "primary-bus" shows up as a CLI option,
> so far (before this series) it looks like it's an internal property)?

That was tried in v2 and since SMMUv3 is not a pci device by itself(it is a 
sysbus device) reusing the default "bus" property to establish an association
with a PCI bus created problems,
https://lore.kernel.org/qemu-devel/877c2ut0zk.fsf@pond.sub.org/

 
> > +    if (!bus || !object_dynamic_cast(bus->parent,
> TYPE_PCI_HOST_BRIDGE)) {
> Also looking at smmu_base_realize, it has NULL pointer check already.
> Which also rises question, shouldn't smmu_base_realize check for
> TYPE_PCI_HOST_BRIDGE as well (aka can smmu be attached to anything
> else but a host bridge)?

Not at the moment in Qemu. Though the SMMUv3 specification allows it to
be associated with non-pci devices as well.

Thanks,
Shameer

Re: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association

[Igor Mammedov]

On Thu, 5 Jun 2025 11:29:59 +0000
Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote:

> > -----Original Message-----
> > From: Igor Mammedov <imammedo@redhat.com>
> > Sent: Thursday, June 5, 2025 11:55 AM
> > To: Shameer Kolothum via <qemu-devel@nongnu.org>
> > Cc: Shameerali Kolothum Thodi
> > <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
> > eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
> > nicolinc@nvidia.com; ddutile@redhat.com; berrange@redhat.com;
> > nathanc@nvidia.com; mochs@nvidia.com; smostafa@google.com; Linuxarm
> > <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
> > jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
> > <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
> > Subject: Re: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root
> > Complex association
> > 
> > On Mon, 2 Jun 2025 16:41:05 +0100
> > Shameer Kolothum via <qemu-devel@nongnu.org> wrote:
> >   
> > > Although this change does not affect functionality at present, it is
> > > required when we add support for user-creatable SMMUv3 devices in
> > > future patches.
> > >
> > > Signed-off-by: Shameer Kolothum  
> > <shameerali.kolothum.thodi@huawei.com>  
> > > ---
> > >  hw/arm/smmuv3.c | 8 ++++++++
> > >  1 file changed, 8 insertions(+)
> > >
> > > diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> > > index ab67972353..7e934336c2 100644
> > > --- a/hw/arm/smmuv3.c
> > > +++ b/hw/arm/smmuv3.c
> > > @@ -24,6 +24,7 @@
> > >  #include "hw/qdev-properties.h"
> > >  #include "hw/qdev-core.h"
> > >  #include "hw/pci/pci.h"
> > > +#include "hw/pci/pci_bridge.h"
> > >  #include "cpu.h"
> > >  #include "exec/target_page.h"
> > >  #include "trace.h"
> > > @@ -1881,6 +1882,13 @@ static void smmu_realize(DeviceState *d, Error  
> > **errp)  
> > >      SMMUv3Class *c = ARM_SMMUV3_GET_CLASS(s);
> > >      SysBusDevice *dev = SYS_BUS_DEVICE(d);
> > >      Error *local_err = NULL;
> > > +    Object *bus;
> > > +
> > > +    bus = object_property_get_link(OBJECT(d), "primary-bus",  
> > &error_abort);
> > I'd replace this with direct field access like in smmu_base_realize  
> 
> Ok.
>  
> > in QEMU with PCI, usually we specify bus to attach to with 'bus' property,
> > wouldn't it better to rename "primary-bus" to 'bus' to be consistent with
> > the rest of PCI code (and before "primary-bus" shows up as a CLI option,
> > so far (before this series) it looks like it's an internal property)?  
> 
> That was tried in v2 and since SMMUv3 is not a pci device by itself(it is a 
> sysbus device) reusing the default "bus" property to establish an association
> with a PCI bus created problems,
> https://lore.kernel.org/qemu-devel/877c2ut0zk.fsf@pond.sub.org/

that was an approach was trying to workaround by patching dc->bus_type,
which is obviously wrong.

I'm not talking about changing device type or something similar,
but about renaming 'primary-bus' property name to 'bus'
so it would be consistent interface wise with PCI or other QEMU devices
that are attached to a bus.

> > > +    if (!bus || !object_dynamic_cast(bus->parent,  
> > TYPE_PCI_HOST_BRIDGE)) {
> > Also looking at smmu_base_realize, it has NULL pointer check already.
> > Which also rises question, shouldn't smmu_base_realize check for
> > TYPE_PCI_HOST_BRIDGE as well (aka can smmu be attached to anything
> > else but a host bridge)?  
> 
> Not at the moment in Qemu. Though the SMMUv3 specification allows it to
> be associated with non-pci devices as well.

then perhaps move, the check to smmu_base_realize() for now?

if smmu + non-pci ever materialize, it can be refactored at that time. 
 
> Thanks,
> Shameer
> 

RE: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association

[Shameerali Kolothum Thodi via]

> -----Original Message-----
> From: Igor Mammedov <imammedo@redhat.com>
> Sent: Thursday, June 5, 2025 1:20 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Shameer Kolothum via <qemu-devel@nongnu.org>; qemu-
> arm@nongnu.org; eric.auger@redhat.com; peter.maydell@linaro.org;
> jgg@nvidia.com; nicolinc@nvidia.com; ddutile@redhat.com;
> berrange@redhat.com; nathanc@nvidia.com; mochs@nvidia.com;
> smostafa@google.com; Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>;
> zhangfei.gao@linaro.org
> Subject: Re: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root
> Complex association

[..]

> > > in QEMU with PCI, usually we specify bus to attach to with 'bus'
> property,
> > > wouldn't it better to rename "primary-bus" to 'bus' to be consistent
> with
> > > the rest of PCI code (and before "primary-bus" shows up as a CLI option,
> > > so far (before this series) it looks like it's an internal property)?
> >
> > That was tried in v2 and since SMMUv3 is not a pci device by itself(it is a
> > sysbus device) reusing the default "bus" property to establish an
> association
> > with a PCI bus created problems,
> > https://lore.kernel.org/qemu-devel/877c2ut0zk.fsf@pond.sub.org/
> 
> that was an approach was trying to workaround by patching dc->bus_type,
> which is obviously wrong.
> 
> I'm not talking about changing device type or something similar,
> but about renaming 'primary-bus' property name to 'bus'

I have tried that earlier and gets this,
-device arm-smmuv3,bus=pcie.0,id=smmuv3.0: Device 'arm-smmuv3' can't go on PCIE bus

IIRC, the above mentioned patched dc->bus_type = TYPE_PCIE_BUS was used 
to avoid that.

Or am I missing something here?

> so it would be consistent interface wise with PCI or other QEMU devices
> that are attached to a bus.
> 
> > > > +    if (!bus || !object_dynamic_cast(bus->parent,
> > > TYPE_PCI_HOST_BRIDGE)) {
> > > Also looking at smmu_base_realize, it has NULL pointer check already.
> > > Which also rises question, shouldn't smmu_base_realize check for
> > > TYPE_PCI_HOST_BRIDGE as well (aka can smmu be attached to anything
> > > else but a host bridge)?
> >
> > Not at the moment in Qemu. Though the SMMUv3 specification allows it
> to
> > be associated with non-pci devices as well.
> 
> then perhaps move, the check to smmu_base_realize() for now?
> 
> if smmu + non-pci ever materialize, it can be refactored at that time.

Ok.

Thanks,
Shameer

Re: [PATCH v3 6/6] hw/arm/virt: Allow user-creatable SMMUv3 dev instantiation

[Eric Auger]

On 6/2/25 5:41 PM, Shameer Kolothum wrote:
> Allow cold-plug of smmuv3 device to virt if there is no machine
> wide legacy smmuv3 or a virtio-iommu is specified.
>
> Device tree support for new smmuv3 dev is limited to the case where
> it is associated with the default pcie.0 RC.
>
> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
> ---
>  hw/arm/smmuv3.c      |  2 ++
>  hw/arm/virt.c        | 50 ++++++++++++++++++++++++++++++++++++++++++++
>  hw/core/sysbus-fdt.c |  3 +++
>  3 files changed, 55 insertions(+)
>
> diff --git a/hw/arm/smmuv3.c b/hw/arm/smmuv3.c
> index 7e934336c2..d1f66dcd8f 100644
> --- a/hw/arm/smmuv3.c
> +++ b/hw/arm/smmuv3.c
> @@ -2004,6 +2004,8 @@ static void smmuv3_class_init(ObjectClass *klass, const void *data)
>      device_class_set_parent_realize(dc, smmu_realize,
>                                      &c->parent_realize);
>      device_class_set_props(dc, smmuv3_properties);
> +    dc->hotpluggable = false;
> +    dc->user_creatable = true;
>  }
>  
>  static int smmuv3_notify_flag_changed(IOMMUMemoryRegion *iommu,
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index eeace4754d..04bc940fc3 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -56,6 +56,7 @@
>  #include "qemu/cutils.h"
>  #include "qemu/error-report.h"
>  #include "qemu/module.h"
> +#include "hw/pci/pci_bus.h"
>  #include "hw/pci-host/gpex.h"
>  #include "hw/virtio/virtio-pci.h"
>  #include "hw/core/sysbus-fdt.h"
> @@ -1443,6 +1444,28 @@ static void create_smmuv3_dt_bindings(const VirtMachineState *vms, hwaddr base,
>      g_free(node);
>  }
>  
> +static void create_smmuv3_dev_dtb(VirtMachineState *vms,
> +                                  DeviceState *dev, PCIBus *bus)
> +{
> +    PlatformBusDevice *pbus = PLATFORM_BUS_DEVICE(vms->platform_bus_dev);
> +    SysBusDevice *sbdev = SYS_BUS_DEVICE(dev);
> +    int irq = platform_bus_get_irqn(pbus, sbdev, 0);
> +    hwaddr base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
> +    MachineState *ms = MACHINE(vms);
> +
> +    if (strcmp("pcie.0", bus->qbus.name)) {
> +        warn_report("SMMUv3 device only supported with pcie.0 for DT");
> +        return;
> +    }
> +    base += vms->memmap[VIRT_PLATFORM_BUS].base;
> +    irq += vms->irqmap[VIRT_PLATFORM_BUS];
> +
> +    vms->iommu_phandle = qemu_fdt_alloc_phandle(ms->fdt);
> +    create_smmuv3_dt_bindings(vms, base, SMMU_IO_LEN, irq);
> +    qemu_fdt_setprop_cells(ms->fdt, vms->pciehb_nodename, "iommu-map",
> +                           0x0, vms->iommu_phandle, 0x0, 0x10000);
> +}
wondering whether you couldn't have put that code in hw/core/sysbus-fdt.c
just like add_calxeda_midway_xgmac_fdt_node(). vms can be retrieved from
the object hierarchy I guess.

Actually the original infra was made in that spirit. Now I know that
Peter is not a big fan of all those VFIO platform related stuff ;-) so I
have no strong opinion either.
> +
>  static void create_smmu(const VirtMachineState *vms,
>                          PCIBus *bus)
>  {
> @@ -2931,6 +2954,13 @@ static void virt_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev,
>          qlist_append_str(reserved_regions, resv_prop_str);
>          qdev_prop_set_array(dev, "reserved-regions", reserved_regions);
>          g_free(resv_prop_str);
> +    } else if (object_dynamic_cast(OBJECT(dev), TYPE_ARM_SMMUV3)) {
> +        if (vms->legacy_smmuv3_present || vms->iommu == VIRT_IOMMU_VIRTIO) {
> +            error_setg(errp, "virt machine already has %s set. "
> +                       "Doesn't support incompatible iommus",
> +                       (vms->legacy_smmuv3_present) ?
> +                       "iommu=smmuv3" : "virtio-iommu");
> +        }
>      }
>  }
>  
> @@ -2954,6 +2984,25 @@ static void virt_machine_device_plug_cb(HotplugHandler *hotplug_dev,
>          virtio_md_pci_plug(VIRTIO_MD_PCI(dev), MACHINE(hotplug_dev), errp);
>      }
>  
> +    if (object_dynamic_cast(OBJECT(dev), TYPE_ARM_SMMUV3)) {
> +        if (!vms->legacy_smmuv3_present && vms->platform_bus_dev) {
> +            PCIBus *bus;
> +
> +            bus = PCI_BUS(object_property_get_link(OBJECT(dev), "primary-bus",
> +                                                   &error_abort));
> +            if (pci_bus_bypass_iommu(bus)) {
> +                error_setg(errp, "Bypass option cannot be set for SMMUv3 "
> +                           "associated PCIe RC");
> +                return;
> +            }
> +
> +            create_smmuv3_dev_dtb(vms, dev, bus);
> +            if (vms->iommu != VIRT_IOMMU_SMMUV3) {
is it worth to check given the check done in
virt_machine_device_pre_plug_cb()?
        if (vms->legacy_smmuv3_present || vms->iommu == VIRT_IOMMU_VIRTIO) {

Cheers

Eric
> +                vms->iommu = VIRT_IOMMU_SMMUV3;
> +            }
> +        }
> +    }
> +
>      if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_IOMMU_PCI)) {
>          PCIDevice *pdev = PCI_DEVICE(dev);
>  
> @@ -3156,6 +3205,7 @@ static void virt_machine_class_init(ObjectClass *oc, const void *data)
>      machine_class_allow_dynamic_sysbus_dev(mc, TYPE_RAMFB_DEVICE);
>      machine_class_allow_dynamic_sysbus_dev(mc, TYPE_VFIO_PLATFORM);
>      machine_class_allow_dynamic_sysbus_dev(mc, TYPE_UEFI_VARS_SYSBUS);
> +    machine_class_allow_dynamic_sysbus_dev(mc, TYPE_ARM_SMMUV3);
>  #ifdef CONFIG_TPM
>      machine_class_allow_dynamic_sysbus_dev(mc, TYPE_TPM_TIS_SYSBUS);
>  #endif
> diff --git a/hw/core/sysbus-fdt.c b/hw/core/sysbus-fdt.c
> index c339a27875..e80776080b 100644
> --- a/hw/core/sysbus-fdt.c
> +++ b/hw/core/sysbus-fdt.c
> @@ -31,6 +31,7 @@
>  #include "qemu/error-report.h"
>  #include "system/device_tree.h"
>  #include "system/tpm.h"
> +#include "hw/arm/smmuv3.h"
>  #include "hw/platform-bus.h"
>  #include "hw/vfio/vfio-platform.h"
>  #include "hw/vfio/vfio-calxeda-xgmac.h"
> @@ -518,6 +519,8 @@ static const BindingEntry bindings[] = {
>  #ifdef CONFIG_TPM
>      TYPE_BINDING(TYPE_TPM_TIS_SYSBUS, add_tpm_tis_fdt_node),
>  #endif
> +    /* No generic DT support for smmuv3 dev. Support added for arm virt only */
> +    TYPE_BINDING(TYPE_ARM_SMMUV3, no_fdt_node),
>      TYPE_BINDING(TYPE_RAMFB_DEVICE, no_fdt_node),
>      TYPE_BINDING(TYPE_UEFI_VARS_SYSBUS, add_uefi_vars_node),
>      TYPE_BINDING("", NULL), /* last element */

Re: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root Complex association

[Igor Mammedov]

On Thu, 5 Jun 2025 12:36:52 +0000
Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote:

> > -----Original Message-----
> > From: Igor Mammedov <imammedo@redhat.com>
> > Sent: Thursday, June 5, 2025 1:20 PM
> > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> > Cc: Shameer Kolothum via <qemu-devel@nongnu.org>; qemu-
> > arm@nongnu.org; eric.auger@redhat.com; peter.maydell@linaro.org;
> > jgg@nvidia.com; nicolinc@nvidia.com; ddutile@redhat.com;
> > berrange@redhat.com; nathanc@nvidia.com; mochs@nvidia.com;
> > smostafa@google.com; Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
> > <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
> > Jonathan Cameron <jonathan.cameron@huawei.com>;
> > zhangfei.gao@linaro.org
> > Subject: Re: [PATCH v3 1/6] hw/arm/smmuv3: Check SMMUv3 has PCIe Root
> > Complex association  
> 
> [..]
> 
> > > > in QEMU with PCI, usually we specify bus to attach to with 'bus'  
> > property,  
> > > > wouldn't it better to rename "primary-bus" to 'bus' to be consistent  
> > with  
> > > > the rest of PCI code (and before "primary-bus" shows up as a CLI option,
> > > > so far (before this series) it looks like it's an internal property)?  
> > >
> > > That was tried in v2 and since SMMUv3 is not a pci device by itself(it is a
> > > sysbus device) reusing the default "bus" property to establish an  
> > association  
> > > with a PCI bus created problems,
> > > https://lore.kernel.org/qemu-devel/877c2ut0zk.fsf@pond.sub.org/  
> > 
> > that was an approach was trying to workaround by patching dc->bus_type,
> > which is obviously wrong.
> > 
> > I'm not talking about changing device type or something similar,
> > but about renaming 'primary-bus' property name to 'bus'  
> 
> I have tried that earlier and gets this,
> -device arm-smmuv3,bus=pcie.0,id=smmuv3.0: Device 'arm-smmuv3' can't go on PCIE bus
> 
> IIRC, the above mentioned patched dc->bus_type = TYPE_PCIE_BUS was used 
> to avoid that.
> 
> Or am I missing something here?

sigh, but you a right.
qdev_device_add_from_qdict() && co can't handle that without heavy refactoring.

 
> > so it would be consistent interface wise with PCI or other QEMU devices
> > that are attached to a bus.
> >   
> > > > > +    if (!bus || !object_dynamic_cast(bus->parent,  
> > > > TYPE_PCI_HOST_BRIDGE)) {
> > > > Also looking at smmu_base_realize, it has NULL pointer check already.
> > > > Which also rises question, shouldn't smmu_base_realize check for
> > > > TYPE_PCI_HOST_BRIDGE as well (aka can smmu be attached to anything
> > > > else but a host bridge)?  
> > >
> > > Not at the moment in Qemu. Though the SMMUv3 specification allows it  
> > to  
> > > be associated with non-pci devices as well.  
> > 
> > then perhaps move, the check to smmu_base_realize() for now?
> > 
> > if smmu + non-pci ever materialize, it can be refactored at that time.  
> 
> Ok.
> 
> Thanks,
> Shameer
> 

Re: [PATCH v3 0/6] hw/arm/virt: Add support for user creatable SMMUv3 device

[Nathan Chen]

On 6/4/2025 7:34 PM, Donald Dutile wrote:
> On 6/4/25 10:02 PM, Nathan Chen wrote:
>> On 6/2/2025 8:41 AM, Shameer Kolothum wrote:
>>> This patch series introduces support for a user-creatable SMMUv3 device
>>> (-device arm-smmuv3) in QEMU.
>>
>> Tested-by: Nathan Chen <nathanc@nvidia.com>
>>
>> I am able to create 16 SMMUv3 devices in a qemu VM with emulated 
>> devices properly associated with the guest SMMUs in guest sysfs - 
>> verified with some guest SMMUs having two or three emulated NICs 
>> assigned to them while other guest SMMUs have a minimum of one assigned.
>>
>> Thanks,
>> Nathan
>>
> Nathan,
> Great test!
> Can you share the xml &/or qemu cmdline, to demonstrate this broad 
> capability?
> Also would be a good reference when (we know it isn't 'if') a new/ 
> different config
> that wasn't tested shows a bug, and we'll have more info to work with.

Yes, here is the qemu command line I used. smmuv3.15 is associated with 
two devices, smmuv3.0 is associated with two devices, and smmuv3.1 is 
associated with three devices:

qemu-system-aarch64 \
         -machine hmat=on -machine virt,accel=kvm,gic-version=3,ras=on \
         -cpu host -smp cpus=4 -m size=16G,slots=4,maxmem=32G -nographic \
         -bios /usr/share/AAVMF/AAVMF_CODE.fd \
         -device nvme,drive=nvme0,serial=deadbeaf1,bus=pcie.0 \
         -drive 
file=/home/nvidia/nathanc/noble-server-cloudimg-arm64.qcow2,index=0,media=disk,format=qcow2,if=none,id=nvme0 
\
         -device 
e1000,romfile=/root/nathanc/efi-e1000.rom,netdev=net0,bus=pcie.0 \
         -netdev 
user,id=net0,hostfwd=tcp::5558-:22,hostfwd=tcp::5586-:5586 \
         -netdev user,id=net1 \
         -netdev user,id=net2 \
         -netdev user,id=net3 \
         -netdev user,id=net4 \
         -netdev user,id=net5 \
         -netdev user,id=net6 \
         -netdev user,id=net7 \
         -netdev user,id=net8 \
         -netdev user,id=net9 \
         -netdev user,id=net10 \
         -netdev user,id=net11 \
         -netdev user,id=net12 \
         -netdev user,id=net13 \
         -netdev user,id=net14 \
         -netdev user,id=net15 \
         -netdev user,id=net16 \
         -netdev user,id=net17 \
         -netdev user,id=net18 \
         -netdev user,id=net19 \
         -netdev user,id=net20 \
         -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0 \
         -device virtio-net-pci,bus=pcie.0,netdev=net1 \
         -device virtio-net-pci,bus=pcie.0,netdev=net20 \
         -device pxb-pcie,id=pcie.1,bus_nr=200,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1 \
         -device pcie-root-port,id=pcie.port1,bus=pcie.1,slot=1,chassis=1 \
         -device virtio-net-pci,bus=pcie.port1,netdev=net2 \
         -device 
pcie-root-port,id=pcie.port17,bus=pcie.1,slot=17,chassis=17 \
         -device virtio-net-pci,bus=pcie.port17,netdev=net18 \
         -device 
pcie-root-port,id=pcie.port18,bus=pcie.1,slot=18,chassis=18 \
         -device virtio-net-pci,bus=pcie.port18,netdev=net19 \
         -device pxb-pcie,id=pcie.2,bus_nr=196,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.2,id=smmuv3.2 \
         -device pcie-root-port,id=pcie.port2,bus=pcie.2,slot=2,chassis=2 \
         -device virtio-net-pci,bus=pcie.port2,netdev=net3 \
         -device pxb-pcie,id=pcie.3,bus_nr=192,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.3,id=smmuv3.3 \
         -device pcie-root-port,id=pcie.port3,bus=pcie.3,slot=3,chassis=3 \
         -device virtio-net-pci,bus=pcie.port3,netdev=net4 \
         -device pxb-pcie,id=pcie.4,bus_nr=188,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.4,id=smmuv3.4 \
         -device pcie-root-port,id=pcie.port4,bus=pcie.4,slot=4,chassis=4 \
         -device virtio-net-pci,bus=pcie.port4,netdev=net5 \
         -device pxb-pcie,id=pcie.5,bus_nr=184,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.5,id=smmuv3.5 \
         -device pcie-root-port,id=pcie.port5,bus=pcie.5,slot=5,chassis=5 \
         -device virtio-net-pci,bus=pcie.port5,netdev=net6 \
         -device pxb-pcie,id=pcie.6,bus_nr=180,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.6,id=smmuv3.6 \
         -device pcie-root-port,id=pcie.port6,bus=pcie.6,slot=6,chassis=6 \
         -device virtio-net-pci,bus=pcie.port6,netdev=net7 \
         -device pxb-pcie,id=pcie.7,bus_nr=176,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.7,id=smmuv3.7 \
         -device pcie-root-port,id=pcie.port7,bus=pcie.7,slot=7,chassis=7 \
         -device virtio-net-pci,bus=pcie.port7,netdev=net8 \
         -device pxb-pcie,id=pcie.8,bus_nr=172,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.8,id=smmuv3.8 \
         -device pcie-root-port,id=pcie.port8,bus=pcie.8,slot=8,chassis=8 \
         -device virtio-net-pci,bus=pcie.port8,netdev=net9 \
         -device pxb-pcie,id=pcie.9,bus_nr=168,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.9,id=smmuv3.9 \
         -device pcie-root-port,id=pcie.port9,bus=pcie.9,slot=9,chassis=9 \
         -device virtio-net-pci,bus=pcie.port9,netdev=net10 \
         -device pxb-pcie,id=pcie.10,bus_nr=164,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.10,id=smmuv3.10 \
         -device 
pcie-root-port,id=pcie.port10,bus=pcie.10,slot=10,chassis=10 \
         -device virtio-net-pci,bus=pcie.port10,netdev=net11 \
         -device pxb-pcie,id=pcie.11,bus_nr=160,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.11,id=smmuv3.11 \
         -device 
pcie-root-port,id=pcie.port11,bus=pcie.11,slot=11,chassis=11 \
         -device virtio-net-pci,bus=pcie.port11,netdev=net12 \
         -device pxb-pcie,id=pcie.12,bus_nr=156,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.12,id=smmuv3.12 \
         -device 
pcie-root-port,id=pcie.port12,bus=pcie.12,slot=12,chassis=12 \
         -device virtio-net-pci,bus=pcie.port12,netdev=net13 \
         -device pxb-pcie,id=pcie.13,bus_nr=152,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.13,id=smmuv3.13 \
         -device 
pcie-root-port,id=pcie.port13,bus=pcie.13,slot=13,chassis=13 \
         -device virtio-net-pci,bus=pcie.port13,netdev=net14 \
         -device pxb-pcie,id=pcie.14,bus_nr=148,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.14,id=smmuv3.14 \
         -device 
pcie-root-port,id=pcie.port14,bus=pcie.14,slot=14,chassis=14 \
         -device virtio-net-pci,bus=pcie.port14,netdev=net15 \
         -device pxb-pcie,id=pcie.15,bus_nr=144,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.15,id=smmuv3.15 \
         -device 
pcie-root-port,id=pcie.port15,bus=pcie.15,slot=15,chassis=15 \
         -device virtio-net-pci,bus=pcie.port15,netdev=net16 \
         -device 
pcie-root-port,id=pcie.port16,bus=pcie.15,slot=16,chassis=16 \
         -device virtio-net-pci,bus=pcie.port16,netdev=net17

Here is the guest topology:

$ lspci -tv
-+-[0000:00]-+-00.0  Red Hat, Inc. QEMU PCIe Host bridge
  |           +-01.0  Red Hat, Inc. QEMU NVM Express Controller
  |           +-02.0  Intel Corporation 82540EM Gigabit Ethernet Controller
  |           +-03.0  Red Hat, Inc. Virtio network device
  |           +-04.0  Red Hat, Inc. Virtio network device
  |           +-05.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-06.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-07.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-08.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-09.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0a.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0b.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0c.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0d.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0e.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0f.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-10.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-11.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-12.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           \-13.0  Red Hat, Inc. QEMU PCIe Expander bridge
  +-[0000:90]-+-00.0-[91]----00.0  Red Hat, Inc. Virtio 1.0 network device
  |           \-01.0-[92]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:94]---00.0-[95]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:98]---00.0-[99]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:9c]---00.0-[9d]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:a0]---00.0-[a1]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:a4]---00.0-[a5]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:a8]---00.0-[a9]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:ac]---00.0-[ad]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:b0]---00.0-[b1]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:b4]---00.0-[b5]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:b8]---00.0-[b9]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:bc]---00.0-[bd]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:c0]---00.0-[c1]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:c4]---00.0-[c5]----00.0  Red Hat, Inc. Virtio 1.0 network device
  \-[0000:c8]-+-00.0-[c9]----00.0  Red Hat, Inc. Virtio 1.0 network device
              +-01.0-[ca]----00.0  Red Hat, Inc. Virtio 1.0 network device
              \-02.0-[cb]----00.0  Red Hat, Inc. Virtio 1.0 network device

Thanks,
Nathan

Re: [PATCH v3 0/6] hw/arm/virt: Add support for user creatable SMMUv3 device

[Donald Dutile]

On 6/5/25 1:58 PM, Nathan Chen wrote:
> 
> 
> On 6/4/2025 7:34 PM, Donald Dutile wrote:
>> On 6/4/25 10:02 PM, Nathan Chen wrote:
>>> On 6/2/2025 8:41 AM, Shameer Kolothum wrote:
>>>> This patch series introduces support for a user-creatable SMMUv3 device
>>>> (-device arm-smmuv3) in QEMU.
>>>
>>> Tested-by: Nathan Chen <nathanc@nvidia.com>
>>>
>>> I am able to create 16 SMMUv3 devices in a qemu VM with emulated devices properly associated with the guest SMMUs in guest sysfs - verified with some guest SMMUs having two or three emulated NICs assigned to them while other guest SMMUs have a minimum of one assigned.
>>>
>>> Thanks,
>>> Nathan
>>>
>> Nathan,
>> Great test!
>> Can you share the xml &/or qemu cmdline, to demonstrate this broad capability?
>> Also would be a good reference when (we know it isn't 'if') a new/ different config
>> that wasn't tested shows a bug, and we'll have more info to work with.
> 
> Yes, here is the qemu command line I used. smmuv3.15 is associated with two devices, smmuv3.0 is associated with two devices, and smmuv3.1 is associated with three devices:
> 
> qemu-system-aarch64 \
>          -machine hmat=on -machine virt,accel=kvm,gic-version=3,ras=on \
>          -cpu host -smp cpus=4 -m size=16G,slots=4,maxmem=32G -nographic \
>          -bios /usr/share/AAVMF/AAVMF_CODE.fd \
>          -device nvme,drive=nvme0,serial=deadbeaf1,bus=pcie.0 \
>          -drive file=/home/nvidia/nathanc/noble-server-cloudimg-arm64.qcow2,index=0,media=disk,format=qcow2,if=none,id=nvme0 \
>          -device e1000,romfile=/root/nathanc/efi-e1000.rom,netdev=net0,bus=pcie.0 \
>          -netdev user,id=net0,hostfwd=tcp::5558-:22,hostfwd=tcp::5586-:5586 \
>          -netdev user,id=net1 \
>          -netdev user,id=net2 \
>          -netdev user,id=net3 \
>          -netdev user,id=net4 \
>          -netdev user,id=net5 \
>          -netdev user,id=net6 \
>          -netdev user,id=net7 \
>          -netdev user,id=net8 \
>          -netdev user,id=net9 \
>          -netdev user,id=net10 \
>          -netdev user,id=net11 \
>          -netdev user,id=net12 \
>          -netdev user,id=net13 \
>          -netdev user,id=net14 \
>          -netdev user,id=net15 \
>          -netdev user,id=net16 \
>          -netdev user,id=net17 \
>          -netdev user,id=net18 \
>          -netdev user,id=net19 \
>          -netdev user,id=net20 \
>          -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0 \
>          -device virtio-net-pci,bus=pcie.0,netdev=net1 \
>          -device virtio-net-pci,bus=pcie.0,netdev=net20 \
>          -device pxb-pcie,id=pcie.1,bus_nr=200,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1 \
>          -device pcie-root-port,id=pcie.port1,bus=pcie.1,slot=1,chassis=1 \
>          -device virtio-net-pci,bus=pcie.port1,netdev=net2 \
>          -device pcie-root-port,id=pcie.port17,bus=pcie.1,slot=17,chassis=17 \
>          -device virtio-net-pci,bus=pcie.port17,netdev=net18 \
>          -device pcie-root-port,id=pcie.port18,bus=pcie.1,slot=18,chassis=18 \
>          -device virtio-net-pci,bus=pcie.port18,netdev=net19 \
>          -device pxb-pcie,id=pcie.2,bus_nr=196,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.2,id=smmuv3.2 \
>          -device pcie-root-port,id=pcie.port2,bus=pcie.2,slot=2,chassis=2 \
>          -device virtio-net-pci,bus=pcie.port2,netdev=net3 \
>          -device pxb-pcie,id=pcie.3,bus_nr=192,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.3,id=smmuv3.3 \
>          -device pcie-root-port,id=pcie.port3,bus=pcie.3,slot=3,chassis=3 \
>          -device virtio-net-pci,bus=pcie.port3,netdev=net4 \
>          -device pxb-pcie,id=pcie.4,bus_nr=188,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.4,id=smmuv3.4 \
>          -device pcie-root-port,id=pcie.port4,bus=pcie.4,slot=4,chassis=4 \
>          -device virtio-net-pci,bus=pcie.port4,netdev=net5 \
>          -device pxb-pcie,id=pcie.5,bus_nr=184,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.5,id=smmuv3.5 \
>          -device pcie-root-port,id=pcie.port5,bus=pcie.5,slot=5,chassis=5 \
>          -device virtio-net-pci,bus=pcie.port5,netdev=net6 \
>          -device pxb-pcie,id=pcie.6,bus_nr=180,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.6,id=smmuv3.6 \
>          -device pcie-root-port,id=pcie.port6,bus=pcie.6,slot=6,chassis=6 \
>          -device virtio-net-pci,bus=pcie.port6,netdev=net7 \
>          -device pxb-pcie,id=pcie.7,bus_nr=176,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.7,id=smmuv3.7 \
>          -device pcie-root-port,id=pcie.port7,bus=pcie.7,slot=7,chassis=7 \
>          -device virtio-net-pci,bus=pcie.port7,netdev=net8 \
>          -device pxb-pcie,id=pcie.8,bus_nr=172,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.8,id=smmuv3.8 \
>          -device pcie-root-port,id=pcie.port8,bus=pcie.8,slot=8,chassis=8 \
>          -device virtio-net-pci,bus=pcie.port8,netdev=net9 \
>          -device pxb-pcie,id=pcie.9,bus_nr=168,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.9,id=smmuv3.9 \
>          -device pcie-root-port,id=pcie.port9,bus=pcie.9,slot=9,chassis=9 \
>          -device virtio-net-pci,bus=pcie.port9,netdev=net10 \
>          -device pxb-pcie,id=pcie.10,bus_nr=164,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.10,id=smmuv3.10 \
>          -device pcie-root-port,id=pcie.port10,bus=pcie.10,slot=10,chassis=10 \
>          -device virtio-net-pci,bus=pcie.port10,netdev=net11 \
>          -device pxb-pcie,id=pcie.11,bus_nr=160,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.11,id=smmuv3.11 \
>          -device pcie-root-port,id=pcie.port11,bus=pcie.11,slot=11,chassis=11 \
>          -device virtio-net-pci,bus=pcie.port11,netdev=net12 \
>          -device pxb-pcie,id=pcie.12,bus_nr=156,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.12,id=smmuv3.12 \
>          -device pcie-root-port,id=pcie.port12,bus=pcie.12,slot=12,chassis=12 \
>          -device virtio-net-pci,bus=pcie.port12,netdev=net13 \
>          -device pxb-pcie,id=pcie.13,bus_nr=152,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.13,id=smmuv3.13 \
>          -device pcie-root-port,id=pcie.port13,bus=pcie.13,slot=13,chassis=13 \
>          -device virtio-net-pci,bus=pcie.port13,netdev=net14 \
>          -device pxb-pcie,id=pcie.14,bus_nr=148,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.14,id=smmuv3.14 \
>          -device pcie-root-port,id=pcie.port14,bus=pcie.14,slot=14,chassis=14 \
>          -device virtio-net-pci,bus=pcie.port14,netdev=net15 \
>          -device pxb-pcie,id=pcie.15,bus_nr=144,bus=pcie.0 \
>          -device arm-smmuv3,primary-bus=pcie.15,id=smmuv3.15 \
>          -device pcie-root-port,id=pcie.port15,bus=pcie.15,slot=15,chassis=15 \
>          -device virtio-net-pci,bus=pcie.port15,netdev=net16 \
>          -device pcie-root-port,id=pcie.port16,bus=pcie.15,slot=16,chassis=16 \
>          -device virtio-net-pci,bus=pcie.port16,netdev=net17
> 
> Here is the guest topology:
> 
> $ lspci -tv
> -+-[0000:00]-+-00.0  Red Hat, Inc. QEMU PCIe Host bridge
>   |           +-01.0  Red Hat, Inc. QEMU NVM Express Controller
>   |           +-02.0  Intel Corporation 82540EM Gigabit Ethernet Controller
>   |           +-03.0  Red Hat, Inc. Virtio network device
>   |           +-04.0  Red Hat, Inc. Virtio network device
>   |           +-05.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-06.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-07.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-08.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-09.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-0a.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-0b.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-0c.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-0d.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-0e.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-0f.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-10.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-11.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           +-12.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   |           \-13.0  Red Hat, Inc. QEMU PCIe Expander bridge
>   +-[0000:90]-+-00.0-[91]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   |           \-01.0-[92]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:94]---00.0-[95]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:98]---00.0-[99]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:9c]---00.0-[9d]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:a0]---00.0-[a1]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:a4]---00.0-[a5]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:a8]---00.0-[a9]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:ac]---00.0-[ad]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:b0]---00.0-[b1]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:b4]---00.0-[b5]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:b8]---00.0-[b9]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:bc]---00.0-[bd]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:c0]---00.0-[c1]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   +-[0000:c4]---00.0-[c5]----00.0  Red Hat, Inc. Virtio 1.0 network device
>   \-[0000:c8]-+-00.0-[c9]----00.0  Red Hat, Inc. Virtio 1.0 network device
>               +-01.0-[ca]----00.0  Red Hat, Inc. Virtio 1.0 network device
>               \-02.0-[cb]----00.0  Red Hat, Inc. Virtio 1.0 network device
> 
> Thanks,
> Nathan
> 
Needs a device-assigned NVIDIA GPU! ;-)
Thanks for the config!
- Don