[PATCH 0/8] amd_iommu: Allow IOMMU to attach on non system bus

[PATCH 0/8] amd_iommu: Allow IOMMU to attach on non system bus

[Sairaj Kodilkar]

Overview
--------
This patch series enables AMD IOMMU to be attached to non-system root buses
(e. g. , pcie-pxb, pci-root-bus), allowing device isolation between VFIO
passthrough devices and emulated devices.  This is a prerequisite for
hardware-accelerated virtual IOMMU (vIOMMU) support. 


Problem Statement
-----------------
The current AMD IOMMU implementation has the following limitations:

1. Only supports attachment to the default pcie.0 system bus

2. Builds IVRS (I/O Virtualization Reporting Structure) by iterating over
   all system buses, preventing device isolation

3. Cannot separate VFIO passthrough devices from emulated devices on
   different buses

This creates a fundamental issue for hardware-accelerated vIOMMU, which
requires VFIO devices to be isolated from emulated devices, as the vIOMMU
can only operate on passthrough devices. 


Solution Approach
-----------------
This series introduces the following changes:

1. Allow AMD IOMMU to be attached to arbitrary root buses using the "bus="
   parameter, while Maintaining the backward compatibility by selecting the
   pcie.0 by default if this parameter is missing.

2. Implement root bus discovery by traversing the parent hierarchy to find
   the first root bus

3. Register IOMMU operations on the discovered root bus

4. Modify IVRS generation to build IVHD (I/O Virtualization Hardware
   Definition) entries based on the specified root bus, traversing child
   buses and devices in a top-down manner

5. Introduce x86_iommu_device_get() and the get_x86_iommu() callback so the
   KVM layer can determine which IOMMU serves a given device.

   kvm_arch_fixup_msi_route() and kvm_arch_add_msi_route_post() call
   x86_iommu_get_default() for every MSI route. That returns the single
   machine IOMMU and is correct only when that IOMMU is attached to pcie.0
   and serves every device.

   Once the IOMMU is attached below another root bus, it serves only the
   devices under that bus. The KVM layer must program interrupt-remapping
   routes through that IOMMU and leave devices outside its scope untranslated.
   The PCI core can already find the bus that owns a device, but PCIIOMMUOps
   cannot return the corresponding X86IOMMUState. get_x86_iommu() provides
   that link. x86_iommu_device_get() uses it, and both AMD and Intel IOMMU
   implement the callback because this KVM path is shared.


Command line
------------
qemu-system-x86_64 \
-enable-kvm -m 10G -smp 8 \
-netdev user,id=USER0,hostfwd=tcp::3333-:22 \
-cpu EPYC-Genoa,x2apic=on,kvm-msi-ext-dest-id=on,+kvm-pv-unhalt,kvm-pv-tlb-flush,kvm-pv-ipi,kvm-pv-sched-yield  \
-machine q35,kernel_irqchip=split \
-global kvm-pit.lost_tick_policy=discard \
-nographic \
-blockdev node-name=drive0,driver=qcow2,file.driver=file,file.filename=<IMG> \
-device virtio-blk-pci,num-queues=8,drive=drive0 \
-device pxb-pcie,id=pcie.1,bus=pcie.0,bus_nr=0x50 \
-device pcie-root-port,id=rp1,bus=pcie.1,chassis=3,slot=1 \
-device AMDVI-PCI,id=iommupci1,bus=rp1 \
-device amd-iommu,intremap=on,dma-remap=on,xtsup=on,pci-id=iommupci1 \
-device pcie-root-port,id=rp2,bus=pcie.2,chassis=4,slot=1 \
-device virtio-net-pci,id=vnet1,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER0,bus=rp2 \


Example
-------
Example1, for below PCI configuration, IOMMU will serve all the devices
under pxb-pci (i.e. Device1 and Device2) since its a first root bus,
but will not serve Device3 and Device4, effectively isolating the
devices under pxb-pci bus.

pcie.0 -+- pxb-pci -+- pcie-root-port1 --- AMD IOMMU
        |           +- pcie-root-port2 --- Device1
        |           \- pcie-root-port3 --- Device2
        +- Device3
        \- Device4

Example2, for below PCI configuration, IOMMU will serve all the devices
Device[1-4], since first root bus in the parent hierarchy is pcie.0

pcie.0 -+- pcie-root-port1 --- AMD IOMMU
        +- pcie-root-port2 --- Device1
        +- pcie-root-port3 --- Device2
        +- Device3
        \- Device4

Example3, Below three PCI topologies will cause failure as AMD IOMMU can
only attach to **PCIE** hierarchy.

[A]
pcie.0 -+- pcie-pci-bridge --- AMD IOMMU
        \- Device1

[B]
pcie.0 -+- pci-bridge --- AMD IOMMU
        \- Device1

[C]
pcie.0 -+- pci-bridge --- pcie-root-port --- AMD IOMMU
        \- Device1

Sairaj Kodilkar (8):
  amd_iommu: Assign ioapic_as only when interrupt remapping is
    supported.
  x86-iommu: Introduce x86_iommu_device_get() for x86-iommu retrieval
  amd_iommu: Add support for get_x86_iommu callback
  intel_iommu: Add support for get_x86_iommu callback
  i386/kvm: Use x86_iommu_device_get() for per-device IOMMU lookup
  amd_iommu: Introduce root bus
  acpi-build: Create PCI topology using amd iommu root bus
  amd_iommu: Allow IOMMU to attachment on non-system (pcie.0) bus

 hw/i386/acpi-build.c        | 142 +++++++++++++++++++-----------------
 hw/i386/amd_iommu.c         |  75 +++++++++++++++++--
 hw/i386/amd_iommu.h         |   3 +
 hw/i386/intel_iommu.c       |   8 ++
 hw/i386/x86-iommu-stub.c    |   5 ++
 hw/i386/x86-iommu.c         |  16 ++++
 hw/intc/ioapic.c            |   3 +-
 include/hw/i386/x86-iommu.h |   6 ++
 include/hw/pci/pci.h        |  12 +++
 target/i386/kvm/kvm.c       |  15 +++-
 10 files changed, 208 insertions(+), 77 deletions(-)


base-commit: 562bae590f194fb590beb5c65da44fc35ab9f64a
-- 
2.34.1

[PATCH 1/8] amd_iommu: Assign ioapic_as only when interrupt remapping is supported.

[Sairaj Kodilkar]

When interrupt remapping is disabled (intremap=off), the IOAPIC should
use the system address space instead of the IOMMU address space.
Currently, the IOAPIC is unconditionally assigned to the IOMMU address
space, which causes IOAPIC-triggered interrupts to be incorrectly
translated by the IOMMU even when interrupt remapping is disabled.

Signed-off-by: Sairaj Kodilkar <sarunkod@amd.com>
---
 hw/i386/amd_iommu.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/hw/i386/amd_iommu.c b/hw/i386/amd_iommu.c
index 578c27ccbef6..614c7a131482 100644
--- a/hw/i386/amd_iommu.c
+++ b/hw/i386/amd_iommu.c
@@ -2733,8 +2733,16 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)
     memory_region_add_subregion_overlap(&s->mr_sys, AMDVI_INT_ADDR_FIRST,
                                         &s->mr_ir, 1);
 
-    /* Pseudo address space under root PCI bus. */
-    x86ms->ioapic_as = amdvi_host_dma_iommu(bus, s, AMDVI_IOAPIC_SB_DEVID);
+    /**
+     * Pseudo address space under root PCI bus.
+     * The linux kernel disables the intremap when it cannot find IOAPIC under
+     * AMD IOMMU IVRS, hence create a ioapic_as with root bus even though AMD
+     * IOMMU is not serving devices attached to root bus.
+     */
+    if (x86_iommu_ir_supported(X86_IOMMU_DEVICE(s))) {
+        x86ms->ioapic_as = amdvi_host_dma_iommu(pcms->pcibus, s,
+                                                AMDVI_IOAPIC_SB_DEVID);
+    }
 
     if (kvm_enabled() && x86ms->apic_id_limit > 255 && !s->xtsup) {
         error_report("AMD IOMMU with x2APIC configuration requires xtsup=on");

base-commit: 562bae590f194fb590beb5c65da44fc35ab9f64a
-- 
2.34.1

[PATCH 2/8] x86-iommu: Introduce x86_iommu_device_get() for x86-iommu retrieval

[Sairaj Kodilkar]

kvm_arch_fixup_msi_route() and kvm_arch_add_msi_route_post() use
x86_iommu_get_default() whenever an IOMMU is present. This selects the
correct unit only when a single IOMMU is attached to the system bus.

The following patches allow an IOMMU to attach to a different bus and
isolate the subset of devices below it. Each IOMMU then serves only part
of the PCI topology, so interrupt remapping must determine the IOMMU that
actually serves a given device.

Add x86_iommu_device_get() to locate the IOMMU bus for a PCI device and
retrieve its X86IOMMUState through a new get_x86_iommu callback.

Signed-off-by: Sairaj Kodilkar <sarunkod@amd.com>
---
 hw/i386/x86-iommu-stub.c    |  5 +++++
 hw/i386/x86-iommu.c         | 16 ++++++++++++++++
 include/hw/i386/x86-iommu.h |  6 ++++++
 include/hw/pci/pci.h        | 12 ++++++++++++
 4 files changed, 39 insertions(+)

diff --git a/hw/i386/x86-iommu-stub.c b/hw/i386/x86-iommu-stub.c
index 781b5ff92227..b0383d2290b1 100644
--- a/hw/i386/x86-iommu-stub.c
+++ b/hw/i386/x86-iommu-stub.c
@@ -32,6 +32,11 @@ X86IOMMUState *x86_iommu_get_default(void)
     return NULL;
 }
 
+X86IOMMUState *x86_iommu_device_get(PCIDevice *dev)
+{
+    return NULL;
+}
+
 bool x86_iommu_ir_supported(X86IOMMUState *s)
 {
     return false;
diff --git a/hw/i386/x86-iommu.c b/hw/i386/x86-iommu.c
index a57445404ef8..601db5cfea09 100644
--- a/hw/i386/x86-iommu.c
+++ b/hw/i386/x86-iommu.c
@@ -26,6 +26,7 @@
 #include "qemu/error-report.h"
 #include "trace.h"
 #include "system/kvm.h"
+#include "hw/pci/pci_bus.h"
 
 void x86_iommu_iec_register_notifier(X86IOMMUState *iommu,
                                      iec_notify_fn fn, void *data)
@@ -90,6 +91,21 @@ X86IOMMUState *x86_iommu_get_default(void)
     return NULL;
 }
 
+X86IOMMUState *x86_iommu_device_get(PCIDevice *dev)
+{
+    PCIBus *iommu_bus;
+    X86IOMMUState *iommu = NULL;
+
+    pci_device_get_iommu_bus_devfn(dev, &iommu_bus, NULL, NULL);
+
+    if (iommu_bus && iommu_bus->iommu_ops->get_x86_iommu) {
+        iommu_bus->iommu_ops->get_x86_iommu(iommu_bus->iommu_opaque,
+                                            (void **)&iommu);
+    }
+
+    return iommu;
+}
+
 static void x86_iommu_realize(DeviceState *dev, Error **errp)
 {
     X86IOMMUState *x86_iommu = X86_IOMMU_DEVICE(dev);
diff --git a/include/hw/i386/x86-iommu.h b/include/hw/i386/x86-iommu.h
index 0a1e1ba72571..eb575271e9d7 100644
--- a/include/hw/i386/x86-iommu.h
+++ b/include/hw/i386/x86-iommu.h
@@ -134,6 +134,12 @@ struct X86IOMMU_MSIMessage {
  */
 X86IOMMUState *x86_iommu_get_default(void);
 
+/**
+ * x86_iommu_device_get - get IOMMU device for given PCI device
+ * @return: pointer to default IOMMU device
+ */
+X86IOMMUState *x86_iommu_device_get(PCIDevice *dev);
+
 /**
  * x86_iommu_iec_register_notifier - register IEC (Interrupt Entry
  *                                   Cache) notifiers
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index f2448e941a0b..f3c4c3a4db56 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -698,6 +698,18 @@ typedef struct PCIIOMMUOps {
      * Returns: the guest physical address of the MSI doorbell.
      */
     uint64_t (*get_msi_direct_gpa)(PCIBus *bus, void *opaque, int devfn);
+
+    /**
+     * @get_x86_iommu: get parent x86-iommu for given hardware (AMD/Intel)
+     *
+     * Callback is required if IOMMU defines x86->int_remap()
+     *
+     * @opaque: the data passed to pci_setup_iommu()
+     *
+     * @x86_iommu: A pointer to store the x86-iommu object
+     */
+    int (*get_x86_iommu)(void *opaque, void **x86_iommu);
+
 } PCIIOMMUOps;
 
 bool pci_device_get_iommu_bus_devfn(PCIDevice *dev, PCIBus **piommu_bus,
-- 
2.34.1

[PATCH 3/8] amd_iommu: Add support for get_x86_iommu callback

[Sairaj Kodilkar]

Return X86IOMMUState during get_x86_iommu callback.

Signed-off-by: Sairaj Kodilkar <sarunkod@amd.com>
---
 hw/i386/amd_iommu.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/hw/i386/amd_iommu.c b/hw/i386/amd_iommu.c
index 614c7a131482..5c1fb0879c0f 100644
--- a/hw/i386/amd_iommu.c
+++ b/hw/i386/amd_iommu.c
@@ -2472,8 +2472,16 @@ static AddressSpace *amdvi_host_dma_iommu(PCIBus *bus, void *opaque, int devfn)
     return &amdvi_dev_as->as;
 }
 
+static int amdvi_get_x86_iommu(void *opaque, void **x86_iommu)
+{
+    AMDVIState *s = opaque;
+    *x86_iommu = X86_IOMMU_DEVICE(s);
+    return 0;
+}
+
 static const PCIIOMMUOps amdvi_iommu_ops = {
     .get_address_space = amdvi_host_dma_iommu,
+    .get_x86_iommu = amdvi_get_x86_iommu,
 };
 
 static const MemoryRegionOps mmio_mem_ops = {
-- 
2.34.1

[PATCH 4/8] intel_iommu: Add support for get_x86_iommu callback

[Sairaj Kodilkar]

Return X86IOMMUState during get_x86_iommu callback.

Signed-off-by: Sairaj Kodilkar <sarunkod@amd.com>
---
 hw/i386/intel_iommu.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c
index 82c3c3b2c35f..92a2d6eebb61 100644
--- a/hw/i386/intel_iommu.c
+++ b/hw/i386/intel_iommu.c
@@ -5461,6 +5461,13 @@ static int vtd_pri_request_page(PCIBus *bus, void *opaque, int devfn,
     return 0;
 }
 
+static int vtd_get_x86_iommu(void *opaque, void **x86_iommu)
+{
+    IntelIOMMUState *s = opaque;
+    *x86_iommu = X86_IOMMU_DEVICE(s);
+    return 0;
+}
+
 static void vtd_init_iotlb_notifier(PCIBus *bus, void *opaque, int devfn,
                                     IOMMUNotifier *n, IOMMUNotify fn,
                                     void *user_opaque)
@@ -5535,6 +5542,7 @@ static PCIIOMMUOps vtd_iommu_ops = {
     .pri_unregister_notifier = vtd_pri_unregister_notifier,
     .pri_request_page = vtd_pri_request_page,
     .get_viommu_flags = vtd_get_viommu_flags,
+    .get_x86_iommu = vtd_get_x86_iommu,
 };
 
 static bool vtd_decide_config(IntelIOMMUState *s, Error **errp)
-- 
2.34.1

[PATCH 5/8] i386/kvm: Use x86_iommu_device_get() for per-device IOMMU lookup

[Sairaj Kodilkar]

Currently, kvm_arch_fixup_msi_route() unconditionally uses the default
system IOMMU for interrupt remapping when IOMMU is present. This
approach fails when an IOMMU is attached to a non-system bus and serves
only a subset of devices, because the kernel does not program IRT
(Interrupt Remapping Table) entries for devices outside the IOMMU's
scope.

This patch changes the behavior to perform per-device IOMMU lookup:
- For PCI devices: Use pci_device_get_x86_iommu() to find the specific
  IOMMU serving that device, if any
- For non-PCI devices (IOAPIC): Fall back to the default IOMMU
  only if interrupt remapping is enabled (ioapic_as != address_space_memory)

This ensures MSI route translation occurs only for devices actually
managed by an IOMMU, enabling proper interrupt remapping in configurations
where IOMMUs are attached to arbitrary buses.

NOTE: This patch assumes that only non-PCI device calling is IOAPIC and
checks ioapic_as to determine interrupt remapping.

Signed-off-by: Sairaj Kodilkar <sarunkod@amd.com>
---
 hw/intc/ioapic.c      |  3 ++-
 target/i386/kvm/kvm.c | 15 ++++++++++++---
 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/hw/intc/ioapic.c b/hw/intc/ioapic.c
index 1b3f1e824ac4..f1b881ee6118 100644
--- a/hw/intc/ioapic.c
+++ b/hw/intc/ioapic.c
@@ -438,8 +438,9 @@ static void ioapic_machine_done_notify(Notifier *notifier, void *data)
 #ifdef ACCEL_GSI_IRQFD_POSSIBLE
     IOAPICCommonState *s = container_of(notifier, IOAPICCommonState,
                                         machine_done);
+    X86MachineState *x86ms = X86_MACHINE(qdev_get_machine());
 
-    if (accel_irqchip_is_split()) {
+    if (accel_irqchip_is_split() && x86ms->ioapic_as != &address_space_memory) {
         X86IOMMUState *iommu = x86_iommu_get_default();
         if (iommu) {
             /* Register this IOAPIC with IOMMU IEC notifier, so that
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 644c45fb0a02..ffe977ceb418 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -6746,7 +6746,15 @@ uint64_t kvm_swizzle_msi_ext_dest_id(uint64_t address)
 int kvm_arch_fixup_msi_route(struct kvm_irq_routing_entry *route,
                              uint64_t address, uint32_t data, PCIDevice *dev)
 {
-    X86IOMMUState *iommu = x86_iommu_get_default();
+    X86IOMMUState *iommu = NULL;
+    MachineState *ms = MACHINE(qdev_get_machine());
+    X86MachineState *x86ms = X86_MACHINE(ms);
+
+    if (dev) {
+        iommu = x86_iommu_device_get(dev);
+    } else if (x86ms->ioapic_as != &address_space_memory) {
+        iommu = x86_iommu_get_default();
+    }
 
     if (iommu) {
         X86IOMMUClass *class = X86_IOMMU_DEVICE_GET_CLASS(iommu);
@@ -6848,6 +6856,7 @@ int kvm_arch_add_msi_route_post(struct kvm_irq_routing_entry *route,
 {
     static bool notify_list_inited = false;
     MSIRouteEntry *entry;
+    X86IOMMUState *iommu = NULL;
 
     if (!dev) {
         /* These are (possibly) IOAPIC routes only used for split
@@ -6867,13 +6876,13 @@ int kvm_arch_add_msi_route_post(struct kvm_irq_routing_entry *route,
     if (!notify_list_inited) {
         /* For the first time we do add route, add ourselves into
          * IOMMU's IEC notify list if needed. */
-        X86IOMMUState *iommu = x86_iommu_get_default();
+        iommu = x86_iommu_device_get(dev);
         if (iommu) {
             x86_iommu_iec_register_notifier(iommu,
                                             kvm_update_msi_routes_all,
                                             NULL);
+            notify_list_inited = true;
         }
-        notify_list_inited = true;
     }
     return 0;
 }
-- 
2.34.1

[PATCH 6/8] amd_iommu: Introduce root bus

[Sairaj Kodilkar]

Following patch will create IVRS entries starting from the root bus

Signed-off-by: Sairaj Kodilkar <sarunkod@amd.com>
---
 hw/i386/amd_iommu.c | 2 ++
 hw/i386/amd_iommu.h | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/hw/i386/amd_iommu.c b/hw/i386/amd_iommu.c
index 5c1fb0879c0f..5e4d5d1f1fcd 100644
--- a/hw/i386/amd_iommu.c
+++ b/hw/i386/amd_iommu.c
@@ -2713,6 +2713,8 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)
         }
     }
 
+    s->root_bus = bus;
+
     s->iotlb = g_hash_table_new_full(amdvi_iotlb_hash,
                                      amdvi_iotlb_equal, g_free, g_free);
 
diff --git a/hw/i386/amd_iommu.h b/hw/i386/amd_iommu.h
index 687691ec1cce..04a5bbd33740 100644
--- a/hw/i386/amd_iommu.h
+++ b/hw/i386/amd_iommu.h
@@ -373,6 +373,9 @@ struct AMDVIState {
 
     /* DMA address translation */
     bool dma_remap;
+
+    /* root bus to register IOMMU ops */
+    PCIBus *root_bus;
 };
 
 uint64_t amdvi_extended_feature_register(AMDVIState *s);
-- 
2.34.1

[PATCH 7/8] acpi-build: Create PCI topology using amd iommu root bus

[Sairaj Kodilkar]

Current AMD IOMMU ACPI build adds all the devices and buses under the
IVHD. In order to support the IOMMU attached on a non system bus and
serving only child devices of that bus, start exploring devices from the
child bus. Also, use child list provided by the PCIBus, instead of
iterating over all the objects in the qemu to find pci bridges.

Signed-off-by: Sairaj Kodilkar <sarunkod@amd.com>
---
 hw/i386/acpi-build.c | 142 ++++++++++++++++++++++---------------------
 1 file changed, 74 insertions(+), 68 deletions(-)

diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 8837b69687d1..ed944c819406 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -1760,83 +1760,89 @@ insert_ivhd(PCIBus *bus, PCIDevice *dev, void *opaque)
     /* "Select" IVHD entry, type 0x2 */
     entry = PCI_BUILD_BDF(pci_bus_num(bus), dev->devfn) << 8 | 0x2;
     build_append_int_noprefix(table_data, entry, 4);
+}
 
-    if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_BRIDGE)) {
-        PCIBus *sec_bus = pci_bridge_get_sec_bus(PCI_BRIDGE(dev));
-        uint8_t sec = pci_bus_num(sec_bus);
-        uint8_t sub = dev->config[PCI_SUBORDINATE_BUS];
-
-        if (pci_bus_is_express(sec_bus)) {
-            /*
-             * Walk the bus if there are subordinates, otherwise use a range
-             * to cover an entire leaf bus.  We could potentially also use a
-             * range for traversed buses, but we'd need to take care not to
-             * create both Select and Range entries covering the same device.
-             * This is easier and potentially more compact.
-             *
-             * An example bare metal system seems to use Select entries for
-             * root ports without a slot (ie. built-ins) and Range entries
-             * when there is a slot.  The same system also only hard-codes
-             * the alias range for an onboard PCIe-to-PCI bridge, apparently
-             * making no effort to support nested bridges.  We attempt to
-             * be more thorough here.
-             */
-            if (sec == sub) { /* leaf bus */
-                /* "Start of Range" IVHD entry, type 0x3 */
-                entry = PCI_BUILD_BDF(sec, PCI_DEVFN(0, 0)) << 8 | 0x3;
-                build_append_int_noprefix(table_data, entry, 4);
-                /* "End of Range" IVHD entry, type 0x4 */
-                entry = PCI_BUILD_BDF(sub, PCI_DEVFN(31, 7)) << 8 | 0x4;
-                build_append_int_noprefix(table_data, entry, 4);
-            } else {
-                pci_for_each_device(sec_bus, sec, insert_ivhd, table_data);
-            }
-        } else {
-            /*
-             * If the secondary bus is conventional, then we need to create an
-             * Alias range for everything downstream.  The range covers the
-             * first devfn on the secondary bus to the last devfn on the
-             * subordinate bus.  The alias target depends on legacy versus
-             * express bridges, just as in pci_device_iommu_address_space().
-             * DeviceIDa vs DeviceIDb as per the AMD IOMMU spec.
-             */
-            uint16_t dev_id_a, dev_id_b;
-
-            dev_id_a = PCI_BUILD_BDF(sec, PCI_DEVFN(0, 0));
+/*
+ * Insert IVHD entry for device and recurse, insert alias, or insert range as
+ * necessary for the PCI topology.
+ */
+static void add_ivhd_entries_from_bus(PCIBus *bus, GArray *table_data)
+{
+    uint8_t sub;
+    uint32_t entry;
+    PCIBus *child_bus;
+    PCIDevice *parent_dev;
 
-            if (pci_is_express(dev) &&
-                pcie_cap_get_type(dev) == PCI_EXP_TYPE_PCI_BRIDGE) {
-                dev_id_b = dev_id_a;
-            } else {
-                dev_id_b = PCI_BUILD_BDF(pci_bus_num(bus), dev->devfn);
-            }
+    if (!bus->parent_dev) {
+        goto select_devices;
+    }
 
-            /* "Alias Start of Range" IVHD entry, type 0x43, 8 bytes */
-            build_append_int_noprefix(table_data, dev_id_a << 8 | 0x43, 4);
-            build_append_int_noprefix(table_data, dev_id_b << 8 | 0x0, 4);
+    parent_dev = bus->parent_dev;
+    sub = parent_dev->config[PCI_SUBORDINATE_BUS];
 
-            /* "End of Range" IVHD entry, type 0x4 */
-            entry = PCI_BUILD_BDF(sub, PCI_DEVFN(31, 7)) << 8 | 0x4;
-            build_append_int_noprefix(table_data, entry, 4);
+    if (pci_bus_is_express(bus)) {
+        /*
+         * Walk the bus if there are subordinates, otherwise use a range
+         * to cover an entire leaf bus.  We could potentially also use a
+         * range for traversed buses, but we'd need to take care not to
+         * create both Select and Range entries covering the same device.
+         * This is easier and potentially more compact.
+         *
+         * An example bare metal system seems to use Select entries for
+         * root ports without a slot (ie. built-ins) and Range entries
+         * when there is a slot.  The same system also only hard-codes
+         * the alias range for an onboard PCIe-to-PCI bridge, apparently
+         * making no effort to support nested bridges.  We attempt to
+         * be more thorough here.
+         */
+        if (sub != pci_bus_num(bus)) {
+            goto select_devices;
         }
-    }
-}
+        /* "Start of Range" IVHD entry, type 0x3 */
+        entry = PCI_BUILD_BDF(sub, PCI_DEVFN(0, 0)) << 8 | 0x3;
+        build_append_int_noprefix(table_data, entry, 4);
+        /* "End of Range" IVHD entry, type 0x4 */
+        entry = PCI_BUILD_BDF(sub, PCI_DEVFN(31, 7)) << 8 | 0x4;
+        build_append_int_noprefix(table_data, entry, 4);
 
-/* For all PCI host bridges, walk and insert IVHD entries */
-static int
-ivrs_host_bridges(Object *obj, void *opaque)
-{
-    GArray *ivhd_blob = opaque;
+        return;
+    } else {
+        /*
+         * If the secondary bus is conventional, then we need to create an
+         * Alias range for everything downstream.  The range covers the
+         * first devfn on the secondary bus to the last devfn on the
+         * subordinate bus.  The alias target depends on legacy versus
+         * express bridges, just as in pci_device_iommu_address_space().
+         * DeviceIDa vs DeviceIDb as per the AMD IOMMU spec.
+         */
+        uint16_t dev_id_a, dev_id_b;
 
-    if (object_dynamic_cast(obj, TYPE_PCI_HOST_BRIDGE)) {
-        PCIBus *bus = PCI_HOST_BRIDGE(obj)->bus;
+        dev_id_a = PCI_BUILD_BDF(pci_bus_num(bus), PCI_DEVFN(0, 0));
 
-        if (bus && !pci_bus_bypass_iommu(bus)) {
-            pci_for_each_device_under_bus(bus, insert_ivhd, ivhd_blob);
+        if (pci_is_express(parent_dev) &&
+            pcie_cap_get_type(parent_dev) == PCI_EXP_TYPE_PCI_BRIDGE) {
+            dev_id_b = dev_id_a;
+        } else {
+            dev_id_b = PCI_BUILD_BDF(pci_bus_num(pci_get_bus(parent_dev)),
+                                     parent_dev->devfn);
         }
+
+        /* "Alias Start of Range" IVHD entry, type 0x43, 8 bytes */
+        build_append_int_noprefix(table_data, dev_id_a << 8 | 0x43, 4);
+        build_append_int_noprefix(table_data, dev_id_b << 8 | 0x0, 4);
+
+        /* "End of Range" IVHD entry, type 0x4 */
+        entry = PCI_BUILD_BDF(sub, PCI_DEVFN(31, 7)) << 8 | 0x4;
+        build_append_int_noprefix(table_data, entry, 4);
+        return;
     }
 
-    return 0;
+select_devices:
+    pci_for_each_device_under_bus(bus, insert_ivhd, table_data);
+
+    QLIST_FOREACH(child_bus, &bus->child, sibling) {
+        add_ivhd_entries_from_bus(child_bus, table_data);
+    }
 }
 
 static void
@@ -1867,8 +1873,8 @@ build_amd_iommu(GArray *table_data, BIOSLinker *linker, const char *oem_id,
      * blob further below.  Fall back to an entry covering all devices, which
      * is sufficient when no aliases are present.
      */
-    object_child_foreach_recursive(object_get_root(),
-                                   ivrs_host_bridges, ivhd_blob);
+    assert(s->root_bus);
+    add_ivhd_entries_from_bus(s->root_bus, ivhd_blob);
 
     if (!ivhd_blob->len) {
         /*
-- 
2.34.1

[PATCH 8/8] amd_iommu: Allow IOMMU to attachment on non-system (pcie.0) bus

[Sairaj Kodilkar]

Allow the AMD IOMMU to attach on the pcie-root-port using pci device's "bus"
property. This helps in isolating the set of devices by creating a pcie
hierarchy under the pci expansion bus (pxb-pci). The IOMMU always tries
to find the first root bus by traversing up the PCI hierarchy and serves
all the devices under that bus.

Example1, for below PCI configuration, IOMMU will serve all the devices
under pxb-pci (i.e. Device1 and Device2) since its a first root bus,
but will not serve Device3 and Device3, effectively isolating the
devices under pxb-pci bus.

pcie.0 -+- pxb-pci -+- pcie-root-port1 --- AMD IOMMU
        |           +- pcie-root-port2 --- Device1
        |           \- pcie-root-port3 --- Device2
        +- Device3
        \- Device4

Example2, for below PCI configuration, IOMMU will serve all the devices
Device[1-4], since first root bus in the parent hierarchy is pcie.0

pcie.0 -+- pcie-root-port1 --- AMD IOMMU
        +- pcie-root-port2 --- Device1
        +- pcie-root-port3 --- Device2
        +- Device3
        \- Device4

AMD IOMMU uses this root bus to register pci IOMMU ops and build the IVRS
table.

Signed-off-by: Sairaj Kodilkar <sarunkod@amd.com>
---
 hw/i386/amd_iommu.c | 55 +++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 51 insertions(+), 4 deletions(-)

diff --git a/hw/i386/amd_iommu.c b/hw/i386/amd_iommu.c
index 5e4d5d1f1fcd..e0968127ec40 100644
--- a/hw/i386/amd_iommu.c
+++ b/hw/i386/amd_iommu.c
@@ -24,6 +24,7 @@
 #include "hw/i386/pc.h"
 #include "hw/pci/msi.h"
 #include "hw/pci/pci_bus.h"
+#include "hw/pci/pci_bridge.h"
 #include "migration/vmstate.h"
 #include "amd_iommu.h"
 #include "qapi/error.h"
@@ -2680,6 +2681,50 @@ static const VMStateDescription vmstate_amdvi_sysbus_migratable = {
      }
 };
 
+/* Go up in the PCI hierarchy and find the root bus pcie.0 or pxb-pci */
+static PCIBus *get_root_bus(struct PCIBus *bus)
+{
+    while (bus) {
+
+        if (!pci_bus_is_express(bus)) {
+            error_report("attached to non PCI express bus %s",
+                         bus->qbus.name);
+            goto err_out;
+        }
+
+        if (bus->iommu_ops) {
+            error_report("different IOMMU already serves bus %s",
+                         bus->qbus.name);
+            goto err_out;
+        }
+
+        if (pci_bus_is_root(bus) &&
+            object_dynamic_cast(OBJECT(bus)->parent, TYPE_PCI_HOST_BRIDGE)) {
+
+            if (bus->parent_dev &&
+                !object_dynamic_cast(OBJECT(bus), TYPE_PXB_PCIE_BUS)) {
+
+                    error_report("only supports PXB-PCI as extra root bus");
+                    goto err_out;
+            }
+            /* We found the valid root bus so return it to the user*/
+            goto out;
+        }
+
+        if (bus->parent_dev) {
+            bus = pci_get_bus(bus->parent_dev);
+        } else {
+            error_report("could not find valid PCIE bus");
+            goto err_out;
+        }
+    };
+
+err_out:
+    exit(EXIT_FAILURE);
+out:
+    return bus;
+}
+
 static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)
 {
     DeviceClass *dc = (DeviceClass *) object_get_class(OBJECT(dev));
@@ -2687,7 +2732,7 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)
     MachineState *ms = MACHINE(qdev_get_machine());
     PCMachineState *pcms = PC_MACHINE(ms);
     X86MachineState *x86ms = X86_MACHINE(ms);
-    PCIBus *bus = pcms->pcibus;
+    PCIBus *iommu_bus;
 
     if (s->pci_id) {
         PCIDevice *pdev = NULL;
@@ -2703,17 +2748,19 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)
             return;
         }
 
+        iommu_bus = pci_get_bus(pdev);
         s->pci = AMD_IOMMU_PCI(pdev);
         dc->vmsd = &vmstate_amdvi_sysbus_migratable;
     } else {
+        iommu_bus = pcms->pcibus;
         s->pci = AMD_IOMMU_PCI(object_new(TYPE_AMD_IOMMU_PCI));
         /* This device should take care of IOMMU PCI properties */
-        if (!qdev_realize(DEVICE(s->pci), &bus->qbus, errp)) {
+        if (!qdev_realize(DEVICE(s->pci), &iommu_bus->qbus, errp)) {
             return;
         }
     }
 
-    s->root_bus = bus;
+    s->root_bus = get_root_bus(iommu_bus);
 
     s->iotlb = g_hash_table_new_full(amdvi_iotlb_hash,
                                      amdvi_iotlb_equal, g_free, g_free);
@@ -2767,7 +2814,7 @@ static void amdvi_sysbus_realize(DeviceState *dev, Error **errp)
         }
     }
 
-    pci_setup_iommu(bus, &amdvi_iommu_ops, s);
+    pci_setup_iommu(s->root_bus, &amdvi_iommu_ops, s);
     amdvi_init(s);
 }
 
-- 
2.34.1

Re: [PATCH 4/8] intel_iommu: Add support for get_x86_iommu callback

[Clément MATHIEU--DRIF]

Hi,

Reviewed-by: Clement Mathieu--Drif <clement.mathieu--drif@bull.com>

On Wed, 2026-10-14 at 11:04 +0530, Sairaj Kodilkar wrote:
> Caution: External email. Do not open attachments or click links, unless this email comes from a known sender and you know the content is safe.
> 
> 
> Return X86IOMMUState during get_x86_iommu callback.
> 
> Signed-off-by: Sairaj Kodilkar <[sarunkod@amd.com](mailto:sarunkod@amd.com)>  
> ---  
>  hw/i386/intel_iommu.c | 8 ++++++++  
>  1 file changed, 8 insertions(+)
> 
> diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c  
> index 82c3c3b2c35f..92a2d6eebb61 100644  
> --- a/hw/i386/intel_iommu.c  
> +++ b/hw/i386/intel_iommu.c  
> @@ -5461,6 +5461,13 @@ static int vtd_pri_request_page(PCIBus *bus, void *opaque, int devfn,  
>      return 0;  
>  }
> 
> +static int vtd_get_x86_iommu(void *opaque, void **x86_iommu)  
> +{  
> +    IntelIOMMUState *s = opaque;  
> +    *x86_iommu = X86_IOMMU_DEVICE(s);  
> +    return 0;  
> +}  
> +  
>  static void vtd_init_iotlb_notifier(PCIBus *bus, void *opaque, int devfn,  
>                                      IOMMUNotifier *n, IOMMUNotify fn,  
>                                      void *user_opaque)  
> @@ -5535,6 +5542,7 @@ static PCIIOMMUOps vtd_iommu_ops = {  
>      .pri_unregister_notifier = vtd_pri_unregister_notifier,  
>      .pri_request_page = vtd_pri_request_page,  
>      .get_viommu_flags = vtd_get_viommu_flags,  
> +    .get_x86_iommu = vtd_get_x86_iommu,  
>  };
> 
>  static bool vtd_decide_config(IntelIOMMUState *s, Error **errp)  
> --  
> 2.34.1
>