[PATCH 0/8] amd_iommu: Allow IOMMU to attach on non system bus

[Sairaj Kodilkar <sarunkod@amd.com>]

Overview
--------
This patch series enables AMD IOMMU to be attached to non-system root buses
(e. g. , pcie-pxb, pci-root-bus), allowing device isolation between VFIO
passthrough devices and emulated devices.  This is a prerequisite for
hardware-accelerated virtual IOMMU (vIOMMU) support. 


Problem Statement
-----------------
The current AMD IOMMU implementation has the following limitations:

1. Only supports attachment to the default pcie.0 system bus

2. Builds IVRS (I/O Virtualization Reporting Structure) by iterating over
   all system buses, preventing device isolation

3. Cannot separate VFIO passthrough devices from emulated devices on
   different buses

This creates a fundamental issue for hardware-accelerated vIOMMU, which
requires VFIO devices to be isolated from emulated devices, as the vIOMMU
can only operate on passthrough devices. 


Solution Approach
-----------------
This series introduces the following changes:

1. Allow AMD IOMMU to be attached to arbitrary root buses using the "bus="
   parameter, while Maintaining the backward compatibility by selecting the
   pcie.0 by default if this parameter is missing.

2. Implement root bus discovery by traversing the parent hierarchy to find
   the first root bus

3. Register IOMMU operations on the discovered root bus

4. Modify IVRS generation to build IVHD (I/O Virtualization Hardware
   Definition) entries based on the specified root bus, traversing child
   buses and devices in a top-down manner

5. Introduce x86_iommu_device_get() and the get_x86_iommu() callback so the
   KVM layer can determine which IOMMU serves a given device.

   kvm_arch_fixup_msi_route() and kvm_arch_add_msi_route_post() call
   x86_iommu_get_default() for every MSI route. That returns the single
   machine IOMMU and is correct only when that IOMMU is attached to pcie.0
   and serves every device.

   Once the IOMMU is attached below another root bus, it serves only the
   devices under that bus. The KVM layer must program interrupt-remapping
   routes through that IOMMU and leave devices outside its scope untranslated.
   The PCI core can already find the bus that owns a device, but PCIIOMMUOps
   cannot return the corresponding X86IOMMUState. get_x86_iommu() provides
   that link. x86_iommu_device_get() uses it, and both AMD and Intel IOMMU
   implement the callback because this KVM path is shared.


Command line
------------
qemu-system-x86_64 \
-enable-kvm -m 10G -smp 8 \
-netdev user,id=USER0,hostfwd=tcp::3333-:22 \
-cpu EPYC-Genoa,x2apic=on,kvm-msi-ext-dest-id=on,+kvm-pv-unhalt,kvm-pv-tlb-flush,kvm-pv-ipi,kvm-pv-sched-yield  \
-machine q35,kernel_irqchip=split \
-global kvm-pit.lost_tick_policy=discard \
-nographic \
-blockdev node-name=drive0,driver=qcow2,file.driver=file,file.filename=<IMG> \
-device virtio-blk-pci,num-queues=8,drive=drive0 \
-device pxb-pcie,id=pcie.1,bus=pcie.0,bus_nr=0x50 \
-device pcie-root-port,id=rp1,bus=pcie.1,chassis=3,slot=1 \
-device AMDVI-PCI,id=iommupci1,bus=rp1 \
-device amd-iommu,intremap=on,dma-remap=on,xtsup=on,pci-id=iommupci1 \
-device pcie-root-port,id=rp2,bus=pcie.2,chassis=4,slot=1 \
-device virtio-net-pci,id=vnet1,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER0,bus=rp2 \


Example
-------
Example1, for below PCI configuration, IOMMU will serve all the devices
under pxb-pci (i.e. Device1 and Device2) since its a first root bus,
but will not serve Device3 and Device4, effectively isolating the
devices under pxb-pci bus.

pcie.0 -+- pxb-pci -+- pcie-root-port1 --- AMD IOMMU
        |           +- pcie-root-port2 --- Device1
        |           \- pcie-root-port3 --- Device2
        +- Device3
        \- Device4

Example2, for below PCI configuration, IOMMU will serve all the devices
Device[1-4], since first root bus in the parent hierarchy is pcie.0

pcie.0 -+- pcie-root-port1 --- AMD IOMMU
        +- pcie-root-port2 --- Device1
        +- pcie-root-port3 --- Device2
        +- Device3
        \- Device4

Example3, Below three PCI topologies will cause failure as AMD IOMMU can
only attach to **PCIE** hierarchy.

[A]
pcie.0 -+- pcie-pci-bridge --- AMD IOMMU
        \- Device1

[B]
pcie.0 -+- pci-bridge --- AMD IOMMU
        \- Device1

[C]
pcie.0 -+- pci-bridge --- pcie-root-port --- AMD IOMMU
        \- Device1

Sairaj Kodilkar (8):
  amd_iommu: Assign ioapic_as only when interrupt remapping is
    supported.
  x86-iommu: Introduce x86_iommu_device_get() for x86-iommu retrieval
  amd_iommu: Add support for get_x86_iommu callback
  intel_iommu: Add support for get_x86_iommu callback
  i386/kvm: Use x86_iommu_device_get() for per-device IOMMU lookup
  amd_iommu: Introduce root bus
  acpi-build: Create PCI topology using amd iommu root bus
  amd_iommu: Allow IOMMU to attachment on non-system (pcie.0) bus

 hw/i386/acpi-build.c        | 142 +++++++++++++++++++-----------------
 hw/i386/amd_iommu.c         |  75 +++++++++++++++++--
 hw/i386/amd_iommu.h         |   3 +
 hw/i386/intel_iommu.c       |   8 ++
 hw/i386/x86-iommu-stub.c    |   5 ++
 hw/i386/x86-iommu.c         |  16 ++++
 hw/intc/ioapic.c            |   3 +-
 include/hw/i386/x86-iommu.h |   6 ++
 include/hw/pci/pci.h        |  12 +++
 target/i386/kvm/kvm.c       |  15 +++-
 10 files changed, 208 insertions(+), 77 deletions(-)


base-commit: 562bae590f194fb590beb5c65da44fc35ab9f64a
-- 
2.34.1