[PATCH 0/4] Integrate IOMMUs with PCI hosts that have ATUs

[PATCH 0/4] Integrate IOMMUs with PCI hosts that have ATUs

[Jason Chien]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 1126 bytes --]

The struct PCIIOMMUOps is designed for use by an IOMMU, but many PCI hosts
also utilize it to implement their ATUs, preventing coexistence with IOMMUs.
Overwriting a PCI host’s PCIIOMMUOps disrupts its translation rules.

This patch series introduces a mechanism to route inbound transactions from
PCI hosts to the IOMMU, enabling proper integration.

The final patch depends on another patch series:
https://lists.nongnu.org/archive/html/qemu-riscv/2025-03/msg00003.html

Jason Chien (4):
  include/hw/pci: Introduce a callback to set the downstream mr of PCI
    hosts
  hw/pci: Introduce an API to set PCI host downstream mr for IOMMU
    integration
  hw/pci-host/designware: Implement PCIIOMMUOps.set_downstream_mr()
  hw/riscv/riscv-iommu: Connect the IOMMU with PCI hosts that have ATUs

 hw/pci-host/designware.c         | 18 +++++++++++++++---
 hw/pci/pci.c                     |  8 ++++++++
 hw/riscv/riscv-iommu.c           | 15 ++++++++++-----
 include/hw/pci-host/designware.h |  2 ++
 include/hw/pci/pci.h             | 21 +++++++++++++++++++++
 5 files changed, 56 insertions(+), 8 deletions(-)

-- 
2.43.2

[PATCH 1/4] include/hw/pci: Introduce a callback to set the downstream mr of PCI hosts

[Jason Chien]

Many PCI hosts utilize struct PCIIOMMUOps to implement their ATUs, preventing
coexistence with IOMMUs, which need to register struct PCIIOMMUOps as well.

To resolve this, set_downstream_mr() is introduced, allowing IOMMUs to
configure the downstream memory region of the PCI host, enabling both to
coexist.

Signed-off-by: Jason Chien <jason.chien@sifive.com>
---
 include/hw/pci/pci.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index 4002bbeebd..fcf648da19 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -391,6 +391,18 @@ typedef struct PCIIOMMUOps {
      * @devfn: device and function number
      */
     AddressSpace * (*get_address_space)(PCIBus *bus, void *opaque, int devfn);
+    /**
+     * @set_downstream_mr: set the downstream memory region for the PCI host.
+     *
+     * Optional callback that should be implemented if a PCI host registers
+     * this PCIIOMMUOps. It allows an IOMMU to designate its memory region as
+     * the downstream memory region of the PCI host.
+     *
+     * @opaque: the data passed to pci_setup_iommu().
+     *
+     * @mr: the downstream memory region
+     */
+    void (*set_downstream_mr)(void *opaque, MemoryRegion *mr);
     /**
      * @set_iommu_device: attach a HostIOMMUDevice to a vIOMMU
      *
-- 
2.43.2

[PATCH 2/4] hw/pci: Introduce an API to set PCI host downstream mr for IOMMU integration

[Jason Chien]

When an IOMMU detects that a PCI host has registered struct PCIIOMMUOps,
it should call pci_setup_iommu_downstream_mr(), which invokes
PCIIOMMUOps.set_downstream_mr() to configure the PCI host's downstream
memory region, directing inbound transactions to the IOMMU.

Signed-off-by: Jason Chien <jason.chien@sifive.com>
---
 hw/pci/pci.c         | 8 ++++++++
 include/hw/pci/pci.h | 9 +++++++++
 2 files changed, 17 insertions(+)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 1d42847ef0..983290ef0b 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -2858,6 +2858,14 @@ void pci_device_unset_iommu_device(PCIDevice *dev)
     }
 }
 
+void pci_setup_iommu_downstream_mr(PCIBus *bus, MemoryRegion *mr)
+{
+    assert(bus->iommu_ops);
+    assert(bus->iommu_ops->set_downstream_mr);
+
+    bus->iommu_ops->set_downstream_mr(bus->iommu_opaque, mr);
+}
+
 void pci_setup_iommu(PCIBus *bus, const PCIIOMMUOps *ops, void *opaque)
 {
     /*
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index fcf648da19..1ad5dc7d9d 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -442,6 +442,15 @@ bool pci_device_set_iommu_device(PCIDevice *dev, HostIOMMUDevice *hiod,
                                  Error **errp);
 void pci_device_unset_iommu_device(PCIDevice *dev);
 
+/**
+ * pci_setup_iommu_downstream_mr: Designate a downstream memory region
+ * for a PCIBus
+ *
+ * @bus: the #PCIBus being updated.
+ * @mr: the designated memory region.
+ */
+void pci_setup_iommu_downstream_mr(PCIBus *bus, MemoryRegion *mr);
+
 /**
  * pci_setup_iommu: Initialize specific IOMMU handlers for a PCIBus
  *
-- 
2.43.2

[PATCH 3/4] hw/pci-host/designware: Implement PCIIOMMUOps.set_downstream_mr()

[Jason Chien]

The original DesignWare PCIe host implementation could not connect with
an IOMMU, as both attempted to register PCIIOMMUOps.

This commit resolves the conflict by hooking PCIIOMMUOps.set_downstream_mr()
through designware_pcie_host_set_mr(), allowing the IOMMU to designate the
PCIe host's downstream memory region via pci_setup_iommu_downstream_mr()
without competing for PCIIOMMUOps.

Signed-off-by: Jason Chien <jason.chien@sifive.com>
---
 hw/pci-host/designware.c         | 18 +++++++++++++++---
 include/hw/pci-host/designware.h |  2 ++
 2 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/hw/pci-host/designware.c b/hw/pci-host/designware.c
index c07740bfaa..fafbf90259 100644
--- a/hw/pci-host/designware.c
+++ b/hw/pci-host/designware.c
@@ -404,7 +404,6 @@ static void designware_pcie_root_realize(PCIDevice *dev, Error **errp)
 {
     DesignwarePCIERoot *root = DESIGNWARE_PCIE_ROOT(dev);
     DesignwarePCIEHost *host = designware_pcie_root_to_host(root);
-    MemoryRegion *host_mem = get_system_memory();
     MemoryRegion *address_space = &host->pci.memory;
     PCIBridge *br = PCI_BRIDGE(dev);
     DesignwarePCIEViewport *viewport;
@@ -445,7 +444,7 @@ static void designware_pcie_root_realize(PCIDevice *dev, Error **errp)
         viewport->cr[0]   = DESIGNWARE_PCIE_ATU_TYPE_MEM;
 
         source      = &host->pci.address_space_root;
-        destination = host_mem;
+        destination = &host->bridge_mr;
         direction   = "Inbound";
 
         /*
@@ -470,7 +469,7 @@ static void designware_pcie_root_realize(PCIDevice *dev, Error **errp)
 
         destination = &host->pci.memory;
         direction   = "Outbound";
-        source      = host_mem;
+        source      = get_system_memory();
 
         /*
          * Configure MemoryRegion implementing CPU -> PCI memory
@@ -675,8 +674,16 @@ static AddressSpace *designware_pcie_host_set_iommu(PCIBus *bus, void *opaque,
     return &s->pci.address_space;
 }
 
+static void designware_pcie_host_set_mr(void *opaque, MemoryRegion *mr)
+{
+    DesignwarePCIEHost *s = DESIGNWARE_PCIE_HOST(opaque);
+
+    memory_region_add_subregion_overlap(&s->bridge_mr, 0, mr, INT32_MAX);
+}
+
 static const PCIIOMMUOps designware_iommu_ops = {
     .get_address_space = designware_pcie_host_set_iommu,
+    .set_downstream_mr = designware_pcie_host_set_mr,
 };
 
 static void designware_pcie_host_realize(DeviceState *dev, Error **errp)
@@ -713,6 +720,11 @@ static void designware_pcie_host_realize(DeviceState *dev, Error **errp)
                                      TYPE_DESIGNWARE_PCIE_ROOT_BUS);
     pci->bus->flags |= PCI_BUS_EXTENDED_CONFIG_SPACE;
 
+    memory_region_init(&s->bridge_mr, OBJECT(s),
+                       "pcie-bus-bridge-memory", UINT64_MAX);
+    memory_region_add_subregion(&s->bridge_mr, 0x0, get_system_memory());
+    address_space_init(&s->bridge_as, &s->bridge_mr, "pcie-bus-bridge-space");
+
     memory_region_init(&s->pci.address_space_root,
                        OBJECT(s),
                        "pcie-bus-address-space-root",
diff --git a/include/hw/pci-host/designware.h b/include/hw/pci-host/designware.h
index a35a3bd06c..30f1598dbe 100644
--- a/include/hw/pci-host/designware.h
+++ b/include/hw/pci-host/designware.h
@@ -97,6 +97,8 @@ struct DesignwarePCIEHost {
     } pci;
 
     MemoryRegion mmio;
+    AddressSpace bridge_as;
+    MemoryRegion bridge_mr;
 };
 
 #endif /* DESIGNWARE_H */
-- 
2.43.2

[PATCH 4/4] hw/riscv/riscv-iommu: Connect the IOMMU with PCI hosts that have ATUs

[Jason Chien]

When the IOMMU detects that bus->iommu_ops has been registered, indicating
the presence of an ATU, it sets the bus's downstream memory region to ensure
transactions are directed to the IOMMU.

Signed-off-by: Jason Chien <jason.chien@sifive.com>
---
 hw/riscv/riscv-iommu.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/hw/riscv/riscv-iommu.c b/hw/riscv/riscv-iommu.c
index d46beb2d64..9701fe3831 100644
--- a/hw/riscv/riscv-iommu.c
+++ b/hw/riscv/riscv-iommu.c
@@ -2628,11 +2628,16 @@ static const PCIIOMMUOps riscv_iommu_ops = {
 void riscv_iommu_pci_setup_iommu(RISCVIOMMUState *iommu, PCIBus *bus,
         Error **errp)
 {
-    if (bus->iommu_ops &&
-        bus->iommu_ops->get_address_space == riscv_iommu_find_as) {
-        /* Allow multiple IOMMUs on the same PCIe bus, link known devices */
-        RISCVIOMMUState *last = (RISCVIOMMUState *)bus->iommu_opaque;
-        QLIST_INSERT_AFTER(last, iommu, iommus);
+    if (bus->iommu_ops) {
+        if (bus->iommu_ops->get_address_space == riscv_iommu_find_as) {
+            /* Allow multiple IOMMUs on the same PCIe bus, link known devices */
+            RISCVIOMMUState *last = (RISCVIOMMUState *)bus->iommu_opaque;
+            QLIST_INSERT_AFTER(last, iommu, iommus);
+        } else {
+            /* The bus has an ATU. Set its downsteam memory region. */
+            AddressSpace *as = riscv_iommu_space(iommu, 0);
+            pci_setup_iommu_downstream_mr(bus, as->root);
+        }
     } else if (!bus->iommu_ops && !bus->iommu_opaque) {
         pci_setup_iommu(bus, &riscv_iommu_ops, iommu);
     } else {
-- 
2.43.2

Re: [PATCH 4/4] hw/riscv/riscv-iommu: Connect the IOMMU with PCI hosts that have ATUs

[Daniel Henrique Barboza]

On 3/7/25 5:39 PM, Jason Chien wrote:
> When the IOMMU detects that bus->iommu_ops has been registered, indicating
> the presence of an ATU, it sets the bus's downstream memory region to ensure
> transactions are directed to the IOMMU.
> 
> Signed-off-by: Jason Chien <jason.chien@sifive.com>
> ---


Reviewed-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>


>   hw/riscv/riscv-iommu.c | 15 ++++++++++-----
>   1 file changed, 10 insertions(+), 5 deletions(-)
> 
> diff --git a/hw/riscv/riscv-iommu.c b/hw/riscv/riscv-iommu.c
> index d46beb2d64..9701fe3831 100644
> --- a/hw/riscv/riscv-iommu.c
> +++ b/hw/riscv/riscv-iommu.c
> @@ -2628,11 +2628,16 @@ static const PCIIOMMUOps riscv_iommu_ops = {
>   void riscv_iommu_pci_setup_iommu(RISCVIOMMUState *iommu, PCIBus *bus,
>           Error **errp)
>   {
> -    if (bus->iommu_ops &&
> -        bus->iommu_ops->get_address_space == riscv_iommu_find_as) {
> -        /* Allow multiple IOMMUs on the same PCIe bus, link known devices */
> -        RISCVIOMMUState *last = (RISCVIOMMUState *)bus->iommu_opaque;
> -        QLIST_INSERT_AFTER(last, iommu, iommus);
> +    if (bus->iommu_ops) {
> +        if (bus->iommu_ops->get_address_space == riscv_iommu_find_as) {
> +            /* Allow multiple IOMMUs on the same PCIe bus, link known devices */
> +            RISCVIOMMUState *last = (RISCVIOMMUState *)bus->iommu_opaque;
> +            QLIST_INSERT_AFTER(last, iommu, iommus);
> +        } else {
> +            /* The bus has an ATU. Set its downsteam memory region. */
> +            AddressSpace *as = riscv_iommu_space(iommu, 0);
> +            pci_setup_iommu_downstream_mr(bus, as->root);
> +        }
>       } else if (!bus->iommu_ops && !bus->iommu_opaque) {
>           pci_setup_iommu(bus, &riscv_iommu_ops, iommu);
>       } else {

Re: [PATCH 0/4] Integrate IOMMUs with PCI hosts that have ATUs

[Jason Chien]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

Ping

Jason Chien <jason.chien@sifive.com> 於 2025年3月8日 週六 上午4:40寫道：

> The struct PCIIOMMUOps is designed for use by an IOMMU, but many PCI hosts
> also utilize it to implement their ATUs, preventing coexistence with
> IOMMUs.
> Overwriting a PCI host’s PCIIOMMUOps disrupts its translation rules.
>
> This patch series introduces a mechanism to route inbound transactions from
> PCI hosts to the IOMMU, enabling proper integration.
>
> The final patch depends on another patch series:
> https://lists.nongnu.org/archive/html/qemu-riscv/2025-03/msg00003.html
>
> Jason Chien (4):
>   include/hw/pci: Introduce a callback to set the downstream mr of PCI
>     hosts
>   hw/pci: Introduce an API to set PCI host downstream mr for IOMMU
>     integration
>   hw/pci-host/designware: Implement PCIIOMMUOps.set_downstream_mr()
>   hw/riscv/riscv-iommu: Connect the IOMMU with PCI hosts that have ATUs
>
>  hw/pci-host/designware.c         | 18 +++++++++++++++---
>  hw/pci/pci.c                     |  8 ++++++++
>  hw/riscv/riscv-iommu.c           | 15 ++++++++++-----
>  include/hw/pci-host/designware.h |  2 ++
>  include/hw/pci/pci.h             | 21 +++++++++++++++++++++
>  5 files changed, 56 insertions(+), 8 deletions(-)
>
> --
> 2.43.2
>
>

[-- Attachment #2: Type: text/html, Size: 1823 bytes --]

Re: [PATCH 0/4] Integrate IOMMUs with PCI hosts that have ATUs

[Michael S. Tsirkin]

On Thu, Mar 20, 2025 at 01:19:18AM +0800, Jason Chien wrote:
> Ping
> 
> Jason Chien <jason.chien@sifive.com> 於 2025年3月8日週六上午4:40寫道：
> 
>     The struct PCIIOMMUOps is designed for use by an IOMMU, but many PCI hosts
>     also utilize it to implement their ATUs, preventing coexistence with
>     IOMMUs.
>     Overwriting a PCI host’s PCIIOMMUOps disrupts its translation rules.
> 
>     This patch series introduces a mechanism to route inbound transactions from
>     PCI hosts to the IOMMU, enabling proper integration.
> 
>     The final patch depends on another patch series:
>     https://lists.nongnu.org/archive/html/qemu-riscv/2025-03/msg00003.html
> 
>     Jason Chien (4):
>       include/hw/pci: Introduce a callback to set the downstream mr of PCI
>         hosts
>       hw/pci: Introduce an API to set PCI host downstream mr for IOMMU
>         integration
>       hw/pci-host/designware: Implement PCIIOMMUOps.set_downstream_mr()
>       hw/riscv/riscv-iommu: Connect the IOMMU with PCI hosts that have ATUs
> 
>      hw/pci-host/designware.c         | 18 +++++++++++++++---
>      hw/pci/pci.c                     |  8 ++++++++
>      hw/riscv/riscv-iommu.c           | 15 ++++++++++-----
>      include/hw/pci-host/designware.h |  2 ++
>      include/hw/pci/pci.h             | 21 +++++++++++++++++++++
>      5 files changed, 56 insertions(+), 8 deletions(-)


Hello!
I tagged this for 10.1, as qemu entered code freeze.

>     --
>     2.43.2
> 
> 

Re: [PATCH 0/4] Integrate IOMMUs with PCI hosts that have ATUs

[Michael S. Tsirkin]

On Sat, Mar 08, 2025 at 04:39:33AM +0800, Jason Chien wrote:
> The struct PCIIOMMUOps is designed for use by an IOMMU, but many PCI hosts
> also utilize it to implement their ATUs, preventing coexistence with IOMMUs.
> Overwriting a PCI host’s PCIIOMMUOps disrupts its translation rules.
> 
> This patch series introduces a mechanism to route inbound transactions from
> PCI hosts to the IOMMU, enabling proper integration.
> 
> The final patch depends on another patch series:
> https://lists.nongnu.org/archive/html/qemu-riscv/2025-03/msg00003.html


PCI things:

Reviewed-by: Michael S. Tsirkin <mst@redhat.com>


given the dependency, pls merge through risc-v tree.

> Jason Chien (4):
>   include/hw/pci: Introduce a callback to set the downstream mr of PCI
>     hosts
>   hw/pci: Introduce an API to set PCI host downstream mr for IOMMU
>     integration
>   hw/pci-host/designware: Implement PCIIOMMUOps.set_downstream_mr()
>   hw/riscv/riscv-iommu: Connect the IOMMU with PCI hosts that have ATUs
> 
>  hw/pci-host/designware.c         | 18 +++++++++++++++---
>  hw/pci/pci.c                     |  8 ++++++++
>  hw/riscv/riscv-iommu.c           | 15 ++++++++++-----
>  include/hw/pci-host/designware.h |  2 ++
>  include/hw/pci/pci.h             | 21 +++++++++++++++++++++
>  5 files changed, 56 insertions(+), 8 deletions(-)
> 
> -- 
> 2.43.2