[PATCH v11 0/7] SMMU handling for PCIe Passthrough on ARM

All of lore.kernel.org
 help / color / mirror / Atom feed

* [PATCH v11 0/7] SMMU handling for PCIe Passthrough on ARM
@ 2025-05-28  9:12 Mykyta Poturai
  2025-05-28  9:12 ` [PATCH v11 2/7] iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling Mykyta Poturai
                   ` (6 more replies)
  0 siblings, 7 replies; 22+ messages in thread
From: Mykyta Poturai @ 2025-05-28  9:12 UTC (permalink / raw)
  To: xen-devel@lists.xenproject.org
  Cc: Mykyta Poturai, Stefano Stabellini, Julien Grall,
	Bertrand Marquis, Michal Orzel, Volodymyr Babchuk, Jan Beulich,
	Roger Pau Monné, Rahul Singh, Andrew Cooper, Anthony PERARD

This series introduces SMMU handling for PCIe passthrough on ARM. These patches
should be able to be upstreamed independently from the vPCI series [1]. See [2]
for notes about test cases.

[1] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00660.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg01135.html

v10->v11:
* see individual patches

v9->v10:
* drop iommu/arm: Add iommu_dt_xlate()
* see individual patches

v8->v9:
* see individual patches

v7->v8:
* no changes

v6->v7:
* drop ("xen/arm: don't pass iommu properties to hwdom for iommu-map")

v5->v6:
* don't revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: enable dom0 to use PCI devices with pci-passthrough=no")

v4->v5:
* drop ("xen/arm: Improve readability of check for registered devices")
* drop ("xen/arm: Move is_protected flag to struct device")
* add ("xen/arm: don't pass iommu properties to hwdom for iommu-map")
* add ("xen/arm: Fix mapping for PCI bridge mmio region")
* revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: Map ITS doorbell register to IOMMU page tables.")
* fix test case #1 with PCI device in dom0

v3->v4:
* split a change from ("xen/arm: Move is_protected flag to struct device") into
  a new separate patch
* see individual patches for further details

v2->v3:
* drop "pci/arm: Use iommu_add_dt_pci_device()"
* drop "RFC: pci/arm: don't do iommu call for phantom functions"
* move invocation of sideband ID mapping function to add_device()
  platform_ops/iommu_ops hook

Oleksandr Andrushchenko (1):
  xen/arm: smmuv2: Add PCI devices support for SMMUv2

Oleksandr Tyshchenko (1):
  iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API

Rahul Singh (3):
  xen/arm: smmuv3: Add PCI devices support for SMMUv3
  xen/arm: Fix mapping for PCI bridge mmio region
  xen/arm: Map ITS doorbell register to IOMMU page tables

Stewart Hildebrand (2):
  iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling
  xen/arm: enable dom0 to use PCI devices with pci-passthrough=no

 xen/arch/arm/device.c                 |   2 +-
 xen/arch/arm/include/asm/iommu.h      |  15 ++
 xen/arch/arm/include/asm/pci.h        |   2 +
 xen/arch/arm/pci/pci.c                |  14 +-
 xen/arch/arm/vgic-v3-its.c            |  24 ++-
 xen/arch/x86/include/asm/pci.h        |   6 +
 xen/common/device-tree/device-tree.c  |  91 ++++++++++
 xen/drivers/passthrough/arm/iommu.c   |  13 ++
 xen/drivers/passthrough/arm/smmu-v3.c | 119 +++++++++++--
 xen/drivers/passthrough/arm/smmu.c    | 246 +++++++++++++++-----------
 xen/drivers/passthrough/device_tree.c |  49 +++++
 xen/drivers/pci/physdev.c             |   4 +-
 xen/include/xen/device_tree.h         |  23 +++
 xen/include/xen/iommu.h               |  21 ++-
 xen/include/xen/pci.h                 |   5 +
 15 files changed, 517 insertions(+), 117 deletions(-)

-- 
2.34.1


^ permalink raw reply	[flat|nested] 22+ messages in thread

* [PATCH v11 2/7] iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling
  2025-05-28  9:12 [PATCH v11 0/7] SMMU handling for PCIe Passthrough on ARM Mykyta Poturai
@ 2025-05-28  9:12 ` Mykyta Poturai
  2025-05-28  9:12 ` [PATCH v11 1/7] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API Mykyta Poturai
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 22+ messages in thread
From: Mykyta Poturai @ 2025-05-28  9:12 UTC (permalink / raw)
  To: xen-devel@lists.xenproject.org
  Cc: Stewart Hildebrand, Stefano Stabellini, Julien Grall,
	Bertrand Marquis, Michal Orzel

From: Stewart Hildebrand <stewart.hildebrand@amd.com>

Handle phantom functions in iommu_add_dt_pci_sideband_ids(). Each phantom
function will have a unique requestor ID (RID)/BDF. On ARM, we need to
map/translate the RID/BDF to an AXI stream ID for each phantom function
according to the pci-iommu device tree mapping [1]. The RID/BDF -> AXI stream ID
mapping in DT could allow phantom devices (i.e. devices with phantom functions)
to use different AXI stream IDs based on the (phantom) function.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt

Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
v10->v11:
* no changes

v9->v10:
* Add Stefano's RB

v8->v9:
* replace DT_NO_IOMMU with 1

v7->v8:
* no change

v6->v7:
* no change

v5->v6:
* no change

v4->v5:
* no change

v3->v4:
* s/iommu_dt_pci_map_id/dt_map_id/

v2->v3:
* new patch title (was: iommu/arm: iommu_add_dt_pci_device phantom handling)
* rework loop to reduce duplication
* s/iommu_fwspec_free(pci_to_dev(pdev))/iommu_fwspec_free(dev)/

v1->v2:
* new patch

---
TODO: investigate Jan's comment [2]
[2] https://lore.kernel.org/xen-devel/806a2978-19fb-4d31-ab6a-35ea7317c8de@suse.com/
---
 xen/drivers/passthrough/device_tree.c | 33 ++++++++++++++++-----------
 1 file changed, 20 insertions(+), 13 deletions(-)

diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
index 37e1437b65..f5850a2607 100644
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -169,6 +169,7 @@ int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
     struct device *dev = pci_to_dev(pdev);
     const struct dt_device_node *np;
     int rc;
+    unsigned int devfn = pdev->devfn;
 
     if ( !iommu_enabled )
         return 1;
@@ -183,21 +184,27 @@ int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
     if ( !np )
         return -ENODEV;
 
-    /*
-     * According to the Documentation/devicetree/bindings/pci/pci-iommu.txt
-     * from Linux.
-     */
-    rc = dt_map_id(np, PCI_BDF(pdev->bus, pdev->devfn), "iommu-map",
-                   "iommu-map-mask", &iommu_spec.np, iommu_spec.args);
-    if ( rc )
-        return (rc == -ENODEV) ? 1 : rc;
+    do {
+        /*
+         * According to the Documentation/devicetree/bindings/pci/pci-iommu.txt
+         * from Linux.
+         */
+        rc = dt_map_id(np, PCI_BDF(pdev->bus, devfn), "iommu-map",
+                       "iommu-map-mask", &iommu_spec.np, iommu_spec.args);
+        if ( rc )
+            return (rc == -ENODEV) ? 1 : rc;
 
-    rc = iommu_dt_xlate(dev, &iommu_spec, ops);
-    if ( rc < 0 )
-    {
-        iommu_fwspec_free(dev);
-        return -EINVAL;
+        rc = iommu_dt_xlate(dev, &iommu_spec, ops);
+        if ( rc < 0 )
+        {
+            iommu_fwspec_free(dev);
+            return -EINVAL;
+        }
+
+        devfn += pdev->phantom_stride;
     }
+    while ( (devfn != pdev->devfn) &&
+            (PCI_SLOT(devfn) == PCI_SLOT(pdev->devfn)) );
 
     return rc;
 }
-- 
2.34.1


^ permalink raw reply related	[flat|nested] 22+ messages in thread

* [PATCH v11 1/7] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
  2025-05-28  9:12 [PATCH v11 0/7] SMMU handling for PCIe Passthrough on ARM Mykyta Poturai
  2025-05-28  9:12 ` [PATCH v11 2/7] iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling Mykyta Poturai
@ 2025-05-28  9:12 ` Mykyta Poturai
  2025-06-02  8:09   ` Jan Beulich
  2025-05-28  9:12 ` [PATCH v11 3/7] xen/arm: smmuv2: Add PCI devices support for SMMUv2 Mykyta Poturai
                   ` (4 subsequent siblings)
  6 siblings, 1 reply; 22+ messages in thread
From: Mykyta Poturai @ 2025-05-28  9:12 UTC (permalink / raw)
  To: xen-devel@lists.xenproject.org
  Cc: Oleksandr Tyshchenko, Stefano Stabellini, Julien Grall,
	Bertrand Marquis, Michal Orzel, Volodymyr Babchuk, Jan Beulich,
	Roger Pau Monné, Stewart Hildebrand, Mykyta Poturai

From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

The main purpose of this patch is to add a way to register PCI device
(which is behind the IOMMU) using the generic PCI-IOMMU DT bindings [1]
before assigning that device to a domain.

This behaves similarly to the existing iommu_add_dt_device API, except it
handles PCI devices, and it is to be invoked from the add_device hook in the
SMMU driver.

The function dt_map_id to translate an ID through a downstream mapping
(which is also suitable for mapping Requester ID) was borrowed from Linux
(v5.10-rc6) and updated according to the Xen code base.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
Regarding pci_for_each_dma_alias question: getting host bridge node
directly seems like a simpler solution with the same result. AFAIU
with pci_for_each_dma_alias in linux we would arrive to the host brige
node anyway, but also try to call dt_map_id for each device along the
way. I am not sure why exactly it is done this way in linux, as
according to the pci-iommu.txt, iommu-map node can only be present in
the PCI root.

v10-v11:
* add Stefano's RB

v9->v10:
* move iommu_add_pci_sidebands_ids to arm/iommu.c
* replace be32_to_cpup with be32_to_cpu

v8->v9:
* replace DT_NO_IOMMU with 1
* guard iommu_add_pci_sideband_ids with CONFIG_ARM

v7->v8:
* ENOSYS->EOPNOTSUPP
* move iommu_add_pci_sideband_ids to iommu.c to fix x86 build
* simplify iommu_add_pci_sideband_ids
* add docstrings to iommu_add_{dt_}pci_sideband_ids

v6->v7:
* put iommu_add_pci_sideband_ids under ifdef
* remove ifdef CONFIG_APCI
* style: add newline for symmetry

v5->v6:
* pass ops to iommu_dt_xlate()

v4->v5:
* style: add newlines after variable declarations and before return in iommu.h
* drop device_is_protected() check in iommu_add_dt_pci_sideband_ids()
* rebase on top of ("dynamic node programming using overlay dtbo") series
* fix typo in commit message
* remove #ifdef around dt_map_id() prototype
* move dt_map_id() to xen/common/device_tree.c
* add function name in error prints
* use dprintk for debug prints
* use GENMASK and #include <xen/bitops.h>
* fix typo in comment
* remove unnecessary (int) cast in loop condition
* assign *id_out and return success in case of no translation in dt_map_id()
* don't initialize local variable unnecessarily
* return error in case of ACPI/no DT in iommu_add_{dt_}pci_sideband_ids()

v3->v4:
* wrap #include <asm/acpi.h> and if ( acpi_disabled ) in #ifdef CONFIG_ACPI
* fix Michal's remarks about style, parenthesis, and print formats
* remove !ops->dt_xlate check since it is already in iommu_dt_xlate helper
* rename s/iommu_dt_pci_map_id/dt_map_id/ because it is generic, not specific
  to iommu
* update commit description

v2->v3:
* new patch title (was: iommu/arm: Introduce iommu_add_dt_pci_device API)
* renamed function
  from: iommu_add_dt_pci_device
  to: iommu_add_dt_pci_sideband_ids
* removed stale ops->add_device check
* iommu.h: add empty stub iommu_add_dt_pci_sideband_ids for !HAS_DEVICE_TREE
* iommu.h: add iommu_add_pci_sideband_ids helper
* iommu.h: don't wrap prototype in #ifdef CONFIG_HAS_PCI
* s/iommu_fwspec_free(pci_to_dev(pdev))/iommu_fwspec_free(dev)/

v1->v2:
* remove extra devfn parameter since pdev fully describes the device
* remove ops->add_device() call from iommu_add_dt_pci_device(). Instead, rely on
  the existing iommu call in iommu_add_device().
* move the ops->add_device and ops->dt_xlate checks earlier

downstream->v1:
* rebase
* add const qualifier to struct dt_device_node *np arg in dt_map_id()
* add const qualifier to struct dt_device_node *np declaration in iommu_add_pci_device()
* use stdint.h types instead of u8/u32/etc...
* rename functions:
  s/dt_iommu_xlate/iommu_dt_xlate/
  s/dt_map_id/iommu_dt_pci_map_id/
  s/iommu_add_pci_device/iommu_add_dt_pci_device/
* add device_is_protected check in iommu_add_dt_pci_device
* wrap prototypes in CONFIG_HAS_PCI

(cherry picked from commit 734e3bf6ee77e7947667ab8fa96c25b349c2e1da from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
---
 xen/arch/arm/include/asm/iommu.h      | 15 +++++
 xen/common/device-tree/device-tree.c  | 91 +++++++++++++++++++++++++++
 xen/drivers/passthrough/arm/iommu.c   | 13 ++++
 xen/drivers/passthrough/device_tree.c | 42 +++++++++++++
 xen/include/xen/device_tree.h         | 23 +++++++
 xen/include/xen/iommu.h               | 21 ++++++-
 6 files changed, 204 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/include/asm/iommu.h b/xen/arch/arm/include/asm/iommu.h
index d57bd8a38c..ad15477e24 100644
--- a/xen/arch/arm/include/asm/iommu.h
+++ b/xen/arch/arm/include/asm/iommu.h
@@ -34,6 +34,21 @@ int __must_check arm_iommu_unmap_page(struct domain *d, dfn_t dfn,
                                       unsigned int order,
                                       unsigned int *flush_flags);
 
+/*
+ * This function is not strictly ARM-specific, but it is only used by ARM
+ * as of now. So put it here to avoid creating dead code on other
+ * architectures. When usage is extended to other architectures, it should
+ * be moved to the generic header.
+ *
+ *
+ * Fills out the device's IOMMU fwspec with IOMMU ids.
+ *
+ * Return values:
+ *  0 : iommu_fwspec is filled out successfully.
+ * <0 : error while filling out the iommu_fwspec.
+ * >0 : IOMMU is not enabled/present or device is not connected to it.
+ */
+int iommu_add_pci_sideband_ids(struct pci_dev *pdev);
 #endif /* __ARCH_ARM_IOMMU_H__ */
 
 /*
diff --git a/xen/common/device-tree/device-tree.c b/xen/common/device-tree/device-tree.c
index 886e6c7712..7bede20fa6 100644
--- a/xen/common/device-tree/device-tree.c
+++ b/xen/common/device-tree/device-tree.c
@@ -7,6 +7,7 @@
  * benh@kernel.crashing.org
  */
 
+#include <xen/bitops.h>
 #include <xen/types.h>
 #include <xen/init.h>
 #include <xen/guest_access.h>
@@ -2221,6 +2222,96 @@ int dt_get_pci_domain_nr(struct dt_device_node *node)
     return (u16)domain;
 }
 
+int dt_map_id(const struct dt_device_node *np, uint32_t id,
+              const char *map_name, const char *map_mask_name,
+              struct dt_device_node **target, uint32_t *id_out)
+{
+    uint32_t map_mask, masked_id, map_len;
+    const __be32 *map = NULL;
+
+    if ( !np || !map_name || (!target && !id_out) )
+        return -EINVAL;
+
+    map = dt_get_property(np, map_name, &map_len);
+    if ( !map )
+    {
+        if ( target )
+            return -ENODEV;
+
+        /* Otherwise, no map implies no translation */
+        *id_out = id;
+        return 0;
+    }
+
+    if ( !map_len || (map_len % (4 * sizeof(*map))) )
+    {
+        printk(XENLOG_ERR "%s(): %s: Error: Bad %s length: %u\n", __func__,
+               np->full_name, map_name, map_len);
+        return -EINVAL;
+    }
+
+    /* The default is to select all bits. */
+    map_mask = GENMASK(31, 0);
+
+    /*
+     * Can be overridden by "{iommu,msi}-map-mask" property.
+     * If dt_property_read_u32() fails, the default is used.
+     */
+    if ( map_mask_name )
+        dt_property_read_u32(np, map_mask_name, &map_mask);
+
+    masked_id = map_mask & id;
+    for ( ; map_len > 0; map_len -= 4 * sizeof(*map), map += 4 )
+    {
+        struct dt_device_node *phandle_node;
+        uint32_t id_base = be32_to_cpu(*(map + 0));
+        uint32_t phandle = be32_to_cpu(*(map + 1));
+        uint32_t out_base = be32_to_cpu(*(map + 2));
+        uint32_t id_len = be32_to_cpu(*(map + 3));
+
+        if ( id_base & ~map_mask )
+        {
+            printk(XENLOG_ERR "%s(): %s: Invalid %s translation - %s-mask (0x%"PRIx32") ignores id-base (0x%"PRIx32")\n",
+                   __func__, np->full_name, map_name, map_name, map_mask,
+                   id_base);
+            return -EFAULT;
+        }
+
+        if ( (masked_id < id_base) || (masked_id >= (id_base + id_len)) )
+            continue;
+
+        phandle_node = dt_find_node_by_phandle(phandle);
+        if ( !phandle_node )
+            return -ENODEV;
+
+        if ( target )
+        {
+            if ( !*target )
+                *target = phandle_node;
+
+            if ( *target != phandle_node )
+                continue;
+        }
+
+        if ( id_out )
+            *id_out = masked_id - id_base + out_base;
+
+        dprintk(XENLOG_DEBUG, "%s: %s, using mask %08"PRIx32", id-base: %08"PRIx32", out-base: %08"PRIx32", length: %08"PRIx32", id: %08"PRIx32" -> %08"PRIx32"\n",
+               np->full_name, map_name, map_mask, id_base, out_base, id_len, id,
+               masked_id - id_base + out_base);
+        return 0;
+    }
+
+    dprintk(XENLOG_DEBUG, "%s: no %s translation for id 0x%"PRIx32" on %s\n",
+           np->full_name, map_name, id,
+           (target && *target) ? (*target)->full_name : NULL);
+
+    if ( id_out )
+        *id_out = id;
+
+    return 0;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c
index fc453180f0..100545e23f 100644
--- a/xen/drivers/passthrough/arm/iommu.c
+++ b/xen/drivers/passthrough/arm/iommu.c
@@ -15,6 +15,7 @@
  * GNU General Public License for more details.
  */
 
+#include <xen/acpi.h>
 #include <xen/device_tree.h>
 #include <xen/iommu.h>
 #include <xen/lib.h>
@@ -151,3 +152,15 @@ bool arch_iommu_use_permitted(const struct domain *d)
 {
     return true;
 }
+
+int iommu_add_pci_sideband_ids(struct pci_dev *pdev)
+{
+    int ret = -EOPNOTSUPP;
+
+#ifdef CONFIG_HAS_PCI
+    if ( acpi_disabled )
+        ret = iommu_add_dt_pci_sideband_ids(pdev);
+#endif
+
+    return ret;
+}
diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
index 4a1971c3fc..37e1437b65 100644
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -161,6 +161,48 @@ static int iommu_dt_xlate(struct device *dev,
     return ops->dt_xlate(dev, iommu_spec);
 }
 
+#ifdef CONFIG_HAS_PCI
+int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+    struct dt_phandle_args iommu_spec = { .args_count = 1 };
+    struct device *dev = pci_to_dev(pdev);
+    const struct dt_device_node *np;
+    int rc;
+
+    if ( !iommu_enabled )
+        return 1;
+
+    if ( !ops )
+        return -EINVAL;
+
+    if ( dev_iommu_fwspec_get(dev) )
+        return -EEXIST;
+
+    np = pci_find_host_bridge_node(pdev);
+    if ( !np )
+        return -ENODEV;
+
+    /*
+     * According to the Documentation/devicetree/bindings/pci/pci-iommu.txt
+     * from Linux.
+     */
+    rc = dt_map_id(np, PCI_BDF(pdev->bus, pdev->devfn), "iommu-map",
+                   "iommu-map-mask", &iommu_spec.np, iommu_spec.args);
+    if ( rc )
+        return (rc == -ENODEV) ? 1 : rc;
+
+    rc = iommu_dt_xlate(dev, &iommu_spec, ops);
+    if ( rc < 0 )
+    {
+        iommu_fwspec_free(dev);
+        return -EINVAL;
+    }
+
+    return rc;
+}
+#endif /* CONFIG_HAS_PCI */
+
 int iommu_remove_dt_device(struct dt_device_node *np)
 {
     const struct iommu_ops *ops = iommu_get_ops();
diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h
index 6dc1fb5159..3de7ff9085 100644
--- a/xen/include/xen/device_tree.h
+++ b/xen/include/xen/device_tree.h
@@ -947,6 +947,29 @@ int dt_count_phandle_with_args(const struct dt_device_node *np,
  */
 int dt_get_pci_domain_nr(struct dt_device_node *node);
 
+/**
+ * dt_map_id - Translate an ID through a downstream mapping.
+ * @np: root complex device node.
+ * @id: device ID to map.
+ * @map_name: property name of the map to use.
+ * @map_mask_name: optional property name of the mask to use.
+ * @target: optional pointer to a target device node.
+ * @id_out: optional pointer to receive the translated ID.
+ *
+ * Given a device ID, look up the appropriate implementation-defined
+ * platform ID and/or the target device which receives transactions on that
+ * ID, as per the "iommu-map" and "msi-map" bindings. Either of @target or
+ * @id_out may be NULL if only the other is required. If @target points to
+ * a non-NULL device node pointer, only entries targeting that node will be
+ * matched; if it points to a NULL value, it will receive the device node of
+ * the first matching target phandle, with a reference held.
+ *
+ * Return: 0 on success or a standard error code on failure.
+ */
+int dt_map_id(const struct dt_device_node *np, uint32_t id,
+              const char *map_name, const char *map_mask_name,
+              struct dt_device_node **target, uint32_t *id_out);
+
 struct dt_device_node *dt_find_node_by_phandle(dt_phandle handle);
 
 #ifdef CONFIG_DEVICE_TREE_DEBUG
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 832775754b..ebfada1d88 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -241,7 +241,8 @@ int iommu_dt_domain_init(struct domain *d);
 int iommu_release_dt_devices(struct domain *d);
 
 /*
- * Helper to add master device to the IOMMU using generic IOMMU DT bindings.
+ * Helpers to add master device to the IOMMU using generic (PCI-)IOMMU
+ * DT bindings.
  *
  * Return values:
  *  0 : device is protected by an IOMMU
@@ -251,6 +252,19 @@ int iommu_release_dt_devices(struct domain *d);
  */
 int iommu_add_dt_device(struct dt_device_node *np);
 
+/*
+ * Fills out the device's IOMMU fwspec with IOMMU ids from the DT.
+ * Ids are specified in the iommu-map property in the host bridge node.
+ * More information on the iommu-map property format can be found in
+ * Documentation/devicetree/bindings/pci/pci-iommu.txt from Linux.
+ *
+ * Return values:
+ *  0 : iommu_fwspec is filled out successfully.
+ * <0 : error while filling out the iommu_fwspec.
+ * >0 : IOMMU is not enabled/present or device is not connected to it.
+ */
+int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev);
+
 int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
                        XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl);
 
@@ -286,6 +300,11 @@ static inline int iommu_release_dt_devices(struct domain *d)
     return 0;
 }
 
+static inline int iommu_add_dt_pci_sideband_ids(struct pci_dev *pdev)
+{
+    return -EOPNOTSUPP;
+}
+
 #endif /* HAS_PASSTHROUGH */
 
 #endif /* HAS_DEVICE_TREE */
-- 
2.34.1


^ permalink raw reply related	[flat|nested] 22+ messages in thread

* [PATCH v11 3/7] xen/arm: smmuv2: Add PCI devices support for SMMUv2
  2025-05-28  9:12 [PATCH v11 0/7] SMMU handling for PCIe Passthrough on ARM Mykyta Poturai
  2025-05-28  9:12 ` [PATCH v11 2/7] iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling Mykyta Poturai
  2025-05-28  9:12 ` [PATCH v11 1/7] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API Mykyta Poturai
@ 2025-05-28  9:12 ` Mykyta Poturai
  2025-05-29  0:53   ` Stefano Stabellini
  2025-05-28  9:12 ` [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3 Mykyta Poturai
                   ` (3 subsequent siblings)
  6 siblings, 1 reply; 22+ messages in thread
From: Mykyta Poturai @ 2025-05-28  9:12 UTC (permalink / raw)
  To: xen-devel@lists.xenproject.org
  Cc: Oleksandr Andrushchenko, Julien Grall, Rahul Singh,
	Stefano Stabellini, Bertrand Marquis, Michal Orzel,
	Volodymyr Babchuk, Oleksandr Tyshchenko, Stewart Hildebrand,
	Mykyta Poturai

From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>

Implement support for PCI devices in the SMMU driver. Make arm_smmu_master
structure to hold a pointer to the device to allow it to hold PCI devices.
Trigger iommu-map parsing when new PCI device is added. Add checks to
assign/deassign functions to ensure PCI devices are handled correctly.
Implement basic quarantining.

All pci devices are automatically assigned to hardware domain if it exists
to ensure it can probe them.

TODO:
Implement scratch page quarantining support.

Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
v10-v11:
* remove unused code
* remove unnecessary blocks

v9->v10:
* remove unused code
* return on error in arm_smmu_dt_add_device_generic

v8->v9:
* no change

v7->v8:
* no change

v6->v7:
* use d->pci_lock in arm_smmu_assign_dev()
* remove !is_hardware_domain and pdev->domain == d checks in assign to
  support future dom0less use case when dom0 is using vPCI
* remove stale todo in dev_get_dev_node
* don't print ""
* remove redundant dt_device_is_protected check
* remove assign/deassing prints
* change assign logic to remove reassign reimplementation
* check if pdev->domain exists before assigning to it
* explain pdev->devfn check
* make reassign check stricter and update comment

v5->v6:
* check for hardware_domain == NULL (dom0less test case)
* locking: assign pdev->domain before list_add()

v4->v5:
* assign device to pdev->domain (usually dom0) by default in add_device() hook
* deassign from hwdom
* rebase on top of ("dynamic node programming using overlay dtbo") series
* remove TODO in comment about device prints
* add TODO regarding locking
* fixup after dropping ("xen/arm: Move is_protected flag to struct device")

v3->v4:
* add new device_is_protected check in add_device hook to match SMMUv3 and
  IPMMU-VMSA drivers

v2->v3:
* invoke iommu_add_pci_sideband_ids() from add_device hook

v1->v2:
* ignore add_device/assign_device/reassign_device calls for phantom functions
  (i.e. devfn != pdev->devfn)

downstream->v1:
* wrap unused function in #ifdef 0
* remove the remove_device() stub since it was submitted separately to the list
  [XEN][PATCH v6 12/19] xen/smmu: Add remove_device callback for smmu_iommu ops
  https://lists.xenproject.org/archives/html/xen-devel/2023-05/msg00204.html
* arm_smmu_(de)assign_dev: return error instead of crashing system
* update condition in arm_smmu_reassign_dev
* style fixup
* add && !is_hardware_domain(d) into condition in arm_smmu_assign_dev()

(cherry picked from commit 0c11a7f65f044c26d87d1e27ac6283ef1f9cfb7a from
 the downstream branch spider-master from
 https://github.com/xen-troops/xen.git)
---
 xen/drivers/passthrough/arm/smmu.c | 246 +++++++++++++++++------------
 1 file changed, 146 insertions(+), 100 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
index 0f8d47dc98..307c2f6837 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -120,11 +120,21 @@ enum irqreturn {
 
 typedef enum irqreturn irqreturn_t;
 
-/* Device logger functions
- * TODO: Handle PCI
- */
-#define dev_print(dev, lvl, fmt, ...)						\
-	 printk(lvl "smmu: %s: " fmt, dt_node_full_name(dev_to_dt(dev)), ## __VA_ARGS__)
+/* Device logger functions */
+#ifndef CONFIG_HAS_PCI
+#define dev_print(dev, lvl, fmt, ...)    \
+    printk(lvl "smmu: %s: " fmt, dev_name(dev), ## __VA_ARGS__)
+#else
+#define dev_print(dev, lvl, fmt, ...) ({                                \
+    if ( !dev_is_pci((dev)) )                                           \
+        printk(lvl "smmu: %s: " fmt, dev_name((dev)), ## __VA_ARGS__);  \
+    else                                                                \
+    {                                                                   \
+        struct pci_dev *pdev = dev_to_pci((dev));                       \
+        printk(lvl "smmu: %pp: " fmt, &pdev->sbdf, ## __VA_ARGS__);     \
+    }                                                                   \
+})
+#endif
 
 #define dev_dbg(dev, fmt, ...) dev_print(dev, XENLOG_DEBUG, fmt, ## __VA_ARGS__)
 #define dev_notice(dev, fmt, ...) dev_print(dev, XENLOG_INFO, fmt, ## __VA_ARGS__)
@@ -172,20 +182,6 @@ static void __iomem *devm_ioremap_resource(struct device *dev,
 #define IOMMU_FAULT_READ	0
 #define IOMMU_FAULT_WRITE	1
 
-/*
- * Xen: PCI functions
- * TODO: It should be implemented when PCI will be supported
- */
-#define to_pci_dev(dev)	(NULL)
-static inline int pci_for_each_dma_alias(struct pci_dev *pdev,
-					 int (*fn) (struct pci_dev *pdev,
-						    u16 alias, void *data),
-					 void *data)
-{
-	BUG();
-	return 0;
-}
-
 /* Xen: misc */
 #define PHYS_MASK_SHIFT		PADDR_BITS
 
@@ -619,7 +615,7 @@ struct arm_smmu_master_cfg {
 	for (i = 0; idx = (cfg)->smendx[i], (i) < (num); ++(i))
 
 struct arm_smmu_master {
-	struct device_node		*of_node;
+	struct device			*dev;
 	struct rb_node			node;
 	struct arm_smmu_master_cfg	cfg;
 };
@@ -711,7 +707,7 @@ arm_smmu_get_fwspec(struct arm_smmu_master_cfg *cfg)
 {
 	struct arm_smmu_master *master = container_of(cfg,
 			                                      struct arm_smmu_master, cfg);
-	return dev_iommu_fwspec_get(&master->of_node->dev);
+	return dev_iommu_fwspec_get(master->dev);
 }
 
 static void parse_driver_options(struct arm_smmu_device *smmu)
@@ -730,21 +726,11 @@ static void parse_driver_options(struct arm_smmu_device *smmu)
 
 static struct device_node *dev_get_dev_node(struct device *dev)
 {
-#if 0 /* Xen: TODO: Add support for PCI */
-	if (dev_is_pci(dev)) {
-		struct pci_bus *bus = to_pci_dev(dev)->bus;
-
-		while (!pci_is_root_bus(bus))
-			bus = bus->parent;
-		return bus->bridge->parent->of_node;
-	}
-#endif
-
 	return dev->of_node;
 }
 
 static struct arm_smmu_master *find_smmu_master(struct arm_smmu_device *smmu,
-						struct device_node *dev_node)
+						struct device *dev)
 {
 	struct rb_node *node = smmu->masters.rb_node;
 
@@ -753,9 +739,9 @@ static struct arm_smmu_master *find_smmu_master(struct arm_smmu_device *smmu,
 
 		master = container_of(node, struct arm_smmu_master, node);
 
-		if (dev_node < master->of_node)
+		if (dev < master->dev)
 			node = node->rb_left;
-		else if (dev_node > master->of_node)
+		else if (dev > master->dev)
 			node = node->rb_right;
 		else
 			return master;
@@ -790,9 +776,9 @@ static int insert_smmu_master(struct arm_smmu_device *smmu,
 			= container_of(*new, struct arm_smmu_master, node);
 
 		parent = *new;
-		if (master->of_node < this->of_node)
+		if (master->dev < this->dev)
 			new = &((*new)->rb_left);
-		else if (master->of_node > this->of_node)
+		else if (master->dev > this->dev)
 			new = &((*new)->rb_right);
 		else
 			return -EEXIST;
@@ -824,28 +810,30 @@ static int arm_smmu_dt_add_device_legacy(struct arm_smmu_device *smmu,
 	struct arm_smmu_master *master;
 	struct device_node *dev_node = dev_get_dev_node(dev);
 
-	master = find_smmu_master(smmu, dev_node);
+	master = find_smmu_master(smmu, dev);
 	if (master) {
 		dev_err(dev,
-			"rejecting multiple registrations for master device %s\n",
-			dev_node->name);
+			"rejecting multiple registrations for master device\n");
 		return -EBUSY;
 	}
 
 	master = devm_kzalloc(dev, sizeof(*master), GFP_KERNEL);
 	if (!master)
 		return -ENOMEM;
-	master->of_node = dev_node;
+	master->dev = dev;
 
-	/* Xen: Let Xen know that the device is protected by an SMMU */
-	dt_device_set_protected(dev_node);
+	if ( !dev_is_pci(dev) )
+	{
+		/* Xen: Let Xen know that the device is protected by an SMMU */
+		dt_device_set_protected(dev_node);
+	}
 
 	for (i = 0; i < fwspec->num_ids; ++i) {
 		if (!(smmu->features & ARM_SMMU_FEAT_STREAM_MATCH) &&
 		     (fwspec->ids[i] >= smmu->num_mapping_groups)) {
 			dev_err(dev,
-				"stream ID for master device %s greater than maximum allowed (%d)\n",
-				dev_node->name, smmu->num_mapping_groups);
+				"SMMU stream ID %d is greater than maximum allowed (%d)\n",
+				fwspec->ids[i], smmu->num_mapping_groups);
 			return -ERANGE;
 		}
 		master->cfg.smendx[i] = INVALID_SMENDX;
@@ -860,7 +848,7 @@ static int arm_smmu_dt_remove_device_legacy(struct arm_smmu_device *smmu,
 	struct device_node *dev_node = dev_get_dev_node(dev);
 	int ret;
 
-	master = find_smmu_master(smmu, dev_node);
+	master = find_smmu_master(smmu, dev);
 	if (master == NULL) {
 		dev_err(dev,
 			"No registrations found for master device %s\n",
@@ -872,8 +860,9 @@ static int arm_smmu_dt_remove_device_legacy(struct arm_smmu_device *smmu,
 	if (ret)
 		return ret;
 
-	/* Protected by dt_host_lock and dtdevs_lock as caller holds these locks. */
-	dev_node->is_protected = false;
+	if ( !dev_is_pci(dev) )
+		/* Protected by dt_host_lock and dtdevs_lock as caller holds these locks. */
+		dev_node->is_protected = false;
 
 	kfree(master);
 	return 0;
@@ -902,6 +891,12 @@ static int register_smmu_master(struct arm_smmu_device *smmu,
 					     fwspec);
 }
 
+/* Forward declaration */
+static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
+			       struct device *dev, u32 flag);
+static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
+				 struct device *dev);
+
 /*
  * The driver which supports generic IOMMU DT bindings must have this
  * callback implemented.
@@ -926,6 +921,25 @@ static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev)
 {
 	struct arm_smmu_device *smmu;
 	struct iommu_fwspec *fwspec;
+	int ret;
+
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+		int ret;
+
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		ret = iommu_add_pci_sideband_ids(pdev);
+		if ( ret < 0 ) {
+			iommu_fwspec_free(dev);
+			return ret;
+		}
+	}
+#endif
 
 	fwspec = dev_iommu_fwspec_get(dev);
 	if (fwspec == NULL)
@@ -935,7 +949,25 @@ static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev)
 	if (smmu == NULL)
 		return -ENXIO;
 
-	return arm_smmu_dt_add_device_legacy(smmu, dev, fwspec);
+	ret = arm_smmu_dt_add_device_legacy(smmu, dev, fwspec);
+	if ( ret )
+		return ret;
+
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/*
+		 * During PHYSDEVOP_pci_device_add, Xen does not assign the
+		 * device, so we must do it here.
+		 */
+		if ( pdev->domain )
+			ret = arm_smmu_assign_dev(pdev->domain, devfn, dev, 0);
+	}
+#endif
+
+	return ret;
 }
 
 static int arm_smmu_dt_xlate_generic(struct device *dev,
@@ -958,11 +990,10 @@ static struct arm_smmu_device *find_smmu_for_device(struct device *dev)
 {
 	struct arm_smmu_device *smmu;
 	struct arm_smmu_master *master = NULL;
-	struct device_node *dev_node = dev_get_dev_node(dev);
 
 	spin_lock(&arm_smmu_devices_lock);
 	list_for_each_entry(smmu, &arm_smmu_devices, list) {
-		master = find_smmu_master(smmu, dev_node);
+		master = find_smmu_master(smmu, dev);
 		if (master)
 			break;
 	}
@@ -2054,65 +2085,26 @@ static bool arm_smmu_capable(enum iommu_cap cap)
 }
 #endif
 
-static int __arm_smmu_get_pci_sid(struct pci_dev *pdev, u16 alias, void *data)
-{
-	*((u16 *)data) = alias;
-	return 0; /* Continue walking */
-}
-
-static void __arm_smmu_release_pci_iommudata(void *data)
-{
-	kfree(data);
-}
-
 static int arm_smmu_add_device(struct device *dev)
 {
 	struct arm_smmu_device *smmu;
+	struct arm_smmu_master *master;
 	struct arm_smmu_master_cfg *cfg;
 	struct iommu_group *group;
 	void (*releasefn)(void *data) = NULL;
-	int ret;
 
 	smmu = find_smmu_for_device(dev);
 	if (!smmu)
 		return -ENODEV;
 
-	if (dev_is_pci(dev)) {
-		struct pci_dev *pdev = to_pci_dev(dev);
-		struct iommu_fwspec *fwspec;
-
-		cfg = kzalloc(sizeof(*cfg), GFP_KERNEL);
-		if (!cfg) {
-			return -ENOMEM;
-		}
-
-		ret = iommu_fwspec_init(dev, smmu->dev);
-		if (ret) {
-			kfree(cfg);
-			return ret;
-		}
-		fwspec = dev_iommu_fwspec_get(dev);
-
-		/*
-		 * Assume Stream ID == Requester ID for now.
-		 * We need a way to describe the ID mappings in FDT.
-		 */
-		pci_for_each_dma_alias(pdev, __arm_smmu_get_pci_sid,
-				       &fwspec->ids[0]);
-		releasefn = __arm_smmu_release_pci_iommudata;
-		cfg->smmu = smmu;
-	} else {
-		struct arm_smmu_master *master;
-
-		master = find_smmu_master(smmu, dev->of_node);
-		if (!master) {
-			return -ENODEV;
-		}
-
-		cfg = &master->cfg;
-		cfg->smmu = smmu;
+	master = find_smmu_master(smmu, dev);
+	if (!master) {
+		return -ENODEV;
 	}
 
+	cfg = &master->cfg;
+	cfg->smmu = smmu;
+
 	group = iommu_group_alloc();
 	if (IS_ERR(group)) {
 		dev_err(dev, "Failed to allocate IOMMU group\n");
@@ -2772,6 +2764,42 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
 			return -ENOMEM;
 	}
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		ASSERT(pcidevs_locked());
+
+		write_lock(&pdev->domain->pci_lock);
+		list_del(&pdev->domain_list);
+		write_unlock(&pdev->domain->pci_lock);
+
+		pdev->domain = d;
+
+		write_lock(&d->pci_lock);
+		list_add(&pdev->domain_list, &d->pdev_list);
+		write_unlock(&d->pci_lock);
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+		{
+			struct iommu_domain *domain = dev_iommu_domain(dev);
+			if ( !iommu_quarantine )
+				return 0;
+
+			if ( domain && domain->priv )
+				arm_smmu_deassign_dev(domain->priv->cfg.domain, devfn, dev);
+
+			return 0;
+		}
+	}
+#endif
+
 	if (!dev_iommu_group(dev)) {
 		ret = arm_smmu_add_device(dev);
 		if (ret)
@@ -2821,11 +2849,27 @@ out:
 	return ret;
 }
 
-static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
+static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
+				 struct device *dev)
 {
 	struct iommu_domain *domain = dev_iommu_domain(dev);
 	struct arm_smmu_xen_domain *xen_domain;
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+			return 0;
+	}
+#endif
+
 	xen_domain = dom_iommu(d)->arch.priv;
 
 	if (!domain || domain->priv->cfg.domain != d) {
@@ -2852,14 +2896,16 @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t,
 {
 	int ret = 0;
 
-	/* Don't allow remapping on other domain than hwdom */
-	if ( t && !is_hardware_domain(t) )
+	/* Don't allow remapping on other domain than hwdom
+	 * or dom_io for PCI devices
+	 */
+	if ( t && !is_hardware_domain(t) && (t != dom_io || !dev_is_pci(dev)) )
 		return -EPERM;
 
 	if (t == s)
 		return 0;
 
-	ret = arm_smmu_deassign_dev(s, dev);
+	ret = arm_smmu_deassign_dev(s, devfn, dev);
 	if (ret)
 		return ret;
 
-- 
2.34.1


^ permalink raw reply related	[flat|nested] 22+ messages in thread

* [PATCH v11 5/7] xen/arm: Fix mapping for PCI bridge mmio region
  2025-05-28  9:12 [PATCH v11 0/7] SMMU handling for PCIe Passthrough on ARM Mykyta Poturai
                   ` (3 preceding siblings ...)
  2025-05-28  9:12 ` [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3 Mykyta Poturai
@ 2025-05-28  9:12 ` Mykyta Poturai
  2025-05-28  9:12 ` [PATCH v11 7/7] xen/arm: Map ITS doorbell register to IOMMU page tables Mykyta Poturai
  2025-05-28  9:12 ` [PATCH v11 6/7] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no Mykyta Poturai
  6 siblings, 0 replies; 22+ messages in thread
From: Mykyta Poturai @ 2025-05-28  9:12 UTC (permalink / raw)
  To: xen-devel@lists.xenproject.org
  Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
	Michal Orzel, Volodymyr Babchuk, Stewart Hildebrand, Julien Grall

From: Rahul Singh <rahul.singh@arm.com>

Current code skip the mapping for PCI bridge MMIO region to dom0 when
pci_passthrough_enabled flag is set. Mapping should be skip when
has_vpci(d) is enabled for the domain, as we need to skip the mapping
only when VPCI handler are registered for ECAM.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Acked-by: Julien Grall <jgrall@amazon.com>
---
This patch was originally picked up from [1]

v10->v11:
* no change

v9->v10:
* no change

v8->v9:
* no change

v7->v8:
* no change

v6->v7:
* add Julien's A-b

v5->v6:
* drop unrelated change in xen/arch/arm/domain_build.c:handle_linux_pci_domain()

v4->v5:
* new patch

changes since picking up from [1]:
* rebase on top of "dynamic node programming using overlay dtbo" series
* replace !is_pci_passthrough_enabled() check with !IS_ENABLED(CONFIG_HAS_PCI)
  instead of removing

[1] https://lists.xenproject.org/archives/html/xen-devel/2023-07/msg00483.html
---
 xen/arch/arm/device.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/arch/arm/device.c b/xen/arch/arm/device.c
index 5e1c1cc326..11523750ae 100644
--- a/xen/arch/arm/device.c
+++ b/xen/arch/arm/device.c
@@ -268,7 +268,7 @@ int handle_device(struct domain *d, struct dt_device_node *dev, p2m_type_t p2mt,
         .d = d,
         .p2mt = p2mt,
         .skip_mapping = !own_device ||
-                        (is_pci_passthrough_enabled() &&
+                        (has_vpci(d) &&
                         (device_get_class(dev) == DEVICE_PCI_HOSTBRIDGE)),
         .iomem_ranges = iomem_ranges,
         .irq_ranges = irq_ranges
-- 
2.34.1


^ permalink raw reply related	[flat|nested] 22+ messages in thread

* [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3
  2025-05-28  9:12 [PATCH v11 0/7] SMMU handling for PCIe Passthrough on ARM Mykyta Poturai
                   ` (2 preceding siblings ...)
  2025-05-28  9:12 ` [PATCH v11 3/7] xen/arm: smmuv2: Add PCI devices support for SMMUv2 Mykyta Poturai
@ 2025-05-28  9:12 ` Mykyta Poturai
  2025-05-29  0:56   ` Stefano Stabellini
                     ` (2 more replies)
  2025-05-28  9:12 ` [PATCH v11 5/7] xen/arm: Fix mapping for PCI bridge mmio region Mykyta Poturai
                   ` (2 subsequent siblings)
  6 siblings, 3 replies; 22+ messages in thread
From: Mykyta Poturai @ 2025-05-28  9:12 UTC (permalink / raw)
  To: xen-devel@lists.xenproject.org
  Cc: Rahul Singh, Bertrand Marquis, Stefano Stabellini, Julien Grall,
	Michal Orzel, Volodymyr Babchuk, Stewart Hildebrand,
	Mykyta Poturai

From: Rahul Singh <rahul.singh@arm.com>

Implement support for PCI devices in the SMMU driver. Trigger iommu-map
parsing when new PCI device is added. Add checks to assign/deassign
functions to ensure PCI devices are handled correctly. Implement basic
quarantining.

All pci devices are automatically assigned to hardware domain if it exists
to ensure it can probe them.

TODO:
Implement scratch page quarantining support.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
v10->v11:
* no changes

v9->v10:
* return if iommu_add_pci_sidband_ids fails

v8->v9:
* no change

v7->v8:
* no change

v6->v7:
* address TODO: use d->pci_lock in arm_smmu_assign_dev()
* remove !is_hardware_domain and pdev->domain == d checks in assign to
  support future dom0less use case when dom0 is using vPCI
* check if pdev->domain exists before assigning to it
* don't print ""
* change assign logic to remove reassign reimplementation
* explain pdev->devfn check
* make reassign check stricter and update comment

v5->v6:
* check for hardware_domain == NULL (dom0less test case)
* locking: assign pdev->domain before list_add()

v4->v5:
* deassign from hwdom
* add TODO regarding locking
* fixup after dropping ("xen/arm: Move is_protected flag to struct device")

v3->v4:
* no change

v2->v3:
* rebase
* invoke iommu_add_pci_sideband_ids() from add_device hook

v1->v2:
* ignore add_device/assign_device/reassign_device calls for phantom functions
  (i.e. devfn != pdev->devfn)

downstream->v1:
* rebase
* move 2 replacements of s/dt_device_set_protected(dev_to_dt(dev))/device_set_protected(dev)/
  from this commit to ("xen/arm: Move is_protected flag to struct device")
  so as to not break ability to bisect
* adjust patch title (remove stray space)
* arm_smmu_(de)assign_dev: return error instead of crashing system
* remove arm_smmu_remove_device() stub
* update condition in arm_smmu_reassign_dev
* style fixup

(cherry picked from commit 7ed6c3ab250d899fe6e893a514278e406a2893e8 from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
---
 xen/drivers/passthrough/arm/smmu-v3.c | 119 +++++++++++++++++++++++---
 1 file changed, 108 insertions(+), 11 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
index df16235057..a3bbfda993 100644
--- a/xen/drivers/passthrough/arm/smmu-v3.c
+++ b/xen/drivers/passthrough/arm/smmu-v3.c
@@ -1469,14 +1469,37 @@ static bool arm_smmu_sid_in_range(struct arm_smmu_device *smmu, u32 sid)
 }
 /* Forward declaration */
 static struct arm_smmu_device *arm_smmu_get_by_dev(const struct device *dev);
+static int arm_smmu_assign_dev(struct domain *d, u8 devfn, struct device *dev,
+			       u32 flag);
+static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
+				 struct device *dev);
 
 static int arm_smmu_add_device(u8 devfn, struct device *dev)
 {
 	int i, ret;
 	struct arm_smmu_device *smmu;
 	struct arm_smmu_master *master;
-	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct iommu_fwspec *fwspec;
+
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+		int ret;
+				
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		ret = iommu_add_pci_sideband_ids(pdev);
+		if ( ret < 0 ) {
+			iommu_fwspec_free(dev);
+			return ret;
+		}
+	}
+#endif
 
+	fwspec = dev_iommu_fwspec_get(dev);
 	if (!fwspec)
 		return -ENODEV;
 
@@ -1521,17 +1544,38 @@ static int arm_smmu_add_device(u8 devfn, struct device *dev)
 	 */
 	arm_smmu_enable_pasid(master);
 
-	if (dt_device_is_protected(dev_to_dt(dev))) {
-		dev_err(dev, "Already added to SMMUv3\n");
-		return -EEXIST;
-	}
+	if ( !dev_is_pci(dev) )
+	{
+		if (dt_device_is_protected(dev_to_dt(dev))) {
+			dev_err(dev, "Already added to SMMUv3\n");
+			return -EEXIST;
+		}
 
-	/* Let Xen know that the master device is protected by an IOMMU. */
-	dt_device_set_protected(dev_to_dt(dev));
+		/* Let Xen know that the master device is protected by an IOMMU. */
+		dt_device_set_protected(dev_to_dt(dev));
+	}
 
 	dev_info(dev, "Added master device (SMMUv3 %s StreamIds %u)\n",
 			dev_name(fwspec->iommu_dev), fwspec->num_ids);
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/*
+		 * During PHYSDEVOP_pci_device_add, Xen does not assign the
+		 * device, so we must do it here.
+		 */
+		if ( pdev->domain )
+		{
+			ret = arm_smmu_assign_dev(pdev->domain, devfn, dev, 0);
+			if (ret)
+				goto err_free_master;
+		}
+	}
+#endif
+
 	return 0;
 
 err_free_master:
@@ -2624,6 +2668,42 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
 	struct arm_smmu_domain *smmu_domain;
 	struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		ASSERT(pcidevs_locked());
+
+		write_lock(&pdev->domain->pci_lock);
+		list_del(&pdev->domain_list);
+		write_unlock(&pdev->domain->pci_lock);
+
+		pdev->domain = d;
+
+		write_lock(&d->pci_lock);
+		list_add(&pdev->domain_list, &d->pdev_list);
+		write_unlock(&d->pci_lock);
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+		{
+			struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+			if ( !iommu_quarantine )
+				return 0;
+
+			if ( master && master->domain )
+				arm_smmu_deassign_dev(master->domain->d, devfn, dev);
+
+			return 0;
+		}
+	}
+#endif
+
 	spin_lock(&xen_domain->lock);
 
 	/*
@@ -2657,7 +2737,7 @@ out:
 	return ret;
 }
 
-static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
+static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn, struct device *dev)
 {
 	struct iommu_domain *io_domain = arm_smmu_get_domain(d, dev);
 	struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
@@ -2669,6 +2749,21 @@ static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
 		return -ESRCH;
 	}
 
+#ifdef CONFIG_HAS_PCI
+	if ( dev_is_pci(dev) )
+	{
+		struct pci_dev *pdev = dev_to_pci(dev);
+
+		/* Ignore calls for phantom functions */
+		if ( devfn != pdev->devfn )
+			return 0;
+
+		/* dom_io is used as a sentinel for quarantined devices */
+		if ( d == dom_io )
+			return 0;
+	}
+#endif
+
 	spin_lock(&xen_domain->lock);
 
 	arm_smmu_detach_dev(master);
@@ -2687,14 +2782,16 @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t,
 {
 	int ret = 0;
 
-	/* Don't allow remapping on other domain than hwdom */
-	if ( t && !is_hardware_domain(t) )
+	/* Don't allow remapping on other domain than hwdom
+	 * or dom_io for PCI devices
+	 */
+	if ( t && !is_hardware_domain(t) && (t != dom_io || !dev_is_pci(dev)) )
 		return -EPERM;
 
 	if (t == s)
 		return 0;
 
-	ret = arm_smmu_deassign_dev(s, dev);
+	ret = arm_smmu_deassign_dev(s, devfn, dev);
 	if (ret)
 		return ret;
 
-- 
2.34.1


^ permalink raw reply related	[flat|nested] 22+ messages in thread

* [PATCH v11 6/7] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
  2025-05-28  9:12 [PATCH v11 0/7] SMMU handling for PCIe Passthrough on ARM Mykyta Poturai
                   ` (5 preceding siblings ...)
  2025-05-28  9:12 ` [PATCH v11 7/7] xen/arm: Map ITS doorbell register to IOMMU page tables Mykyta Poturai
@ 2025-05-28  9:12 ` Mykyta Poturai
  2025-06-02  8:11   ` Jan Beulich
  6 siblings, 1 reply; 22+ messages in thread
From: Mykyta Poturai @ 2025-05-28  9:12 UTC (permalink / raw)
  To: xen-devel@lists.xenproject.org
  Cc: Stewart Hildebrand, Stefano Stabellini, Julien Grall,
	Bertrand Marquis, Michal Orzel, Volodymyr Babchuk, Andrew Cooper,
	Anthony PERARD, Jan Beulich, Roger Pau Monné, Mykyta Poturai

From: Stewart Hildebrand <stewart.hildebrand@amd.com>

Enable the use of IOMMU + PCI in dom0 without having to specify
"pci-passthrough=yes". Due to possible platform specific dependencies
of the PCI host, we rely on dom0 to initialize it and perform
a PHYSDEVOP_pci_device_add/remove call to add each device to SMMU.
PHYSDEVOP_pci_device_reset is left untouched as it does not have the
pci_passthrough_enabled check.

Because pci_passthrough is not always enabled on all architectures, add
a new function arch_pci_device_physdevop that checks if we need to enable
a subset of the PCI subsystem related to managing IOMMU configuration
for PCI devices.

Enable pci_init() for initializing Xen's internal PCI subsystem, and
allow PHYSDEVOP_pci_device_add when pci-passthrough is disabled.

Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
hmm. Since
  dec9e02f3190 ("xen: avoid generation of stub <asm/pci.h> header")
Should we also move is_pci_passthrough_enabled() back to xen/arch/arm/include/asm/pci.h ?
Not sure if PPC/RISC-V will plan on using this check.

v10->v11:
* always_inline -> inline
* add comments
* clarify reset sub-op handling in the commit message

v9->v10:
* move iommu_enabled check in a separate arch function
* add Stefano's RB

v8->v9:
* move iommu_enabled check inside is_pci_passthrough_enabled()

v7->v8:
* bring back x86 definition of is_pci_passthrough_enabled()

v6->v7:
* remove x86 definition of is_pci_passthrough_enabled()
* update comments
* make pci_physdev_op checks stricter

v5->v6:
* new patch - this effectively replaces
  ("Revert "xen/arm: Add cmdline boot option "pci-passthrough = <boolean>""")
---
 xen/arch/arm/include/asm/pci.h |  2 ++
 xen/arch/arm/pci/pci.c         | 14 +++++++++++++-
 xen/arch/x86/include/asm/pci.h |  6 ++++++
 xen/drivers/pci/physdev.c      |  4 ++--
 xen/include/xen/pci.h          |  5 +++++
 5 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/include/asm/pci.h b/xen/arch/arm/include/asm/pci.h
index 7f77226c9b..d6e05f16b0 100644
--- a/xen/arch/arm/include/asm/pci.h
+++ b/xen/arch/arm/include/asm/pci.h
@@ -128,6 +128,8 @@ int pci_host_bridge_mappings(struct domain *d);
 
 bool pci_check_bar(const struct pci_dev *pdev, mfn_t start, mfn_t end);
 
+bool arch_pci_device_physdevop(void);
+
 #else   /*!CONFIG_HAS_PCI*/
 
 struct pci_dev;
diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
index 8d9692c92e..ca825ee3a6 100644
--- a/xen/arch/arm/pci/pci.c
+++ b/xen/arch/arm/pci/pci.c
@@ -16,6 +16,7 @@
 #include <xen/device_tree.h>
 #include <xen/errno.h>
 #include <xen/init.h>
+#include <xen/iommu.h>
 #include <xen/param.h>
 #include <xen/pci.h>
 
@@ -75,6 +76,17 @@ static int __init acpi_pci_init(void)
 }
 #endif
 
+/* 
+ * Platform-specific PCI host dependencies require dom0 to handle
+ * initialization and issue PHYSDEVOP_pci_device_add/remove calls for SMMU
+ * device registration. This check is used to enable the minimal PCI
+ * subsystem required for dom0 operation when PCI passthrough is disabled.
+ */
+bool arch_pci_device_physdevop(void)
+{
+    return iommu_enabled;
+}
+
 /* By default pci passthrough is disabled. */
 bool __read_mostly pci_passthrough_enabled;
 boolean_param("pci-passthrough", pci_passthrough_enabled);
@@ -85,7 +97,7 @@ static int __init pci_init(void)
      * Enable PCI passthrough when has been enabled explicitly
      * (pci-passthrough=on).
      */
-    if ( !pci_passthrough_enabled )
+    if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
         return 0;
 
     if ( pci_add_segment(0) )
diff --git a/xen/arch/x86/include/asm/pci.h b/xen/arch/x86/include/asm/pci.h
index fd5480d67d..61d8043fa3 100644
--- a/xen/arch/x86/include/asm/pci.h
+++ b/xen/arch/x86/include/asm/pci.h
@@ -67,4 +67,10 @@ static inline bool pci_check_bar(const struct pci_dev *pdev,
     return is_memory_hole(start, end);
 }
 
+/* PCI passthrough is always enabled on x86 so no special handling is needed */
+static inline bool arch_pci_device_physdevop(void)
+{
+    return false;
+}
+
 #endif /* __X86_PCI_H__ */
diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
index 0161a85e1e..21c8a3a90e 100644
--- a/xen/drivers/pci/physdev.c
+++ b/xen/drivers/pci/physdev.c
@@ -19,7 +19,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         struct pci_dev_info pdev_info;
         nodeid_t node = NUMA_NO_NODE;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop())
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
@@ -57,7 +57,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
     case PHYSDEVOP_pci_device_remove: {
         struct physdev_pci_device dev;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop())
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index ef60196653..130c2a8c1a 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -79,6 +79,11 @@ static inline bool is_pci_passthrough_enabled(void)
     return false;
 }
 
+static inline bool arch_pci_device_physdevop(void)
+{
+    return false;
+}
+
 #endif
 
 struct pci_dev_info {
-- 
2.34.1


^ permalink raw reply related	[flat|nested] 22+ messages in thread

* [PATCH v11 7/7] xen/arm: Map ITS doorbell register to IOMMU page tables
  2025-05-28  9:12 [PATCH v11 0/7] SMMU handling for PCIe Passthrough on ARM Mykyta Poturai
                   ` (4 preceding siblings ...)
  2025-05-28  9:12 ` [PATCH v11 5/7] xen/arm: Fix mapping for PCI bridge mmio region Mykyta Poturai
@ 2025-05-28  9:12 ` Mykyta Poturai
  2025-05-28  9:12 ` [PATCH v11 6/7] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no Mykyta Poturai
  6 siblings, 0 replies; 22+ messages in thread
From: Mykyta Poturai @ 2025-05-28  9:12 UTC (permalink / raw)
  To: xen-devel@lists.xenproject.org
  Cc: Rahul Singh, Stefano Stabellini, Julien Grall, Bertrand Marquis,
	Michal Orzel, Volodymyr Babchuk, Stewart Hildebrand,
	Mykyta Poturai

From: Rahul Singh <rahul.singh@arm.com>

When ITS is enabled and PCI devices that are behind an SMMU generate an
MSI interrupt, SMMU fault will be observed as there is currently no
mapping in p2m table for the ITS translation register (GITS_TRANSLATER).

A mapping is required in the iommu page tables so that the device can
generate the MSI interrupt writing to the GITS_TRANSLATER register.

The GITS_TRANSLATER register is a 32-bit register, and there is nothing
else in a page containing it, so map that page.

Add new host_addr parameter to vgic_v3_its_init_virtual to prepare the
foundation for DomU MSI support where guest doorbell address can differ
for the host's. Note that the 1:1 check in arm_iommu_map_page remains
for now, as virtual ITSes are currently only created for hwdom where the
doorbell mapping is always 1:1.

Signed-off-by: Rahul Singh <rahul.singh@arm.com>
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
This patch was originally picked up from [1], and commit description
loosely borrowed from [2].

Example SMMUv3 fault (qemu-system-aarch64 virt model), ITS base 0x8080000:

(XEN) SMMUv3: /smmuv3@9050000: event 0x10 received:
(XEN) SMMUv3: /smmuv3@9050000:  0x0000000800000010
(XEN) SMMUv3: /smmuv3@9050000:  0x0000008000000000
(XEN) SMMUv3: /smmuv3@9050000:  0x0000000008090040
(XEN) SMMUv3: /smmuv3@9050000:  0x0000000000000000

Example SMMUv2 fault (AMD/Xilinx Versal), ITS base 0xf9020000:

(XEN) smmu: /axi/smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0xf9030040, fsynr=0x12, cb=0

v10->v11:
* add Stefano's RB

v9->v10:
* map vITS doorbell to host ITS doorbell instead of always 1:1
* use simpler addr to dfn conversion

v8->v9:
* no changes

v7->v8:
* no changes

v6->v7:
* add tlb flush after mapping
* style: update formatting
* revert back to printk with XENLOG_G_ERR

v5->v6:
* switch to iommu_map() interface
* fix page_count argument
* style fixup
* use gprintk instead of printk
* add my Signed-off-by
* move to vgic_v3_its_init_virtual()

v4->v5:
* new patch

[1] https://lists.xenproject.org/archives/html/xen-devel/2023-07/msg00483.html
[2] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/commit/6232a0d53377009bb7fbc3c3ab81d0153734be6b
---
 xen/arch/arm/vgic-v3-its.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index c65c1dbf52..bc738614bb 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -1445,11 +1445,13 @@ static const struct mmio_handler_ops vgic_its_mmio_handler = {
 };
 
 static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr,
+                                    paddr_t host_addr,
                                     unsigned int devid_bits,
                                     unsigned int evid_bits)
 {
     struct virt_its *its;
     uint64_t base_attr;
+    paddr_t host_doorbell_addr = host_addr + ITS_DOORBELL_OFFSET;
 
     its = xzalloc(struct virt_its);
     if ( !its )
@@ -1478,6 +1480,26 @@ static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr,
 
     register_mmio_handler(d, &vgic_its_mmio_handler, guest_addr, SZ_64K, its);
 
+    if ( is_iommu_enabled(its->d) )
+    {
+        mfn_t mfn = maddr_to_mfn(host_doorbell_addr);
+        unsigned int flush_flags = 0;
+        int ret = iommu_map(its->d, _dfn(PFN_DOWN(its->doorbell_address)),
+                            mfn, 1, IOMMUF_writable, &flush_flags);
+
+        if ( ret < 0 )
+        {
+            printk(XENLOG_G_ERR
+                    "GICv3: Map ITS translation register for %pd failed.\n",
+                    its->d);
+            return ret;
+        }
+
+        ret = iommu_iotlb_flush(its->d, _dfn(PFN_DOWN(its->doorbell_address)), 1, flush_flags);
+        if ( ret < 0 )
+            return ret;
+    }
+
     /* Register the virtual ITS to be able to clean it up later. */
     list_add_tail(&its->vits_list, &d->arch.vgic.vits_list);
 
@@ -1522,7 +1544,7 @@ int vgic_v3_its_init_domain(struct domain *d)
              * base and thus doorbell address.
              * Use the same number of device ID and event ID bits as the host.
              */
-            ret = vgic_v3_its_init_virtual(d, hw_its->addr,
+            ret = vgic_v3_its_init_virtual(d, hw_its->addr, hw_its->addr,
                                            hw_its->devid_bits,
                                            hw_its->evid_bits);
             if ( ret )
-- 
2.34.1


^ permalink raw reply related	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 3/7] xen/arm: smmuv2: Add PCI devices support for SMMUv2
  2025-05-28  9:12 ` [PATCH v11 3/7] xen/arm: smmuv2: Add PCI devices support for SMMUv2 Mykyta Poturai
@ 2025-05-29  0:53   ` Stefano Stabellini
  0 siblings, 0 replies; 22+ messages in thread
From: Stefano Stabellini @ 2025-05-29  0:53 UTC (permalink / raw)
  To: Mykyta Poturai
  Cc: xen-devel@lists.xenproject.org, Oleksandr Andrushchenko,
	Julien Grall, Rahul Singh, Stefano Stabellini, Bertrand Marquis,
	Michal Orzel, Volodymyr Babchuk, Oleksandr Tyshchenko,
	Stewart Hildebrand

On Wed, 28 May 2025, Mykyta Poturai wrote:
> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
> 
> Implement support for PCI devices in the SMMU driver. Make arm_smmu_master
> structure to hold a pointer to the device to allow it to hold PCI devices.
> Trigger iommu-map parsing when new PCI device is added. Add checks to
> assign/deassign functions to ensure PCI devices are handled correctly.
> Implement basic quarantining.
> 
> All pci devices are automatically assigned to hardware domain if it exists
> to ensure it can probe them.
> 
> TODO:
> Implement scratch page quarantining support.
> 
> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
> Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
> v10-v11:
> * remove unused code
> * remove unnecessary blocks
> 
> v9->v10:
> * remove unused code
> * return on error in arm_smmu_dt_add_device_generic
> 
> v8->v9:
> * no change
> 
> v7->v8:
> * no change
> 
> v6->v7:
> * use d->pci_lock in arm_smmu_assign_dev()
> * remove !is_hardware_domain and pdev->domain == d checks in assign to
>   support future dom0less use case when dom0 is using vPCI
> * remove stale todo in dev_get_dev_node
> * don't print ""
> * remove redundant dt_device_is_protected check
> * remove assign/deassing prints
> * change assign logic to remove reassign reimplementation
> * check if pdev->domain exists before assigning to it
> * explain pdev->devfn check
> * make reassign check stricter and update comment
> 
> v5->v6:
> * check for hardware_domain == NULL (dom0less test case)
> * locking: assign pdev->domain before list_add()
> 
> v4->v5:
> * assign device to pdev->domain (usually dom0) by default in add_device() hook
> * deassign from hwdom
> * rebase on top of ("dynamic node programming using overlay dtbo") series
> * remove TODO in comment about device prints
> * add TODO regarding locking
> * fixup after dropping ("xen/arm: Move is_protected flag to struct device")
> 
> v3->v4:
> * add new device_is_protected check in add_device hook to match SMMUv3 and
>   IPMMU-VMSA drivers
> 
> v2->v3:
> * invoke iommu_add_pci_sideband_ids() from add_device hook
> 
> v1->v2:
> * ignore add_device/assign_device/reassign_device calls for phantom functions
>   (i.e. devfn != pdev->devfn)
> 
> downstream->v1:
> * wrap unused function in #ifdef 0
> * remove the remove_device() stub since it was submitted separately to the list
>   [XEN][PATCH v6 12/19] xen/smmu: Add remove_device callback for smmu_iommu ops
>   https://lists.xenproject.org/archives/html/xen-devel/2023-05/msg00204.html
> * arm_smmu_(de)assign_dev: return error instead of crashing system
> * update condition in arm_smmu_reassign_dev
> * style fixup
> * add && !is_hardware_domain(d) into condition in arm_smmu_assign_dev()
> 
> (cherry picked from commit 0c11a7f65f044c26d87d1e27ac6283ef1f9cfb7a from
>  the downstream branch spider-master from
>  https://github.com/xen-troops/xen.git)
> ---
>  xen/drivers/passthrough/arm/smmu.c | 246 +++++++++++++++++------------
>  1 file changed, 146 insertions(+), 100 deletions(-)
> 
> diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
> index 0f8d47dc98..307c2f6837 100644
> --- a/xen/drivers/passthrough/arm/smmu.c
> +++ b/xen/drivers/passthrough/arm/smmu.c
> @@ -120,11 +120,21 @@ enum irqreturn {
>  
>  typedef enum irqreturn irqreturn_t;
>  
> -/* Device logger functions
> - * TODO: Handle PCI
> - */
> -#define dev_print(dev, lvl, fmt, ...)						\
> -	 printk(lvl "smmu: %s: " fmt, dt_node_full_name(dev_to_dt(dev)), ## __VA_ARGS__)
> +/* Device logger functions */
> +#ifndef CONFIG_HAS_PCI
> +#define dev_print(dev, lvl, fmt, ...)    \
> +    printk(lvl "smmu: %s: " fmt, dev_name(dev), ## __VA_ARGS__)
> +#else
> +#define dev_print(dev, lvl, fmt, ...) ({                                \
> +    if ( !dev_is_pci((dev)) )                                           \
> +        printk(lvl "smmu: %s: " fmt, dev_name((dev)), ## __VA_ARGS__);  \
> +    else                                                                \
> +    {                                                                   \
> +        struct pci_dev *pdev = dev_to_pci((dev));                       \
> +        printk(lvl "smmu: %pp: " fmt, &pdev->sbdf, ## __VA_ARGS__);     \
> +    }                                                                   \
> +})
> +#endif
>  
>  #define dev_dbg(dev, fmt, ...) dev_print(dev, XENLOG_DEBUG, fmt, ## __VA_ARGS__)
>  #define dev_notice(dev, fmt, ...) dev_print(dev, XENLOG_INFO, fmt, ## __VA_ARGS__)
> @@ -172,20 +182,6 @@ static void __iomem *devm_ioremap_resource(struct device *dev,
>  #define IOMMU_FAULT_READ	0
>  #define IOMMU_FAULT_WRITE	1
>  
> -/*
> - * Xen: PCI functions
> - * TODO: It should be implemented when PCI will be supported
> - */
> -#define to_pci_dev(dev)	(NULL)
> -static inline int pci_for_each_dma_alias(struct pci_dev *pdev,
> -					 int (*fn) (struct pci_dev *pdev,
> -						    u16 alias, void *data),
> -					 void *data)
> -{
> -	BUG();
> -	return 0;
> -}
> -
>  /* Xen: misc */
>  #define PHYS_MASK_SHIFT		PADDR_BITS
>  
> @@ -619,7 +615,7 @@ struct arm_smmu_master_cfg {
>  	for (i = 0; idx = (cfg)->smendx[i], (i) < (num); ++(i))
>  
>  struct arm_smmu_master {
> -	struct device_node		*of_node;
> +	struct device			*dev;
>  	struct rb_node			node;
>  	struct arm_smmu_master_cfg	cfg;
>  };
> @@ -711,7 +707,7 @@ arm_smmu_get_fwspec(struct arm_smmu_master_cfg *cfg)
>  {
>  	struct arm_smmu_master *master = container_of(cfg,
>  			                                      struct arm_smmu_master, cfg);
> -	return dev_iommu_fwspec_get(&master->of_node->dev);
> +	return dev_iommu_fwspec_get(master->dev);
>  }
>  
>  static void parse_driver_options(struct arm_smmu_device *smmu)
> @@ -730,21 +726,11 @@ static void parse_driver_options(struct arm_smmu_device *smmu)
>  
>  static struct device_node *dev_get_dev_node(struct device *dev)
>  {
> -#if 0 /* Xen: TODO: Add support for PCI */
> -	if (dev_is_pci(dev)) {
> -		struct pci_bus *bus = to_pci_dev(dev)->bus;
> -
> -		while (!pci_is_root_bus(bus))
> -			bus = bus->parent;
> -		return bus->bridge->parent->of_node;
> -	}
> -#endif
> -
>  	return dev->of_node;
>  }
>  
>  static struct arm_smmu_master *find_smmu_master(struct arm_smmu_device *smmu,
> -						struct device_node *dev_node)
> +						struct device *dev)
>  {
>  	struct rb_node *node = smmu->masters.rb_node;
>  
> @@ -753,9 +739,9 @@ static struct arm_smmu_master *find_smmu_master(struct arm_smmu_device *smmu,
>  
>  		master = container_of(node, struct arm_smmu_master, node);
>  
> -		if (dev_node < master->of_node)
> +		if (dev < master->dev)
>  			node = node->rb_left;
> -		else if (dev_node > master->of_node)
> +		else if (dev > master->dev)
>  			node = node->rb_right;
>  		else
>  			return master;
> @@ -790,9 +776,9 @@ static int insert_smmu_master(struct arm_smmu_device *smmu,
>  			= container_of(*new, struct arm_smmu_master, node);
>  
>  		parent = *new;
> -		if (master->of_node < this->of_node)
> +		if (master->dev < this->dev)
>  			new = &((*new)->rb_left);
> -		else if (master->of_node > this->of_node)
> +		else if (master->dev > this->dev)
>  			new = &((*new)->rb_right);
>  		else
>  			return -EEXIST;
> @@ -824,28 +810,30 @@ static int arm_smmu_dt_add_device_legacy(struct arm_smmu_device *smmu,
>  	struct arm_smmu_master *master;
>  	struct device_node *dev_node = dev_get_dev_node(dev);
>  
> -	master = find_smmu_master(smmu, dev_node);
> +	master = find_smmu_master(smmu, dev);
>  	if (master) {
>  		dev_err(dev,
> -			"rejecting multiple registrations for master device %s\n",
> -			dev_node->name);
> +			"rejecting multiple registrations for master device\n");
>  		return -EBUSY;
>  	}
>  
>  	master = devm_kzalloc(dev, sizeof(*master), GFP_KERNEL);
>  	if (!master)
>  		return -ENOMEM;
> -	master->of_node = dev_node;
> +	master->dev = dev;
>  
> -	/* Xen: Let Xen know that the device is protected by an SMMU */
> -	dt_device_set_protected(dev_node);
> +	if ( !dev_is_pci(dev) )
> +	{
> +		/* Xen: Let Xen know that the device is protected by an SMMU */
> +		dt_device_set_protected(dev_node);
> +	}
>  
>  	for (i = 0; i < fwspec->num_ids; ++i) {
>  		if (!(smmu->features & ARM_SMMU_FEAT_STREAM_MATCH) &&
>  		     (fwspec->ids[i] >= smmu->num_mapping_groups)) {
>  			dev_err(dev,
> -				"stream ID for master device %s greater than maximum allowed (%d)\n",
> -				dev_node->name, smmu->num_mapping_groups);
> +				"SMMU stream ID %d is greater than maximum allowed (%d)\n",
> +				fwspec->ids[i], smmu->num_mapping_groups);
>  			return -ERANGE;
>  		}
>  		master->cfg.smendx[i] = INVALID_SMENDX;
> @@ -860,7 +848,7 @@ static int arm_smmu_dt_remove_device_legacy(struct arm_smmu_device *smmu,
>  	struct device_node *dev_node = dev_get_dev_node(dev);
>  	int ret;
>  
> -	master = find_smmu_master(smmu, dev_node);
> +	master = find_smmu_master(smmu, dev);
>  	if (master == NULL) {
>  		dev_err(dev,
>  			"No registrations found for master device %s\n",
> @@ -872,8 +860,9 @@ static int arm_smmu_dt_remove_device_legacy(struct arm_smmu_device *smmu,
>  	if (ret)
>  		return ret;
>  
> -	/* Protected by dt_host_lock and dtdevs_lock as caller holds these locks. */
> -	dev_node->is_protected = false;
> +	if ( !dev_is_pci(dev) )
> +		/* Protected by dt_host_lock and dtdevs_lock as caller holds these locks. */
> +		dev_node->is_protected = false;
>  
>  	kfree(master);
>  	return 0;
> @@ -902,6 +891,12 @@ static int register_smmu_master(struct arm_smmu_device *smmu,
>  					     fwspec);
>  }
>  
> +/* Forward declaration */
> +static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
> +			       struct device *dev, u32 flag);
> +static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
> +				 struct device *dev);
> +
>  /*
>   * The driver which supports generic IOMMU DT bindings must have this
>   * callback implemented.
> @@ -926,6 +921,25 @@ static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev)
>  {
>  	struct arm_smmu_device *smmu;
>  	struct iommu_fwspec *fwspec;
> +	int ret;
> +
> +#ifdef CONFIG_HAS_PCI
> +	if ( dev_is_pci(dev) )
> +	{
> +		struct pci_dev *pdev = dev_to_pci(dev);
> +		int ret;
> +
> +		/* Ignore calls for phantom functions */
> +		if ( devfn != pdev->devfn )
> +			return 0;
> +
> +		ret = iommu_add_pci_sideband_ids(pdev);
> +		if ( ret < 0 ) {
> +			iommu_fwspec_free(dev);
> +			return ret;
> +		}
> +	}
> +#endif
>  
>  	fwspec = dev_iommu_fwspec_get(dev);
>  	if (fwspec == NULL)
> @@ -935,7 +949,25 @@ static int arm_smmu_dt_add_device_generic(u8 devfn, struct device *dev)
>  	if (smmu == NULL)
>  		return -ENXIO;
>  
> -	return arm_smmu_dt_add_device_legacy(smmu, dev, fwspec);
> +	ret = arm_smmu_dt_add_device_legacy(smmu, dev, fwspec);
> +	if ( ret )
> +		return ret;
> +
> +#ifdef CONFIG_HAS_PCI
> +	if ( dev_is_pci(dev) )
> +	{
> +		struct pci_dev *pdev = dev_to_pci(dev);
> +
> +		/*
> +		 * During PHYSDEVOP_pci_device_add, Xen does not assign the
> +		 * device, so we must do it here.
> +		 */
> +		if ( pdev->domain )
> +			ret = arm_smmu_assign_dev(pdev->domain, devfn, dev, 0);
> +	}
> +#endif
> +
> +	return ret;
>  }
>  
>  static int arm_smmu_dt_xlate_generic(struct device *dev,
> @@ -958,11 +990,10 @@ static struct arm_smmu_device *find_smmu_for_device(struct device *dev)
>  {
>  	struct arm_smmu_device *smmu;
>  	struct arm_smmu_master *master = NULL;
> -	struct device_node *dev_node = dev_get_dev_node(dev);
>  
>  	spin_lock(&arm_smmu_devices_lock);
>  	list_for_each_entry(smmu, &arm_smmu_devices, list) {
> -		master = find_smmu_master(smmu, dev_node);
> +		master = find_smmu_master(smmu, dev);
>  		if (master)
>  			break;
>  	}
> @@ -2054,65 +2085,26 @@ static bool arm_smmu_capable(enum iommu_cap cap)
>  }
>  #endif
>  
> -static int __arm_smmu_get_pci_sid(struct pci_dev *pdev, u16 alias, void *data)
> -{
> -	*((u16 *)data) = alias;
> -	return 0; /* Continue walking */
> -}
> -
> -static void __arm_smmu_release_pci_iommudata(void *data)
> -{
> -	kfree(data);
> -}
> -
>  static int arm_smmu_add_device(struct device *dev)
>  {
>  	struct arm_smmu_device *smmu;
> +	struct arm_smmu_master *master;
>  	struct arm_smmu_master_cfg *cfg;
>  	struct iommu_group *group;
>  	void (*releasefn)(void *data) = NULL;
> -	int ret;
>  
>  	smmu = find_smmu_for_device(dev);
>  	if (!smmu)
>  		return -ENODEV;
>  
> -	if (dev_is_pci(dev)) {
> -		struct pci_dev *pdev = to_pci_dev(dev);
> -		struct iommu_fwspec *fwspec;
> -
> -		cfg = kzalloc(sizeof(*cfg), GFP_KERNEL);
> -		if (!cfg) {
> -			return -ENOMEM;
> -		}
> -
> -		ret = iommu_fwspec_init(dev, smmu->dev);
> -		if (ret) {
> -			kfree(cfg);
> -			return ret;
> -		}
> -		fwspec = dev_iommu_fwspec_get(dev);
> -
> -		/*
> -		 * Assume Stream ID == Requester ID for now.
> -		 * We need a way to describe the ID mappings in FDT.
> -		 */
> -		pci_for_each_dma_alias(pdev, __arm_smmu_get_pci_sid,
> -				       &fwspec->ids[0]);
> -		releasefn = __arm_smmu_release_pci_iommudata;
> -		cfg->smmu = smmu;
> -	} else {
> -		struct arm_smmu_master *master;
> -
> -		master = find_smmu_master(smmu, dev->of_node);
> -		if (!master) {
> -			return -ENODEV;
> -		}
> -
> -		cfg = &master->cfg;
> -		cfg->smmu = smmu;
> +	master = find_smmu_master(smmu, dev);
> +	if (!master) {
> +		return -ENODEV;
>  	}
>  
> +	cfg = &master->cfg;
> +	cfg->smmu = smmu;
> +
>  	group = iommu_group_alloc();
>  	if (IS_ERR(group)) {
>  		dev_err(dev, "Failed to allocate IOMMU group\n");
> @@ -2772,6 +2764,42 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
>  			return -ENOMEM;
>  	}
>  
> +#ifdef CONFIG_HAS_PCI
> +	if ( dev_is_pci(dev) )
> +	{
> +		struct pci_dev *pdev = dev_to_pci(dev);
> +
> +		/* Ignore calls for phantom functions */
> +		if ( devfn != pdev->devfn )
> +			return 0;
> +
> +		ASSERT(pcidevs_locked());
> +
> +		write_lock(&pdev->domain->pci_lock);
> +		list_del(&pdev->domain_list);
> +		write_unlock(&pdev->domain->pci_lock);
> +
> +		pdev->domain = d;
> +
> +		write_lock(&d->pci_lock);
> +		list_add(&pdev->domain_list, &d->pdev_list);
> +		write_unlock(&d->pci_lock);
> +
> +		/* dom_io is used as a sentinel for quarantined devices */
> +		if ( d == dom_io )
> +		{
> +			struct iommu_domain *domain = dev_iommu_domain(dev);
> +			if ( !iommu_quarantine )
> +				return 0;
> +
> +			if ( domain && domain->priv )
> +				arm_smmu_deassign_dev(domain->priv->cfg.domain, devfn, dev);
> +
> +			return 0;
> +		}
> +	}
> +#endif
> +
>  	if (!dev_iommu_group(dev)) {
>  		ret = arm_smmu_add_device(dev);
>  		if (ret)
> @@ -2821,11 +2849,27 @@ out:
>  	return ret;
>  }
>  
> -static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
> +static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
> +				 struct device *dev)
>  {
>  	struct iommu_domain *domain = dev_iommu_domain(dev);
>  	struct arm_smmu_xen_domain *xen_domain;
>  
> +#ifdef CONFIG_HAS_PCI
> +	if ( dev_is_pci(dev) )
> +	{
> +		struct pci_dev *pdev = dev_to_pci(dev);
> +
> +		/* Ignore calls for phantom functions */
> +		if ( devfn != pdev->devfn )
> +			return 0;
> +
> +		/* dom_io is used as a sentinel for quarantined devices */
> +		if ( d == dom_io )
> +			return 0;
> +	}
> +#endif
> +
>  	xen_domain = dom_iommu(d)->arch.priv;
>  
>  	if (!domain || domain->priv->cfg.domain != d) {
> @@ -2852,14 +2896,16 @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t,
>  {
>  	int ret = 0;
>  
> -	/* Don't allow remapping on other domain than hwdom */
> -	if ( t && !is_hardware_domain(t) )
> +	/* Don't allow remapping on other domain than hwdom
> +	 * or dom_io for PCI devices
> +	 */
> +	if ( t && !is_hardware_domain(t) && (t != dom_io || !dev_is_pci(dev)) )
>  		return -EPERM;
>  
>  	if (t == s)
>  		return 0;
>  
> -	ret = arm_smmu_deassign_dev(s, dev);
> +	ret = arm_smmu_deassign_dev(s, devfn, dev);
>  	if (ret)
>  		return ret;
>  
> -- 
> 2.34.1
> 


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3
  2025-05-28  9:12 ` [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3 Mykyta Poturai
@ 2025-05-29  0:56   ` Stefano Stabellini
  2025-05-29  2:10   ` Demi Marie Obenour
  2025-05-29  7:55   ` Julien Grall
  2 siblings, 0 replies; 22+ messages in thread
From: Stefano Stabellini @ 2025-05-29  0:56 UTC (permalink / raw)
  To: Mykyta Poturai
  Cc: xen-devel@lists.xenproject.org, Rahul Singh, Bertrand Marquis,
	Stefano Stabellini, Julien Grall, Michal Orzel, Volodymyr Babchuk,
	Stewart Hildebrand

On Wed, 28 May 2025, Mykyta Poturai wrote:
> From: Rahul Singh <rahul.singh@arm.com>
> 
> Implement support for PCI devices in the SMMU driver. Trigger iommu-map
> parsing when new PCI device is added. Add checks to assign/deassign
> functions to ensure PCI devices are handled correctly. Implement basic
> quarantining.
> 
> All pci devices are automatically assigned to hardware domain if it exists
> to ensure it can probe them.
> 
> TODO:
> Implement scratch page quarantining support.
> 
> Signed-off-by: Rahul Singh <rahul.singh@arm.com>
> Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
> v10->v11:
> * no changes
> 
> v9->v10:
> * return if iommu_add_pci_sidband_ids fails
> 
> v8->v9:
> * no change
> 
> v7->v8:
> * no change
> 
> v6->v7:
> * address TODO: use d->pci_lock in arm_smmu_assign_dev()
> * remove !is_hardware_domain and pdev->domain == d checks in assign to
>   support future dom0less use case when dom0 is using vPCI
> * check if pdev->domain exists before assigning to it
> * don't print ""
> * change assign logic to remove reassign reimplementation
> * explain pdev->devfn check
> * make reassign check stricter and update comment
> 
> v5->v6:
> * check for hardware_domain == NULL (dom0less test case)
> * locking: assign pdev->domain before list_add()
> 
> v4->v5:
> * deassign from hwdom
> * add TODO regarding locking
> * fixup after dropping ("xen/arm: Move is_protected flag to struct device")
> 
> v3->v4:
> * no change
> 
> v2->v3:
> * rebase
> * invoke iommu_add_pci_sideband_ids() from add_device hook
> 
> v1->v2:
> * ignore add_device/assign_device/reassign_device calls for phantom functions
>   (i.e. devfn != pdev->devfn)
> 
> downstream->v1:
> * rebase
> * move 2 replacements of s/dt_device_set_protected(dev_to_dt(dev))/device_set_protected(dev)/
>   from this commit to ("xen/arm: Move is_protected flag to struct device")
>   so as to not break ability to bisect
> * adjust patch title (remove stray space)
> * arm_smmu_(de)assign_dev: return error instead of crashing system
> * remove arm_smmu_remove_device() stub
> * update condition in arm_smmu_reassign_dev
> * style fixup
> 
> (cherry picked from commit 7ed6c3ab250d899fe6e893a514278e406a2893e8 from
>  the downstream branch poc/pci-passthrough from
>  https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
> ---
>  xen/drivers/passthrough/arm/smmu-v3.c | 119 +++++++++++++++++++++++---
>  1 file changed, 108 insertions(+), 11 deletions(-)
> 
> diff --git a/xen/drivers/passthrough/arm/smmu-v3.c b/xen/drivers/passthrough/arm/smmu-v3.c
> index df16235057..a3bbfda993 100644
> --- a/xen/drivers/passthrough/arm/smmu-v3.c
> +++ b/xen/drivers/passthrough/arm/smmu-v3.c
> @@ -1469,14 +1469,37 @@ static bool arm_smmu_sid_in_range(struct arm_smmu_device *smmu, u32 sid)
>  }
>  /* Forward declaration */
>  static struct arm_smmu_device *arm_smmu_get_by_dev(const struct device *dev);
> +static int arm_smmu_assign_dev(struct domain *d, u8 devfn, struct device *dev,
> +			       u32 flag);
> +static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn,
> +				 struct device *dev);
>  
>  static int arm_smmu_add_device(u8 devfn, struct device *dev)
>  {
>  	int i, ret;
>  	struct arm_smmu_device *smmu;
>  	struct arm_smmu_master *master;
> -	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
> +	struct iommu_fwspec *fwspec;
> +
> +#ifdef CONFIG_HAS_PCI
> +	if ( dev_is_pci(dev) )
> +	{
> +		struct pci_dev *pdev = dev_to_pci(dev);
> +		int ret;
> +				
> +		/* Ignore calls for phantom functions */
> +		if ( devfn != pdev->devfn )
> +			return 0;
> +
> +		ret = iommu_add_pci_sideband_ids(pdev);
> +		if ( ret < 0 ) {
> +			iommu_fwspec_free(dev);
> +			return ret;
> +		}
> +	}
> +#endif
>  
> +	fwspec = dev_iommu_fwspec_get(dev);
>  	if (!fwspec)
>  		return -ENODEV;
>  
> @@ -1521,17 +1544,38 @@ static int arm_smmu_add_device(u8 devfn, struct device *dev)
>  	 */
>  	arm_smmu_enable_pasid(master);
>  
> -	if (dt_device_is_protected(dev_to_dt(dev))) {
> -		dev_err(dev, "Already added to SMMUv3\n");
> -		return -EEXIST;
> -	}
> +	if ( !dev_is_pci(dev) )
> +	{
> +		if (dt_device_is_protected(dev_to_dt(dev))) {
> +			dev_err(dev, "Already added to SMMUv3\n");
> +			return -EEXIST;
> +		}
>  
> -	/* Let Xen know that the master device is protected by an IOMMU. */
> -	dt_device_set_protected(dev_to_dt(dev));
> +		/* Let Xen know that the master device is protected by an IOMMU. */
> +		dt_device_set_protected(dev_to_dt(dev));
> +	}
>  
>  	dev_info(dev, "Added master device (SMMUv3 %s StreamIds %u)\n",
>  			dev_name(fwspec->iommu_dev), fwspec->num_ids);
>  
> +#ifdef CONFIG_HAS_PCI
> +	if ( dev_is_pci(dev) )
> +	{
> +		struct pci_dev *pdev = dev_to_pci(dev);
> +
> +		/*
> +		 * During PHYSDEVOP_pci_device_add, Xen does not assign the
> +		 * device, so we must do it here.
> +		 */
> +		if ( pdev->domain )
> +		{
> +			ret = arm_smmu_assign_dev(pdev->domain, devfn, dev, 0);
> +			if (ret)
> +				goto err_free_master;
> +		}
> +	}
> +#endif
> +
>  	return 0;
>  
>  err_free_master:
> @@ -2624,6 +2668,42 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
>  	struct arm_smmu_domain *smmu_domain;
>  	struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
>  
> +#ifdef CONFIG_HAS_PCI
> +	if ( dev_is_pci(dev) )
> +	{
> +		struct pci_dev *pdev = dev_to_pci(dev);
> +
> +		/* Ignore calls for phantom functions */
> +		if ( devfn != pdev->devfn )
> +			return 0;
> +
> +		ASSERT(pcidevs_locked());
> +
> +		write_lock(&pdev->domain->pci_lock);
> +		list_del(&pdev->domain_list);
> +		write_unlock(&pdev->domain->pci_lock);
> +
> +		pdev->domain = d;
> +
> +		write_lock(&d->pci_lock);
> +		list_add(&pdev->domain_list, &d->pdev_list);
> +		write_unlock(&d->pci_lock);
> +
> +		/* dom_io is used as a sentinel for quarantined devices */
> +		if ( d == dom_io )
> +		{
> +			struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> +			if ( !iommu_quarantine )
> +				return 0;
> +
> +			if ( master && master->domain )
> +				arm_smmu_deassign_dev(master->domain->d, devfn, dev);
> +
> +			return 0;
> +		}
> +	}
> +#endif
> +
>  	spin_lock(&xen_domain->lock);
>  
>  	/*
> @@ -2657,7 +2737,7 @@ out:
>  	return ret;
>  }
>  
> -static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
> +static int arm_smmu_deassign_dev(struct domain *d, uint8_t devfn, struct device *dev)
>  {
>  	struct iommu_domain *io_domain = arm_smmu_get_domain(d, dev);
>  	struct arm_smmu_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
> @@ -2669,6 +2749,21 @@ static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
>  		return -ESRCH;
>  	}
>  
> +#ifdef CONFIG_HAS_PCI
> +	if ( dev_is_pci(dev) )
> +	{
> +		struct pci_dev *pdev = dev_to_pci(dev);
> +
> +		/* Ignore calls for phantom functions */
> +		if ( devfn != pdev->devfn )
> +			return 0;
> +
> +		/* dom_io is used as a sentinel for quarantined devices */
> +		if ( d == dom_io )
> +			return 0;
> +	}
> +#endif
> +
>  	spin_lock(&xen_domain->lock);
>  
>  	arm_smmu_detach_dev(master);
> @@ -2687,14 +2782,16 @@ static int arm_smmu_reassign_dev(struct domain *s, struct domain *t,
>  {
>  	int ret = 0;
>  
> -	/* Don't allow remapping on other domain than hwdom */
> -	if ( t && !is_hardware_domain(t) )
> +	/* Don't allow remapping on other domain than hwdom
> +	 * or dom_io for PCI devices
> +	 */
> +	if ( t && !is_hardware_domain(t) && (t != dom_io || !dev_is_pci(dev)) )
>  		return -EPERM;
>  
>  	if (t == s)
>  		return 0;
>  
> -	ret = arm_smmu_deassign_dev(s, dev);
> +	ret = arm_smmu_deassign_dev(s, devfn, dev);
>  	if (ret)
>  		return ret;
>  
> -- 
> 2.34.1
> 


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3
  2025-05-28  9:12 ` [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3 Mykyta Poturai
  2025-05-29  0:56   ` Stefano Stabellini
@ 2025-05-29  2:10   ` Demi Marie Obenour
  2025-06-04 12:48     ` Julien Grall
  2025-05-29  7:55   ` Julien Grall
  2 siblings, 1 reply; 22+ messages in thread
From: Demi Marie Obenour @ 2025-05-29  2:10 UTC (permalink / raw)
  To: xen-devel


[-- Attachment #1.1.1: Type: text/plain, Size: 613 bytes --]

On 5/28/25 05:12, Mykyta Poturai wrote:
> From: Rahul Singh <rahul.singh@arm.com>
> 
> Implement support for PCI devices in the SMMU driver. Trigger iommu-map
> parsing when new PCI device is added. Add checks to assign/deassign
> functions to ensure PCI devices are handled correctly. Implement basic
> quarantining.
> 
> All pci devices are automatically assigned to hardware domain if it exists
> to ensure it can probe them.
This is only safe for devices present at boot time.  It’s not safe for
hotplugged devices, which should be quarantined.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3
  2025-05-28  9:12 ` [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3 Mykyta Poturai
  2025-05-29  0:56   ` Stefano Stabellini
  2025-05-29  2:10   ` Demi Marie Obenour
@ 2025-05-29  7:55   ` Julien Grall
  2 siblings, 0 replies; 22+ messages in thread
From: Julien Grall @ 2025-05-29  7:55 UTC (permalink / raw)
  To: Mykyta Poturai, xen-devel@lists.xenproject.org
  Cc: Rahul Singh, Bertrand Marquis, Stefano Stabellini, Michal Orzel,
	Volodymyr Babchuk, Stewart Hildebrand

Hi Mykyta,

I have only skimmed through the patch. But there is something that
caught my eye in both this patch and the one for the SMMUv2.

On 28/05/2025 10:12, Mykyta Poturai wrote:
> +		/* dom_io is used as a sentinel for quarantined devices */
> +		if ( d == dom_io )
> +		{
> +			struct arm_smmu_master *master = dev_iommu_priv_get(dev);
> +			if ( !iommu_quarantine )
> +				return 0;
> +
> +			if ( master && master->domain )
> +				arm_smmu_deassign_dev(master->domain->d, devfn, dev);

arm_smmu_deassign_dev() can return an error. Can you explain why this is 
fine to ignore it?

> +
> +			return 0;
> +		}
> +	}
> +#endif
> +

Cheers,

-- 
Julien Grall



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 1/7] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API
  2025-05-28  9:12 ` [PATCH v11 1/7] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API Mykyta Poturai
@ 2025-06-02  8:09   ` Jan Beulich
  0 siblings, 0 replies; 22+ messages in thread
From: Jan Beulich @ 2025-06-02  8:09 UTC (permalink / raw)
  To: Mykyta Poturai
  Cc: Oleksandr Tyshchenko, Stefano Stabellini, Julien Grall,
	Bertrand Marquis, Michal Orzel, Volodymyr Babchuk,
	Roger Pau Monné, Stewart Hildebrand,
	xen-devel@lists.xenproject.org

On 28.05.2025 11:12, Mykyta Poturai wrote:
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> 
> The main purpose of this patch is to add a way to register PCI device
> (which is behind the IOMMU) using the generic PCI-IOMMU DT bindings [1]
> before assigning that device to a domain.
> 
> This behaves similarly to the existing iommu_add_dt_device API, except it
> handles PCI devices, and it is to be invoked from the add_device hook in the
> SMMU driver.
> 
> The function dt_map_id to translate an ID through a downstream mapping
> (which is also suitable for mapping Requester ID) was borrowed from Linux
> (v5.10-rc6) and updated according to the Xen code base.
> 
> [1] https://www.kernel.org/doc/Documentation/devicetree/bindings/pci/pci-iommu.txt
> 
> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

While I'm not happy about all this DT stuff in ...

> ---
>  xen/arch/arm/include/asm/iommu.h      | 15 +++++
>  xen/common/device-tree/device-tree.c  | 91 +++++++++++++++++++++++++++
>  xen/drivers/passthrough/arm/iommu.c   | 13 ++++
>  xen/drivers/passthrough/device_tree.c | 42 +++++++++++++
>  xen/include/xen/device_tree.h         | 23 +++++++
>  xen/include/xen/iommu.h               | 21 ++++++-

... this file, it's only adding to what's there already, so
Acked-by: Jan Beulich <jbeulich@suse.com>

Jan


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 6/7] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
  2025-05-28  9:12 ` [PATCH v11 6/7] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no Mykyta Poturai
@ 2025-06-02  8:11   ` Jan Beulich
  2025-06-03 13:31     ` Mykyta Poturai
  0 siblings, 1 reply; 22+ messages in thread
From: Jan Beulich @ 2025-06-02  8:11 UTC (permalink / raw)
  To: Mykyta Poturai
  Cc: Stewart Hildebrand, Stefano Stabellini, Julien Grall,
	Bertrand Marquis, Michal Orzel, Volodymyr Babchuk, Andrew Cooper,
	Anthony PERARD, Roger Pau Monné,
	xen-devel@lists.xenproject.org

On 28.05.2025 11:12, Mykyta Poturai wrote:
> From: Stewart Hildebrand <stewart.hildebrand@amd.com>
> 
> Enable the use of IOMMU + PCI in dom0 without having to specify
> "pci-passthrough=yes". Due to possible platform specific dependencies
> of the PCI host, we rely on dom0 to initialize it and perform
> a PHYSDEVOP_pci_device_add/remove call to add each device to SMMU.
> PHYSDEVOP_pci_device_reset is left untouched as it does not have the
> pci_passthrough_enabled check.

Just to re-raise the question here: Is this actually correct?

> --- a/xen/drivers/pci/physdev.c
> +++ b/xen/drivers/pci/physdev.c
> @@ -19,7 +19,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>          struct pci_dev_info pdev_info;
>          nodeid_t node = NUMA_NO_NODE;
>  
> -        if ( !is_pci_passthrough_enabled() )
> +        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop())
>              return -EOPNOTSUPP;
>  
>          ret = -EFAULT;
> @@ -57,7 +57,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>      case PHYSDEVOP_pci_device_remove: {
>          struct physdev_pci_device dev;
>  
> -        if ( !is_pci_passthrough_enabled() )
> +        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop())
>              return -EOPNOTSUPP;

Nit (style): You're losing a relevant blank each.

Jan


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 6/7] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
  2025-06-02  8:11   ` Jan Beulich
@ 2025-06-03 13:31     ` Mykyta Poturai
  2025-06-04  5:52       ` Jan Beulich
  0 siblings, 1 reply; 22+ messages in thread
From: Mykyta Poturai @ 2025-06-03 13:31 UTC (permalink / raw)
  To: Jan Beulich
  Cc: Stewart Hildebrand, Stefano Stabellini, Julien Grall,
	Bertrand Marquis, Michal Orzel, Volodymyr Babchuk, Andrew Cooper,
	Anthony PERARD, Roger Pau Monné,
	xen-devel@lists.xenproject.org

On 02.06.25 11:11, Jan Beulich wrote:
> On 28.05.2025 11:12, Mykyta Poturai wrote:
>> From: Stewart Hildebrand <stewart.hildebrand@amd.com>
>>
>> Enable the use of IOMMU + PCI in dom0 without having to specify
>> "pci-passthrough=yes". Due to possible platform specific dependencies
>> of the PCI host, we rely on dom0 to initialize it and perform
>> a PHYSDEVOP_pci_device_add/remove call to add each device to SMMU.
>> PHYSDEVOP_pci_device_reset is left untouched as it does not have the
>> pci_passthrough_enabled check.
> 
> Just to re-raise the question here: Is this actually correct?
> 

I'm afraid I don't quite understand your concerns here.

The purpose of this patch is to relax the pci_passthrough_enabled checks 
and make PCI physdev ops work with passthrough disabled.
The reset op worked independently of PCI passthrough being on or off and 
will continue to do so after this patch.
If your concerns are about the correctness of allowing reset to always 
work, you specifically requested this behavior in the patches 
implementing it here[1].

>> --- a/xen/drivers/pci/physdev.c
>> +++ b/xen/drivers/pci/physdev.c
>> @@ -19,7 +19,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>>           struct pci_dev_info pdev_info;
>>           nodeid_t node = NUMA_NO_NODE;
>>   
>> -        if ( !is_pci_passthrough_enabled() )
>> +        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop())
>>               return -EOPNOTSUPP;
>>   
>>           ret = -EFAULT;
>> @@ -57,7 +57,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
>>       case PHYSDEVOP_pci_device_remove: {
>>           struct physdev_pci_device dev;
>>   
>> -        if ( !is_pci_passthrough_enabled() )
>> +        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop())
>>               return -EOPNOTSUPP;
> 
> Nit (style): You're losing a relevant blank each.

I will update this, thanks.

> Jan

[1]: 
https://patchew.org/Xen/20240816110820.75672-1-Jiqian.Chen@amd.com/20240816110820.75672-2-Jiqian.Chen@amd.com/#1e0eee6c-0dcd-4ed4-970f-3d7e569cec09@suse.com

-- 
Mykyta

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 6/7] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
  2025-06-03 13:31     ` Mykyta Poturai
@ 2025-06-04  5:52       ` Jan Beulich
  2025-07-01  8:29         ` Mykyta Poturai
  0 siblings, 1 reply; 22+ messages in thread
From: Jan Beulich @ 2025-06-04  5:52 UTC (permalink / raw)
  To: Mykyta Poturai
  Cc: Stewart Hildebrand, Stefano Stabellini, Julien Grall,
	Bertrand Marquis, Michal Orzel, Volodymyr Babchuk, Andrew Cooper,
	Anthony PERARD, Roger Pau Monné,
	xen-devel@lists.xenproject.org

On 03.06.2025 15:31, Mykyta Poturai wrote:
> On 02.06.25 11:11, Jan Beulich wrote:
>> On 28.05.2025 11:12, Mykyta Poturai wrote:
>>> From: Stewart Hildebrand <stewart.hildebrand@amd.com>
>>>
>>> Enable the use of IOMMU + PCI in dom0 without having to specify
>>> "pci-passthrough=yes". Due to possible platform specific dependencies
>>> of the PCI host, we rely on dom0 to initialize it and perform
>>> a PHYSDEVOP_pci_device_add/remove call to add each device to SMMU.
>>> PHYSDEVOP_pci_device_reset is left untouched as it does not have the
>>> pci_passthrough_enabled check.
>>
>> Just to re-raise the question here: Is this actually correct?
> 
> I'm afraid I don't quite understand your concerns here.
> 
> The purpose of this patch is to relax the pci_passthrough_enabled checks 
> and make PCI physdev ops work with passthrough disabled.
> The reset op worked independently of PCI passthrough being on or off and 
> will continue to do so after this patch.
> If your concerns are about the correctness of allowing reset to always 
> work, you specifically requested this behavior in the patches 
> implementing it here[1].

Right, yet even there I had already asked for possible differing opinions.
Plus the case I had mentioned was specifically Dom0, which fits here.

Jan


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3
  2025-05-29  2:10   ` Demi Marie Obenour
@ 2025-06-04 12:48     ` Julien Grall
  2025-06-04 18:57       ` Demi Marie Obenour
  2025-06-10 13:13       ` Mykyta Poturai
  0 siblings, 2 replies; 22+ messages in thread
From: Julien Grall @ 2025-06-04 12:48 UTC (permalink / raw)
  To: Demi Marie Obenour, xen-devel
  Cc: Bertrand Marquis, Stefano Stabellini, Michal Orzel,
	Volodymyr Babchuk, Stewart Hildebrand, Mykyta Poturai

Hi Demi,

When replying to a thread, please keep the folks in CC unless there is a 
reason to drop them. Sending to just xen-devel is likely going to be 
lost. I personally don't always keep an eyes on discussion where I am 
not CCed, there are too many!

So adding re-adding the CC for you and keeping your reply as-is.

On 29/05/2025 03:10, Demi Marie Obenour wrote:
> On 5/28/25 05:12, Mykyta Poturai wrote:
>> From: Rahul Singh <rahul.singh@arm.com>
>>
>> Implement support for PCI devices in the SMMU driver. Trigger iommu-map
>> parsing when new PCI device is added. Add checks to assign/deassign
>> functions to ensure PCI devices are handled correctly. Implement basic
>> quarantining.
>>
>> All pci devices are automatically assigned to hardware domain if it exists
>> to ensure it can probe them.
> This is only safe for devices present at boot time.  It’s not safe for
> hotplugged devices, which should be quarantined.

-- 
Julien Grall



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3
  2025-06-04 12:48     ` Julien Grall
@ 2025-06-04 18:57       ` Demi Marie Obenour
  2025-06-04 19:52         ` Julien Grall
  2025-06-10 13:13       ` Mykyta Poturai
  1 sibling, 1 reply; 22+ messages in thread
From: Demi Marie Obenour @ 2025-06-04 18:57 UTC (permalink / raw)
  To: Julien Grall, xen-devel
  Cc: Bertrand Marquis, Stefano Stabellini, Michal Orzel,
	Volodymyr Babchuk, Stewart Hildebrand, Mykyta Poturai


[-- Attachment #1.1.1: Type: text/plain, Size: 1213 bytes --]

On 6/4/25 08:48, Julien Grall wrote:
> Hi Demi,
> 
> When replying to a thread, please keep the folks in CC unless there is a 
> reason to drop them. Sending to just xen-devel is likely going to be 
> lost. I personally don't always keep an eyes on discussion where I am 
> not CCed, there are too many!
> 
> So adding re-adding the CC for you and keeping your reply as-is.
> 
> On 29/05/2025 03:10, Demi Marie Obenour wrote:
>> On 5/28/25 05:12, Mykyta Poturai wrote:
>>> From: Rahul Singh <rahul.singh@arm.com>
>>>
>>> Implement support for PCI devices in the SMMU driver. Trigger iommu-map
>>> parsing when new PCI device is added. Add checks to assign/deassign
>>> functions to ensure PCI devices are handled correctly. Implement basic
>>> quarantining.
>>>
>>> All pci devices are automatically assigned to hardware domain if it exists
>>> to ensure it can probe them.
>> This is only safe for devices present at boot time.  It’s not safe for
>> hotplugged devices, which should be quarantined.

Thank you!  This makes me wonder if Thunderbird is a good choice for an
email client, or if I should use something like Mutt or Aerc.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3
  2025-06-04 18:57       ` Demi Marie Obenour
@ 2025-06-04 19:52         ` Julien Grall
  0 siblings, 0 replies; 22+ messages in thread
From: Julien Grall @ 2025-06-04 19:52 UTC (permalink / raw)
  To: Demi Marie Obenour, xen-devel
  Cc: Bertrand Marquis, Stefano Stabellini, Michal Orzel,
	Volodymyr Babchuk, Stewart Hildebrand, Mykyta Poturai

Hi Demi,

On 04/06/2025 19:57, Demi Marie Obenour wrote:
> On 6/4/25 08:48, Julien Grall wrote:
>> Hi Demi,
>>
>> When replying to a thread, please keep the folks in CC unless there is a
>> reason to drop them. Sending to just xen-devel is likely going to be
>> lost. I personally don't always keep an eyes on discussion where I am
>> not CCed, there are too many!
>>
>> So adding re-adding the CC for you and keeping your reply as-is.
>>
>> On 29/05/2025 03:10, Demi Marie Obenour wrote:
>>> On 5/28/25 05:12, Mykyta Poturai wrote:
>>>> From: Rahul Singh <rahul.singh@arm.com>
>>>>
>>>> Implement support for PCI devices in the SMMU driver. Trigger iommu-map
>>>> parsing when new PCI device is added. Add checks to assign/deassign
>>>> functions to ensure PCI devices are handled correctly. Implement basic
>>>> quarantining.
>>>>
>>>> All pci devices are automatically assigned to hardware domain if it exists
>>>> to ensure it can probe them.
>>> This is only safe for devices present at boot time.  It’s not safe for
>>> hotplugged devices, which should be quarantined.
> 
> Thank you!  This makes me wonder if Thunderbird is a good choice for an
> email client, or if I should use something like Mutt or Aerc.

I have been using Thunderbird for the past 10 years without any issue.

The UI tends to offer me two options "reply" and "reply all". I need to 
click on the latter to CC everyone.

I am not sure whether you can hide the "reply" (so drop all the CC) only.

Cheers,

-- 
Julien Grall



^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3
  2025-06-04 12:48     ` Julien Grall
  2025-06-04 18:57       ` Demi Marie Obenour
@ 2025-06-10 13:13       ` Mykyta Poturai
  1 sibling, 0 replies; 22+ messages in thread
From: Mykyta Poturai @ 2025-06-10 13:13 UTC (permalink / raw)
  To: Julien Grall, Demi Marie Obenour, xen-devel@lists.xenproject.org
  Cc: Bertrand Marquis, Stefano Stabellini, Michal Orzel,
	Volodymyr Babchuk, Stewart Hildebrand

On 29/05/2025 03:10, Demi Marie Obenour wrote:

>> On 5/28/25 05:12, Mykyta Poturai wrote:
>>> From: Rahul Singh <rahul.singh@arm.com>
>>>
>>> Implement support for PCI devices in the SMMU driver. Trigger iommu-map
>>> parsing when new PCI device is added. Add checks to assign/deassign
>>> functions to ensure PCI devices are handled correctly. Implement basic
>>> quarantining.
>>>
>>> All pci devices are automatically assigned to hardware domain if it 
>>> exists
>>> to ensure it can probe them.
>> This is only safe for devices present at boot time.  It’s not safe for
>> hotplugged devices, which should be quarantined.
> 

Hi Demi,

Could you please explain a little more in detail what do you think the 
issues will be here? As far as I understand quarantining is only 
relevant for transitioning devices between domains. Maybe I am missing 
something here.

-- 
Mykyta

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 6/7] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
  2025-06-04  5:52       ` Jan Beulich
@ 2025-07-01  8:29         ` Mykyta Poturai
  2025-07-01  8:49           ` Jan Beulich
  0 siblings, 1 reply; 22+ messages in thread
From: Mykyta Poturai @ 2025-07-01  8:29 UTC (permalink / raw)
  To: Jan Beulich
  Cc: Stewart Hildebrand, Stefano Stabellini, Julien Grall,
	Bertrand Marquis, Michal Orzel, Volodymyr Babchuk, Andrew Cooper,
	Anthony PERARD, Roger Pau Monné,
	xen-devel@lists.xenproject.org

On 04.06.25 08:52, Jan Beulich wrote:
> On 03.06.2025 15:31, Mykyta Poturai wrote:
>> On 02.06.25 11:11, Jan Beulich wrote:
>>> On 28.05.2025 11:12, Mykyta Poturai wrote:
>>>> From: Stewart Hildebrand <stewart.hildebrand@amd.com>
>>>>
>>>> Enable the use of IOMMU + PCI in dom0 without having to specify
>>>> "pci-passthrough=yes". Due to possible platform specific dependencies
>>>> of the PCI host, we rely on dom0 to initialize it and perform
>>>> a PHYSDEVOP_pci_device_add/remove call to add each device to SMMU.
>>>> PHYSDEVOP_pci_device_reset is left untouched as it does not have the
>>>> pci_passthrough_enabled check.
>>>
>>> Just to re-raise the question here: Is this actually correct?
>>
>> I'm afraid I don't quite understand your concerns here.
>>
>> The purpose of this patch is to relax the pci_passthrough_enabled checks
>> and make PCI physdev ops work with passthrough disabled.
>> The reset op worked independently of PCI passthrough being on or off and
>> will continue to do so after this patch.
>> If your concerns are about the correctness of allowing reset to always
>> work, you specifically requested this behavior in the patches
>> implementing it here[1].
> 
> Right, yet even there I had already asked for possible differing opinions.
> Plus the case I had mentioned was specifically Dom0, which fits here.
> 
> Jan

So I've done some testing to see the actual behavior with different 
combinations of pci-passthrough and iommu switches. With passthrough=off 
and iommu=on the reset works fine. But with both of them off, it fails 
because PHYSDEVOP_pci_device_add is not adding anything and therefore 
pci_get_pdev can't find the pdev.

I am not sure which behavior would be the correct one here for 
passthrought=off and iommu=off.

1. Leave it as is, reset returns -ENODEV and pciback probe fails
2. Add the same check as in add/remove, reset will return -EOPNOTSUPP 
and pciback probe will also fail
3. Add the same check as in add/remove but return 0 so pciback can probe 
the device.

Maybe you have some thoughts on this. I can't come up with an actual 
good reason for using pciback without pci-passthrough enabled, outside 
of maybe "not breaking some abstract scripts". And EOPNOTSUPP seems more 
descriptive than ENODEV so I strive towards option 2 if everyone okay 
with that.

-- 
Mykyta

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH v11 6/7] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no
  2025-07-01  8:29         ` Mykyta Poturai
@ 2025-07-01  8:49           ` Jan Beulich
  0 siblings, 0 replies; 22+ messages in thread
From: Jan Beulich @ 2025-07-01  8:49 UTC (permalink / raw)
  To: Mykyta Poturai
  Cc: Stewart Hildebrand, Stefano Stabellini, Julien Grall,
	Bertrand Marquis, Michal Orzel, Volodymyr Babchuk, Andrew Cooper,
	Anthony PERARD, Roger Pau Monné,
	xen-devel@lists.xenproject.org

On 01.07.2025 10:29, Mykyta Poturai wrote:
> On 04.06.25 08:52, Jan Beulich wrote:
>> On 03.06.2025 15:31, Mykyta Poturai wrote:
>>> On 02.06.25 11:11, Jan Beulich wrote:
>>>> On 28.05.2025 11:12, Mykyta Poturai wrote:
>>>>> From: Stewart Hildebrand <stewart.hildebrand@amd.com>
>>>>>
>>>>> Enable the use of IOMMU + PCI in dom0 without having to specify
>>>>> "pci-passthrough=yes". Due to possible platform specific dependencies
>>>>> of the PCI host, we rely on dom0 to initialize it and perform
>>>>> a PHYSDEVOP_pci_device_add/remove call to add each device to SMMU.
>>>>> PHYSDEVOP_pci_device_reset is left untouched as it does not have the
>>>>> pci_passthrough_enabled check.
>>>>
>>>> Just to re-raise the question here: Is this actually correct?
>>>
>>> I'm afraid I don't quite understand your concerns here.
>>>
>>> The purpose of this patch is to relax the pci_passthrough_enabled checks
>>> and make PCI physdev ops work with passthrough disabled.
>>> The reset op worked independently of PCI passthrough being on or off and
>>> will continue to do so after this patch.
>>> If your concerns are about the correctness of allowing reset to always
>>> work, you specifically requested this behavior in the patches
>>> implementing it here[1].
>>
>> Right, yet even there I had already asked for possible differing opinions.
>> Plus the case I had mentioned was specifically Dom0, which fits here.
> 
> So I've done some testing to see the actual behavior with different 
> combinations of pci-passthrough and iommu switches. With passthrough=off 
> and iommu=on the reset works fine. But with both of them off, it fails 
> because PHYSDEVOP_pci_device_add is not adding anything and therefore 
> pci_get_pdev can't find the pdev.
> 
> I am not sure which behavior would be the correct one here for 
> passthrought=off and iommu=off.
> 
> 1. Leave it as is, reset returns -ENODEV and pciback probe fails
> 2. Add the same check as in add/remove, reset will return -EOPNOTSUPP 
> and pciback probe will also fail
> 3. Add the same check as in add/remove but return 0 so pciback can probe 
> the device.
> 
> Maybe you have some thoughts on this. I can't come up with an actual 
> good reason for using pciback without pci-passthrough enabled, outside 
> of maybe "not breaking some abstract scripts". And EOPNOTSUPP seems more 
> descriptive than ENODEV so I strive towards option 2 if everyone okay 
> with that.

I think I'd favor option 2, too. Without pass-through, PHYSDEVOP_pci_device_reset
is pretty meaningless aiui. vPCI in particular builds on top of pass-through aiui,
even if that isn't expressed like that right now (e.g. by having HAS_VPCI select
HAS_PASSTHROUGH).

Jan


^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, other threads:[~2025-07-01  8:50 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-05-28  9:12 [PATCH v11 0/7] SMMU handling for PCIe Passthrough on ARM Mykyta Poturai
2025-05-28  9:12 ` [PATCH v11 2/7] iommu/arm: iommu_add_dt_pci_sideband_ids phantom handling Mykyta Poturai
2025-05-28  9:12 ` [PATCH v11 1/7] iommu/arm: Introduce iommu_add_dt_pci_sideband_ids API Mykyta Poturai
2025-06-02  8:09   ` Jan Beulich
2025-05-28  9:12 ` [PATCH v11 3/7] xen/arm: smmuv2: Add PCI devices support for SMMUv2 Mykyta Poturai
2025-05-29  0:53   ` Stefano Stabellini
2025-05-28  9:12 ` [PATCH v11 4/7] xen/arm: smmuv3: Add PCI devices support for SMMUv3 Mykyta Poturai
2025-05-29  0:56   ` Stefano Stabellini
2025-05-29  2:10   ` Demi Marie Obenour
2025-06-04 12:48     ` Julien Grall
2025-06-04 18:57       ` Demi Marie Obenour
2025-06-04 19:52         ` Julien Grall
2025-06-10 13:13       ` Mykyta Poturai
2025-05-29  7:55   ` Julien Grall
2025-05-28  9:12 ` [PATCH v11 5/7] xen/arm: Fix mapping for PCI bridge mmio region Mykyta Poturai
2025-05-28  9:12 ` [PATCH v11 7/7] xen/arm: Map ITS doorbell register to IOMMU page tables Mykyta Poturai
2025-05-28  9:12 ` [PATCH v11 6/7] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no Mykyta Poturai
2025-06-02  8:11   ` Jan Beulich
2025-06-03 13:31     ` Mykyta Poturai
2025-06-04  5:52       ` Jan Beulich
2025-07-01  8:29         ` Mykyta Poturai
2025-07-01  8:49           ` Jan Beulich

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.