[PATCH v14 0/2] SMMU handling for PCIe Passthrough on ARM

[PATCH v14 0/2] SMMU handling for PCIe Passthrough on ARM

[Mykyta Poturai]

This series introduces SMMU handling for PCIe passthrough on ARM. These patches
should be able to be upstreamed independently from the vPCI series [1]. See [2]
for notes about test cases.

[1] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00660.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg01135.html

v13->v14:
* see individual patches

v12->v13:
* removed merged patches
* move addition of pci-passthrough enabled check in reset physdev op to a
separate patch

v11->v12:
* see individual patches

v10->v11:
* see individual patches

v9->v10:
* drop iommu/arm: Add iommu_dt_xlate()
* see individual patches

v8->v9:
* see individual patches

v7->v8:
* no changes

v6->v7:
* drop ("xen/arm: don't pass iommu properties to hwdom for iommu-map")

v5->v6:
* don't revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: enable dom0 to use PCI devices with pci-passthrough=no")

v4->v5:
* drop ("xen/arm: Improve readability of check for registered devices")
* drop ("xen/arm: Move is_protected flag to struct device")
* add ("xen/arm: don't pass iommu properties to hwdom for iommu-map")
* add ("xen/arm: Fix mapping for PCI bridge mmio region")
* revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: Map ITS doorbell register to IOMMU page tables.")
* fix test case #1 with PCI device in dom0

v3->v4:
* split a change from ("xen/arm: Move is_protected flag to struct device") into
  a new separate patch
* see individual patches for further details

v2->v3:
* drop "pci/arm: Use iommu_add_dt_pci_device()"
* drop "RFC: pci/arm: don't do iommu call for phantom functions"
* move invocation of sideband ID mapping function to add_device()
  platform_ops/iommu_ops hook

Mykyta Poturai (1):
  arm/pci: add pci_passhtrough_enabled check to pci_device_reset

Stewart Hildebrand (1):
  xen/arm: enable dom0 to use PCI devices with pci-passthrough=no

 xen/arch/arm/include/asm/pci.h |  2 ++
 xen/arch/arm/pci/pci.c         | 14 +++++++++++++-
 xen/arch/x86/include/asm/pci.h | 10 ++++++++++
 xen/drivers/pci/physdev.c      |  7 +++++--
 xen/include/xen/pci.h          |  5 +++++
 5 files changed, 35 insertions(+), 3 deletions(-)

-- 
2.34.1

[PATCH v14 1/2] arm/pci: add pci_passhtrough_enabled check to pci_device_reset

[Mykyta Poturai]

On Arm without pci-passthrough=on Xen does not know anything about
present PCI devices due to PHYSDEVOP_pci_device_add not executing. This
causes PHYSDEVOP_pci_device_reset to fail when trying to find the device
that is being reset and return ENODEV.

Add explicit is_pci_passthrough_enabled check to return a more
meaningful EOPNOTSUPP code.
This change should not affect x86 behavior as pci-passthrough is always
enabled there.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
v13->v14:
* clarify the commit message

v13:
* new patch
---
 xen/drivers/pci/physdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
index 0161a85e1e..78de67ec64 100644
--- a/xen/drivers/pci/physdev.c
+++ b/xen/drivers/pci/physdev.c
@@ -74,6 +74,9 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         struct pci_dev *pdev;
         pci_sbdf_t sbdf;
 
+        if ( !is_pci_passthrough_enabled() )
+            return -EOPNOTSUPP;
+
         ret = -EFAULT;
         if ( copy_from_guest(&dev_reset, arg, 1) != 0 )
             break;
-- 
2.34.1

[PATCH v14 2/2] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no

[Mykyta Poturai]

From: Stewart Hildebrand <stewart.hildebrand@amd.com>

Enable the use of IOMMU + PCI in dom0 without having to specify
"pci-passthrough=yes". Due to possible platform specific dependencies
of the PCI host, we rely on dom0 to initialize it and perform
a PCI PHYSDEVOP calls to add each device to SMMU.

Because pci_passthrough is not always enabled on all architectures, add
a new function arch_pci_device_physdevop that checks if we need to enable
a subset of the PCI subsystem related to managing IOMMU configuration
for PCI devices.

Enable pci_init() for initializing Xen's internal PCI subsystem, and
allow PCI PHYSDEV ops when pci-passthrough is disabled.

Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
This is the last patch from SMMU-PCIe on ARM series
https://patchew.org/Xen/cover.1751439885.git.mykyta._5Fpoturai@epam.com/
hmm. Since
  dec9e02f3190 ("xen: avoid generation of stub <asm/pci.h> header")
Should we also move is_pci_passthrough_enabled() back to xen/arch/arm/include/asm/pci.h ?
Not sure if PPC/RISC-V will plan on using this check.

v13->v14:
* make x86 arch_pci_device_physdevop more clear

v12->v13:
* move introduction of reset op check to a separate patch
* update the commit message

v11->v12:
* add enabled checks to pci_device_reset
* fix style issues

v10->v11:
* always_inline -> inline
* add comments
* clarify reset sub-op handling in the commit message

v9->v10:
* move iommu_enabled check in a separate arch function
* add Stefano's RB

v8->v9:
* move iommu_enabled check inside is_pci_passthrough_enabled()

v7->v8:
* bring back x86 definition of is_pci_passthrough_enabled()

v6->v7:
* remove x86 definition of is_pci_passthrough_enabled()
* update comments
* make pci_physdev_op checks stricter

v5->v6:
* new patch - this effectively replaces
  ("Revert "xen/arm: Add cmdline boot option "pci-passthrough = <boolean>""")
---
 xen/arch/arm/include/asm/pci.h |  2 ++
 xen/arch/arm/pci/pci.c         | 14 +++++++++++++-
 xen/arch/x86/include/asm/pci.h | 10 ++++++++++
 xen/drivers/pci/physdev.c      |  6 +++---
 xen/include/xen/pci.h          |  5 +++++
 5 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/include/asm/pci.h b/xen/arch/arm/include/asm/pci.h
index 37a6f14dd4..08ffcd4438 100644
--- a/xen/arch/arm/include/asm/pci.h
+++ b/xen/arch/arm/include/asm/pci.h
@@ -151,6 +151,8 @@ void pci_generic_init_bus_range_child(struct dt_device_node *dev,
                                       struct pci_host_bridge *bridge,
                                       struct pci_config_window *cfg);
 
+bool arch_pci_device_physdevop(void);
+
 #else   /*!CONFIG_HAS_PCI*/
 
 struct pci_dev;
diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
index 8d9692c92e..beb1f971fa 100644
--- a/xen/arch/arm/pci/pci.c
+++ b/xen/arch/arm/pci/pci.c
@@ -16,6 +16,7 @@
 #include <xen/device_tree.h>
 #include <xen/errno.h>
 #include <xen/init.h>
+#include <xen/iommu.h>
 #include <xen/param.h>
 #include <xen/pci.h>
 
@@ -75,6 +76,17 @@ static int __init acpi_pci_init(void)
 }
 #endif
 
+/*
+ * Platform-specific PCI host dependencies require dom0 to handle
+ * initialization and issue PHYSDEVOP_pci_device_add/remove calls for SMMU
+ * device registration. This check is used to enable the minimal PCI
+ * subsystem required for dom0 operation when PCI passthrough is disabled.
+ */
+bool arch_pci_device_physdevop(void)
+{
+    return iommu_enabled;
+}
+
 /* By default pci passthrough is disabled. */
 bool __read_mostly pci_passthrough_enabled;
 boolean_param("pci-passthrough", pci_passthrough_enabled);
@@ -85,7 +97,7 @@ static int __init pci_init(void)
      * Enable PCI passthrough when has been enabled explicitly
      * (pci-passthrough=on).
      */
-    if ( !pci_passthrough_enabled )
+    if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
         return 0;
 
     if ( pci_add_segment(0) )
diff --git a/xen/arch/x86/include/asm/pci.h b/xen/arch/x86/include/asm/pci.h
index 2e67cba8b9..665b321165 100644
--- a/xen/arch/x86/include/asm/pci.h
+++ b/xen/arch/x86/include/asm/pci.h
@@ -57,6 +57,16 @@ static always_inline bool is_pci_passthrough_enabled(void)
     return true;
 }
 
+/*
+ * Since PCI passthrough is always enabled on x86, physdevop handling doesn't
+ * need special arch-specific behavior. Current call sites work with either
+ * return value, but true is more consistent with passthrough being enabled.
+ */
+static inline bool arch_pci_device_physdevop(void)
+{
+    return true;
+}
+
 void arch_pci_init_pdev(struct pci_dev *pdev);
 
 bool pci_check_bar(const struct pci_dev *pdev, mfn_t start, mfn_t end);
diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
index 78de67ec64..d46501b884 100644
--- a/xen/drivers/pci/physdev.c
+++ b/xen/drivers/pci/physdev.c
@@ -19,7 +19,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         struct pci_dev_info pdev_info;
         nodeid_t node = NUMA_NO_NODE;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
@@ -57,7 +57,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
     case PHYSDEVOP_pci_device_remove: {
         struct physdev_pci_device dev;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
@@ -74,7 +74,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         struct pci_dev *pdev;
         pci_sbdf_t sbdf;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index ef60196653..130c2a8c1a 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -79,6 +79,11 @@ static inline bool is_pci_passthrough_enabled(void)
     return false;
 }
 
+static inline bool arch_pci_device_physdevop(void)
+{
+    return false;
+}
+
 #endif
 
 struct pci_dev_info {
-- 
2.34.1

Re: [PATCH v14 1/2] arm/pci: add pci_passhtrough_enabled check to pci_device_reset

[Jan Beulich]

On 24.07.2025 11:44, Mykyta Poturai wrote:
> On Arm without pci-passthrough=on Xen does not know anything about
> present PCI devices due to PHYSDEVOP_pci_device_add not executing. This
> causes PHYSDEVOP_pci_device_reset to fail when trying to find the device
> that is being reset and return ENODEV.
> 
> Add explicit is_pci_passthrough_enabled check to return a more
> meaningful EOPNOTSUPP code.

I.e. just like PHYSDEVOP_pci_device_{add,remove} already have it. Which
also helps ...

> This change should not affect x86 behavior as pci-passthrough is always
> enabled there.

... proving the no-effect aspect for x86 here.

> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

Reviewed-by: Jan Beulich <jbeulich@suse.com>

However, the subject prefix is entirely misleading. The change helps Arm,
but is in arch-neutral code. Without "physdev" or "physdevop" anywhere in
the subject, whether "pci_device_reset" refers to a function (and you
merely omitted the parentheses) or something else doesn't become clear.

Also, nit: "is_pci_passthrough_enabled()" or "pci-passthrough-enabled" in
the subject, both to correct spelling and to make clear that either you
explicitly mean an existing predicate, or you merely mean to describe that
verbally.

Jan

Re: [PATCH v14 2/2] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no

[Jan Beulich]

On 24.07.2025 11:44, Mykyta Poturai wrote:
> From: Stewart Hildebrand <stewart.hildebrand@amd.com>
> 
> Enable the use of IOMMU + PCI in dom0 without having to specify
> "pci-passthrough=yes". Due to possible platform specific dependencies
> of the PCI host, we rely on dom0 to initialize it and perform
> a PCI PHYSDEVOP calls to add each device to SMMU.
> 
> Because pci_passthrough is not always enabled on all architectures, add
> a new function arch_pci_device_physdevop that checks if we need to enable
> a subset of the PCI subsystem related to managing IOMMU configuration
> for PCI devices.
> 
> Enable pci_init() for initializing Xen's internal PCI subsystem, and
> allow PCI PHYSDEV ops when pci-passthrough is disabled.
> 
> Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

Acked-by: Jan Beulich <jbeulich@suse.com>

Nevertheless I'd like to point out that ...

> --- a/xen/arch/x86/include/asm/pci.h
> +++ b/xen/arch/x86/include/asm/pci.h
> @@ -57,6 +57,16 @@ static always_inline bool is_pci_passthrough_enabled(void)
>      return true;
>  }
>  
> +/*
> + * Since PCI passthrough is always enabled on x86, physdevop handling doesn't
> + * need special arch-specific behavior. Current call sites work with either
> + * return value, but true is more consistent with passthrough being enabled.
> + */
> +static inline bool arch_pci_device_physdevop(void)
> +{
> +    return true;
> +}

... the comment still isn't quite right. PCI-passthrough isn't always enabled
on x86. Both AMD_IOMMU and INTEL_IOMMU Kconfig options can be used to turn
respective support off. Things then work as if no IOMMU was found, which
means largely "no pass-through".

Jan