[PATCH v13 0/2] SMMU handling for PCIe Passthrough on ARM

[PATCH v13 0/2] SMMU handling for PCIe Passthrough on ARM

[Mykyta Poturai]

This series introduces SMMU handling for PCIe passthrough on ARM. These patches
should be able to be upstreamed independently from the vPCI series [1]. See [2]
for notes about test cases.

[1] https://lists.xenproject.org/archives/html/xen-devel/2023-10/msg00660.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2023-06/msg01135.html

v12->v13:
* removed merged patches
* move addition of pci-passthrough enabled check in reset physdev op to a
separate patch

v11->v12:
* see individual patches

v10->v11:
* see individual patches

v9->v10:
* drop iommu/arm: Add iommu_dt_xlate()
* see individual patches

v8->v9:
* see individual patches

v7->v8:
* no changes

v6->v7:
* drop ("xen/arm: don't pass iommu properties to hwdom for iommu-map")

v5->v6:
* don't revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: enable dom0 to use PCI devices with pci-passthrough=no")

v4->v5:
* drop ("xen/arm: Improve readability of check for registered devices")
* drop ("xen/arm: Move is_protected flag to struct device")
* add ("xen/arm: don't pass iommu properties to hwdom for iommu-map")
* add ("xen/arm: Fix mapping for PCI bridge mmio region")
* revert ("xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"")
* add ("xen/arm: Map ITS doorbell register to IOMMU page tables.")
* fix test case #1 with PCI device in dom0

v3->v4:
* split a change from ("xen/arm: Move is_protected flag to struct device") into
  a new separate patch
* see individual patches for further details

v2->v3:
* drop "pci/arm: Use iommu_add_dt_pci_device()"
* drop "RFC: pci/arm: don't do iommu call for phantom functions"
* move invocation of sideband ID mapping function to add_device()
  platform_ops/iommu_ops hook

Mykyta Poturai (1):
  arm/pci: add pci_passhtrough_enabled check to pci_device_reset

Stewart Hildebrand (1):
  xen/arm: enable dom0 to use PCI devices with pci-passthrough=no

 xen/arch/arm/include/asm/pci.h |  2 ++
 xen/arch/arm/pci/pci.c         | 14 +++++++++++++-
 xen/arch/x86/include/asm/pci.h |  6 ++++++
 xen/drivers/pci/physdev.c      |  7 +++++--
 xen/include/xen/pci.h          |  5 +++++
 5 files changed, 31 insertions(+), 3 deletions(-)

-- 
2.34.1

[PATCH v13 1/2] arm/pci: add pci_passhtrough_enabled check to pci_device_reset

[Mykyta Poturai]

Without pci-passthrough=on Xen does not know anything about present PCI
devices due to PHYSDEVOP_pci_device_add not executing. This causes
PHYSDEVOP_pci_device_reset to fail when trying to find the device that
is being reset and return ENODEV.

Add explicit is_pci_passthrough_enabled check to return a more
meaningful EOPNOTSUPP code.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
 xen/drivers/pci/physdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
index 0161a85e1e..78de67ec64 100644
--- a/xen/drivers/pci/physdev.c
+++ b/xen/drivers/pci/physdev.c
@@ -74,6 +74,9 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         struct pci_dev *pdev;
         pci_sbdf_t sbdf;
 
+        if ( !is_pci_passthrough_enabled() )
+            return -EOPNOTSUPP;
+
         ret = -EFAULT;
         if ( copy_from_guest(&dev_reset, arg, 1) != 0 )
             break;
-- 
2.34.1

Re: [PATCH v13 1/2] arm/pci: add pci_passhtrough_enabled check to pci_device_reset

[Jan Beulich]

On 16.07.2025 09:43, Mykyta Poturai wrote:
> Without pci-passthrough=on Xen does not know anything about present PCI
> devices due to PHYSDEVOP_pci_device_add not executing.

While the latter half of the sentence is true, Xen may know of PCI devices
be other means. It scans the bus itself after all on x86, under certain
conditions. Furthermore "pci-passthrough" is an Arm-only option, when the
change here affects x86 as much. The description therefore will need some
improvement; the code change itself is okay (on the assumption that it can
be justified properly).

Jan

Re: [PATCH v13 1/2] arm/pci: add pci_passhtrough_enabled check to pci_device_reset

[Mykyta Poturai]

On 16.07.25 11:36, Jan Beulich wrote:
> On 16.07.2025 09:43, Mykyta Poturai wrote:
>> Without pci-passthrough=on Xen does not know anything about present PCI
>> devices due to PHYSDEVOP_pci_device_add not executing.
> 
> While the latter half of the sentence is true, Xen may know of PCI devices
> be other means. It scans the bus itself after all on x86, under certain
> conditions. Furthermore "pci-passthrough" is an Arm-only option, when the
> change here affects x86 as much. The description therefore will need some
> improvement; the code change itself is okay (on the assumption that it can
> be justified properly).
> 
> Jan

Will adding an additional note that all of this only relates to Arm and 
will not change x86 functionality be sufficient? As far as I understand, 
this sentence is completely true for Arm, even the yet-to-be-upstreamed 
bus scanning routines in Xen are guarded by is_pci_passthroug_enabled in 
the latest patches I've seen them in.

-- 
Mykyta

Re: [PATCH v13 1/2] arm/pci: add pci_passhtrough_enabled check to pci_device_reset

[Jan Beulich]

On 18.07.2025 13:52, Mykyta Poturai wrote:
> On 16.07.25 11:36, Jan Beulich wrote:
>> On 16.07.2025 09:43, Mykyta Poturai wrote:
>>> Without pci-passthrough=on Xen does not know anything about present PCI
>>> devices due to PHYSDEVOP_pci_device_add not executing.
>>
>> While the latter half of the sentence is true, Xen may know of PCI devices
>> be other means. It scans the bus itself after all on x86, under certain
>> conditions. Furthermore "pci-passthrough" is an Arm-only option, when the
>> change here affects x86 as much. The description therefore will need some
>> improvement; the code change itself is okay (on the assumption that it can
>> be justified properly).
> 
> Will adding an additional note that all of this only relates to Arm and 
> will not change x86 functionality be sufficient?

Probably.

Jan

> As far as I understand, 
> this sentence is completely true for Arm, even the yet-to-be-upstreamed 
> bus scanning routines in Xen are guarded by is_pci_passthroug_enabled in 
> the latest patches I've seen them in.
> 

[PATCH v13 2/2] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no

[Mykyta Poturai]

From: Stewart Hildebrand <stewart.hildebrand@amd.com>

Enable the use of IOMMU + PCI in dom0 without having to specify
"pci-passthrough=yes". Due to possible platform specific dependencies
of the PCI host, we rely on dom0 to initialize it and perform
a PCI PHYSDEVOP calls to add each device to SMMU.

Because pci_passthrough is not always enabled on all architectures, add
a new function arch_pci_device_physdevop that checks if we need to enable
a subset of the PCI subsystem related to managing IOMMU configuration
for PCI devices.

Enable pci_init() for initializing Xen's internal PCI subsystem, and
allow PCI PHYSDEV ops when pci-passthrough is disabled.

Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
This is the last patch from SMMU-PCIe on ARM series
https://patchew.org/Xen/cover.1751439885.git.mykyta._5Fpoturai@epam.com/
hmm. Since
  dec9e02f3190 ("xen: avoid generation of stub <asm/pci.h> header")
Should we also move is_pci_passthrough_enabled() back to xen/arch/arm/include/asm/pci.h ?
Not sure if PPC/RISC-V will plan on using this check.

v12->v13:
* move introduction of reset op check to a separate patch
* update the commit message

v11->v12:
* add enabled checks to pci_device_reset
* fix style issues

v10->v11:
* always_inline -> inline
* add comments
* clarify reset sub-op handling in the commit message

v9->v10:
* move iommu_enabled check in a separate arch function
* add Stefano's RB

v8->v9:
* move iommu_enabled check inside is_pci_passthrough_enabled()

v7->v8:
* bring back x86 definition of is_pci_passthrough_enabled()

v6->v7:
* remove x86 definition of is_pci_passthrough_enabled()
* update comments
* make pci_physdev_op checks stricter

v5->v6:
* new patch - this effectively replaces
  ("Revert "xen/arm: Add cmdline boot option "pci-passthrough = <boolean>""")
---
 xen/arch/arm/include/asm/pci.h |  2 ++
 xen/arch/arm/pci/pci.c         | 14 +++++++++++++-
 xen/arch/x86/include/asm/pci.h |  6 ++++++
 xen/drivers/pci/physdev.c      |  6 +++---
 xen/include/xen/pci.h          |  5 +++++
 5 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/include/asm/pci.h b/xen/arch/arm/include/asm/pci.h
index 37a6f14dd4..08ffcd4438 100644
--- a/xen/arch/arm/include/asm/pci.h
+++ b/xen/arch/arm/include/asm/pci.h
@@ -151,6 +151,8 @@ void pci_generic_init_bus_range_child(struct dt_device_node *dev,
                                       struct pci_host_bridge *bridge,
                                       struct pci_config_window *cfg);
 
+bool arch_pci_device_physdevop(void);
+
 #else   /*!CONFIG_HAS_PCI*/
 
 struct pci_dev;
diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
index 8d9692c92e..beb1f971fa 100644
--- a/xen/arch/arm/pci/pci.c
+++ b/xen/arch/arm/pci/pci.c
@@ -16,6 +16,7 @@
 #include <xen/device_tree.h>
 #include <xen/errno.h>
 #include <xen/init.h>
+#include <xen/iommu.h>
 #include <xen/param.h>
 #include <xen/pci.h>
 
@@ -75,6 +76,17 @@ static int __init acpi_pci_init(void)
 }
 #endif
 
+/*
+ * Platform-specific PCI host dependencies require dom0 to handle
+ * initialization and issue PHYSDEVOP_pci_device_add/remove calls for SMMU
+ * device registration. This check is used to enable the minimal PCI
+ * subsystem required for dom0 operation when PCI passthrough is disabled.
+ */
+bool arch_pci_device_physdevop(void)
+{
+    return iommu_enabled;
+}
+
 /* By default pci passthrough is disabled. */
 bool __read_mostly pci_passthrough_enabled;
 boolean_param("pci-passthrough", pci_passthrough_enabled);
@@ -85,7 +97,7 @@ static int __init pci_init(void)
      * Enable PCI passthrough when has been enabled explicitly
      * (pci-passthrough=on).
      */
-    if ( !pci_passthrough_enabled )
+    if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
         return 0;
 
     if ( pci_add_segment(0) )
diff --git a/xen/arch/x86/include/asm/pci.h b/xen/arch/x86/include/asm/pci.h
index 2e67cba8b9..3830232246 100644
--- a/xen/arch/x86/include/asm/pci.h
+++ b/xen/arch/x86/include/asm/pci.h
@@ -64,4 +64,10 @@ bool pci_check_bar(const struct pci_dev *pdev, mfn_t start, mfn_t end);
 struct rangeset;
 int pci_sanitize_bar_memory(struct rangeset *r);
 
+/* PCI passthrough is always enabled on x86 so no special handling is needed */
+static inline bool arch_pci_device_physdevop(void)
+{
+    return false;
+}
+
 #endif /* __X86_PCI_H__ */
diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
index 78de67ec64..d46501b884 100644
--- a/xen/drivers/pci/physdev.c
+++ b/xen/drivers/pci/physdev.c
@@ -19,7 +19,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         struct pci_dev_info pdev_info;
         nodeid_t node = NUMA_NO_NODE;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
@@ -57,7 +57,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
     case PHYSDEVOP_pci_device_remove: {
         struct physdev_pci_device dev;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
@@ -74,7 +74,7 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         struct pci_dev *pdev;
         pci_sbdf_t sbdf;
 
-        if ( !is_pci_passthrough_enabled() )
+        if ( !is_pci_passthrough_enabled() && !arch_pci_device_physdevop() )
             return -EOPNOTSUPP;
 
         ret = -EFAULT;
diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index ef60196653..130c2a8c1a 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -79,6 +79,11 @@ static inline bool is_pci_passthrough_enabled(void)
     return false;
 }
 
+static inline bool arch_pci_device_physdevop(void)
+{
+    return false;
+}
+
 #endif
 
 struct pci_dev_info {
-- 
2.34.1

Re: [PATCH v13 2/2] xen/arm: enable dom0 to use PCI devices with pci-passthrough=no

[Jan Beulich]

On 16.07.2025 09:43, Mykyta Poturai wrote:
> --- a/xen/arch/x86/include/asm/pci.h
> +++ b/xen/arch/x86/include/asm/pci.h
> @@ -64,4 +64,10 @@ bool pci_check_bar(const struct pci_dev *pdev, mfn_t start, mfn_t end);
>  struct rangeset;
>  int pci_sanitize_bar_memory(struct rangeset *r);
>  
> +/* PCI passthrough is always enabled on x86 so no special handling is needed */
> +static inline bool arch_pci_device_physdevop(void)
> +{
> +    return false;
> +}

The comment is somewhat odd, as it talks about pass-through when the function
is all about physdevop. A connection wants making imo. Plus isn't it benign
right now whether the function returned false or true? From an abstract
perspective, it returning true would perhaps make more sense (as opposed to
the generic stub in include/xen/pci.h). IOW I think the return value wants
changing _and_ the comment wants to clarify that either value could be used
with the present call sites (thus helping people finding themselves in need
of altering the return value, in order to use the function elsewhere).

I also think this would better go directly below is_pci_passthrough_enabled().

Jan