[PATCH v2 0/3] dom0less pci passthrough support on Arm part 1

[PATCH v2 0/3] dom0less pci passthrough support on Arm part 1

[Mykyta Poturai]

This series adds basic PCI device enumeration in Xen. This will allow us to not
rely on Dom0 enumeration for supported controllers, which will enable PCI
passthrough for dom0less setups.

Enumeration is disabled by default and can be enabled with "pci-scan" cmdline
option. Discovered devices are added to Xen and assigned to dom_io by default.

Edward Pickup (1):
  arm/pci: Add pci-scan boot argument

Luca Fancellu (2):
  xen/pci: modify pci_add_device to handle device add by Xen
  xen/pci: add discovered PCI device at boot

 docs/misc/xen-command-line.pandoc           |  7 ++
 xen/arch/arm/include/asm/pci.h              |  3 +
 xen/arch/arm/pci/pci-host-common.c          |  1 +
 xen/arch/arm/pci/pci.c                      | 25 +++++++-
 xen/arch/x86/physdev.c                      |  9 ++-
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  2 +-
 xen/drivers/passthrough/pci.c               | 71 ++++++++++++++-------
 xen/drivers/passthrough/vtd/iommu.c         |  2 +-
 xen/drivers/pci/physdev.c                   |  3 +-
 xen/include/xen/pci.h                       | 10 +--
 10 files changed, 94 insertions(+), 39 deletions(-)

-- 
2.34.1

[PATCH v2 1/3] arm/pci: Add pci-scan boot argument

[Mykyta Poturai]

From: Edward Pickup <Edward.Pickup@arm.com>

This patch adds a Xen boot arguments that, if enabled, causes a call to
existing code to scan pci devices enumerated by the firmware.

This will be needed ahead of dom0less support for pci passthrough on
arm.

Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
(cherry picked from commit bce463e1588a45e1bfdf59fc0d5f88b16604e439 from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)

v1->v2:
* remove is_pci_scan_enabled wrapper
* make pci_scan_enabled ro_after_init
* drop debug prints
* drop Edward's SOB

changes since cherry-pick:
* s/always_inline/inline/
* replace additional kconfig option with config DEBUG

Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
---
 docs/misc/xen-command-line.pandoc  |  7 +++++++
 xen/arch/arm/include/asm/pci.h     |  3 +++
 xen/arch/arm/pci/pci-host-common.c |  1 +
 xen/arch/arm/pci/pci.c             | 24 ++++++++++++++++++++++--
 4 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
index a75b6c9301..762a1a4f5f 100644
--- a/docs/misc/xen-command-line.pandoc
+++ b/docs/misc/xen-command-line.pandoc
@@ -2059,6 +2059,13 @@ This option can be specified more than once (up to 8 times at present).
 
 Flag to enable or disable support for PCI passthrough
 
+### pci-scan (arm)
+> `= <boolean>`
+
+> Default: `false`
+
+Flag to enable or disable Xen PCI scan at boot.
+
 ### pcid (x86)
 > `= <boolean> | xpti=<bool>`
 
diff --git a/xen/arch/arm/include/asm/pci.h b/xen/arch/arm/include/asm/pci.h
index 08ffcd4438..7289f7688b 100644
--- a/xen/arch/arm/include/asm/pci.h
+++ b/xen/arch/arm/include/asm/pci.h
@@ -23,6 +23,7 @@
 #define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
 
 extern bool pci_passthrough_enabled;
+extern bool pci_scan_enabled;
 
 struct rangeset;
 
@@ -155,6 +156,8 @@ bool arch_pci_device_physdevop(void);
 
 #else   /*!CONFIG_HAS_PCI*/
 
+#define pci_scan_enabled false
+
 struct pci_dev;
 
 static inline void arch_pci_init_pdev(struct pci_dev *pdev) {}
diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
index 487c545f3a..d3481b05eb 100644
--- a/xen/arch/arm/pci/pci-host-common.c
+++ b/xen/arch/arm/pci/pci-host-common.c
@@ -284,6 +284,7 @@ pci_host_common_probe(struct dt_device_node *dev,
     }
 
     pci_add_host_bridge(bridge);
+    pci_add_segment(bridge->segment);
 
     return bridge;
 
diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
index beb1f971fa..1b34e17517 100644
--- a/xen/arch/arm/pci/pci.c
+++ b/xen/arch/arm/pci/pci.c
@@ -91,8 +91,14 @@ bool arch_pci_device_physdevop(void)
 bool __read_mostly pci_passthrough_enabled;
 boolean_param("pci-passthrough", pci_passthrough_enabled);
 
+/* By default pci scan is disabled. */
+bool __ro_after_init pci_scan_enabled;
+boolean_param("pci-scan", pci_scan_enabled);
+
 static int __init pci_init(void)
 {
+    int ret;
+
     /*
      * Enable PCI passthrough when has been enabled explicitly
      * (pci-passthrough=on).
@@ -104,9 +110,23 @@ static int __init pci_init(void)
         panic("Could not initialize PCI segment 0\n");
 
     if ( acpi_disabled )
-        return dt_pci_init();
+        ret = dt_pci_init();
     else
-        return acpi_pci_init();
+        ret = acpi_pci_init();
+
+    if ( ret < 0 )
+        return ret;
+
+    if ( pci_scan_enabled )
+    {
+        ret = scan_pci_devices();
+
+        if ( ret < 0 )
+            return ret;
+
+    }
+
+    return 0;
 }
 __initcall(pci_init);
 
-- 
2.34.1

Re: [PATCH v2 1/3] arm/pci: Add pci-scan boot argument

[Stefano Stabellini]

On Wed, 20 Aug 2025, Mykyta Poturai wrote:
> From: Edward Pickup <Edward.Pickup@arm.com>
> 
> This patch adds a Xen boot arguments that, if enabled, causes a call to
> existing code to scan pci devices enumerated by the firmware.
> 
> This will be needed ahead of dom0less support for pci passthrough on
> arm.
> 
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
> (cherry picked from commit bce463e1588a45e1bfdf59fc0d5f88b16604e439 from
>  the downstream branch poc/pci-passthrough from
>  https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
> 
> v1->v2:
> * remove is_pci_scan_enabled wrapper
> * make pci_scan_enabled ro_after_init
> * drop debug prints
> * drop Edward's SOB
> 
> changes since cherry-pick:
> * s/always_inline/inline/
> * replace additional kconfig option with config DEBUG
> 
> Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
> ---
>  docs/misc/xen-command-line.pandoc  |  7 +++++++
>  xen/arch/arm/include/asm/pci.h     |  3 +++
>  xen/arch/arm/pci/pci-host-common.c |  1 +
>  xen/arch/arm/pci/pci.c             | 24 ++++++++++++++++++++++--
>  4 files changed, 33 insertions(+), 2 deletions(-)
> 
> diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
> index a75b6c9301..762a1a4f5f 100644
> --- a/docs/misc/xen-command-line.pandoc
> +++ b/docs/misc/xen-command-line.pandoc
> @@ -2059,6 +2059,13 @@ This option can be specified more than once (up to 8 times at present).
>  
>  Flag to enable or disable support for PCI passthrough
>  
> +### pci-scan (arm)
> +> `= <boolean>`
> +
> +> Default: `false`
> +
> +Flag to enable or disable Xen PCI scan at boot.
> +
>  ### pcid (x86)
>  > `= <boolean> | xpti=<bool>`
>  
> diff --git a/xen/arch/arm/include/asm/pci.h b/xen/arch/arm/include/asm/pci.h
> index 08ffcd4438..7289f7688b 100644
> --- a/xen/arch/arm/include/asm/pci.h
> +++ b/xen/arch/arm/include/asm/pci.h
> @@ -23,6 +23,7 @@
>  #define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
>  
>  extern bool pci_passthrough_enabled;
> +extern bool pci_scan_enabled;
>  
>  struct rangeset;
>  
> @@ -155,6 +156,8 @@ bool arch_pci_device_physdevop(void);
>  
>  #else   /*!CONFIG_HAS_PCI*/
>  
> +#define pci_scan_enabled false
> +
>  struct pci_dev;
>  
>  static inline void arch_pci_init_pdev(struct pci_dev *pdev) {}
> diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c
> index 487c545f3a..d3481b05eb 100644
> --- a/xen/arch/arm/pci/pci-host-common.c
> +++ b/xen/arch/arm/pci/pci-host-common.c
> @@ -284,6 +284,7 @@ pci_host_common_probe(struct dt_device_node *dev,
>      }
>  
>      pci_add_host_bridge(bridge);
> +    pci_add_segment(bridge->segment);
>  
>      return bridge;
>  
> diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
> index beb1f971fa..1b34e17517 100644
> --- a/xen/arch/arm/pci/pci.c
> +++ b/xen/arch/arm/pci/pci.c
> @@ -91,8 +91,14 @@ bool arch_pci_device_physdevop(void)
>  bool __read_mostly pci_passthrough_enabled;
>  boolean_param("pci-passthrough", pci_passthrough_enabled);
>  
> +/* By default pci scan is disabled. */
> +bool __ro_after_init pci_scan_enabled;
> +boolean_param("pci-scan", pci_scan_enabled);
> +
>  static int __init pci_init(void)
>  {
> +    int ret;
> +
>      /*
>       * Enable PCI passthrough when has been enabled explicitly
>       * (pci-passthrough=on).
> @@ -104,9 +110,23 @@ static int __init pci_init(void)
>          panic("Could not initialize PCI segment 0\n");
>  
>      if ( acpi_disabled )
> -        return dt_pci_init();
> +        ret = dt_pci_init();
>      else
> -        return acpi_pci_init();
> +        ret = acpi_pci_init();
> +
> +    if ( ret < 0 )
> +        return ret;
> +
> +    if ( pci_scan_enabled )
> +    {
> +        ret = scan_pci_devices();
> +
> +        if ( ret < 0 )
> +            return ret;
> +
> +    }
> +
> +    return 0;
>  }
>  __initcall(pci_init);
>  
> -- 
> 2.34.1
> 

[PATCH v2 2/3] xen/pci: modify pci_add_device to handle device add by Xen

[Mykyta Poturai]

From: Luca Fancellu <luca.fancellu@arm.com>

Currently pci_add_device is called from hypercalls requested by Dom0
to add pci devices and when the device has no domain associated with
it, it is assumed that hardware_domain is the owner.

On the dom0less scenario, the enumeration is performed by the
firmware and Xen at boot time might want to assign some pci devices
to guests, so it has to firstly add the device and then assign it to
the final guest.

Modify pci_add_device to have the owner domain passed as a parameter
to the function, so that when it is called from the hypercall the
owner would be the caller domain, otherwise when Xen is calling it,
it would be another domain since hw domain could not be there
(dom0less guests without Dom0 use case).

Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
(cherry picked from commit f0c85d9043f7c9402e85b73361c8a13c683428ca from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)

v1->v2:
* remove dom_io check
* fixup pci_add_device parameters
* use current->domain instead of hardware_domain

changes since cherry-pick:
* s/hardware_domain/d/ in some write_unlock calls in xen/drivers/passthrough/pci.c
---
 xen/arch/x86/physdev.c        |  9 ++++-----
 xen/drivers/passthrough/pci.c | 29 +++++++++++++++++++----------
 xen/drivers/pci/physdev.c     |  3 ++-
 xen/include/xen/pci.h         |  5 +++--
 4 files changed, 28 insertions(+), 18 deletions(-)

diff --git a/xen/arch/x86/physdev.c b/xen/arch/x86/physdev.c
index 4dfa1c0191..04d179e81b 100644
--- a/xen/arch/x86/physdev.c
+++ b/xen/arch/x86/physdev.c
@@ -446,8 +446,8 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         if ( copy_from_guest(&manage_pci, arg, 1) != 0 )
             break;
 
-        ret = pci_add_device(0, manage_pci.bus, manage_pci.devfn,
-                             NULL, NUMA_NO_NODE);
+        ret = pci_add_device(0, manage_pci.bus, manage_pci.devfn, NULL,
+                             NUMA_NO_NODE, current->domain);
         break;
     }
 
@@ -477,9 +477,8 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         pdev_info.is_virtfn = manage_pci_ext.is_virtfn;
         pdev_info.physfn.bus = manage_pci_ext.physfn.bus;
         pdev_info.physfn.devfn = manage_pci_ext.physfn.devfn;
-        ret = pci_add_device(0, manage_pci_ext.bus,
-                             manage_pci_ext.devfn,
-                             &pdev_info, NUMA_NO_NODE);
+        ret = pci_add_device(0, manage_pci_ext.bus, manage_pci_ext.devfn,
+                             &pdev_info, NUMA_NO_NODE, current->domain);
         break;
     }
 
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 3edcfa8a04..09b424d1b3 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -654,8 +654,9 @@ unsigned int pci_size_mem_bar(pci_sbdf_t sbdf, unsigned int pos,
     return is64bits ? 2 : 1;
 }
 
-int pci_add_device(u16 seg, u8 bus, u8 devfn,
-                   const struct pci_dev_info *info, nodeid_t node)
+int pci_add_device(uint16_t seg, uint8_t bus, uint8_t devfn,
+                   const struct pci_dev_info *info, nodeid_t node,
+                   struct domain *d)
 {
     struct pci_seg *pseg;
     struct pci_dev *pdev;
@@ -663,6 +664,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
     const char *type;
     int ret;
 
+    if ( !d )
+        return -EINVAL;
+
     if ( !info )
         type = "device";
     else if ( info->is_virtfn )
@@ -767,9 +771,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
     ret = 0;
     if ( !pdev->domain )
     {
-        pdev->domain = hardware_domain;
-        write_lock(&hardware_domain->pci_lock);
-        list_add(&pdev->domain_list, &hardware_domain->pdev_list);
+        pdev->domain = d;
+        write_lock(&d->pci_lock);
+        list_add(&pdev->domain_list, &pdev->domain->pdev_list);
 
         /*
          * For devices not discovered by Xen during boot, add vPCI handlers
@@ -779,25 +783,30 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
         if ( ret )
         {
             list_del(&pdev->domain_list);
-            write_unlock(&hardware_domain->pci_lock);
+            write_unlock(&d->pci_lock);
             pdev->domain = NULL;
             printk(XENLOG_ERR "Setup of vPCI failed: %d\n", ret);
             goto out;
         }
-        write_unlock(&hardware_domain->pci_lock);
+        write_unlock(&d->pci_lock);
         ret = iommu_add_device(pdev);
         if ( ret )
         {
-            write_lock(&hardware_domain->pci_lock);
+            write_lock(&d->pci_lock);
             vpci_deassign_device(pdev);
             list_del(&pdev->domain_list);
-            write_unlock(&hardware_domain->pci_lock);
+            write_unlock(&d->pci_lock);
             pdev->domain = NULL;
             goto out;
         }
     }
-    else
+    else if ( pdev->domain == d )
         iommu_enable_device(pdev);
+    else
+    {
+        ret = -EINVAL;
+        goto out;
+    }
 
     pci_enable_acs(pdev);
 
diff --git a/xen/drivers/pci/physdev.c b/xen/drivers/pci/physdev.c
index d46501b884..cd3a36903b 100644
--- a/xen/drivers/pci/physdev.c
+++ b/xen/drivers/pci/physdev.c
@@ -50,7 +50,8 @@ ret_t pci_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         }
 #endif
 
-        ret = pci_add_device(add.seg, add.bus, add.devfn, &pdev_info, node);
+        ret = pci_add_device(add.seg, add.bus, add.devfn,
+                             &pdev_info, node, current->domain);
         break;
     }
 
diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index 130c2a8c1a..89f3281b7c 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -226,8 +226,9 @@ void setup_hwdom_pci_devices(struct domain *d,
 int pci_release_devices(struct domain *d);
 int pci_add_segment(u16 seg);
 const unsigned long *pci_get_ro_map(u16 seg);
-int pci_add_device(u16 seg, u8 bus, u8 devfn,
-                   const struct pci_dev_info *info, nodeid_t node);
+int pci_add_device(uint16_t seg, uint8_t bus, uint8_t devfn,
+                   const struct pci_dev_info *info, nodeid_t node,
+                   struct domain *d);
 int pci_remove_device(u16 seg, u8 bus, u8 devfn);
 int pci_ro_device(int seg, int bus, int devfn);
 int pci_hide_device(unsigned int seg, unsigned int bus, unsigned int devfn);
-- 
2.34.1

Re: [PATCH v2 2/3] xen/pci: modify pci_add_device to handle device add by Xen

[Stefano Stabellini]

On Wed, 20 Aug 2025, Mykyta Poturai wrote:
> From: Luca Fancellu <luca.fancellu@arm.com>
> 
> Currently pci_add_device is called from hypercalls requested by Dom0
> to add pci devices and when the device has no domain associated with
> it, it is assumed that hardware_domain is the owner.
> 
> On the dom0less scenario, the enumeration is performed by the
> firmware and Xen at boot time might want to assign some pci devices
> to guests, so it has to firstly add the device and then assign it to
> the final guest.
> 
> Modify pci_add_device to have the owner domain passed as a parameter
> to the function, so that when it is called from the hypercall the
> owner would be the caller domain, otherwise when Xen is calling it,
> it would be another domain since hw domain could not be there
> (dom0less guests without Dom0 use case).
> 
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

Re: [PATCH v2 2/3] xen/pci: modify pci_add_device to handle device add by Xen

[Jan Beulich]

On 20.08.2025 14:28, Mykyta Poturai wrote:
> From: Luca Fancellu <luca.fancellu@arm.com>
> 
> Currently pci_add_device is called from hypercalls requested by Dom0
> to add pci devices and when the device has no domain associated with
> it, it is assumed that hardware_domain is the owner.
> 
> On the dom0less scenario, the enumeration is performed by the
> firmware and Xen at boot time might want to assign some pci devices
> to guests, so it has to firstly add the device and then assign it to
> the final guest.
> 
> Modify pci_add_device to have the owner domain passed as a parameter
> to the function, so that when it is called from the hypercall the
> owner would be the caller domain, otherwise when Xen is calling it,
> it would be another domain since hw domain could not be there
> (dom0less guests without Dom0 use case).
> 
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> (cherry picked from commit f0c85d9043f7c9402e85b73361c8a13c683428ca from
>  the downstream branch poc/pci-passthrough from
>  https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
> 
> v1->v2:
> * remove dom_io check
> * fixup pci_add_device parameters
> * use current->domain instead of hardware_domain

What I'm missing (as per my v1 comment) is discussion of the hardware_domain
-> current->domain change, including the XSM aspect. Because of the XSM aspect,
please also Cc the XSM maintainer going forward (I'm adding him here as well).

> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -654,8 +654,9 @@ unsigned int pci_size_mem_bar(pci_sbdf_t sbdf, unsigned int pos,
>      return is64bits ? 2 : 1;
>  }
>  
> -int pci_add_device(u16 seg, u8 bus, u8 devfn,
> -                   const struct pci_dev_info *info, nodeid_t node)
> +int pci_add_device(uint16_t seg, uint8_t bus, uint8_t devfn,
> +                   const struct pci_dev_info *info, nodeid_t node,
> +                   struct domain *d)
>  {
>      struct pci_seg *pseg;
>      struct pci_dev *pdev;
> @@ -663,6 +664,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>      const char *type;
>      int ret;
>  
> +    if ( !d )
> +        return -EINVAL;

This should't be needed. Very remotely ASSERT(d) could be added here, but
we don't normally do so elsewhere.

> @@ -767,9 +771,9 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
>      ret = 0;
>      if ( !pdev->domain )
>      {
> -        pdev->domain = hardware_domain;
> -        write_lock(&hardware_domain->pci_lock);
> -        list_add(&pdev->domain_list, &hardware_domain->pdev_list);
> +        pdev->domain = d;
> +        write_lock(&d->pci_lock);
> +        list_add(&pdev->domain_list, &pdev->domain->pdev_list);

Why pdev->domain instead of the shorter and more efficient d?

Jan

[PATCH v2 3/3] xen/pci: add discovered PCI device at boot

[Mykyta Poturai]

From: Luca Fancellu <luca.fancellu@arm.com>

In dom0less mode, there is no dom0 that can call PCI physdev ops to
register PCI devices to iommu, so it needs to be done by Xen.
pci_add_device requires some default domain, we don't have hwdom, and
the guests are not yet created during the PCI init phase, so use dom_io
as a temporary sentinel before devices are assigned to their target
domains.

Rename setup_hwdom_pci_devices to setup_pci_devices and add dom0less
handling to it.

In pci_add_device there is a call to xsm that doesn't consider the
requester of the function to be Xen itself, so add a check to skip
the call if the owner domain is dom_io, since it means the call is
coming directly from Xen.

Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
---
(cherry picked from commit eff51e50021b75f5a50533f7de681b2ce044f5bd from
 the downstream branch poc/pci-passthrough from
 https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)

v1->v2:
* integrate add_discovered_pci_devices into existing routines
* better explain the need for dom_io
---
 xen/arch/arm/pci/pci.c                      |  1 +
 xen/drivers/passthrough/amd/pci_amd_iommu.c |  2 +-
 xen/drivers/passthrough/pci.c               | 42 ++++++++++++++-------
 xen/drivers/passthrough/vtd/iommu.c         |  2 +-
 xen/include/xen/pci.h                       |  5 +--
 5 files changed, 33 insertions(+), 19 deletions(-)

diff --git a/xen/arch/arm/pci/pci.c b/xen/arch/arm/pci/pci.c
index 1b34e17517..909fbdd694 100644
--- a/xen/arch/arm/pci/pci.c
+++ b/xen/arch/arm/pci/pci.c
@@ -124,6 +124,7 @@ static int __init pci_init(void)
         if ( ret < 0 )
             return ret;
 
+        setup_pci_devices(dom_io, NULL);
     }
 
     return 0;
diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c b/xen/drivers/passthrough/amd/pci_amd_iommu.c
index 3a14770855..f3a83a0ab7 100644
--- a/xen/drivers/passthrough/amd/pci_amd_iommu.c
+++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c
@@ -401,7 +401,7 @@ static void __hwdom_init cf_check amd_iommu_hwdom_init(struct domain *d)
 
     /* Make sure workarounds are applied (if needed) before adding devices. */
     arch_iommu_hwdom_init(d);
-    setup_hwdom_pci_devices(d, amd_iommu_add_device);
+    setup_pci_devices(d, amd_iommu_add_device);
 }
 
 static void amd_iommu_disable_domain_device(const struct domain *domain,
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 09b424d1b3..6ddc6811df 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -35,6 +35,7 @@
 #include <xen/msi.h>
 #include <xsm/xsm.h>
 #include "ats.h"
+#include "xen/dom0less-build.h"
 
 struct pci_seg {
     struct list_head alldevs_list;
@@ -676,9 +677,12 @@ int pci_add_device(uint16_t seg, uint8_t bus, uint8_t devfn,
     else
         type = "device";
 
-    ret = xsm_resource_plug_pci(XSM_PRIV, (seg << 16) | (bus << 8) | devfn);
-    if ( ret )
-        return ret;
+    if ( d != dom_io )
+    {
+        ret = xsm_resource_plug_pci(XSM_PRIV, (seg << 16) | (bus << 8) | devfn);
+        if ( ret )
+            return ret;
+    }
 
     ret = -ENOMEM;
 
@@ -1181,19 +1185,21 @@ int __init scan_pci_devices(void)
     return ret;
 }
 
-struct setup_hwdom {
+struct setup_ctxt {
     struct domain *d;
     int (*handler)(uint8_t devfn, struct pci_dev *pdev);
 };
 
-static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt,
+static void __hwdom_init setup_one_pci_device(const struct setup_ctxt *ctxt,
                                                 struct pci_dev *pdev)
 {
     u8 devfn = pdev->devfn;
-    int err;
+    int err = 0;
 
     do {
-        err = ctxt->handler(devfn, pdev);
+        if ( ctxt->handler )
+            err = ctxt->handler(devfn, pdev);
+
         if ( err )
         {
             printk(XENLOG_ERR "setup %pp for d%d failed (%d)\n",
@@ -1213,10 +1219,10 @@ static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt,
                ctxt->d->domain_id, err);
 }
 
-static int __hwdom_init cf_check _setup_hwdom_pci_devices(
+static int __hwdom_init cf_check _setup_pci_devices(
     struct pci_seg *pseg, void *arg)
 {
-    struct setup_hwdom *ctxt = arg;
+    struct setup_ctxt *ctxt = arg;
     int bus, devfn;
 
     for ( bus = 0; bus < 256; bus++ )
@@ -1229,18 +1235,26 @@ static int __hwdom_init cf_check _setup_hwdom_pci_devices(
             if ( !pdev )
                 continue;
 
+            if ( is_dom0less_mode() ) {
+                int ret = pci_add_device(pdev->seg, pdev->bus, pdev->devfn, NULL,
+                                         NUMA_NO_NODE, ctxt->d);
+                if (ret)
+                    printk(XENLOG_ERR "Failed to add PCI device %pp: %d\n", &pdev->sbdf, ret);
+                continue;
+            }
+
             if ( !pdev->domain )
             {
                 pdev->domain = ctxt->d;
                 write_lock(&ctxt->d->pci_lock);
                 list_add(&pdev->domain_list, &ctxt->d->pdev_list);
                 write_unlock(&ctxt->d->pci_lock);
-                setup_one_hwdom_device(ctxt, pdev);
+                setup_one_pci_device(ctxt, pdev);
             }
             else if ( pdev->domain == dom_xen )
             {
                 pdev->domain = ctxt->d;
-                setup_one_hwdom_device(ctxt, pdev);
+                setup_one_pci_device(ctxt, pdev);
                 pdev->domain = dom_xen;
             }
             else if ( pdev->domain != ctxt->d )
@@ -1266,13 +1280,13 @@ static int __hwdom_init cf_check _setup_hwdom_pci_devices(
     return 0;
 }
 
-void __hwdom_init setup_hwdom_pci_devices(
+void __hwdom_init setup_pci_devices(
     struct domain *d, int (*handler)(uint8_t devfn, struct pci_dev *pdev))
 {
-    struct setup_hwdom ctxt = { .d = d, .handler = handler };
+    struct setup_ctxt ctxt = { .d = d, .handler = handler };
 
     pcidevs_lock();
-    pci_segments_iterate(_setup_hwdom_pci_devices, &ctxt);
+    pci_segments_iterate(_setup_pci_devices, &ctxt);
     pcidevs_unlock();
 }
 
diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c
index c55f02c97e..1096c16327 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1451,7 +1451,7 @@ static void __hwdom_init cf_check intel_iommu_hwdom_init(struct domain *d)
 {
     struct acpi_drhd_unit *drhd;
 
-    setup_hwdom_pci_devices(d, setup_hwdom_device);
+    setup_pci_devices(d, setup_hwdom_device);
     setup_hwdom_rmrr(d);
     /* Make sure workarounds are applied before enabling the IOMMU(s). */
     arch_iommu_hwdom_init(d);
diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index 89f3281b7c..61f69a8a1b 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -220,9 +220,8 @@ int scan_pci_devices(void);
 enum pdev_type pdev_type(u16 seg, u8 bus, u8 devfn);
 int find_upstream_bridge(u16 seg, u8 *bus, u8 *devfn, u8 *secbus);
 
-void setup_hwdom_pci_devices(struct domain *d,
-                             int (*handler)(uint8_t devfn,
-                                            struct pci_dev *pdev));
+void setup_pci_devices(struct domain *d, int (*handler)(uint8_t devfn,
+                                                        struct pci_dev *pdev));
 int pci_release_devices(struct domain *d);
 int pci_add_segment(u16 seg);
 const unsigned long *pci_get_ro_map(u16 seg);
-- 
2.34.1

Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot

[Jan Beulich]

On 20.08.2025 14:28, Mykyta Poturai wrote:
> From: Luca Fancellu <luca.fancellu@arm.com>
> 
> In dom0less mode, there is no dom0 that can call PCI physdev ops to
> register PCI devices to iommu, so it needs to be done by Xen.
> pci_add_device requires some default domain, we don't have hwdom, and
> the guests are not yet created during the PCI init phase, so use dom_io
> as a temporary sentinel before devices are assigned to their target
> domains.
> 
> Rename setup_hwdom_pci_devices to setup_pci_devices and add dom0less
> handling to it.
> 
> In pci_add_device there is a call to xsm that doesn't consider the
> requester of the function to be Xen itself, so add a check to skip
> the call if the owner domain is dom_io, since it means the call is
> coming directly from Xen.
> 
> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> (cherry picked from commit eff51e50021b75f5a50533f7de681b2ce044f5bd from
>  the downstream branch poc/pci-passthrough from
>  https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git)
> 
> v1->v2:
> * integrate add_discovered_pci_devices into existing routines
> * better explain the need for dom_io

What I continue to miss is an explanation of why devices can't go to their
ultimate "destination" domain right away (once those have been created),
i.e. why the dom_io intermediary is necessary in the first place.

> --- a/xen/drivers/passthrough/pci.c
> +++ b/xen/drivers/passthrough/pci.c
> @@ -35,6 +35,7 @@
>  #include <xen/msi.h>
>  #include <xsm/xsm.h>
>  #include "ats.h"
> +#include "xen/dom0less-build.h"

No, please don't, at the very least not this way (using quotes rather than
angle brackets). I may guess that it's for is_dom0less_mode(), but even
then I wonder whether that declaration wouldn't better move elsewhere. It
simply feels somewhat wrong to include this header here.

> @@ -1181,19 +1185,21 @@ int __init scan_pci_devices(void)
>      return ret;
>  }
>  
> -struct setup_hwdom {
> +struct setup_ctxt {
>      struct domain *d;
>      int (*handler)(uint8_t devfn, struct pci_dev *pdev);
>  };
>  
> -static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt,
> +static void __hwdom_init setup_one_pci_device(const struct setup_ctxt *ctxt,
>                                                  struct pci_dev *pdev)

Nit: Indentation then also needds to change on this following line.

>  {
>      u8 devfn = pdev->devfn;
> -    int err;
> +    int err = 0;

This doesn't suffice, as ...

>      do {
> -        err = ctxt->handler(devfn, pdev);
> +        if ( ctxt->handler )
> +            err = ctxt->handler(devfn, pdev);
> +
>          if ( err )
>          {
>              printk(XENLOG_ERR "setup %pp for d%d failed (%d)\n",

... below here we may continue the loop even if we got an error. "err"
needs setting unconditionally in the loop body, and hence maybe better
with a conditional expression.

> @@ -1229,18 +1235,26 @@ static int __hwdom_init cf_check _setup_hwdom_pci_devices(
>              if ( !pdev )
>                  continue;
>  
> +            if ( is_dom0less_mode() ) {

We're in a __hwdom_init function. You can't call an __init one from here.

Also nit (style): Brace placement.

> +                int ret = pci_add_device(pdev->seg, pdev->bus, pdev->devfn, NULL,
> +                                         NUMA_NO_NODE, ctxt->d);
> +                if (ret)

Nit (style): Missing blanks.

> --- a/xen/include/xen/pci.h
> +++ b/xen/include/xen/pci.h
> @@ -220,9 +220,8 @@ int scan_pci_devices(void);
>  enum pdev_type pdev_type(u16 seg, u8 bus, u8 devfn);
>  int find_upstream_bridge(u16 seg, u8 *bus, u8 *devfn, u8 *secbus);
>  
> -void setup_hwdom_pci_devices(struct domain *d,
> -                             int (*handler)(uint8_t devfn,
> -                                            struct pci_dev *pdev));
> +void setup_pci_devices(struct domain *d, int (*handler)(uint8_t devfn,
> +                                                        struct pci_dev *pdev));

I think in this case the 2nd parameter would better remain on the following
line, to limit overall indentation.

Jan

Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot

[Mykyta Poturai]

On 21.08.25 12:08, Jan Beulich wrote:
> On 20.08.2025 14:28, Mykyta Poturai wrote:
>> From: Luca Fancellu <luca.fancellu@arm.com>
>>
>> In dom0less mode, there is no dom0 that can call PCI physdev ops to
>> register PCI devices to iommu, so it needs to be done by Xen.
>> pci_add_device requires some default domain, we don't have hwdom, and
>> the guests are not yet created during the PCI init phase, so use dom_io
>> as a temporary sentinel before devices are assigned to their target
>> domains.
>>
>> Rename setup_hwdom_pci_devices to setup_pci_devices and add dom0less
>> handling to it.
>>
>> In pci_add_device there is a call to xsm that doesn't consider the
>> requester of the function to be Xen itself, so add a check to skip
>> the call if the owner domain is dom_io, since it means the call is
>> coming directly from Xen.
>>
>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>> ---
>> (cherry picked from commit eff51e50021b75f5a50533f7de681b2ce044f5bd from
>>   the downstream branch poc/pci-passthrough from
>>   https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git
>>
>> v1->v2:
>> * integrate add_discovered_pci_devices into existing routines
>> * better explain the need for dom_io
> 
> What I continue to miss is an explanation of why devices can't go to their
> ultimate "destination" domain right away (once those have been created),
> i.e. why the dom_io intermediary is necessary in the first place.
> 
> Jan

I've done some testing and indeed everything seems to work if we call 
pci_add_device directly during domain construction instead of 
reassigning them. Do you think this would be a better approach? If so 
then I guess this series needs to be dropped, and I will prepare a new 
one with direct device assignment to DomUs.

-- 
Mykyta

Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot

[Jan Beulich]

On 22.08.2025 10:03, Mykyta Poturai wrote:
> On 21.08.25 12:08, Jan Beulich wrote:
>> On 20.08.2025 14:28, Mykyta Poturai wrote:
>>> From: Luca Fancellu <luca.fancellu@arm.com>
>>>
>>> In dom0less mode, there is no dom0 that can call PCI physdev ops to
>>> register PCI devices to iommu, so it needs to be done by Xen.
>>> pci_add_device requires some default domain, we don't have hwdom, and
>>> the guests are not yet created during the PCI init phase, so use dom_io
>>> as a temporary sentinel before devices are assigned to their target
>>> domains.
>>>
>>> Rename setup_hwdom_pci_devices to setup_pci_devices and add dom0less
>>> handling to it.
>>>
>>> In pci_add_device there is a call to xsm that doesn't consider the
>>> requester of the function to be Xen itself, so add a check to skip
>>> the call if the owner domain is dom_io, since it means the call is
>>> coming directly from Xen.
>>>
>>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
>>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>>> ---
>>> (cherry picked from commit eff51e50021b75f5a50533f7de681b2ce044f5bd from
>>>   the downstream branch poc/pci-passthrough from
>>>   https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git
>>>
>>> v1->v2:
>>> * integrate add_discovered_pci_devices into existing routines
>>> * better explain the need for dom_io
>>
>> What I continue to miss is an explanation of why devices can't go to their
>> ultimate "destination" domain right away (once those have been created),
>> i.e. why the dom_io intermediary is necessary in the first place.
> 
> I've done some testing and indeed everything seems to work if we call 
> pci_add_device directly during domain construction instead of 
> reassigning them. Do you think this would be a better approach?

I think so, yes, but first and foremost you'll need Arm maintainer buyoff
on either approach (or yet another one).

Jan

> If so 
> then I guess this series needs to be dropped, and I will prepare a new 
> one with direct device assignment to DomUs.

Re: [PATCH v2 3/3] xen/pci: add discovered PCI device at boot

[Stefano Stabellini]

On Fri, 22 Aug 2025, Jan Beulich wrote:
> On 22.08.2025 10:03, Mykyta Poturai wrote:
> > On 21.08.25 12:08, Jan Beulich wrote:
> >> On 20.08.2025 14:28, Mykyta Poturai wrote:
> >>> From: Luca Fancellu <luca.fancellu@arm.com>
> >>>
> >>> In dom0less mode, there is no dom0 that can call PCI physdev ops to
> >>> register PCI devices to iommu, so it needs to be done by Xen.
> >>> pci_add_device requires some default domain, we don't have hwdom, and
> >>> the guests are not yet created during the PCI init phase, so use dom_io
> >>> as a temporary sentinel before devices are assigned to their target
> >>> domains.
> >>>
> >>> Rename setup_hwdom_pci_devices to setup_pci_devices and add dom0less
> >>> handling to it.
> >>>
> >>> In pci_add_device there is a call to xsm that doesn't consider the
> >>> requester of the function to be Xen itself, so add a check to skip
> >>> the call if the owner domain is dom_io, since it means the call is
> >>> coming directly from Xen.
> >>>
> >>> Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
> >>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> >>> ---
> >>> (cherry picked from commit eff51e50021b75f5a50533f7de681b2ce044f5bd from
> >>>   the downstream branch poc/pci-passthrough from
> >>>   https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc.git
> >>>
> >>> v1->v2:
> >>> * integrate add_discovered_pci_devices into existing routines
> >>> * better explain the need for dom_io
> >>
> >> What I continue to miss is an explanation of why devices can't go to their
> >> ultimate "destination" domain right away (once those have been created),
> >> i.e. why the dom_io intermediary is necessary in the first place.
> > 
> > I've done some testing and indeed everything seems to work if we call 
> > pci_add_device directly during domain construction instead of 
> > reassigning them. Do you think this would be a better approach?
> 
> I think so, yes, but first and foremost you'll need Arm maintainer buyoff
> on either approach (or yet another one).

I am OK with that in principle