qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v5 00/11] hw/arm/virt: Add support for user creatable SMMUv3 device
@ 2025-06-23  9:42 Shameer Kolothum via
  2025-06-23  9:42 ` [PATCH v5 01/11] hw/arm/smmu-common: Check SMMU has PCIe Root Complex association Shameer Kolothum via
                   ` (12 more replies)
  0 siblings, 13 replies; 39+ messages in thread
From: Shameer Kolothum via @ 2025-06-23  9:42 UTC (permalink / raw)
  To: qemu-arm, qemu-devel
  Cc: eric.auger, peter.maydell, jgg, nicolinc, ddutile, berrange,
	imammedo, nathanc, mochs, smostafa, gustavo.romero, linuxarm,
	wangzhou1, jiangkunkun, jonathan.cameron, zhangfei.gao

Hi All,

Changes from v4:
https://lore.kernel.org/qemu-devel/20250613144449.60156-1-shameerali.kolothum.thodi@huawei.com/

Major changes from v4:

1. Added stricter validation for PCI buses associated with the SMMU.
   Only the default PCIe Root Complex (pcie.0) and additional root
   complexes created using pxb-pcie (see patch #1) are allowed.

2. While testing this series with a setup involving multiple PCIe root
   complexes using pxb-pcie, I encountered an issue related to IOMMU
   ops resolution. Consider the below configuration, where an
   arm-smmuv3 device is associated with the default root complex pcie.0,
   and an additional pxb-pcie-based root complex (pcie.1) is added
   without any associated SMMU:

   -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.1 \
   ...
   -device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \
   -device pcie-root-port,id=pcie.port1,chassis=2,bus=pcie.1 \
   -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.2 \
   ...
   -device virtio-net-pci,bus=pcie.0,netdev=net0,id=virtionet.0 \
   -device virtio-net-pci,bus=pcie.port1,netdev=net1,id=virtionet.1

   The guest boots fine, and virtionet.0(behind the SMMUV3) bring up
   is successful. However, attempting to bring up virtionet.1
   (behind pcie.1, which lacks a connected SMMU) results in a failure:

   root@ubuntu:/# dhclient enp9s0
   arm-smmu-v3 arm-smmu-v3.0.auto: event 0x02 received:
   arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000090000000002
   arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000
   arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000
   arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000
   arm-smmu-v3 arm-smmu-v3.0.auto: event: C_BAD_STREAMID client: (unassigned sid) sid: 0x900 ssid: 0x0
   virtio_net virtio1 enp9s0: NETDEV WATCHDOG: CPU: 2: transmit queue 0 timed out 5172 ms
   virtio_net virtio1 enp9s0: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 5172000 usecs ago
   ...

   Debug shows that QEMU currently registers IOMMU ops for bus using
   pci_setup_iommu(). However, when retrieving IOMMU ops for a device
   via pci_device_get_iommu_bus_devfn(), the function walks up to the 
   parent_dev and fetches the IOMMU ops from the parent, even if the
   current root bus has none configured.

   This works today because existing IOMMU models in QEMU are globally 
   scoped, and pxb-pcie based extra root complexes can use the
   bypass_iommu property to skip translation as needed.

   However, with this series introducing support for associating
   arm-smmuv3 devices with specific PCIe root complexes, this
   becomes problematic. In QEMU, pxb-pcie is implemented as a synthetic
   root complex whose parent_dev is pcie.0. As a result, even though
   pcie.1 has no SMMU attached, pci_device_get_iommu_bus_devfn()
   incorrectly returns the IOMMU ops associated with pcie.0 due to
   the fallback mechanism via parent_dev. This causes devices on
   pcie.1 to erroneously use the address space from pcie.0's SMMU,
   leading to failures like the one above.

   To address this, patch #6 in the series introduces a new helper 
   function pci_setup_iommu_per_bus(), which explicitly sets the 
   iommu_per_bus field in the PCIBus structure. This allows 
   pci_device_get_iommu_bus_devfn() to retrieve IOMMU ops based 
   on the specific bus.

   Not sure this is the correct approach or not. If there is a better
   way to handle this, please let me know .

3. Picked up few R-by tags where the patch content has not changed much.

4. Dropped T-by from Nathan for some patches as things have changed a bit.
   @Nathan, apprecaite if you have time to rerun the tests.

5. Added a bios table tests for both legacy SMMUv3 and new SMMMv3 devices.
   See last few patches.

Cover letter:

This patch series introduces support for a user-creatable SMMUv3 device
(-device arm-smmuv3) in QEMU.

The implementation is based on feedback received from the RFCv2[0]:
"hw/arm/virt: Add support for user-creatable accelerated SMMUv3"

Currently, QEMU's SMMUv3 emulation (iommu=smmuv3) is tied to the machine
and does not support instantiating multiple SMMUv3 devices—each associated
with a separate PCIe root complex. In contrast, real-world ARM systems
often include multiple SMMUv3 instances, each bound to a different PCIe
root complex.

This series allows to specify multiple SMMUv3 instances as below,

 -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0
  ...
 -device arm-smmuv3,primary-bus=pcie.1,,id=smmuv3.1

The multiple SMMUv3 instance support lays the groundwork for supporting
accelerated SMMUv3, as proposed in the aforementioned RFCv2[0]. The
proposed accelerated support will be an optional property like below,
-device arm-smmuv3,primary-bus=pcie.1,accel=on,..

Please note, the accelerated SMMUv3 support is not part of this series
and will be sent out as a separate series later on top of this one.

This series also,

-Supports either the legacy iommu=smmuv3 option or the new
  "-device arm-smmuv3" model.
  -Adds device tree bindings for the new SMMUv3 device on the arm/virt
   machine only, and only for the default pcie.0 root complex.
   (Note: pxb-pcie root complexes are currently not supported with the
    device tree due to known limitations[1].)

Example usage:
  -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0
  -device virtio-net-pci,bus=pcie.0
  -device pxb-pcie,id=pcie.1,bus_nr=2
  -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1
  -device pcie-root-port,id=pcie.port1,bus=pcie.1
  -device virtio-net-pci,bus=pcie.port1

Please take a look and let me know your feedback.

Thanks,
Shameer
[0]:https://lore.kernel.org/qemu-devel/20250311141045.66620-1-shameerali.kolothum.thodi@huawei.com/
[1]:https://lore.kernel.org/qemu-devel/20230421165037.2506-1-Jonathan.Cameron@huawei.com/

Nicolin Chen (1):
  hw/arm/virt: Add an SMMU_IO_LEN macro

Shameer Kolothum (10):
  hw/arm/smmu-common: Check SMMU has PCIe Root Complex association
  hw/arm/virt-acpi-build: Re-arrange SMMUv3 IORT build
  hw/arm/virt-acpi-build: Update IORT for multiple smmuv3 devices
  hw/arm/virt: Factor out common SMMUV3 dt bindings code
  hw/pci: Introduce pci_setup_iommu_per_bus() for per-bus IOMMU ops
    retrieval
  hw/arm/virt: Allow user-creatable SMMUv3 dev instantiation
  qemu-options.hx: Document the arm-smmuv3 device
  bios-tables-test: Allow for smmuv3 test data.
  qtest/bios-tables-test: Add tests for legacy smmuv3 and smmuv3 device
  qtest/bios-tables-test: Update tables for smmuv3 tests

 hw/arm/smmu-common.c                          |  35 +++-
 hw/arm/smmuv3.c                               |   2 +
 hw/arm/virt-acpi-build.c                      | 164 ++++++++++++++----
 hw/arm/virt.c                                 | 110 +++++++++---
 hw/core/sysbus-fdt.c                          |   3 +
 hw/pci-bridge/pci_expander_bridge.c           |   1 -
 hw/pci/pci.c                                  |  25 +++
 include/hw/arm/smmu-common.h                  |   1 +
 include/hw/arm/virt.h                         |   1 +
 include/hw/pci/pci.h                          |   2 +
 include/hw/pci/pci_bridge.h                   |   1 +
 include/hw/pci/pci_bus.h                      |   1 +
 qemu-options.hx                               |   7 +
 tests/data/acpi/aarch64/virt/DSDT.smmuv3-dev  | Bin 0 -> 10162 bytes
 .../data/acpi/aarch64/virt/DSDT.smmuv3-legacy | Bin 0 -> 10162 bytes
 tests/data/acpi/aarch64/virt/IORT.smmuv3-dev  | Bin 0 -> 364 bytes
 .../data/acpi/aarch64/virt/IORT.smmuv3-legacy | Bin 0 -> 276 bytes
 tests/qtest/bios-tables-test.c                |  86 +++++++++
 18 files changed, 375 insertions(+), 64 deletions(-)
 create mode 100644 tests/data/acpi/aarch64/virt/DSDT.smmuv3-dev
 create mode 100644 tests/data/acpi/aarch64/virt/DSDT.smmuv3-legacy
 create mode 100644 tests/data/acpi/aarch64/virt/IORT.smmuv3-dev
 create mode 100644 tests/data/acpi/aarch64/virt/IORT.smmuv3-legacy

-- 
2.34.1



^ permalink raw reply	[flat|nested] 39+ messages in thread

end of thread, other threads:[~2025-07-02 15:09 UTC | newest]

Thread overview: 39+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-06-23  9:42 [PATCH v5 00/11] hw/arm/virt: Add support for user creatable SMMUv3 device Shameer Kolothum via
2025-06-23  9:42 ` [PATCH v5 01/11] hw/arm/smmu-common: Check SMMU has PCIe Root Complex association Shameer Kolothum via
2025-06-23 11:32   ` Jonathan Cameron via
2025-06-27 11:52   ` Eric Auger
2025-06-30  7:01     ` Shameerali Kolothum Thodi via
2025-07-01  6:31       ` Eric Auger
2025-06-23  9:42 ` [PATCH v5 02/11] hw/arm/virt-acpi-build: Re-arrange SMMUv3 IORT build Shameer Kolothum via
2025-06-27 11:54   ` Eric Auger
2025-06-23  9:42 ` [PATCH v5 03/11] hw/arm/virt-acpi-build: Update IORT for multiple smmuv3 devices Shameer Kolothum via
2025-06-23  9:42 ` [PATCH v5 04/11] hw/arm/virt: Factor out common SMMUV3 dt bindings code Shameer Kolothum via
2025-06-23  9:42 ` [PATCH v5 05/11] hw/arm/virt: Add an SMMU_IO_LEN macro Shameer Kolothum via
2025-06-23 11:35   ` Jonathan Cameron via
2025-06-23  9:42 ` [PATCH v5 06/11] hw/pci: Introduce pci_setup_iommu_per_bus() for per-bus IOMMU ops retrieval Shameer Kolothum via
2025-06-23 11:39   ` Jonathan Cameron via
2025-06-27 12:04   ` Eric Auger
2025-06-30  7:05     ` Shameerali Kolothum Thodi via
2025-06-23  9:42 ` [PATCH v5 07/11] hw/arm/virt: Allow user-creatable SMMUv3 dev instantiation Shameer Kolothum via
2025-06-23 11:46   ` Jonathan Cameron via
2025-06-27 12:05   ` Eric Auger
2025-06-23  9:42 ` [PATCH v5 08/11] qemu-options.hx: Document the arm-smmuv3 device Shameer Kolothum via
2025-06-23 11:47   ` Jonathan Cameron via
2025-06-27 12:08   ` Eric Auger
2025-06-23  9:42 ` [PATCH v5 09/11] bios-tables-test: Allow for smmuv3 test data Shameer Kolothum via
2025-06-23 11:49   ` Jonathan Cameron via
2025-06-27 12:14   ` Eric Auger
2025-06-23  9:42 ` [PATCH v5 10/11] qtest/bios-tables-test: Add tests for legacy smmuv3 and smmuv3 device Shameer Kolothum via
2025-06-23 11:57   ` Jonathan Cameron via
2025-06-27 12:34   ` Eric Auger
2025-06-30  7:08     ` Shameerali Kolothum Thodi via
2025-06-23  9:42 ` [PATCH v5 11/11] qtest/bios-tables-test: Update tables for smmuv3 tests Shameer Kolothum via
2025-06-23 12:00   ` Jonathan Cameron via
2025-06-27 12:36   ` Eric Auger
2025-06-30  7:11     ` Shameerali Kolothum Thodi via
2025-07-01  6:35       ` Eric Auger
2025-06-27 12:36 ` [PATCH v5 00/11] hw/arm/virt: Add support for user creatable SMMUv3 device Eric Auger
2025-06-30  7:12   ` Shameerali Kolothum Thodi via
2025-07-01  6:37     ` Eric Auger
2025-07-02  1:01 ` Nathan Chen
2025-07-02 15:08   ` Shameerali Kolothum Thodi via

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).