Re: [PATCH v2 00/19] iommufd: Add VIOMMU infrastructure (Part-1)

Yi Liu
2024-09-25T10:27:18Z

Hi Nic,

On 2024/8/28 00:59, Nicolin Chen wrote:
> This series introduces a new VIOMMU infrastructure and related ioctls.
>
> IOMMUFD has been using the HWPT infrastructure for all cases, including a
> nested IO page table support. Yet, there're limitations for an HWPT-based
> structure to support some advanced HW-accelerated features, such as CMDQV
> on NVIDIA Grace, and HW-accelerated vIOMMU on AMD. Even for a multi-IOMMU
> environment, it is not straightforward for nested HWPTs to share the same
> parent HWPT (stage-2 IO pagetable), with the HWPT infrastructure alone.

could you elaborate a bit for the last sentence in the above paragraph?

>
> The new VIOMMU object is an additional layer, between the nested HWPT and
> its parent HWPT, to give to both the IOMMUFD core and an IOMMU driver an
> additional structure to support HW-accelerated feature:
> ----------------------------
> ---------------- | | paging_hwpt0 |
> | hwpt_nested0 |--->| viommu0 ------------------
> ---------------- | | HW-accel feats |
> ----------------------------
>
> On a multi-IOMMU system, the VIOMMU object can be instanced to the number
> of vIOMMUs in a guest VM, while holding the same parent HWPT to share the
> stage-2 IO pagetable. Each VIOMMU then just need to only allocate its own
> VMID to attach the shared stage-2 IO pagetable to the physical IOMMU:
> ----------------------------
> ---------------- | | paging_hwpt0 |
> | hwpt_nested0 |--->| viommu0 ------------------
> ---------------- | | VMID0 |
> ----------------------------
> ----------------------------
> ---------------- | | paging_hwpt0 |
> | hwpt_nested1 |--->| viommu1 ------------------
> ---------------- | | VMID1 |
> ----------------------------
>
> As an initial part-1, add ioctls to support a VIOMMU-based invalidation:
> IOMMUFD_CMD_VIOMMU_ALLOC to allocate a VIOMMU object
> IOMMUFD_CMD_VIOMMU_SET/UNSET_VDEV_ID to set/clear device's virtual ID
> (Resue IOMMUFD_CMD_HWPT_INVALIDATE for a VIOMMU object to flush cache
> by a given driver data)
>
> Worth noting that the VDEV_ID is for a per-VIOMMU device list for drivers
> to look up the device's physical instance from its virtual ID in a VM. It
> is essential for a VIOMMU-based invalidation where the request contains a
> device's virtual ID for its device cache flush, e.g. ATC invalidation.
>
> As for the implementation of the series, add an IOMMU_VIOMMU_TYPE_DEFAULT
> type for a core-allocated-core-managed VIOMMU object, allowing drivers to
> simply hook a default viommu ops for viommu-based invalidation alone. And
> provide some viommu helpers to drivers for VDEV_ID translation and parent
> domain lookup. Add VIOMMU invalidation support to ARM SMMUv3 driver for a
> real world use case. This adds supports of arm-smmuv-v3's CMDQ_OP_ATC_INV
> and CMDQ_OP_CFGI_CD/ALL commands, supplementing HWPT-based invalidations.
>
> In the future, drivers will also be able to choose a driver-managed type
> to hold its own structure by adding a new type to enum iommu_viommu_type.
> More VIOMMU-based structures and ioctls will be introduced in part-2/3 to
> support a driver-managed VIOMMU, e.g. VQUEUE object for a HW accelerated
> queue, VIRQ (or VEVENT) object for IRQ injections. Although we repurposed
> the VIOMMU object from an earlier RFC discussion, for a referece:
> https://lore.kernel.org/all/cover.1712978212.git.nicolinc@nvidia.com/
>
> This series is on Github:
> https://github.com/nicolinc/iommufd/commits/iommufd_viommu_p1-v2
> Paring QEMU branch for testing:
> https://github.com/nicolinc/qemu/commits/wip/for_iommufd_viommu_p1-v2
>
> Changelog
> v2
> * Limited vdev_id to one per idev
> * Added a rw_sem to protect the vdev_id list
> * Reworked driver-level APIs with proper lockings
> * Added a new viommu_api file for IOMMUFD_DRIVER config
> * Dropped useless iommu_dev point from the viommu structure
> * Added missing index numnbers to new types in the uAPI header
> * Dropped IOMMU_VIOMMU_INVALIDATE uAPI; Instead, reuse the HWPT one
> * Reworked mock_viommu_cache_invalidate() using the new iommu helper
> * Reordered details of set/unset_vdev_id handlers for proper lockings
> * Added arm_smmu_cache_invalidate_user patch from Jason's nesting series
> v1
> https://lore.kernel.org/all/cover.1723061377.git.nicolinc@nvidia.com/
>
> Thanks!
> Nicolin
>
> Jason Gunthorpe (3):
> iommu: Add iommu_copy_struct_from_full_user_array helper
> iommu/arm-smmu-v3: Allow ATS for IOMMU_DOMAIN_NESTED
> iommu/arm-smmu-v3: Update comments about ATS and bypass
>
> Nicolin Chen (16):
> iommufd: Reorder struct forward declarations
> iommufd/viommu: Add IOMMUFD_OBJ_VIOMMU and IOMMU_VIOMMU_ALLOC ioctl
> iommu: Pass in a viommu pointer to domain_alloc_user op
> iommufd: Allow pt_id to carry viommu_id for IOMMU_HWPT_ALLOC
> iommufd/selftest: Add IOMMU_VIOMMU_ALLOC test coverage
> iommufd/viommu: Add IOMMU_VIOMMU_SET/UNSET_VDEV_ID ioctl
> iommufd/selftest: Add IOMMU_VIOMMU_SET/UNSET_VDEV_ID test coverage
> iommufd/viommu: Add cache_invalidate for IOMMU_VIOMMU_TYPE_DEFAULT
> iommufd: Allow hwpt_id to carry viommu_id for IOMMU_HWPT_INVALIDATE
> iommufd/viommu: Add vdev_id helpers for IOMMU drivers
> iommufd/selftest: Add mock_viommu_invalidate_user op
> iommufd/selftest: Add IOMMU_TEST_OP_DEV_CHECK_CACHE test command
> iommufd/selftest: Add VIOMMU coverage for IOMMU_HWPT_INVALIDATE ioctl
> iommufd/viommu: Add iommufd_viommu_to_parent_domain helper
> iommu/arm-smmu-v3: Add arm_smmu_cache_invalidate_user
> iommu/arm-smmu-v3: Add arm_smmu_viommu_cache_invalidate
>
> drivers/iommu/amd/iommu.c | 1 +
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 218 ++++++++++++++-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 +
> drivers/iommu/intel/iommu.c | 1 +
> drivers/iommu/iommufd/Makefile | 5 +-
> drivers/iommu/iommufd/device.c | 12 +
> drivers/iommu/iommufd/hw_pagetable.c | 59 +++-
> drivers/iommu/iommufd/iommufd_private.h | 37 +++
> drivers/iommu/iommufd/iommufd_test.h | 30 ++
> drivers/iommu/iommufd/main.c | 12 +
> drivers/iommu/iommufd/selftest.c | 101 ++++++-
> drivers/iommu/iommufd/viommu.c | 196 +++++++++++++
> drivers/iommu/iommufd/viommu_api.c | 53 ++++
> include/linux/iommu.h | 56 +++-
> include/linux/iommufd.h | 51 +++-
> include/uapi/linux/iommufd.h | 117 +++++++-
> tools/testing/selftests/iommu/iommufd.c | 259 +++++++++++++++++-
> tools/testing/selftests/iommu/iommufd_utils.h | 126 +++++++++
> 18 files changed, 1299 insertions(+), 38 deletions(-)
> create mode 100644 drivers/iommu/iommufd/viommu.c
> create mode 100644 drivers/iommu/iommufd/viommu_api.c
>

--
Regards,
Yi Liu