Re: [PATCH RFC v2 00/13] IOMMUFD Generic interface

[Eric Auger <eric.auger@redhat.com>]

Hi,

On 9/13/22 03:55, Tian, Kevin wrote:
> We didn't close the open of how to get this merged in LPC due to the
> audio issue. Then let's use mails.
>
> Overall there are three options on the table:
>
> 1) Require vfio-compat to be 100% compatible with vfio-type1
>
>    Probably not a good choice given the amount of work to fix the remaining
>    gaps. And this will block support of new IOMMU features for a longer time.
>
> 2) Leave vfio-compat as what it is in this series
>
>    Treat it as a vehicle to validate the iommufd logic instead of immediately
>    replacing vfio-type1. Functionally most vfio applications can work w/o
>    change if putting aside the difference on locked mm accounting, p2p, etc.
>
>    Then work on new features and 100% vfio-type1 compat. in parallel.
>
> 3) Focus on iommufd native uAPI first
>
>    Require vfio_device cdev and adoption in Qemu. Only for new vfio app.
>
>    Then work on new features and vfio-compat in parallel.
>
> I'm fine with either 2) or 3). Per a quick chat with Alex he prefers to 3).

I am also inclined to pursue 3) as this was the initial Jason's guidance
and pre-requisite to integrate new features. In the past we concluded
vfio-compat would mostly be used for testing purpose. Our QEMU
integration fully is based on device based API.

Thanks

Eric
>
> Jason, how about your opinion?
>
> Thanks
> Kevin
>
>> From: Jason Gunthorpe <jgg@nvidia.com>
>> Sent: Saturday, September 3, 2022 3:59 AM
>>
>> iommufd is the user API to control the IOMMU subsystem as it relates to
>> managing IO page tables that point at user space memory.
>>
>> It takes over from drivers/vfio/vfio_iommu_type1.c (aka the VFIO
>> container) which is the VFIO specific interface for a similar idea.
>>
>> We see a broad need for extended features, some being highly IOMMU
>> device
>> specific:
>>  - Binding iommu_domain's to PASID/SSID
>>  - Userspace page tables, for ARM, x86 and S390
>>  - Kernel bypass'd invalidation of user page tables
>>  - Re-use of the KVM page table in the IOMMU
>>  - Dirty page tracking in the IOMMU
>>  - Runtime Increase/Decrease of IOPTE size
>>  - PRI support with faults resolved in userspace
>>
>> As well as a need to access these features beyond just VFIO, from VDPA for
>> instance. Other classes of accelerator HW are touching on these areas now
>> too.
>>
>> The pre-v1 series proposed re-using the VFIO type 1 data structure,
>> however it was suggested that if we are doing this big update then we
>> should also come with an improved data structure that solves the
>> limitations that VFIO type1 has. Notably this addresses:
>>
>>  - Multiple IOAS/'containers' and multiple domains inside a single FD
>>
>>  - Single-pin operation no matter how many domains and containers use
>>    a page
>>
>>  - A fine grained locking scheme supporting user managed concurrency for
>>    multi-threaded map/unmap
>>
>>  - A pre-registration mechanism to optimize vIOMMU use cases by
>>    pre-pinning pages
>>
>>  - Extended ioctl API that can manage these new objects and exposes
>>    domains directly to user space
>>
>>  - domains are sharable between subsystems, eg VFIO and VDPA
>>
>> The bulk of this code is a new data structure design to track how the
>> IOVAs are mapped to PFNs.
>>
>> iommufd intends to be general and consumable by any driver that wants to
>> DMA to userspace. From a driver perspective it can largely be dropped in
>> in-place of iommu_attach_device() and provides a uniform full feature set
>> to all consumers.
>>
>> As this is a larger project this series is the first step. This series
>> provides the iommfd "generic interface" which is designed to be suitable
>> for applications like DPDK and VMM flows that are not optimized to
>> specific HW scenarios. It is close to being a drop in replacement for the
>> existing VFIO type 1.
>>
>> Several follow-on series are being prepared:
>>
>> - Patches integrating with qemu in native mode:
>>   https://github.com/yiliu1765/qemu/commits/qemu-iommufd-6.0-rc2
>>
>> - A completed integration with VFIO now exists that covers "emulated" mdev
>>   use cases now, and can pass testing with qemu/etc in compatability mode:
>>   https://github.com/jgunthorpe/linux/commits/vfio_iommufd
>>
>> - A draft providing system iommu dirty tracking on top of iommufd,
>>   including iommu driver implementations:
>>   https://github.com/jpemartins/linux/commits/x86-iommufd
>>
>>   This pairs with patches for providing a similar API to support VFIO-device
>>   tracking to give a complete vfio solution:
>>   https://lore.kernel.org/kvm/20220901093853.60194-1-yishaih@nvidia.com/
>>
>> - Userspace page tables aka 'nested translation' for ARM and Intel iommu
>>   drivers:
>>   https://github.com/nicolinc/iommufd/commits/iommufd_nesting
>>
>> - "device centric" vfio series to expose the vfio_device FD directly as a
>>   normal cdev, and provide an extended API allowing dynamically changing
>>   the IOAS binding:
>>   https://github.com/yiliu1765/iommufd/commits/iommufd-v6.0-rc2-
>> nesting-0901
>>
>> - Drafts for PASID and PRI interfaces are included above as well
>>
>> Overall enough work is done now to show the merit of the new API design
>> and at least draft solutions to many of the main problems.
>>
>> Several people have contributed directly to this work: Eric Auger, Joao
>> Martins, Kevin Tian, Lu Baolu, Nicolin Chen, Yi L Liu. Many more have
>> participated in the discussions that lead here, and provided ideas. Thanks
>> to all!
>>
>> The v1 iommufd series has been used to guide a large amount of preparatory
>> work that has now been merged. The general theme is to organize things in
>> a way that makes injecting iommufd natural:
>>
>>  - VFIO live migration support with mlx5 and hisi_acc drivers.
>>    These series need a dirty tracking solution to be really usable.
>>    https://lore.kernel.org/kvm/20220224142024.147653-1-
>> yishaih@nvidia.com/
>>    https://lore.kernel.org/kvm/20220308184902.2242-1-
>> shameerali.kolothum.thodi@huawei.com/
>>
>>  - Significantly rework the VFIO gvt mdev and remove struct
>>    mdev_parent_ops
>>    https://lore.kernel.org/lkml/20220411141403.86980-1-hch@lst.de/
>>
>>  - Rework how PCIe no-snoop blocking works
>>    https://lore.kernel.org/kvm/0-v3-2cf356649677+a32-
>> intel_no_snoop_jgg@nvidia.com/
>>
>>  - Consolidate dma ownership into the iommu core code
>>    https://lore.kernel.org/linux-iommu/20220418005000.897664-1-
>> baolu.lu@linux.intel.com/
>>
>>  - Make all vfio driver interfaces use struct vfio_device consistently
>>    https://lore.kernel.org/kvm/0-v4-8045e76bf00b+13d-
>> vfio_mdev_no_group_jgg@nvidia.com/
>>
>>  - Remove the vfio_group from the kvm/vfio interface
>>    https://lore.kernel.org/kvm/0-v3-f7729924a7ea+25e33-
>> vfio_kvm_no_group_jgg@nvidia.com/
>>
>>  - Simplify locking in vfio
>>    https://lore.kernel.org/kvm/0-v2-d035a1842d81+1bf-
>> vfio_group_locking_jgg@nvidia.com/
>>
>>  - Remove the vfio notifiter scheme that faces drivers
>>    https://lore.kernel.org/kvm/0-v4-681e038e30fd+78-
>> vfio_unmap_notif_jgg@nvidia.com/
>>
>>  - Improve the driver facing API for vfio pin/unpin pages to make the
>>    presence of struct page clear
>>    https://lore.kernel.org/kvm/20220723020256.30081-1-
>> nicolinc@nvidia.com/
>>
>>  - Clean up in the Intel IOMMU driver
>>    https://lore.kernel.org/linux-iommu/20220301020159.633356-1-
>> baolu.lu@linux.intel.com/
>>    https://lore.kernel.org/linux-iommu/20220510023407.2759143-1-
>> baolu.lu@linux.intel.com/
>>    https://lore.kernel.org/linux-iommu/20220514014322.2927339-1-
>> baolu.lu@linux.intel.com/
>>    https://lore.kernel.org/linux-iommu/20220706025524.2904370-1-
>> baolu.lu@linux.intel.com/
>>    https://lore.kernel.org/linux-iommu/20220702015610.2849494-1-
>> baolu.lu@linux.intel.com/
>>
>>  - Rework s390 vfio drivers
>>    https://lore.kernel.org/kvm/20220707135737.720765-1-
>> farman@linux.ibm.com/
>>
>>  - Normalize vfio ioctl handling
>>    https://lore.kernel.org/kvm/0-v2-0f9e632d54fb+d6-
>> vfio_ioctl_split_jgg@nvidia.com/
>>
>> This is about 168 patches applied since March, thank you to everyone
>> involved in all this work!
>>
>> Currently there are a number of supporting series still in progress:
>>  - Simplify and consolidate iommu_domain/device compatability checking
>>    https://lore.kernel.org/linux-iommu/20220815181437.28127-1-
>> nicolinc@nvidia.com/
>>
>>  - Align iommu SVA support with the domain-centric model
>>    https://lore.kernel.org/linux-iommu/20220826121141.50743-1-
>> baolu.lu@linux.intel.com/
>>
>>  - VFIO API for dirty tracking (aka dma logging) managed inside a PCI
>>    device, with mlx5 implementation
>>    https://lore.kernel.org/kvm/20220901093853.60194-1-yishaih@nvidia.com
>>
>>  - Introduce a struct device sysfs presence for struct vfio_device
>>    https://lore.kernel.org/kvm/20220901143747.32858-1-
>> kevin.tian@intel.com/
>>
>>  - Complete restructuring the vfio mdev model
>>    https://lore.kernel.org/kvm/20220822062208.152745-1-hch@lst.de/
>>
>>  - DMABUF exporter support for VFIO to allow PCI P2P with VFIO
>>    https://lore.kernel.org/r/0-v2-472615b3877e+28f7-
>> vfio_dma_buf_jgg@nvidia.com
>>
>>  - Isolate VFIO container code in preperation for iommufd to provide an
>>    alternative implementation of it all
>>    https://lore.kernel.org/kvm/0-v1-a805b607f1fb+17b-
>> vfio_container_split_jgg@nvidia.com
>>
>>  - Start to provide iommu_domain ops for power
>>    https://lore.kernel.org/all/20220714081822.3717693-1-aik@ozlabs.ru/
>>
>> Right now there is no more preperatory work sketched out, so this is the
>> last of it.
>>
>> This series remains RFC as there are still several important FIXME's to
>> deal with first, but things are on track for non-RFC in the near future.
>>
>> This is on github: https://github.com/jgunthorpe/linux/commits/iommufd
>>
>> v2:
>>  - Rebase to v6.0-rc3
>>  - Improve comments
>>  - Change to an iterative destruction approach to avoid cycles
>>  - Near rewrite of the vfio facing implementation, supported by a complete
>>    implementation on the vfio side
>>  - New IOMMU_IOAS_ALLOW_IOVAS API as discussed. Allows userspace to
>>    assert that ranges of IOVA must always be mappable. To be used by a
>> VMM
>>    that has promised a guest a certain availability of IOVA. May help
>>    guide PPC's multi-window implementation.
>>  - Rework how unmap_iova works, user can unmap the whole ioas now
>>  - The no-snoop / wbinvd support is implemented
>>  - Bug fixes
>>  - Test suite improvements
>>  - Lots of smaller changes (the interdiff is 3k lines)
>> v1: https://lore.kernel.org/r/0-v1-e79cd8d168e8+6-
>> iommufd_jgg@nvidia.com
>>
>> # S390 in-kernel page table walker
>> Cc: Niklas Schnelle <schnelle@linux.ibm.com>
>> Cc: Matthew Rosato <mjrosato@linux.ibm.com>
>> # AMD Dirty page tracking
>> Cc: Joao Martins <joao.m.martins@oracle.com>
>> # ARM SMMU Dirty page tracking
>> Cc: Keqian Zhu <zhukeqian1@huawei.com>
>> Cc: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
>> # ARM SMMU nesting
>> Cc: Eric Auger <eric.auger@redhat.com>
>> Cc: Jean-Philippe Brucker <jean-philippe@linaro.org>
>> # Map/unmap performance
>> Cc: Daniel Jordan <daniel.m.jordan@oracle.com>
>> # VDPA
>> Cc: "Michael S. Tsirkin" <mst@redhat.com>
>> Cc: Jason Wang <jasowang@redhat.com>
>> # Power
>> Cc: David Gibson <david@gibson.dropbear.id.au>
>> # vfio
>> Cc: Alex Williamson <alex.williamson@redhat.com>
>> Cc: Cornelia Huck <cohuck@redhat.com>
>> Cc: kvm@vger.kernel.org
>> # iommu
>> Cc: iommu@lists.linux.dev
>> # Collaborators
>> Cc: "Chaitanya Kulkarni" <chaitanyak@nvidia.com>
>> Cc: Nicolin Chen <nicolinc@nvidia.com>
>> Cc: Lu Baolu <baolu.lu@linux.intel.com>
>> Cc: Kevin Tian <kevin.tian@intel.com>
>> Cc: Yi Liu <yi.l.liu@intel.com>
>> # s390
>> Cc: Eric Farman <farman@linux.ibm.com>
>> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
>>
>> Jason Gunthorpe (12):
>>   interval-tree: Add a utility to iterate over spans in an interval tree
>>   iommufd: File descriptor, context, kconfig and makefiles
>>   kernel/user: Allow user::locked_vm to be usable for iommufd
>>   iommufd: PFN handling for iopt_pages
>>   iommufd: Algorithms for PFN storage
>>   iommufd: Data structure to provide IOVA to PFN mapping
>>   iommufd: IOCTLs for the io_pagetable
>>   iommufd: Add a HW pagetable object
>>   iommufd: Add kAPI toward external drivers for physical devices
>>   iommufd: Add kAPI toward external drivers for kernel access
>>   iommufd: vfio container FD ioctl compatibility
>>   iommufd: Add a selftest
>>
>> Kevin Tian (1):
>>   iommufd: Overview documentation
>>
>>  .clang-format                                 |    1 +
>>  Documentation/userspace-api/index.rst         |    1 +
>>  .../userspace-api/ioctl/ioctl-number.rst      |    1 +
>>  Documentation/userspace-api/iommufd.rst       |  224 +++
>>  MAINTAINERS                                   |   10 +
>>  drivers/iommu/Kconfig                         |    1 +
>>  drivers/iommu/Makefile                        |    2 +-
>>  drivers/iommu/iommufd/Kconfig                 |   22 +
>>  drivers/iommu/iommufd/Makefile                |   13 +
>>  drivers/iommu/iommufd/device.c                |  580 +++++++
>>  drivers/iommu/iommufd/hw_pagetable.c          |   68 +
>>  drivers/iommu/iommufd/io_pagetable.c          |  984 ++++++++++++
>>  drivers/iommu/iommufd/io_pagetable.h          |  186 +++
>>  drivers/iommu/iommufd/ioas.c                  |  338 ++++
>>  drivers/iommu/iommufd/iommufd_private.h       |  266 ++++
>>  drivers/iommu/iommufd/iommufd_test.h          |   74 +
>>  drivers/iommu/iommufd/main.c                  |  392 +++++
>>  drivers/iommu/iommufd/pages.c                 | 1301 +++++++++++++++
>>  drivers/iommu/iommufd/selftest.c              |  626 ++++++++
>>  drivers/iommu/iommufd/vfio_compat.c           |  423 +++++
>>  include/linux/interval_tree.h                 |   47 +
>>  include/linux/iommufd.h                       |  101 ++
>>  include/linux/sched/user.h                    |    2 +-
>>  include/uapi/linux/iommufd.h                  |  279 ++++
>>  kernel/user.c                                 |    1 +
>>  lib/interval_tree.c                           |   98 ++
>>  tools/testing/selftests/Makefile              |    1 +
>>  tools/testing/selftests/iommu/.gitignore      |    2 +
>>  tools/testing/selftests/iommu/Makefile        |   11 +
>>  tools/testing/selftests/iommu/config          |    2 +
>>  tools/testing/selftests/iommu/iommufd.c       | 1396 +++++++++++++++++
>>  31 files changed, 7451 insertions(+), 2 deletions(-)
>>  create mode 100644 Documentation/userspace-api/iommufd.rst
>>  create mode 100644 drivers/iommu/iommufd/Kconfig
>>  create mode 100644 drivers/iommu/iommufd/Makefile
>>  create mode 100644 drivers/iommu/iommufd/device.c
>>  create mode 100644 drivers/iommu/iommufd/hw_pagetable.c
>>  create mode 100644 drivers/iommu/iommufd/io_pagetable.c
>>  create mode 100644 drivers/iommu/iommufd/io_pagetable.h
>>  create mode 100644 drivers/iommu/iommufd/ioas.c
>>  create mode 100644 drivers/iommu/iommufd/iommufd_private.h
>>  create mode 100644 drivers/iommu/iommufd/iommufd_test.h
>>  create mode 100644 drivers/iommu/iommufd/main.c
>>  create mode 100644 drivers/iommu/iommufd/pages.c
>>  create mode 100644 drivers/iommu/iommufd/selftest.c
>>  create mode 100644 drivers/iommu/iommufd/vfio_compat.c
>>  create mode 100644 include/linux/iommufd.h
>>  create mode 100644 include/uapi/linux/iommufd.h
>>  create mode 100644 tools/testing/selftests/iommu/.gitignore
>>  create mode 100644 tools/testing/selftests/iommu/Makefile
>>  create mode 100644 tools/testing/selftests/iommu/config
>>  create mode 100644 tools/testing/selftests/iommu/iommufd.c
>>
>>
>> base-commit: b90cb1053190353cc30f0fef0ef1f378ccc063c5
>> --
>> 2.37.3