Re: [PATCH v7 00/17] Add a host IOMMU device abstraction to check with vIOMMU

["Michael S. Tsirkin" <mst@redhat.com>]

On Mon, Jun 24, 2024 at 05:12:13PM +0200, Cédric Le Goater wrote:
> On 6/24/24 12:26 PM, Michael S. Tsirkin wrote:
> > On Wed, Jun 05, 2024 at 04:30:26PM +0800, Zhenzhong Duan wrote:
> > > Hi,
> > > 
> > > This series introduce a HostIOMMUDevice abstraction and sub-classes.
> > > Also HostIOMMUDeviceCaps structure in HostIOMMUDevice and a new interface
> > > between vIOMMU and HostIOMMUDevice.
> > 
> > Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
> > 
> > Who is merging this? Me? Or Alex?
> 
> I will and I will include this series also :
> 
>   [v4] VIRTIO-IOMMU/VFIO: Fix host iommu geometry
>   https://lore.kernel.org/all/20240614095402.904691-1-eric.auger@redhat.com
> 
> Thanks,
> 
> C.


Sounds good. I sent ack for merging, both reviewed:

Reviewed-by: Michael S. Tsirkin <mst@redhat.com>

> 
> > 
> > 
> > 
> > > A HostIOMMUDevice is an abstraction for an assigned device that is protected
> > > by a physical IOMMU (aka host IOMMU). The userspace interaction with this
> > > physical IOMMU can be done either through the VFIO IOMMU type 1 legacy
> > > backend or the new iommufd backend. The assigned device can be a VFIO device
> > > or a VDPA device. The HostIOMMUDevice is needed to interact with the host
> > > IOMMU that protects the assigned device. It is especially useful when the
> > > device is also protected by a virtual IOMMU as this latter use the translation
> > > services of the physical IOMMU and is constrained by it. In that context the
> > > HostIOMMUDevice can be passed to the virtual IOMMU to collect physical IOMMU
> > > capabilities such as the supported address width. In the future, the virtual
> > > IOMMU will use the HostIOMMUDevice to program the guest page tables in the
> > > first translation stage of the physical IOMMU.
> > > 
> > > HostIOMMUDeviceClass::realize() is introduced to initialize
> > > HostIOMMUDeviceCaps and other fields of HostIOMMUDevice variants.
> > > 
> > > HostIOMMUDeviceClass::get_cap() is introduced to query host IOMMU
> > > device capabilities.
> > > 
> > > The class tree is as below:
> > > 
> > >                                HostIOMMUDevice
> > >                                       | .caps
> > >                                       | .realize()
> > >                                       | .get_cap()
> > >                                       |
> > >              .-----------------------------------------------.
> > >              |                        |                      |
> > > HostIOMMUDeviceLegacyVFIO  {HostIOMMUDeviceLegacyVDPA}  HostIOMMUDeviceIOMMUFD
> > >              |                        |                      | [.iommufd]
> > >                                                              | [.devid]
> > >                                                              | [.ioas_id]
> > >                                                              | [.attach_hwpt()]
> > >                                                              | [.detach_hwpt()]
> > >                                                              |
> > >                                              .----------------------.
> > >                                              |                      |
> > >                           HostIOMMUDeviceIOMMUFDVFIO  {HostIOMMUDeviceIOMMUFDVDPA}
> > >                                            | [.vdev]                | {.vdev}
> > > 
> > > * The attributes in [] will be implemented in nesting series.
> > > * The classes in {} will be implemented in future.
> > > * .vdev in different class points to different agent device,
> > > * i.e., VFIODevice or VDPADevice.
> > > 
> > > PATCH1-4: Introduce HostIOMMUDevice and its sub classes
> > > PATCH5-10: Implement .realize() and .get_cap() handler
> > > PATCH11-14: Create HostIOMMUDevice instance and pass to vIOMMU
> > > PATCH15-17: Implement compatibility check between host IOMMU and vIOMMU(intel_iommu)
> > > 
> > > Test done:
> > > make check
> > > vfio device hotplug/unplug with different backend on linux
> > > reboot, kexec
> > > build test on linux and windows11
> > > 
> > > Qemu code can be found at:
> > > https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_preq_v7
> > > 
> > > Besides the compatibility check in this series, in nesting series, this
> > > host IOMMU device is extended for much wider usage. For anyone interested
> > > on the nesting series, here is the link:
> > > https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_rfcv2
> > > 
> > > Thanks
> > > Zhenzhong
> > > 
> > > Changelog:
> > > v7:
> > > - drop config CONFIG_HOST_IOMMU_DEVICE (Cédric)
> > > - introduce HOST_IOMMU_DEVICE_CAP_AW_BITS_MAX (Eric)
> > > - use iova_ranges method in iommufd.realize() (Eric)
> > > - introduce HostIOMMUDevice::name to facilitate tracing (Eric)
> > > - implement a custom destroy hash function (Cédric)
> > > - drop VTDHostIOMMUDevice and save HostIOMMUDevice in hash table (Eric)
> > > - move patch5 after patch1 (Eric)
> > > - squash patch3 and 4, squash patch12 and 13 (Eric)
> > > - refine comments (Eric)
> > > - collect Eric's R-B
> > > 
> > > v6:
> > > - open coded host_iommu_device_get_cap() to avoid #ifdef in intel_iommu.c (Cédric)
> > > 
> > > v5:
> > > - pci_device_set_iommu_device return true (Cédric)
> > > - fix build failure on windows (thanks Cédric found that issue)
> > > 
> > > v4:
> > > - move properties vdev, iommufd and devid to nesting series where need it (Cédric)
> > > - fix 32bit build with clz64 (Cédric)
> > > - change check_cap naming to get_cap (Cédric)
> > > - return bool if error is passed through errp (Cédric)
> > > - drop HostIOMMUDevice[LegacyVFIO|IOMMUFD|IOMMUFDVFIO] declaration (Cédric)
> > > - drop HOST_IOMMU_DEVICE_CAP_IOMMUFD (Cédric)
> > > - replace include directive with forward declaration (Cédric)
> > > 
> > > v3:
> > > - refine declaration and doc for HostIOMMUDevice (Cédric, Philippe)
> > > - introduce HostIOMMUDeviceCaps, .realize() and .check_cap() (Cédric)
> > > - introduce helper range_get_last_bit() for range operation (Cédric)
> > > - separate pci_device_get_iommu_bus_devfn() in a prereq patch (Cédric)
> > > - replace HIOD_ abbreviation with HOST_IOMMU_DEVICE_ (Cédric)
> > > - add header in include/sysemu/iommufd.h (Cédric)
> > > 
> > > v2:
> > > - use QOM to abstract host IOMMU device and its sub-classes (Cédric)
> > > - move host IOMMU device creation in attach_device() (Cédric)
> > > - refine pci_device_set/unset_iommu_device doc further (Eric)
> > > - define host IOMMU info format of different backend
> > > - implement get_host_iommu_info() for different backend (Cédric)
> > > - drop cap/ecap update logic (MST)
> > > - check aw-bits from get_host_iommu_info() in legacy mode
> > > 
> > > v1:
> > > - use HostIOMMUDevice handle instead of union in VFIODevice (Eric)
> > > - change host_iommu_device_init to host_iommu_device_create
> > > - allocate HostIOMMUDevice in host_iommu_device_create callback
> > >    and set the VFIODevice base_hdev handle (Eric)
> > > - refine pci_device_set/unset_iommu_device doc (Eric)
> > > - use HostIOMMUDevice handle instead of union in VTDHostIOMMUDevice (Eric)
> > > - convert HostIOMMUDevice to sub object pointer in vtd_check_hdev
> > > 
> > > rfcv2:
> > > - introduce common abstract HostIOMMUDevice and sub struct for different BEs (Eric, Cédric)
> > > - remove iommufd_device.[ch] (Cédric)
> > > - remove duplicate iommufd/devid define from VFIODevice (Eric)
> > > - drop the p in aliased_pbus and aliased_pdevfn (Eric)
> > > - assert devfn and iommu_bus in pci_device_get_iommu_bus_devfn (Cédric, Eric)
> > > - use errp in iommufd_device_get_info (Eric)
> > > - split and simplify cap/ecap check/sync code in intel_iommu.c (Cédric)
> > > - move VTDHostIOMMUDevice declaration to intel_iommu_internal.h (Cédric)
> > > - make '(vtd->cap_reg >> 16) & 0x3fULL' a MACRO and add missed '+1' (Cédric)
> > > - block migration if vIOMMU cap/ecap updated based on host IOMMU cap/ecap
> > > - add R-B
> > > 
> > > Yi Liu (2):
> > >    hw/pci: Introduce pci_device_[set|unset]_iommu_device()
> > >    intel_iommu: Implement [set|unset]_iommu_device() callbacks
> > > 
> > > Zhenzhong Duan (15):
> > >    backends: Introduce HostIOMMUDevice abstract
> > >    backends/host_iommu_device: Introduce HostIOMMUDeviceCaps
> > >    vfio/container: Introduce TYPE_HOST_IOMMU_DEVICE_LEGACY_VFIO device
> > >    backends/iommufd: Introduce TYPE_HOST_IOMMU_DEVICE_IOMMUFD[_VFIO]
> > >      devices
> > >    range: Introduce range_get_last_bit()
> > >    vfio/container: Implement HostIOMMUDeviceClass::realize() handler
> > >    backends/iommufd: Introduce helper function
> > >      iommufd_backend_get_device_info()
> > >    vfio/iommufd: Implement HostIOMMUDeviceClass::realize() handler
> > >    vfio/container: Implement HostIOMMUDeviceClass::get_cap() handler
> > >    backends/iommufd: Implement HostIOMMUDeviceClass::get_cap() handler
> > >    vfio: Create host IOMMU device instance
> > >    hw/pci: Introduce helper function pci_device_get_iommu_bus_devfn()
> > >    vfio/pci: Pass HostIOMMUDevice to vIOMMU
> > >    intel_iommu: Extract out vtd_cap_init() to initialize cap/ecap
> > >    intel_iommu: Check compatibility with host IOMMU capabilities
> > > 
> > >   MAINTAINERS                           |   2 +
> > >   include/hw/i386/intel_iommu.h         |   2 +
> > >   include/hw/pci/pci.h                  |  38 ++++-
> > >   include/hw/vfio/vfio-common.h         |   8 +
> > >   include/hw/vfio/vfio-container-base.h |   3 +
> > >   include/qemu/range.h                  |  11 ++
> > >   include/sysemu/host_iommu_device.h    |  91 ++++++++++++
> > >   include/sysemu/iommufd.h              |  19 +++
> > >   backends/host_iommu_device.c          |  33 +++++
> > >   backends/iommufd.c                    |  76 ++++++++--
> > >   hw/i386/intel_iommu.c                 | 203 ++++++++++++++++++++------
> > >   hw/pci/pci.c                          |  75 +++++++++-
> > >   hw/vfio/common.c                      |  16 +-
> > >   hw/vfio/container.c                   |  41 +++++-
> > >   hw/vfio/helpers.c                     |  17 +++
> > >   hw/vfio/iommufd.c                     |  37 ++++-
> > >   hw/vfio/pci.c                         |  19 ++-
> > >   backends/meson.build                  |   1 +
> > >   18 files changed, 623 insertions(+), 69 deletions(-)
> > >   create mode 100644 include/sysemu/host_iommu_device.h
> > >   create mode 100644 backends/host_iommu_device.c
> > > 
> > > -- 
> > > 2.34.1
> >