Re: [PATCH v4 00/20] intel_iommu: Enable stage-1 translation for passthrough device

[Eric Auger <eric.auger@redhat.com>]

On 8/21/25 10:50 AM, Yi Liu wrote:
> On 2025/8/21 15:19, Duan, Zhenzhong wrote:
>> Kindly ping, any more comments?
>
> Do you have enough comments for a new version. I plan to have a look
> either this version or a new version next week. :)
same for me ;-)

Eric
>
> Regards,
> Yi Liu
>
>> Thanks
>> Zhenzhong
>>
>>> -----Original Message-----
>>> From: Duan, Zhenzhong <zhenzhong.duan@intel.com>
>>> Subject: [PATCH v4 00/20] intel_iommu: Enable stage-1 translation for
>>> passthrough device
>>>
>>> Hi,
>>>
>>> For passthrough device with intel_iommu.x-flts=on, we don't do
>>> shadowing
>>> of
>>> guest page table for passthrough device but pass stage-1 page table
>>> to host
>>> side to construct a nested domain. There was some effort to enable this
>>> feature
>>> in old days, see [1] for details.
>>>
>>> The key design is to utilize the dual-stage IOMMU translation (also
>>> known as
>>> IOMMU nested translation) capability in host IOMMU. As the below
>>> diagram
>>> shows,
>>> guest I/O page table pointer in GPA (guest physical address) is
>>> passed to host
>>> and be used to perform the stage-1 address translation. Along with it,
>>> modifications to present mappings in the guest I/O page table should be
>>> followed
>>> with an IOTLB invalidation.
>>>
>>>         .-------------.  .---------------------------.
>>>         |   vIOMMU    |  | Guest I/O page table      |
>>>         |             |  '---------------------------'
>>>         .----------------/
>>>         | PASID Entry |--- PASID cache flush --+
>>>         '-------------'                        |
>>>         |             |                        V
>>>         |             |           I/O page table pointer in GPA
>>>         '-------------'
>>>     Guest
>>>     ------| Shadow |---------------------------|--------
>>>           v        v                           v
>>>     Host
>>>         .-------------.  .------------------------.
>>>         |   pIOMMU    |  | Stage1 for GIOVA->GPA  |
>>>         |             |  '------------------------'
>>>         .----------------/  |
>>>         | PASID Entry |     V (Nested xlate)
>>>         '----------------\.--------------------------------------.
>>>         |             |   | Stage2 for GPA->HPA, unmanaged domain|
>>>         |             |   '--------------------------------------'
>>>         '-------------'
>>> For history reason, there are different namings in different VTD
>>> spec rev,
>>> Where:
>>> - Stage1 = First stage = First level = flts
>>> - Stage2 = Second stage = Second level = slts
>>> <Intel VT-d Nested translation>
>>>
>>> This series reuse VFIO device's default hwpt as nested parent
>>> instead of
>>> creating new one. This way avoids duplicate code of a new memory
>>> listener,
>>> all existing feature from VFIO listener can be shared, e.g., ram
>>> discard,
>>> dirty tracking, etc. Two limitations are: 1) not supporting VFIO device
>>> under a PCI bridge with emulated device, because emulated device wants
>>> IOMMU AS and VFIO device stick to system AS; 2) not supporting kexec or
>>> reboot from "intel_iommu=on,sm_on" to "intel_iommu=on,sm_off", because
>>> VFIO device's default hwpt is created with NEST_PARENT flag, kernel
>>> inhibit RO mappings when switch to shadow mode.
>>>
>>> This series is also a prerequisite work for vSVA, i.e. Sharing guest
>>> application address space with passthrough devices.
>>>
>>> There are some interactions between VFIO and vIOMMU
>>> * vIOMMU registers PCIIOMMUOps [set|unset]_iommu_device to PCI
>>>   subsystem. VFIO calls them to register/unregister HostIOMMUDevice
>>>   instance to vIOMMU at vfio device realize stage.
>>> * vIOMMU registers PCIIOMMUOps get_viommu_cap to PCI subsystem.
>>>   VFIO calls it to get vIOMMU exposed capabilities.
>>> * vIOMMU calls HostIOMMUDeviceIOMMUFD interface [at|de]tach_hwpt
>>>   to bind/unbind device to IOMMUFD backed domains, either nested
>>>   domain or not.
>>>
>>> See below diagram:
>>>
>>>         VFIO Device                                 Intel IOMMU
>>>     .-----------------.                         .-------------------.
>>>     |                 |                         |
>>> |
>>>     |       .---------|PCIIOMMUOps              |.-------------.    |
>>>     |       | IOMMUFD |(set/unset_iommu_device) || Host IOMMU  |
>>> |
>>>     |       | Device  |------------------------>|| Device list |    |
>>>     |       .---------|(get_viommu_cap)         |.-------------.    |
>>>     |                 |                         |       |
>>> |
>>>     |                 |                         |       V
>>> |
>>>     |       .---------|  HostIOMMUDeviceIOMMUFD |  .-------------.  |
>>>     |       | IOMMUFD |            (attach_hwpt)|  | Host IOMMU
>>> |  |
>>>     |       | link    |<------------------------|  |   Device    |  |
>>>     |       .---------|            (detach_hwpt)|  .-------------.  |
>>>     |                 |                         |       |
>>> |
>>>     |                 |                         |       ...
>>> |
>>>     .-----------------.                         .-------------------.
>>>
>>> Below is an example to enable stage-1 translation for passthrough
>>> device:
>>>
>>>     -M q35,...
>>>     -device intel-iommu,x-scalable-mode=on,x-flts=on...
>>>     -object iommufd,id=iommufd0 -device vfio-pci,iommufd=iommufd0,...
>>>
>>> Test done:
>>> - VFIO devices hotplug/unplug
>>> - different VFIO devices linked to different iommufds
>>> - vhost net device ping test
>>>
>>> PATCH1-6:  Some preparing work
>>> PATCH7-8:  Compatibility check between vIOMMU and Host IOMMU
>>> PATCH9-17: Implement stage-1 page table for passthrough device
>>> PATCH18-19:Workaround for ERRATA_772415_SPR17
>>> PATCH20:   Enable stage-1 translation for passthrough device
>>>
>>> Qemu code can be found at [2]
>>>
>>> Fault report isn't supported in this series, we presume guest kernel
>>> always
>>> construct correct stage1 page table for passthrough device. For
>>> emulated
>>> devices, the emulation code already provided stage1 fault injection.
>>>
>>> TODO:
>>> - Fault report to guest when HW stage1 faults
>>>
>>> [1]
>>> https://patchwork.kernel.org/project/kvm/cover/20210302203827.437645-1
>>> -yi.l.liu@intel.com/
>>> [2] https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting.v4
>>>
>>> Thanks
>>> Zhenzhong
>>>
>>> Changelog:
>>> v4:
>>> - s/VIOMMU_CAP_STAGE1/VIOMMU_CAP_HW_NESTED (Eric, Nicolin,
>>> Donald, Shameer)
>>> - clarify get_viommu_cap() return pure emulated caps and explain
>>> reason in
>>> commit log (Eric)
>>> - retrieve the ce only if vtd_as->pasid in
>>> vtd_as_to_iommu_pasid_locked (Eric)
>>> - refine doc comment and commit log in patch10-11 (Eric)
>>>
>>> v3:
>>> - define enum type for VIOMMU_CAP_* (Eric)
>>> - drop inline flag in the patch which uses the helper (Eric)
>>> - use extract64 in new introduced MACRO (Eric)
>>> - polish comments and fix typo error (Eric)
>>> - split workaround patch for ERRATA_772415_SPR17 to two patches (Eric)
>>> - optimize bind/unbind error path processing
>>>
>>> v2:
>>> - introduce get_viommu_cap() to get STAGE1 flag to create nested parent
>>> hwpt (Liuyi)
>>> - reuse VFIO's default hwpt as parent hwpt of nested translation
>>> (Nicolin,
>>> Liuyi)
>>> - abandon support of VFIO device under pcie-to-pci bridge to
>>> simplify design
>>> (Liuyi)
>>> - bypass RO mapping in VFIO's default hwpt if ERRATA_772415_SPR17
>>> (Liuyi)
>>> - drop vtd_dev_to_context_entry optimization (Liuyi)
>>>
>>> v1:
>>> - simplify vendor specific checking in vtd_check_hiod (Cedric, Nicolin)
>>> - rebase to master
>>>
>>> rfcv3:
>>> - s/hwpt_id/id in iommufd_backend_invalidate_cache()'s parameter
>>> (Shameer)
>>> - hide vtd vendor specific caps in a wrapper union (Eric, Nicolin)
>>> - simplify return value check of get_cap() (Eric)
>>> - drop realize_late (Cedric, Eric)
>>> - split patch13:intel_iommu: Add PASID cache management infrastructure
>>> (Eric)
>>> - s/vtd_pasid_cache_reset/vtd_pasid_cache_reset_locked (Eric)
>>> - s/vtd_pe_get_domain_id/vtd_pe_get_did (Eric)
>>> - refine comments (Eric, Donald)
>>>
>>> rfcv2:
>>> - Drop VTDPASIDAddressSpace and use VTDAddressSpace (Eric, Liuyi)
>>> - Move HWPT uAPI patches ahead(patch1-8) so arm nesting could easily
>>> rebase
>>> - add two cleanup patches(patch9-10)
>>> - VFIO passes iommufd/devid/hwpt_id to vIOMMU instead of
>>> iommufd/devid/ioas_id
>>> - add vtd_as_[from|to]_iommu_pasid() helper to translate between vtd_as
>>> and
>>>   iommu pasid, this is important for dropping VTDPASIDAddressSpace
>>>
>>>
>>> Yi Liu (3):
>>>   intel_iommu: Replay pasid bindings after context cache invalidation
>>>   intel_iommu: Propagate PASID-based iotlb invalidation to host
>>>   intel_iommu: Replay all pasid bindings when either SRTP or TE bit is
>>>     changed
>>>
>>> Zhenzhong Duan (17):
>>>   intel_iommu: Rename vtd_ce_get_rid2pasid_entry to
>>>     vtd_ce_get_pasid_entry
>>>   hw/pci: Introduce pci_device_get_viommu_cap()
>>>   intel_iommu: Implement get_viommu_cap() callback
>>>   vfio/iommufd: Force creating nested parent domain
>>>   hw/pci: Export pci_device_get_iommu_bus_devfn() and return bool
>>>   intel_iommu: Introduce a new structure VTDHostIOMMUDevice
>>>   intel_iommu: Check for compatibility with IOMMUFD backed device when
>>>     x-flts=on
>>>   intel_iommu: Fail passthrough device under PCI bridge if x-flts=on
>>>   intel_iommu: Introduce two helpers vtd_as_from/to_iommu_pasid_locked
>>>   intel_iommu: Handle PASID entry removal and update
>>>   intel_iommu: Handle PASID entry addition
>>>   intel_iommu: Introduce a new pasid cache invalidation type
>>> FORCE_RESET
>>>   intel_iommu: Stick to system MR for IOMMUFD backed host device when
>>>     x-fls=on
>>>   intel_iommu: Bind/unbind guest page table to host
>>>   vfio: Add a new element bypass_ro in VFIOContainerBase
>>>   Workaround for ERRATA_772415_SPR17
>>>   intel_iommu: Enable host device when x-flts=on in scalable mode
>>>
>>> MAINTAINERS                           |   1 +
>>> hw/i386/intel_iommu_internal.h        |  68 +-
>>> include/hw/i386/intel_iommu.h         |   9 +-
>>> include/hw/iommu.h                    |  17 +
>>> include/hw/pci/pci.h                  |  27 +
>>> include/hw/vfio/vfio-container-base.h |   1 +
>>> hw/i386/intel_iommu.c                 | 941
>>> +++++++++++++++++++++++++-
>>> hw/pci/pci.c                          |  23 +-
>>> hw/vfio/iommufd.c                     |  22 +-
>>> hw/vfio/listener.c                    |  13 +-
>>> hw/i386/trace-events                  |   8 +
>>> 11 files changed, 1088 insertions(+), 42 deletions(-)
>>> create mode 100644 include/hw/iommu.h
>>>
>>>
>>> base-commit: 92c05be4dfb59a71033d4c57dac944b29f7dabf0
>>> -- 
>>> 2.47.1
>>
>