From: Eric Auger <eric.auger@redhat.com>
To: Zhenzhong Duan <zhenzhong.duan@intel.com>, qemu-devel@nongnu.org
Cc: alex.williamson@redhat.com, clg@redhat.com, jgg@nvidia.com,
nicolinc@nvidia.com, joao.m.martins@oracle.com,
peterx@redhat.com, jasowang@redhat.com, kevin.tian@intel.com,
yi.l.liu@intel.com, yi.y.sun@intel.com, chao.p.peng@intel.com
Subject: Re: [PATCH v1 00/22] vfio: Adopt iommufd
Date: Thu, 14 Sep 2023 11:04:11 +0200 [thread overview]
Message-ID: <8cd9a176-17b5-b6de-dc54-ba10124e9fe3@redhat.com> (raw)
In-Reply-To: <20230830103754.36461-1-zhenzhong.duan@intel.com>
Hi Zhenzhong
On 8/30/23 12:37, Zhenzhong Duan wrote:
> Hi All,
>
> As the kernel side iommufd cdev and hot reset feature have been queued,
> also hwpt alloc has been added in Jason's for_next branch [1], I'd like
> to update a new version matching kernel side update and with rfc flag
> removed. Qemu code can be found at [2], look forward more comments!
>
>
> We have done wide test with different combinations, e.g:
>
> - PCI device were tested
> - FD passing and hot reset with some trick.
> - device hotplug test with legacy and iommufd backends
> - with or without vIOMMU for legacy and iommufd backends
> - divices linked to different iommufds
> - VFIO migration with a E800 net card(no dirty sync support) passthrough
> - platform, ccw and ap were only compile-tested due to environment limit
>
>
> Given some iommufd kernel limitations, the iommufd backend is
> not yet fully on par with the legacy backend w.r.t. features like:
> - p2p mappings (you will see related error traces)
> - dirty page sync
> - and etc.
>
>
> Changelog:
> v1:
> - Alloc hwpt instead of using auto hwpt
> - elaborate iommufd code per Nicolin
> - consolidate two patches and drop as.c
> - typo error fix and function rename
>
> I didn't list change log of rfc stage, see [3] if anyone is interested.
>
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git
> [2] https://github.com/yiliu1765/qemu/commits/zhenzhong/iommufd_cdev_v1
> [3] https://lists.nongnu.org/archive/html/qemu-devel/2023-07/msg02529.html
Do you have a branch to share?
It does not apply to upstream
Thanks
Eric
>
>
> --------------------------------------------------------------------------
>
> With the introduction of iommufd, the Linux kernel provides a generic
> interface for userspace drivers to propagate their DMA mappings to kernel
> for assigned devices. This series does the porting of the VFIO devices
> onto the /dev/iommu uapi and let it coexist with the legacy implementation.
>
> This QEMU integration is the result of a collaborative work between
> Yi Liu, Yi Sun, Nicolin Chen and Eric Auger.
>
> At QEMU level, interactions with the /dev/iommu are abstracted by a new
> iommufd object (compiled in with the CONFIG_IOMMUFD option).
>
> Any QEMU device (e.g. vfio device) wishing to use /dev/iommu must be
> linked with an iommufd object. In this series, the vfio-pci device is
> granted with such capability (other VFIO devices are not yet ready):
>
> It gets a new optional parameter named iommufd which allows to pass
> an iommufd object:
>
> -object iommufd,id=iommufd0
> -device vfio-pci,host=0000:02:00.0,iommufd=iommufd0
>
> Note the /dev/iommu and vfio cdev can be externally opened by a
> management layer. In such a case the fd is passed:
>
> -object iommufd,id=iommufd0,fd=22
> -device vfio-pci,iommufd=iommufd0,fd=23
>
> If the fd parameter is not passed, the fd is opened by QEMU.
> See https://www.mail-archive.com/qemu-devel@nongnu.org/msg937155.html
> for detailed discuss on this requirement.
>
> If no iommufd option is passed to the vfio-pci device, iommufd is not
> used and the end-user gets the behavior based on the legacy vfio iommu
> interfaces:
>
> -device vfio-pci,host=0000:02:00.0
>
> While the legacy kernel interface is group-centric, the new iommufd
> interface is device-centric, relying on device fd and iommufd.
>
> To support both interfaces in the QEMU VFIO device we reworked the vfio
> container abstraction so that the generic VFIO code can use either
> backend.
>
> The VFIOContainer object becomes a base object derived into
> a) the legacy VFIO container and
> b) the new iommufd based container.
>
> The base object implements generic code such as code related to
> memory_listener and address space management whereas the derived
> objects implement callbacks specific to either BE, legacy and
> iommufd. Indeed each backend has its own way to setup secure context
> and dma management interface. The below diagram shows how it looks
> like with both BEs.
>
> VFIO AddressSpace/Memory
> +-------+ +----------+ +-----+ +-----+
> | pci | | platform | | ap | | ccw |
> +---+---+ +----+-----+ +--+--+ +--+--+ +----------------------+
> | | | | | AddressSpace |
> | | | | +------------+---------+
> +---V-----------V-----------V--------V----+ /
> | VFIOAddressSpace | <------------+
> | | | MemoryListener
> | VFIOContainer list |
> +-------+----------------------------+----+
> | |
> | |
> +-------V------+ +--------V----------+
> | iommufd | | vfio legacy |
> | container | | container |
> +-------+------+ +--------+----------+
> | |
> | /dev/iommu | /dev/vfio/vfio
> | /dev/vfio/devices/vfioX | /dev/vfio/$group_id
> Userspace | |
> ============+============================+===========================
> Kernel | device fd |
> +---------------+ | group/container fd
> | (BIND_IOMMUFD | | (SET_CONTAINER/SET_IOMMU)
> | ATTACH_IOAS) | | device fd
> | | |
> | +-------V------------V-----------------+
> iommufd | | vfio |
> (map/unmap | +---------+--------------------+-------+
> ioas_copy) | | | map/unmap
> | | |
> +------V------+ +-----V------+ +------V--------+
> | iommfd core | | device | | vfio iommu |
> +-------------+ +------------+ +---------------+
>
> [Secure Context setup]
> - iommufd BE: uses device fd and iommufd to setup secure context
> (bind_iommufd, attach_ioas)
> - vfio legacy BE: uses group fd and container fd to setup secure context
> (set_container, set_iommu)
> [Device access]
> - iommufd BE: device fd is opened through /dev/vfio/devices/vfioX
> - vfio legacy BE: device fd is retrieved from group fd ioctl
> [DMA Mapping flow]
> 1. VFIOAddressSpace receives MemoryRegion add/del via MemoryListener
> 2. VFIO populates DMA map/unmap via the container BEs
> *) iommufd BE: uses iommufd
> *) vfio legacy BE: uses container fd
>
>
> Thanks,
> Yi, Yi, Eric, Zhenzhong
>
>
> Eric Auger (8):
> scripts/update-linux-headers: Add iommufd.h
> vfio/common: Introduce vfio_container_add|del_section_window()
> vfio/container: Introduce vfio_[attach/detach]_device
> vfio/platform: Use vfio_[attach/detach]_device
> vfio/ap: Use vfio_[attach/detach]_device
> vfio/ccw: Use vfio_[attach/detach]_device
> backends/iommufd: Introduce the iommufd object
> vfio/pci: Allow the selection of a given iommu backend
>
> Yi Liu (5):
> vfio/common: Move IOMMU agnostic helpers to a separate file
> vfio/common: Move legacy VFIO backend code into separate container.c
> vfio: Add base container
> util/char_dev: Add open_cdev()
> vfio/iommufd: Implement the iommufd backend
>
> Zhenzhong Duan (9):
> Update linux-header to support iommufd cdev and hwpt alloc
> vfio/common: Extract out vfio_kvm_device_[add/del]_fd
> vfio/common: Add a vfio device iterator
> vfio/common: Refactor vfio_viommu_preset() to be group agnostic
> vfio/common: Simplify vfio_viommu_preset()
> Add iommufd configure option
> vfio/iommufd: Add vfio device iterator callback for iommufd
> vfio/pci: Adapt vfio pci hot reset support with iommufd BE
> vfio/pci: Make vfio cdev pre-openable by passing a file handle
>
> MAINTAINERS | 13 +
> backends/Kconfig | 4 +
> backends/iommufd.c | 291 ++++
> backends/meson.build | 3 +
> backends/trace-events | 13 +
> hw/vfio/ap.c | 68 +-
> hw/vfio/ccw.c | 120 +-
> hw/vfio/common.c | 1948 +++----------------------
> hw/vfio/container-base.c | 160 ++
> hw/vfio/container.c | 1208 +++++++++++++++
> hw/vfio/helpers.c | 626 ++++++++
> hw/vfio/iommufd.c | 554 +++++++
> hw/vfio/meson.build | 6 +
> hw/vfio/pci.c | 319 +++-
> hw/vfio/platform.c | 43 +-
> hw/vfio/spapr.c | 22 +-
> hw/vfio/trace-events | 21 +-
> include/hw/vfio/vfio-common.h | 111 +-
> include/hw/vfio/vfio-container-base.h | 158 ++
> include/qemu/char_dev.h | 16 +
> include/standard-headers/linux/fuse.h | 3 +
> include/sysemu/iommufd.h | 49 +
> linux-headers/linux/iommufd.h | 444 ++++++
> linux-headers/linux/kvm.h | 13 +-
> linux-headers/linux/vfio.h | 148 +-
> meson.build | 6 +
> meson_options.txt | 2 +
> qapi/qom.json | 18 +-
> qemu-options.hx | 13 +
> scripts/meson-buildoptions.sh | 3 +
> scripts/update-linux-headers.sh | 3 +-
> util/chardev_open.c | 61 +
> util/meson.build | 1 +
> 33 files changed, 4395 insertions(+), 2073 deletions(-)
> create mode 100644 backends/iommufd.c
> create mode 100644 hw/vfio/container-base.c
> create mode 100644 hw/vfio/container.c
> create mode 100644 hw/vfio/helpers.c
> create mode 100644 hw/vfio/iommufd.c
> create mode 100644 include/hw/vfio/vfio-container-base.h
> create mode 100644 include/qemu/char_dev.h
> create mode 100644 include/sysemu/iommufd.h
> create mode 100644 linux-headers/linux/iommufd.h
> create mode 100644 util/chardev_open.c
>
next prev parent reply other threads:[~2023-09-14 9:05 UTC|newest]
Thread overview: 109+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-30 10:37 [PATCH v1 00/22] vfio: Adopt iommufd Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 01/22] scripts/update-linux-headers: Add iommufd.h Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 02/22] Update linux-header to support iommufd cdev and hwpt alloc Zhenzhong Duan
2023-09-14 14:46 ` Eric Auger
2023-09-15 3:02 ` Duan, Zhenzhong
2023-09-20 11:04 ` Eric Auger
2023-09-20 11:15 ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 03/22] vfio/common: Move IOMMU agnostic helpers to a separate file Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 04/22] vfio/common: Introduce vfio_container_add|del_section_window() Zhenzhong Duan
2023-09-20 11:23 ` Eric Auger
2023-09-20 12:18 ` Duan, Zhenzhong
2023-09-21 8:28 ` Cédric Le Goater
2023-09-21 10:14 ` Duan, Zhenzhong
2023-09-21 10:55 ` Cédric Le Goater
2023-09-27 2:08 ` Duan, Zhenzhong
2023-09-27 6:50 ` Cédric Le Goater
2023-08-30 10:37 ` [PATCH v1 05/22] vfio/common: Extract out vfio_kvm_device_[add/del]_fd Zhenzhong Duan
2023-09-20 11:49 ` Eric Auger
2023-09-21 2:04 ` Duan, Zhenzhong
2023-09-21 8:42 ` Cédric Le Goater
2023-09-21 10:22 ` Duan, Zhenzhong
2023-09-21 10:53 ` Cédric Le Goater
2023-09-20 21:39 ` Alex Williamson
2023-09-21 6:03 ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 06/22] vfio/common: Add a vfio device iterator Zhenzhong Duan
2023-09-20 12:25 ` Eric Auger
2023-09-21 2:27 ` Duan, Zhenzhong
2023-09-20 22:16 ` Alex Williamson
2023-09-21 2:16 ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 07/22] vfio/common: Refactor vfio_viommu_preset() to be group agnostic Zhenzhong Duan
2023-09-20 13:00 ` Eric Auger
2023-09-21 2:52 ` Duan, Zhenzhong
2023-09-20 22:51 ` Alex Williamson
2023-09-21 6:13 ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 08/22] vfio/common: Move legacy VFIO backend code into separate container.c Zhenzhong Duan
2023-09-20 13:12 ` Eric Auger
2023-09-21 3:02 ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 09/22] vfio/container: Introduce vfio_[attach/detach]_device Zhenzhong Duan
2023-09-20 13:33 ` Eric Auger
2023-09-21 3:08 ` Duan, Zhenzhong
2023-09-21 9:44 ` Cédric Le Goater
2023-09-21 10:26 ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 10/22] vfio/platform: Use vfio_[attach/detach]_device Zhenzhong Duan
2023-09-21 12:17 ` Cédric Le Goater
2023-08-30 10:37 ` [PATCH v1 11/22] vfio/ap: " Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 12/22] vfio/ccw: " Zhenzhong Duan
2023-09-21 12:19 ` Cédric Le Goater
2023-09-21 13:00 ` Duan, Zhenzhong
2023-09-21 13:24 ` Cédric Le Goater
2023-08-30 10:37 ` [PATCH v1 13/22] vfio: Add base container Zhenzhong Duan
2023-09-19 17:23 ` Cédric Le Goater
2023-09-20 8:48 ` Duan, Zhenzhong
2023-09-20 12:57 ` Cédric Le Goater
2023-09-20 13:58 ` Eric Auger
2023-09-21 2:51 ` Duan, Zhenzhong
2023-09-20 13:53 ` Eric Auger
2023-09-21 3:12 ` Duan, Zhenzhong
2023-09-20 17:31 ` Eric Auger
2023-09-21 3:35 ` Duan, Zhenzhong
2023-09-21 6:28 ` Eric Auger
2023-09-21 17:20 ` Eric Auger
2023-09-22 2:52 ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 14/22] vfio/common: Simplify vfio_viommu_preset() Zhenzhong Duan
2023-09-19 16:01 ` Cédric Le Goater
2023-09-20 2:59 ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 15/22] Add iommufd configure option Zhenzhong Duan
2023-09-19 17:07 ` Cédric Le Goater
2023-09-20 3:42 ` Duan, Zhenzhong
2023-09-20 12:19 ` Cédric Le Goater
2023-09-20 12:51 ` Jason Gunthorpe
2023-09-20 13:01 ` Daniel P. Berrangé
2023-09-20 13:07 ` Jason Gunthorpe
2023-09-20 13:02 ` Cédric Le Goater
2023-09-20 17:37 ` Eric Auger
2023-09-20 17:49 ` Jason Gunthorpe
2023-09-20 18:17 ` Alex Williamson
2023-09-20 18:19 ` Jason Gunthorpe
2023-09-21 3:43 ` Duan, Zhenzhong
2023-09-26 6:05 ` Tian, Kevin
2023-09-21 4:00 ` Duan, Zhenzhong
2023-09-21 2:11 ` Duan, Zhenzhong
2023-09-20 18:01 ` Alex Williamson
2023-09-20 18:12 ` Jason Gunthorpe
2023-09-20 20:29 ` Alex Williamson
2023-09-20 18:15 ` Daniel P. Berrangé
2023-08-30 10:37 ` [PATCH v1 16/22] backends/iommufd: Introduce the iommufd object Zhenzhong Duan
2023-09-22 7:15 ` Cédric Le Goater
2023-09-22 8:39 ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 17/22] util/char_dev: Add open_cdev() Zhenzhong Duan
2023-09-20 12:39 ` Daniel P. Berrangé
2023-09-20 12:53 ` Jason Gunthorpe
2023-09-20 12:56 ` Daniel P. Berrangé
2023-09-21 2:37 ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 18/22] vfio/iommufd: Implement the iommufd backend Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 19/22] vfio/iommufd: Add vfio device iterator callback for iommufd Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 20/22] vfio/pci: Adapt vfio pci hot reset support with iommufd BE Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 21/22] vfio/pci: Allow the selection of a given iommu backend Zhenzhong Duan
2023-09-06 18:10 ` Jason Gunthorpe
2023-09-06 19:09 ` Alex Williamson
2023-09-07 1:10 ` Jason Gunthorpe
2023-09-07 2:27 ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 22/22] vfio/pci: Make vfio cdev pre-openable by passing a file handle Zhenzhong Duan
2023-09-14 9:04 ` Eric Auger [this message]
2023-09-14 9:27 ` [PATCH v1 00/22] vfio: Adopt iommufd Duan, Zhenzhong
2023-09-15 12:42 ` Cédric Le Goater
2023-09-15 13:14 ` Duan, Zhenzhong
2023-09-18 11:51 ` Jason Gunthorpe
2023-09-18 12:23 ` Cédric Le Goater
2023-09-18 17:56 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8cd9a176-17b5-b6de-dc54-ba10124e9fe3@redhat.com \
--to=eric.auger@redhat.com \
--cc=alex.williamson@redhat.com \
--cc=chao.p.peng@intel.com \
--cc=clg@redhat.com \
--cc=jasowang@redhat.com \
--cc=jgg@nvidia.com \
--cc=joao.m.martins@oracle.com \
--cc=kevin.tian@intel.com \
--cc=nicolinc@nvidia.com \
--cc=peterx@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=yi.l.liu@intel.com \
--cc=yi.y.sun@intel.com \
--cc=zhenzhong.duan@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).