qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
From: Eric Auger <eric.auger@redhat.com>
To: Zhenzhong Duan <zhenzhong.duan@intel.com>, qemu-devel@nongnu.org
Cc: alex.williamson@redhat.com, clg@redhat.com, jgg@nvidia.com,
	nicolinc@nvidia.com, joao.m.martins@oracle.com,
	peterx@redhat.com, jasowang@redhat.com, kevin.tian@intel.com,
	yi.l.liu@intel.com, yi.y.sun@intel.com, chao.p.peng@intel.com
Subject: Re: [PATCH v1 00/22] vfio: Adopt iommufd
Date: Thu, 14 Sep 2023 11:04:11 +0200	[thread overview]
Message-ID: <8cd9a176-17b5-b6de-dc54-ba10124e9fe3@redhat.com> (raw)
In-Reply-To: <20230830103754.36461-1-zhenzhong.duan@intel.com>

Hi Zhenzhong

On 8/30/23 12:37, Zhenzhong Duan wrote:
> Hi All,
>
> As the kernel side iommufd cdev and hot reset feature have been queued,
> also hwpt alloc has been added in Jason's for_next branch [1], I'd like
> to update a new version matching kernel side update and with rfc flag
> removed. Qemu code can be found at [2], look forward more comments!
>
>
> We have done wide test with different combinations, e.g:
>
> - PCI device were tested
> - FD passing and hot reset with some trick.
> - device hotplug test with legacy and iommufd backends
> - with or without vIOMMU for legacy and iommufd backends
> - divices linked to different iommufds
> - VFIO migration with a E800 net card(no dirty sync support) passthrough
> - platform, ccw and ap were only compile-tested due to environment limit
>
>
> Given some iommufd kernel limitations, the iommufd backend is
> not yet fully on par with the legacy backend w.r.t. features like:
> - p2p mappings (you will see related error traces)
> - dirty page sync
> - and etc.
>
>
> Changelog:
> v1:
> - Alloc hwpt instead of using auto hwpt
> - elaborate iommufd code per Nicolin
> - consolidate two patches and drop as.c
> - typo error fix and function rename
>
> I didn't list change log of rfc stage, see [3] if anyone is interested.
>
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git
> [2] https://github.com/yiliu1765/qemu/commits/zhenzhong/iommufd_cdev_v1
> [3] https://lists.nongnu.org/archive/html/qemu-devel/2023-07/msg02529.html

Do you have a branch to share?

It does not apply to upstream

Thanks

Eric
>
>
> --------------------------------------------------------------------------
>
> With the introduction of iommufd, the Linux kernel provides a generic
> interface for userspace drivers to propagate their DMA mappings to kernel
> for assigned devices. This series does the porting of the VFIO devices
> onto the /dev/iommu uapi and let it coexist with the legacy implementation.
>
> This QEMU integration is the result of a collaborative work between
> Yi Liu, Yi Sun, Nicolin Chen and Eric Auger.
>
> At QEMU level, interactions with the /dev/iommu are abstracted by a new
> iommufd object (compiled in with the CONFIG_IOMMUFD option).
>
> Any QEMU device (e.g. vfio device) wishing to use /dev/iommu must be
> linked with an iommufd object. In this series, the vfio-pci device is
> granted with such capability (other VFIO devices are not yet ready):
>
> It gets a new optional parameter named iommufd which allows to pass
> an iommufd object:
>
>     -object iommufd,id=iommufd0
>     -device vfio-pci,host=0000:02:00.0,iommufd=iommufd0
>
> Note the /dev/iommu and vfio cdev can be externally opened by a
> management layer. In such a case the fd is passed:
>   
>     -object iommufd,id=iommufd0,fd=22
>     -device vfio-pci,iommufd=iommufd0,fd=23
>
> If the fd parameter is not passed, the fd is opened by QEMU.
> See https://www.mail-archive.com/qemu-devel@nongnu.org/msg937155.html
> for detailed discuss on this requirement.
>
> If no iommufd option is passed to the vfio-pci device, iommufd is not
> used and the end-user gets the behavior based on the legacy vfio iommu
> interfaces:
>
>     -device vfio-pci,host=0000:02:00.0
>
> While the legacy kernel interface is group-centric, the new iommufd
> interface is device-centric, relying on device fd and iommufd.
>
> To support both interfaces in the QEMU VFIO device we reworked the vfio
> container abstraction so that the generic VFIO code can use either
> backend.
>
> The VFIOContainer object becomes a base object derived into
> a) the legacy VFIO container and
> b) the new iommufd based container.
>
> The base object implements generic code such as code related to
> memory_listener and address space management whereas the derived
> objects implement callbacks specific to either BE, legacy and
> iommufd. Indeed each backend has its own way to setup secure context
> and dma management interface. The below diagram shows how it looks
> like with both BEs.
>
>                     VFIO                           AddressSpace/Memory
>     +-------+  +----------+  +-----+  +-----+
>     |  pci  |  | platform |  |  ap |  | ccw |
>     +---+---+  +----+-----+  +--+--+  +--+--+     +----------------------+
>         |           |           |        |        |   AddressSpace       |
>         |           |           |        |        +------------+---------+
>     +---V-----------V-----------V--------V----+               /
>     |           VFIOAddressSpace              | <------------+
>     |                  |                      |  MemoryListener
>     |          VFIOContainer list             |
>     +-------+----------------------------+----+
>             |                            |
>             |                            |
>     +-------V------+            +--------V----------+
>     |   iommufd    |            |    vfio legacy    |
>     |  container   |            |     container     |
>     +-------+------+            +--------+----------+
>             |                            |
>             | /dev/iommu                 | /dev/vfio/vfio
>             | /dev/vfio/devices/vfioX    | /dev/vfio/$group_id
> Userspace   |                            |
> ============+============================+===========================
> Kernel      |  device fd                 |
>             +---------------+            | group/container fd
>             | (BIND_IOMMUFD |            | (SET_CONTAINER/SET_IOMMU)
>             |  ATTACH_IOAS) |            | device fd
>             |               |            |
>             |       +-------V------------V-----------------+
>     iommufd |       |                vfio                  |
> (map/unmap  |       +---------+--------------------+-------+
> ioas_copy)  |                 |                    | map/unmap
>             |                 |                    |
>      +------V------+    +-----V------+      +------V--------+
>      | iommfd core |    |  device    |      |  vfio iommu   |
>      +-------------+    +------------+      +---------------+
>
> [Secure Context setup]
> - iommufd BE: uses device fd and iommufd to setup secure context
>               (bind_iommufd, attach_ioas)
> - vfio legacy BE: uses group fd and container fd to setup secure context
>                   (set_container, set_iommu)
> [Device access]
> - iommufd BE: device fd is opened through /dev/vfio/devices/vfioX
> - vfio legacy BE: device fd is retrieved from group fd ioctl
> [DMA Mapping flow]
> 1. VFIOAddressSpace receives MemoryRegion add/del via MemoryListener
> 2. VFIO populates DMA map/unmap via the container BEs
>    *) iommufd BE: uses iommufd
>    *) vfio legacy BE: uses container fd
>
>
> Thanks,
> Yi, Yi, Eric, Zhenzhong
>
>
> Eric Auger (8):
>   scripts/update-linux-headers: Add iommufd.h
>   vfio/common: Introduce vfio_container_add|del_section_window()
>   vfio/container: Introduce vfio_[attach/detach]_device
>   vfio/platform: Use vfio_[attach/detach]_device
>   vfio/ap: Use vfio_[attach/detach]_device
>   vfio/ccw: Use vfio_[attach/detach]_device
>   backends/iommufd: Introduce the iommufd object
>   vfio/pci: Allow the selection of a given iommu backend
>
> Yi Liu (5):
>   vfio/common: Move IOMMU agnostic helpers to a separate file
>   vfio/common: Move legacy VFIO backend code into separate container.c
>   vfio: Add base container
>   util/char_dev: Add open_cdev()
>   vfio/iommufd: Implement the iommufd backend
>
> Zhenzhong Duan (9):
>   Update linux-header to support iommufd cdev and hwpt alloc
>   vfio/common: Extract out vfio_kvm_device_[add/del]_fd
>   vfio/common: Add a vfio device iterator
>   vfio/common: Refactor vfio_viommu_preset() to be group agnostic
>   vfio/common: Simplify vfio_viommu_preset()
>   Add iommufd configure option
>   vfio/iommufd: Add vfio device iterator callback for iommufd
>   vfio/pci: Adapt vfio pci hot reset support with iommufd BE
>   vfio/pci: Make vfio cdev pre-openable by passing a file handle
>
>  MAINTAINERS                           |   13 +
>  backends/Kconfig                      |    4 +
>  backends/iommufd.c                    |  291 ++++
>  backends/meson.build                  |    3 +
>  backends/trace-events                 |   13 +
>  hw/vfio/ap.c                          |   68 +-
>  hw/vfio/ccw.c                         |  120 +-
>  hw/vfio/common.c                      | 1948 +++----------------------
>  hw/vfio/container-base.c              |  160 ++
>  hw/vfio/container.c                   | 1208 +++++++++++++++
>  hw/vfio/helpers.c                     |  626 ++++++++
>  hw/vfio/iommufd.c                     |  554 +++++++
>  hw/vfio/meson.build                   |    6 +
>  hw/vfio/pci.c                         |  319 +++-
>  hw/vfio/platform.c                    |   43 +-
>  hw/vfio/spapr.c                       |   22 +-
>  hw/vfio/trace-events                  |   21 +-
>  include/hw/vfio/vfio-common.h         |  111 +-
>  include/hw/vfio/vfio-container-base.h |  158 ++
>  include/qemu/char_dev.h               |   16 +
>  include/standard-headers/linux/fuse.h |    3 +
>  include/sysemu/iommufd.h              |   49 +
>  linux-headers/linux/iommufd.h         |  444 ++++++
>  linux-headers/linux/kvm.h             |   13 +-
>  linux-headers/linux/vfio.h            |  148 +-
>  meson.build                           |    6 +
>  meson_options.txt                     |    2 +
>  qapi/qom.json                         |   18 +-
>  qemu-options.hx                       |   13 +
>  scripts/meson-buildoptions.sh         |    3 +
>  scripts/update-linux-headers.sh       |    3 +-
>  util/chardev_open.c                   |   61 +
>  util/meson.build                      |    1 +
>  33 files changed, 4395 insertions(+), 2073 deletions(-)
>  create mode 100644 backends/iommufd.c
>  create mode 100644 hw/vfio/container-base.c
>  create mode 100644 hw/vfio/container.c
>  create mode 100644 hw/vfio/helpers.c
>  create mode 100644 hw/vfio/iommufd.c
>  create mode 100644 include/hw/vfio/vfio-container-base.h
>  create mode 100644 include/qemu/char_dev.h
>  create mode 100644 include/sysemu/iommufd.h
>  create mode 100644 linux-headers/linux/iommufd.h
>  create mode 100644 util/chardev_open.c
>



  parent reply	other threads:[~2023-09-14  9:05 UTC|newest]

Thread overview: 109+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-08-30 10:37 [PATCH v1 00/22] vfio: Adopt iommufd Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 01/22] scripts/update-linux-headers: Add iommufd.h Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 02/22] Update linux-header to support iommufd cdev and hwpt alloc Zhenzhong Duan
2023-09-14 14:46   ` Eric Auger
2023-09-15  3:02     ` Duan, Zhenzhong
2023-09-20 11:04       ` Eric Auger
2023-09-20 11:15         ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 03/22] vfio/common: Move IOMMU agnostic helpers to a separate file Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 04/22] vfio/common: Introduce vfio_container_add|del_section_window() Zhenzhong Duan
2023-09-20 11:23   ` Eric Auger
2023-09-20 12:18     ` Duan, Zhenzhong
2023-09-21  8:28   ` Cédric Le Goater
2023-09-21 10:14     ` Duan, Zhenzhong
2023-09-21 10:55       ` Cédric Le Goater
2023-09-27  2:08       ` Duan, Zhenzhong
2023-09-27  6:50         ` Cédric Le Goater
2023-08-30 10:37 ` [PATCH v1 05/22] vfio/common: Extract out vfio_kvm_device_[add/del]_fd Zhenzhong Duan
2023-09-20 11:49   ` Eric Auger
2023-09-21  2:04     ` Duan, Zhenzhong
2023-09-21  8:42     ` Cédric Le Goater
2023-09-21 10:22       ` Duan, Zhenzhong
2023-09-21 10:53         ` Cédric Le Goater
2023-09-20 21:39   ` Alex Williamson
2023-09-21  6:03     ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 06/22] vfio/common: Add a vfio device iterator Zhenzhong Duan
2023-09-20 12:25   ` Eric Auger
2023-09-21  2:27     ` Duan, Zhenzhong
2023-09-20 22:16   ` Alex Williamson
2023-09-21  2:16     ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 07/22] vfio/common: Refactor vfio_viommu_preset() to be group agnostic Zhenzhong Duan
2023-09-20 13:00   ` Eric Auger
2023-09-21  2:52     ` Duan, Zhenzhong
2023-09-20 22:51   ` Alex Williamson
2023-09-21  6:13     ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 08/22] vfio/common: Move legacy VFIO backend code into separate container.c Zhenzhong Duan
2023-09-20 13:12   ` Eric Auger
2023-09-21  3:02     ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 09/22] vfio/container: Introduce vfio_[attach/detach]_device Zhenzhong Duan
2023-09-20 13:33   ` Eric Auger
2023-09-21  3:08     ` Duan, Zhenzhong
2023-09-21  9:44   ` Cédric Le Goater
2023-09-21 10:26     ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 10/22] vfio/platform: Use vfio_[attach/detach]_device Zhenzhong Duan
2023-09-21 12:17   ` Cédric Le Goater
2023-08-30 10:37 ` [PATCH v1 11/22] vfio/ap: " Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 12/22] vfio/ccw: " Zhenzhong Duan
2023-09-21 12:19   ` Cédric Le Goater
2023-09-21 13:00     ` Duan, Zhenzhong
2023-09-21 13:24       ` Cédric Le Goater
2023-08-30 10:37 ` [PATCH v1 13/22] vfio: Add base container Zhenzhong Duan
2023-09-19 17:23   ` Cédric Le Goater
2023-09-20  8:48     ` Duan, Zhenzhong
2023-09-20 12:57       ` Cédric Le Goater
2023-09-20 13:58         ` Eric Auger
2023-09-21  2:51         ` Duan, Zhenzhong
2023-09-20 13:53     ` Eric Auger
2023-09-21  3:12       ` Duan, Zhenzhong
2023-09-20 17:31     ` Eric Auger
2023-09-21  3:35       ` Duan, Zhenzhong
2023-09-21  6:28         ` Eric Auger
2023-09-21 17:20         ` Eric Auger
2023-09-22  2:52           ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 14/22] vfio/common: Simplify vfio_viommu_preset() Zhenzhong Duan
2023-09-19 16:01   ` Cédric Le Goater
2023-09-20  2:59     ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 15/22] Add iommufd configure option Zhenzhong Duan
2023-09-19 17:07   ` Cédric Le Goater
2023-09-20  3:42     ` Duan, Zhenzhong
2023-09-20 12:19       ` Cédric Le Goater
2023-09-20 12:51         ` Jason Gunthorpe
2023-09-20 13:01           ` Daniel P. Berrangé
2023-09-20 13:07             ` Jason Gunthorpe
2023-09-20 13:02           ` Cédric Le Goater
2023-09-20 17:37             ` Eric Auger
2023-09-20 17:49               ` Jason Gunthorpe
2023-09-20 18:17                 ` Alex Williamson
2023-09-20 18:19                   ` Jason Gunthorpe
2023-09-21  3:43                     ` Duan, Zhenzhong
2023-09-26  6:05                     ` Tian, Kevin
2023-09-21  4:00             ` Duan, Zhenzhong
2023-09-21  2:11         ` Duan, Zhenzhong
2023-09-20 18:01       ` Alex Williamson
2023-09-20 18:12         ` Jason Gunthorpe
2023-09-20 20:29           ` Alex Williamson
2023-09-20 18:15         ` Daniel P. Berrangé
2023-08-30 10:37 ` [PATCH v1 16/22] backends/iommufd: Introduce the iommufd object Zhenzhong Duan
2023-09-22  7:15   ` Cédric Le Goater
2023-09-22  8:39     ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 17/22] util/char_dev: Add open_cdev() Zhenzhong Duan
2023-09-20 12:39   ` Daniel P. Berrangé
2023-09-20 12:53     ` Jason Gunthorpe
2023-09-20 12:56       ` Daniel P. Berrangé
2023-09-21  2:37     ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 18/22] vfio/iommufd: Implement the iommufd backend Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 19/22] vfio/iommufd: Add vfio device iterator callback for iommufd Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 20/22] vfio/pci: Adapt vfio pci hot reset support with iommufd BE Zhenzhong Duan
2023-08-30 10:37 ` [PATCH v1 21/22] vfio/pci: Allow the selection of a given iommu backend Zhenzhong Duan
2023-09-06 18:10   ` Jason Gunthorpe
2023-09-06 19:09     ` Alex Williamson
2023-09-07  1:10       ` Jason Gunthorpe
2023-09-07  2:27         ` Duan, Zhenzhong
2023-08-30 10:37 ` [PATCH v1 22/22] vfio/pci: Make vfio cdev pre-openable by passing a file handle Zhenzhong Duan
2023-09-14  9:04 ` Eric Auger [this message]
2023-09-14  9:27   ` [PATCH v1 00/22] vfio: Adopt iommufd Duan, Zhenzhong
2023-09-15 12:42 ` Cédric Le Goater
2023-09-15 13:14   ` Duan, Zhenzhong
2023-09-18 11:51   ` Jason Gunthorpe
2023-09-18 12:23     ` Cédric Le Goater
2023-09-18 17:56       ` Jason Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8cd9a176-17b5-b6de-dc54-ba10124e9fe3@redhat.com \
    --to=eric.auger@redhat.com \
    --cc=alex.williamson@redhat.com \
    --cc=chao.p.peng@intel.com \
    --cc=clg@redhat.com \
    --cc=jasowang@redhat.com \
    --cc=jgg@nvidia.com \
    --cc=joao.m.martins@oracle.com \
    --cc=kevin.tian@intel.com \
    --cc=nicolinc@nvidia.com \
    --cc=peterx@redhat.com \
    --cc=qemu-devel@nongnu.org \
    --cc=yi.l.liu@intel.com \
    --cc=yi.y.sun@intel.com \
    --cc=zhenzhong.duan@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).