Re: [PATCH] docs/devel: Add VFIO iommufd backend documentation

["Cédric Le Goater" <clg@redhat.com>]

On 11/17/23 10:35, Zhenzhong Duan wrote:
> Suggested-by: Cédric Le Goater <clg@redhat.com>
> Signed-off-by: Zhenzhong Duan <zhenzhong.duan@intel.com>

The content looks good but it lacks formatting. Please try to generate
the docs.

Thanks,

C.


> ---
>   MAINTAINERS                    |   1 +
>   docs/devel/index-internals.rst |   1 +
>   docs/devel/vfio-iommufd.rst    | 115 +++++++++++++++++++++++++++++++++
>   3 files changed, 117 insertions(+)
>   create mode 100644 docs/devel/vfio-iommufd.rst
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index d86ba56a49..07990456ed 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2175,6 +2175,7 @@ F: backends/iommufd.c
>   F: include/sysemu/iommufd.h
>   F: include/qemu/chardev_open.h
>   F: util/chardev_open.c
> +F: docs/devel/vfio-iommufd.rst
>   
>   vhost
>   M: Michael S. Tsirkin <mst@redhat.com>
> diff --git a/docs/devel/index-internals.rst b/docs/devel/index-internals.rst
> index 6f81df92bc..3def4a138b 100644
> --- a/docs/devel/index-internals.rst
> +++ b/docs/devel/index-internals.rst
> @@ -18,5 +18,6 @@ Details about QEMU's various subsystems including how to add features to them.
>      s390-dasd-ipl
>      tracing
>      vfio-migration
> +   vfio-iommufd
>      writing-monitor-commands
>      virtio-backends
> diff --git a/docs/devel/vfio-iommufd.rst b/docs/devel/vfio-iommufd.rst
> new file mode 100644
> index 0000000000..59804a7f26
> --- /dev/null
> +++ b/docs/devel/vfio-iommufd.rst
> @@ -0,0 +1,115 @@
> +===============================
> +IOMMUFD BACKEND usage with VFIO
> +===============================
> +
> +(Same meaning for backend/container/BE)
> +
> +With the introduction of iommufd, the Linux kernel provides a generic
> +interface for user space drivers to propagate their DMA mappings to kernel
> +for assigned devices. While the legacy kernel interface is group-centric,
> +the new iommufd interface is device-centric, relying on device fd and iommufd.
> +
> +To support both interfaces in the QEMU VFIO device, introduce a base container
> +to abstract the common part of VFIO legacy and iommufd container. So that the
> +generic VFIO code can use either container.
> +
> +The base container implements generic functions such as memory_listener and
> +address space management whereas the derived container implements callbacks
> +specific to either legacy or iommufd. Each container has its own way to setup
> +secure context and dma management interface. The below diagram shows how it
> +looks like with both containers.
> +
> +                    VFIO                           AddressSpace/Memory
> +    +-------+  +----------+  +-----+  +-----+
> +    |  pci  |  | platform |  |  ap |  | ccw |
> +    +---+---+  +----+-----+  +--+--+  +--+--+     +----------------------+
> +        |           |           |        |        |   AddressSpace       |
> +        |           |           |        |        +------------+---------+
> +    +---V-----------V-----------V--------V----+               /
> +    |           VFIOAddressSpace              | <------------+
> +    |                  |                      |  MemoryListener
> +    |        VFIOContainerBase list           |
> +    +-------+----------------------------+----+
> +            |                            |
> +            |                            |
> +    +-------V------+            +--------V----------+
> +    |   iommufd    |            |    vfio legacy    |
> +    |  container   |            |     container     |
> +    +-------+------+            +--------+----------+
> +            |                            |
> +            | /dev/iommu                 | /dev/vfio/vfio
> +            | /dev/vfio/devices/vfioX    | /dev/vfio/$group_id
> +Userspace   |                            |
> +============+============================+===========================
> +Kernel      |  device fd                 |
> +            +---------------+            | group/container fd
> +            | (BIND_IOMMUFD |            | (SET_CONTAINER/SET_IOMMU)
> +            |  ATTACH_IOAS) |            | device fd
> +            |               |            |
> +            |       +-------V------------V-----------------+
> +    iommufd |       |                vfio                  |
> +(map/unmap  |       +---------+--------------------+-------+
> +ioas_copy)  |                 |                    | map/unmap
> +            |                 |                    |
> +     +------V------+    +-----V------+      +------V--------+
> +     | iommfd core |    |  device    |      |  vfio iommu   |
> +     +-------------+    +------------+      +---------------+
> +
> +[Secure Context setup]
> +- iommufd BE: uses device fd and iommufd to setup secure context
> +              (bind_iommufd, attach_ioas)
> +- vfio legacy BE: uses group fd and container fd to setup secure context
> +                  (set_container, set_iommu)
> +
> +[Device access]
> +- iommufd BE: device fd is opened through /dev/vfio/devices/vfioX
> +- vfio legacy BE: device fd is retrieved from group fd ioctl
> +
> +[DMA Mapping flow]
> +1. VFIOAddressSpace receives MemoryRegion add/del via MemoryListener
> +2. VFIO populates DMA map/unmap via the container BEs
> +   *) iommufd BE: uses iommufd
> +   *) vfio legacy BE: uses container fd
> +
> +
> +Example configuration
> +=====================
> +
> +Step 1: configure the host device
> +---------------------------------
> +
> +It's exactly same as the VFIO device with legacy VFIO container.
> +
> +Step 2: configure QEMU
> +----------------------
> +
> +Interactions with the /dev/iommu are abstracted by a new
> +iommufd object (compiled in with the CONFIG_IOMMUFD option).
> +
> +Any QEMU device (e.g. VFIO device) wishing to use /dev/iommu must be
> +linked with an iommufd object. It gets a new optional property named
> +iommufd which allows to pass an iommufd object. Take vfio-pci device
> +for example:
> +
> +    -object iommufd,id=iommufd0
> +    -device vfio-pci,host=0000:02:00.0,iommufd=iommufd0
> +
> +Note the /dev/iommu and VFIO cdev can be externally opened by a
> +management layer. In such a case the fd is passed, the fd supports
> +a string naming the fd or a number, for example:
> +
> +    -object iommufd,id=iommufd0,fd=22
> +    -device vfio-pci,iommufd=iommufd0,fd=23
> +
> +If the fd property is not passed, the fd is opened by QEMU.
> +
> +If no iommufd property is passed to the vfio-pci device, iommufd is
> +not used and the user gets the behavior based on the legacy VFIO
> +container:
> +
> +    -device vfio-pci,host=0000:02:00.0
> +
> +Supported platform
> +==================
> +
> +Supports X86, ARM and S390X currently.