Re: [dpdk-dev] [RFC v1 0/2] Add device emulation support in DPDK

[Stephen Hemminger <stephen@networkplumber.org>]

On Fri, 14 Aug 2020 19:16:04 +0000
Chenbo Xia <chenbo.xia@intel.com> wrote:

> This series enables DPDK to be an alternative I/O device emulation library of
> building virtualized devices in separate processes outside QEMU. It introduces
> a new library (librte_vfio_user), a new device class (emudev) and one pilot
> device provider (avf_emudev) with its backend of Ethdev PMD (avfbe_ethdev).
> 
> *librte_vfio_user* is a server implementation of VFIO-over-socket[1] (also
> known as vfio-user) which is a protocol that allows a device to be virtualized
> in a separate process outside of QEMU. 
> 
> *emudev* is a device type for emulated devices. It is up to device provider to
> choose the transport. In avf_emudev case, it uses vfio-user as transport
> communicate with its client (e.g., QEMU).
> 
> *avf_emudev* is the emudev provider of AVF which is a device specification for
> Intel Virtual Function cross generation. It’s implemented by an AVF emudev
> driver which offers a few APIs for avfbe_ethdev or app logic to operate. 
> 
> *avfbe_ethdev* is a normal ethdev PMD to supply the basic I/O as backend data
> path of avf_emudev. One simple usage of avfbe_ethdev could be a para-virtualized
> backend connected with network application logic.
> 
> Background & Motivation
> -----------------------
> In order to reduce the attack surface, QEMU community is disaggregating QEMU by
> removing part of device emulation from it. The disaggregated/multi-process QEMU
> is using VFIO-over-socket/vfio-user as the main transport mechanism to disaggregate
> I/O services from QEMU[2]. Vfio-user essentially implements the VFIO device model
> presented to the user process by a set of messages over a unix-domain socket. The
> main difference between application using vfio-user and application using vfio
> kernel module is that device manipulation is based on socket messages for vfio-user
> but system calls for vfio kernel module. The vfio-user devices consist of a generic
> VFIO device type, living in QEMU, which is called the client[3], and the core device
> implementation (emulated device), living outside of QEMU, which is called the server. 
> 
> With the introduction and support of vfio-user in QEMU, QEMU is explicitly adding
> support for external emulated device and data path. We are trying to leverage that
> and introducing vfio-user support in DPDK. By doing so, DPDK is enabled to be an
> alternative I/O device emulation library of building virtualized devices along with
> high-performance data path in separate processes outside QEMU. It will be easy for
> hardware vendors to provide virtualized solutions of their hardware devices by
> implementing emulated device in DPDK.
> 
> Except for vfio-user introduced in DPDK, this series also introduces the first
> emulated device implementation. That is emulated AVF device (avf_emudev) implemented
> by AVF emulation driver (avf_emudev driver). Emulated AVF device demos how emulated
> device could be implemented in DPDK. SPDK is also investigating to implement use case
> for NVMe. 
> 
> Design overview
> ---------------
> 
>                     +------------------------------------------------------+
>                     |   +---------------+      +---------------+           |
>                     |   |  avf_emudev   |      |  avfbe_ethdev |           |
>                     |   |    driver     |      |     driver    |           |
>                     |   +---------------+      +---------------+           |
>                     |           |                       |                  |
>                     | ------------------------------------------- VDEV BUS |
>                     |           |                       |                  |
>                     |   +---------------+       +--------------+           |
> +--------------+    |   | vdev:         |       | vdev:        |           |
> | +----------+ |    |   | /path/to/vfio |       | avf_emudev_# |           |
> | | Generic  | |    |   +---------------+       +--------------+           |
> | | vfio-dev | |    |           |                                          |
> | +----------+ |    |           |                                          |
> | +----------+ |    |      +----------+                                    |
> | | vfio-user| |    |      | vfio-user|                                    |
> | | client   | |<---|----->| server   |                                    |
> | +----------+ |    |      +----------+                                    |
> | QEMU         |    | DPDK                                                 |
> +--------------+    +------------------------------------------------------+
> 
> - vfio-user. Vfio-user in DPDK is referred to the vfio-user protocol implementation
> playing server role. It provides transport between emulated device and generic VFIO
> device in QEMU. Emulated device in DPDK and generic VFIO device in QEMU are working
> together to present VFIO device model to VM. This series introduces vfio-user
> implementation as a library called librte_vfio_user which is under lib/librte_vfio_user.
> 
> - vdev:/path/to/vfio. It defines the emudev device and binds to vdev bus driver. The
> emudev device is defined by DPDK applications through command line as '--vdev=emu_iavf,
> path=/path/to/socket, id=#' in avf_emudev case. Parameters in command line include device
> name (emu_iavf) which is used to identify corresponding driver (in this case, avf_emudev
> driver which implements emudev device of AVF), path=/path/to/socket which is used to open
> the transport interface to vfio-user client in QEMU, and id which is the index of emudev
> device.
> 
> - avf_emudev driver. It implements emulated AVF device which is the emudev provider of
> AVF. The avf_emudev_driver offers a few APIs implementation exposed by emudev device APIs
> for avfbe_ethdev_pmd or application logic to operate. These APIs are described in
> lib/librte_emudev/rte_emudev.h.
> 
> - vdev: avf_emudev_#. The vdev device is defined by DPDK application through command line
> as '--vdev=net_avfbe,id=#,avf_emu_id=#'.It is associated with emudev provider of AVF by
> 'avf_emu_id=#'.
>      
> - avfbe_ethdev driver. It is a normal ethdev PMD to supply the basic I/O as backend data
> path of avf_emudev.
> 
> Why not rawdev for emulated device
> ----------------------------------
> Instead of introducing new class emudev, emulated device could be presented as rawdev.
> However, existing rawdev APIs cannot meet the requirements of emulated device. There are
> three API categories for emudev. They are emudev device lifecycle management, backend
> facing APIs, and emudev device provider facing APIs respectively. Existing rawdev APIs
> could only cover lifecycle management APIs and some of backend facing APIs. Other APIs,
> even if added to rawdev API are not required by other rawdev applications.
>  
> References
> ----------
> [1]: https://patchew.org/QEMU/1594913503-52271-1-git-send-email-thanos.makatos@nutanix.com/
> [2]: https://wiki.qemu.org/Features/MultiProcessQEMU
> [3]: https://github.com/elmarco/qemu/blob/wip/vfio-user/hw/vfio/libvfio-user.c
> 
> Chenbo Xia (2):
>   vfio_user: Add library for vfio over socket
>   emudev: Add library for emulated device
> 
>  lib/librte_emudev/rte_emudev.h       | 315 +++++++++++++++++++++++++
>  lib/librte_vfio_user/rte_vfio_user.h | 335 +++++++++++++++++++++++++++
>  2 files changed, 650 insertions(+)
>  create mode 100644 lib/librte_emudev/rte_emudev.h
>  create mode 100644 lib/librte_vfio_user/rte_vfio_user.h
> 

This looks good, but it would be good to have an example or way to integrate
it into a test framework.  One of the agree upon principles by the tech board
is "all new features should have test cases". There have been a lot of exceptions though.