Re: Requirements for out-of-process device emulation

[Alex Williamson <alex.williamson@redhat.com>]

On Fri, 9 Oct 2020 17:18:15 +0100
Stefan Hajnoczi <stefanha@redhat.com> wrote:

> Device emulation
> ----------------
> Device resources
> ````````````````
> Devices provide resources that drivers interact with such as hardware
> registers, memory, or interrupts. The fundamental requirement of
> out-of-process device emulation is exposing device resources.
> 
> The following types of device resources are needed:
> 
> Synchronous MMIO/PIO accesses
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> The most basic device emulation operation is the hardware register
> access. This is a memory-mapped I/O (MMIO) or programmed I/O (PIO)
> access to the device. A read loads a value from a device register. A
> write stores a value to a device register. These operations are
> synchronous because the vCPU is paused until completion.
> Asynchronous doorbells
> 
> Devices often have doorbell registers, allowing the driver to inform the
> device that new requests are ready for processing. The vCPU does not
> need to wait since the access is a posted write.
> 
> The kvm.ko ioeventfd mechanism can be used to implement asynchronous
> doorbells.
> 
> Shared device memory
> ~~~~~~~~~~~~~~~~~~~~
> Devices may have memory-like regions that the CPU can access (such as
> PCI Memory BARs). The device emulation process therefore needs to share
> a region of its memory space with the VMM so the guest can access it.
> This mechanism also allows device emulation to busy wait (poll) instead
> of using synchronous MMIO/PIO accesses or asynchronous doorbells for
> notifications.
> 
> Direct Memory Access (DMA)
> ~~~~~~~~~~~~~~~~~~~~~~~~~~
> Devices often require read and write access to a memory address space
> belonging to the CPU. This allows network cards to transmit packet
> payloads that are located in guest RAM, for example.
> 
> Early out-of-process device emulation interfaces simply shared guest
> RAM. The allowed DMA to any guest physical memory address. More advanced
> IOMMU and address space identifier mechanisms are now becoming
> ubiquitous. Therefore, new out-of-process device emulation interfaces
> should incorporate IOMMU functionality.
> 
> The key requirement for IOMMU mechanisms is allowing the VMM to grant
> access to a region of memory so the device emulation process can read
> from and/or write to it.
> 
> Interrupts
> ~~~~~~~~~~
> Devices notify the CPU using interrupts. An interrupt is simply a
> message sent by the device emulation process to the VMM. Interrupt
> configuration is flexible on modern devices, meaning the driver may be
> able to select the number of interrupts and a mapping (using one
> interrupt with multiple event sources). This can be implemented using
> the Linux eventfd mechanism or via in-band device emulation protocol
> messages, for example.
> 
> Extensibility for new bus types
> ```````````````````````````````
> It should be possible to support multiple bus types. vhost-user only
> supports vhost devices. VFIO is more extensible but currently focussed
> on PCI devices.

Wait a sec, the vfio API essentially deconstructs devices into exactly
the resources you've outlined above.  We not only have a vfio-pci
device convention within vfio, but we've defined vfio-platform,
vfio-amba, vfio-ccw, vfio-ap, and we'll likely be adding vfio-fsl-mc in
the next kernel.  The core device, group, and container model within
vfio is completely device/bus agnostic.  So while it's true that
vfio-pci is the most mature and featureful convention, that's largely a
reflection that PCI is the most ubiquitous device interface currently
available.  Thanks,

Alex