Re: MMIO/PIO dispatch file descriptors (ioregionfd) design discussion

[Jason Wang <jasowang@redhat.com>]

On 2020/11/26 上午3:21, Elena Afanasova wrote:
> Hello,
>
> I'm an Outreachy intern with QEMU and I’m working on implementing the 
> ioregionfd API in KVM.
> So I’d like to resume the ioregionfd design discussion. The latest 
> version of the ioregionfd API document is provided below.
>
> Overview
> --------
> ioregionfd is a KVM dispatch mechanism for handling MMIO/PIO accesses 
> over a
> file descriptor without returning from ioctl(KVM_RUN). This allows device
> emulation to run in another task separate from the vCPU task.
>
> This is achieved through KVM ioctls for registering MMIO/PIO regions 
> and a wire
> protocol that KVM uses to communicate with a task handling an MMIO/PIO 
> access.
>
> The traditional ioctl(KVM_RUN) dispatch mechanism with device 
> emulation in a
> separate task looks like this:
>
>    kvm.ko  <---ioctl(KVM_RUN)---> VMM vCPU task <---messages---> 
> device task
>
> ioregionfd improves performance by eliminating the need for the vCPU 
> task to
> forward MMIO/PIO exits to device emulation tasks:


I wonder at which cases we care performance like this. (Note that 
vhost-user suppots set|get_config() for a while).


>
>    kvm.ko  <---ioctl(KVM_RUN)---> VMM vCPU task
>      ^
>      `---ioregionfd---> device task


It's better to draw a device task via the KVM_RUN path to show the 
possible advantage.


>
> Both multi-threaded and multi-process VMMs can take advantage of 
> ioregionfd to
> run device emulation in dedicated threads and processes, respectively.
>
> This mechanism is similar to ioeventfd except it supports all read and 
> write
> accesses, whereas ioeventfd only supports posted doorbell writes.
>
> Traditional ioctl(KVM_RUN) dispatch and ioeventfd continue to work 
> alongside
> the new mechanism, but only one mechanism handles a MMIO/PIO access.
>
> KVM_CREATE_IOREGIONFD
> ---------------------
> :Capability: KVM_CAP_IOREGIONFD
> :Architectures: all
> :Type: system ioctl
> :Parameters: none
> :Returns: an ioregionfd file descriptor, -1 on error
>
> This ioctl creates a new ioregionfd and returns the file descriptor. 
> The fd can
> be used to handle MMIO/PIO accesses instead of returning from 
> ioctl(KVM_RUN)
> with KVM_EXIT_MMIO or KVM_EXIT_PIO. One or more MMIO or PIO regions 
> must be
> registered with KVM_SET_IOREGION in order to receive MMIO/PIO accesses 
> on the
> fd. An ioregionfd can be used with multiple VMs and its lifecycle is 
> not tied
> to a specific VM.
>
> When the last file descriptor for an ioregionfd is closed, all regions
> registered with KVM_SET_IOREGION are dropped and guest accesses to those
> regions cause ioctl(KVM_RUN) to return again.


I may miss something, but I don't see any special requirement of this 
fd. The fd just a transport of a protocol between KVM and userspace 
process. So instead of mandating a new type, it might be better to allow 
any type of fd to be attached. (E.g pipe or socket).


>
> KVM_SET_IOREGION
> ----------------
> :Capability: KVM_CAP_IOREGIONFD
> :Architectures: all
> :Type: vm ioctl
> :Parameters: struct kvm_ioregion (in)
> :Returns: 0 on success, -1 on error
>
> This ioctl adds, modifies, or removes an ioregionfd MMIO or PIO 
> region. Guest
> read and write accesses are dispatched through the given ioregionfd 
> instead of
> returning from ioctl(KVM_RUN).
>
> ::
>
>   struct kvm_ioregion {
>       __u64 guest_paddr; /* guest physical address */
>       __u64 memory_size; /* bytes */
>       __u64 user_data;
>       __s32 fd; /* previously created with KVM_CREATE_IOREGIONFD */
>       __u32 flags;
>       __u8  pad[32];
>   };
>
>   /* for kvm_ioregion::flags */
>   #define KVM_IOREGION_PIO           (1u << 0)
>   #define KVM_IOREGION_POSTED_WRITES (1u << 1)
>
> If a new region would split an existing region -1 is returned and errno is
> EINVAL.
>
> Regions can be deleted by setting fd to -1. If no existing region matches
> guest_paddr and memory_size then -1 is returned and errno is ENOENT.
>
> Existing regions can be modified as long as guest_paddr and memory_size
> match an existing region.
>
> MMIO is the default. The KVM_IOREGION_PIO flag selects PIO instead.
>
> The user_data value is included in messages KVM writes to the 
> ioregionfd upon
> guest access. KVM does not interpret user_data.
>
> Both read and write guest accesses wait for a response before entering the
> guest again. The KVM_IOREGION_POSTED_WRITES flag does not wait for a 
> response
> and immediately enters the guest again. This is suitable for accesses 
> that do
> not require synchronous emulation, such as posted doorbell register 
> writes.
> Note that guest writes may block the vCPU despite 
> KVM_IOREGION_POSTED_WRITES if
> the device is too slow in reading from the ioregionfd.
>
> Wire protocol
> -------------
> The protocol spoken over the file descriptor is as follows. The device 
> reads
> commands from the file descriptor with the following layout::
>
>   struct ioregionfd_cmd {
>       __u32 info;
>       __u32 padding;
>       __u64 user_data;
>       __u64 offset;
>       __u64 data;
>   };
>
> The info field layout is as follows::
>
>   bits:  | 31 ... 8 |  6   | 5 ... 4 | 3 ... 0 |
>   field: | reserved | resp |   size  |   cmd   |
>
> The cmd field identifies the operation to perform::
>
>   #define IOREGIONFD_CMD_READ  0
>   #define IOREGIONFD_CMD_WRITE 1
>
> The size field indicates the size of the access::
>
>   #define IOREGIONFD_SIZE_8BIT  0
>   #define IOREGIONFD_SIZE_16BIT 1
>   #define IOREGIONFD_SIZE_32BIT 2
>   #define IOREGIONFD_SIZE_64BIT 3
>
> If the command is IOREGIONFD_CMD_WRITE then the resp bit indicates 
> whether or
> not a response must be sent.
>
> The user_data field contains the opaque value provided to 
> KVM_SET_IOREGION.
> Applications can use this to uniquely identify the region that is being
> accessed.
>
> The offset field contains the byte offset being accessed within a region
> that was registered with KVM_SET_IOREGION.
>
> If the command is IOREGIONFD_CMD_WRITE then data contains the value
> being written. The data value is a 64-bit integer in host endianness,
> regardless of the access size.
>
> The device sends responses by writing the following structure to the
> file descriptor::
>
>   struct ioregionfd_resp {
>       __u64 data;
>       __u8 pad[24];
>   };
>
> The data field contains the value read by an IOREGIONFD_CMD_READ
> command. This field is zero for other commands. The data value is a 64-bit
> integer in host endianness, regardless of the access size.
>
> Ordering
> --------
> Guest accesses are delivered in order, including posted writes.
>
> Signals
> -------
> The vCPU task can be interrupted by a signal while waiting for an 
> ioregionfd
> response. In this case ioctl(KVM_RUN) returns with -EINTR. Guest entry is
> deferred until ioctl(KVM_RUN) is called again and the response has 
> been written
> to the ioregionfd.
>
> Security
> --------
> Device emulation processes may be untrusted in multi-process VMM 
> architectures.
> Therefore the control plane and the data plane of ioregionfd are 
> separate. A
> task that only has access to an ioregionfd is unable to add/modify/remove
> regions since that requires ioctls on a KVM vm fd. This ensures that 
> device
> emulation processes can only service MMIO/PIO accesses for regions 
> that the VMM
> registered on their behalf.
>
> Multi-queue scalability
> -----------------------
> The protocol is synchronous - only one command/response cycle is in 
> flight at a
> time - but the vCPU will be blocked until the response has been processed
> anyway. If another vCPU accesses an MMIO or PIO region belonging to 
> the same
> ioregionfd during this time then it waits for the first access to 
> complete.
>
> Per-queue ioregionfds can be set up to take advantage of concurrency on
> multi-queue devices.
>
> Polling
> -------
> Userspace can poll ioregionfd by submitting an io_uring IORING_OP_READ 
> request
> and polling the cq ring to detect when the read has completed. 
> Although this
> dispatch mechanism incurs more overhead than polling directly on guest 
> RAM, it
> captures each write access and supports reads.
>
> Does it obsolete ioeventfd?
> ---------------------------
> No, although KVM_IOREGION_POSTED_WRITES offers somewhat similar 
> functionality
> to ioeventfd, there are differences. The datamatch functionality of 
> ioeventfd
> is not available and would need to be implemented by the device emulation
> program.


This means another dispatching layer in the device emulation.

Thanks


> Due to the counter semantics of eventfds there is automatic coalescing
> of repeated accesses with ioeventfd. Overall ioeventfd is lighter 
> weight but
> also more limited.