Re: MMIO/PIO dispatch file descriptors (ioregionfd) design discussion

Re: MMIO/PIO dispatch file descriptors (ioregionfd) design discussion

[elena]

On Wed, Nov 25, 2020 at 12:44:07PM -0800, Elena Afanasova wrote:
> Hello,
>

Hi

Sorry for top-posting, just wanted to provide a quik update.
We are currently working on the support for ioregionfd in Qemu and will
be posting the patches soon. Plus the KVM patches will be posted based
of the RFC v3 with some fixes if there are no objections from Elena's side
who originally posted KVM RFC patchset.


Thanks!
Elena Ufimtseva

> I'm an Outreachy intern with QEMU and I’m working on implementing the ioregionfd 
> API in KVM. So I’d like to resume the ioregionfd design discussion. The latest 
> version of the ioregionfd API document is provided below.
> 
> Overview
> --------
> ioregionfd is a KVM dispatch mechanism for handling MMIO/PIO accesses over a
> file descriptor without returning from ioctl(KVM_RUN). This allows device
> emulation to run in another task separate from the vCPU task.
> 
> This is achieved through KVM ioctls for registering MMIO/PIO regions and a wire
> protocol that KVM uses to communicate with a task handling an MMIO/PIO access.
> 
> The traditional ioctl(KVM_RUN) dispatch mechanism with device emulation in a
> separate task looks like this:
> 
>    kvm.ko  <---ioctl(KVM_RUN)---> VMM vCPU task <---messages---> device task
> 
> ioregionfd improves performance by eliminating the need for the vCPU task to
> forward MMIO/PIO exits to device emulation tasks:
> 
>    kvm.ko  <---ioctl(KVM_RUN)---> VMM vCPU task
>      ^
>      `---ioregionfd---> device task
> 
> Both multi-threaded and multi-process VMMs can take advantage of ioregionfd to
> run device emulation in dedicated threads and processes, respectively.
> 
> This mechanism is similar to ioeventfd except it supports all read and write
> accesses, whereas ioeventfd only supports posted doorbell writes.
> 
> Traditional ioctl(KVM_RUN) dispatch and ioeventfd continue to work alongside
> the new mechanism, but only one mechanism handles a MMIO/PIO access.
> 
> KVM_CREATE_IOREGIONFD
> ---------------------
> :Capability: KVM_CAP_IOREGIONFD
> :Architectures: all
> :Type: system ioctl
> :Parameters: none
> :Returns: an ioregionfd file descriptor, -1 on error
> 
> This ioctl creates a new ioregionfd and returns the file descriptor. The fd can
> be used to handle MMIO/PIO accesses instead of returning from ioctl(KVM_RUN)
> with KVM_EXIT_MMIO or KVM_EXIT_PIO. One or more MMIO or PIO regions must be
> registered with KVM_SET_IOREGION in order to receive MMIO/PIO accesses on the
> fd. An ioregionfd can be used with multiple VMs and its lifecycle is not tied
> to a specific VM.
> 
> When the last file descriptor for an ioregionfd is closed, all regions
> registered with KVM_SET_IOREGION are dropped and guest accesses to those
> regions cause ioctl(KVM_RUN) to return again.
> 
> KVM_SET_IOREGION
> ----------------
> :Capability: KVM_CAP_IOREGIONFD
> :Architectures: all
> :Type: vm ioctl
> :Parameters: struct kvm_ioregion (in)
> :Returns: 0 on success, -1 on error
> 
> This ioctl adds, modifies, or removes an ioregionfd MMIO or PIO region. Guest
> read and write accesses are dispatched through the given ioregionfd instead of
> returning from ioctl(KVM_RUN).
> 
> ::
> 
>   struct kvm_ioregion {
>       __u64 guest_paddr; /* guest physical address */
>       __u64 memory_size; /* bytes */
>       __u64 user_data;
>       __s32 fd; /* previously created with KVM_CREATE_IOREGIONFD */
>       __u32 flags;
>       __u8  pad[32];
>   };
> 
>   /* for kvm_ioregion::flags */
>   #define KVM_IOREGION_PIO           (1u << 0)
>   #define KVM_IOREGION_POSTED_WRITES (1u << 1)
> 
> If a new region would split an existing region -1 is returned and errno is
> EINVAL.
> 
> Regions can be deleted by setting fd to -1. If no existing region matches
> guest_paddr and memory_size then -1 is returned and errno is ENOENT.
> 
> Existing regions can be modified as long as guest_paddr and memory_size
> match an existing region.
> 
> MMIO is the default. The KVM_IOREGION_PIO flag selects PIO instead.
> 
> The user_data value is included in messages KVM writes to the ioregionfd upon
> guest access. KVM does not interpret user_data.
> 
> Both read and write guest accesses wait for a response before entering the
> guest again. The KVM_IOREGION_POSTED_WRITES flag does not wait for a response
> and immediately enters the guest again. This is suitable for accesses that do
> not require synchronous emulation, such as posted doorbell register writes.
> Note that guest writes may block the vCPU despite KVM_IOREGION_POSTED_WRITES if
> the device is too slow in reading from the ioregionfd.
> 
> Wire protocol
> -------------
> The protocol spoken over the file descriptor is as follows. The device reads
> commands from the file descriptor with the following layout::
> 
>   struct ioregionfd_cmd {
>       __u32 info;
>       __u32 padding;
>       __u64 user_data;
>       __u64 offset;
>       __u64 data;
>   };
> 
> The info field layout is as follows::
> 
>   bits:  | 31 ... 8 |  6   | 5 ... 4 | 3 ... 0 |
>   field: | reserved | resp |   size  |   cmd   |
> 
> The cmd field identifies the operation to perform::
> 
>   #define IOREGIONFD_CMD_READ  0
>   #define IOREGIONFD_CMD_WRITE 1
> 
> The size field indicates the size of the access::
> 
>   #define IOREGIONFD_SIZE_8BIT  0
>   #define IOREGIONFD_SIZE_16BIT 1
>   #define IOREGIONFD_SIZE_32BIT 2
>   #define IOREGIONFD_SIZE_64BIT 3
> 
> If the command is IOREGIONFD_CMD_WRITE then the resp bit indicates whether or
> not a response must be sent.
> 
> The user_data field contains the opaque value provided to KVM_SET_IOREGION.
> Applications can use this to uniquely identify the region that is being
> accessed.
> 
> The offset field contains the byte offset being accessed within a region
> that was registered with KVM_SET_IOREGION.
> 
> If the command is IOREGIONFD_CMD_WRITE then data contains the value
> being written. The data value is a 64-bit integer in host endianness,
> regardless of the access size.
> 
> The device sends responses by writing the following structure to the
> file descriptor::
> 
>   struct ioregionfd_resp {
>       __u64 data;
>       __u8 pad[24];
>   };
> 
> The data field contains the value read by an IOREGIONFD_CMD_READ
> command. This field is zero for other commands. The data value is a 64-bit
> integer in host endianness, regardless of the access size.
> 
> Ordering
> --------
> Guest accesses are delivered in order, including posted writes.
> 
> Signals
> -------
> The vCPU task can be interrupted by a signal while waiting for an ioregionfd
> response. In this case ioctl(KVM_RUN) returns with -EINTR. Guest entry is
> deferred until ioctl(KVM_RUN) is called again and the response has been written
> to the ioregionfd.
> 
> Security
> --------
> Device emulation processes may be untrusted in multi-process VMM architectures.
> Therefore the control plane and the data plane of ioregionfd are separate. A
> task that only has access to an ioregionfd is unable to add/modify/remove
> regions since that requires ioctls on a KVM vm fd. This ensures that device
> emulation processes can only service MMIO/PIO accesses for regions that the VMM
> registered on their behalf.
> 
> Multi-queue scalability
> -----------------------
> The protocol is synchronous - only one command/response cycle is in flight at a
> time - but the vCPU will be blocked until the response has been processed
> anyway. If another vCPU accesses an MMIO or PIO region belonging to the same
> ioregionfd during this time then it waits for the first access to complete.
> 
> Per-queue ioregionfds can be set up to take advantage of concurrency on
> multi-queue devices.
> 
> Polling
> -------
> Userspace can poll ioregionfd by submitting an io_uring IORING_OP_READ request
> and polling the cq ring to detect when the read has completed. Although this
> dispatch mechanism incurs more overhead than polling directly on guest RAM, it
> captures each write access and supports reads.
> 
> Does it obsolete ioeventfd?
> ---------------------------
> No, although KVM_IOREGION_POSTED_WRITES offers somewhat similar functionality
> to ioeventfd, there are differences. The datamatch functionality of ioeventfd
> is not available and would need to be implemented by the device emulation
> program. Due to the counter semantics of eventfds there is automatic coalescing
> of repeated accesses with ioeventfd. Overall ioeventfd is lighter weight but
> also more limited.
> 

Re: MMIO/PIO dispatch file descriptors (ioregionfd) design discussion

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1790 bytes --]

On Mon, Oct 11, 2021 at 10:34:29PM -0700, elena wrote:
> On Wed, Nov 25, 2020 at 12:44:07PM -0800, Elena Afanasova wrote:
> > Hello,
> >
> 
> Hi
> 
> Sorry for top-posting, just wanted to provide a quik update.
> We are currently working on the support for ioregionfd in Qemu and will
> be posting the patches soon. Plus the KVM patches will be posted based
> of the RFC v3 with some fixes if there are no objections from Elena's side
> who originally posted KVM RFC patchset.

Hi Elena,
I'm curious what approach you want to propose for QEMU integration. A
while back I thought about the QEMU API. It's possible to implement it
along the lines of the memory_region_add_eventfd() API where each
ioregionfd is explicitly added by device emulation code. An advantage of
this approach is that a MemoryRegion can have multiple ioregionfds, but
I'm not sure if that is a useful feature.

An alternative is to cover the entire MemoryRegion with one ioregionfd.
That way the device emulation code can use ioregionfd without much fuss
since there is a 1:1 mapping between MemoryRegions, which are already
there in existing devices. There is no need to think deeply about which
ioregionfds to create for a device.

A new API called memory_region_set_aio_context(MemoryRegion *mr,
AioContext *ctx) would cause ioregionfd (or a userspace fallback for
non-KVM cases) to execute the MemoryRegion->read/write() accessors from
the given AioContext. The details of ioregionfd are hidden behind the
memory_region_set_aio_context() API, so the device emulation code
doesn't need to know the capabilities of ioregionfd.

The second approach seems promising if we want more devices to use
ioregionfd inside QEMU because it requires less ioregionfd-specific
code.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: MMIO/PIO dispatch file descriptors (ioregionfd) design discussion

[Elena]

On Mon, Oct 25, 2021 at 01:42:56PM +0100, Stefan Hajnoczi wrote:
> On Mon, Oct 11, 2021 at 10:34:29PM -0700, elena wrote:
> > On Wed, Nov 25, 2020 at 12:44:07PM -0800, Elena Afanasova wrote:
> > > Hello,
> > >
> > 
> > Hi
> > 
> > Sorry for top-posting, just wanted to provide a quik update.
> > We are currently working on the support for ioregionfd in Qemu and will
> > be posting the patches soon. Plus the KVM patches will be posted based
> > of the RFC v3 with some fixes if there are no objections from Elena's side
> > who originally posted KVM RFC patchset.
> 
> Hi Elena,

Hello Stefan.

> I'm curious what approach you want to propose for QEMU integration. A
> while back I thought about the QEMU API. It's possible to implement it
> along the lines of the memory_region_add_eventfd() API where each
> ioregionfd is explicitly added by device emulation code. An advantage of
> this approach is that a MemoryRegion can have multiple ioregionfds, but
> I'm not sure if that is a useful feature.
>

This is the approach that is currently in the works. Agree, I dont see
much of the application here at this point to have multiple ioregions
per MemoryRegion.
I added Memory API/eventfd approach to the vfio-user as well to try
things out.

> An alternative is to cover the entire MemoryRegion with one ioregionfd.
> That way the device emulation code can use ioregionfd without much fuss
> since there is a 1:1 mapping between MemoryRegions, which are already
> there in existing devices. There is no need to think deeply about which
> ioregionfds to create for a device.
>
> A new API called memory_region_set_aio_context(MemoryRegion *mr,
> AioContext *ctx) would cause ioregionfd (or a userspace fallback for
> non-KVM cases) to execute the MemoryRegion->read/write() accessors from
> the given AioContext. The details of ioregionfd are hidden behind the
> memory_region_set_aio_context() API, so the device emulation code
> doesn't need to know the capabilities of ioregionfd.

> 
> The second approach seems promising if we want more devices to use
> ioregionfd inside QEMU because it requires less ioregionfd-specific
> code.
> 
I like this approach as well.
As you have mentioned, the device emulation code with first approach
does have to how to handle the region accesses. The second approach will
make things more transparent. Let me see how can I modify what there is
there now and may ask further questions.

Thank you for your input Stefan.
Elena

> Stefan

Re: MMIO/PIO dispatch file descriptors (ioregionfd) design discussion

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2714 bytes --]

On Mon, Oct 25, 2021 at 08:21:22AM -0700, Elena wrote:
> On Mon, Oct 25, 2021 at 01:42:56PM +0100, Stefan Hajnoczi wrote:
> > On Mon, Oct 11, 2021 at 10:34:29PM -0700, elena wrote:
> > > On Wed, Nov 25, 2020 at 12:44:07PM -0800, Elena Afanasova wrote:
> > > > Hello,
> > > >
> > > 
> > > Hi
> > > 
> > > Sorry for top-posting, just wanted to provide a quik update.
> > > We are currently working on the support for ioregionfd in Qemu and will
> > > be posting the patches soon. Plus the KVM patches will be posted based
> > > of the RFC v3 with some fixes if there are no objections from Elena's side
> > > who originally posted KVM RFC patchset.
> > 
> > Hi Elena,
> 
> Hello Stefan.
> 
> > I'm curious what approach you want to propose for QEMU integration. A
> > while back I thought about the QEMU API. It's possible to implement it
> > along the lines of the memory_region_add_eventfd() API where each
> > ioregionfd is explicitly added by device emulation code. An advantage of
> > this approach is that a MemoryRegion can have multiple ioregionfds, but
> > I'm not sure if that is a useful feature.
> >
> 
> This is the approach that is currently in the works. Agree, I dont see
> much of the application here at this point to have multiple ioregions
> per MemoryRegion.
> I added Memory API/eventfd approach to the vfio-user as well to try
> things out.
> 
> > An alternative is to cover the entire MemoryRegion with one ioregionfd.
> > That way the device emulation code can use ioregionfd without much fuss
> > since there is a 1:1 mapping between MemoryRegions, which are already
> > there in existing devices. There is no need to think deeply about which
> > ioregionfds to create for a device.
> >
> > A new API called memory_region_set_aio_context(MemoryRegion *mr,
> > AioContext *ctx) would cause ioregionfd (or a userspace fallback for
> > non-KVM cases) to execute the MemoryRegion->read/write() accessors from
> > the given AioContext. The details of ioregionfd are hidden behind the
> > memory_region_set_aio_context() API, so the device emulation code
> > doesn't need to know the capabilities of ioregionfd.
> 
> > 
> > The second approach seems promising if we want more devices to use
> > ioregionfd inside QEMU because it requires less ioregionfd-specific
> > code.
> > 
> I like this approach as well.
> As you have mentioned, the device emulation code with first approach
> does have to how to handle the region accesses. The second approach will
> make things more transparent. Let me see how can I modify what there is
> there now and may ask further questions.

Thanks, I look forward to patches you are working on!

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: MMIO/PIO dispatch file descriptors (ioregionfd) design discussion

[John Levon]

On Mon, Oct 25, 2021 at 08:21:22AM -0700, Elena wrote:

> > I'm curious what approach you want to propose for QEMU integration. A
> > while back I thought about the QEMU API. It's possible to implement it
> > along the lines of the memory_region_add_eventfd() API where each
> > ioregionfd is explicitly added by device emulation code. An advantage of
> > this approach is that a MemoryRegion can have multiple ioregionfds, but
> > I'm not sure if that is a useful feature.
> >
> 
> This is the approach that is currently in the works. Agree, I dont see
> much of the application here at this point to have multiple ioregions
> per MemoryRegion.
> I added Memory API/eventfd approach to the vfio-user as well to try
> things out.
> 
> > An alternative is to cover the entire MemoryRegion with one ioregionfd.
> > That way the device emulation code can use ioregionfd without much fuss
> > since there is a 1:1 mapping between MemoryRegions, which are already
> > there in existing devices. There is no need to think deeply about which
> > ioregionfds to create for a device.
> >
> > A new API called memory_region_set_aio_context(MemoryRegion *mr,
> > AioContext *ctx) would cause ioregionfd (or a userspace fallback for
> > non-KVM cases) to execute the MemoryRegion->read/write() accessors from
> > the given AioContext. The details of ioregionfd are hidden behind the
> > memory_region_set_aio_context() API, so the device emulation code
> > doesn't need to know the capabilities of ioregionfd.
> 
> > 
> > The second approach seems promising if we want more devices to use
> > ioregionfd inside QEMU because it requires less ioregionfd-specific
> > code.
> > 
> I like this approach as well.
> As you have mentioned, the device emulation code with first approach
> does have to how to handle the region accesses. The second approach will
> make things more transparent. Let me see how can I modify what there is
> there now and may ask further questions.

Sorry I'm a bit late to this discussion, I'm not clear on the above WRT
vfio-user. If an ioregionfd has to cover a whole BAR0 (?), how would this
interact with partly-mmap()able regions like we do with SPDK/vfio-user/NVMe?

thanks
john

Re: MMIO/PIO dispatch file descriptors (ioregionfd) design discussion

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2649 bytes --]

On Tue, Oct 26, 2021 at 08:01:39PM +0100, John Levon wrote:
> On Mon, Oct 25, 2021 at 08:21:22AM -0700, Elena wrote:
> 
> > > I'm curious what approach you want to propose for QEMU integration. A
> > > while back I thought about the QEMU API. It's possible to implement it
> > > along the lines of the memory_region_add_eventfd() API where each
> > > ioregionfd is explicitly added by device emulation code. An advantage of
> > > this approach is that a MemoryRegion can have multiple ioregionfds, but
> > > I'm not sure if that is a useful feature.
> > >
> > 
> > This is the approach that is currently in the works. Agree, I dont see
> > much of the application here at this point to have multiple ioregions
> > per MemoryRegion.
> > I added Memory API/eventfd approach to the vfio-user as well to try
> > things out.
> > 
> > > An alternative is to cover the entire MemoryRegion with one ioregionfd.
> > > That way the device emulation code can use ioregionfd without much fuss
> > > since there is a 1:1 mapping between MemoryRegions, which are already
> > > there in existing devices. There is no need to think deeply about which
> > > ioregionfds to create for a device.
> > >
> > > A new API called memory_region_set_aio_context(MemoryRegion *mr,
> > > AioContext *ctx) would cause ioregionfd (or a userspace fallback for
> > > non-KVM cases) to execute the MemoryRegion->read/write() accessors from
> > > the given AioContext. The details of ioregionfd are hidden behind the
> > > memory_region_set_aio_context() API, so the device emulation code
> > > doesn't need to know the capabilities of ioregionfd.
> > 
> > > 
> > > The second approach seems promising if we want more devices to use
> > > ioregionfd inside QEMU because it requires less ioregionfd-specific
> > > code.
> > > 
> > I like this approach as well.
> > As you have mentioned, the device emulation code with first approach
> > does have to how to handle the region accesses. The second approach will
> > make things more transparent. Let me see how can I modify what there is
> > there now and may ask further questions.
> 
> Sorry I'm a bit late to this discussion, I'm not clear on the above WRT
> vfio-user. If an ioregionfd has to cover a whole BAR0 (?), how would this
> interact with partly-mmap()able regions like we do with SPDK/vfio-user/NVMe?

The ioregionfd doesn't need to cover an entire BAR. QEMU's MemoryRegions
form a hierarchy, so it's possible to sub-divide the BAR into several
MemoryRegions.

This means it's still possible to have mmap() sub-regions or even
ioeventfds sprinkled in between.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: MMIO/PIO dispatch file descriptors (ioregionfd) design discussion

[John Levon]

On Wed, Oct 27, 2021 at 11:15:35AM +0100, Stefan Hajnoczi wrote:

> > > I like this approach as well.
> > > As you have mentioned, the device emulation code with first approach
> > > does have to how to handle the region accesses. The second approach will
> > > make things more transparent. Let me see how can I modify what there is
> > > there now and may ask further questions.
> > 
> > Sorry I'm a bit late to this discussion, I'm not clear on the above WRT
> > vfio-user. If an ioregionfd has to cover a whole BAR0 (?), how would this
> > interact with partly-mmap()able regions like we do with SPDK/vfio-user/NVMe?
> 
> The ioregionfd doesn't need to cover an entire BAR. QEMU's MemoryRegions
> form a hierarchy, so it's possible to sub-divide the BAR into several
> MemoryRegions.

I think you're saying that when vfio-user client in qemu calls
VFIO_USER_DEVICE_GET_REGION_IO_FDS, it would create a sub-MR corresponding to
each one, before asking KVM to configure them?

thanks
john

Re: MMIO/PIO dispatch file descriptors (ioregionfd) design discussion

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1300 bytes --]

On Wed, Oct 27, 2021 at 01:22:28PM +0100, John Levon wrote:
> On Wed, Oct 27, 2021 at 11:15:35AM +0100, Stefan Hajnoczi wrote:
> 
> > > > I like this approach as well.
> > > > As you have mentioned, the device emulation code with first approach
> > > > does have to how to handle the region accesses. The second approach will
> > > > make things more transparent. Let me see how can I modify what there is
> > > > there now and may ask further questions.
> > > 
> > > Sorry I'm a bit late to this discussion, I'm not clear on the above WRT
> > > vfio-user. If an ioregionfd has to cover a whole BAR0 (?), how would this
> > > interact with partly-mmap()able regions like we do with SPDK/vfio-user/NVMe?
> > 
> > The ioregionfd doesn't need to cover an entire BAR. QEMU's MemoryRegions
> > form a hierarchy, so it's possible to sub-divide the BAR into several
> > MemoryRegions.
> 
> I think you're saying that when vfio-user client in qemu calls
> VFIO_USER_DEVICE_GET_REGION_IO_FDS, it would create a sub-MR corresponding to
> each one, before asking KVM to configure them?

Yes. Actually I wasn't thinking of the vfio-user client but just about
QEMU device emulation code in general. What you suggested sounds like a
clean mapping from MemoryRegions to vfio-user.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]