ioregionfd with io_uring IORING_OP_URING_CMD

ioregionfd with io_uring IORING_OP_URING_CMD

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1737 bytes --]

Hi,
During Elena Afanasova's Outreachy project we discussed whether
ioregionfd should be a custom struct file_operations (anon inode) or a
userspace-provided file (socketpair, UNIX domain socket, etc).

Back then it seemed more flexible and simpler to let userspace provide
the file. It may be worth revisiting this decision in light of the
recent io_uring IORING_OP_URING_CMD feature, which fits for this
performance-critical interface.

IORING_OP_URING_CMD involves a new struct file_operations->uring_cmd()
callback. It's a flexible userspace interface like ioctl(2) but designed
to be asynchronous. ioregionfd can provide a uring_cmd() that reads a
ioregionfd response from userspace and then writes the next ioregionfd
request to userspace.

This single operation merges the request/response so only 1 syscall is
necessary per KVM MMIO/PIO exit instead of a read() + write(). Bypassing
the net/socket infrastructure is likely to help too.

It would be interesting to benchmark this and compare it against the
existing userspace-provided file approach. Although it's not the same
scenario, results for the Linux NVMe driver using ->uring_cmd() are
promising:
https://www.snia.org/educational-library/enabling-asynchronous-i-o-passthru-nvme-native-applications-2021

The downside is it requires more code than general purpose I/O. In
addition to ->uring_cmd(), it's also worth implementing struct
file_operations read/write/poll so traditional file I/O syscalls work
for simple applications that don't want to use io_uring.

It's possible to add ->uring_cmd() later but as a userspace developer I
would prefer the ->uring_cmd() approach, so I'm not sure it's worth
committing to the existing userspace-provided file approach?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: ioregionfd with io_uring IORING_OP_URING_CMD

[Elena]

On Mon, Jun 06, 2022 at 03:57:47PM +0100, Stefan Hajnoczi wrote:
> Hi,
> During Elena Afanasova's Outreachy project we discussed whether
> ioregionfd should be a custom struct file_operations (anon inode) or a
> userspace-provided file (socketpair, UNIX domain socket, etc).
>

Hello Stefan,

> Back then it seemed more flexible and simpler to let userspace provide
> the file. It may be worth revisiting this decision in light of the
> recent io_uring IORING_OP_URING_CMD feature, which fits for this
> performance-critical interface.
>
And Paolo was asking about io_uring in the review of the initial
patches.

> IORING_OP_URING_CMD involves a new struct file_operations->uring_cmd()
> callback. It's a flexible userspace interface like ioctl(2) but designed
> to be asynchronous. ioregionfd can provide a uring_cmd() that reads a
> ioregionfd response from userspace and then writes the next ioregionfd
> request to userspace.
> 
> This single operation merges the request/response so only 1 syscall is
> necessary per KVM MMIO/PIO exit instead of a read() + write(). Bypassing
> the net/socket infrastructure is likely to help too.
> 
> It would be interesting to benchmark this and compare it against the
> existing userspace-provided file approach. Although it's not the same
> scenario, results for the Linux NVMe driver using ->uring_cmd() are
> promising:
> https://www.snia.org/educational-library/enabling-asynchronous-i-o-passthru-nvme-native-applications-2021
> 
Yes, looks interesting, we were thinking about adding this.

> The downside is it requires more code than general purpose I/O. In
> addition to ->uring_cmd(), it's also worth implementing struct
> file_operations read/write/poll so traditional file I/O syscalls work
> for simple applications that don't want to use io_uring.
> 
> It's possible to add ->uring_cmd() later but as a userspace developer I
> would prefer the ->uring_cmd() approach, so I'm not sure it's worth
> committing to the existing userspace-provided file approach?

Makes total sense. I am going to start working on this and will
come back with more questions.

Thank you!
Elena
> 
> Stefan

Re: ioregionfd with io_uring IORING_OP_URING_CMD

[Stefan Hajnoczi]

On Tue, 7 Jun 2022 at 14:32, Elena <elena.ufimtseva@oracle.com> wrote:
> On Mon, Jun 06, 2022 at 03:57:47PM +0100, Stefan Hajnoczi wrote:
> > The downside is it requires more code than general purpose I/O. In
> > addition to ->uring_cmd(), it's also worth implementing struct
> > file_operations read/write/poll so traditional file I/O syscalls work
> > for simple applications that don't want to use io_uring.
> >
> > It's possible to add ->uring_cmd() later but as a userspace developer I
> > would prefer the ->uring_cmd() approach, so I'm not sure it's worth
> > committing to the existing userspace-provided file approach?
>
> Makes total sense. I am going to start working on this and will
> come back with more questions.

Good to hear!

Userspace needs a way to create these fds. I think a new
ioctl(KVM_CREATE_IOREGIONFD) is needed. Then the fd can be passed back
to KVM_SET_IOREGION.

Stefan