Re: [PATCH 00/17] io_uring passthru over nvme

Christoph Hellwig
2022-03-10T08:29:38Z

What branch is this against?

Do you have a git tree available?

On Tue, Mar 08, 2022 at 08:50:48PM +0530, Kanchan Joshi wrote:
> This is a streamlined series with new way of doing uring-cmd, and connects
> nvme-passthrough (over char device /dev/ngX) to it.
> uring-cmd enables using io_uring for any arbitrary command (ioctl,
> fsctl etc.) exposed by the command provider (e.g. driver, fs etc.).
>
> To store the command inline within the sqe, Jens added an option to setup
> the ring with 128-byte SQEs.This gives 80 bytes of space (16 bytes at
> the end of the first sqe + 64 bytes in the second sqe). With inline
> command in sqe, the application avoids explicit allocation and, in the
> kernel, we avoid doing copy_from_user. Command-opcode, length etc.
> are stored in per-op fields of io_uring_sqe.
>
> Non-inline submission (when command is a user-space pointer rather than
> housed inside sqe) is also supported.
>
> io_uring sends this command down by newly introduced ->async_cmd()
> handler in file_operations. The handler does what is required to
> submit, and indicates queued completion.The infra has been added to
> process the completion when it arrives.
>
> Overall the patches wire up the following capabilities for this path:
> - async
> - fixed-buffer
> - plugging
> - bio-cache
> - sync and async polling.
>
> This scales well. 512b randread perf (KIOPS) comparing
> uring-passthru-over-char (/dev/ng0n1) to
> uring-over-block(/dev/nvme0n1)
>
> QD uring pt uring-poll pt-poll
> 8 538 589 831 902
> 64 967 1131 1351 1378
> 256 1043 1230 1376 1429
>
> Testing/perf is done with this custom fio that turnes regular-io into
> passthru-io on supplying "uring_cmd=1" option.
> https://github.com/joshkan/fio/tree/big-sqe-pt.v1
>
> Example command-line:
> fio -iodepth=256 -rw=randread -ioengine=io_uring -bs=512 -numjobs=1
> -runtime=60 -group_reporting -iodepth_batch_submit=64
> -iodepth_batch_complete_min=1 -iodepth_batch_complete_max=64
> -fixedbufs=1 -hipri=1 -sqthread_poll=0 -filename=/dev/ng0n1
> -name=io_uring_256 -uring_cmd=1
>
>
> Anuj Gupta (3):
> io_uring: prep for fixed-buffer enabled uring-cmd
> nvme: enable passthrough with fixed-buffer
> nvme: enable non-inline passthru commands
>
> Jens Axboe (5):
> io_uring: add support for 128-byte SQEs
> fs: add file_operations->async_cmd()
> io_uring: add infra and support for IORING_OP_URING_CMD
> io_uring: plug for async bypass
> block: wire-up support for plugging
>
> Kanchan Joshi (5):
> nvme: wire-up support for async-passthru on char-device.
> io_uring: add support for uring_cmd with fixed-buffer
> block: factor out helper for bio allocation from cache
> nvme: enable bio-cache for fixed-buffer passthru
> io_uring: add support for non-inline uring-cmd
>
> Keith Busch (2):
> nvme: modify nvme_alloc_request to take an additional parameter
> nvme: allow user passthrough commands to poll
>
> Pankaj Raghav (2):
> io_uring: add polling support for uring-cmd
> nvme: wire-up polling for uring-passthru
>
> block/bio.c | 43 ++--
> block/blk-map.c | 45 +++++
> block/blk-mq.c | 93 ++++-----
> drivers/nvme/host/core.c | 21 +-
> drivers/nvme/host/ioctl.c | 336 +++++++++++++++++++++++++++-----
> drivers/nvme/host/multipath.c | 2 +
> drivers/nvme/host/nvme.h | 11 +-
> drivers/nvme/host/pci.c | 4 +-
> drivers/nvme/target/passthru.c | 2 +-
> fs/io_uring.c | 188 ++++++++++++++++--
> include/linux/bio.h | 1 +
> include/linux/blk-mq.h | 4 +
> include/linux/fs.h | 2 +
> include/linux/io_uring.h | 43 ++++
> include/uapi/linux/io_uring.h | 21 +-
> include/uapi/linux/nvme_ioctl.h | 4 +
> 16 files changed, 689 insertions(+), 131 deletions(-)
>
> --
> 2.25.1
---end quoted text---