Re: [PATCH 00/17] io_uring passthru over nvme

[Christoph Hellwig <hch@lst.de>]

What branch is this against?

Do you have a git tree available?

On Tue, Mar 08, 2022 at 08:50:48PM +0530, Kanchan Joshi wrote:
> This is a streamlined series with new way of doing uring-cmd, and connects
> nvme-passthrough (over char device /dev/ngX) to it.
> uring-cmd enables using io_uring for any arbitrary command (ioctl,
> fsctl etc.) exposed by the command provider (e.g. driver, fs etc.).
> 
> To store the command inline within the sqe, Jens added an option to setup
> the ring with 128-byte SQEs.This gives 80 bytes of space (16 bytes at
> the end of the first sqe + 64 bytes in the second sqe). With inline
> command in sqe, the application avoids explicit allocation and, in the
> kernel, we avoid doing copy_from_user. Command-opcode, length etc.
> are stored in per-op fields of io_uring_sqe.
> 
> Non-inline submission (when command is a user-space pointer rather than
> housed inside sqe) is also supported.
> 
> io_uring sends this command down by newly introduced ->async_cmd()
> handler in file_operations. The handler does what is required to
> submit, and indicates queued completion.The infra has been added to
> process the completion when it arrives.
> 
> Overall the patches wire up the following capabilities for this path:
> - async
> - fixed-buffer
> - plugging
> - bio-cache
> - sync and async polling.
> 
> This scales well. 512b randread perf (KIOPS) comparing
> uring-passthru-over-char (/dev/ng0n1) to
> uring-over-block(/dev/nvme0n1)
> 
> QD    uring    pt    uring-poll    pt-poll
> 8      538     589      831         902
> 64     967     1131     1351        1378
> 256    1043    1230     1376        1429
> 
> Testing/perf is done with this custom fio that turnes regular-io into
> passthru-io on supplying "uring_cmd=1" option.
> https://github.com/joshkan/fio/tree/big-sqe-pt.v1
> 
> Example command-line:
> fio -iodepth=256 -rw=randread -ioengine=io_uring -bs=512 -numjobs=1
> -runtime=60 -group_reporting -iodepth_batch_submit=64
> -iodepth_batch_complete_min=1 -iodepth_batch_complete_max=64
> -fixedbufs=1 -hipri=1 -sqthread_poll=0 -filename=/dev/ng0n1
> -name=io_uring_256 -uring_cmd=1
> 
> 
> Anuj Gupta (3):
>   io_uring: prep for fixed-buffer enabled uring-cmd
>   nvme: enable passthrough with fixed-buffer
>   nvme: enable non-inline passthru commands
> 
> Jens Axboe (5):
>   io_uring: add support for 128-byte SQEs
>   fs: add file_operations->async_cmd()
>   io_uring: add infra and support for IORING_OP_URING_CMD
>   io_uring: plug for async bypass
>   block: wire-up support for plugging
> 
> Kanchan Joshi (5):
>   nvme: wire-up support for async-passthru on char-device.
>   io_uring: add support for uring_cmd with fixed-buffer
>   block: factor out helper for bio allocation from cache
>   nvme: enable bio-cache for fixed-buffer passthru
>   io_uring: add support for non-inline uring-cmd
> 
> Keith Busch (2):
>   nvme: modify nvme_alloc_request to take an additional parameter
>   nvme: allow user passthrough commands to poll
> 
> Pankaj Raghav (2):
>   io_uring: add polling support for uring-cmd
>   nvme: wire-up polling for uring-passthru
> 
>  block/bio.c                     |  43 ++--
>  block/blk-map.c                 |  45 +++++
>  block/blk-mq.c                  |  93 ++++-----
>  drivers/nvme/host/core.c        |  21 +-
>  drivers/nvme/host/ioctl.c       | 336 +++++++++++++++++++++++++++-----
>  drivers/nvme/host/multipath.c   |   2 +
>  drivers/nvme/host/nvme.h        |  11 +-
>  drivers/nvme/host/pci.c         |   4 +-
>  drivers/nvme/target/passthru.c  |   2 +-
>  fs/io_uring.c                   | 188 ++++++++++++++++--
>  include/linux/bio.h             |   1 +
>  include/linux/blk-mq.h          |   4 +
>  include/linux/fs.h              |   2 +
>  include/linux/io_uring.h        |  43 ++++
>  include/uapi/linux/io_uring.h   |  21 +-
>  include/uapi/linux/nvme_ioctl.h |   4 +
>  16 files changed, 689 insertions(+), 131 deletions(-)
> 
> -- 
> 2.25.1
---end quoted text---