[PATCH v2 00/14] ublk: allow off-daemon zero-copy buffer registration

[Caleb Sander Mateos <csander@purestorage.com>]

Currently ublk zero-copy requires ublk request buffers to be registered
and unregistered by the ublk I/O's daemon task. However, as currently
implemented, there is no reason for this restriction. Registration looks
up the request via the ublk device's tagset rather than the daemon-local
ublk_io structure and takes an atomic reference to prevent racing with
dispatch or completion of the request. Ming has expressed interest in
relaxing this restriction[1] so the ublk server can offload the I/O
operation that uses the zero-copy buffer to another thread.

Additionally, optimize the buffer registration for the common case where
the buffer is registered and unregistered by the daemon task. Skip the
expensive atomic reference count increment and decrement and the several
pointer dereferences involved in looking up the request on the tagset.
On our ublk server threads, this results in a 24% decrease in the CPU
time spent in the ublk functions on a 4K zero-copy read workload, from
8.75% to 6.68%.

Two preliminary fixes are included as well:
- Move the ublk request reference count from ublk_rq_data (a tail
  allocation of struct request) to ublk_io to prevent a use after free
  if the struct request is freed concurrently with taking a reference.
- Don't allocate physically contiguous memory for __queues, which can be
  very large. Doubling the size of struct ublk_io caused ENOMEM errors
  when adding a ublk device before this change.

[1]: https://lore.kernel.org/linux-block/aAmYJxaV1-yWEMRo@fedora/

v2:
- Add 2 fix patches
- Optimize buffer unregistration too
- Cache-align ublk_io
- Restore check for zero copy support in ublk_unregister_io_buf()
- Check for registered buffer count overflow

Caleb Sander Mateos (14):
  ublk: use vmalloc for ublk_device's __queues
  ublk: remove struct ublk_rq_data
  ublk: check cmd_op first
  ublk: handle UBLK_IO_FETCH_REQ earlier
  ublk: remove task variable from __ublk_ch_uring_cmd()
  ublk: consolidate UBLK_IO_FLAG_{ACTIVE,OWNED_BY_SRV} checks
  ublk: move ublk_prep_cancel() to case UBLK_IO_COMMIT_AND_FETCH_REQ
  ublk: don't take ublk_queue in ublk_unregister_io_buf()
  ublk: allow UBLK_IO_(UN)REGISTER_IO_BUF on any task
  ublk: return early if blk_should_fake_timeout()
  ublk: optimize UBLK_IO_REGISTER_IO_BUF on daemon task
  ublk: optimize UBLK_IO_UNREGISTER_IO_BUF on daemon task
  ublk: remove ubq checks from ublk_{get,put}_req_ref()
  ublk: cache-align struct ublk_io

 drivers/block/ublk_drv.c      | 283 +++++++++++++++++++++-------------
 include/uapi/linux/ublk_cmd.h |  10 ++
 2 files changed, 187 insertions(+), 106 deletions(-)

-- 
2.45.2