[PATCH v2 00/14] fuse: add io-uring buffer rings and zero-copy

[Joanne Koong <joannelkoong@gmail.com>]

This series adds buffer ring and zero-copy capabilities to fuse over io-uring.

Using buffer rings has advantages over the non-buffer-ring (iovec) path:
- Reduced memory usage: in the iovec path, each entry has its own
  dedicated payload buffer, requiring N buffers for N entries where each
  buffer must be large enough to accommodate the maximum possible
  payload size. With buffer rings, payload buffers are pooled and
  selected on demand. Entries only hold a buffer while actively
  processing a request with payload data. When incremental buffer
  consumption is added, this will allow non-overlapping regions of a
  single buffer to be used simultaneously across multiple requests,
  further reducing memory requirements.
- Foundation for pinned buffers: the buffer ring headers and payloads
  are now each passed in as a contiguous memory allocation, which allows
  fuse to easily pin and vmap the entire region in one operation during
  queue setup. This will eliminate the per-request overhead of having to
  pin/unpin user pages and translate virtual addresses and is a
  prerequisite for future optimizations like performing data copies
  outside of the server's task context.

This series adds the capability to pin the underlying header and payload
buffers by setting init flags at registration time, depending on the user's
mlock limit.

Zero-copy (only for privileged servers) is also opt-in by setting an init flag
at registration time. Zero-copy eliminates the memory copies between kernel and
userspace for read/write/payload-heavy operations by allowing the server to
directly operate on the client's underlying pages.

This series has a dependency on io-uring registered bvec buffers changes
in [1].

The throughput improvements from pinned buffers and zero-copy depends on how
much of the server's per-request latency is spent on data copying vs backing
I/O. When backing I/O dominates, the saved memcpy is a negligible fraction of
overall latency. Please also note that for the server to read/write
into the zero-copied pages, the read/write must go through io-uring
as an IORING_OP_READ_FIXED / IORING_OP_WRITE_FIXED operation. If the
server's backing I/O is instantaneous (eg served from cache), the
overhead of the additional io_uring operation may negate the savings
from eliminating the memcpy.

In benchmarks using passthrough_hp on a high-performance NVMe-backed
system, pinned headers and pinned payload buffers showed around a 10%
throughput improvement for direct randreads (~2150 MiB/s to ~2400
MiB/s), a 4% improvement for direct sequential reads (~2510 MiB/s to
~2620 MiB/s), a 8% improvement for buffered randreads (~2100 MiB/s to
~2280 MiB/s), and a 6% improvement for buffered sequential reads (~2500
MiB/s to ~2670 MiB/s).

Zero-copy showed around a 35% throughput improvement for direct
randreads (~2150 MiB/s to ~2900 MiB/s), a 15% improvement for direct
sequential reads (~2510 MiB/s to ~2900 MiB/s), a 15% improvement for
buffered randreads (~2100 MiB/s to ~2470 MiB/s), and a 10% improvement
for buffered sequential reads (~2500 MiB/s to ~2750 MiB/s). I didn't see
enough of a clear improvement for writes due to write latency being I/O
dominated.

The benchmarks were run using:
  fio --name=test_run --ioengine=sync --rw=rand{read,write} --bs=1M
  --size=1G --numjobs=2 --ramp_time=30 --group_reporting=1

To run the benchmark, please also add this patch [2].

The libfuse changes can be found in [3]. To test the server, run:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp
--nopassthrough -o io_uring_zero_copy -o io_uring_q_depth=8
Once this series is merged, the libfuse changes will be tidied up and
submitted upstream.

Further optimizations for incremental buffer consumption, request
dispatching in current task context, and backing buffer integration with
IORING_OP_READ/IORING_OP_WRITE operations will be submitted as part of a
separate series.

Thanks,
Joanne 

[1] https://lore.kernel.org/io-uring/20260402160929.2749744-1-joannelkoong@gmail.com/T/#t
[2] https://lore.kernel.org/linux-fsdevel/20260326215127.3857682-2-joannelkoong@gmail.com/
[3] https://github.com/joannekoong/libfuse/commits/zero_copy_v2/

Changelog
---------
v1: https://lore.kernel.org/linux-fsdevel/20260324224532.3733468-1-joannelkoong@gmail.com/
v1 -> v2:
* Drop kernel managed buffers from io-uring infrastructure and instead move
  logic into fuse. To later use buffers with io-uring requests natively will
  require fuse to place the backing buffer as a fixed buffer in a sparse slot
  for the server, but that will be added as an optimization in a separate
  series. This makes the io-uring code cleaner and accomodates for more flexible
  fuse user configurations (eg mlock limits) and easier setup (me)
* Run more benchmarks and get more numbers (me)
* Add visual diagrams and more documentatoin to commit messages and
  documentation patch (Bernd)

Joanne Koong (14):
  fuse: separate next request fetching from sending logic
  fuse: refactor io-uring header copying to ring
  fuse: refactor io-uring header copying from ring
  fuse: use enum types for header copying
  fuse: refactor setting up copy state for payload copying
  fuse: support buffer copying for kernel addresses
  fuse: use named constants for io-uring iovec indices
  fuse: move fuse_uring_abort() from header to dev_uring.c
  fuse: rearrange io-uring iovec and ent allocation logic
  fuse: add io-uring buffer rings
  fuse: add pinned headers capability for io-uring buffer rings
  fuse: add pinned payload buffers capability for io-uring buffer rings
  fuse: add zero-copy over io-uring
  docs: fuse: add io-uring bufring and zero-copy documentation

 .../filesystems/fuse/fuse-io-uring.rst        |  189 +++
 fs/fuse/dev.c                                 |   30 +-
 fs/fuse/dev_uring.c                           | 1042 ++++++++++++++---
 fs/fuse/dev_uring_i.h                         |   86 +-
 fs/fuse/fuse_dev_i.h                          |    8 +-
 include/uapi/linux/fuse.h                     |   36 +-
 6 files changed, 1194 insertions(+), 197 deletions(-)


base-commit: 619fa72e875483dabf7683001496cc0ca4480aa6
-- 
2.52.0