* [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy
@ 2026-08-14 18:59 Joanne Koong
2026-08-14 18:59 ` [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration Joanne Koong
` (7 more replies)
0 siblings, 8 replies; 27+ messages in thread
From: Joanne Koong @ 2026-08-14 18:59 UTC (permalink / raw)
To: miklos; +Cc: jlayton, libaokun, axboe, bernd, amir73il, fuse-devel
This series implements fuse io-uring buffer pools and zero-copy.
Prior to this series, ents and buffers are tightly coupled where each entry
has its own dedicated payload buffer, requiring N buffers for N entries where
each buffer must be large enough to accomodate the maximum payload size. This
is suboptimal as most request types require vastly less bytes than the maximum
payload size and some requests do not require payload buffers at all.
This series allows servers to pass in a buffer pool (a contiguous chunk of
memory) that the kernel will use as it wishes for servicing ents/requests.
This decoupling reduces the memory usage requirements needed to use
fuse-io-uring and lets the kernel do any optimizations for assigning payload
memory to requests.
This series additionally adds zero copy to fuse io-uring. The server can
directly access client pages or page cache folios without copying data through
an intermediary buffer. This requires CAP_SYS_ADMIN privileges and using
buffer pools. The zero copy patch has a dependency on io-uring registered
bvec changes in [1].
This series is on top of commit 7d87a5a284bb and on top of the io-uring bvec
changes and the changes from the series in [2] applied.
The throughput improvements from registered buffers and zero-copy depends on
how much of the server's per-request latency is spent on data copying vs
backing I/O. When backing I/O dominates, the saved memcpy is a negligible
fraction of overall latency. Please also note that for the server to
read/write into the zero-copied pages, the read/write must go through io-uring
as an IORING_OP_READ_FIXED / IORING_OP_WRITE_FIXED operation.
The throughput improvement from zero-copy depends on how much of the
per-request latency is spent on data copying vs backing I/O. The gain
comes from eliminating the payload-buffer memcpy, but accessing the
zero-copied pages requires the server to issue the read/write as an
IORING_OP_READ/WRITE_FIXED operation. The benefit is largest when the
mempcy is a meaningful fraction of per-request latency while backing i/o
is still noticable enough that the extra io-uring op's overhead doesn't
dominate.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered-buf zero-copy (zc vs base)
direct read ~5.1 GB/s ~5.4 GB/s ~8.9 GB/s (+75%)
direct write ~3.4 GB/s ~4.8 GB/s ~5.1 GB/s (+50%)
On a device-bound NVMe (~2 GB/s reads) the read gain shrinks to ~10-16% (no
measurable gains for writes), as backing I/O rather than the eliminated copy
dominates latency. The benefit overall scales with how much of the per-request
latency is the data copy versus backing I/O.
The benchmark script and results can be found in [3]. The libfuse changes can
be found in [4]. To test the server, run:
sudo ~/libfuse/build/example/passthrough_hp ~/src ~/mounts/tmp --nopassthrough
-o io_uring_zero_copy -o io_uring_q_depth=8
Once this series is merged, the libfuse changes will be tidied up and
submitted upstream.
Thanks,
Joanne
[1] https://lore.kernel.org/io-uring/20260612184840.4058966-1-joannelkoong@gmail.com/T/#t
[2] https://lore.kernel.org/fuse-devel/20260715174305.336261-1-joannelkoong@gmail.com/
[3] https://github.com/joannekoong/linux/commits/fuse_zero_copy_benchmarks/
[4] https://github.com/joannekoong/libfuse/commits/zero_copy_v7
Changelog
---------
v6: https://lore.kernel.org/fuse-devel/20260716175908.2339738-1-joannelkoong@gmail.com/
v6 -> v7:
* Use kvmalloc_flex() instead of kmalloc (Bernd)
* Remove registered bufpool flag and add reserved add-bufpool flag, as per
Bernd's preference (Bernd)
v5: https://lore.kernel.org/fuse-devel/20260630211436.2062816-1-joannelkoong@gmail.com/
v5 -> v6:
* Remove WARN_ON tag, add missing barrier (Sashiko)
* Bring back some documentation from v4, rename uapi doc (Amir)
v4: https://lore.kernel.org/fuse-devel/20260612210513.1516038-1-joannelkoong@gmail.com/
v4 -> v5:
* Address Miklos's feedback (separate uring cmd for adding bufpool, pass only
bufpool addr and len, pass back bufpool offset instead of buf id, etc)
* Make zero-copy opt-in on file open
* Do any zeroing for short zero-copy reads instead of accidentally skipping
* that
* Drop Baokun and Jeff's reviewed-bys since the commits had modifications
* Run more rigorous benchmarks, on bare-metal machine
v3: https://lore.kernel.org/fuse-devel/20260522205823.1597313-1-joannelkoong@gmail.com/
v3 -> v4:
* Add reviewed-bys
* Fix documentation typo, add paragraph about aborts to zero-copy commit
message, undo unnecessary padding[6] change, add FUSE_HAS_URING_BUFPOOL
advertisement
v2: https://lore.kernel.org/linux-fsdevel/20260402162840.2989717-1-joannelkoong@gmail.com/
v2 -> v3:
* Rework the uapis to be more ergonomic. Use io-uring registered buffers
infrastructure instead of doing pinning logic in fuse. Get rid of header
pinning as that makes no real perf difference
* Rename from "buffer ring" to buffer pool. Logic is the same, just different
naming
v1: https://lore.kernel.org/linux-fsdevel/20260324224532.3733468-1-joannelkoong@gmail.com/
v1 -> v2:
* Drop kernel managed buffers from io-uring infrastructure and move it to fuse
* Add visual diagrams and more documentatoin to commit messages and
documentation patch
Joanne Koong (6):
fuse: decouple fuse_ring creation from ent registration
fuse: add FUSE_IO_URING_CMD_ADD_QUEUE
fuse: add io-uring buffer pools
fuse: support registered buffer pools in io-uring
fuse: add zero-copy over io-uring
docs: fuse: document io-uring buffer pool and zero-copy uapi
.../filesystems/fuse/fuse-io-uring.rst | 36 +-
Documentation/filesystems/fuse/index.rst | 1 +
.../fuse/uapi/fuse-uapi-io-uring.rst | 126 ++++
fs/fuse/args.h | 2 +
fs/fuse/dev.c | 32 +-
fs/fuse/dev.h | 2 +-
fs/fuse/dev_uring.c | 572 +++++++++++++++---
fs/fuse/dev_uring_i.h | 56 +-
fs/fuse/file.c | 2 +
fs/fuse/fuse_dev_i.h | 2 +
fs/fuse/inode.c | 6 +-
include/uapi/linux/fuse.h | 63 +-
12 files changed, 816 insertions(+), 84 deletions(-)
create mode 100644 Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst
--
2.52.0
^ permalink raw reply [flat|nested] 27+ messages in thread
* [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-14 18:59 [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
@ 2026-08-14 18:59 ` Joanne Koong
2026-08-19 11:34 ` Miklos Szeredi
2026-08-14 18:59 ` [PATCH v7 2/6] fuse: add FUSE_IO_URING_CMD_ADD_QUEUE Joanne Koong
` (6 subsequent siblings)
7 siblings, 1 reply; 27+ messages in thread
From: Joanne Koong @ 2026-08-14 18:59 UTC (permalink / raw)
To: miklos; +Cc: jlayton, libaokun, axboe, bernd, amir73il, fuse-devel
Currently, the connection's fuse_ring is created lazily on the first
FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
thread per queue (one per CPU) and those threads issue their first
REGISTER command concurrently. They then race to create the single
per-connection fuse_ring, which required open-coded handling in
fuse_uring_create() to detect and protect against concurrent creations.
Decouple fuse_ring creation from ent registration and move it to
FUSE_INIT reply processing after a server has negotiated and set
FUSE_OVER_IO_URING. The ring is published before the connection is
marked initialized. fuse_uring_register() no longer creates the ring and
it instead uses the ring set up at init time.
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
fs/fuse/dev.c | 8 +++-----
fs/fuse/dev.h | 2 +-
fs/fuse/dev_uring.c | 26 ++++++++++----------------
fs/fuse/dev_uring_i.h | 5 +++++
fs/fuse/inode.c | 4 +++-
5 files changed, 22 insertions(+), 23 deletions(-)
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 27dafda2a841..d8f97943e973 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -75,6 +75,9 @@ void fuse_chan_set_initialized(struct fuse_chan *fch, struct fuse_chan_param *pa
fch->minor = param->minor;
fch->max_write = param->max_write;
fch->max_pages = param->max_pages;
+
+ if (param->io_uring_enabled)
+ fuse_uring_conn_init(fch);
}
/* Pairs with smp_load_acquire() readers of fch->initialized */
@@ -412,11 +415,6 @@ void fuse_chan_set_fc(struct fuse_chan *fch, struct fuse_conn *fc)
fch->conn = fc;
}
-void fuse_chan_io_uring_enable(struct fuse_chan *fch)
-{
- fch->io_uring = 1;
-}
-
void fuse_pqueue_init(struct fuse_pqueue *fpq)
{
spin_lock_init(&fpq->lock);
diff --git a/fs/fuse/dev.h b/fs/fuse/dev.h
index aed69fd14c41..8d25378c0918 100644
--- a/fs/fuse/dev.h
+++ b/fs/fuse/dev.h
@@ -22,6 +22,7 @@ struct fuse_chan_param {
unsigned int minor;
unsigned int max_write;
unsigned int max_pages;
+ bool io_uring_enabled;
};
struct fuse_chan *fuse_chan_new(void);
@@ -34,7 +35,6 @@ void fuse_chan_max_background_set(struct fuse_chan *fch, unsigned int val);
unsigned int fuse_chan_num_waiting(struct fuse_chan *fch);
void fuse_chan_set_fc(struct fuse_chan *fch, struct fuse_conn *fc);
void fuse_chan_set_initialized(struct fuse_chan *fch, struct fuse_chan_param *param);
-void fuse_chan_io_uring_enable(struct fuse_chan *fch);
ssize_t fuse_chan_send(struct fuse_chan *fch, struct fuse_args *args);
int fuse_chan_send_bg(struct fuse_chan *fch, struct fuse_args *args, gfp_t gfp_flags);
int fuse_chan_send_notify_reply(struct fuse_chan *fch, struct fuse_args *args, u64 unique);
diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
index c8488ebc1d1f..9616778505ba 100644
--- a/fs/fuse/dev_uring.c
+++ b/fs/fuse/dev_uring.c
@@ -238,7 +238,6 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch)
{
struct fuse_ring *ring;
size_t nr_queues = num_possible_cpus();
- struct fuse_ring *res = NULL;
size_t max_payload_size;
ring = kzalloc_obj(*ring, GFP_KERNEL_ACCOUNT);
@@ -258,12 +257,6 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch)
spin_unlock(&fch->lock);
goto out_err;
}
- if (fch->ring) {
- /* race, another thread created the ring in the meantime */
- spin_unlock(&fch->lock);
- res = fch->ring;
- goto out_err;
- }
init_waitqueue_head(&ring->stop_waitq);
@@ -278,7 +271,13 @@ static struct fuse_ring *fuse_uring_create(struct fuse_chan *fch)
out_err:
kfree(ring->queues);
kfree(ring);
- return res;
+ return NULL;
+}
+
+void fuse_uring_conn_init(struct fuse_chan *fch)
+{
+ if (fuse_uring_create(fch))
+ fch->io_uring = 1;
}
static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
@@ -1178,15 +1177,10 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
struct fuse_ring *ring = smp_load_acquire(&fch->ring);
struct fuse_ring_queue *queue;
struct fuse_ring_ent *ent;
- int err;
unsigned int qid = READ_ONCE(cmd_req->qid);
- err = -ENOMEM;
- if (!ring) {
- ring = fuse_uring_create(fch);
- if (!ring)
- return err;
- }
+ if (!ring)
+ return -EINVAL;
if (qid >= ring->nr_queues) {
pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid);
@@ -1197,7 +1191,7 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
if (!queue) {
queue = fuse_uring_create_queue(ring, qid);
if (!queue)
- return err;
+ return -ENOMEM;
}
/*
diff --git a/fs/fuse/dev_uring_i.h b/fs/fuse/dev_uring_i.h
index 55f8d04e4b0b..d721a4fc0215 100644
--- a/fs/fuse/dev_uring_i.h
+++ b/fs/fuse/dev_uring_i.h
@@ -135,6 +135,7 @@ struct fuse_ring {
bool ready;
};
+void fuse_uring_conn_init(struct fuse_chan *fch);
void fuse_uring_stop_queues(struct fuse_ring *ring);
void fuse_uring_abort_end_requests(struct fuse_ring *ring);
int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags);
@@ -174,6 +175,10 @@ static inline bool fuse_uring_ready(struct fuse_chan *fch)
#else /* CONFIG_FUSE_IO_URING */
+static inline void fuse_uring_conn_init(struct fuse_chan *fch)
+{
+}
+
static inline void fuse_uring_abort(struct fuse_chan *fch)
{
}
diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
index f7a0a0860a04..33773c7d129a 100644
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -1272,6 +1272,7 @@ static void process_init_reply(struct fuse_args *args, int error)
struct fuse_mount *fm = ia->fm;
struct fuse_conn *fc = fm->fc;
struct fuse_init_out *arg = &ia->out;
+ bool io_uring_enabled = false;
bool ok = true;
if (error || arg->major != FUSE_KERNEL_VERSION)
@@ -1402,7 +1403,7 @@ static void process_init_reply(struct fuse_args *args, int error)
ok = false;
}
if (flags & FUSE_OVER_IO_URING && fuse_uring_enabled())
- fuse_chan_io_uring_enable(fc->chan);
+ io_uring_enabled = true;
if (flags & FUSE_REQUEST_TIMEOUT)
timeout = arg->request_timeout;
@@ -1433,6 +1434,7 @@ static void process_init_reply(struct fuse_args *args, int error)
.minor = fc->minor,
.max_write = fc->max_write,
.max_pages = fc->max_pages,
+ .io_uring_enabled = io_uring_enabled,
};
fuse_chan_set_initialized(fc->chan, &cp);
}
--
2.52.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v7 2/6] fuse: add FUSE_IO_URING_CMD_ADD_QUEUE
2026-08-14 18:59 [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
2026-08-14 18:59 ` [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration Joanne Koong
@ 2026-08-14 18:59 ` Joanne Koong
2026-08-14 18:59 ` [PATCH v7 3/6] fuse: add io-uring buffer pools Joanne Koong
` (5 subsequent siblings)
7 siblings, 0 replies; 27+ messages in thread
From: Joanne Koong @ 2026-08-14 18:59 UTC (permalink / raw)
To: miklos; +Cc: jlayton, libaokun, axboe, bernd, amir73il, fuse-devel
fuse-over-io-uring queues are currently created lazily, as a side effect
of the first FUSE_IO_URING_CMD_REGISTER command for a given qid. This
ties queue creation to entry registration.
Add a FUSE_IO_URING_CMD_ADD_QUEUE command so a server can create a queue
explicitly, decoupling queue setup from entry registration. This is
additionally a prerequisite for FUSE_IO_URING_CMD_ADD_BUFPOOL, which
attaches a buffer pool to an existing queue and therefore needs the
queue to have been created first.
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
fs/fuse/dev_uring.c | 45 +++++++++++++++++++++++++++++++++------
include/uapi/linux/fuse.h | 8 ++++++-
2 files changed, 45 insertions(+), 8 deletions(-)
diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
index 9616778505ba..3b9fd0daef66 100644
--- a/fs/fuse/dev_uring.c
+++ b/fs/fuse/dev_uring.c
@@ -281,7 +281,8 @@ void fuse_uring_conn_init(struct fuse_chan *fch)
}
static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
- int qid)
+ int qid,
+ bool fail_if_exists)
{
struct fuse_chan *fch = ring->chan;
struct fuse_ring_queue *queue;
@@ -289,11 +290,11 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
queue = kzalloc_obj(*queue, GFP_KERNEL_ACCOUNT);
if (!queue)
- return NULL;
+ return ERR_PTR(-ENOMEM);
pq = fuse_pqueue_alloc();
if (!pq) {
kfree(queue);
- return NULL;
+ return ERR_PTR(-ENOMEM);
}
queue->qid = qid;
@@ -316,7 +317,7 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
spin_unlock(&fch->lock);
kfree(queue->fpq.processing);
kfree(queue);
- return ring->queues[qid];
+ return fail_if_exists ? ERR_PTR(-EEXIST) : ring->queues[qid];
}
/*
@@ -1189,9 +1190,9 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
queue = READ_ONCE(ring->queues[qid]);
if (!queue) {
- queue = fuse_uring_create_queue(ring, qid);
- if (!queue)
- return -ENOMEM;
+ queue = fuse_uring_create_queue(ring, qid, false);
+ if (IS_ERR(queue))
+ return PTR_ERR(queue);
}
/*
@@ -1206,6 +1207,30 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
return fuse_uring_do_register(ent, cmd, issue_flags);
}
+static int fuse_uring_add_queue(struct io_uring_cmd *cmd, struct fuse_chan *fch)
+{
+ const struct fuse_uring_cmd_req *cmd_req =
+ io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
+ struct fuse_ring *ring = smp_load_acquire(&fch->ring);
+ unsigned int qid = READ_ONCE(cmd_req->qid);
+ uint64_t flags = READ_ONCE(cmd_req->flags);
+ struct fuse_ring_queue *queue;
+
+ if (!ring || flags)
+ return -EINVAL;
+
+ if (qid >= ring->nr_queues) {
+ pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid);
+ return -EINVAL;
+ }
+
+ queue = fuse_uring_create_queue(ring, qid, true);
+ if (IS_ERR(queue))
+ return PTR_ERR(queue);
+
+ return 0;
+}
+
/*
* Entry function from io_uring to handle the given passthrough command
* (op code IORING_OP_URING_CMD)
@@ -1272,6 +1297,12 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)
return err;
}
break;
+ case FUSE_IO_URING_CMD_ADD_QUEUE:
+ err = fuse_uring_add_queue(cmd, fch);
+ if (err)
+ pr_info_once("FUSE_IO_URING_CMD_ADD_QUEUE failed err=%d\n",
+ err);
+ return err;
default:
return -EINVAL;
}
diff --git a/include/uapi/linux/fuse.h b/include/uapi/linux/fuse.h
index c13e1f9a2f12..cfb055c0c764 100644
--- a/include/uapi/linux/fuse.h
+++ b/include/uapi/linux/fuse.h
@@ -240,6 +240,9 @@
* - add FUSE_COPY_FILE_RANGE_64
* - add struct fuse_copy_file_range_out
* - add FUSE_NOTIFY_PRUNE
+ *
+ * 7.46
+ * - add FUSE_IO_URING_CMD_ADD_QUEUE
*/
#ifndef _LINUX_FUSE_H
@@ -275,7 +278,7 @@
#define FUSE_KERNEL_VERSION 7
/** Minor version number of this interface */
-#define FUSE_KERNEL_MINOR_VERSION 45
+#define FUSE_KERNEL_MINOR_VERSION 46
/** The node ID of the root inode */
#define FUSE_ROOT_ID 1
@@ -1292,6 +1295,9 @@ enum fuse_uring_cmd {
/* commit fuse request result and fetch next request */
FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2,
+
+ /* add a queue */
+ FUSE_IO_URING_CMD_ADD_QUEUE = 3,
};
/**
--
2.52.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v7 3/6] fuse: add io-uring buffer pools
2026-08-14 18:59 [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
2026-08-14 18:59 ` [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration Joanne Koong
2026-08-14 18:59 ` [PATCH v7 2/6] fuse: add FUSE_IO_URING_CMD_ADD_QUEUE Joanne Koong
@ 2026-08-14 18:59 ` Joanne Koong
2026-08-14 18:59 ` [PATCH v7 4/6] fuse: support registered buffer pools in io-uring Joanne Koong
` (4 subsequent siblings)
7 siblings, 0 replies; 27+ messages in thread
From: Joanne Koong @ 2026-08-14 18:59 UTC (permalink / raw)
To: miklos; +Cc: jlayton, libaokun, axboe, bernd, amir73il, fuse-devel
Right now, ents and buffers are tightly coupled in fuse io-uring where
each entry has its own dedicated payload buffer, requiring N buffers for
N entries where each buffer must be large enough to accomodate the
maximum payload size. This is suboptimal as most request types (lookup,
open, release, getattr, etc) require vastly less bytes than the maximum
payload size and some requests (unlink, rmdir, fsync, flush, etc) do not
require payload buffers at all.
Instead of requiring a 1:1 coupling between ents and payload buffers,
allow the server to pass in a buffer pool (a contiguous chunk of memory)
that the kernel will use as it wishes for servicing ents/requests.
Entries only reserve a "buffer" from the pool while actively processing
a request that requires a payload buffer. This decoupling and letting
the kernel delegate memory from the pool for requests allows the kernel to
optimize memory usage and reduces the memory usage requirements needed
to use fuse-over-io-uring.
A pool is registered per queue with the new
FUSE_IO_URING_CMD_ADD_BUFPOOL command. The server passes the pool's base
address and length in fuse_uring_cmd_req.bufpool.{uaddr,len}.
Internally, the kernel splits the region into buffers of
ring->max_payload_sz bytes each (nr_bufs = pool len / max_payload_sz). A
queue commits to a payload mode on first use: registering an entry that
carries its own payload selects the legacy per-entry mode, while
ADD_BUFPOOL selects pool mode. The two are mutually exclusive, so
ADD_BUFPOOL must be issued before any payload-carrying entries are
registered on that queue. The queue must have been created before the
bufpool is added, through the FUSE_IO_URING_CMD_ADD_QUEUE command.
The kernel tracks free buffers with a bitmap (a set bit marks a free
buffer). On dispatch, a request that needs a payload claims a free
buffer (find_first_bit + clear). A request that needs none claims
nothing. The buffer's byte offset within the pool is reported to the
server in the new fuse_uring_ent_in_out.offset field so that the server
can locate the payload. On completion the buffer is returned to the pool
or reused directly if the next request on that entry also has a payload.
The FUSE_HAS_IO_URING_BUFPOOL flag advertises kernel support to the
server for bufpools.
Buffer pool request flow
~~~~~~~~~~~~~~~~~~~~~~~~
| Kernel | FUSE daemon
| |
| [request arrives] |
| [claim a free pool buffer] |
| >fuse_uring_select_buffer() |
| [copy headers to ring] |
| [copy payload to buffer] |
| [report buffer offset in ent_in_out] |
| >io_uring_cmd_done() |
| | [read headers]
| | [read/write payload at offset]
| | [process request]
| | >io_uring_submit()
| | COMMIT_AND_FETCH
| >fuse_uring_commit_fetch() |
| [copy reply from ring] |
| [return buffer to the pool] |
| >fuse_uring_recycle_buffer() |
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
fs/fuse/dev_uring.c | 239 +++++++++++++++++++++++++++++++++-----
fs/fuse/dev_uring_i.h | 37 +++++-
fs/fuse/inode.c | 2 +-
include/uapi/linux/fuse.h | 21 +++-
4 files changed, 269 insertions(+), 30 deletions(-)
diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
index 3b9fd0daef66..d300c7f441c4 100644
--- a/fs/fuse/dev_uring.c
+++ b/fs/fuse/dev_uring.c
@@ -9,6 +9,7 @@
#include "dev_uring_i.h"
#include "fuse_trace.h"
+#include <linux/bitmap.h>
#include <linux/fs.h>
#include <linux/io_uring/cmd.h>
@@ -41,6 +42,11 @@ enum fuse_uring_header_type {
FUSE_URING_HEADER_RING_ENT,
};
+static inline bool bufpool_enabled(struct fuse_ring_queue *queue)
+{
+ return queue->payload_mode == FUSE_PAYLOAD_BUFPOOL;
+}
+
static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd,
struct fuse_ring_ent *ring_ent)
{
@@ -222,6 +228,7 @@ void fuse_uring_destruct(struct fuse_chan *fch)
}
kfree(queue->fpq.processing);
+ kfree(queue->bufpool);
kfree(queue);
WRITE_ONCE(ring->queues[qid], NULL);
}
@@ -316,6 +323,7 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
if (ring->queues[qid]) {
spin_unlock(&fch->lock);
kfree(queue->fpq.processing);
+ kfree(queue->bufpool);
kfree(queue);
return fail_if_exists ? ERR_PTR(-EEXIST) : ring->queues[qid];
}
@@ -646,13 +654,14 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,
}
static int setup_fuse_copy_state(struct fuse_copy_state *cs,
- struct fuse_ring *ring, struct fuse_req *req,
+ struct fuse_req *req,
struct fuse_ring_ent *ent, int dir,
struct iov_iter *iter)
{
int err;
- err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter);
+ err = import_ubuf(dir, ent->payload.iov_base, ent->payload.iov_len,
+ iter);
if (err) {
pr_info_ratelimited("fuse: Import of user buffer failed\n");
return err;
@@ -666,8 +675,7 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs,
return 0;
}
-static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
- struct fuse_req *req,
+static int fuse_uring_copy_from_ring(struct fuse_req *req,
struct fuse_ring_ent *ent)
{
struct fuse_copy_state cs;
@@ -681,7 +689,7 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
if (err)
return err;
- err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_SOURCE, &iter);
+ err = setup_fuse_copy_state(&cs, req, ent, ITER_SOURCE, &iter);
if (err)
return err;
@@ -693,7 +701,7 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
/*
* Copy data from the req to the ring buffer
*/
-static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
+static int fuse_uring_args_to_ring(struct fuse_req *req,
struct fuse_ring_ent *ent)
{
struct fuse_copy_state cs;
@@ -707,7 +715,7 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
.commit_id = req->in.h.unique,
};
- err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_DEST, &iter);
+ err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter);
if (err)
return err;
@@ -737,6 +745,10 @@ static int fuse_uring_args_to_ring(struct fuse_ring *ring, struct fuse_req *req,
}
ent_in_out.payload_sz = cs.ring.copied_sz;
+ if (bufpool_enabled(ent->queue) && ent->payload.iov_base)
+ ent_in_out.offset =
+ (uintptr_t)ent->payload.iov_base - ent->queue->bufpool->base_uaddr;
+
return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,
&ent_in_out, sizeof(ent_in_out));
}
@@ -745,7 +757,6 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
struct fuse_req *req)
{
struct fuse_ring_queue *queue = ent->queue;
- struct fuse_ring *ring = queue->ring;
int err;
err = -EIO;
@@ -760,7 +771,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
return err;
/* copy the request */
- err = fuse_uring_args_to_ring(ring, req, ent);
+ err = fuse_uring_args_to_ring(req, ent);
if (unlikely(err)) {
pr_info_ratelimited("Copy to ring failed: %d\n", err);
return err;
@@ -771,6 +782,91 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
sizeof(req->in.h));
}
+static bool fuse_uring_req_has_payload(struct fuse_req *req)
+{
+ struct fuse_args *args = req->args;
+
+ return args->in_numargs > 1 || args->out_numargs;
+}
+
+static int fuse_uring_select_buffer(struct fuse_ring_ent *ent)
+{
+ struct fuse_ring_queue *queue = ent->queue;
+ struct fuse_bufpool *pool = queue->bufpool;
+ unsigned int id;
+
+ lockdep_assert_held(&queue->lock);
+
+ id = find_first_bit(pool->free_map, pool->nr_bufs);
+ if (id >= pool->nr_bufs)
+ return -ENOBUFS;
+
+ WARN_ON_ONCE(ent->payload.iov_base);
+ __clear_bit(id, pool->free_map);
+
+ ent->buf_id = id;
+ ent->payload.iov_base =
+ (void __user *)(pool->base_uaddr + id * pool->buf_size);
+ ent->payload.iov_len = pool->buf_size;
+
+ return 0;
+}
+
+static void fuse_uring_recycle_buffer(struct fuse_ring_ent *ent)
+{
+ struct iovec *ent_payload = &ent->payload;
+ struct fuse_ring_queue *queue = ent->queue;
+ struct fuse_bufpool *pool;
+
+ lockdep_assert_held(&queue->lock);
+
+ if (!bufpool_enabled(queue) || !ent_payload->iov_base)
+ return;
+
+ pool = queue->bufpool;
+
+ /* a buffer should never be recycled twice */
+ WARN_ON_ONCE(test_bit(ent->buf_id, pool->free_map));
+ __set_bit(ent->buf_id, pool->free_map);
+
+ memset(ent_payload, 0, sizeof(*ent_payload));
+ ent->buf_id = 0;
+}
+
+static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent,
+ struct fuse_req *req)
+{
+ bool buffer_selected;
+ bool has_payload;
+
+ if (!bufpool_enabled(ent->queue))
+ return 0;
+
+ buffer_selected = !!ent->payload.iov_base;
+ has_payload = fuse_uring_req_has_payload(req);
+
+ if (has_payload && !buffer_selected)
+ return fuse_uring_select_buffer(ent);
+
+ if (!has_payload && buffer_selected)
+ fuse_uring_recycle_buffer(ent);
+
+ return 0;
+}
+
+static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
+ struct fuse_req *req)
+{
+ if (!bufpool_enabled(ent->queue))
+ return 0;
+
+ /* no payload to copy, can skip selecting a buffer */
+ if (!fuse_uring_req_has_payload(req))
+ return 0;
+
+ return fuse_uring_select_buffer(ent);
+}
+
static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
struct fuse_req *req)
{
@@ -858,9 +954,12 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent)
/* get and assign the next entry while it is still holding the lock */
req = list_first_entry_or_null(req_queue, struct fuse_req, list);
- if (req)
- fuse_uring_add_req_to_ring_ent(ent, req);
+ if (!req || fuse_uring_next_req_update_buffer(ent, req)) {
+ fuse_uring_recycle_buffer(ent);
+ return NULL;
+ }
+ fuse_uring_add_req_to_ring_ent(ent, req);
return req;
}
@@ -872,7 +971,6 @@ static struct fuse_req *fuse_uring_ent_assign_req(struct fuse_ring_ent *ent)
static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
unsigned int issue_flags)
{
- struct fuse_ring *ring = ent->queue->ring;
ssize_t err = -EFAULT;
if (copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->out.h,
@@ -885,7 +983,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
goto out;
}
- err = fuse_uring_copy_from_ring(ring, req, ent);
+ err = fuse_uring_copy_from_ring(req, ent);
out:
fuse_uring_req_end(ent, req, err);
}
@@ -1004,6 +1102,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
if (err != 0) {
pr_info_ratelimited("qid=%d commit_id %llu state %d",
queue->qid, commit_id, ent->state);
+ fuse_uring_recycle_buffer(ent);
spin_unlock(&queue->lock);
fuse_uring_req_end(ent, req, err);
return err;
@@ -1021,6 +1120,11 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
* fuse requests would otherwise not get processed - committing
* and fetching is done in one step vs legacy fuse, which has separated
* read (fetch request) and write (commit result).
+ *
+ * If there is no next request or if all buffers are busy (if using a
+ * bufpool), the cmd is not returned to userspace. The entry is left
+ * available and the cmd only returns to userspace when there's a
+ * next request and an available buffer.
*/
if (fuse_uring_get_next_fuse_req(ent, queue))
fuse_uring_send(ent, cmd, 0, issue_flags);
@@ -1145,11 +1249,23 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
}
payload = &iov[FUSE_URING_IOV_PAYLOAD];
- if (payload->iov_len < ring->max_payload_sz) {
- pr_info_ratelimited("Invalid req payload len %zu\n",
- payload->iov_len);
- return ERR_PTR(err);
+
+ spin_lock(&queue->lock);
+ if (bufpool_enabled(queue)) {
+ if (payload->iov_base || payload->iov_len) {
+ spin_unlock(&queue->lock);
+ return ERR_PTR(err);
+ }
+ } else {
+ if (payload->iov_len < ring->max_payload_sz) {
+ pr_info_ratelimited("Invalid req payload len %zu\n",
+ payload->iov_len);
+ spin_unlock(&queue->lock);
+ return ERR_PTR(err);
+ }
+ queue->payload_mode = FUSE_PAYLOAD_PER_ENT;
}
+ spin_unlock(&queue->lock);
err = -ENOMEM;
ent = kzalloc_obj(*ent, GFP_KERNEL_ACCOUNT);
@@ -1160,7 +1276,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
ent->queue = queue;
ent->headers = headers->iov_base;
- ent->payload = payload->iov_base;
+ if (queue->payload_mode == FUSE_PAYLOAD_PER_ENT)
+ ent->payload = *payload;
atomic_inc(&ring->queue_refs);
return ent;
@@ -1231,6 +1348,67 @@ static int fuse_uring_add_queue(struct io_uring_cmd *cmd, struct fuse_chan *fch)
return 0;
}
+static int fuse_uring_add_bufpool(struct io_uring_cmd *cmd,
+ struct fuse_chan *fch)
+{
+ const struct fuse_uring_cmd_req *cmd_req =
+ io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
+ unsigned int qid = READ_ONCE(cmd_req->qid);
+ uint64_t flags = READ_ONCE(cmd_req->flags);
+ /* paired with the smp_store_release() in fuse_uring_create */
+ struct fuse_ring *ring = smp_load_acquire(&fch->ring);
+ struct fuse_ring_queue *queue;
+ struct fuse_bufpool *pool;
+ uintptr_t pool_uaddr;
+ unsigned int pool_len, nr_bufs;
+ size_t pool_size, buf_size;
+
+ if (!ring || qid >= ring->nr_queues || flags)
+ return -EINVAL;
+
+ /* reserved for future use, must be zero */
+ if (READ_ONCE(cmd_req->bufpool.reserved))
+ return -EINVAL;
+
+ /* Pairs with smp_store_release() in fuse_uring_create_queue() */
+ queue = smp_load_acquire(&ring->queues[qid]);
+ if (!queue)
+ return -EINVAL;
+
+ pool_uaddr = READ_ONCE(cmd_req->bufpool.uaddr);
+ pool_len = READ_ONCE(cmd_req->bufpool.len);
+
+ /* each buffer holds the max payload size */
+ buf_size = queue->ring->max_payload_sz;
+
+ nr_bufs = pool_len / buf_size;
+ if (!nr_bufs)
+ return -EINVAL;
+
+ pool_size = struct_size(pool, free_map, BITS_TO_LONGS(nr_bufs));
+ pool = kzalloc(pool_size, GFP_KERNEL_ACCOUNT);
+ if (!pool)
+ return -ENOMEM;
+
+ pool->base_uaddr = pool_uaddr;
+ pool->buf_size = buf_size;
+ pool->nr_bufs = nr_bufs;
+ /* all buffers are free */
+ bitmap_set(pool->free_map, 0, nr_bufs);
+
+ spin_lock(&queue->lock);
+ if (queue->payload_mode != FUSE_PAYLOAD_UNSET) {
+ spin_unlock(&queue->lock);
+ kfree(pool);
+ return -EINVAL;
+ }
+ queue->bufpool = pool;
+ queue->payload_mode = FUSE_PAYLOAD_BUFPOOL;
+ spin_unlock(&queue->lock);
+
+ return 0;
+}
+
/*
* Entry function from io_uring to handle the given passthrough command
* (op code IORING_OP_URING_CMD)
@@ -1303,6 +1481,12 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)
pr_info_once("FUSE_IO_URING_CMD_ADD_QUEUE failed err=%d\n",
err);
return err;
+ case FUSE_IO_URING_CMD_ADD_BUFPOOL:
+ err = fuse_uring_add_bufpool(cmd, fch);
+ if (err)
+ pr_info_once("FUSE_IO_URING_ADD_BUFPOOL failed err=%d\n",
+ err);
+ return err;
default:
return -EINVAL;
}
@@ -1336,6 +1520,7 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)
spin_lock(&queue->lock);
list_del_init(&ent->list);
+ fuse_uring_recycle_buffer(ent);
spin_unlock(&queue->lock);
io_uring_cmd_done(cmd, err, issue_flags);
@@ -1397,15 +1582,16 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue *fiq, struct fuse_req *req)
req->ring_queue = queue;
ent = list_first_entry_or_null(&queue->ent_avail_queue,
struct fuse_ring_ent, list);
- if (ent)
- fuse_uring_add_req_to_ring_ent(ent, req);
- else
- list_add_tail(&req->list, &queue->fuse_req_queue);
- spin_unlock(&queue->lock);
- if (ent)
- fuse_uring_dispatch_ent(ent);
+ if (!ent || fuse_uring_prep_buffer(ent, req)) {
+ list_add_tail(&req->list, &queue->fuse_req_queue);
+ spin_unlock(&queue->lock);
+ return;
+ }
+ fuse_uring_add_req_to_ring_ent(ent, req);
+ spin_unlock(&queue->lock);
+ fuse_uring_dispatch_ent(ent);
return;
err_unlock:
@@ -1453,10 +1639,9 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req)
*/
req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req,
list);
- if (ent && req) {
+ if (ent && req && !fuse_uring_prep_buffer(ent, req)) {
fuse_uring_add_req_to_ring_ent(ent, req);
spin_unlock(&queue->lock);
-
fuse_uring_dispatch_ent(ent);
} else {
spin_unlock(&queue->lock);
diff --git a/fs/fuse/dev_uring_i.h b/fs/fuse/dev_uring_i.h
index d721a4fc0215..cdf56f8b38b5 100644
--- a/fs/fuse/dev_uring_i.h
+++ b/fs/fuse/dev_uring_i.h
@@ -7,6 +7,8 @@
#ifndef _FS_FUSE_DEV_URING_I_H
#define _FS_FUSE_DEV_URING_I_H
+#include <linux/uio.h>
+
#include "fuse_dev_i.h"
#ifdef CONFIG_FUSE_IO_URING
@@ -36,11 +38,38 @@ enum fuse_ring_req_state {
FRRS_RELEASED,
};
+/* how a queue's payload buffers are provided */
+enum fuse_queue_payload_mode {
+ /* not yet committed (a bufpool may still be added) */
+ FUSE_PAYLOAD_UNSET = 0,
+ /* each entry registers its own payload buffer */
+ FUSE_PAYLOAD_PER_ENT,
+ /* each entry's payload buffer is assigned from a bufpool */
+ FUSE_PAYLOAD_BUFPOOL,
+};
+
+struct fuse_bufpool {
+ /* starting uaddr of the bufpool */
+ uintptr_t base_uaddr;
+
+ /* size of each buffer in the pool */
+ size_t buf_size;
+
+ /* total number of buffers in the pool */
+ unsigned int nr_bufs;
+
+ /* bitmap tracking which buffers are free */
+ unsigned long free_map[];
+};
+
/** A fuse ring entry, part of the ring queue */
struct fuse_ring_ent {
/* userspace buffer */
struct fuse_uring_req_header __user *headers;
- void __user *payload;
+ struct iovec payload;
+
+ /* buffer id in the pool, if bufpools are used. ignored otherwise */
+ unsigned int buf_id;
/* the ring queue that owns the request */
struct fuse_ring_queue *queue;
@@ -99,6 +128,12 @@ struct fuse_ring_queue {
unsigned int active_background;
bool stopped;
+
+ /* how this queue's payload buffers are provided */
+ enum fuse_queue_payload_mode payload_mode;
+
+ /* only allocated when payload_mode == FUSE_PAYLOAD_BUFPOOL */
+ struct fuse_bufpool *bufpool;
};
/*
diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
index 33773c7d129a..9779adc98593 100644
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -1482,7 +1482,7 @@ static struct fuse_init_args *fuse_new_init(struct fuse_mount *fm)
* the reply - server is either sending IORING_OP_URING_CMD or not.
*/
if (fuse_uring_enabled())
- flags |= FUSE_OVER_IO_URING;
+ flags |= FUSE_OVER_IO_URING | FUSE_HAS_IO_URING_BUFPOOL;
ia->in.flags = flags;
ia->in.flags2 = flags >> 32;
diff --git a/include/uapi/linux/fuse.h b/include/uapi/linux/fuse.h
index cfb055c0c764..538d844da099 100644
--- a/include/uapi/linux/fuse.h
+++ b/include/uapi/linux/fuse.h
@@ -243,6 +243,9 @@
*
* 7.46
* - add FUSE_IO_URING_CMD_ADD_QUEUE
+ * - add FUSE_HAS_IO_URING_BUFPOOL
+ * - add fuse_uring_cmd_req bufpool struct
+ * - add bufpool offset field to fuse_uring_ent_in_out struct
*/
#ifndef _LINUX_FUSE_H
@@ -451,6 +454,7 @@ struct fuse_file_lock {
* FUSE_OVER_IO_URING: Indicate that client supports io-uring
* FUSE_REQUEST_TIMEOUT: kernel supports timing out requests.
* init_out.request_timeout contains the timeout (in secs)
+ * FUSE_HAS_IO_URING_BUFPOOL: kernel supports io-uring buffer pools
*/
#define FUSE_ASYNC_READ (1 << 0)
#define FUSE_POSIX_LOCKS (1 << 1)
@@ -498,6 +502,7 @@ struct fuse_file_lock {
#define FUSE_ALLOW_IDMAP (1ULL << 40)
#define FUSE_OVER_IO_URING (1ULL << 41)
#define FUSE_REQUEST_TIMEOUT (1ULL << 42)
+#define FUSE_HAS_IO_URING_BUFPOOL (1ULL << 43)
/**
* CUSE INIT request/reply flags
@@ -1266,7 +1271,9 @@ struct fuse_uring_ent_in_out {
/* size of user payload buffer */
uint32_t payload_sz;
- uint32_t padding;
+
+ /* Offset into the bufpool, if bufpools are used */
+ uint32_t offset;
uint64_t reserved;
};
@@ -1298,6 +1305,9 @@ enum fuse_uring_cmd {
/* add a queue */
FUSE_IO_URING_CMD_ADD_QUEUE = 3,
+
+ /* add a bufpool to a queue */
+ FUSE_IO_URING_CMD_ADD_BUFPOOL = 4,
};
/**
@@ -1312,6 +1322,15 @@ struct fuse_uring_cmd_req {
/* queue the command is for (queue index) */
uint16_t qid;
uint8_t padding[6];
+
+ union {
+ struct {
+ /* base address of bufpool */
+ uint64_t uaddr;
+ uint32_t len;
+ uint32_t reserved;
+ } bufpool;
+ };
};
#endif /* _LINUX_FUSE_H */
--
2.52.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v7 4/6] fuse: support registered buffer pools in io-uring
2026-08-14 18:59 [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
` (2 preceding siblings ...)
2026-08-14 18:59 ` [PATCH v7 3/6] fuse: add io-uring buffer pools Joanne Koong
@ 2026-08-14 18:59 ` Joanne Koong
2026-08-17 10:15 ` Bernd Schubert
2026-08-14 18:59 ` [PATCH v7 5/6] fuse: add zero-copy over io-uring Joanne Koong
` (3 subsequent siblings)
7 siblings, 1 reply; 27+ messages in thread
From: Joanne Koong @ 2026-08-14 18:59 UTC (permalink / raw)
To: miklos; +Cc: jlayton, libaokun, axboe, bernd, amir73il, fuse-devel
Allow servers to use a buffer pool that is also registered through
io-uring. When the server registers a buffer pool with io-uring, the
pages backing the pool are pinned upfront. This eliminates the overhead
of pinning/unpinning user pages and translating virtual addresses per
i/o request. This also allows servers to use the same registered memory
for subsequent backing store I/O (eg read_fixed/write_fixed), keeping
data in the same pinned pages without additional pinning or mapping
overhead required.
To use this, the server needs to set the FUSE_URING_REGISTERED_BUFPOOL
flag when adding a bufpool through the FUSE_IO_URING_CMD_ADD_BUFPOOL
cmd. For every sqe submitted (including the one for adding the bufpool),
it should set sqe->uring_cmd_flags to include IORING_URING_CMD_FIXED,
and pass in the index where the registered bufpool resides to
sqe->buf_index.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered buffers
direct read ~5.1 GB/s ~5.4 GB/s (+~5%)
direct write ~3.4 GB/s ~4.8 GB/s (+~45%)
Registered buffers bring up the write path speed up closer to speed of
reads. There isn't much improvement for reads because it is already fast
enough where it's at the copy-bound ceiling (surpassing that requires
doing zero-copy). On a device-bound NVMe though, the differences are
within noise, as backing I/O dominates per-request latency.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
fs/fuse/dev_uring.c | 112 ++++++++++++++++++++++++++++++++++--------
fs/fuse/dev_uring_i.h | 8 +++
2 files changed, 99 insertions(+), 21 deletions(-)
diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
index d300c7f441c4..17806da93039 100644
--- a/fs/fuse/dev_uring.c
+++ b/fs/fuse/dev_uring.c
@@ -47,6 +47,27 @@ static inline bool bufpool_enabled(struct fuse_ring_queue *queue)
return queue->payload_mode == FUSE_PAYLOAD_BUFPOOL;
}
+static inline bool bufpool_registered(struct fuse_ring_queue *queue)
+{
+ return queue->bufpool && queue->bufpool->registered;
+}
+
+/*
+ * For a registered bufpool, every sqe that drives a payload import (REGISTER,
+ * COMMIT_AND_FETCH) must carry the registered buffer index of the pool.
+ * This also must be called from the command's issue handler, where cmd->sqe is
+ * still valid
+ */
+static inline bool fuse_uring_cmd_index_ok(struct io_uring_cmd *cmd,
+ struct fuse_ring_queue *queue)
+{
+ if (!bufpool_registered(queue))
+ return true;
+
+ return (cmd->flags & IORING_URING_CMD_FIXED) &&
+ READ_ONCE(cmd->sqe->buf_index) == queue->bufpool->registered_index;
+}
+
static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd,
struct fuse_ring_ent *ring_ent)
{
@@ -653,19 +674,42 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,
return 0;
}
+static int fuse_uring_import_payload(struct fuse_ring_ent *ent, int dir,
+ struct iov_iter *iter,
+ unsigned int issue_flags)
+{
+ void __user *base = ent->payload.iov_base;
+ size_t len = ent->payload.iov_len;
+ int err = 0;
+
+ if (!base) {
+ memset(iter, 0, sizeof(*iter));
+ return 0;
+ }
+
+ if (bufpool_registered(ent->queue))
+ err = io_uring_cmd_import_fixed((u64)(uintptr_t)base, len, dir,
+ iter, ent->cmd, issue_flags);
+ else
+ err = import_ubuf(dir, base, len, iter);
+
+ if (err)
+ pr_info_ratelimited("fuse: Import of user buffer failed\n");
+
+ return err;
+}
+
static int setup_fuse_copy_state(struct fuse_copy_state *cs,
struct fuse_req *req,
struct fuse_ring_ent *ent, int dir,
- struct iov_iter *iter)
+ struct iov_iter *iter,
+ unsigned int issue_flags)
{
int err;
- err = import_ubuf(dir, ent->payload.iov_base, ent->payload.iov_len,
- iter);
- if (err) {
- pr_info_ratelimited("fuse: Import of user buffer failed\n");
+ err = fuse_uring_import_payload(ent, dir, iter, issue_flags);
+ if (err)
return err;
- }
fuse_copy_init(cs, dir == ITER_DEST, iter);
@@ -676,7 +720,8 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs,
}
static int fuse_uring_copy_from_ring(struct fuse_req *req,
- struct fuse_ring_ent *ent)
+ struct fuse_ring_ent *ent,
+ unsigned int issue_flags)
{
struct fuse_copy_state cs;
struct fuse_args *args = req->args;
@@ -689,7 +734,8 @@ static int fuse_uring_copy_from_ring(struct fuse_req *req,
if (err)
return err;
- err = setup_fuse_copy_state(&cs, req, ent, ITER_SOURCE, &iter);
+ err = setup_fuse_copy_state(&cs, req, ent, ITER_SOURCE, &iter,
+ issue_flags);
if (err)
return err;
@@ -702,7 +748,8 @@ static int fuse_uring_copy_from_ring(struct fuse_req *req,
* Copy data from the req to the ring buffer
*/
static int fuse_uring_args_to_ring(struct fuse_req *req,
- struct fuse_ring_ent *ent)
+ struct fuse_ring_ent *ent,
+ unsigned int issue_flags)
{
struct fuse_copy_state cs;
struct fuse_args *args = req->args;
@@ -715,7 +762,8 @@ static int fuse_uring_args_to_ring(struct fuse_req *req,
.commit_id = req->in.h.unique,
};
- err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter);
+ err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter,
+ issue_flags);
if (err)
return err;
@@ -754,7 +802,8 @@ static int fuse_uring_args_to_ring(struct fuse_req *req,
}
static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
- struct fuse_req *req)
+ struct fuse_req *req,
+ unsigned int issue_flags)
{
struct fuse_ring_queue *queue = ent->queue;
int err;
@@ -771,7 +820,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
return err;
/* copy the request */
- err = fuse_uring_args_to_ring(req, ent);
+ err = fuse_uring_args_to_ring(req, ent, issue_flags);
if (unlikely(err)) {
pr_info_ratelimited("Copy to ring failed: %d\n", err);
return err;
@@ -868,11 +917,12 @@ static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
}
static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
- struct fuse_req *req)
+ struct fuse_req *req,
+ unsigned int issue_flags)
{
int err;
- err = fuse_uring_copy_to_ring(ent, req);
+ err = fuse_uring_copy_to_ring(ent, req, issue_flags);
if (!err) {
set_bit(FR_SENT, &req->flags);
trace_fuse_request_sent(req);
@@ -983,7 +1033,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
goto out;
}
- err = fuse_uring_copy_from_ring(req, ent);
+ err = fuse_uring_copy_from_ring(req, ent, issue_flags);
out:
fuse_uring_req_end(ent, req, err);
}
@@ -995,7 +1045,8 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
* Else, there is no next fuse request and this returns false.
*/
static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
- struct fuse_ring_queue *queue)
+ struct fuse_ring_queue *queue,
+ unsigned int issue_flags)
{
int err;
struct fuse_req *req;
@@ -1007,7 +1058,7 @@ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
spin_unlock(&queue->lock);
if (req) {
- err = fuse_uring_prepare_send(ent, req);
+ err = fuse_uring_prepare_send(ent, req, issue_flags);
if (err)
goto retry;
}
@@ -1081,6 +1132,11 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
return err;
}
+ if (!fuse_uring_cmd_index_ok(cmd, queue)) {
+ spin_unlock(&queue->lock);
+ return -EINVAL;
+ }
+
/* Find a request based on the unique ID of the fuse request
* This should get revised, as it needs a hash calculation and list
* search. And full struct fuse_pqueue is needed (memory overhead).
@@ -1126,7 +1182,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
* available and the cmd only returns to userspace when there's a
* next request and an available buffer.
*/
- if (fuse_uring_get_next_fuse_req(ent, queue))
+ if (fuse_uring_get_next_fuse_req(ent, queue, issue_flags))
fuse_uring_send(ent, cmd, 0, issue_flags);
return 0;
}
@@ -1252,7 +1308,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
spin_lock(&queue->lock);
if (bufpool_enabled(queue)) {
- if (payload->iov_base || payload->iov_len) {
+ if (payload->iov_base || payload->iov_len ||
+ !fuse_uring_cmd_index_ok(cmd, queue)) {
spin_unlock(&queue->lock);
return ERR_PTR(err);
}
@@ -1362,6 +1419,7 @@ static int fuse_uring_add_bufpool(struct io_uring_cmd *cmd,
uintptr_t pool_uaddr;
unsigned int pool_len, nr_bufs;
size_t pool_size, buf_size;
+ bool registered = cmd->flags & IORING_URING_CMD_FIXED;
if (!ring || qid >= ring->nr_queues || flags)
return -EINVAL;
@@ -1396,6 +1454,17 @@ static int fuse_uring_add_bufpool(struct io_uring_cmd *cmd,
/* all buffers are free */
bitmap_set(pool->free_map, 0, nr_bufs);
+ /*
+ * A registered bufpool is reached through an io_uring fixed buffer, so
+ * the pool is registered iff this command was submitted with
+ * IORING_URING_CMD_FIXED. The registered buffer index is taken from
+ * sqe->buf_index.
+ */
+ if (registered) {
+ pool->registered = true;
+ pool->registered_index = READ_ONCE(cmd->sqe->buf_index);
+ }
+
spin_lock(&queue->lock);
if (queue->payload_mode != FUSE_PAYLOAD_UNSET) {
spin_unlock(&queue->lock);
@@ -1508,9 +1577,10 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)
int err;
if (!tw.cancel) {
- err = fuse_uring_prepare_send(ent, ent->fuse_req);
+ err = fuse_uring_prepare_send(ent, ent->fuse_req, issue_flags);
if (err) {
- if (!fuse_uring_get_next_fuse_req(ent, queue))
+ if (!fuse_uring_get_next_fuse_req(ent, queue,
+ issue_flags))
return;
err = 0;
}
diff --git a/fs/fuse/dev_uring_i.h b/fs/fuse/dev_uring_i.h
index cdf56f8b38b5..e142cae43022 100644
--- a/fs/fuse/dev_uring_i.h
+++ b/fs/fuse/dev_uring_i.h
@@ -49,6 +49,14 @@ enum fuse_queue_payload_mode {
};
struct fuse_bufpool {
+ bool registered;
+
+ /*
+ * io_uring registered buffer table index for this pool, bound at
+ * ADD_BUFPOOL time. Only valid if the bufpool is registered
+ */
+ u16 registered_index;
+
/* starting uaddr of the bufpool */
uintptr_t base_uaddr;
--
2.52.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v7 5/6] fuse: add zero-copy over io-uring
2026-08-14 18:59 [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
` (3 preceding siblings ...)
2026-08-14 18:59 ` [PATCH v7 4/6] fuse: support registered buffer pools in io-uring Joanne Koong
@ 2026-08-14 18:59 ` Joanne Koong
2026-08-17 13:05 ` Bernd Schubert
2026-08-14 18:59 ` [PATCH v7 6/6] docs: fuse: document io-uring buffer pool and zero-copy uapi Joanne Koong
` (2 subsequent siblings)
7 siblings, 1 reply; 27+ messages in thread
From: Joanne Koong @ 2026-08-14 18:59 UTC (permalink / raw)
To: miklos; +Cc: jlayton, libaokun, axboe, bernd, amir73il, fuse-devel
Implement zero-copy in fuse io-uring to eliminate memory copies between
the application, kernel, and server for read/write operations. The
server can directly access client pages or page cache folios without
copying data through an intermediary buffer. When a fuse request arrives,
the kernel registers the relevant pages into a sparse slot in the
server's io_uring registered buffer table. The server can then operate
on these pages directly using io-uring fixed buffer operations (eg
read_fixed/write_fixed) and the kernel unregisters these pages when the
request completes. Non-page-backed args (eg op out headers) will go
through the payload buffer as normal. The server can specify which open
files should have their reads/writes go through zero-copy, by setting
the FOPEN_IO_URING_ZERO_COPY flag when servicing opens.
This requires CAP_SYS_ADMIN and bufpools. This is gated behind
CAP_SYS_ADMIN because zero-copy allows the server direct access to the
client's underlying pages, rather than operating on an intermediary
buffer that the contents of the client's pages were copied into or on
page cache folios.
The request flow for the zero-copy direct-io write path (client writes
data, server reads it) is as follows:
=======================================================================
| Kernel | FUSE server
| |
| "write(fd, buf, 1MB)" |
| |
| >sys_write() |
| >fuse_file_write_iter() |
| >fuse_send_one() |
| [req->args->in_pages = true] |
| [folios hold client write data] |
| |
| >fuse_uring_copy_to_ring() |
| >copy_header_to_ring(IN_OUT) |
| [memcpy fuse_in_header] |
| >copy_header_to_ring(OP) |
| [memcpy write_in header] |
| |
| >fuse_uring_args_to_ring() |
| >setup_fuse_copy_state() |
| [skip_folio_copy = true] |
| |
| >fuse_uring_set_up_zero_copy() |
| [folio_get for each client folio] |
| [build bio_vec array from folios] |
| >io_buffer_register_bvec() |
| [register pages at
ent->zero_copy_index] |
| [ent->zero_copied = true] |
| |
| >fuse_copy_args() |
| [skip_folio_copy => return 0 |
| for page arg, skip data copy] |
| |
| >copy_header_to_ring(RING_ENT) |
| [memcpy ent_in_out] |
| >io_uring_cmd_done() |
| |
| | [CQE received]
| |
| | [issue io_uring READ at
| | ent->zero_copy_index]
| | [reads directly from
| |client's pages (ZERO_COPY)]
| |
| | [write data to backing
| | store]
| | [submit COMMIT AND FETCH]
| |
| >fuse_uring_commit_fetch() |
| >fuse_uring_commit() |
| >fuse_uring_copy_from_ring() |
| >fuse_uring_req_end() |
| >io_buffer_unregister(ent->zero_copy_index) |
| [unregister pages from index] |
| >fuse_zero_copy_release() |
| [folio_put for each folio] |
| [ent->zero_copied = false] |
| >fuse_request_end() |
| [wake up client] |
The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args.
For these requests, the page-backed args are zero-copied while the
non-page-backed args are copied to the buffer selected from the buffer
pool:
zero-copy: pages registered via io_buffer_register_bvec()
non-page-backed: copied to payload buffer via fuse_copy_args()
For a request whose payload is zero-copied, the
registration/unregistration path looks like:
register: fuse_uring_set_up_zero_copy()
folio_get() for each folio
io_buffer_register_bvec(ent->zero_copy_index)
unregister: fuse_uring_req_end()
io_buffer_unregister(ent->zero_copy_index)
-> fuse_zero_copy_release() callback
folio_put() for each folio
Please note that on abort for in-flight zero-copied requests that have
been sent to userspace, the registered bvec slot remains occupied and
its folios remain pinned until the io-uring ring is destroyed, at which
point io-uring unregisters all buffers and the fuse_zero_copy_release()
callback drops the folio references. Unregistering at teardown would
require operating on the ring context directly, whose validity is hard
to ascertain; this is deemed not worth the complexity for the abort
race, since everything is freed when the ring is torn down.
The throughput improvement from zero-copy depends on how much of the
per-request latency is spent on data copying vs backing I/O. The gain
comes from eliminating the payload-buffer memcpy, but accessing the
zero-copied pages requires the server to issue the read/write as an
IORING_OP_READ/WRITE_FIXED operation. The benefit is largest when the
mempcy is a meaningful fraction of per-request latency while backing i/o
is still noticable enough that the extra io-uring op's overhead doesn't
dominate.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered-buf zero-copy (zc vs base)
direct read ~5.1 GB/s ~5.4 GB/s ~8.9 GB/s (+75%)
direct write ~3.4 GB/s ~4.8 GB/s ~5.1 GB/s (+50%)
Reads end up higher than writes because the backing store reads faster
than it writes (the baseline shows the same read>write gap, and the raw
device does too). On a device-bound NVMe (~2 GB/s reads) the read gain
shrinks to ~10-16% (and no measurable gains for writes), as backing I/O
rather than the eliminated copy dominates latency.
The benefit overall scales with how much of the
per-request latency is the data copy versus backing I/O.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
fs/fuse/args.h | 2 +
fs/fuse/dev.c | 24 ++++-
fs/fuse/dev_uring.c | 182 +++++++++++++++++++++++++++++++++++---
fs/fuse/dev_uring_i.h | 6 ++
fs/fuse/file.c | 2 +
fs/fuse/fuse_dev_i.h | 2 +
include/uapi/linux/fuse.h | 34 +++++++
7 files changed, 236 insertions(+), 16 deletions(-)
diff --git a/fs/fuse/args.h b/fs/fuse/args.h
index ecfe51a192af..5173264a1261 100644
--- a/fs/fuse/args.h
+++ b/fs/fuse/args.h
@@ -42,6 +42,8 @@ struct fuse_args {
bool is_pinned:1;
bool invalidate_vmap:1;
bool abort_on_kill:1;
+ /* server requested io-uring zero-copy for this op */
+ bool zero_copy:1;
struct fuse_in_arg in_args[4];
struct fuse_arg out_args[2];
void (*end)(struct fuse_args *args, int error);
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index d8f97943e973..90dceb7da571 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -1248,11 +1248,25 @@ int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
if (folio) {
size = folio_size(folio);
- if (zeroing && count < size)
- folio_zero_range(folio, 0, size);
+ if (zeroing && count < size) {
+ /*
+ * When the copy is skipped the folio already holds the
+ * payload, so only the bytes outside [offset, offset +
+ * count) may be zeroed.
+ *
+ * Otherwise, the whole folio is cleared first so that a
+ * failed copy leaves zeros rather than stale folio
+ * contents.
+ */
+ if (cs->skip_folio_copy)
+ folio_zero_segments(folio, 0, offset,
+ offset + count, size);
+ else
+ folio_zero_range(folio, 0, size);
+ }
}
- while (count) {
+ while (!cs->skip_folio_copy && count) {
if (cs->write && cs->pipebufs && folio) {
/*
* Can't control lifetime of pipe buffers, so always
@@ -1345,6 +1359,10 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs,
for (i = 0; !err && i < numargs; i++) {
struct fuse_arg *arg = &args[i];
if (i == numargs - 1 && argpages)
+ /*
+ * if cs->skip_folio_copy is set, this just does any
+ * needed zeroing. No copying is involved.
+ */
err = fuse_copy_folios(cs, arg->size, zeroing);
else
err = fuse_copy_one(cs, arg->value, arg->size);
diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
index 17806da93039..1547a4f0d9b9 100644
--- a/fs/fuse/dev_uring.c
+++ b/fs/fuse/dev_uring.c
@@ -22,6 +22,8 @@ MODULE_PARM_DESC(enable_uring,
#define FUSE_URING_IOV_HEADERS 0
#define FUSE_URING_IOV_PAYLOAD 1
+#define FUSE_URING_ADD_QUEUE_FLAGS (FUSE_URING_ZERO_COPY)
+
bool fuse_uring_enabled(void)
{
return enable_uring;
@@ -31,6 +33,11 @@ struct fuse_uring_pdu {
struct fuse_ring_ent *ent;
};
+struct fuse_zero_copy_bvs {
+ unsigned int nr_bvs;
+ struct bio_vec bvs[];
+};
+
static const struct fuse_iqueue_ops fuse_io_uring_ops;
enum fuse_uring_header_type {
@@ -113,8 +120,36 @@ static void fuse_uring_flush_bg(struct fuse_ring_queue *queue)
}
}
+static bool can_zero_copy_req(struct fuse_ring_ent *ent, struct fuse_req *req)
+{
+ struct fuse_args *args = req->args;
+
+ if (!ent->queue->zero_copy || !args->zero_copy)
+ return false;
+
+ if (args->opcode != FUSE_READ && args->opcode != FUSE_WRITE)
+ return false;
+
+ return args->in_pages || args->out_pages;
+}
+
+static void zero_copy_unregister(struct io_uring_cmd *cmd,
+ struct fuse_ring_ent *ent,
+ unsigned int issue_flags)
+{
+ if (ent->zero_copied) {
+ int err = io_buffer_unregister(cmd, ent->zero_copy_index,
+ issue_flags);
+
+ if (err)
+ pr_warn_ratelimited("qid=%d zero-copy unregister failed: %d\n",
+ ent->queue->qid, err);
+ ent->zero_copied = false;
+ }
+}
+
static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req,
- int error)
+ int error, unsigned int issue_flags)
{
struct fuse_ring_queue *queue = ent->queue;
struct fuse_ring *ring = queue->ring;
@@ -134,6 +169,8 @@ static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req,
spin_unlock(&queue->lock);
+ zero_copy_unregister(ent->cmd, ent, issue_flags);
+
if (error)
req->out.h.error = error;
@@ -309,7 +346,7 @@ void fuse_uring_conn_init(struct fuse_chan *fch)
}
static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
- int qid,
+ int qid, bool zero_copy,
bool fail_if_exists)
{
struct fuse_chan *fch = ring->chan;
@@ -328,6 +365,7 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
queue->qid = qid;
queue->ring = ring;
spin_lock_init(&queue->lock);
+ queue->zero_copy = zero_copy;
INIT_LIST_HEAD(&queue->ent_avail_queue);
INIT_LIST_HEAD(&queue->ent_commit_queue);
@@ -713,6 +751,9 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs,
fuse_copy_init(cs, dir == ITER_DEST, iter);
+ if (ent->zero_copied)
+ cs->skip_folio_copy = true;
+
cs->is_uring = true;
cs->req = req;
@@ -744,6 +785,62 @@ static int fuse_uring_copy_from_ring(struct fuse_req *req,
return err;
}
+static void fuse_zero_copy_release(void *priv)
+{
+ struct fuse_zero_copy_bvs *zc_bvs = priv;
+ unsigned int i;
+
+ for (i = 0; i < zc_bvs->nr_bvs; i++)
+ folio_put(page_folio(zc_bvs->bvs[i].bv_page));
+
+ kvfree(zc_bvs);
+}
+
+static int fuse_uring_set_up_zero_copy(struct fuse_ring_ent *ent,
+ struct fuse_req *req,
+ unsigned int issue_flags)
+{
+ struct fuse_args_pages *ap;
+ int err, i, ddir = 0;
+ struct fuse_zero_copy_bvs *zc_bvs;
+ struct bio_vec *bvs;
+
+ /* out_pages indicates a read, in_pages indicates a write */
+ if (req->args->out_pages)
+ ddir |= IO_BUF_DEST;
+ if (req->args->in_pages)
+ ddir |= IO_BUF_SOURCE;
+
+ ap = container_of(req->args, typeof(*ap), args);
+
+ zc_bvs = kvmalloc_flex(*zc_bvs, bvs, ap->num_folios,
+ GFP_KERNEL_ACCOUNT);
+ if (!zc_bvs)
+ return -ENOMEM;
+
+ zc_bvs->nr_bvs = ap->num_folios;
+ bvs = zc_bvs->bvs;
+ for (i = 0; i < ap->num_folios; i++) {
+ bvs[i].bv_page = folio_page(ap->folios[i], 0);
+ bvs[i].bv_offset = ap->descs[i].offset;
+ bvs[i].bv_len = ap->descs[i].length;
+ folio_get(ap->folios[i]);
+ }
+
+ err = io_buffer_register_bvec(ent->cmd, bvs, ap->num_folios,
+ fuse_zero_copy_release, zc_bvs,
+ ddir, ent->zero_copy_index,
+ issue_flags);
+ if (err) {
+ fuse_zero_copy_release(zc_bvs);
+ return err;
+ }
+
+ ent->zero_copied = true;
+
+ return 0;
+}
+
/*
* Copy data from the req to the ring buffer
*/
@@ -762,6 +859,13 @@ static int fuse_uring_args_to_ring(struct fuse_req *req,
.commit_id = req->in.h.unique,
};
+ if (can_zero_copy_req(ent, req)) {
+ ent_in_out.flags |= FUSE_URING_ENT_ZERO_COPY;
+ err = fuse_uring_set_up_zero_copy(ent, req, issue_flags);
+ if (err)
+ return err;
+ }
+
err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter,
issue_flags);
if (err)
@@ -793,6 +897,18 @@ static int fuse_uring_args_to_ring(struct fuse_req *req,
}
ent_in_out.payload_sz = cs.ring.copied_sz;
+ /*
+ * on a zero-copied write the pages are registered for the server to
+ * read via a fixed-buffer op rather than copied into the payload
+ * buffer, so copied_sz does not account for it. The server still needs
+ * the total inbound size to know how many bytes to read from the
+ * registered buffer, so add the page arg (always the last in-arg) back
+ * in
+ */
+ if (cs.skip_folio_copy && args->in_pages)
+ ent_in_out.payload_sz +=
+ args->in_args[args->in_numargs - 1].size;
+
if (bufpool_enabled(ent->queue) && ent->payload.iov_base)
ent_in_out.offset =
(uintptr_t)ent->payload.iov_base - ent->queue->bufpool->base_uaddr;
@@ -831,11 +947,25 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
sizeof(req->in.h));
}
-static bool fuse_uring_req_has_payload(struct fuse_req *req)
+static bool fuse_uring_req_has_copyable_payload(struct fuse_ring_ent *ent,
+ struct fuse_req *req)
{
struct fuse_args *args = req->args;
- return args->in_numargs > 1 || args->out_numargs;
+ if (!can_zero_copy_req(ent, req))
+ return args->in_numargs > 1 || args->out_numargs;
+
+ /*
+ * the asymmetry between in_numargs > 2 and out_numargs > 1 is because
+ * the per-op header is extracted before fuse_copy_args() for inargs but
+ * not for outargs
+ */
+ if ((args->in_numargs > 1) && (!args->in_pages || args->in_numargs > 2))
+ return true;
+ if (args->out_numargs && (!args->out_pages || args->out_numargs > 1))
+ return true;
+
+ return false;
}
static int fuse_uring_select_buffer(struct fuse_ring_ent *ent)
@@ -892,7 +1022,7 @@ static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent,
return 0;
buffer_selected = !!ent->payload.iov_base;
- has_payload = fuse_uring_req_has_payload(req);
+ has_payload = fuse_uring_req_has_copyable_payload(ent, req);
if (has_payload && !buffer_selected)
return fuse_uring_select_buffer(ent);
@@ -910,7 +1040,7 @@ static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
return 0;
/* no payload to copy, can skip selecting a buffer */
- if (!fuse_uring_req_has_payload(req))
+ if (!fuse_uring_req_has_copyable_payload(ent, req))
return 0;
return fuse_uring_select_buffer(ent);
@@ -936,7 +1066,7 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
ent->state = FRRS_INVALID;
spin_unlock(&ent->queue->lock);
- fuse_uring_req_end(ent, req, err);
+ fuse_uring_req_end(ent, req, err, issue_flags);
}
return err;
@@ -1035,7 +1165,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
err = fuse_uring_copy_from_ring(req, ent, issue_flags);
out:
- fuse_uring_req_end(ent, req, err);
+ fuse_uring_req_end(ent, req, err, issue_flags);
}
/*
@@ -1160,7 +1290,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
queue->qid, commit_id, ent->state);
fuse_uring_recycle_buffer(ent);
spin_unlock(&queue->lock);
- fuse_uring_req_end(ent, req, err);
+ /*
+ * Unregister any zero copyable pages since ent->cmd is null
+ * when it hits fuse_uring_req_end() in this path
+ */
+ zero_copy_unregister(cmd, ent, issue_flags);
+ fuse_uring_req_end(ent, req, err, issue_flags);
return err;
}
@@ -1284,10 +1419,14 @@ static struct fuse_ring_ent *
fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
struct fuse_ring_queue *queue)
{
+ const struct fuse_uring_cmd_req *cmd_req =
+ io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
struct fuse_ring *ring = queue->ring;
struct fuse_ring_ent *ent;
struct iovec iov[FUSE_URING_IOV_SEGS];
struct iovec *headers, *payload;
+ unsigned int zero_copy_index;
+
int err;
err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);
@@ -1297,6 +1436,10 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
return ERR_PTR(err);
}
+ zero_copy_index = READ_ONCE(cmd_req->ent_zero_copy_buf_index);
+ if (zero_copy_index && !queue->zero_copy)
+ return ERR_PTR(-EINVAL);
+
err = -EINVAL;
headers = &iov[FUSE_URING_IOV_HEADERS];
if (headers->iov_len < sizeof(struct fuse_uring_req_header)) {
@@ -1315,9 +1458,14 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
}
} else {
if (payload->iov_len < ring->max_payload_sz) {
+ spin_unlock(&queue->lock);
pr_info_ratelimited("Invalid req payload len %zu\n",
payload->iov_len);
+ return ERR_PTR(err);
+ }
+ if (queue->zero_copy) {
spin_unlock(&queue->lock);
+ pr_info_ratelimited("Can only use zero copy with bufpools\n");
return ERR_PTR(err);
}
queue->payload_mode = FUSE_PAYLOAD_PER_ENT;
@@ -1335,6 +1483,7 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
ent->headers = headers->iov_base;
if (queue->payload_mode == FUSE_PAYLOAD_PER_ENT)
ent->payload = *payload;
+ ent->zero_copy_index = zero_copy_index;
atomic_inc(&ring->queue_refs);
return ent;
@@ -1364,7 +1513,7 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
queue = READ_ONCE(ring->queues[qid]);
if (!queue) {
- queue = fuse_uring_create_queue(ring, qid, false);
+ queue = fuse_uring_create_queue(ring, qid, false, false);
if (IS_ERR(queue))
return PTR_ERR(queue);
}
@@ -1389,8 +1538,9 @@ static int fuse_uring_add_queue(struct io_uring_cmd *cmd, struct fuse_chan *fch)
unsigned int qid = READ_ONCE(cmd_req->qid);
uint64_t flags = READ_ONCE(cmd_req->flags);
struct fuse_ring_queue *queue;
+ bool zero_copy = flags & FUSE_URING_ZERO_COPY;
- if (!ring || flags)
+ if (!ring)
return -EINVAL;
if (qid >= ring->nr_queues) {
@@ -1398,7 +1548,13 @@ static int fuse_uring_add_queue(struct io_uring_cmd *cmd, struct fuse_chan *fch)
return -EINVAL;
}
- queue = fuse_uring_create_queue(ring, qid, true);
+ if (flags & ~FUSE_URING_ADD_QUEUE_FLAGS)
+ return -EINVAL;
+
+ if (zero_copy && !capable(CAP_SYS_ADMIN))
+ return -EPERM;
+
+ queue = fuse_uring_create_queue(ring, qid, zero_copy, true);
if (IS_ERR(queue))
return PTR_ERR(queue);
@@ -1595,7 +1751,7 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)
io_uring_cmd_done(cmd, err, issue_flags);
- fuse_uring_req_end(ent, ent->fuse_req, err);
+ fuse_uring_req_end(ent, ent->fuse_req, err, issue_flags);
kfree(ent);
if (atomic_dec_and_test(&queue->ring->queue_refs))
wake_up_all(&queue->ring->stop_waitq);
diff --git a/fs/fuse/dev_uring_i.h b/fs/fuse/dev_uring_i.h
index e142cae43022..263d0f8b9714 100644
--- a/fs/fuse/dev_uring_i.h
+++ b/fs/fuse/dev_uring_i.h
@@ -79,6 +79,10 @@ struct fuse_ring_ent {
/* buffer id in the pool, if bufpools are used. ignored otherwise */
unsigned int buf_id;
+ /* true if the request's pages are being zero-copied */
+ bool zero_copied;
+ unsigned int zero_copy_index;
+
/* the ring queue that owns the request */
struct fuse_ring_queue *queue;
@@ -142,6 +146,8 @@ struct fuse_ring_queue {
/* only allocated when payload_mode == FUSE_PAYLOAD_BUFPOOL */
struct fuse_bufpool *bufpool;
+
+ bool zero_copy;
};
/*
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index da5859e8159d..7b883cf170ac 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -605,6 +605,7 @@ void fuse_read_args_fill(struct fuse_io_args *ia, struct file *file, loff_t pos,
args->out_argvar = true;
args->out_numargs = 1;
args->out_args[0].size = count;
+ args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY;
}
static void fuse_release_user_pages(struct fuse_args_pages *ap, ssize_t nres,
@@ -1151,6 +1152,7 @@ static void fuse_write_args_fill(struct fuse_io_args *ia, struct fuse_file *ff,
args->out_numargs = 1;
args->out_args[0].size = sizeof(ia->write.out);
args->out_args[0].value = &ia->write.out;
+ args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY;
}
static unsigned int fuse_write_flags(struct kiocb *iocb)
diff --git a/fs/fuse/fuse_dev_i.h b/fs/fuse/fuse_dev_i.h
index 668c8391d61c..f41749f484df 100644
--- a/fs/fuse/fuse_dev_i.h
+++ b/fs/fuse/fuse_dev_i.h
@@ -325,6 +325,8 @@ struct fuse_copy_state {
bool write:1;
bool move_folios:1;
bool is_uring:1;
+ /* set when the payload is zero-copied. folios are filled in place */
+ bool skip_folio_copy:1;
struct {
unsigned int copied_sz; /* copied size into the user buffer */
} ring;
diff --git a/include/uapi/linux/fuse.h b/include/uapi/linux/fuse.h
index 538d844da099..7435e09c87fe 100644
--- a/include/uapi/linux/fuse.h
+++ b/include/uapi/linux/fuse.h
@@ -246,6 +246,8 @@
* - add FUSE_HAS_IO_URING_BUFPOOL
* - add fuse_uring_cmd_req bufpool struct
* - add bufpool offset field to fuse_uring_ent_in_out struct
+ * - add FUSE_URING_ZERO_COPY, FUSE_URING_ENT_ZERO_COPY, and
+ * FOPEN_IO_URING_ZERO_COPY flag
*/
#ifndef _LINUX_FUSE_H
@@ -389,6 +391,12 @@ struct fuse_file_lock {
* FOPEN_NOFLUSH: don't flush data cache on close (unless FUSE_WRITEBACK_CACHE)
* FOPEN_PARALLEL_DIRECT_WRITES: Allow concurrent direct writes on the same inode
* FOPEN_PASSTHROUGH: passthrough read/write io for this open file
+ * FOPEN_IO_URING_ZERO_COPY: use io-uring zero-copy for reads/writes on this
+ * open file. Honored only when the serving io-uring
+ * queue was set up for zero-copy
+ * (FUSE_URING_ZERO_COPY) and the request carries page
+ * payload. Otherwise reads/writes fall back to
+ * copying.
*/
#define FOPEN_DIRECT_IO (1 << 0)
#define FOPEN_KEEP_CACHE (1 << 1)
@@ -398,6 +406,7 @@ struct fuse_file_lock {
#define FOPEN_NOFLUSH (1 << 5)
#define FOPEN_PARALLEL_DIRECT_WRITES (1 << 6)
#define FOPEN_PASSTHROUGH (1 << 7)
+#define FOPEN_IO_URING_ZERO_COPY (1 << 8)
/**
* INIT request/reply flags
@@ -1259,6 +1268,13 @@ struct fuse_supp_groups {
#define FUSE_URING_IN_OUT_HEADER_SZ 128
#define FUSE_URING_OP_IN_OUT_SZ 128
+/**
+ * fuse_uring_ent_in_out flags
+ *
+ * FUSE_URING_ENT_ZERO_COPY: Set if the ent's payload is zero-copied
+ */
+#define FUSE_URING_ENT_ZERO_COPY (1 << 0)
+
/* Used as part of the fuse_uring_req_header */
struct fuse_uring_ent_in_out {
uint64_t flags;
@@ -1310,6 +1326,14 @@ enum fuse_uring_cmd {
FUSE_IO_URING_CMD_ADD_BUFPOOL = 4,
};
+/*
+ * fuse_uring_cmd_req flags for FUSE_IO_URING_CMD_ADD_QUEUE
+ *
+ * FUSE_URING_ZERO_COPY is only supported for queues with bufpools on privileged
+ * servers
+ */
+#define FUSE_URING_ZERO_COPY (1 << 0)
+
/**
* In the 80B command area of the SQE.
*/
@@ -1330,6 +1354,16 @@ struct fuse_uring_cmd_req {
uint32_t len;
uint32_t reserved;
} bufpool;
+
+ /*
+ * Index of this entry's slot in the server's io_uring
+ * registered buffer table, where the kernel registers the
+ * request's pages for zero-copy. Set for
+ * FUSE_IO_URING_CMD_REGISTER cmds only, and only on queues
+ * created with FUSE_URING_ZERO_COPY. On a non-zero-copy queue
+ * this must be 0
+ */
+ uint16_t ent_zero_copy_buf_index;
};
};
--
2.52.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v7 6/6] docs: fuse: document io-uring buffer pool and zero-copy uapi
2026-08-14 18:59 [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
` (4 preceding siblings ...)
2026-08-14 18:59 ` [PATCH v7 5/6] fuse: add zero-copy over io-uring Joanne Koong
@ 2026-08-14 18:59 ` Joanne Koong
2026-08-14 19:23 ` [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
2026-08-17 15:29 ` Miklos Szeredi
7 siblings, 0 replies; 27+ messages in thread
From: Joanne Koong @ 2026-08-14 18:59 UTC (permalink / raw)
To: miklos; +Cc: jlayton, libaokun, axboe, bernd, amir73il, fuse-devel
Add documentation for fuse over io-uring usage of buffer pools and
zero-copy.
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
.../filesystems/fuse/fuse-io-uring.rst | 36 ++++-
Documentation/filesystems/fuse/index.rst | 1 +
.../fuse/uapi/fuse-uapi-io-uring.rst | 126 ++++++++++++++++++
3 files changed, 161 insertions(+), 2 deletions(-)
create mode 100644 Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst
diff --git a/Documentation/filesystems/fuse/fuse-io-uring.rst b/Documentation/filesystems/fuse/fuse-io-uring.rst
index d73dd0dbd238..29f98057500d 100644
--- a/Documentation/filesystems/fuse/fuse-io-uring.rst
+++ b/Documentation/filesystems/fuse/fuse-io-uring.rst
@@ -11,6 +11,9 @@ and works. For generic details about FUSE see fuse.rst.
This document also covers the current interface, which is
still in development and might change.
+For the userspace protocol, see
+Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst.
+
Limitations
===========
As of now not all requests types are supported through io-uring, userspace
@@ -95,5 +98,34 @@ Sending requests with CQEs
| <fuse_unlink() |
| <sys_unlink() |
-
-
+Buffer pools
+============
+
+Without a buffer pool, every entry needs to pass a dedicated payload buffer
+large enough for the maximum payload size. A buffer pool decouples entries
+from payload buffers. The server hands the kernel one contiguous buffer pool
+of memory and when the kernel sends the server a request, it indicates the
+offset into the pool for that request's payload. Internally, the kernel is
+able to manage/optimize the buffer pool memory however it likes.
+
+A server may also register the pool region with io_uring as a fixed buffer.
+The backing pages are then pinned once, avoiding per-request pinning and
+address translation. This also allows servers to use the same registered
+buffers for subsequent backing store I/O through io-uring, keeping data
+in the same pinned pages without additional pinning / mapping overhead.
+
+Zero-copy
+=========
+
+Zero-copy lets the server read from / write to the client's pages (pinned
+user pages for direct I/O, or page-cache folios for buffered I/O) without an
+intermediary payload copy. This requires CAP_SYS_ADMIN privileges.
+
+When a fuse request arrives for a file that opted into zero-copy, the kernel
+registers the relevant pages (pinned user pages for direct i/o or underlying
+page cache folios for buffered i/o) into a sparse slot in the server's
+io_uring registered buffer table. The server can then operate on these pages
+directly using io-uring fixed buffer operations (eg read_fixed / write_fixed)
+and the kernel unregisters these pages when the request completes.
+Non-page-backed args (eg op out headers) will go through the payload buffer as
+normal.
diff --git a/Documentation/filesystems/fuse/index.rst b/Documentation/filesystems/fuse/index.rst
index 393a845214da..3dada6c4057a 100644
--- a/Documentation/filesystems/fuse/index.rst
+++ b/Documentation/filesystems/fuse/index.rst
@@ -12,3 +12,4 @@ FUSE (Filesystem in Userspace) Technical Documentation
fuse-io
fuse-io-uring
fuse-passthrough
+ uapi/fuse-uapi-io-uring
diff --git a/Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst b/Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst
new file mode 100644
index 000000000000..8367be7ea29d
--- /dev/null
+++ b/Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst
@@ -0,0 +1,126 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================================
+FUSE-over-io-uring uapi documentation
+=====================================
+
+Commands
+========
+
+``enum fuse_uring_cmd``:
+
+``FUSE_IO_URING_CMD_ADD_QUEUE``
+ Create a queue identified by ``fuse_uring_cmd_req.qid``. Queue-wide
+ options are passed in ``fuse_uring_cmd_req.flags``:
+
+ ``FUSE_URING_ZERO_COPY``
+ Enable zero-copy on this queue. Requires ``CAP_SYS_ADMIN`` and a buffer
+ pool, which is added separately via ``ADD_BUFPOOL`` before registering
+ entries (see `Zero-copy`_).
+
+``FUSE_IO_URING_CMD_ADD_BUFPOOL``
+ Register the payload buffer pool for an existing queue. The server provides
+ a single contiguous region in ``fuse_uring_cmd_req.bufpool.uaddr`` /
+ ``.len``. This command must be issued after ``ADD_QUEUE`` and before
+ registering any payload-carrying entries on that queue.
+ ``fuse_uring_cmd_req.flags`` must be 0. Submitting this command with
+ ``IORING_URING_CMD_FIXED`` marks the pool as registered, which avoids per
+ i/o pinning/unpinning and mapping overhead (see `Buffer pools`_).
+
+``FUSE_IO_URING_CMD_REGISTER``
+ Register a ring entry (a long-lived SQE that carries the request header
+ iovec). For a zero-copy queue, ``fuse_uring_cmd_req.ent_zero_copy_buf_index``
+ indicates the reserved registered buffer table slot this entry uses for
+ zero-copy (see `Zero-copy`_).
+
+``FUSE_IO_URING_CMD_COMMIT_AND_FETCH``
+ Commit the reply for a completed request and fetch the next one. The
+ request is identified by ``fuse_uring_cmd_req.commit_id`` (the value the
+ kernel reported in ``fuse_uring_ent_in_out.commit_id``).
+
+Structures
+==========
+
+``struct fuse_uring_cmd_req`` (80-byte SQE command area):
+
+============================ ==================================================
+Field Meaning
+============================ ==================================================
+``flags`` Command-specific flags (see each command).
+``commit_id`` Request id, for ``COMMIT_AND_FETCH``.
+``qid`` Queue index.
+``bufpool.uaddr`` Pool base address, for ``ADD_BUFPOOL``.
+``bufpool.len`` Pool length in bytes, for ``ADD_BUFPOOL``.
+``bufpool.reserved`` Must be 0, for ``ADD_BUFPOOL``.
+``ent_zero_copy_buf_index`` Per-entry zero-copy slot, for ``REGISTER``.
+============================ ==================================================
+
+``struct fuse_uring_ent_in_out`` (reported by the kernel per request):
+
+============================ ==================================================
+Field Meaning
+============================ ==================================================
+``flags`` ``FUSE_URING_ENT_ZERO_COPY`` if zero-copied.
+``commit_id`` Id to echo back in ``COMMIT_AND_FETCH``.
+``payload_sz`` Total payload size in bytes (see `Zero-copy`_).
+``offset`` Payload buffer offset within the pool.
+============================ ==================================================
+
+Buffer pools
+============
+Setup:
+
+* Issue ``ADD_QUEUE`` for the qid.
+* Issue ``ADD_BUFPOOL`` with ``bufpool.uaddr`` and ``bufpool.len`` pointing
+ at the region.
+* Register entries with ``REGISTER``.
+
+For every request that has a payload, the kernel reports where the payload
+lives in ``struct fuse_uring_ent_in_out`` (part of
+``struct fuse_uring_req_header``):
+
+``offset``
+ Byte offset, within the pool region, for this request's payload buffer.
+ The server adds this to the pool base address to locate the payload.
+
+``payload_sz``
+ Number of payload bytes for this request.
+
+To use registered buffers, the server registers the pool region with io_uring
+and submits ``ADD_BUFPOOL`` with ``IORING_URING_CMD_FIXED`` set in
+``sqe->uring_cmd_flags`` and the index of the registered bufpool in
+``sqe->buf_index``. Every SQE the server submits afterwards must follow the
+same fixed-buffer protocol, carrying ``IORING_URING_CMD_FIXED`` and that same
+``sqe->buf_index``. The same registered buffer can be reused for the server's
+backing-store I/O as well (e.g. ``IORING_OP_READ_FIXED`` /
+``IORING_OP_WRITE_FIXED``).
+
+Zero-copy
+=========
+Requirements:
+
+* The server must be privileged (``CAP_SYS_ADMIN``).
+* A zero-copy queue: ``ADD_QUEUE`` with the ``FUSE_URING_ZERO_COPY`` flag set.
+* A buffer pool: ``ADD_BUFPOOL``.
+* For each entry, ``REGISTER`` with ``ent_zero_copy_buf_index`` set to the
+ index this entry uses in the server's io_uring registered-buffer table.
+ This is where the kernel registers the request's pages for the server to
+ access (it is separate from the payload pool). On a non-zero-copy queue this
+ field must be 0.
+
+Zero-copy is selected per open file. The server sets the open-file flag in
+the ``FUSE_OPEN`` / ``FUSE_CREATE`` reply:
+
+``FOPEN_IO_URING_ZERO_COPY``
+ Reads/writes on this open file should use zero-copy.
+
+For a request that is zero-copied, the kernel sets ``FUSE_URING_ENT_ZERO_COPY``
+in ``fuse_uring_ent_in_out.flags`` and places the request's pages at the
+entry's ``ent_zero_copy_buf_index``. The server then issues
+``IORING_OP_READ_FIXED`` / ``IORING_OP_WRITE_FIXED`` against that index to
+transfer the data directly to/from the client's pages.
+
+For such a request, ``payload_sz`` includes the zero-copied page bytes
+(transferred via the registered buffer at ``ent_zero_copy_buf_index``). Any
+non-page-backed args (e.g. op headers) are still copied through the pool
+payload buffer at ``offset``.
--
2.52.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy
2026-08-14 18:59 [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
` (5 preceding siblings ...)
2026-08-14 18:59 ` [PATCH v7 6/6] docs: fuse: document io-uring buffer pool and zero-copy uapi Joanne Koong
@ 2026-08-14 19:23 ` Joanne Koong
2026-08-17 15:29 ` Miklos Szeredi
7 siblings, 0 replies; 27+ messages in thread
From: Joanne Koong @ 2026-08-14 19:23 UTC (permalink / raw)
To: miklos; +Cc: jlayton, libaokun, axboe, bernd, amir73il, fuse-devel
On Fri, Aug 14, 2026 at 12:00 PM Joanne Koong <joannelkoong@gmail.com> wrote:
>
> This series implements fuse io-uring buffer pools and zero-copy.
Hi Miklos,
This series is ready for the upcoming merge window. Bernd reviewed v6
[1] this week and v7 addresses his comments/preferences.
The 4 io-uring patches this series depends on are queued in Jens's
for-7.2/io_uring-fuse branch [2]. There are no changes to these
io-uring patches. There will be a merge conflict when you pull the 2nd
patch "split io_buffer_register_request() logic":
diff --cc io_uring/rsrc.c
index 40807994a8f4,666d3a009ffc..000000000000
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@@ -1057,15 -998,7 +1086,19 @@@ int io_buffer_register_request(struct i
goto unlock;
}
++<<<<<<< HEAD
+ imu->ubuf = 0;
+ imu->len = blk_rq_bytes(rq);
+ imu->folio_shift = PAGE_SHIFT;
+ refcount_set(&imu->refs, 1);
+ imu->release = release;
+ imu->priv = rq;
+ imu->flags = IO_REGBUF_F_KBUF;
+ imu->dir = 1 << rq_data_dir(rq);
+
++=======
+ nr_bvecs = 0;
++>>>>>>> 27eada8066bd (io_uring/rsrc: split io_buffer_register_request() logic)
rq_for_each_bvec(bv, rq, rq_iter)
imu->bvec[nr_bvecs++] = bv;
imu->nr_bvecs = nr_bvecs;
that can be resolved by applying this:
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index e1f4ad026f16..dfc2e63214f7 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -958,7 +958,6 @@ static struct io_mapped_ubuf
*io_kernel_buffer_init(struct io_ring_ctx *ctx,
imu->ubuf = 0;
imu->len = total_bytes;
- imu->acct_pages = 0;
imu->folio_shift = PAGE_SHIFT;
imu->nr_bvecs = nr_bvecs;
refcount_set(&imu->refs, 1);
@@ -997,19 +996,8 @@ int io_buffer_register_request(struct
io_uring_cmd *cmd, struct request *rq,
ret = PTR_ERR(imu);
goto unlock;
}
-<<<<<<< HEAD
- imu->ubuf = 0;
- imu->len = blk_rq_bytes(rq);
- imu->folio_shift = PAGE_SHIFT;
- refcount_set(&imu->refs, 1);
- imu->release = release;
- imu->priv = rq;
- imu->flags = IO_REGBUF_F_KBUF;
- imu->dir = 1 << rq_data_dir(rq);
-=======
nr_bvecs = 0;
->>>>>>> 27eada8066bd (io_uring/rsrc: split io_buffer_register_request() logic)
rq_for_each_bvec(bv, rq, rq_iter)
imu->bvec[nr_bvecs++] = bv;
imu->nr_bvecs = nr_bvecs;
If it makes things easier, this is the github tree on top of the fuse
for-next branch with the io-uring patches (with the merge conflict
fixup) and this series applied [3].
If there's anything else you need, please let me know.
Thanks,
Joanne
[1] https://lore.kernel.org/fuse-devel/20260716175908.2339738-1-joannelkoong@gmail.com/
[2] https://lore.kernel.org/fuse-devel/04ab6f18-f668-43b4-9e03-f63d45460742@kernel.dk/
[3] https://github.com/joannekoong/linux/commits/fuse_zero_copy_v7/
^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: [PATCH v7 4/6] fuse: support registered buffer pools in io-uring
2026-08-14 18:59 ` [PATCH v7 4/6] fuse: support registered buffer pools in io-uring Joanne Koong
@ 2026-08-17 10:15 ` Bernd Schubert
0 siblings, 0 replies; 27+ messages in thread
From: Bernd Schubert @ 2026-08-17 10:15 UTC (permalink / raw)
To: Joanne Koong, miklos; +Cc: jlayton, libaokun, axboe, amir73il, fuse-devel
On 8/14/26 20:59, Joanne Koong wrote:
> Allow servers to use a buffer pool that is also registered through
> io-uring. When the server registers a buffer pool with io-uring, the
> pages backing the pool are pinned upfront. This eliminates the overhead
> of pinning/unpinning user pages and translating virtual addresses per
> i/o request. This also allows servers to use the same registered memory
> for subsequent backing store I/O (eg read_fixed/write_fixed), keeping
> data in the same pinned pages without additional pinning or mapping
> overhead required.
>
> To use this, the server needs to set the FUSE_URING_REGISTERED_BUFPOOL
> flag when adding a bufpool through the FUSE_IO_URING_CMD_ADD_BUFPOOL
> cmd. For every sqe submitted (including the one for adding the bufpool),
> it should set sqe->uring_cmd_flags to include IORING_URING_CMD_FIXED,
> and pass in the index where the registered bufpool resides to
> sqe->buf_index.
>
> Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
> 2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
> engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
> direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
> I/O is not the bottleneck):
>
> baseline registered buffers
> direct read ~5.1 GB/s ~5.4 GB/s (+~5%)
> direct write ~3.4 GB/s ~4.8 GB/s (+~45%)
>
> Registered buffers bring up the write path speed up closer to speed of
> reads. There isn't much improvement for reads because it is already fast
> enough where it's at the copy-bound ceiling (surpassing that requires
> doing zero-copy). On a device-bound NVMe though, the differences are
> within noise, as backing I/O dominates per-request latency.
>
> Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
> ---
> fs/fuse/dev_uring.c | 112 ++++++++++++++++++++++++++++++++++--------
> fs/fuse/dev_uring_i.h | 8 +++
> 2 files changed, 99 insertions(+), 21 deletions(-)
>
> diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
> index d300c7f441c4..17806da93039 100644
> --- a/fs/fuse/dev_uring.c
> +++ b/fs/fuse/dev_uring.c
> @@ -47,6 +47,27 @@ static inline bool bufpool_enabled(struct fuse_ring_queue *queue)
> return queue->payload_mode == FUSE_PAYLOAD_BUFPOOL;
> }
>
> +static inline bool bufpool_registered(struct fuse_ring_queue *queue)
> +{
> + return queue->bufpool && queue->bufpool->registered;
> +}
> +
> +/*
> + * For a registered bufpool, every sqe that drives a payload import (REGISTER,
> + * COMMIT_AND_FETCH) must carry the registered buffer index of the pool.
> + * This also must be called from the command's issue handler, where cmd->sqe is
> + * still valid
> + */
> +static inline bool fuse_uring_cmd_index_ok(struct io_uring_cmd *cmd,
> + struct fuse_ring_queue *queue)
> +{
> + if (!bufpool_registered(queue))
> + return true;
> +
> + return (cmd->flags & IORING_URING_CMD_FIXED) &&
> + READ_ONCE(cmd->sqe->buf_index) == queue->bufpool->registered_index;
> +}
> +
> static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd,
> struct fuse_ring_ent *ring_ent)
> {
> @@ -653,19 +674,42 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent,
> return 0;
> }
>
> +static int fuse_uring_import_payload(struct fuse_ring_ent *ent, int dir,
> + struct iov_iter *iter,
> + unsigned int issue_flags)
> +{
> + void __user *base = ent->payload.iov_base;
> + size_t len = ent->payload.iov_len;
> + int err = 0;
> +
> + if (!base) {
> + memset(iter, 0, sizeof(*iter));
> + return 0;
> + }
> +
> + if (bufpool_registered(ent->queue))
> + err = io_uring_cmd_import_fixed((u64)(uintptr_t)base, len, dir,
> + iter, ent->cmd, issue_flags);
> + else
> + err = import_ubuf(dir, base, len, iter);
> +
> + if (err)
> + pr_info_ratelimited("fuse: Import of user buffer failed\n");
> +
> + return err;
> +}
> +
> static int setup_fuse_copy_state(struct fuse_copy_state *cs,
> struct fuse_req *req,
> struct fuse_ring_ent *ent, int dir,
> - struct iov_iter *iter)
> + struct iov_iter *iter,
> + unsigned int issue_flags)
> {
> int err;
>
> - err = import_ubuf(dir, ent->payload.iov_base, ent->payload.iov_len,
> - iter);
> - if (err) {
> - pr_info_ratelimited("fuse: Import of user buffer failed\n");
> + err = fuse_uring_import_payload(ent, dir, iter, issue_flags);
> + if (err)
> return err;
> - }
>
> fuse_copy_init(cs, dir == ITER_DEST, iter);
>
> @@ -676,7 +720,8 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs,
> }
>
> static int fuse_uring_copy_from_ring(struct fuse_req *req,
> - struct fuse_ring_ent *ent)
> + struct fuse_ring_ent *ent,
> + unsigned int issue_flags)
> {
> struct fuse_copy_state cs;
> struct fuse_args *args = req->args;
> @@ -689,7 +734,8 @@ static int fuse_uring_copy_from_ring(struct fuse_req *req,
> if (err)
> return err;
>
> - err = setup_fuse_copy_state(&cs, req, ent, ITER_SOURCE, &iter);
> + err = setup_fuse_copy_state(&cs, req, ent, ITER_SOURCE, &iter,
> + issue_flags);
> if (err)
> return err;
>
> @@ -702,7 +748,8 @@ static int fuse_uring_copy_from_ring(struct fuse_req *req,
> * Copy data from the req to the ring buffer
> */
> static int fuse_uring_args_to_ring(struct fuse_req *req,
> - struct fuse_ring_ent *ent)
> + struct fuse_ring_ent *ent,
> + unsigned int issue_flags)
> {
> struct fuse_copy_state cs;
> struct fuse_args *args = req->args;
> @@ -715,7 +762,8 @@ static int fuse_uring_args_to_ring(struct fuse_req *req,
> .commit_id = req->in.h.unique,
> };
>
> - err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter);
> + err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter,
> + issue_flags);
> if (err)
> return err;
>
> @@ -754,7 +802,8 @@ static int fuse_uring_args_to_ring(struct fuse_req *req,
> }
>
> static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
> - struct fuse_req *req)
> + struct fuse_req *req,
> + unsigned int issue_flags)
> {
> struct fuse_ring_queue *queue = ent->queue;
> int err;
> @@ -771,7 +820,7 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
> return err;
>
> /* copy the request */
> - err = fuse_uring_args_to_ring(req, ent);
> + err = fuse_uring_args_to_ring(req, ent, issue_flags);
> if (unlikely(err)) {
> pr_info_ratelimited("Copy to ring failed: %d\n", err);
> return err;
> @@ -868,11 +917,12 @@ static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
> }
>
> static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
> - struct fuse_req *req)
> + struct fuse_req *req,
> + unsigned int issue_flags)
> {
> int err;
>
> - err = fuse_uring_copy_to_ring(ent, req);
> + err = fuse_uring_copy_to_ring(ent, req, issue_flags);
> if (!err) {
> set_bit(FR_SENT, &req->flags);
> trace_fuse_request_sent(req);
> @@ -983,7 +1033,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
> goto out;
> }
>
> - err = fuse_uring_copy_from_ring(req, ent);
> + err = fuse_uring_copy_from_ring(req, ent, issue_flags);
> out:
> fuse_uring_req_end(ent, req, err);
> }
> @@ -995,7 +1045,8 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
> * Else, there is no next fuse request and this returns false.
> */
> static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
> - struct fuse_ring_queue *queue)
> + struct fuse_ring_queue *queue,
> + unsigned int issue_flags)
> {
> int err;
> struct fuse_req *req;
> @@ -1007,7 +1058,7 @@ static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
> spin_unlock(&queue->lock);
>
> if (req) {
> - err = fuse_uring_prepare_send(ent, req);
> + err = fuse_uring_prepare_send(ent, req, issue_flags);
> if (err)
> goto retry;
> }
> @@ -1081,6 +1132,11 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
> return err;
> }
>
> + if (!fuse_uring_cmd_index_ok(cmd, queue)) {
> + spin_unlock(&queue->lock);
> + return -EINVAL;
> + }
> +
> /* Find a request based on the unique ID of the fuse request
> * This should get revised, as it needs a hash calculation and list
> * search. And full struct fuse_pqueue is needed (memory overhead).
> @@ -1126,7 +1182,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
> * available and the cmd only returns to userspace when there's a
> * next request and an available buffer.
> */
> - if (fuse_uring_get_next_fuse_req(ent, queue))
> + if (fuse_uring_get_next_fuse_req(ent, queue, issue_flags))
> fuse_uring_send(ent, cmd, 0, issue_flags);
> return 0;
> }
> @@ -1252,7 +1308,8 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
>
> spin_lock(&queue->lock);
> if (bufpool_enabled(queue)) {
> - if (payload->iov_base || payload->iov_len) {
> + if (payload->iov_base || payload->iov_len ||
> + !fuse_uring_cmd_index_ok(cmd, queue)) {
> spin_unlock(&queue->lock);
> return ERR_PTR(err);
> }
> @@ -1362,6 +1419,7 @@ static int fuse_uring_add_bufpool(struct io_uring_cmd *cmd,
> uintptr_t pool_uaddr;
> unsigned int pool_len, nr_bufs;
> size_t pool_size, buf_size;
> + bool registered = cmd->flags & IORING_URING_CMD_FIXED;
>
> if (!ring || qid >= ring->nr_queues || flags)
> return -EINVAL;
> @@ -1396,6 +1454,17 @@ static int fuse_uring_add_bufpool(struct io_uring_cmd *cmd,
> /* all buffers are free */
> bitmap_set(pool->free_map, 0, nr_bufs);
>
> + /*
> + * A registered bufpool is reached through an io_uring fixed buffer, so
> + * the pool is registered iff this command was submitted with
> + * IORING_URING_CMD_FIXED. The registered buffer index is taken from
> + * sqe->buf_index.
> + */
> + if (registered) {
> + pool->registered = true;
> + pool->registered_index = READ_ONCE(cmd->sqe->buf_index);
> + }
> +
> spin_lock(&queue->lock);
> if (queue->payload_mode != FUSE_PAYLOAD_UNSET) {
> spin_unlock(&queue->lock);
> @@ -1508,9 +1577,10 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)
> int err;
>
> if (!tw.cancel) {
> - err = fuse_uring_prepare_send(ent, ent->fuse_req);
> + err = fuse_uring_prepare_send(ent, ent->fuse_req, issue_flags);
> if (err) {
> - if (!fuse_uring_get_next_fuse_req(ent, queue))
> + if (!fuse_uring_get_next_fuse_req(ent, queue,
> + issue_flags))
> return;
> err = 0;
> }
> diff --git a/fs/fuse/dev_uring_i.h b/fs/fuse/dev_uring_i.h
> index cdf56f8b38b5..e142cae43022 100644
> --- a/fs/fuse/dev_uring_i.h
> +++ b/fs/fuse/dev_uring_i.h
> @@ -49,6 +49,14 @@ enum fuse_queue_payload_mode {
> };
>
> struct fuse_bufpool {
> + bool registered;
> +
> + /*
> + * io_uring registered buffer table index for this pool, bound at
> + * ADD_BUFPOOL time. Only valid if the bufpool is registered
> + */
> + u16 registered_index;
> +
> /* starting uaddr of the bufpool */
> uintptr_t base_uaddr;
>
Thanks for the update!
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 5/6] fuse: add zero-copy over io-uring
2026-08-14 18:59 ` [PATCH v7 5/6] fuse: add zero-copy over io-uring Joanne Koong
@ 2026-08-17 13:05 ` Bernd Schubert
0 siblings, 0 replies; 27+ messages in thread
From: Bernd Schubert @ 2026-08-17 13:05 UTC (permalink / raw)
To: Joanne Koong, miklos; +Cc: jlayton, libaokun, axboe, amir73il, fuse-devel
On 8/14/26 20:59, Joanne Koong wrote:
> Implement zero-copy in fuse io-uring to eliminate memory copies between
> the application, kernel, and server for read/write operations. The
> server can directly access client pages or page cache folios without
> copying data through an intermediary buffer. When a fuse request arrives,
> the kernel registers the relevant pages into a sparse slot in the
> server's io_uring registered buffer table. The server can then operate
> on these pages directly using io-uring fixed buffer operations (eg
> read_fixed/write_fixed) and the kernel unregisters these pages when the
> request completes. Non-page-backed args (eg op out headers) will go
> through the payload buffer as normal. The server can specify which open
> files should have their reads/writes go through zero-copy, by setting
> the FOPEN_IO_URING_ZERO_COPY flag when servicing opens.
>
> This requires CAP_SYS_ADMIN and bufpools. This is gated behind
> CAP_SYS_ADMIN because zero-copy allows the server direct access to the
> client's underlying pages, rather than operating on an intermediary
> buffer that the contents of the client's pages were copied into or on
> page cache folios.
>
> The request flow for the zero-copy direct-io write path (client writes
> data, server reads it) is as follows:
> =======================================================================
> | Kernel | FUSE server
> | |
> | "write(fd, buf, 1MB)" |
> | |
> | >sys_write() |
> | >fuse_file_write_iter() |
> | >fuse_send_one() |
> | [req->args->in_pages = true] |
> | [folios hold client write data] |
> | |
> | >fuse_uring_copy_to_ring() |
> | >copy_header_to_ring(IN_OUT) |
> | [memcpy fuse_in_header] |
> | >copy_header_to_ring(OP) |
> | [memcpy write_in header] |
> | |
> | >fuse_uring_args_to_ring() |
> | >setup_fuse_copy_state() |
> | [skip_folio_copy = true] |
> | |
> | >fuse_uring_set_up_zero_copy() |
> | [folio_get for each client folio] |
> | [build bio_vec array from folios] |
> | >io_buffer_register_bvec() |
> | [register pages at
> ent->zero_copy_index] |
> | [ent->zero_copied = true] |
> | |
> | >fuse_copy_args() |
> | [skip_folio_copy => return 0 |
> | for page arg, skip data copy] |
> | |
> | >copy_header_to_ring(RING_ENT) |
> | [memcpy ent_in_out] |
> | >io_uring_cmd_done() |
> | |
> | | [CQE received]
> | |
> | | [issue io_uring READ at
> | | ent->zero_copy_index]
> | | [reads directly from
> | |client's pages (ZERO_COPY)]
> | |
> | | [write data to backing
> | | store]
> | | [submit COMMIT AND FETCH]
> | |
> | >fuse_uring_commit_fetch() |
> | >fuse_uring_commit() |
> | >fuse_uring_copy_from_ring() |
> | >fuse_uring_req_end() |
> | >io_buffer_unregister(ent->zero_copy_index) |
> | [unregister pages from index] |
> | >fuse_zero_copy_release() |
> | [folio_put for each folio] |
> | [ent->zero_copied = false] |
> | >fuse_request_end() |
> | [wake up client] |
>
> The zero-copy read path is analogous.
>
> Some requests may have both page-backed args and non-page-backed args.
> For these requests, the page-backed args are zero-copied while the
> non-page-backed args are copied to the buffer selected from the buffer
> pool:
> zero-copy: pages registered via io_buffer_register_bvec()
> non-page-backed: copied to payload buffer via fuse_copy_args()
>
> For a request whose payload is zero-copied, the
> registration/unregistration path looks like:
>
> register: fuse_uring_set_up_zero_copy()
> folio_get() for each folio
> io_buffer_register_bvec(ent->zero_copy_index)
>
> unregister: fuse_uring_req_end()
> io_buffer_unregister(ent->zero_copy_index)
> -> fuse_zero_copy_release() callback
> folio_put() for each folio
>
> Please note that on abort for in-flight zero-copied requests that have
> been sent to userspace, the registered bvec slot remains occupied and
> its folios remain pinned until the io-uring ring is destroyed, at which
> point io-uring unregisters all buffers and the fuse_zero_copy_release()
> callback drops the folio references. Unregistering at teardown would
> require operating on the ring context directly, whose validity is hard
> to ascertain; this is deemed not worth the complexity for the abort
> race, since everything is freed when the ring is torn down.
>
> The throughput improvement from zero-copy depends on how much of the
> per-request latency is spent on data copying vs backing I/O. The gain
> comes from eliminating the payload-buffer memcpy, but accessing the
> zero-copied pages requires the server to issue the read/write as an
> IORING_OP_READ/WRITE_FIXED operation. The benefit is largest when the
> mempcy is a meaningful fraction of per-request latency while backing i/o
> is still noticable enough that the extra io-uring op's overhead doesn't
> dominate.
>
> Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
> 2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
> engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
> direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
> I/O is not the bottleneck):
>
> baseline registered-buf zero-copy (zc vs base)
> direct read ~5.1 GB/s ~5.4 GB/s ~8.9 GB/s (+75%)
> direct write ~3.4 GB/s ~4.8 GB/s ~5.1 GB/s (+50%)
>
> Reads end up higher than writes because the backing store reads faster
> than it writes (the baseline shows the same read>write gap, and the raw
> device does too). On a device-bound NVMe (~2 GB/s reads) the read gain
> shrinks to ~10-16% (and no measurable gains for writes), as backing I/O
> rather than the eliminated copy dominates latency.
>
> The benefit overall scales with how much of the
> per-request latency is the data copy versus backing I/O.
>
> Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
> ---
> fs/fuse/args.h | 2 +
> fs/fuse/dev.c | 24 ++++-
> fs/fuse/dev_uring.c | 182 +++++++++++++++++++++++++++++++++++---
> fs/fuse/dev_uring_i.h | 6 ++
> fs/fuse/file.c | 2 +
> fs/fuse/fuse_dev_i.h | 2 +
> include/uapi/linux/fuse.h | 34 +++++++
> 7 files changed, 236 insertions(+), 16 deletions(-)
>
> diff --git a/fs/fuse/args.h b/fs/fuse/args.h
> index ecfe51a192af..5173264a1261 100644
> --- a/fs/fuse/args.h
> +++ b/fs/fuse/args.h
> @@ -42,6 +42,8 @@ struct fuse_args {
> bool is_pinned:1;
> bool invalidate_vmap:1;
> bool abort_on_kill:1;
> + /* server requested io-uring zero-copy for this op */
> + bool zero_copy:1;
> struct fuse_in_arg in_args[4];
> struct fuse_arg out_args[2];
> void (*end)(struct fuse_args *args, int error);
> diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
> index d8f97943e973..90dceb7da571 100644
> --- a/fs/fuse/dev.c
> +++ b/fs/fuse/dev.c
> @@ -1248,11 +1248,25 @@ int fuse_copy_folio(struct fuse_copy_state *cs, struct folio **foliop,
>
> if (folio) {
> size = folio_size(folio);
> - if (zeroing && count < size)
> - folio_zero_range(folio, 0, size);
> + if (zeroing && count < size) {
> + /*
> + * When the copy is skipped the folio already holds the
> + * payload, so only the bytes outside [offset, offset +
> + * count) may be zeroed.
> + *
> + * Otherwise, the whole folio is cleared first so that a
> + * failed copy leaves zeros rather than stale folio
> + * contents.
> + */
> + if (cs->skip_folio_copy)
> + folio_zero_segments(folio, 0, offset,
> + offset + count, size);
> + else
> + folio_zero_range(folio, 0, size);
> + }
> }
>
> - while (count) {
> + while (!cs->skip_folio_copy && count) {
> if (cs->write && cs->pipebufs && folio) {
> /*
> * Can't control lifetime of pipe buffers, so always
> @@ -1345,6 +1359,10 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs,
> for (i = 0; !err && i < numargs; i++) {
> struct fuse_arg *arg = &args[i];
> if (i == numargs - 1 && argpages)
> + /*
> + * if cs->skip_folio_copy is set, this just does any
> + * needed zeroing. No copying is involved.
> + */
> err = fuse_copy_folios(cs, arg->size, zeroing);
> else
> err = fuse_copy_one(cs, arg->value, arg->size);
> diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
> index 17806da93039..1547a4f0d9b9 100644
> --- a/fs/fuse/dev_uring.c
> +++ b/fs/fuse/dev_uring.c
> @@ -22,6 +22,8 @@ MODULE_PARM_DESC(enable_uring,
> #define FUSE_URING_IOV_HEADERS 0
> #define FUSE_URING_IOV_PAYLOAD 1
>
> +#define FUSE_URING_ADD_QUEUE_FLAGS (FUSE_URING_ZERO_COPY)
> +
> bool fuse_uring_enabled(void)
> {
> return enable_uring;
> @@ -31,6 +33,11 @@ struct fuse_uring_pdu {
> struct fuse_ring_ent *ent;
> };
>
> +struct fuse_zero_copy_bvs {
> + unsigned int nr_bvs;
> + struct bio_vec bvs[];
> +};
> +
> static const struct fuse_iqueue_ops fuse_io_uring_ops;
>
> enum fuse_uring_header_type {
> @@ -113,8 +120,36 @@ static void fuse_uring_flush_bg(struct fuse_ring_queue *queue)
> }
> }
>
> +static bool can_zero_copy_req(struct fuse_ring_ent *ent, struct fuse_req *req)
> +{
> + struct fuse_args *args = req->args;
> +
> + if (!ent->queue->zero_copy || !args->zero_copy)
> + return false;
> +
> + if (args->opcode != FUSE_READ && args->opcode != FUSE_WRITE)
> + return false;
> +
> + return args->in_pages || args->out_pages;
> +}
> +
> +static void zero_copy_unregister(struct io_uring_cmd *cmd,
> + struct fuse_ring_ent *ent,
> + unsigned int issue_flags)
> +{
> + if (ent->zero_copied) {
> + int err = io_buffer_unregister(cmd, ent->zero_copy_index,
> + issue_flags);
> +
> + if (err)
> + pr_warn_ratelimited("qid=%d zero-copy unregister failed: %d\n",
> + ent->queue->qid, err);
> + ent->zero_copied = false;
> + }
> +}
> +
> static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req,
> - int error)
> + int error, unsigned int issue_flags)
> {
> struct fuse_ring_queue *queue = ent->queue;
> struct fuse_ring *ring = queue->ring;
> @@ -134,6 +169,8 @@ static void fuse_uring_req_end(struct fuse_ring_ent *ent, struct fuse_req *req,
>
> spin_unlock(&queue->lock);
>
> + zero_copy_unregister(ent->cmd, ent, issue_flags);
> +
> if (error)
> req->out.h.error = error;
>
> @@ -309,7 +346,7 @@ void fuse_uring_conn_init(struct fuse_chan *fch)
> }
>
> static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
> - int qid,
> + int qid, bool zero_copy,
> bool fail_if_exists)
> {
> struct fuse_chan *fch = ring->chan;
> @@ -328,6 +365,7 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
> queue->qid = qid;
> queue->ring = ring;
> spin_lock_init(&queue->lock);
> + queue->zero_copy = zero_copy;
>
> INIT_LIST_HEAD(&queue->ent_avail_queue);
> INIT_LIST_HEAD(&queue->ent_commit_queue);
> @@ -713,6 +751,9 @@ static int setup_fuse_copy_state(struct fuse_copy_state *cs,
>
> fuse_copy_init(cs, dir == ITER_DEST, iter);
>
> + if (ent->zero_copied)
> + cs->skip_folio_copy = true;
> +
> cs->is_uring = true;
> cs->req = req;
>
> @@ -744,6 +785,62 @@ static int fuse_uring_copy_from_ring(struct fuse_req *req,
> return err;
> }
>
> +static void fuse_zero_copy_release(void *priv)
> +{
> + struct fuse_zero_copy_bvs *zc_bvs = priv;
> + unsigned int i;
> +
> + for (i = 0; i < zc_bvs->nr_bvs; i++)
> + folio_put(page_folio(zc_bvs->bvs[i].bv_page));
> +
> + kvfree(zc_bvs);
> +}
> +
> +static int fuse_uring_set_up_zero_copy(struct fuse_ring_ent *ent,
> + struct fuse_req *req,
> + unsigned int issue_flags)
> +{
> + struct fuse_args_pages *ap;
> + int err, i, ddir = 0;
> + struct fuse_zero_copy_bvs *zc_bvs;
> + struct bio_vec *bvs;
> +
> + /* out_pages indicates a read, in_pages indicates a write */
> + if (req->args->out_pages)
> + ddir |= IO_BUF_DEST;
> + if (req->args->in_pages)
> + ddir |= IO_BUF_SOURCE;
> +
> + ap = container_of(req->args, typeof(*ap), args);
> +
> + zc_bvs = kvmalloc_flex(*zc_bvs, bvs, ap->num_folios,
> + GFP_KERNEL_ACCOUNT);
> + if (!zc_bvs)
> + return -ENOMEM;
> +
> + zc_bvs->nr_bvs = ap->num_folios;
> + bvs = zc_bvs->bvs;
> + for (i = 0; i < ap->num_folios; i++) {
> + bvs[i].bv_page = folio_page(ap->folios[i], 0);
> + bvs[i].bv_offset = ap->descs[i].offset;
> + bvs[i].bv_len = ap->descs[i].length;
> + folio_get(ap->folios[i]);
> + }
> +
> + err = io_buffer_register_bvec(ent->cmd, bvs, ap->num_folios,
> + fuse_zero_copy_release, zc_bvs,
> + ddir, ent->zero_copy_index,
> + issue_flags);
> + if (err) {
> + fuse_zero_copy_release(zc_bvs);
> + return err;
> + }
> +
> + ent->zero_copied = true;
> +
> + return 0;
> +}
> +
> /*
> * Copy data from the req to the ring buffer
> */
> @@ -762,6 +859,13 @@ static int fuse_uring_args_to_ring(struct fuse_req *req,
> .commit_id = req->in.h.unique,
> };
>
> + if (can_zero_copy_req(ent, req)) {
> + ent_in_out.flags |= FUSE_URING_ENT_ZERO_COPY;
> + err = fuse_uring_set_up_zero_copy(ent, req, issue_flags);
> + if (err)
> + return err;
> + }
> +
> err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter,
> issue_flags);
> if (err)
> @@ -793,6 +897,18 @@ static int fuse_uring_args_to_ring(struct fuse_req *req,
> }
>
> ent_in_out.payload_sz = cs.ring.copied_sz;
> + /*
> + * on a zero-copied write the pages are registered for the server to
> + * read via a fixed-buffer op rather than copied into the payload
> + * buffer, so copied_sz does not account for it. The server still needs
> + * the total inbound size to know how many bytes to read from the
> + * registered buffer, so add the page arg (always the last in-arg) back
> + * in
> + */
> + if (cs.skip_folio_copy && args->in_pages)
> + ent_in_out.payload_sz +=
> + args->in_args[args->in_numargs - 1].size;
> +
> if (bufpool_enabled(ent->queue) && ent->payload.iov_base)
> ent_in_out.offset =
> (uintptr_t)ent->payload.iov_base - ent->queue->bufpool->base_uaddr;
> @@ -831,11 +947,25 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
> sizeof(req->in.h));
> }
>
> -static bool fuse_uring_req_has_payload(struct fuse_req *req)
> +static bool fuse_uring_req_has_copyable_payload(struct fuse_ring_ent *ent,
> + struct fuse_req *req)
I notice a bit late, the new name is a bit confusing, the previous name
was easier to read for the intent, but we can update that any time later.
> {
> struct fuse_args *args = req->args;
>
> - return args->in_numargs > 1 || args->out_numargs;
> + if (!can_zero_copy_req(ent, req))
> + return args->in_numargs > 1 || args->out_numargs;
> +
> + /*
> + * the asymmetry between in_numargs > 2 and out_numargs > 1 is because
> + * the per-op header is extracted before fuse_copy_args() for inargs but
> + * not for outargs
> + */
Also spotted too late and can updated later, I think it should start
with "conditions that don't allow io-uring zero-copy".
> + if ((args->in_numargs > 1) && (!args->in_pages || args->in_numargs > 2))
> + return true;
> + if (args->out_numargs && (!args->out_pages || args->out_numargs > 1))
> + return true;
> +
> + return false;
> }
>
> static int fuse_uring_select_buffer(struct fuse_ring_ent *ent)
> @@ -892,7 +1022,7 @@ static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent,
> return 0;
>
> buffer_selected = !!ent->payload.iov_base;
> - has_payload = fuse_uring_req_has_payload(req);
> + has_payload = fuse_uring_req_has_copyable_payload(ent, req);
>
> if (has_payload && !buffer_selected)
> return fuse_uring_select_buffer(ent);
> @@ -910,7 +1040,7 @@ static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
> return 0;
>
> /* no payload to copy, can skip selecting a buffer */
> - if (!fuse_uring_req_has_payload(req))
> + if (!fuse_uring_req_has_copyable_payload(ent, req))
> return 0;
>
> return fuse_uring_select_buffer(ent);
> @@ -936,7 +1066,7 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
> ent->state = FRRS_INVALID;
> spin_unlock(&ent->queue->lock);
>
> - fuse_uring_req_end(ent, req, err);
> + fuse_uring_req_end(ent, req, err, issue_flags);
> }
>
> return err;
> @@ -1035,7 +1165,7 @@ static void fuse_uring_commit(struct fuse_ring_ent *ent, struct fuse_req *req,
>
> err = fuse_uring_copy_from_ring(req, ent, issue_flags);
> out:
> - fuse_uring_req_end(ent, req, err);
> + fuse_uring_req_end(ent, req, err, issue_flags);
> }
>
> /*
> @@ -1160,7 +1290,12 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
> queue->qid, commit_id, ent->state);
> fuse_uring_recycle_buffer(ent);
> spin_unlock(&queue->lock);
> - fuse_uring_req_end(ent, req, err);
> + /*
> + * Unregister any zero copyable pages since ent->cmd is null
> + * when it hits fuse_uring_req_end() in this path
> + */
> + zero_copy_unregister(cmd, ent, issue_flags);
> + fuse_uring_req_end(ent, req, err, issue_flags);
> return err;
> }
>
> @@ -1284,10 +1419,14 @@ static struct fuse_ring_ent *
> fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
> struct fuse_ring_queue *queue)
> {
> + const struct fuse_uring_cmd_req *cmd_req =
> + io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
> struct fuse_ring *ring = queue->ring;
> struct fuse_ring_ent *ent;
> struct iovec iov[FUSE_URING_IOV_SEGS];
> struct iovec *headers, *payload;
> + unsigned int zero_copy_index;
> +
> int err;
>
> err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);
> @@ -1297,6 +1436,10 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
> return ERR_PTR(err);
> }
>
> + zero_copy_index = READ_ONCE(cmd_req->ent_zero_copy_buf_index);
> + if (zero_copy_index && !queue->zero_copy)
> + return ERR_PTR(-EINVAL);
> +
> err = -EINVAL;
> headers = &iov[FUSE_URING_IOV_HEADERS];
> if (headers->iov_len < sizeof(struct fuse_uring_req_header)) {
> @@ -1315,9 +1458,14 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
> }
> } else {
> if (payload->iov_len < ring->max_payload_sz) {
> + spin_unlock(&queue->lock);
> pr_info_ratelimited("Invalid req payload len %zu\n",
> payload->iov_len);
> + return ERR_PTR(err);
> + }
> + if (queue->zero_copy) {
> spin_unlock(&queue->lock);
> + pr_info_ratelimited("Can only use zero copy with bufpools\n");
> return ERR_PTR(err);
> }
> queue->payload_mode = FUSE_PAYLOAD_PER_ENT;
> @@ -1335,6 +1483,7 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
> ent->headers = headers->iov_base;
> if (queue->payload_mode == FUSE_PAYLOAD_PER_ENT)
> ent->payload = *payload;
> + ent->zero_copy_index = zero_copy_index;
>
> atomic_inc(&ring->queue_refs);
> return ent;
> @@ -1364,7 +1513,7 @@ static int fuse_uring_register(struct io_uring_cmd *cmd,
>
> queue = READ_ONCE(ring->queues[qid]);
> if (!queue) {
> - queue = fuse_uring_create_queue(ring, qid, false);
> + queue = fuse_uring_create_queue(ring, qid, false, false);
> if (IS_ERR(queue))
> return PTR_ERR(queue);
> }
> @@ -1389,8 +1538,9 @@ static int fuse_uring_add_queue(struct io_uring_cmd *cmd, struct fuse_chan *fch)
> unsigned int qid = READ_ONCE(cmd_req->qid);
> uint64_t flags = READ_ONCE(cmd_req->flags);
> struct fuse_ring_queue *queue;
> + bool zero_copy = flags & FUSE_URING_ZERO_COPY;
>
> - if (!ring || flags)
> + if (!ring)
> return -EINVAL;
>
> if (qid >= ring->nr_queues) {
> @@ -1398,7 +1548,13 @@ static int fuse_uring_add_queue(struct io_uring_cmd *cmd, struct fuse_chan *fch)
> return -EINVAL;
> }
>
> - queue = fuse_uring_create_queue(ring, qid, true);
> + if (flags & ~FUSE_URING_ADD_QUEUE_FLAGS)
> + return -EINVAL;
> +
> + if (zero_copy && !capable(CAP_SYS_ADMIN))
> + return -EPERM;
> +
> + queue = fuse_uring_create_queue(ring, qid, zero_copy, true);
> if (IS_ERR(queue))
> return PTR_ERR(queue);
>
> @@ -1595,7 +1751,7 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)
>
> io_uring_cmd_done(cmd, err, issue_flags);
>
> - fuse_uring_req_end(ent, ent->fuse_req, err);
> + fuse_uring_req_end(ent, ent->fuse_req, err, issue_flags);
> kfree(ent);
> if (atomic_dec_and_test(&queue->ring->queue_refs))
> wake_up_all(&queue->ring->stop_waitq);
> diff --git a/fs/fuse/dev_uring_i.h b/fs/fuse/dev_uring_i.h
> index e142cae43022..263d0f8b9714 100644
> --- a/fs/fuse/dev_uring_i.h
> +++ b/fs/fuse/dev_uring_i.h
> @@ -79,6 +79,10 @@ struct fuse_ring_ent {
> /* buffer id in the pool, if bufpools are used. ignored otherwise */
> unsigned int buf_id;
>
> + /* true if the request's pages are being zero-copied */
> + bool zero_copied;
> + unsigned int zero_copy_index;
> +
> /* the ring queue that owns the request */
> struct fuse_ring_queue *queue;
>
> @@ -142,6 +146,8 @@ struct fuse_ring_queue {
>
> /* only allocated when payload_mode == FUSE_PAYLOAD_BUFPOOL */
> struct fuse_bufpool *bufpool;
> +
> + bool zero_copy;
> };
>
> /*
> diff --git a/fs/fuse/file.c b/fs/fuse/file.c
> index da5859e8159d..7b883cf170ac 100644
> --- a/fs/fuse/file.c
> +++ b/fs/fuse/file.c
> @@ -605,6 +605,7 @@ void fuse_read_args_fill(struct fuse_io_args *ia, struct file *file, loff_t pos,
> args->out_argvar = true;
> args->out_numargs = 1;
> args->out_args[0].size = count;
> + args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY;
> }
>
> static void fuse_release_user_pages(struct fuse_args_pages *ap, ssize_t nres,
> @@ -1151,6 +1152,7 @@ static void fuse_write_args_fill(struct fuse_io_args *ia, struct fuse_file *ff,
> args->out_numargs = 1;
> args->out_args[0].size = sizeof(ia->write.out);
> args->out_args[0].value = &ia->write.out;
> + args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY;
> }
>
> static unsigned int fuse_write_flags(struct kiocb *iocb)
> diff --git a/fs/fuse/fuse_dev_i.h b/fs/fuse/fuse_dev_i.h
> index 668c8391d61c..f41749f484df 100644
> --- a/fs/fuse/fuse_dev_i.h
> +++ b/fs/fuse/fuse_dev_i.h
> @@ -325,6 +325,8 @@ struct fuse_copy_state {
> bool write:1;
> bool move_folios:1;
> bool is_uring:1;
> + /* set when the payload is zero-copied. folios are filled in place */
> + bool skip_folio_copy:1;
> struct {
> unsigned int copied_sz; /* copied size into the user buffer */
> } ring;
> diff --git a/include/uapi/linux/fuse.h b/include/uapi/linux/fuse.h
> index 538d844da099..7435e09c87fe 100644
> --- a/include/uapi/linux/fuse.h
> +++ b/include/uapi/linux/fuse.h
> @@ -246,6 +246,8 @@
> * - add FUSE_HAS_IO_URING_BUFPOOL
> * - add fuse_uring_cmd_req bufpool struct
> * - add bufpool offset field to fuse_uring_ent_in_out struct
> + * - add FUSE_URING_ZERO_COPY, FUSE_URING_ENT_ZERO_COPY, and
> + * FOPEN_IO_URING_ZERO_COPY flag
> */
>
> #ifndef _LINUX_FUSE_H
> @@ -389,6 +391,12 @@ struct fuse_file_lock {
> * FOPEN_NOFLUSH: don't flush data cache on close (unless FUSE_WRITEBACK_CACHE)
> * FOPEN_PARALLEL_DIRECT_WRITES: Allow concurrent direct writes on the same inode
> * FOPEN_PASSTHROUGH: passthrough read/write io for this open file
> + * FOPEN_IO_URING_ZERO_COPY: use io-uring zero-copy for reads/writes on this
> + * open file. Honored only when the serving io-uring
> + * queue was set up for zero-copy
> + * (FUSE_URING_ZERO_COPY) and the request carries page
> + * payload. Otherwise reads/writes fall back to
> + * copying.
> */
> #define FOPEN_DIRECT_IO (1 << 0)
> #define FOPEN_KEEP_CACHE (1 << 1)
> @@ -398,6 +406,7 @@ struct fuse_file_lock {
> #define FOPEN_NOFLUSH (1 << 5)
> #define FOPEN_PARALLEL_DIRECT_WRITES (1 << 6)
> #define FOPEN_PASSTHROUGH (1 << 7)
> +#define FOPEN_IO_URING_ZERO_COPY (1 << 8)
>
> /**
> * INIT request/reply flags
> @@ -1259,6 +1268,13 @@ struct fuse_supp_groups {
> #define FUSE_URING_IN_OUT_HEADER_SZ 128
> #define FUSE_URING_OP_IN_OUT_SZ 128
>
> +/**
> + * fuse_uring_ent_in_out flags
> + *
> + * FUSE_URING_ENT_ZERO_COPY: Set if the ent's payload is zero-copied
> + */
> +#define FUSE_URING_ENT_ZERO_COPY (1 << 0)
> +
> /* Used as part of the fuse_uring_req_header */
> struct fuse_uring_ent_in_out {
> uint64_t flags;
> @@ -1310,6 +1326,14 @@ enum fuse_uring_cmd {
> FUSE_IO_URING_CMD_ADD_BUFPOOL = 4,
> };
>
> +/*
> + * fuse_uring_cmd_req flags for FUSE_IO_URING_CMD_ADD_QUEUE
> + *
> + * FUSE_URING_ZERO_COPY is only supported for queues with bufpools on privileged
> + * servers
> + */
> +#define FUSE_URING_ZERO_COPY (1 << 0)
> +
> /**
> * In the 80B command area of the SQE.
> */
> @@ -1330,6 +1354,16 @@ struct fuse_uring_cmd_req {
> uint32_t len;
> uint32_t reserved;
> } bufpool;
> +
> + /*
> + * Index of this entry's slot in the server's io_uring
> + * registered buffer table, where the kernel registers the
> + * request's pages for zero-copy. Set for
> + * FUSE_IO_URING_CMD_REGISTER cmds only, and only on queues
> + * created with FUSE_URING_ZERO_COPY. On a non-zero-copy queue
> + * this must be 0
> + */
> + uint16_t ent_zero_copy_buf_index;
> };
> };
>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy
2026-08-14 18:59 [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
` (6 preceding siblings ...)
2026-08-14 19:23 ` [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
@ 2026-08-17 15:29 ` Miklos Szeredi
2026-08-17 18:23 ` Jens Axboe
7 siblings, 1 reply; 27+ messages in thread
From: Miklos Szeredi @ 2026-08-17 15:29 UTC (permalink / raw)
To: Joanne Koong; +Cc: jlayton, libaokun, axboe, bernd, amir73il, fuse-devel
On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>
> This series implements fuse io-uring buffer pools and zero-copy.
Applied.
I cherry picked the io-uring patches, since they don't seem to be
bound for -next.
Thanks,
Miklos
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy
2026-08-17 15:29 ` Miklos Szeredi
@ 2026-08-17 18:23 ` Jens Axboe
0 siblings, 0 replies; 27+ messages in thread
From: Jens Axboe @ 2026-08-17 18:23 UTC (permalink / raw)
To: Miklos Szeredi, Joanne Koong
Cc: jlayton, libaokun, bernd, amir73il, fuse-devel
On 8/17/26 9:29 AM, Miklos Szeredi wrote:
> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>>
>> This series implements fuse io-uring buffer pools and zero-copy.
>
> Applied.
>
> I cherry picked the io-uring patches, since they don't seem to be
> bound for -next.
Right, I had them staged for 7.2 for Joanne, but then wasn't fully aware
(and gone a lot recently) if the plan was to shove it into 7.3 Hence I
never did a new branch for that, picking them from the 7.2 branch is
fine.
--
Jens Axboe
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-14 18:59 ` [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration Joanne Koong
@ 2026-08-19 11:34 ` Miklos Szeredi
2026-08-19 11:38 ` Bernd Schubert
2026-08-19 17:56 ` Joanne Koong
0 siblings, 2 replies; 27+ messages in thread
From: Miklos Szeredi @ 2026-08-19 11:34 UTC (permalink / raw)
To: Joanne Koong; +Cc: jlayton, libaokun, axboe, bernd, amir73il, fuse-devel
[-- Attachment #1: Type: text/plain, Size: 1195 bytes --]
On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>
> Currently, the connection's fuse_ring is created lazily on the first
> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
> thread per queue (one per CPU) and those threads issue their first
> REGISTER command concurrently. They then race to create the single
> per-connection fuse_ring, which required open-coded handling in
> fuse_uring_create() to detect and protect against concurrent creations.
>
> Decouple fuse_ring creation from ent registration and move it to
> FUSE_INIT reply processing after a server has negotiated and set
> FUSE_OVER_IO_URING. The ring is published before the connection is
> marked initialized. fuse_uring_register() no longer creates the ring and
> it instead uses the ring set up at init time.
I tested this with loraw (a "raw" loopback tester that doesn't use
libfuse) and it fails with
root@kvm:~# ./loraw -u /mnt/fuse
loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
cqe->res is -22 (EINVAL).
Attaching the reproducer. To compile:
cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
gcc loraw.c -oloraw -luring
Thanks,
Miklos
[-- Attachment #2: loraw.c --]
[-- Type: text/x-csrc, Size: 27142 bytes --]
#define _GNU_SOURCE
#define LO_NOTHREAD 1
#include "fuse_kernel.h"
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#include <stdbool.h>
#include <string.h>
#include <limits.h>
#include <dirent.h>
#include <assert.h>
#include <errno.h>
#include <err.h>
#include <inttypes.h>
#include <sched.h>
#include <sys/file.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <sys/mount.h>
#include <sys/mman.h>
#include <pthread.h>
#include <semaphore.h>
#include <sys/sysinfo.h>
#include <sys/sysmacros.h>
#include <liburing.h>
/* returns -1 on error */
#define ER(_expr) \
({ typeof(_expr) _ret = (_expr); if (_ret == (typeof(_expr)) -1) err(1, #_expr); _ret; })
/* returns errno on error */
#define PE(_expr) \
({ typeof(_expr) _ret = (_expr); if (_ret != 0) { errno = _ret; err(1, #_expr); } _ret; })
/* returns -errno on error */
#define NE(_expr) \
({ typeof(_expr) _ret = (_expr); if (_ret < 0) { errno = -_ret; err(1, #_expr); } _ret; })
/* returns NULL on error */
#define NL(_expr) \
({ typeof(_expr) _ret = (_expr); if (_ret == NULL) { errx(1, #_expr " returned NULL"); } _ret; })
struct lo_inode {
struct lo_inode *next; /* protected by lo->mutex */
struct lo_inode *prev; /* protected by lo->mutex */
int fd;
int backing_id;
dev_t dev;
uint64_t refcount; /* protected by lo->mutex */
struct fuse_attr attr;
};
struct lo_file {
union {
struct lo_file *next;
int fd;
struct {
DIR *dp;
struct dirent *entry;
off_t offset;
};
};
};
struct lo_config {
int debug;
int single;
int bind;
int map;
uint64_t timeout;
const char *source;
int nothread;
int direct;
int uring;
int queue_depth;
size_t req_size;
int passthrough;
int passthrough2;
const char *mnt;
};
struct lo_data {
pthread_mutex_t mutex;
#ifdef LO_NOTHREAD
#define LO_INODE_MAX 65536
struct lo_inode inodes[LO_INODE_MAX];
struct lo_inode *free_inodes;
#define LO_FILE_MAX 65536
struct lo_file files[LO_FILE_MAX];
struct lo_file *free_files;
sem_t sem;
#endif
struct lo_config c;
struct lo_inode root;
int devfd;
int inited;
};
struct lo_chan {
int fd;
void *inbuf;
void *outbuf;
size_t bufsize;
};
struct lo_ring_req {
struct io_uring *ring;
int qid;
struct fuse_uring_req_header *rreq;
struct iovec iov[2];
uint64_t unique;
};
struct lo_req {
struct lo_data *lo;
int is_ch;
union {
struct lo_chan ch;
struct lo_ring_req rr;
};
};
#ifdef LO_NOTHREAD
static inline int lo_nothread(struct lo_data *lo)
{
return lo->c.nothread;
}
static inline void lo_mutex_init_nt(struct lo_data *lo)
{
sem_init(&lo->sem, 1, 1);
}
static inline void lo_mutex_lock_nt(struct lo_data *lo)
{
sem_wait(&lo->sem);
}
static inline void lo_mutex_unlock_nt(struct lo_data *lo)
{
sem_post(&lo->sem);
}
static inline struct lo_inode *lo_alloc_inode_nt(struct lo_data *lo)
{
struct lo_inode *inode;
lo_mutex_lock_nt(lo);
inode = lo->free_inodes;
if (inode)
lo->free_inodes = inode->next;
lo_mutex_unlock_nt(lo);
memset(inode, 0, sizeof(*inode));
return inode;
}
static inline struct lo_file *lo_alloc_file_nt(struct lo_data *lo)
{
struct lo_file *lf;
lo_mutex_lock_nt(lo);
lf = lo->free_files;
if (lf)
lo->free_files = lf->next;
lo_mutex_unlock_nt(lo);
memset(lf, 0, sizeof(*lf));
return lf;
}
static inline void lo_free_inode_locked_nt(struct lo_data *lo,
struct lo_inode *inode)
{
inode->next = lo->free_inodes;
lo->free_inodes = inode;
}
static inline void lo_free_file_locked_nt(struct lo_data *lo,
struct lo_file *lf)
{
lf->next = lo->free_files;
lo->free_files = lf;
}
static inline void lo_free_inode_nt(struct lo_data *lo, struct lo_inode *inode)
{
lo_mutex_lock_nt(lo);
lo_free_inode_locked_nt(lo, inode);
lo_mutex_unlock_nt(lo);
}
static inline void lo_free_file_nt(struct lo_data *lo, struct lo_file *lf)
{
lo_mutex_lock_nt(lo);
lo_free_file_locked_nt(lo, lf);
lo_mutex_unlock_nt(lo);
}
static inline struct lo_data *lo_alloc_lo_nt(void)
{
struct lo_data *lo;
unsigned int i;
lo = ER(mmap(NULL, sizeof(struct lo_data), PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_ANONYMOUS, -1, 0));
for (i = 0; i < LO_INODE_MAX; i++)
lo_free_inode_locked_nt(lo, &lo->inodes[i]);
for (i = 0; i < LO_FILE_MAX; i++)
lo_free_file_locked_nt(lo, &lo->files[i]);
return lo;
}
#else
#define lo_nothread(lo) ((void) lo, 0)
#define lo_alloc_inode_nt(lo) NULL
#define lo_alloc_file_nt(lo) NULL
#define lo_free_inode_nt(lo, inode) abort()
#define lo_free_file_nt(lo, lf) abort()
#define lo_alloc_lo_nt() NULL
#define lo_mutex_init_nt(lo) abort()
#define lo_mutex_lock_nt(lo) abort()
#define lo_mutex_unlock_nt(lo) abort()
#endif
static void lo_mutex_init(struct lo_data *lo)
{
if (!lo_nothread(lo))
pthread_mutex_init(&lo->mutex, NULL);
else
lo_mutex_init_nt(lo);
}
static void lo_mutex_lock(struct lo_data *lo)
{
if (!lo_nothread(lo))
pthread_mutex_lock(&lo->mutex);
else
lo_mutex_lock_nt(lo);
}
static void lo_mutex_unlock(struct lo_data *lo)
{
if (!lo_nothread(lo))
pthread_mutex_unlock(&lo->mutex);
else
lo_mutex_unlock_nt(lo);
}
static struct lo_inode *lo_alloc_inode(struct lo_data *lo)
{
if (!lo_nothread(lo))
return calloc(1, sizeof(struct lo_inode));
else
return lo_alloc_inode_nt(lo);
}
static struct lo_file *lo_alloc_file(struct lo_data *lo)
{
if (!lo_nothread(lo))
return calloc(1, sizeof(struct lo_file));
else
return lo_alloc_file_nt(lo);
}
static void lo_free_inode(struct lo_data *lo, struct lo_inode *inode)
{
if (!lo_nothread(lo))
free(inode);
else
lo_free_inode_nt(lo, inode);
}
static void lo_free_file(struct lo_data *lo, struct lo_file *lf)
{
if (!lo_nothread(lo))
free(lf);
else
lo_free_file_nt(lo, lf);
}
static struct lo_inode *lo_inode(struct lo_data *lo, uint64_t ino)
{
if (ino == FUSE_ROOT_ID)
return &lo->root;
else
return (struct lo_inode *) (uintptr_t) ino;
}
static int lo_debug(struct lo_req *req)
{
return req->lo->c.debug;
}
static void lo_reply_ch(struct lo_req *req, int error, size_t argsize)
{
struct lo_chan *lc = &req->ch;
struct fuse_in_header *inh = lc->inbuf;
struct fuse_out_header *outh = lc->outbuf;
outh->len = sizeof(struct fuse_out_header) + argsize;
outh->error = -error;
outh->unique = inh->unique;
ER(write(lc->fd, lc->outbuf, outh->len));
}
static void lo_queue_uring(struct lo_req *req, int cmd_op)
{
struct io_uring_sqe *sqe;
struct fuse_uring_cmd_req *ureq;
sqe = NL(io_uring_get_sqe(req->rr.ring));
sqe->opcode = IORING_OP_URING_CMD;
sqe->flags = IOSQE_FIXED_FILE;
sqe->fd = 0;
sqe->rw_flags = 0;
sqe->ioprio = 0;
sqe->off = 0;
sqe->cmd_op = cmd_op;
sqe->__pad1 = 0;
ureq = (struct fuse_uring_cmd_req *) sqe->cmd;
ureq->qid = req->rr.qid;
ureq->commit_id = req->rr.rreq->ring_ent_in_out.commit_id;
ureq->flags = 0;
io_uring_sqe_set_data(sqe, req);
}
static void lo_reply_uring(struct lo_req *req, int error, size_t argsize)
{
struct fuse_uring_req_header *rreq = req->rr.rreq;
struct fuse_out_header *out = (struct fuse_out_header *)&rreq->in_out;
struct fuse_uring_ent_in_out *ent_in_out = &rreq->ring_ent_in_out;
ent_in_out->payload_sz = argsize;
out->len = sizeof(struct fuse_out_header) + argsize;
out->error = -error;
out->unique = req->rr.unique;
lo_queue_uring(req, FUSE_IO_URING_CMD_COMMIT_AND_FETCH);
NE(io_uring_submit(req->rr.ring));
}
static void lo_reply(struct lo_req *req, int error, size_t argsize)
{
if (lo_debug(req)) {
fprintf(stderr, " error: %i, outsize: %zu\n", error,
sizeof(struct fuse_out_header) + argsize);
}
if (req->is_ch)
lo_reply_ch(req, error, argsize);
else
lo_reply_uring(req, error, argsize);
}
static void *lo_out_arg(struct lo_req *req)
{
if (req->is_ch)
return ((struct fuse_out_header *) req->ch.outbuf) + 1;
else
return req->rr.iov[1].iov_base;
}
static bool lo_overflow(struct lo_req *req, size_t size)
{
if (req->is_ch)
return size > req->ch.bufsize - sizeof(struct fuse_out_header);
else
return size > req->rr.iov[1].iov_len;
}
static void lo_convert_stat(const struct statx *stat, struct fuse_attr *attr)
{
memset(attr, 0, sizeof(*attr));
attr->ino = stat->stx_ino;
attr->mode = stat->stx_mode;
attr->nlink = stat->stx_nlink;
attr->uid = stat->stx_uid;
attr->gid = stat->stx_gid;
attr->rdev = makedev(stat->stx_rdev_major, stat->stx_rdev_minor);
attr->size = stat->stx_size;
attr->blksize = stat->stx_blksize;
attr->blocks = stat->stx_blocks;
attr->atime = stat->stx_atime.tv_sec;
attr->mtime = stat->stx_mtime.tv_sec;
attr->ctime = stat->stx_ctime.tv_sec;
attr->atimensec = stat->stx_atime.tv_nsec;
attr->mtimensec = stat->stx_mtime.tv_nsec;
attr->ctimensec = stat->stx_ctime.tv_nsec;
}
static void lo_getattr(struct lo_req *req, struct fuse_in_header *inh,
struct fuse_getattr_in *inarg)
{
struct lo_data *lo = req->lo;
struct fuse_attr_out *outarg = lo_out_arg(req);
(void) inarg;
if (lo_debug(req))
fprintf(stderr, "lo_getattr(ino=%"PRIu64")\n", inh->nodeid);
outarg->attr_valid = lo->c.timeout;
outarg->attr_valid_nsec = 0;
outarg->dummy = 0;
outarg->attr = lo_inode(lo, inh->nodeid)->attr;
lo_reply(req, 0, sizeof(*outarg));
}
static struct lo_inode *lo_find(struct lo_data *lo, struct statx *st)
{
struct lo_inode *p;
struct lo_inode *ret = NULL;
lo_mutex_lock(lo);
for (p = lo->root.next; p != &lo->root; p = p->next) {
if (p->attr.ino == st->stx_ino && p->dev == makedev(st->stx_dev_major, st->stx_dev_minor)) {
assert(p->refcount > 0);
ret = p;
ret->refcount++;
break;
}
}
lo_mutex_unlock(lo);
return ret;
}
static void lo_lookup(struct lo_req *req, struct fuse_in_header *inh,
char *name)
{
struct lo_data *lo = req->lo;
struct lo_inode *inode, *parent = lo_inode(lo, inh->nodeid);
struct fuse_entry_out *outarg = lo_out_arg(req);
struct statx stat;
int newfd, res, saverr;
if (lo_debug(req)) {
fprintf(stderr, " lo_lookup(parent=%"PRIu64", name=%s)\n",
inh->nodeid, name);
}
newfd = openat(parent->fd, name, O_PATH | O_NOFOLLOW);
if (newfd == -1)
goto out_err;
res = statx(newfd, "", AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW, STATX_BASIC_STATS, &stat);
if (res == -1)
goto out_err;
inode = lo_find(lo, &stat);
if (inode) {
close(newfd);
newfd = -1;
} else {
struct lo_inode *prev, *next;
saverr = ENOMEM;
inode = lo_alloc_inode(lo);
if (!inode)
goto out_err;
inode->refcount = 1;
inode->fd = newfd;
inode->dev = makedev(stat.stx_dev_major, stat.stx_dev_minor);
lo_convert_stat(&stat, &inode->attr);
lo_mutex_lock(lo);
prev = &lo->root;
next = prev->next;
next->prev = inode;
inode->next = next;
inode->prev = prev;
prev->next = inode;
lo_mutex_unlock(lo);
}
memset(outarg, 0, sizeof(*outarg));
outarg->nodeid = (uintptr_t) inode;
outarg->entry_valid = lo->c.timeout;
outarg->attr_valid = lo->c.timeout;
outarg->attr = inode->attr;
#if 0
/* FIXME: name gets overwritten with fuse_uring */
if (lo_debug(req)) {
fprintf(stderr, " %"PRIu64"/%s -> %"PRIu64"\n",
inh->nodeid, name, outarg->nodeid);
}
#endif
lo_reply(req, 0, sizeof(*outarg));
return;
out_err:
saverr = errno;
if (newfd != -1)
close(newfd);
lo_reply(req, saverr, 0);
}
static int lo_backing_open(struct lo_data *lo, int fd)
{
struct fuse_backing_map map = { .fd = fd };
int backing_id;
backing_id = ER(ioctl(lo->devfd, FUSE_DEV_IOC_BACKING_OPEN, &map));
if (lo->c.debug)
fprintf(stderr, "backing_open(%i) = %i\n", fd, backing_id);
return backing_id;
}
static void lo_backing_close(struct lo_data *lo, int backing_id)
{
if (lo->c.debug)
fprintf(stderr, "backing_close(%i)\n", backing_id);
ER(ioctl(lo->devfd, FUSE_DEV_IOC_BACKING_CLOSE, &backing_id));
}
static void lo_open(struct lo_req *req, struct fuse_in_header *inh,
struct fuse_open_in *inarg)
{
struct lo_data *lo = req->lo;
struct lo_inode *inode = lo_inode(lo, inh->nodeid);
struct fuse_open_out *outarg = lo_out_arg(req);
struct lo_file *lf;
char buf[64];
int fd;
int o_direct = lo->c.direct ? O_DIRECT : 0;
if (lo->c.passthrough) {
if (!inode->backing_id)
inode->backing_id = lo_backing_open(lo, inode->fd);
memset(outarg, 0, sizeof(*outarg));
outarg->open_flags = FOPEN_PASSTHROUGH;
outarg->backing_id = inode->backing_id;
lo_reply(req, 0, sizeof(*outarg));
return;
}
sprintf(buf, "/proc/self/fd/%i", inode->fd);
fd = open(buf, (inarg->flags & O_ACCMODE) | o_direct);
if (fd == -1) {
lo_reply(req, errno, 0);
return;
}
lf = NL(lo_alloc_file(lo));
memset(outarg, 0, sizeof(*outarg));
outarg->fh = (uintptr_t) lf;
outarg->open_flags = FOPEN_KEEP_CACHE;
if (lo->c.passthrough2) {
int backing_id = lo_backing_open(lo, fd);
outarg->open_flags = FOPEN_PASSTHROUGH;
outarg->backing_id = backing_id;
close(fd);
lf->fd = backing_id;
} else {
lf->fd = fd;
}
lo_reply(req, 0, sizeof(*outarg));
}
static struct lo_file *lo_file(uint64_t fh)
{
return (void *) (uintptr_t) fh;
}
static void lo_release(struct lo_req *req, struct fuse_in_header *inh,
struct fuse_release_in *inarg)
{
struct lo_file *lf = lo_file(inarg->fh);
(void) inh;
/* No lo_file for passthrough */
if (lf) {
if (req->lo->c.passthrough2)
lo_backing_close(req->lo, lf->fd);
else
close(lf->fd);
lo_free_file(req->lo, lf);
}
lo_reply(req, 0, 0);
}
static void lo_read(struct lo_req *req, struct fuse_in_header *inh,
struct fuse_read_in *inarg)
{
char *outarg = lo_out_arg(req);
struct lo_file *lf = lo_file(inarg->fh);
ssize_t res;
(void) inh;
if (lo_overflow(req, inarg->size)) {
lo_reply(req, EOVERFLOW, 0);
return;
}
res = pread(lf->fd, outarg, inarg->size, inarg->offset);
if (res == -1) {
lo_reply(req, errno, 0);
return;
}
lo_reply(req, 0, res);
}
static void lo_opendir(struct lo_req *req, struct fuse_in_header *inh,
struct fuse_open_in *inarg)
{
struct lo_data *lo = req->lo;
struct lo_inode *inode = lo_inode(lo, inh->nodeid);
struct fuse_open_out *outarg = lo_out_arg(req);
struct lo_file *lf;
int fd;
DIR *dp;
(void) inarg;
fd = openat(inode->fd, ".", O_RDONLY);
if (fd == -1) {
lo_reply(req, errno, 0);
return;
}
dp = fdopendir(fd);
if (dp == NULL) {
int saverr = errno;
close(fd);
lo_reply(req, saverr, 0);
return;
}
lf = NL(lo_alloc_file(lo));
memset(outarg, 0, sizeof(*outarg));
outarg->fh = (uintptr_t) lf;
outarg->open_flags = 0;
lf->dp = dp;
lo_reply(req, 0, sizeof(*outarg));
}
static void lo_releasedir(struct lo_req *req, struct fuse_in_header *inh,
struct fuse_release_in *inarg)
{
struct lo_file *lf = lo_file(inarg->fh);
(void) inh;
closedir(lf->dp);
lo_free_file(req->lo, lf);
lo_reply(req, 0, 0);
}
static void lo_readdir(struct lo_req *req, struct fuse_in_header *inh,
struct fuse_read_in *inarg)
{
void *p = lo_out_arg(req);
struct lo_file *lf = lo_file(inarg->fh);
size_t rem = inarg->size, orig_size = inarg->size;
int err = 0;
const char *name;
size_t namelen, entlen, entlen_padded;
struct fuse_dirent *dirent;
off_t nextoff;
(void) inh;
if (lo_overflow(req, inarg->size)) {
lo_reply(req, EOVERFLOW, 0);
return;
}
if ((off_t) inarg->offset != lf->offset) {
seekdir(lf->dp, inarg->offset);
lf->entry = NULL;
lf->offset = inarg->offset;
}
while (1) {
if (!lf->entry) {
errno = 0;
lf->entry = readdir(lf->dp);
if (!lf->entry) {
if (errno) {
err = errno;
break;
} else {
break;
}
}
}
nextoff = lf->entry->d_off;
name = lf->entry->d_name;
namelen = strlen(name);
entlen = FUSE_NAME_OFFSET + namelen;
entlen_padded = FUSE_DIRENT_ALIGN(entlen);
if (entlen_padded > rem)
break;
dirent = (struct fuse_dirent *) p;
dirent->ino = lf->entry->d_ino;
dirent->off = nextoff;
dirent->namelen = namelen;
dirent->type = lf->entry->d_type;
memcpy(dirent->name, name, namelen);
memset(dirent->name + namelen, 0, entlen_padded - entlen);
p += entlen_padded;
rem -= entlen_padded;
lf->entry = NULL;
lf->offset = nextoff;
}
if (err && rem == orig_size)
lo_reply(req, err, 0);
else
lo_reply(req, 0, orig_size - rem);
}
static void unref_inode(struct lo_data *lo, struct lo_inode *inode, uint64_t n)
{
if (!inode)
return;
lo_mutex_lock(lo);
assert(inode->refcount >= n);
inode->refcount -= n;
if (!inode->refcount) {
struct lo_inode *prev, *next;
prev = inode->prev;
next = inode->next;
next->prev = prev;
prev->next = next;
lo_mutex_unlock(lo);
if (inode->backing_id)
lo_backing_close(lo, inode->backing_id);
close(inode->fd);
lo_free_inode(lo, inode);
} else {
lo_mutex_unlock(lo);
}
}
static void lo_forget_one(struct lo_data *lo, uint64_t nodeid,
uint64_t nlookup)
{
struct lo_inode *inode = lo_inode(lo, nodeid);
if (lo->c.debug) {
fprintf(stderr, " forget %"PRIu64" %"PRIu64" -%"PRIu64"\n",
nodeid, inode->refcount, nlookup);
}
unref_inode(lo, inode, nlookup);
}
static void lo_forget(struct lo_data *lo, struct fuse_in_header *inh,
struct fuse_forget_in *inarg)
{
lo_forget_one(lo, inh->nodeid, inarg->nlookup);
}
static void lo_batch_forget(struct lo_data *lo, struct fuse_in_header *inh,
struct fuse_batch_forget_in *inarg,
struct fuse_forget_one *param)
{
unsigned int i;
(void) inh;
if (!param)
param = (void *) (inarg + 1);
for (i = 0; i < inarg->count; i++)
lo_forget_one(lo, param[i].nodeid, param[i].nlookup);
}
static void lo_init(struct lo_req *req, struct fuse_in_header *inh,
struct fuse_init_in *inarg)
{
struct fuse_init_out *outarg = lo_out_arg(req);
uint64_t inflags = inarg->flags;
uint64_t outflags;
(void) inh;
memset(outarg, 0, sizeof(*outarg));
if (inflags & FUSE_INIT_EXT)
inflags |= (uint64_t) inarg->flags2 << 32;
outflags = inflags & (FUSE_PARALLEL_DIROPS | FUSE_ASYNC_READ | FUSE_ASYNC_DIO | FUSE_INIT_EXT);
if (req->lo->c.passthrough || req->lo->c.passthrough2) {
if (!(inflags & FUSE_PASSTHROUGH))
errx(1, "passthrough mode not supported");
outflags |= FUSE_PASSTHROUGH;
outarg->max_stack_depth = 1;
}
if (req->lo->c.uring && !(inflags & FUSE_OVER_IO_URING))
errx(1, "uring mode not supported");
outarg->flags = outflags;
if (outflags & FUSE_INIT_EXT)
outarg->flags2 = outflags >> 32;
outarg->major = FUSE_KERNEL_VERSION;
outarg->minor = FUSE_KERNEL_MINOR_VERSION;
outarg->max_readahead = inarg->max_readahead;
req->lo->inited = 1;
lo_reply(req, 0, sizeof(*outarg));
}
static size_t lo_getreq(struct lo_chan *lc)
{
ssize_t res;
res = ER(read(lc->fd, lc->inbuf, lc->bufsize));
if ((size_t) res < sizeof(struct fuse_in_header))
errx(1, "short read from fuse device");
return res;
}
static void lo_process(struct lo_req *req, struct fuse_in_header *inh,
void *arg, void *payload, size_t len, struct io_uring_cqe *cqe)
{
if (lo_debug(req)) {
fprintf(stderr,
"%cunique: %"PRIu64", opcode: %i, nodeid: %"PRIu64", insize: %zu\n",
req->is_ch ? ' ' : cqe ? '.' : '*',
inh->unique, inh->opcode, inh->nodeid, len);
}
switch (inh->opcode) {
case FUSE_INIT:
assert(req->is_ch || !len);
lo_init(req, inh, arg);
break;
case FUSE_LOOKUP:
lo_lookup(req, inh, payload ? payload : arg);
break;
case FUSE_GETATTR:
assert(req->is_ch || !len);
lo_getattr(req, inh, arg);
break;
case FUSE_OPEN:
assert(req->is_ch || !len);
lo_open(req, inh, arg);
break;
case FUSE_RELEASE:
assert(req->is_ch || !len);
lo_release(req, inh, arg);
break;
case FUSE_READ:
assert(req->is_ch || !len);
lo_read(req, inh, arg);
break;
case FUSE_OPENDIR:
assert(req->is_ch || !len);
lo_opendir(req, inh, arg);
break;
case FUSE_RELEASEDIR:
assert(req->is_ch || !len);
lo_releasedir(req, inh, arg);
break;
case FUSE_READDIR:
assert(req->is_ch || !len);
lo_readdir(req, inh, arg);
break;
case FUSE_FORGET:
assert(req->is_ch || !len);
lo_forget(req->lo, inh, arg);
break;
case FUSE_BATCH_FORGET:
lo_batch_forget(req->lo, inh, arg, payload);
break;
default:
lo_reply(req, ENOSYS, 0);
}
}
static void lo_alloc_bufs(struct lo_chan *lc)
{
size_t outbuf_align = 0x20000;
size_t outbuf_allocsize = 0x60000;
size_t outbuf_offset = outbuf_align - sizeof(struct fuse_out_header);
lc->bufsize = 0x21000;
PE(posix_memalign(&lc->inbuf, 0x1000, lc->bufsize));
assert(outbuf_offset + lc->bufsize <= outbuf_allocsize);
PE(posix_memalign(&lc->outbuf, outbuf_align, outbuf_allocsize));
lc->outbuf += outbuf_offset;
}
struct lo_thread_data {
struct lo_data *lo;
int cpu;
};
static void lo_start_uring(struct lo_thread_data *ltd)
{
int fd, i;
struct lo_data *lo = ltd->lo;
struct io_uring ring;
struct lo_req *req;
fd = lo->devfd;
NE(io_uring_queue_init(lo->c.queue_depth, &ring, IORING_SETUP_SQE128));
NE(io_uring_register_files(&ring, &fd, 1));
for (i = 0; i < lo->c.queue_depth; i++) {
struct io_uring_sqe *sqe;
struct fuse_uring_cmd_req *ureq;
req = NL(calloc(1, sizeof(*req)));
req->lo = lo;
req->is_ch = 0;
req->rr.ring = &ring,
req->rr.qid = ltd->cpu;
/* Allocate header buffer (page-aligned) */
PE(posix_memalign((void **) &req->rr.rreq, 0x1000, sizeof(struct fuse_uring_req_header)));
/* Allocate payload buffer (page-aligned) */
req->rr.iov[1].iov_len = lo->c.req_size;
PE(posix_memalign((void **) &req->rr.iov[1].iov_base, 0x1000, req->rr.iov[1].iov_len));
if (lo->c.debug) {
fprintf(stderr, "NEW req=%p rreq=%p qid=%i\n",
req, req->rr.rreq, req->rr.qid);
}
sqe = NL(io_uring_get_sqe(&ring));
sqe->opcode = IORING_OP_URING_CMD;
sqe->flags = IOSQE_FIXED_FILE;
sqe->fd = 0;
sqe->cmd_op = FUSE_IO_URING_CMD_REGISTER;
req->rr.iov[0].iov_base = req->rr.rreq;
req->rr.iov[0].iov_len = sizeof(struct fuse_uring_req_header);
sqe->addr = (unsigned long) req->rr.iov;
sqe->len = 2;
ureq = (struct fuse_uring_cmd_req *) sqe->cmd;
ureq->qid = req->rr.qid;
io_uring_sqe_set_data(sqe, req);
}
NE(io_uring_submit(&ring));
for (;;) {
struct fuse_uring_req_header *rreq;
struct io_uring_cqe *cqe;
struct fuse_in_header *in;
struct fuse_uring_ent_in_out *ent_in_out;
NE(io_uring_wait_cqe(&ring, &cqe));
req = io_uring_cqe_get_data(cqe);
rreq = req->rr.rreq;
in = (struct fuse_in_header *)&rreq->in_out;
ent_in_out = &rreq->ring_ent_in_out;
if (lo->c.debug) {
fprintf(stderr, "CQE res=%d, commit_id=%lu, payload_sz=%u req=%p\n",
cqe->res, ent_in_out->commit_id, ent_in_out->payload_sz, req);
}
assert(!cqe->res);
req->rr.unique = in->unique;
lo_process(req, in, rreq->op_in, req->rr.iov[1].iov_base, ent_in_out->payload_sz, cqe);
io_uring_cqe_seen(req->rr.ring, cqe);
}
}
static void lo_loop(struct lo_data *lo, int fd)
{
struct lo_req req = {
.lo = lo,
.is_ch =1,
.ch.fd = fd,
};
lo_alloc_bufs(&req.ch);
while (1) {
struct fuse_in_header *inh = req.ch.inbuf;
void *arg = inh + 1;
size_t len;
len = lo_getreq(&req.ch);
lo_process(&req, inh, arg, NULL, len, NULL);
}
}
static void lo_process_init(struct lo_data *lo)
{
struct lo_req req = {
.lo = lo,
.is_ch = 1,
.ch.fd = lo->devfd,
};
struct fuse_in_header *inh;
void *arg;
size_t len;
lo_alloc_bufs(&req.ch);
inh = req.ch.inbuf;
arg = inh + 1;
len = lo_getreq(&req.ch);
lo_process(&req, inh, arg, NULL, len, NULL);
assert(lo->inited);
}
static void lo_start_common(struct lo_thread_data *ltd)
{
struct lo_data *lo = ltd->lo;
int devfd = lo->devfd;
int fd = devfd;
if (ltd->lo->c.uring) {
lo_start_uring(ltd);
return;
}
if (ltd->lo->c.bind) {
fd = ER(open("/dev/fuse", O_RDWR));
ER(ioctl(fd, FUSE_DEV_IOC_CLONE, &devfd));
}
lo_loop(lo, fd);
}
static void *lo_start_one(void *data)
{
struct lo_thread_data *ltd = data;
cpu_set_t one;
CPU_ZERO(&one);
CPU_SET(ltd->cpu, &one);
PE(pthread_setaffinity_np(pthread_self(), sizeof(one), &one));
lo_start_common(ltd);
return NULL;
}
static int lo_start_one_nt(void *data)
{
struct lo_thread_data *ltd = data;
cpu_set_t one;
CPU_ZERO(&one);
CPU_SET(ltd->cpu, &one);
ER(sched_setaffinity(0, sizeof(one), &one));
lo_start_common(data);
return 0;
}
static void lo_start_threads(struct lo_data *lo)
{
int i, n;
cpu_set_t set;
struct lo_thread_data *ltd;
ER(sched_getaffinity(0, sizeof(set), &set));
n = CPU_COUNT(&set);
for (i = 0; n && i < CPU_SETSIZE; i++) {
if (CPU_ISSET(i, &set)) {
ltd = NL(malloc(sizeof(*ltd)));
ltd->lo = lo;
ltd->cpu = i;
if (!lo_nothread(lo)) {
pthread_t id;
PE(pthread_create(&id, NULL, lo_start_one, ltd));
} else {
void *stack, *top;
size_t stack_size = 1048576;
stack = NL(malloc(stack_size));
top = stack + stack_size;
ER(clone(lo_start_one_nt, top, CLONE_FILES, ltd));
}
n--;
}
}
}
static void lo_mount(struct lo_data *lo)
{
int fs_fd, mnt_fd;
int ret;
fs_fd = ER(fsopen("fuse", 0));
ret = fsconfig(fs_fd, FSCONFIG_SET_FD, "fd", NULL, lo->devfd);
if (ret == -1) {
char opt[64];
snprintf(opt, sizeof(opt), "%i", lo->devfd);
ER(fsconfig(fs_fd, FSCONFIG_SET_STRING, "fd", opt, 0));
}
ER(fsconfig(fs_fd, FSCONFIG_SET_STRING, "rootmode", "40000", 0));
ER(fsconfig(fs_fd, FSCONFIG_SET_STRING, "user_id", "0", 0));
ER(fsconfig(fs_fd, FSCONFIG_SET_STRING, "group_id", "0", 0));
ER(fsconfig(fs_fd, FSCONFIG_CMD_CREATE, 0, 0, 0));
mnt_fd = ER(fsmount(fs_fd, 0, 0));
ER(move_mount(mnt_fd, "", AT_FDCWD, lo->c.mnt, MOVE_MOUNT_F_EMPTY_PATH));
close(mnt_fd);
close(fs_fd);
}
static void lo_usage(char *argv[])
{
errx(1, "usage: %s [-d] [-s] [-b] [-r] [-t] mountpoint", argv[0]);
}
int main(int argc, char *argv[])
{
struct lo_data *lo;
struct lo_config c = {};
char *devname = "/dev/fuse";
struct statx stat;
int ctr;
int delay_threads = 0;
if (argc < 2)
lo_usage(argv);
c.source = "/";
c.timeout = 999999;
for (ctr = 1; ctr < argc; ctr++) {
char *arg = argv[ctr];
if (arg[0] == '-') {
switch (arg[1]) {
case 'd':
c.debug = 1;
break;
case 's':
c.single = 1;
break;
case 'b':
c.bind = 1;
break;
case 'r':
c.direct = 1;
break;
case 'u':
c.uring = 1;
c.queue_depth = 24;
c.req_size = 1048576;
break;
case 'p':
c.passthrough = 1;
break;
case 'q':
c.passthrough2 = 1;
break;
#ifdef LO_NOTHREAD
case 't':
c.nothread = 1;
break;
#endif
default:
lo_usage(argv);
}
} else if (!c.mnt) {
c.mnt = arg;
} else {
lo_usage(argv);
}
}
if (c.nothread)
lo = lo_alloc_lo_nt();
else
lo = NL(calloc(1, sizeof(struct lo_data)));
lo->c = c;
lo_mutex_init(lo);
/* Don't mask creation mode, kernel already did that */
umask(0);
lo->root.next = lo->root.prev = &lo->root;
lo->root.refcount = 2;
lo->root.fd = ER(open(lo->c.source, O_PATH));
ER(statx(lo->root.fd, "", AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW, STATX_BASIC_STATS, &stat));
lo_convert_stat(&stat, &lo->root.attr);
lo->devfd = ER(open(devname, O_RDWR));
if (!lo->c.single) {
if (!lo->c.uring &&
ioctl(lo->devfd, FUSE_DEV_IOC_SYNC_INIT) == 0)
lo_start_threads(lo);
else
delay_threads = 1;
}
lo_mount(lo);
if (lo->c.uring)
lo_process_init(lo);
if (delay_threads)
lo_start_threads(lo);
lo_loop(lo, lo->devfd);
return 0;
}
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-19 11:34 ` Miklos Szeredi
@ 2026-08-19 11:38 ` Bernd Schubert
2026-08-19 17:56 ` Joanne Koong
1 sibling, 0 replies; 27+ messages in thread
From: Bernd Schubert @ 2026-08-19 11:38 UTC (permalink / raw)
To: Miklos Szeredi, Joanne Koong
Cc: jlayton, libaokun, axboe, amir73il, fuse-devel
On 8/19/26 13:34, Miklos Szeredi wrote:
> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>>
>> Currently, the connection's fuse_ring is created lazily on the first
>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
>> thread per queue (one per CPU) and those threads issue their first
>> REGISTER command concurrently. They then race to create the single
>> per-connection fuse_ring, which required open-coded handling in
>> fuse_uring_create() to detect and protect against concurrent creations.
>>
>> Decouple fuse_ring creation from ent registration and move it to
>> FUSE_INIT reply processing after a server has negotiated and set
>> FUSE_OVER_IO_URING. The ring is published before the connection is
>> marked initialized. fuse_uring_register() no longer creates the ring and
>> it instead uses the ring set up at init time.
>
> I tested this with loraw (a "raw" loopback tester that doesn't use
> libfuse) and it fails with
>
> root@kvm:~# ./loraw -u /mnt/fuse
> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
>
> cqe->res is -22 (EINVAL).
>
> Attaching the reproducer. To compile:
>
> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
> gcc loraw.c -oloraw -luring
Unless Joannes replies earlier, I will take a look in about 5 to 7 hours
- reply expected in the night.
Thanks,
Bernd
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-19 11:34 ` Miklos Szeredi
2026-08-19 11:38 ` Bernd Schubert
@ 2026-08-19 17:56 ` Joanne Koong
2026-08-19 20:05 ` Bernd Schubert
1 sibling, 1 reply; 27+ messages in thread
From: Joanne Koong @ 2026-08-19 17:56 UTC (permalink / raw)
To: Miklos Szeredi; +Cc: jlayton, libaokun, axboe, bernd, amir73il, fuse-devel
On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
> >
> > Currently, the connection's fuse_ring is created lazily on the first
> > FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
> > thread per queue (one per CPU) and those threads issue their first
> > REGISTER command concurrently. They then race to create the single
> > per-connection fuse_ring, which required open-coded handling in
> > fuse_uring_create() to detect and protect against concurrent creations.
> >
> > Decouple fuse_ring creation from ent registration and move it to
> > FUSE_INIT reply processing after a server has negotiated and set
> > FUSE_OVER_IO_URING. The ring is published before the connection is
> > marked initialized. fuse_uring_register() no longer creates the ring and
> > it instead uses the ring set up at init time.
>
> I tested this with loraw (a "raw" loopback tester that doesn't use
> libfuse) and it fails with
>
> root@kvm:~# ./loraw -u /mnt/fuse
> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
>
> cqe->res is -22 (EINVAL).
>
> Attaching the reproducer. To compile:
>
> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
> gcc loraw.c -oloraw -luring
>
Thanks for attaching the repro.
This is happening because this patch uses the FUSE_OVER_IO_URING init
reply as a signal that the ring should be created, but I missed that
the FUSE_OVER_IO_URING reply is *optional*.
Prior to this patch, there's two scenarios:
a) server sets FUSE_OVER_IO_URING reply at init time - requests will
automatically block until fuse-io-uring is completely set up
b) server does not set FUSE_OVER_IO_URING but later sends uring
register request - requests will continue along /dev/fuse path until
fuse-io-uring is completely set up
Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
I think the best way to fix this is to have the ring creation happen
when the kernel receives the first io-uring command instead of at
FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
Subject: [PATCH] fuse: create fuse_ring on the first io-uring command
Commit 6330b1f61ed1 ("fuse: decouple fuse_ring creation from ent
registration") moved fuse_ring creation to FUSE_INIT reply processing,
gated on the server setting FUSE_OVER_IO_URING in its reply flags.
However, that flag is optional. Libfuse sets it but servers not going
through libfuse may not.
Create the ring on the first io-uring command instead, independent of
what the server negotiated. Ring creation stays decoupled from ent
registration, which FUSE_IO_URING_CMD_ADD_QUEUE depends on since it
needs the ring to exist before any entry is registered.
Fixes: 6330b1f61ed1 ("fuse: decouple fuse_ring creation from ent registration")
Reported-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
---
fs/fuse/dev.c | 2 +-
fs/fuse/dev_uring.c | 18 +++++++++++-------
fs/fuse/dev_uring_i.h | 5 -----
3 files changed, 12 insertions(+), 13 deletions(-)
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 4fec31fc0b84..a665d76292c9 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -77,7 +77,7 @@ void fuse_chan_set_initialized(struct fuse_chan
*fch, struct fuse_chan_param *pa
fch->max_pages = param->max_pages;
if (param->io_uring_enabled)
- fuse_uring_conn_init(fch);
+ fch->io_uring = 1;
}
/* Pairs with smp_load_acquire() readers of fch->initialized */
diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
index e22a48c9a678..23c26099d159 100644
--- a/fs/fuse/dev_uring.c
+++ b/fs/fuse/dev_uring.c
@@ -303,6 +303,7 @@ static struct fuse_ring *fuse_uring_create(struct
fuse_chan *fch)
{
struct fuse_ring *ring;
size_t nr_queues = num_possible_cpus();
+ struct fuse_ring *res = NULL;
size_t max_payload_size;
ring = kzalloc_obj(*ring, GFP_KERNEL_ACCOUNT);
@@ -322,6 +323,12 @@ static struct fuse_ring *fuse_uring_create(struct
fuse_chan *fch)
spin_unlock(&fch->lock);
goto out_err;
}
+ if (fch->ring) {
+ /* race, another thread created the ring in the meantime */
+ spin_unlock(&fch->lock);
+ res = fch->ring;
+ goto out_err;
+ }
init_waitqueue_head(&ring->stop_waitq);
@@ -336,13 +343,7 @@ static struct fuse_ring *fuse_uring_create(struct
fuse_chan *fch)
out_err:
kfree(ring->queues);
kfree(ring);
- return NULL;
-}
-
-void fuse_uring_conn_init(struct fuse_chan *fch)
-{
- if (fuse_uring_create(fch))
- fch->io_uring = 1;
+ return res;
}
static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
@@ -1685,6 +1686,9 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd,
unsigned int issue_flags)
if (!smp_load_acquire(&fch->initialized))
return -EAGAIN;
+ if (!smp_load_acquire(&fch->ring) && !fuse_uring_create(fch))
+ return -ENOMEM;
+
switch (cmd_op) {
case FUSE_IO_URING_CMD_REGISTER:
err = fuse_uring_register(cmd, issue_flags, fch);
diff --git a/fs/fuse/dev_uring_i.h b/fs/fuse/dev_uring_i.h
index 263d0f8b9714..3233b07430d2 100644
--- a/fs/fuse/dev_uring_i.h
+++ b/fs/fuse/dev_uring_i.h
@@ -184,7 +184,6 @@ struct fuse_ring {
bool ready;
};
-void fuse_uring_conn_init(struct fuse_chan *fch);
void fuse_uring_stop_queues(struct fuse_ring *ring);
void fuse_uring_abort_end_requests(struct fuse_ring *ring);
int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags);
@@ -224,10 +223,6 @@ static inline bool fuse_uring_ready(struct fuse_chan *fch)
#else /* CONFIG_FUSE_IO_URING */
-static inline void fuse_uring_conn_init(struct fuse_chan *fch)
-{
-}
-
static inline void fuse_uring_abort(struct fuse_chan *fch)
{
}
--
2.52.0
If you'd prefer an inline replacement for the original commit instead
of a fixup patch on top of the tree, please let me know and I'd be
happy to send that over. Whatever would be easiest for you.
Thanks,
Joanne
^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-19 17:56 ` Joanne Koong
@ 2026-08-19 20:05 ` Bernd Schubert
2026-08-19 20:29 ` Joanne Koong
2026-08-20 8:02 ` Baokun Li
0 siblings, 2 replies; 27+ messages in thread
From: Bernd Schubert @ 2026-08-19 20:05 UTC (permalink / raw)
To: Joanne Koong, Miklos Szeredi
Cc: jlayton, libaokun, axboe, amir73il, fuse-devel
On 8/19/26 19:56, Joanne Koong wrote:
> On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>>
>> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>>>
>>> Currently, the connection's fuse_ring is created lazily on the first
>>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
>>> thread per queue (one per CPU) and those threads issue their first
>>> REGISTER command concurrently. They then race to create the single
>>> per-connection fuse_ring, which required open-coded handling in
>>> fuse_uring_create() to detect and protect against concurrent creations.
>>>
>>> Decouple fuse_ring creation from ent registration and move it to
>>> FUSE_INIT reply processing after a server has negotiated and set
>>> FUSE_OVER_IO_URING. The ring is published before the connection is
>>> marked initialized. fuse_uring_register() no longer creates the ring and
>>> it instead uses the ring set up at init time.
>>
>> I tested this with loraw (a "raw" loopback tester that doesn't use
>> libfuse) and it fails with
>>
>> root@kvm:~# ./loraw -u /mnt/fuse
>> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
>>
>> cqe->res is -22 (EINVAL).
>>
>> Attaching the reproducer. To compile:
>>
>> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
>> gcc loraw.c -oloraw -luring
>>
>
> Thanks for attaching the repro.
>
> This is happening because this patch uses the FUSE_OVER_IO_URING init
> reply as a signal that the ring should be created, but I missed that
> the FUSE_OVER_IO_URING reply is *optional*.
>
> Prior to this patch, there's two scenarios:
> a) server sets FUSE_OVER_IO_URING reply at init time - requests will
> automatically block until fuse-io-uring is completely set up
> b) server does not set FUSE_OVER_IO_URING but later sends uring
> register request - requests will continue along /dev/fuse path until
> fuse-io-uring is completely set up
>
> Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
>
> I think the best way to fix this is to have the ring creation happen
> when the kernel receives the first io-uring command instead of at
> FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
> that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
I don't think we should allow io-uring without FUSE_OVER_IO_URING and
I really thought that was disabled.
<... checking the code ...>
I'm on a ublk branch without your commits a applied, i.e. plain upstream 7.2 fuse
fuse_uring_cmd()
/* Once a connection has io-uring enabled on it, it can't be disabled */
if (!enable_uring && !fch->io_uring) {
pr_info_ratelimited("fuse-io-uring is disabled\n");
return -EOPNOTSUPP;
}
In process_init_reply()
if (flags & FUSE_OVER_IO_URING && fuse_uring_enabled())
fuse_chan_io_uring_enable(fc->chan);
And this is also absolutely needed to block requests in fuse_block_alloc(),
which is a requirement to avoid lock order issues between queue->lock and
fch->bg_lock (at least I believe that has not been solved yet).
I'm going to try Miklos' reproducer in a bit, but one way or the other
let's please not allow io-uring without FUSE_OVER_IO_URING reply.
Thanks,
Bernd
>
> Subject: [PATCH] fuse: create fuse_ring on the first io-uring command
>
> Commit 6330b1f61ed1 ("fuse: decouple fuse_ring creation from ent
> registration") moved fuse_ring creation to FUSE_INIT reply processing,
> gated on the server setting FUSE_OVER_IO_URING in its reply flags.
>
> However, that flag is optional. Libfuse sets it but servers not going
> through libfuse may not.
>
> Create the ring on the first io-uring command instead, independent of
> what the server negotiated. Ring creation stays decoupled from ent
> registration, which FUSE_IO_URING_CMD_ADD_QUEUE depends on since it
> needs the ring to exist before any entry is registered.
>
> Fixes: 6330b1f61ed1 ("fuse: decouple fuse_ring creation from ent registration")
> Reported-by: Miklos Szeredi <mszeredi@redhat.com>
> Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
> ---
> fs/fuse/dev.c | 2 +-
> fs/fuse/dev_uring.c | 18 +++++++++++-------
> fs/fuse/dev_uring_i.h | 5 -----
> 3 files changed, 12 insertions(+), 13 deletions(-)
>
> diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
> index 4fec31fc0b84..a665d76292c9 100644
> --- a/fs/fuse/dev.c
> +++ b/fs/fuse/dev.c
> @@ -77,7 +77,7 @@ void fuse_chan_set_initialized(struct fuse_chan
> *fch, struct fuse_chan_param *pa
> fch->max_pages = param->max_pages;
>
> if (param->io_uring_enabled)
> - fuse_uring_conn_init(fch);
> + fch->io_uring = 1;
> }
>
> /* Pairs with smp_load_acquire() readers of fch->initialized */
> diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
> index e22a48c9a678..23c26099d159 100644
> --- a/fs/fuse/dev_uring.c
> +++ b/fs/fuse/dev_uring.c
> @@ -303,6 +303,7 @@ static struct fuse_ring *fuse_uring_create(struct
> fuse_chan *fch)
> {
> struct fuse_ring *ring;
> size_t nr_queues = num_possible_cpus();
> + struct fuse_ring *res = NULL;
> size_t max_payload_size;
>
> ring = kzalloc_obj(*ring, GFP_KERNEL_ACCOUNT);
> @@ -322,6 +323,12 @@ static struct fuse_ring *fuse_uring_create(struct
> fuse_chan *fch)
> spin_unlock(&fch->lock);
> goto out_err;
> }
> + if (fch->ring) {
> + /* race, another thread created the ring in the meantime */
> + spin_unlock(&fch->lock);
> + res = fch->ring;
> + goto out_err;
> + }
>
> init_waitqueue_head(&ring->stop_waitq);
>
> @@ -336,13 +343,7 @@ static struct fuse_ring *fuse_uring_create(struct
> fuse_chan *fch)
> out_err:
> kfree(ring->queues);
> kfree(ring);
> - return NULL;
> -}
> -
> -void fuse_uring_conn_init(struct fuse_chan *fch)
> -{
> - if (fuse_uring_create(fch))
> - fch->io_uring = 1;
> + return res;
> }
>
> static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
> @@ -1685,6 +1686,9 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd,
> unsigned int issue_flags)
> if (!smp_load_acquire(&fch->initialized))
> return -EAGAIN;
>
> + if (!smp_load_acquire(&fch->ring) && !fuse_uring_create(fch))
> + return -ENOMEM;
> +
> switch (cmd_op) {
> case FUSE_IO_URING_CMD_REGISTER:
> err = fuse_uring_register(cmd, issue_flags, fch);
> diff --git a/fs/fuse/dev_uring_i.h b/fs/fuse/dev_uring_i.h
> index 263d0f8b9714..3233b07430d2 100644
> --- a/fs/fuse/dev_uring_i.h
> +++ b/fs/fuse/dev_uring_i.h
> @@ -184,7 +184,6 @@ struct fuse_ring {
> bool ready;
> };
>
> -void fuse_uring_conn_init(struct fuse_chan *fch);
> void fuse_uring_stop_queues(struct fuse_ring *ring);
> void fuse_uring_abort_end_requests(struct fuse_ring *ring);
> int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags);
> @@ -224,10 +223,6 @@ static inline bool fuse_uring_ready(struct fuse_chan *fch)
>
> #else /* CONFIG_FUSE_IO_URING */
>
> -static inline void fuse_uring_conn_init(struct fuse_chan *fch)
> -{
> -}
> -
> static inline void fuse_uring_abort(struct fuse_chan *fch)
> {
> }
> --
> 2.52.0
>
>
> If you'd prefer an inline replacement for the original commit instead
> of a fixup patch on top of the tree, please let me know and I'd be
> happy to send that over. Whatever would be easiest for you.
>
> Thanks,
> Joanne
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-19 20:05 ` Bernd Schubert
@ 2026-08-19 20:29 ` Joanne Koong
2026-08-19 20:52 ` Bernd Schubert
2026-08-20 8:02 ` Baokun Li
1 sibling, 1 reply; 27+ messages in thread
From: Joanne Koong @ 2026-08-19 20:29 UTC (permalink / raw)
To: Bernd Schubert
Cc: Miklos Szeredi, jlayton, libaokun, axboe, amir73il, fuse-devel
On Wed, Aug 19, 2026 at 1:05 PM Bernd Schubert <bernd@bsbernd.com> wrote:
>
>
>
> On 8/19/26 19:56, Joanne Koong wrote:
> > On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
> >>
> >> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
> >>>
> >>> Currently, the connection's fuse_ring is created lazily on the first
> >>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
> >>> thread per queue (one per CPU) and those threads issue their first
> >>> REGISTER command concurrently. They then race to create the single
> >>> per-connection fuse_ring, which required open-coded handling in
> >>> fuse_uring_create() to detect and protect against concurrent creations.
> >>>
> >>> Decouple fuse_ring creation from ent registration and move it to
> >>> FUSE_INIT reply processing after a server has negotiated and set
> >>> FUSE_OVER_IO_URING. The ring is published before the connection is
> >>> marked initialized. fuse_uring_register() no longer creates the ring and
> >>> it instead uses the ring set up at init time.
> >>
> >> I tested this with loraw (a "raw" loopback tester that doesn't use
> >> libfuse) and it fails with
> >>
> >> root@kvm:~# ./loraw -u /mnt/fuse
> >> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
> >>
> >> cqe->res is -22 (EINVAL).
> >>
> >> Attaching the reproducer. To compile:
> >>
> >> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
> >> gcc loraw.c -oloraw -luring
> >>
> >
> > Thanks for attaching the repro.
> >
> > This is happening because this patch uses the FUSE_OVER_IO_URING init
> > reply as a signal that the ring should be created, but I missed that
> > the FUSE_OVER_IO_URING reply is *optional*.
> >
> > Prior to this patch, there's two scenarios:
> > a) server sets FUSE_OVER_IO_URING reply at init time - requests will
> > automatically block until fuse-io-uring is completely set up
> > b) server does not set FUSE_OVER_IO_URING but later sends uring
> > register request - requests will continue along /dev/fuse path until
> > fuse-io-uring is completely set up
> >
> > Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
> >
> > I think the best way to fix this is to have the ring creation happen
> > when the kernel receives the first io-uring command instead of at
> > FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
> > that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
>
> I don't think we should allow io-uring without FUSE_OVER_IO_URING and
> I really thought that was disabled.
This is pre-existing behavior that's been there since the beginning
(kernel version 6.14) [1]. I don't think we can change this now, or
it'll break backwards compatibility, like Miklos's loraw program.
>
> <... checking the code ...>
>
> I'm on a ublk branch without your commits a applied, i.e. plain upstream 7.2 fuse
>
> fuse_uring_cmd()
> /* Once a connection has io-uring enabled on it, it can't be disabled */
> if (!enable_uring && !fch->io_uring) {
> pr_info_ratelimited("fuse-io-uring is disabled\n");
> return -EOPNOTSUPP;
> }
>
>
>
> In process_init_reply()
>
> if (flags & FUSE_OVER_IO_URING && fuse_uring_enabled())
> fuse_chan_io_uring_enable(fc->chan);
>
The condition check in fuse_uring_cmd() is an && and not an ||. It
doesn't enforce that the server must have sent over FUSE_OVER_IO_URING
in the init reply to use fuse io-uring. With enable_uring=1 (which is
needed for the kernel to advertise FUSE_OVER_IO_URING in the init
request in the first place), the fuse io-uring logic proceeds
regardless of whether the server replied with the FUSE_OVER_IO_URING
flag or not.
Thanks,
Joanne
[1] https://elixir.bootlin.com/linux/v6.14-rc1/source/fs/fuse/inode.c#L1455
>
> And this is also absolutely needed to block requests in fuse_block_alloc(),
> which is a requirement to avoid lock order issues between queue->lock and
> fch->bg_lock (at least I believe that has not been solved yet).
>
>
> I'm going to try Miklos' reproducer in a bit, but one way or the other
> let's please not allow io-uring without FUSE_OVER_IO_URING reply.
>
>
> Thanks,
> Bernd
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-19 20:29 ` Joanne Koong
@ 2026-08-19 20:52 ` Bernd Schubert
2026-08-19 21:35 ` Bernd Schubert
0 siblings, 1 reply; 27+ messages in thread
From: Bernd Schubert @ 2026-08-19 20:52 UTC (permalink / raw)
To: Joanne Koong
Cc: Miklos Szeredi, jlayton, libaokun, axboe, amir73il, fuse-devel
On 8/19/26 22:29, Joanne Koong wrote:
> On Wed, Aug 19, 2026 at 1:05 PM Bernd Schubert <bernd@bsbernd.com> wrote:
>>
>>
>>
>> On 8/19/26 19:56, Joanne Koong wrote:
>>> On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>>>>
>>>> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>>>>>
>>>>> Currently, the connection's fuse_ring is created lazily on the first
>>>>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
>>>>> thread per queue (one per CPU) and those threads issue their first
>>>>> REGISTER command concurrently. They then race to create the single
>>>>> per-connection fuse_ring, which required open-coded handling in
>>>>> fuse_uring_create() to detect and protect against concurrent creations.
>>>>>
>>>>> Decouple fuse_ring creation from ent registration and move it to
>>>>> FUSE_INIT reply processing after a server has negotiated and set
>>>>> FUSE_OVER_IO_URING. The ring is published before the connection is
>>>>> marked initialized. fuse_uring_register() no longer creates the ring and
>>>>> it instead uses the ring set up at init time.
>>>>
>>>> I tested this with loraw (a "raw" loopback tester that doesn't use
>>>> libfuse) and it fails with
>>>>
>>>> root@kvm:~# ./loraw -u /mnt/fuse
>>>> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
>>>>
>>>> cqe->res is -22 (EINVAL).
>>>>
>>>> Attaching the reproducer. To compile:
>>>>
>>>> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
>>>> gcc loraw.c -oloraw -luring
>>>>
>>>
>>> Thanks for attaching the repro.
>>>
>>> This is happening because this patch uses the FUSE_OVER_IO_URING init
>>> reply as a signal that the ring should be created, but I missed that
>>> the FUSE_OVER_IO_URING reply is *optional*.
>>>
>>> Prior to this patch, there's two scenarios:
>>> a) server sets FUSE_OVER_IO_URING reply at init time - requests will
>>> automatically block until fuse-io-uring is completely set up
>>> b) server does not set FUSE_OVER_IO_URING but later sends uring
>>> register request - requests will continue along /dev/fuse path until
>>> fuse-io-uring is completely set up
>>>
>>> Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
>>>
>>> I think the best way to fix this is to have the ring creation happen
>>> when the kernel receives the first io-uring command instead of at
>>> FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
>>> that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
>>
>> I don't think we should allow io-uring without FUSE_OVER_IO_URING and
>> I really thought that was disabled.
>
> This is pre-existing behavior that's been there since the beginning
> (kernel version 6.14) [1]. I don't think we can change this now, or
> it'll break backwards compatibility, like Miklos's loraw program.
>
>>
>> <... checking the code ...>
>>
>> I'm on a ublk branch without your commits a applied, i.e. plain upstream 7.2 fuse
>>
>> fuse_uring_cmd()
>> /* Once a connection has io-uring enabled on it, it can't be disabled */
>> if (!enable_uring && !fch->io_uring) {
>> pr_info_ratelimited("fuse-io-uring is disabled\n");
>> return -EOPNOTSUPP;
>> }
>>
>>
>>
>> In process_init_reply()
>>
>> if (flags & FUSE_OVER_IO_URING && fuse_uring_enabled())
>> fuse_chan_io_uring_enable(fc->chan);
>>
>
> The condition check in fuse_uring_cmd() is an && and not an ||. It
Aaaarg. That makes the entire condition useless :(
> doesn't enforce that the server must have sent over FUSE_OVER_IO_URING
> in the init reply to use fuse io-uring. With enable_uring=1 (which is
> needed for the kernel to advertise FUSE_OVER_IO_URING in the init
> request in the first place), the fuse io-uring logic proceeds
> regardless of whether the server replied with the FUSE_OVER_IO_URING
> flag or not.
>
> Thanks,
> Joanne
>
> [1] https://elixir.bootlin.com/linux/v6.14-rc1/source/fs/fuse/inode.c#L1455
The comment there is definitely outdated, at least it was *supposed* to.
Well, then we have a problem because we have a lock order inversion
issue. After adding in the reduce-queue series I had planned to work on
distributing fch->num_background among queues, so that each queue gets
its own num_background, so that holding two locks like in
fuse_uring_queue_bq_req() wouldn't be needed anymore.
Also again, without FUSE_OVER_IO_URING reply the condition in
fuse_block_alloc() doesn't make any sense.
Thanks,
Bernd
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-19 20:52 ` Bernd Schubert
@ 2026-08-19 21:35 ` Bernd Schubert
0 siblings, 0 replies; 27+ messages in thread
From: Bernd Schubert @ 2026-08-19 21:35 UTC (permalink / raw)
To: Joanne Koong
Cc: Miklos Szeredi, jlayton, libaokun, axboe, amir73il, fuse-devel
On 8/19/26 22:52, Bernd Schubert wrote:
>
>
> On 8/19/26 22:29, Joanne Koong wrote:
>> On Wed, Aug 19, 2026 at 1:05 PM Bernd Schubert <bernd@bsbernd.com> wrote:
>>>
>>>
>>>
>>> On 8/19/26 19:56, Joanne Koong wrote:
>>>> On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>>>>>
>>>>> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>>>>>>
>>>>>> Currently, the connection's fuse_ring is created lazily on the first
>>>>>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
>>>>>> thread per queue (one per CPU) and those threads issue their first
>>>>>> REGISTER command concurrently. They then race to create the single
>>>>>> per-connection fuse_ring, which required open-coded handling in
>>>>>> fuse_uring_create() to detect and protect against concurrent creations.
>>>>>>
>>>>>> Decouple fuse_ring creation from ent registration and move it to
>>>>>> FUSE_INIT reply processing after a server has negotiated and set
>>>>>> FUSE_OVER_IO_URING. The ring is published before the connection is
>>>>>> marked initialized. fuse_uring_register() no longer creates the ring and
>>>>>> it instead uses the ring set up at init time.
>>>>>
>>>>> I tested this with loraw (a "raw" loopback tester that doesn't use
>>>>> libfuse) and it fails with
>>>>>
>>>>> root@kvm:~# ./loraw -u /mnt/fuse
>>>>> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
>>>>>
>>>>> cqe->res is -22 (EINVAL).
>>>>>
>>>>> Attaching the reproducer. To compile:
>>>>>
>>>>> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
>>>>> gcc loraw.c -oloraw -luring
>>>>>
>>>>
>>>> Thanks for attaching the repro.
>>>>
>>>> This is happening because this patch uses the FUSE_OVER_IO_URING init
>>>> reply as a signal that the ring should be created, but I missed that
>>>> the FUSE_OVER_IO_URING reply is *optional*.
>>>>
>>>> Prior to this patch, there's two scenarios:
>>>> a) server sets FUSE_OVER_IO_URING reply at init time - requests will
>>>> automatically block until fuse-io-uring is completely set up
>>>> b) server does not set FUSE_OVER_IO_URING but later sends uring
>>>> register request - requests will continue along /dev/fuse path until
>>>> fuse-io-uring is completely set up
>>>>
>>>> Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
>>>>
>>>> I think the best way to fix this is to have the ring creation happen
>>>> when the kernel receives the first io-uring command instead of at
>>>> FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
>>>> that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
>>>
>>> I don't think we should allow io-uring without FUSE_OVER_IO_URING and
>>> I really thought that was disabled.
>>
>> This is pre-existing behavior that's been there since the beginning
>> (kernel version 6.14) [1]. I don't think we can change this now, or
>> it'll break backwards compatibility, like Miklos's loraw program.
>>
>>>
>>> <... checking the code ...>
>>>
>>> I'm on a ublk branch without your commits a applied, i.e. plain upstream 7.2 fuse
>>>
>>> fuse_uring_cmd()
>>> /* Once a connection has io-uring enabled on it, it can't be disabled */
>>> if (!enable_uring && !fch->io_uring) {
>>> pr_info_ratelimited("fuse-io-uring is disabled\n");
>>> return -EOPNOTSUPP;
>>> }
>>>
>>>
>>>
>>> In process_init_reply()
>>>
>>> if (flags & FUSE_OVER_IO_URING && fuse_uring_enabled())
>>> fuse_chan_io_uring_enable(fc->chan);
>>>
>>
>> The condition check in fuse_uring_cmd() is an && and not an ||. It
>
> Aaaarg. That makes the entire condition useless :(
>
>> doesn't enforce that the server must have sent over FUSE_OVER_IO_URING
>> in the init reply to use fuse io-uring. With enable_uring=1 (which is
>> needed for the kernel to advertise FUSE_OVER_IO_URING in the init
>> request in the first place), the fuse io-uring logic proceeds
>> regardless of whether the server replied with the FUSE_OVER_IO_URING
>> flag or not.
>>
>> Thanks,
>> Joanne
>>
>> [1] https://elixir.bootlin.com/linux/v6.14-rc1/source/fs/fuse/inode.c#L1455
>
> The comment there is definitely outdated, at least it was *supposed* to.
>
> Well, then we have a problem because we have a lock order inversion
> issue. After adding in the reduce-queue series I had planned to work on
> distributing fch->num_background among queues, so that each queue gets
> its own num_background, so that holding two locks like in
> fuse_uring_queue_bq_req() wouldn't be needed anymore.
>
> Also again, without FUSE_OVER_IO_URING reply the condition in
> fuse_block_alloc() doesn't make any sense.
Lock order inversion is this
CPU 0 - io-uring completion | CPU 1 - legacy background completion
| fuse_request_end()
| takes fch->bg_lock
| fuse_request_bg_finish()
fuse_uring_cmd() |
fuse_uring_commit_fetch() |
fuse_uring_commit() |
fuse_uring_req_end() |
takes queue->lock |
wants fch->bg_lock |
| flush_bg_queue()
| fuse_send_one()
| fiq->ops->send_req()
| fuse_uring_queue_fuse_req()
| wants queue->lock
(I queried AI about it to quickly generate that graph, but that is
exactly what I had seen back in development and which is why fuse
requests are supposed to be blocked until queue initialization is
complete, i.e. to avoid switching from /dev/fuse to io-uring at
run time.
The wrong "&&" in fuse_uring_cmd() also makes it possible that people
bypass the module option to enable io-uring, by just setting the flag
in their userspace implementation. Yikes :/
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-19 20:05 ` Bernd Schubert
2026-08-19 20:29 ` Joanne Koong
@ 2026-08-20 8:02 ` Baokun Li
2026-08-20 16:16 ` Joanne Koong
1 sibling, 1 reply; 27+ messages in thread
From: Baokun Li @ 2026-08-20 8:02 UTC (permalink / raw)
To: Bernd Schubert, Joanne Koong, Miklos Szeredi
Cc: jlayton, axboe, amir73il, fuse-devel
Hi all,
On 2026/8/20 04:05, Bernd Schubert wrote:
>
> On 8/19/26 19:56, Joanne Koong wrote:
>> On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>>> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>>>> Currently, the connection's fuse_ring is created lazily on the first
>>>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
>>>> thread per queue (one per CPU) and those threads issue their first
>>>> REGISTER command concurrently. They then race to create the single
>>>> per-connection fuse_ring, which required open-coded handling in
>>>> fuse_uring_create() to detect and protect against concurrent creations.
>>>>
>>>> Decouple fuse_ring creation from ent registration and move it to
>>>> FUSE_INIT reply processing after a server has negotiated and set
>>>> FUSE_OVER_IO_URING. The ring is published before the connection is
>>>> marked initialized. fuse_uring_register() no longer creates the ring and
>>>> it instead uses the ring set up at init time.
>>> I tested this with loraw (a "raw" loopback tester that doesn't use
>>> libfuse) and it fails with
>>>
>>> root@kvm:~# ./loraw -u /mnt/fuse
>>> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
>>>
>>> cqe->res is -22 (EINVAL).
>>>
>>> Attaching the reproducer. To compile:
>>>
>>> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
>>> gcc loraw.c -oloraw -luring
>>>
>> Thanks for attaching the repro.
>>
>> This is happening because this patch uses the FUSE_OVER_IO_URING init
>> reply as a signal that the ring should be created, but I missed that
>> the FUSE_OVER_IO_URING reply is *optional*.
>>
>> Prior to this patch, there's two scenarios:
>> a) server sets FUSE_OVER_IO_URING reply at init time - requests will
>> automatically block until fuse-io-uring is completely set up
>> b) server does not set FUSE_OVER_IO_URING but later sends uring
>> register request - requests will continue along /dev/fuse path until
>> fuse-io-uring is completely set up
>>
>> Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
>>
>> I think the best way to fix this is to have the ring creation happen
>> when the kernel receives the first io-uring command instead of at
>> FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
>> that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
> I don't think we should allow io-uring without FUSE_OVER_IO_URING and
> I really thought that was disabled.
I share Bernd's concern here. Allowing io-uring without
FUSE_OVER_IO_URING means enabling a capability beyond what was
negotiated. We should honor the negotiated feature set, and print
the negotiated flags to dmesg at INIT time so issues like this are
easy to spot.
>
> <... checking the code ...>
>
> I'm on a ublk branch without your commits a applied, i.e. plain upstream 7.2 fuse
>
> fuse_uring_cmd()
> /* Once a connection has io-uring enabled on it, it can't be disabled */
> if (!enable_uring && !fch->io_uring) {
> pr_info_ratelimited("fuse-io-uring is disabled\n");
> return -EOPNOTSUPP;
> }
BTW, the current code clears fch->io_uring on REGISTER failure:
if (err) {
fch->io_uring = 0;
wake_up_all(&fch->blocked_waitq);
return err;
}
But by then other entries may have already registered successfully,
fiq->ops is switched to fuse_io_uring_ops, and requests are flowing
through the uring path. Clearing fch->io_uring here may reject all
subsequent io_uring commands with -EOPNOTSUPP while the data path
is still running on uring ops.
fch->io_uring is a protocol fact — the daemon negotiated
FUSE_OVER_IO_URING at INIT time. A single REGISTER failure should
not regress it, at least not once the ring is ready. The daemon
can retry; the negotiated capability should be permanent.
Thanks,
Baokun
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-20 8:02 ` Baokun Li
@ 2026-08-20 16:16 ` Joanne Koong
2026-08-20 17:20 ` Bernd Schubert
2026-08-21 3:04 ` Baokun Li
0 siblings, 2 replies; 27+ messages in thread
From: Joanne Koong @ 2026-08-20 16:16 UTC (permalink / raw)
To: Baokun Li
Cc: Bernd Schubert, Miklos Szeredi, jlayton, axboe, amir73il,
fuse-devel
On Thu, Aug 20, 2026 at 1:02 AM Baokun Li <libaokun@linux.alibaba.com> wrote:
>
> Hi all,
>
> On 2026/8/20 04:05, Bernd Schubert wrote:
> >
> > On 8/19/26 19:56, Joanne Koong wrote:
> >> On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
> >>> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
> >>>> Currently, the connection's fuse_ring is created lazily on the first
> >>>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
> >>>> thread per queue (one per CPU) and those threads issue their first
> >>>> REGISTER command concurrently. They then race to create the single
> >>>> per-connection fuse_ring, which required open-coded handling in
> >>>> fuse_uring_create() to detect and protect against concurrent creations.
> >>>>
> >>>> Decouple fuse_ring creation from ent registration and move it to
> >>>> FUSE_INIT reply processing after a server has negotiated and set
> >>>> FUSE_OVER_IO_URING. The ring is published before the connection is
> >>>> marked initialized. fuse_uring_register() no longer creates the ring and
> >>>> it instead uses the ring set up at init time.
> >>> I tested this with loraw (a "raw" loopback tester that doesn't use
> >>> libfuse) and it fails with
> >>>
> >>> root@kvm:~# ./loraw -u /mnt/fuse
> >>> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
> >>>
> >>> cqe->res is -22 (EINVAL).
> >>>
> >>> Attaching the reproducer. To compile:
> >>>
> >>> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
> >>> gcc loraw.c -oloraw -luring
> >>>
> >> Thanks for attaching the repro.
> >>
> >> This is happening because this patch uses the FUSE_OVER_IO_URING init
> >> reply as a signal that the ring should be created, but I missed that
> >> the FUSE_OVER_IO_URING reply is *optional*.
> >>
> >> Prior to this patch, there's two scenarios:
> >> a) server sets FUSE_OVER_IO_URING reply at init time - requests will
> >> automatically block until fuse-io-uring is completely set up
> >> b) server does not set FUSE_OVER_IO_URING but later sends uring
> >> register request - requests will continue along /dev/fuse path until
> >> fuse-io-uring is completely set up
> >>
> >> Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
> >>
> >> I think the best way to fix this is to have the ring creation happen
> >> when the kernel receives the first io-uring command instead of at
> >> FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
> >> that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
> > I don't think we should allow io-uring without FUSE_OVER_IO_URING and
> > I really thought that was disabled.
>
> I share Bernd's concern here. Allowing io-uring without
> FUSE_OVER_IO_URING means enabling a capability beyond what was
> negotiated. We should honor the negotiated feature set, and print
> the negotiated flags to dmesg at INIT time so issues like this are
> easy to spot.
Not sure if you missed this reply [1], but will copy and paste it here:
This is pre-existing behavior that's been there since the beginning
(kernel version 6.14). I don't think we can change this now, or
it'll break backwards compatibility, like Miklos's loraw program.
>
> >
> > <... checking the code ...>
> >
> > I'm on a ublk branch without your commits a applied, i.e. plain upstream 7.2 fuse
> >
> > fuse_uring_cmd()
> > /* Once a connection has io-uring enabled on it, it can't be disabled */
> > if (!enable_uring && !fch->io_uring) {
> > pr_info_ratelimited("fuse-io-uring is disabled\n");
> > return -EOPNOTSUPP;
> > }
>
> BTW, the current code clears fch->io_uring on REGISTER failure:
>
> if (err) {
> fch->io_uring = 0;
> wake_up_all(&fch->blocked_waitq);
> return err;
> }
>
> But by then other entries may have already registered successfully,
> fiq->ops is switched to fuse_io_uring_ops, and requests are flowing
> through the uring path. Clearing fch->io_uring here may reject all
> subsequent io_uring commands with -EOPNOTSUPP while the data path
> is still running on uring ops.
>
This is also pre-existing behavior. The check in fuse_uring_cmd() is a
&& not an ||. It'll only return -EOPNOTSUPP if admin disables
enable_uring.
Thanks,
Joanne
[1] https://lore.kernel.org/fuse-devel/CAJnrk1Z_ioW02xbrUekpzSJNerHL=XH-VYqHmUXFzyUD4HDNdQ@mail.gmail.com/
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-20 16:16 ` Joanne Koong
@ 2026-08-20 17:20 ` Bernd Schubert
2026-08-20 17:46 ` Joanne Koong
2026-08-21 3:24 ` Baokun Li
2026-08-21 3:04 ` Baokun Li
1 sibling, 2 replies; 27+ messages in thread
From: Bernd Schubert @ 2026-08-20 17:20 UTC (permalink / raw)
To: Joanne Koong, Baokun Li
Cc: Miklos Szeredi, jlayton, axboe, amir73il, fuse-devel
On 8/20/26 18:16, Joanne Koong wrote:
> On Thu, Aug 20, 2026 at 1:02 AM Baokun Li <libaokun@linux.alibaba.com> wrote:
>>
>> Hi all,
>>
>> On 2026/8/20 04:05, Bernd Schubert wrote:
>>>
>>> On 8/19/26 19:56, Joanne Koong wrote:
>>>> On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>>>>> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>>>>>> Currently, the connection's fuse_ring is created lazily on the first
>>>>>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
>>>>>> thread per queue (one per CPU) and those threads issue their first
>>>>>> REGISTER command concurrently. They then race to create the single
>>>>>> per-connection fuse_ring, which required open-coded handling in
>>>>>> fuse_uring_create() to detect and protect against concurrent creations.
>>>>>>
>>>>>> Decouple fuse_ring creation from ent registration and move it to
>>>>>> FUSE_INIT reply processing after a server has negotiated and set
>>>>>> FUSE_OVER_IO_URING. The ring is published before the connection is
>>>>>> marked initialized. fuse_uring_register() no longer creates the ring and
>>>>>> it instead uses the ring set up at init time.
>>>>> I tested this with loraw (a "raw" loopback tester that doesn't use
>>>>> libfuse) and it fails with
>>>>>
>>>>> root@kvm:~# ./loraw -u /mnt/fuse
>>>>> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
>>>>>
>>>>> cqe->res is -22 (EINVAL).
>>>>>
>>>>> Attaching the reproducer. To compile:
>>>>>
>>>>> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
>>>>> gcc loraw.c -oloraw -luring
>>>>>
>>>> Thanks for attaching the repro.
>>>>
>>>> This is happening because this patch uses the FUSE_OVER_IO_URING init
>>>> reply as a signal that the ring should be created, but I missed that
>>>> the FUSE_OVER_IO_URING reply is *optional*.
>>>>
>>>> Prior to this patch, there's two scenarios:
>>>> a) server sets FUSE_OVER_IO_URING reply at init time - requests will
>>>> automatically block until fuse-io-uring is completely set up
>>>> b) server does not set FUSE_OVER_IO_URING but later sends uring
>>>> register request - requests will continue along /dev/fuse path until
>>>> fuse-io-uring is completely set up
>>>>
>>>> Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
>>>>
>>>> I think the best way to fix this is to have the ring creation happen
>>>> when the kernel receives the first io-uring command instead of at
>>>> FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
>>>> that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
>>> I don't think we should allow io-uring without FUSE_OVER_IO_URING and
>>> I really thought that was disabled.
>>
>> I share Bernd's concern here. Allowing io-uring without
>> FUSE_OVER_IO_URING means enabling a capability beyond what was
>> negotiated. We should honor the negotiated feature set, and print
>> the negotiated flags to dmesg at INIT time so issues like this are
>> easy to spot.
>
> Not sure if you missed this reply [1], but will copy and paste it here:
>
> This is pre-existing behavior that's been there since the beginning
> (kernel version 6.14). I don't think we can change this now, or
> it'll break backwards compatibility, like Miklos's loraw program.
>
I think we need to discuss this. I had replied that the current
accidental scheme we
- deadlock (lock order), with bg_lock being one issue, but I bet there
is more
- module option bypass
- bypass of what fuse-client/kernel announces
I.e. if a fuse-server did implement the accidental scheme, it was broken
anyway.
If wanted to be paranoid, we would switch to FUSE_OVER_IO_URING2 flag
and ignore FUSE_OVER_IO_URING. I hope you don't insist on allowing
fuse-server to set flags that fuse-server doesn't even announce...
Thanks,
Bernd
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-20 17:20 ` Bernd Schubert
@ 2026-08-20 17:46 ` Joanne Koong
2026-08-20 18:27 ` Bernd Schubert
2026-08-21 3:38 ` Baokun Li
2026-08-21 3:24 ` Baokun Li
1 sibling, 2 replies; 27+ messages in thread
From: Joanne Koong @ 2026-08-20 17:46 UTC (permalink / raw)
To: Bernd Schubert
Cc: Baokun Li, Miklos Szeredi, jlayton, axboe, amir73il, fuse-devel
On Thu, Aug 20, 2026 at 10:20 AM Bernd Schubert <bernd@bsbernd.com> wrote:
>
>
>
> On 8/20/26 18:16, Joanne Koong wrote:
> > On Thu, Aug 20, 2026 at 1:02 AM Baokun Li <libaokun@linux.alibaba.com> wrote:
> >>
> >> Hi all,
> >>
> >> On 2026/8/20 04:05, Bernd Schubert wrote:
> >>>
> >>> On 8/19/26 19:56, Joanne Koong wrote:
> >>>> On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
> >>>>> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
> >>>>>> Currently, the connection's fuse_ring is created lazily on the first
> >>>>>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
> >>>>>> thread per queue (one per CPU) and those threads issue their first
> >>>>>> REGISTER command concurrently. They then race to create the single
> >>>>>> per-connection fuse_ring, which required open-coded handling in
> >>>>>> fuse_uring_create() to detect and protect against concurrent creations.
> >>>>>>
> >>>>>> Decouple fuse_ring creation from ent registration and move it to
> >>>>>> FUSE_INIT reply processing after a server has negotiated and set
> >>>>>> FUSE_OVER_IO_URING. The ring is published before the connection is
> >>>>>> marked initialized. fuse_uring_register() no longer creates the ring and
> >>>>>> it instead uses the ring set up at init time.
> >>>>> I tested this with loraw (a "raw" loopback tester that doesn't use
> >>>>> libfuse) and it fails with
> >>>>>
> >>>>> root@kvm:~# ./loraw -u /mnt/fuse
> >>>>> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
> >>>>>
> >>>>> cqe->res is -22 (EINVAL).
> >>>>>
> >>>>> Attaching the reproducer. To compile:
> >>>>>
> >>>>> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
> >>>>> gcc loraw.c -oloraw -luring
> >>>>>
> >>>> Thanks for attaching the repro.
> >>>>
> >>>> This is happening because this patch uses the FUSE_OVER_IO_URING init
> >>>> reply as a signal that the ring should be created, but I missed that
> >>>> the FUSE_OVER_IO_URING reply is *optional*.
> >>>>
> >>>> Prior to this patch, there's two scenarios:
> >>>> a) server sets FUSE_OVER_IO_URING reply at init time - requests will
> >>>> automatically block until fuse-io-uring is completely set up
> >>>> b) server does not set FUSE_OVER_IO_URING but later sends uring
> >>>> register request - requests will continue along /dev/fuse path until
> >>>> fuse-io-uring is completely set up
> >>>>
> >>>> Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
> >>>>
> >>>> I think the best way to fix this is to have the ring creation happen
> >>>> when the kernel receives the first io-uring command instead of at
> >>>> FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
> >>>> that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
> >>> I don't think we should allow io-uring without FUSE_OVER_IO_URING and
> >>> I really thought that was disabled.
> >>
> >> I share Bernd's concern here. Allowing io-uring without
> >> FUSE_OVER_IO_URING means enabling a capability beyond what was
> >> negotiated. We should honor the negotiated feature set, and print
> >> the negotiated flags to dmesg at INIT time so issues like this are
> >> easy to spot.
> >
> > Not sure if you missed this reply [1], but will copy and paste it here:
> >
> > This is pre-existing behavior that's been there since the beginning
> > (kernel version 6.14). I don't think we can change this now, or
> > it'll break backwards compatibility, like Miklos's loraw program.
> >
>
> I think we need to discuss this. I had replied that the current
> accidental scheme we
>
> - deadlock (lock order), with bg_lock being one issue, but I bet there
> is more
> - module option bypass
> - bypass of what fuse-client/kernel announces
>
> I.e. if a fuse-server did implement the accidental scheme, it was broken
> anyway.
>
> If wanted to be paranoid, we would switch to FUSE_OVER_IO_URING2 flag
> and ignore FUSE_OVER_IO_URING. I hope you don't insist on allowing
I don't think a new FUSE_OVER_IO_URING2 flag helps. If the kernel
ignores FUSE_OVER_IO_URING as you describe, servers using older
versions of libfuse will break. If it accepts both flags, it's
behaviorally identical to just enforcing FUSE_OVER_IO_URING.
> fuse-server to set flags that fuse-server doesn't even announce...
>
I think this is a call better left up to Miklos. I don't know how
rigorously it is enforced in linux that nothing should break backwards
compatibility. The code was released in March 2025 as part of kernel
version 6.14, so it's been roughly a year and a half, which I guess
isn't that long in the grand scheme of things, but if Miklos has his
loraw server program that relies on this, it's probably likely there's
other users out there who have servers that would similarly just break
if we switch the policy now.
I haven't had time to look deeply at the lockdep thing you wrote
about, but from a first glance, couldn't we fix it at the source?
flush_bg_queue() calls ->send_req() while holding the fch->bg_lock
which is what creates the bg_lock -> queue->lock deadlock. If it
instead only does the background accounting under the lock and moves
the requests to a caller-provided list where the caller only sends
*after* dropping the bg_lock, doesn't that solve the deadlock? This
would fix it for every server regardless of whether it sent
FUSE_OVER_IO_URING or not. I'll try to get some time to look at this
next week.
Thanks,
Joanne
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-20 17:46 ` Joanne Koong
@ 2026-08-20 18:27 ` Bernd Schubert
2026-08-21 3:38 ` Baokun Li
1 sibling, 0 replies; 27+ messages in thread
From: Bernd Schubert @ 2026-08-20 18:27 UTC (permalink / raw)
To: Joanne Koong
Cc: Baokun Li, Miklos Szeredi, jlayton, axboe, amir73il, fuse-devel
On 8/20/26 19:46, Joanne Koong wrote:
> On Thu, Aug 20, 2026 at 10:20 AM Bernd Schubert <bernd@bsbernd.com> wrote:
>>
>>
>>
>> On 8/20/26 18:16, Joanne Koong wrote:
>>> On Thu, Aug 20, 2026 at 1:02 AM Baokun Li <libaokun@linux.alibaba.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> On 2026/8/20 04:05, Bernd Schubert wrote:
>>>>>
>>>>> On 8/19/26 19:56, Joanne Koong wrote:
>>>>>> On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>>>>>>> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>>>>>>>> Currently, the connection's fuse_ring is created lazily on the first
>>>>>>>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
>>>>>>>> thread per queue (one per CPU) and those threads issue their first
>>>>>>>> REGISTER command concurrently. They then race to create the single
>>>>>>>> per-connection fuse_ring, which required open-coded handling in
>>>>>>>> fuse_uring_create() to detect and protect against concurrent creations.
>>>>>>>>
>>>>>>>> Decouple fuse_ring creation from ent registration and move it to
>>>>>>>> FUSE_INIT reply processing after a server has negotiated and set
>>>>>>>> FUSE_OVER_IO_URING. The ring is published before the connection is
>>>>>>>> marked initialized. fuse_uring_register() no longer creates the ring and
>>>>>>>> it instead uses the ring set up at init time.
>>>>>>> I tested this with loraw (a "raw" loopback tester that doesn't use
>>>>>>> libfuse) and it fails with
>>>>>>>
>>>>>>> root@kvm:~# ./loraw -u /mnt/fuse
>>>>>>> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
>>>>>>>
>>>>>>> cqe->res is -22 (EINVAL).
>>>>>>>
>>>>>>> Attaching the reproducer. To compile:
>>>>>>>
>>>>>>> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
>>>>>>> gcc loraw.c -oloraw -luring
>>>>>>>
>>>>>> Thanks for attaching the repro.
>>>>>>
>>>>>> This is happening because this patch uses the FUSE_OVER_IO_URING init
>>>>>> reply as a signal that the ring should be created, but I missed that
>>>>>> the FUSE_OVER_IO_URING reply is *optional*.
>>>>>>
>>>>>> Prior to this patch, there's two scenarios:
>>>>>> a) server sets FUSE_OVER_IO_URING reply at init time - requests will
>>>>>> automatically block until fuse-io-uring is completely set up
>>>>>> b) server does not set FUSE_OVER_IO_URING but later sends uring
>>>>>> register request - requests will continue along /dev/fuse path until
>>>>>> fuse-io-uring is completely set up
>>>>>>
>>>>>> Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
>>>>>>
>>>>>> I think the best way to fix this is to have the ring creation happen
>>>>>> when the kernel receives the first io-uring command instead of at
>>>>>> FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
>>>>>> that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
>>>>> I don't think we should allow io-uring without FUSE_OVER_IO_URING and
>>>>> I really thought that was disabled.
>>>>
>>>> I share Bernd's concern here. Allowing io-uring without
>>>> FUSE_OVER_IO_URING means enabling a capability beyond what was
>>>> negotiated. We should honor the negotiated feature set, and print
>>>> the negotiated flags to dmesg at INIT time so issues like this are
>>>> easy to spot.
>>>
>>> Not sure if you missed this reply [1], but will copy and paste it here:
>>>
>>> This is pre-existing behavior that's been there since the beginning
>>> (kernel version 6.14). I don't think we can change this now, or
>>> it'll break backwards compatibility, like Miklos's loraw program.
>>>
>>
>> I think we need to discuss this. I had replied that the current
>> accidental scheme we
>>
>> - deadlock (lock order), with bg_lock being one issue, but I bet there
>> is more
>> - module option bypass
>> - bypass of what fuse-client/kernel announces
>>
>> I.e. if a fuse-server did implement the accidental scheme, it was broken
>> anyway.
>>
>> If wanted to be paranoid, we would switch to FUSE_OVER_IO_URING2 flag
>> and ignore FUSE_OVER_IO_URING. I hope you don't insist on allowing
>
> I don't think a new FUSE_OVER_IO_URING2 flag helps. If the kernel
> ignores FUSE_OVER_IO_URING as you describe, servers using older
> versions of libfuse will break. If it accepts both flags, it's
> behaviorally identical to just enforcing FUSE_OVER_IO_URING.
Well, setting a flag that is not announced by kernel is not ok
either. Yes, it is change in behavior, but it is debatable.
>
>> fuse-server to set flags that fuse-server doesn't even announce...
>>
>
> I think this is a call better left up to Miklos. I don't know how
> rigorously it is enforced in linux that nothing should break backwards
> compatibility. The code was released in March 2025 as part of kernel
> version 6.14, so it's been roughly a year and a half, which I guess
> isn't that long in the grand scheme of things, but if Miklos has his
> loraw server program that relies on this, it's probably likely there's
> other users out there who have servers that would similarly just break
> if we switch the policy now.
>
> I haven't had time to look deeply at the lockdep thing you wrote
> about, but from a first glance, couldn't we fix it at the source?
> flush_bg_queue() calls ->send_req() while holding the fch->bg_lock
> which is what creates the bg_lock -> queue->lock deadlock. If it
> instead only does the background accounting under the lock and moves
> the requests to a caller-provided list where the caller only sends
> *after* dropping the bg_lock, doesn't that solve the deadlock? This
> would fix it for every server regardless of whether it sent
> FUSE_OVER_IO_URING or not. I'll try to get some time to look at this
> next week.
I have a basic patch for that, but I fear testing will bring up ore
issues when we switch at run time.
Let's discuss in a few min.
Thanks,
Bernd
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-20 16:16 ` Joanne Koong
2026-08-20 17:20 ` Bernd Schubert
@ 2026-08-21 3:04 ` Baokun Li
1 sibling, 0 replies; 27+ messages in thread
From: Baokun Li @ 2026-08-21 3:04 UTC (permalink / raw)
To: Joanne Koong
Cc: Bernd Schubert, Miklos Szeredi, jlayton, axboe, amir73il,
fuse-devel
On 2026/8/21 00:16, Joanne Koong wrote:
> On Thu, Aug 20, 2026 at 1:02 AM Baokun Li <libaokun@linux.alibaba.com> wrote:
>> Hi all,
>>
>> On 2026/8/20 04:05, Bernd Schubert wrote:
>>> On 8/19/26 19:56, Joanne Koong wrote:
>>>> On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>>>>> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>>>>>> Currently, the connection's fuse_ring is created lazily on the first
>>>>>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
>>>>>> thread per queue (one per CPU) and those threads issue their first
>>>>>> REGISTER command concurrently. They then race to create the single
>>>>>> per-connection fuse_ring, which required open-coded handling in
>>>>>> fuse_uring_create() to detect and protect against concurrent creations.
>>>>>>
>>>>>> Decouple fuse_ring creation from ent registration and move it to
>>>>>> FUSE_INIT reply processing after a server has negotiated and set
>>>>>> FUSE_OVER_IO_URING. The ring is published before the connection is
>>>>>> marked initialized. fuse_uring_register() no longer creates the ring and
>>>>>> it instead uses the ring set up at init time.
>>>>> I tested this with loraw (a "raw" loopback tester that doesn't use
>>>>> libfuse) and it fails with
>>>>>
>>>>> root@kvm:~# ./loraw -u /mnt/fuse
>>>>> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
>>>>>
>>>>> cqe->res is -22 (EINVAL).
>>>>>
>>>>> Attaching the reproducer. To compile:
>>>>>
>>>>> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
>>>>> gcc loraw.c -oloraw -luring
>>>>>
>>>> Thanks for attaching the repro.
>>>>
>>>> This is happening because this patch uses the FUSE_OVER_IO_URING init
>>>> reply as a signal that the ring should be created, but I missed that
>>>> the FUSE_OVER_IO_URING reply is *optional*.
>>>>
>>>> Prior to this patch, there's two scenarios:
>>>> a) server sets FUSE_OVER_IO_URING reply at init time - requests will
>>>> automatically block until fuse-io-uring is completely set up
>>>> b) server does not set FUSE_OVER_IO_URING but later sends uring
>>>> register request - requests will continue along /dev/fuse path until
>>>> fuse-io-uring is completely set up
>>>>
>>>> Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
>>>>
>>>> I think the best way to fix this is to have the ring creation happen
>>>> when the kernel receives the first io-uring command instead of at
>>>> FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
>>>> that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
>>> I don't think we should allow io-uring without FUSE_OVER_IO_URING and
>>> I really thought that was disabled.
>> I share Bernd's concern here. Allowing io-uring without
>> FUSE_OVER_IO_URING means enabling a capability beyond what was
>> negotiated. We should honor the negotiated feature set, and print
>> the negotiated flags to dmesg at INIT time so issues like this are
>> easy to spot.
> Not sure if you missed this reply [1], but will copy and paste it here:
>
> This is pre-existing behavior that's been there since the beginning
> (kernel version 6.14). I don't think we can change this now, or
> it'll break backwards compatibility, like Miklos's loraw program.
Yes, I saw it, but pre-existing doesn't necessarily mean correct.
Enabling capabilities beyond what was negotiated easily introduces
inconsistencies.
In our recent hardening work we found that the fch->io_uring switch
causes anomalies in several mechanisms (mostly in our downstream tree),
which is why I replied in Bernd's thread — he raised the
enable_uring / fch->io_uring bypass there.
>>> <... checking the code ...>
>>>
>>> I'm on a ublk branch without your commits a applied, i.e. plain upstream 7.2 fuse
>>>
>>> fuse_uring_cmd()
>>> /* Once a connection has io-uring enabled on it, it can't be disabled */
>>> if (!enable_uring && !fch->io_uring) {
>>> pr_info_ratelimited("fuse-io-uring is disabled\n");
>>> return -EOPNOTSUPP;
>>> }
>> BTW, the current code clears fch->io_uring on REGISTER failure:
>>
>> if (err) {
>> fch->io_uring = 0;
>> wake_up_all(&fch->blocked_waitq);
>> return err;
>> }
>>
>> But by then other entries may have already registered successfully,
>> fiq->ops is switched to fuse_io_uring_ops, and requests are flowing
>> through the uring path. Clearing fch->io_uring here may reject all
>> subsequent io_uring commands with -EOPNOTSUPP while the data path
>> is still running on uring ops.
>>
> This is also pre-existing behavior. The check in fuse_uring_cmd() is a
> && not an ||. It'll only return -EOPNOTSUPP if admin disables
> enable_uring.
Yes, that's exactly the problem. Right now we have several flags
with unclear boundaries:
enable_uring: user-controlled, can be toggled at any time
fch->io_uring: set to 1 at INIT negotiation, but cleared to 0
on any entry registration failure — even if other entries
have already registered successfully and fiq->ops has been
switched to uring ops. The two states are inconsistent.
ring->ready: set when at least one entry has registered
I think the clean design would be: at INIT time, decide based on
enable_uring and the negotiated flags whether to initialize uring
and set fch->io_uring. After that, enable_uring is no longer
consulted — fch->io_uring is write-once. When ring->ready (at
least one entry registered), take the io_uring path; otherwise
fall back to the classic /dev/fuse path. This way each
variable has a single, clear semantic.
Thanks,
Baokun
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-20 17:20 ` Bernd Schubert
2026-08-20 17:46 ` Joanne Koong
@ 2026-08-21 3:24 ` Baokun Li
1 sibling, 0 replies; 27+ messages in thread
From: Baokun Li @ 2026-08-21 3:24 UTC (permalink / raw)
To: Bernd Schubert, Joanne Koong
Cc: Miklos Szeredi, jlayton, axboe, amir73il, fuse-devel
On 2026/8/21 01:20, Bernd Schubert wrote:
>
> On 8/20/26 18:16, Joanne Koong wrote:
>> On Thu, Aug 20, 2026 at 1:02 AM Baokun Li <libaokun@linux.alibaba.com> wrote:
>>> Hi all,
>>>
>>> On 2026/8/20 04:05, Bernd Schubert wrote:
>>>> On 8/19/26 19:56, Joanne Koong wrote:
>>>>> On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>>>>>> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>>>>>>> Currently, the connection's fuse_ring is created lazily on the first
>>>>>>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
>>>>>>> thread per queue (one per CPU) and those threads issue their first
>>>>>>> REGISTER command concurrently. They then race to create the single
>>>>>>> per-connection fuse_ring, which required open-coded handling in
>>>>>>> fuse_uring_create() to detect and protect against concurrent creations.
>>>>>>>
>>>>>>> Decouple fuse_ring creation from ent registration and move it to
>>>>>>> FUSE_INIT reply processing after a server has negotiated and set
>>>>>>> FUSE_OVER_IO_URING. The ring is published before the connection is
>>>>>>> marked initialized. fuse_uring_register() no longer creates the ring and
>>>>>>> it instead uses the ring set up at init time.
>>>>>> I tested this with loraw (a "raw" loopback tester that doesn't use
>>>>>> libfuse) and it fails with
>>>>>>
>>>>>> root@kvm:~# ./loraw -u /mnt/fuse
>>>>>> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
>>>>>>
>>>>>> cqe->res is -22 (EINVAL).
>>>>>>
>>>>>> Attaching the reproducer. To compile:
>>>>>>
>>>>>> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
>>>>>> gcc loraw.c -oloraw -luring
>>>>>>
>>>>> Thanks for attaching the repro.
>>>>>
>>>>> This is happening because this patch uses the FUSE_OVER_IO_URING init
>>>>> reply as a signal that the ring should be created, but I missed that
>>>>> the FUSE_OVER_IO_URING reply is *optional*.
>>>>>
>>>>> Prior to this patch, there's two scenarios:
>>>>> a) server sets FUSE_OVER_IO_URING reply at init time - requests will
>>>>> automatically block until fuse-io-uring is completely set up
>>>>> b) server does not set FUSE_OVER_IO_URING but later sends uring
>>>>> register request - requests will continue along /dev/fuse path until
>>>>> fuse-io-uring is completely set up
>>>>>
>>>>> Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
>>>>>
>>>>> I think the best way to fix this is to have the ring creation happen
>>>>> when the kernel receives the first io-uring command instead of at
>>>>> FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
>>>>> that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
>>>> I don't think we should allow io-uring without FUSE_OVER_IO_URING and
>>>> I really thought that was disabled.
>>> I share Bernd's concern here. Allowing io-uring without
>>> FUSE_OVER_IO_URING means enabling a capability beyond what was
>>> negotiated. We should honor the negotiated feature set, and print
>>> the negotiated flags to dmesg at INIT time so issues like this are
>>> easy to spot.
>> Not sure if you missed this reply [1], but will copy and paste it here:
>>
>> This is pre-existing behavior that's been there since the beginning
>> (kernel version 6.14). I don't think we can change this now, or
>> it'll break backwards compatibility, like Miklos's loraw program.
>>
> I think we need to discuss this. I had replied that the current
> accidental scheme we
>
> - deadlock (lock order), with bg_lock being one issue, but I bet there
> is more
> - module option bypass
> - bypass of what fuse-client/kernel announces
>
> I.e. if a fuse-server did implement the accidental scheme, it was broken
> anyway.
Agreed. A server that skips FUSE_OVER_IO_URING was never in a
well-defined state — no blocking guarantee, no lock ordering
protection, nothing. There is nothing to preserve compatibility
with.
Thanks,
Baokun
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration
2026-08-20 17:46 ` Joanne Koong
2026-08-20 18:27 ` Bernd Schubert
@ 2026-08-21 3:38 ` Baokun Li
1 sibling, 0 replies; 27+ messages in thread
From: Baokun Li @ 2026-08-21 3:38 UTC (permalink / raw)
To: Joanne Koong, Bernd Schubert
Cc: Miklos Szeredi, jlayton, axboe, amir73il, fuse-devel
On 2026/8/21 01:46, Joanne Koong wrote:
> On Thu, Aug 20, 2026 at 10:20 AM Bernd Schubert <bernd@bsbernd.com> wrote:
>>
>>
>> On 8/20/26 18:16, Joanne Koong wrote:
>>> On Thu, Aug 20, 2026 at 1:02 AM Baokun Li <libaokun@linux.alibaba.com> wrote:
>>>> Hi all,
>>>>
>>>> On 2026/8/20 04:05, Bernd Schubert wrote:
>>>>> On 8/19/26 19:56, Joanne Koong wrote:
>>>>>> On Wed, Aug 19, 2026 at 4:35 AM Miklos Szeredi <miklos@szeredi.hu> wrote:
>>>>>>> On Fri, 14 Aug 2026 at 21:00, Joanne Koong <joannelkoong@gmail.com> wrote:
>>>>>>>> Currently, the connection's fuse_ring is created lazily on the first
>>>>>>>> FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
>>>>>>>> thread per queue (one per CPU) and those threads issue their first
>>>>>>>> REGISTER command concurrently. They then race to create the single
>>>>>>>> per-connection fuse_ring, which required open-coded handling in
>>>>>>>> fuse_uring_create() to detect and protect against concurrent creations.
>>>>>>>>
>>>>>>>> Decouple fuse_ring creation from ent registration and move it to
>>>>>>>> FUSE_INIT reply processing after a server has negotiated and set
>>>>>>>> FUSE_OVER_IO_URING. The ring is published before the connection is
>>>>>>>> marked initialized. fuse_uring_register() no longer creates the ring and
>>>>>>>> it instead uses the ring set up at init time.
>>>>>>> I tested this with loraw (a "raw" loopback tester that doesn't use
>>>>>>> libfuse) and it fails with
>>>>>>>
>>>>>>> root@kvm:~# ./loraw -u /mnt/fuse
>>>>>>> loraw: loraw.c:1010: lo_start_uring: Assertion `!cqe->res' failed.
>>>>>>>
>>>>>>> cqe->res is -22 (EINVAL).
>>>>>>>
>>>>>>> Attaching the reproducer. To compile:
>>>>>>>
>>>>>>> cp $(KERNEL_TREE)/include/uapi/linux/fuse.h fuse_kernel.h
>>>>>>> gcc loraw.c -oloraw -luring
>>>>>>>
>>>>>> Thanks for attaching the repro.
>>>>>>
>>>>>> This is happening because this patch uses the FUSE_OVER_IO_URING init
>>>>>> reply as a signal that the ring should be created, but I missed that
>>>>>> the FUSE_OVER_IO_URING reply is *optional*.
>>>>>>
>>>>>> Prior to this patch, there's two scenarios:
>>>>>> a) server sets FUSE_OVER_IO_URING reply at init time - requests will
>>>>>> automatically block until fuse-io-uring is completely set up
>>>>>> b) server does not set FUSE_OVER_IO_URING but later sends uring
>>>>>> register request - requests will continue along /dev/fuse path until
>>>>>> fuse-io-uring is completely set up
>>>>>>
>>>>>> Libfuse sets FUSE_OVER_IO_URING in the reply, but the loraw.c server does not.
>>>>>>
>>>>>> I think the best way to fix this is to have the ring creation happen
>>>>>> when the kernel receives the first io-uring command instead of at
>>>>>> FUSE_INIT or at FUSE_IO_URING_CMD_REGISTER ent creation time, given
>>>>>> that FUSE_IO_URING_ADD_QUEUE needs the ring to exist:
>>>>> I don't think we should allow io-uring without FUSE_OVER_IO_URING and
>>>>> I really thought that was disabled.
>>>> I share Bernd's concern here. Allowing io-uring without
>>>> FUSE_OVER_IO_URING means enabling a capability beyond what was
>>>> negotiated. We should honor the negotiated feature set, and print
>>>> the negotiated flags to dmesg at INIT time so issues like this are
>>>> easy to spot.
>>> Not sure if you missed this reply [1], but will copy and paste it here:
>>>
>>> This is pre-existing behavior that's been there since the beginning
>>> (kernel version 6.14). I don't think we can change this now, or
>>> it'll break backwards compatibility, like Miklos's loraw program.
>>>
>> I think we need to discuss this. I had replied that the current
>> accidental scheme we
>>
>> - deadlock (lock order), with bg_lock being one issue, but I bet there
>> is more
>> - module option bypass
>> - bypass of what fuse-client/kernel announces
>>
>> I.e. if a fuse-server did implement the accidental scheme, it was broken
>> anyway.
>>
>> If wanted to be paranoid, we would switch to FUSE_OVER_IO_URING2 flag
>> and ignore FUSE_OVER_IO_URING. I hope you don't insist on allowing
> I don't think a new FUSE_OVER_IO_URING2 flag helps. If the kernel
> ignores FUSE_OVER_IO_URING as you describe, servers using older
> versions of libfuse will break. If it accepts both flags, it's
> behaviorally identical to just enforcing FUSE_OVER_IO_URING.
>
>> fuse-server to set flags that fuse-server doesn't even announce...
>>
> I think this is a call better left up to Miklos. I don't know how
> rigorously it is enforced in linux that nothing should break backwards
> compatibility. The code was released in March 2025 as part of kernel
> version 6.14, so it's been roughly a year and a half, which I guess
> isn't that long in the grand scheme of things, but if Miklos has his
> loraw server program that relies on this, it's probably likely there's
> other users out there who have servers that would similarly just break
> if we switch the policy now.
>
> I haven't had time to look deeply at the lockdep thing you wrote
> about, but from a first glance, couldn't we fix it at the source?
> flush_bg_queue() calls ->send_req() while holding the fch->bg_lock
> which is what creates the bg_lock -> queue->lock deadlock. If it
> instead only does the background accounting under the lock and moves
> the requests to a caller-provided list where the caller only sends
> *after* dropping the bg_lock, doesn't that solve the deadlock? This
> would fix it for every server regardless of whether it sent
> FUSE_OVER_IO_URING or not. I'll try to get some time to look at this
> next week.
Makes sense — separating dequeue from send eliminates the nesting
structurally.
Cheers,
Baokun
^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2026-08-21 3:38 UTC | newest]
Thread overview: 27+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-14 18:59 [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
2026-08-14 18:59 ` [PATCH v7 1/6] fuse: decouple fuse_ring creation from ent registration Joanne Koong
2026-08-19 11:34 ` Miklos Szeredi
2026-08-19 11:38 ` Bernd Schubert
2026-08-19 17:56 ` Joanne Koong
2026-08-19 20:05 ` Bernd Schubert
2026-08-19 20:29 ` Joanne Koong
2026-08-19 20:52 ` Bernd Schubert
2026-08-19 21:35 ` Bernd Schubert
2026-08-20 8:02 ` Baokun Li
2026-08-20 16:16 ` Joanne Koong
2026-08-20 17:20 ` Bernd Schubert
2026-08-20 17:46 ` Joanne Koong
2026-08-20 18:27 ` Bernd Schubert
2026-08-21 3:38 ` Baokun Li
2026-08-21 3:24 ` Baokun Li
2026-08-21 3:04 ` Baokun Li
2026-08-14 18:59 ` [PATCH v7 2/6] fuse: add FUSE_IO_URING_CMD_ADD_QUEUE Joanne Koong
2026-08-14 18:59 ` [PATCH v7 3/6] fuse: add io-uring buffer pools Joanne Koong
2026-08-14 18:59 ` [PATCH v7 4/6] fuse: support registered buffer pools in io-uring Joanne Koong
2026-08-17 10:15 ` Bernd Schubert
2026-08-14 18:59 ` [PATCH v7 5/6] fuse: add zero-copy over io-uring Joanne Koong
2026-08-17 13:05 ` Bernd Schubert
2026-08-14 18:59 ` [PATCH v7 6/6] docs: fuse: document io-uring buffer pool and zero-copy uapi Joanne Koong
2026-08-14 19:23 ` [PATCH v7 0/6] fuse: add io-uring buffer pools and zero-copy Joanne Koong
2026-08-17 15:29 ` Miklos Szeredi
2026-08-17 18:23 ` Jens Axboe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.