Re: [PATCH 1/3] fuse: add FUSE-over-io_uring enable opt and init

[Brian Song <hibriansong@gmail.com>]

On 8/19/25 7:23 PM, Brian Song wrote:
>
>
> On 8/19/25 6:26 PM, Bernd Schubert wrote:
>>
>>
>> On 8/19/25 03:15, Brian Song wrote:
>>>
>>>
>>> On 8/18/25 7:04 PM, Bernd Schubert wrote:
>>>>
>>>>
>>>> On 8/17/25 01:13, Brian Song wrote:
>>>>>
>>>>>
>>>>> On 8/14/25 11:46 PM, Brian Song wrote:
>>>>>> From: Brian Song <hibriansong@gmail.com>
>>>>>>
>>>>>> This patch adds a new export option for storage-export-daemon to
>>>>>> enable
>>>>>> or disable FUSE-over-io_uring via the switch io-uring=on|off (disable
>>>>>> by default). It also implements the protocol handshake with the Linux
>>>>>> kernel during the FUSE-over-io_uring initialization phase.
>>>>>>
>>>>>> See: https://docs.kernel.org/filesystems/fuse-io-uring.html
>>>>>>
>>>>>> The kernel documentation describes in detail how FUSE-over-io_uring
>>>>>> works. This patch implements the Initial SQE stage shown in
>>>>>> thediagram:
>>>>>> it initializes one queue per IOThread, each currently supporting a
>>>>>> single submission queue entry (SQE). When the FUSE driver sends the
>>>>>> first FUSE request (FUSE_INIT), storage-export-daemon calls
>>>>>> fuse_uring_start() to complete initialization, ultimately submitting
>>>>>> the SQE with the FUSE_IO_URING_CMD_REGISTER command to confirm
>>>>>> successful initialization with the kernel.
>>>>>>
>>>>>> Suggested-by: Kevin Wolf <kwolf@redhat.com>
>>>>>> Suggested-by: Stefan Hajnoczi <stefanha@redhat.com>
>>>>>> Signed-off-by: Brian Song <hibriansong@gmail.com>
>>>>>> ---
>>>>>>     block/export/fuse.c                  | 161 +++++++++++++++++++
>>>>>> +++++---
>>>>>>     docs/tools/qemu-storage-daemon.rst   |  11 +-
>>>>>>     qapi/block-export.json               |   5 +-
>>>>>>     storage-daemon/qemu-storage-daemon.c |   1 +
>>>>>>     util/fdmon-io_uring.c                |   5 +-
>>>>>>     5 files changed, 159 insertions(+), 24 deletions(-)
>>>>>>
>>>>>> diff --git a/block/export/fuse.c b/block/export/fuse.c
>>>>>> index c0ad4696ce..59fa79f486 100644
>>>>>> --- a/block/export/fuse.c
>>>>>> +++ b/block/export/fuse.c
>>>>>> @@ -48,6 +48,11 @@
>>>>>>     #include <linux/fs.h>
>>>>>>     #endif
>>>>>>
>>>>>> +#define FUSE_DEFAULT_MAX_PAGES_PER_REQ 32
>>>>>> +
>>>>>> +/* room needed in buffer to accommodate header */
>>>>>> +#define FUSE_BUFFER_HEADER_SIZE 0x1000
>>>>>> +
>>>>>>     /* Prevent overly long bounce buffer allocations */
>>>>>>     #define FUSE_MAX_READ_BYTES (MIN(BDRV_REQUEST_MAX_BYTES, 1 *
>>>>>> 1024 * 1024))
>>>>>>     /*
>>>>>> @@ -63,12 +68,31 @@
>>>>>>         (FUSE_MAX_WRITE_BYTES - FUSE_IN_PLACE_WRITE_BYTES)
>>>>>>
>>>>>>     typedef struct FuseExport FuseExport;
>>>>>> +typedef struct FuseQueue FuseQueue;
>>>>>> +
>>>>>> +typedef struct FuseRingEnt {
>>>>>> +    /* back pointer */
>>>>>> +    FuseQueue *q;
>>>>>> +
>>>>>> +    /* commit id of a fuse request */
>>>>>> +    uint64_t req_commit_id;
>>>>>> +
>>>>>> +    /* fuse request header and payload */
>>>>>> +    struct fuse_uring_req_header req_header;
>>>>>> +    void *op_payload;
>>>>>> +    size_t req_payload_sz;
>>>>>> +
>>>>>> +    /* The vector passed to the kernel */
>>>>>> +    struct iovec iov[2];
>>>>>> +
>>>>>> +    CqeHandler fuse_cqe_handler;
>>>>>> +} FuseRingEnt;
>>>>>>
>>>>>>     /*
>>>>>>      * One FUSE "queue", representing one FUSE FD from which
>>>>>> requests are fetched
>>>>>>      * and processed.  Each queue is tied to an AioContext.
>>>>>>      */
>>>>>> -typedef struct FuseQueue {
>>>>>> +struct FuseQueue {
>>>>>>         FuseExport *exp;
>>>>>>
>>>>>>         AioContext *ctx;
>>>>>> @@ -109,7 +133,12 @@ typedef struct FuseQueue {
>>>>>>          * Free this buffer with qemu_vfree().
>>>>>>          */
>>>>>>         void *spillover_buf;
>>>>>> -} FuseQueue;
>>>>>> +
>>>>>> +#ifdef CONFIG_LINUX_IO_URING
>>>>>> +    int qid;
>>>>>> +    FuseRingEnt ent;
>>>>>> +#endif
>>>>>> +};
>>>>>>
>>>>>>     /*
>>>>>>      * Verify that FuseQueue.request_buf plus the spill-over
>>>>>> buffer together
>>>>>> @@ -148,6 +177,7 @@ struct FuseExport {
>>>>>>         bool growable;
>>>>>>         /* Whether allow_other was used as a mount option or not */
>>>>>>         bool allow_other;
>>>>>> +    bool is_uring;
>>>>>>
>>>>>>         mode_t st_mode;
>>>>>>         uid_t st_uid;
>>>>>> @@ -257,6 +287,93 @@ static const BlockDevOps
>>>>>> fuse_export_blk_dev_ops = {
>>>>>>         .drained_poll  = fuse_export_drained_poll,
>>>>>>     };
>>>>>>
>>>>>> +#ifdef CONFIG_LINUX_IO_URING
>>>>>> +
>>>>>> +static void fuse_uring_sqe_set_req_data(struct fuse_uring_cmd_req
>>>>>> *req,
>>>>>> +                    const unsigned int qid,
>>>>>> +                    const unsigned int commit_id)
>>>>>> +{
>>>>>> +    req->qid = qid;
>>>>>> +    req->commit_id = commit_id;
>>>>>> +    req->flags = 0;
>>>>>> +}
>>>>>> +
>>>>>> +static void fuse_uring_sqe_prepare(struct io_uring_sqe *sqe,
>>>>>> FuseQueue *q,
>>>>>> +               __u32 cmd_op)
>>>>>> +{
>>>>>> +    sqe->opcode = IORING_OP_URING_CMD;
>>>>>> +
>>>>>> +    sqe->fd = q->fuse_fd;
>>>>>> +    sqe->rw_flags = 0;
>>>>>> +    sqe->ioprio = 0;
>>>>>> +    sqe->off = 0;
>>>>>> +
>>>>>> +    sqe->cmd_op = cmd_op;
>>>>>> +    sqe->__pad1 = 0;
>>>>>> +}
>>>>>> +
>>>>>> +static void fuse_uring_prep_sqe_register(struct io_uring_sqe
>>>>>> *sqe, void *opaque)
>>>>>> +{
>>>>>> +    FuseQueue *q = opaque;
>>>>>> +    struct fuse_uring_cmd_req *req = (void *)&sqe->cmd[0];
>>>>>> +
>>>>>> +    fuse_uring_sqe_prepare(sqe, q, FUSE_IO_URING_CMD_REGISTER);
>>>>>> +
>>>>>> +    sqe->addr = (uint64_t)(q->ent.iov);
>>>>>> +    sqe->len = 2;
>>>>>> +
>>>>>> +    fuse_uring_sqe_set_req_data(req, q->qid, 0);
>>>>>> +}
>>>>>> +
>>>>>> +static void fuse_uring_submit_register(void *opaque)
>>>>>> +{
>>>>>> +    FuseQueue *q = opaque;
>>>>>> +    FuseExport *exp = q->exp;
>>>>>> +
>>>>>> +
>>>>>> +    aio_add_sqe(fuse_uring_prep_sqe_register, q, &(q-
>>>>>> >ent.fuse_cqe_handler));
>>>>>
>>>>> I think there might be a tricky issue with the io_uring integration in
>>>>> QEMU. Currently, when the number of IOThreads goes above ~6 or 7,
>>>>> there’s a pretty high chance of a hang. I added some debug logging in
>>>>> the kernel’s fuse_uring_cmd() registration part, and noticed that the
>>>>> number of register calls is less than the total number of entries
>>>>> in the
>>>>> queue. In theory, we should be registering each entry for each queue.
>>>>
>>>> Did you also try to add logging at the top of fuse_uring_cmd()? I
>>>> wonder
>>>> if there is a start up race and if initial commands are just getting
>>>> refused. I had run into issues you are describing in some versions of
>>>> the -rfc patches, but thought that everything was fixed for that.
>>>> I.e. not excluded that there is still a kernel issue left.
>>>>
>>>> Thanks,
>>>> Bernd
>>>>
>>>>
>>>
>>> Yes. I added a printk at the beginning of fuse_uring_cmd(), another at
>>> the beginning of fuse_uring_register(), and one more at the end of
>>> fuse_uring_do_register(). Then I created and registered 20 queues, each
>>> with a single ring entry. It printed 37 times(diff every time) with
>>> opcode FUSE_IO_URING_CMD_REGISTER (would expect 20), and only 6 queues
>>> were registered successfully. The rest of fuse_uring_cmd (x31) exited
>>> inside the if (!fc->initialized) branch in fuse_uring_cmd()
>>>
>>> dmesg: https://gist.github.com/
>>> hibriansong/4eda6e7e92601df497282dcd56fd5470
>>
>> Thank you for the logs, could you try this?
>>
>> diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
>> index 2aa20707f40b..cea57ad5d3ab 100644
>> --- a/fs/fuse/dev_uring.c
>> +++ b/fs/fuse/dev_uring.c
>> @@ -1324,6 +1324,9 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd,
>> unsigned int issue_flags)
>>          if (!fc->connected)
>>                  return -ENOTCONN;
>> +       /* Matches smp_wmb() in fuse_set_initialized() */
>> +       smp_rmb();
>> +
>>          /*
>>           * fuse_uring_register() needs the ring to be initialized,
>>           * we need to know the max payload size
>>
>>
>>
>> Thanks,
>> Bernd
>
> I realized the issue actually comes from QEMU handling the FUSE_INIT
> request. After I processed outargs, I didn't send the response back to
> the kernel before starting the fuse-over-io_uring initialization. So
> it's possible that the 20 registration requests submitted via
> io_uring_cmd() reach the kernel before process_init_reply() has run and
> set fc->initialized = 1, which causes fuse_uring_cmd to bail out
> repeatedly.
>
> I also noticed that in libfuse, they first send the init request
> response, then allocate queues and submit the register SQEs. But even
> there, during the fuse-over-io_uring init after sending the response, if
> the kernel hasn't finished process_init_reply() and set fc->initialized
> = 1, wouldn't they run into a similar issue fuse_uring_cmd repeatedly
> bailing on register requests because fc->initialized isn't set yet?

Hi Bernd,

Nvm, I think writing to /dev/fuse fd is blocking.

Thanks so much for your feedback!

Best,
Brian