Fuse over io_uring mode cannot handle iodepth > 1 case properly like the default mode

Fuse over io_uring mode cannot handle iodepth > 1 case properly like the default mode

[Gang He]

Hi Bernd,

Sorry for interruption.
I tested your fuse over io_uring patch set with libfuse null example,
the fuse over io_uring mode has better performance than the default
mode. e.g., the fio command is as below,
fio -direct=1 --filename=/mnt/singfile --rw=read  -iodepth=1
--ioengine=libaio --bs=4k --size=4G --runtime=60 --numjobs=1
-name=test_fuse1

But, if I increased fio iodepth option, the fuse over io_uring mode
has worse performance than the default mode. e.g., the fio command is
as below,
fio -direct=1 --filename=/mnt/singfile --rw=read  -iodepth=4
--ioengine=libaio --bs=4k --size=4G --runtime=60 --numjobs=1
-name=test_fuse2

The test result showed the fuse over io_uring mode cannot handle this
case properly. could you take a look at this issue? or this is design
issue?

I went through the related source code, I do not understand each
fuse_ring_queue thread has only one  available ring entry? this design
will cause the above issue?
the related code is as follows,
dev_uring.c
1099
1100     queue = ring->queues[qid];
1101     if (!queue) {
1102         queue = fuse_uring_create_queue(ring, qid);
1103         if (!queue)
1104             return err;
1105     }
1106
1107     /*
1108      * The created queue above does not need to be destructed in
1109      * case of entry errors below, will be done at ring destruction time.
1110      */
1111
1112     ent = fuse_uring_create_ring_ent(cmd, queue);
1113     if (IS_ERR(ent))
1114         return PTR_ERR(ent);
1115
1116     fuse_uring_do_register(ent, cmd, issue_flags);
1117
1118     return 0;
1119 }


Thanks
Gang

Re: Fuse over io_uring mode cannot handle iodepth > 1 case properly like the default mode

[Bernd Schubert]

On August 15, 2025 9:45:34 AM GMT+02:00, Gang He <dchg2000@gmail.com> wrote:
>Hi Bernd,
>
>Sorry for interruption.
>I tested your fuse over io_uring patch set with libfuse null example,
>the fuse over io_uring mode has better performance than the default
>mode. e.g., the fio command is as below,
>fio -direct=1 --filename=/mnt/singfile --rw=read  -iodepth=1
>--ioengine=libaio --bs=4k --size=4G --runtime=60 --numjobs=1
>-name=test_fuse1
>
>But, if I increased fio iodepth option, the fuse over io_uring mode
>has worse performance than the default mode. e.g., the fio command is
>as below,
>fio -direct=1 --filename=/mnt/singfile --rw=read  -iodepth=4
>--ioengine=libaio --bs=4k --size=4G --runtime=60 --numjobs=1
>-name=test_fuse2
>
>The test result showed the fuse over io_uring mode cannot handle this
>case properly. could you take a look at this issue? or this is design
>issue?
>
>I went through the related source code, I do not understand each
>fuse_ring_queue thread has only one  available ring entry? this design
>will cause the above issue?
>the related code is as follows,
>dev_uring.c
>1099
>1100     queue = ring->queues[qid];
>1101     if (!queue) {
>1102         queue = fuse_uring_create_queue(ring, qid);
>1103         if (!queue)
>1104             return err;
>1105     }
>1106
>1107     /*
>1108      * The created queue above does not need to be destructed in
>1109      * case of entry errors below, will be done at ring destruction time.
>1110      */
>1111
>1112     ent = fuse_uring_create_ring_ent(cmd, queue);
>1113     if (IS_ERR(ent))
>1114         return PTR_ERR(ent);
>1115
>1116     fuse_uring_do_register(ent, cmd, issue_flags);
>1117
>1118     return 0;
>1119 }
>
>
>Thanks
>Gang


Hi Gang,

we are just slowly traveling back with my family from Germany to France - sorry for delayed responses.

Each queue can have up to N ring entries - I think I put in max 65535.

The code you are looking at will just add new entries to per queue lists.

I don't know why higher fio io-depth results in lower performance. A possible reason is that /dev/fuse request get distributed to multiple threads, while fuse-io-uring might all go the same thread/ring. I had posted patches recently that add request  balancing between queues. 

Cheers,
Bernd



Cheers,
Bernd

Re: Fuse over io_uring mode cannot handle iodepth > 1 case properly like the default mode

[Gang He]

Hi Bernd,

Bernd Schubert <bernd@bsbernd.com> 于2025年8月16日周六 04:56写道：
>
> On August 15, 2025 9:45:34 AM GMT+02:00, Gang He <dchg2000@gmail.com> wrote:
> >Hi Bernd,
> >
> >Sorry for interruption.
> >I tested your fuse over io_uring patch set with libfuse null example,
> >the fuse over io_uring mode has better performance than the default
> >mode. e.g., the fio command is as below,
> >fio -direct=1 --filename=/mnt/singfile --rw=read  -iodepth=1
> >--ioengine=libaio --bs=4k --size=4G --runtime=60 --numjobs=1
> >-name=test_fuse1
> >
> >But, if I increased fio iodepth option, the fuse over io_uring mode
> >has worse performance than the default mode. e.g., the fio command is
> >as below,
> >fio -direct=1 --filename=/mnt/singfile --rw=read  -iodepth=4
> >--ioengine=libaio --bs=4k --size=4G --runtime=60 --numjobs=1
> >-name=test_fuse2
> >
> >The test result showed the fuse over io_uring mode cannot handle this
> >case properly. could you take a look at this issue? or this is design
> >issue?
> >
> >I went through the related source code, I do not understand each
> >fuse_ring_queue thread has only one  available ring entry? this design
> >will cause the above issue?
> >the related code is as follows,
> >dev_uring.c
> >1099
> >1100     queue = ring->queues[qid];
> >1101     if (!queue) {
> >1102         queue = fuse_uring_create_queue(ring, qid);
> >1103         if (!queue)
> >1104             return err;
> >1105     }
> >1106
> >1107     /*
> >1108      * The created queue above does not need to be destructed in
> >1109      * case of entry errors below, will be done at ring destruction time.
> >1110      */
> >1111
> >1112     ent = fuse_uring_create_ring_ent(cmd, queue);
> >1113     if (IS_ERR(ent))
> >1114         return PTR_ERR(ent);
> >1115
> >1116     fuse_uring_do_register(ent, cmd, issue_flags);
> >1117
> >1118     return 0;
> >1119 }
> >
> >
> >Thanks
> >Gang
>
>
> Hi Gang,
>
> we are just slowly traveling back with my family from Germany to France - sorry for delayed responses.
>
> Each queue can have up to N ring entries - I think I put in max 65535.
>
> The code you are looking at will just add new entries to per queue lists.
>
> I don't know why higher fio io-depth results in lower performance. A possible reason is that /dev/fuse request get distributed to multiple threads, while fuse-io-uring might all go the same thread/ring. I had posted patches recently that add request  balancing between queues.
Io-depth > 1 case means asynchronous IO implementation, but from the
code in the fuse_uring_commit_fetch() function, this function
completes one IO request, then fetches the next request. This logic
will block handling more IO requests before the last request is being
processed in this thread. Can each thread accept more IO requests
before the last request in the thread is being processed? Maybe this
is the root cause for fio (iodepth>1) test case.

Thanks
Gang


>
> Cheers,
> Bernd
>
>
>
> Cheers,
> Bernd

Re: Fuse over io_uring mode cannot handle iodepth > 1 case properly like the default mode

[Bernd Schubert]

On 8/18/25 03:39, Gang He wrote:
> Hi Bernd,
> 
> Bernd Schubert <bernd@bsbernd.com> 于2025年8月16日周六 04:56写道：
>>
>> On August 15, 2025 9:45:34 AM GMT+02:00, Gang He <dchg2000@gmail.com> wrote:
>>> Hi Bernd,
>>>
>>> Sorry for interruption.
>>> I tested your fuse over io_uring patch set with libfuse null example,
>>> the fuse over io_uring mode has better performance than the default
>>> mode. e.g., the fio command is as below,
>>> fio -direct=1 --filename=/mnt/singfile --rw=read  -iodepth=1
>>> --ioengine=libaio --bs=4k --size=4G --runtime=60 --numjobs=1
>>> -name=test_fuse1
>>>
>>> But, if I increased fio iodepth option, the fuse over io_uring mode
>>> has worse performance than the default mode. e.g., the fio command is
>>> as below,
>>> fio -direct=1 --filename=/mnt/singfile --rw=read  -iodepth=4
>>> --ioengine=libaio --bs=4k --size=4G --runtime=60 --numjobs=1
>>> -name=test_fuse2
>>>
>>> The test result showed the fuse over io_uring mode cannot handle this
>>> case properly. could you take a look at this issue? or this is design
>>> issue?
>>>
>>> I went through the related source code, I do not understand each
>>> fuse_ring_queue thread has only one  available ring entry? this design
>>> will cause the above issue?
>>> the related code is as follows,
>>> dev_uring.c
>>> 1099
>>> 1100     queue = ring->queues[qid];
>>> 1101     if (!queue) {
>>> 1102         queue = fuse_uring_create_queue(ring, qid);
>>> 1103         if (!queue)
>>> 1104             return err;
>>> 1105     }
>>> 1106
>>> 1107     /*
>>> 1108      * The created queue above does not need to be destructed in
>>> 1109      * case of entry errors below, will be done at ring destruction time.
>>> 1110      */
>>> 1111
>>> 1112     ent = fuse_uring_create_ring_ent(cmd, queue);
>>> 1113     if (IS_ERR(ent))
>>> 1114         return PTR_ERR(ent);
>>> 1115
>>> 1116     fuse_uring_do_register(ent, cmd, issue_flags);
>>> 1117
>>> 1118     return 0;
>>> 1119 }
>>>
>>>
>>> Thanks
>>> Gang
>>
>>
>> Hi Gang,
>>
>> we are just slowly traveling back with my family from Germany to France - sorry for delayed responses.
>>
>> Each queue can have up to N ring entries - I think I put in max 65535.
>>
>> The code you are looking at will just add new entries to per queue lists.
>>
>> I don't know why higher fio io-depth results in lower performance. A possible reason is that /dev/fuse request get distributed to multiple threads, while fuse-io-uring might all go the same thread/ring. I had posted patches recently that add request  balancing between queues.
> Io-depth > 1 case means asynchronous IO implementation, but from the
> code in the fuse_uring_commit_fetch() function, this function
> completes one IO request, then fetches the next request. This logic
> will block handling more IO requests before the last request is being
> processed in this thread. Can each thread accept more IO requests
> before the last request in the thread is being processed? Maybe this
> is the root cause for fio (iodepth>1) test case.


Well, there is a missing io-uring kernel feature - io_uring_cmd_done() 
can only complete one SQE at a time. There is no way right now
to to batch multiple "struct io_uring_cmd". Although I personally
doubt that this is the limit you are running into.

Re: Fuse over io_uring mode cannot handle iodepth > 1 case properly like the default mode

[Gang He]

Hi Bernd,

Bernd Schubert <bernd@bsbernd.com> 于2025年8月19日周二 01:31写道：
>
>
>
> On 8/18/25 03:39, Gang He wrote:
> > Hi Bernd,
> >
> > Bernd Schubert <bernd@bsbernd.com> 于2025年8月16日周六 04:56写道：
> >>
> >> On August 15, 2025 9:45:34 AM GMT+02:00, Gang He <dchg2000@gmail.com> wrote:
> >>> Hi Bernd,
> >>>
> >>> Sorry for interruption.
> >>> I tested your fuse over io_uring patch set with libfuse null example,
> >>> the fuse over io_uring mode has better performance than the default
> >>> mode. e.g., the fio command is as below,
> >>> fio -direct=1 --filename=/mnt/singfile --rw=read  -iodepth=1
> >>> --ioengine=libaio --bs=4k --size=4G --runtime=60 --numjobs=1
> >>> -name=test_fuse1
> >>>
> >>> But, if I increased fio iodepth option, the fuse over io_uring mode
> >>> has worse performance than the default mode. e.g., the fio command is
> >>> as below,
> >>> fio -direct=1 --filename=/mnt/singfile --rw=read  -iodepth=4
> >>> --ioengine=libaio --bs=4k --size=4G --runtime=60 --numjobs=1
> >>> -name=test_fuse2
> >>>
> >>> The test result showed the fuse over io_uring mode cannot handle this
> >>> case properly. could you take a look at this issue? or this is design
> >>> issue?
> >>>
> >>> I went through the related source code, I do not understand each
> >>> fuse_ring_queue thread has only one  available ring entry? this design
> >>> will cause the above issue?
> >>> the related code is as follows,
> >>> dev_uring.c
> >>> 1099
> >>> 1100     queue = ring->queues[qid];
> >>> 1101     if (!queue) {
> >>> 1102         queue = fuse_uring_create_queue(ring, qid);
> >>> 1103         if (!queue)
> >>> 1104             return err;
> >>> 1105     }
> >>> 1106
> >>> 1107     /*
> >>> 1108      * The created queue above does not need to be destructed in
> >>> 1109      * case of entry errors below, will be done at ring destruction time.
> >>> 1110      */
> >>> 1111
> >>> 1112     ent = fuse_uring_create_ring_ent(cmd, queue);
> >>> 1113     if (IS_ERR(ent))
> >>> 1114         return PTR_ERR(ent);
> >>> 1115
> >>> 1116     fuse_uring_do_register(ent, cmd, issue_flags);
> >>> 1117
> >>> 1118     return 0;
> >>> 1119 }
> >>>
> >>>
> >>> Thanks
> >>> Gang
> >>
> >>
> >> Hi Gang,
> >>
> >> we are just slowly traveling back with my family from Germany to France - sorry for delayed responses.
> >>
> >> Each queue can have up to N ring entries - I think I put in max 65535.
> >>
> >> The code you are looking at will just add new entries to per queue lists.
> >>
> >> I don't know why higher fio io-depth results in lower performance. A possible reason is that /dev/fuse request get distributed to multiple threads, while fuse-io-uring might all go the same thread/ring. I had posted patches recently that add request  balancing between queues.
> > Io-depth > 1 case means asynchronous IO implementation, but from the
> > code in the fuse_uring_commit_fetch() function, this function
> > completes one IO request, then fetches the next request. This logic
> > will block handling more IO requests before the last request is being
> > processed in this thread. Can each thread accept more IO requests
> > before the last request in the thread is being processed? Maybe this
> > is the root cause for fio (iodepth>1) test case.
>
>
> Well, there is a missing io-uring kernel feature - io_uring_cmd_done()
> can only complete one SQE at a time. There is no way right now
> to to batch multiple "struct io_uring_cmd". Although I personally
> doubt that this is the limit you are running into.
OK, I got this design background, but I want to know if we can handle
the next fuse request immediately after the
io_uring_cmd_done() function is called in the kernel space, rather
than in the fuse_uring_commit_fetch() function.
I do not know my suggestion makes sense in the io_uring mechanism, but
from the coding view, we should handle
the next fuse request as soon as possible.
Second, if we cannot change this design, we should send the fuse
request to another thread queue when the current thread queue is
handling the fuse request, but maybe this change will bring
performance drop in the numa case for the single thread fio testing.

Thanks
Gang

Re: Fuse over io_uring mode cannot handle iodepth > 1 case properly like the default mode

[Bernd Schubert]

[Added back fsdevel into CC]

Hi Gang,

On 9/18/25 05:05, Gang He wrote:
> Hi Bernd,
> 
> Sorry for interruption again.
> About enable fuse over io_uring feature, I can back-port the related
> kernel patch set.
> But, for user-space libfuse, when will you or the team release a
> tag(e.g. 3.18)? then I can upgrade the libfuse directly, rather than
> back-port patches.
> Second, Fuse over io_uring mode cannot handle iodepth > 1 case
> quickly, I feel this is a by-design issue, the iouring uses the own
> thread queue to handle the requests from the user space. I write a
> kernel patch, which can fix this case, but for the most cases, we
> still use the current design. the patch is attached, could you take a
> look at it, to see if this patch make sense, or not.


I will try to find time to release libfuse-3.18 today, one io-uring
related patch is missing (re-init of some fields in struct fuse_req) and
then just the release process.

Regarding queue balancing, please have a look here:

https://github.com/bsbernd/linux/commits/reduced-nr-ring-queues_3.1/


The tricky part and which is why I didn't publish v2 of this series is
to keep performance for blocking/sync requests (like DIO, metadata,
etc). At DDN we run with an additional workaround patch that disables
migration in fuse_request_end(), which results in kind of
wake_up_on_current_cpu(). Adding in wake_up_on_current_cpu() to fuse is
simple, I have the patch, problem is more that wake_up_on_current_cpu()
is not perfect. I'm just in the middle to walk through scheduler code.
For example, wake_up_on_current_cpu() gets perfect after increasing
/sys/kernel/debug/sched/migration_cost_ns
Now I don't want to change system wide migration costs, but it needs to
be time limited change when waking up from fuse. Additionally it needs
to be a hint, depending on if the queue has more pending work - that is
the part that gets even more complex with reduced queues and queue
balancing.

Also see https://lkml.org/lkml/2023/5/3/646 and related threads for
reference.

Thanks,
Bernd

Re: Fuse over io_uring mode cannot handle iodepth > 1 case properly like the default mode

[Bernd Schubert]

On 9/18/25 10:38, Bernd Schubert wrote:
> [Added back fsdevel into CC]
> 

I'm getting there, upcoming patch as part of the queue reduction series:

fuse: {io-uring} Queue background requests on a different core

Running background IO on a different core makes quite a difference.

fio --directory=/tmp/dest --name=iops.\$jobnum --rw=randread \
--bs=4k --size=1G --numjobs=1 --iodepth=4 --time_based\
--runtime=30s --group_reporting --ioengine=io_uring\
 --direct=1

unpatched
   READ: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s) ...

patched
Run status group 0 (all jobs):
   READ: bw=674MiB/s (707MB/s), 674MiB/s-674MiB/s (707MB/s-707MB/s) ...


fuse-over-io-uring RFC v1 and v2 versions had

https://lore.kernel.org/all/20240529-fuse-uring-for-6-9-rfc2-out-v1-17-d149476b1d65@ddn.com/

I dropped it on request as was an optimization hack. I had
planned to add it back earlier, but was totally occupied
from January to beginning of September...



Thanks,
Bernd

Re: Fuse over io_uring mode cannot handle iodepth > 1 case properly like the default mode

[Gang He]

Hi Bernd,

Great to hear this information.
One question,
Does this change impact the existing performance advantages？
e.g.,
Single-threaded synchronous IO read and write？

Thanks
Gang

Bernd Schubert <bernd@bsbernd.com> 于2025年9月25日周四 05:09写道：
>
>
>
> On 9/18/25 10:38, Bernd Schubert wrote:
> > [Added back fsdevel into CC]
> >
>
> I'm getting there, upcoming patch as part of the queue reduction series:
>
> fuse: {io-uring} Queue background requests on a different core
>
> Running background IO on a different core makes quite a difference.
>
> fio --directory=/tmp/dest --name=iops.\$jobnum --rw=randread \
> --bs=4k --size=1G --numjobs=1 --iodepth=4 --time_based\
> --runtime=30s --group_reporting --ioengine=io_uring\
>  --direct=1
>
> unpatched
>    READ: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s) ...
>
> patched
> Run status group 0 (all jobs):
>    READ: bw=674MiB/s (707MB/s), 674MiB/s-674MiB/s (707MB/s-707MB/s) ...
>
>
> fuse-over-io-uring RFC v1 and v2 versions had
>
> https://lore.kernel.org/all/20240529-fuse-uring-for-6-9-rfc2-out-v1-17-d149476b1d65@ddn.com/
>
> I dropped it on request as was an optimization hack. I had
> planned to add it back earlier, but was totally occupied
> from January to beginning of September...
>
>
>
> Thanks,
> Bernd