FUSE: [Regression] Fuse legacy path performance scaling lost in v6.14 vs v6.8/6.11 (iodepth scaling with io_uring)

FUSE: [Regression] Fuse legacy path performance scaling lost in v6.14 vs v6.8/6.11 (iodepth scaling with io_uring)

[Abhishek Gupta]

Hello Team,

I am observing a performance regression in the FUSE subsystem on
Kernel 6.14 compared to 6.8/6.11 when using the legacy/standard FUSE
interface (userspace daemon using standard read on /dev/fuse).

Summary of Issue: On Kernel 6.8 & 6.11, increasing iodepth in fio
(using ioengine=io_uring) results in near-linear performance scaling.
On Kernel 6.14, using the exact same userspace binary, increasing
iodepth yields no performance improvement (behavior resembles
iodepth=1).

Environment:
- Workload: GCSFuse (userspace daemon) + Fio
- Fio Config: Random Read, ioengine=io_uring, direct=1, iodepth=4.
- CPU: Intel.
- Daemon: Go-based. It uses a serialized reader loop on /dev/fuse that
immediately spawns a Go routine per request. So, it can serve requests
in parallel.
- Kernel Config: CONFIG_FUSE_IO_URING=y is enabled, but the daemon is
not registering for the ring (legacy mode).

Benchmark Observations:
- Kernel 6.8/6.11: With iodepth=4, we observe ~3.5-4x throughput
compared to iodepth=1.
- Kernel 6.14: With iodepth=4, throughput is identical to iodepth=1.
Parallelism is effectively lost.

Is this a known issue? I would appreciate any insights or pointers on
this issue.

Thanks & Regards,
Abhishek

Re: FUSE: [Regression] Fuse legacy path performance scaling lost in v6.14 vs v6.8/6.11 (iodepth scaling with io_uring)

[Bernd Schubert]

Hi Abhishek,

On 11/26/25 16:07, Abhishek Gupta wrote:
> [You don't often get email from abhishekmgupta@google.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> Hello Team,
> 
> I am observing a performance regression in the FUSE subsystem on
> Kernel 6.14 compared to 6.8/6.11 when using the legacy/standard FUSE
> interface (userspace daemon using standard read on /dev/fuse).
> 
> Summary of Issue: On Kernel 6.8 & 6.11, increasing iodepth in fio
> (using ioengine=io_uring) results in near-linear performance scaling.
> On Kernel 6.14, using the exact same userspace binary, increasing
> iodepth yields no performance improvement (behavior resembles
> iodepth=1).
> 
> Environment:
> - Workload: GCSFuse (userspace daemon) + Fio
> - Fio Config: Random Read, ioengine=io_uring, direct=1, iodepth=4.
> - CPU: Intel.
> - Daemon: Go-based. It uses a serialized reader loop on /dev/fuse that
> immediately spawns a Go routine per request. So, it can serve requests
> in parallel.
> - Kernel Config: CONFIG_FUSE_IO_URING=y is enabled, but the daemon is
> not registering for the ring (legacy mode).
> 
> Benchmark Observations:
> - Kernel 6.8/6.11: With iodepth=4, we observe ~3.5-4x throughput
> compared to iodepth=1.
> - Kernel 6.14: With iodepth=4, throughput is identical to iodepth=1.
> Parallelism is effectively lost.
> 
> Is this a known issue? I would appreciate any insights or pointers on
> this issue.

Could you give your exact fio line? I'm not aware of such a regression.

bschubert2@imesrv3 ~>fio --directory=/tmp/dest --name=iops.\$jobnum --rw=randread --bs=4k --size=1G --numjobs=1 --iodepth=1 --time_based --runtime=30s --group_reporting --ioengine=io_uring  --direct=1
iops.$jobnum: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.36
Starting 1 process
iops.$jobnum: Laying out IO file (1 file / 1024MiB)
...
Run status group 0 (all jobs):
    READ: bw=178MiB/s (186MB/s), 178MiB/s-178MiB/s (186MB/s-186MB/s), io=5331MiB (5590MB), run=30001-30001msec

bschubert2@imesrv3 ~>fio --directory=/tmp/dest --name=iops.\$jobnum --rw=randread --bs=4k --size=1G --numjobs=1 --iodepth=4 --time_based --runtime=30s --group_reporting --ioengine=io_uring  --direct=1
iops.$jobnum: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
fio-3.36
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=673MiB/s][r=172k IOPS][eta 00m:00s]
iops.$jobnum: (groupid=0, jobs=1): err= 0: pid=52012: Wed Nov 26 20:08:17 2025
...
Run status group 0 (all jobs):
   READ: bw=673MiB/s (706MB/s), 673MiB/s-673MiB/s (706MB/s-706MB/s), io=19.7GiB (21.2GB), run=30001-30001msec


This is with libfuse `example/passthrough_hp -o allow_other --nopassthrough --foreground /tmp/source /tmp/dest`


Thanks,
Bernd

Re: FUSE: [Regression] Fuse legacy path performance scaling lost in v6.14 vs v6.8/6.11 (iodepth scaling with io_uring)

[Abhishek Gupta]

Hi Bernd,

Thanks for looking into this.
Please find below the fio output on 6.11 & 6.14 kernel versions.


On kernel 6.11

~/gcsfuse$ uname -a
Linux abhishek-c4-192-west4a 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP
Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

iodepth = 1
:~/fio-fio-3.38$ ./fio --name=randread --rw=randread
--ioengine=io_uring --thread
--filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
--filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
--iodepth=1 --group_reporting=1 --direct=1
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.38
Starting 1 thread
...
Run status group 0 (all jobs):
   READ: bw=3311KiB/s (3391kB/s), 3311KiB/s-3311KiB/s
(3391kB/s-3391kB/s), io=48.5MiB (50.9MB), run=15001-15001msec

iodepth=4
:~/fio-fio-3.38$ ./fio --name=randread --rw=randread
--ioengine=io_uring --thread
--filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
--filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
--iodepth=4 --group_reporting=1 --direct=1
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=io_uring, iodepth=4
fio-3.38
Starting 1 thread
...
Run status group 0 (all jobs):
   READ: bw=11.0MiB/s (11.6MB/s), 11.0MiB/s-11.0MiB/s
(11.6MB/s-11.6MB/s), io=166MiB (174MB), run=15002-15002msec


On kernel 6.14

:~$ uname -a
Linux abhishek-west4a-2504 6.14.0-1019-gcp #20-Ubuntu SMP Wed Oct 15
00:41:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

iodepth=1
:~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
--filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
--filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
--iodepth=1 --group_reporting=1 --direct=1
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.38
Starting 1 thread
...
Run status group 0 (all jobs):
   READ: bw=3576KiB/s (3662kB/s), 3576KiB/s-3576KiB/s
(3662kB/s-3662kB/s), io=52.4MiB (54.9MB), run=15001-15001msec

iodepth=4
:~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
--filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
--filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
--iodepth=4 --group_reporting=1 --direct=1
randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=io_uring, iodepth=4
fio-3.38
...
Run status group 0 (all jobs):
   READ: bw=3863KiB/s (3956kB/s), 3863KiB/s-3863KiB/s
(3956kB/s-3956kB/s), io=56.6MiB (59.3MB), run=15001-15001msec

Thanks,
Abhishek


On Thu, Nov 27, 2025 at 12:41 AM Bernd Schubert <bschubert@ddn.com> wrote:
>
> Hi Abhishek,
>
> On 11/26/25 16:07, Abhishek Gupta wrote:
> > [You don't often get email from abhishekmgupta@google.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> >
> > Hello Team,
> >
> > I am observing a performance regression in the FUSE subsystem on
> > Kernel 6.14 compared to 6.8/6.11 when using the legacy/standard FUSE
> > interface (userspace daemon using standard read on /dev/fuse).
> >
> > Summary of Issue: On Kernel 6.8 & 6.11, increasing iodepth in fio
> > (using ioengine=io_uring) results in near-linear performance scaling.
> > On Kernel 6.14, using the exact same userspace binary, increasing
> > iodepth yields no performance improvement (behavior resembles
> > iodepth=1).
> >
> > Environment:
> > - Workload: GCSFuse (userspace daemon) + Fio
> > - Fio Config: Random Read, ioengine=io_uring, direct=1, iodepth=4.
> > - CPU: Intel.
> > - Daemon: Go-based. It uses a serialized reader loop on /dev/fuse that
> > immediately spawns a Go routine per request. So, it can serve requests
> > in parallel.
> > - Kernel Config: CONFIG_FUSE_IO_URING=y is enabled, but the daemon is
> > not registering for the ring (legacy mode).
> >
> > Benchmark Observations:
> > - Kernel 6.8/6.11: With iodepth=4, we observe ~3.5-4x throughput
> > compared to iodepth=1.
> > - Kernel 6.14: With iodepth=4, throughput is identical to iodepth=1.
> > Parallelism is effectively lost.
> >
> > Is this a known issue? I would appreciate any insights or pointers on
> > this issue.
>
> Could you give your exact fio line? I'm not aware of such a regression.
>
> bschubert2@imesrv3 ~>fio --directory=/tmp/dest --name=iops.\$jobnum --rw=randread --bs=4k --size=1G --numjobs=1 --iodepth=1 --time_based --runtime=30s --group_reporting --ioengine=io_uring  --direct=1
> iops.$jobnum: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
> fio-3.36
> Starting 1 process
> iops.$jobnum: Laying out IO file (1 file / 1024MiB)
> ...
> Run status group 0 (all jobs):
>     READ: bw=178MiB/s (186MB/s), 178MiB/s-178MiB/s (186MB/s-186MB/s), io=5331MiB (5590MB), run=30001-30001msec
>
> bschubert2@imesrv3 ~>fio --directory=/tmp/dest --name=iops.\$jobnum --rw=randread --bs=4k --size=1G --numjobs=1 --iodepth=4 --time_based --runtime=30s --group_reporting --ioengine=io_uring  --direct=1
> iops.$jobnum: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
> fio-3.36
> Starting 1 process
> Jobs: 1 (f=1): [r(1)][100.0%][r=673MiB/s][r=172k IOPS][eta 00m:00s]
> iops.$jobnum: (groupid=0, jobs=1): err= 0: pid=52012: Wed Nov 26 20:08:17 2025
> ...
> Run status group 0 (all jobs):
>    READ: bw=673MiB/s (706MB/s), 673MiB/s-673MiB/s (706MB/s-706MB/s), io=19.7GiB (21.2GB), run=30001-30001msec
>
>
> This is with libfuse `example/passthrough_hp -o allow_other --nopassthrough --foreground /tmp/source /tmp/dest`
>
>
> Thanks,
> Bernd

Re: FUSE: [Regression] Fuse legacy path performance scaling lost in v6.14 vs v6.8/6.11 (iodepth scaling with io_uring)

[Bernd Schubert]

Hi Abhishek,

On 11/27/25 14:37, Abhishek Gupta wrote:
> Hi Bernd,
> 
> Thanks for looking into this.
> Please find below the fio output on 6.11 & 6.14 kernel versions.
> 
> 
> On kernel 6.11
> 
> ~/gcsfuse$ uname -a
> Linux abhishek-c4-192-west4a 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP
> Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
> 
> iodepth = 1
> :~/fio-fio-3.38$ ./fio --name=randread --rw=randread
> --ioengine=io_uring --thread
> --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> --iodepth=1 --group_reporting=1 --direct=1
> randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=io_uring, iodepth=1
> fio-3.38
> Starting 1 thread
> ...
> Run status group 0 (all jobs):
>    READ: bw=3311KiB/s (3391kB/s), 3311KiB/s-3311KiB/s
> (3391kB/s-3391kB/s), io=48.5MiB (50.9MB), run=15001-15001msec
> 
> iodepth=4
> :~/fio-fio-3.38$ ./fio --name=randread --rw=randread
> --ioengine=io_uring --thread
> --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> --iodepth=4 --group_reporting=1 --direct=1
> randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=io_uring, iodepth=4
> fio-3.38
> Starting 1 thread
> ...
> Run status group 0 (all jobs):
>    READ: bw=11.0MiB/s (11.6MB/s), 11.0MiB/s-11.0MiB/s
> (11.6MB/s-11.6MB/s), io=166MiB (174MB), run=15002-15002msec
> 
> 
> On kernel 6.14
> 
> :~$ uname -a
> Linux abhishek-west4a-2504 6.14.0-1019-gcp #20-Ubuntu SMP Wed Oct 15
> 00:41:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
> 
> iodepth=1
> :~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
> --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> --iodepth=1 --group_reporting=1 --direct=1
> randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=io_uring, iodepth=1
> fio-3.38
> Starting 1 thread
> ...
> Run status group 0 (all jobs):
>    READ: bw=3576KiB/s (3662kB/s), 3576KiB/s-3576KiB/s
> (3662kB/s-3662kB/s), io=52.4MiB (54.9MB), run=15001-15001msec
> 
> iodepth=4
> :~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
> --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> --iodepth=4 --group_reporting=1 --direct=1
> randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=io_uring, iodepth=4
> fio-3.38
> ...
> Run status group 0 (all jobs):
>    READ: bw=3863KiB/s (3956kB/s), 3863KiB/s-3863KiB/s
> (3956kB/s-3956kB/s), io=56.6MiB (59.3MB), run=15001-15001msec

assuming I would find some time over the weekend and with the fact that
I don't know anything about google cloud, how can I reproduce this?


Thanks,
Bernd

Re: FUSE: [Regression] Fuse legacy path performance scaling lost in v6.14 vs v6.8/6.11 (iodepth scaling with io_uring)

[Abhishek Gupta]

Hi Bernd,

Apologies for the delay in responding.

Here are the steps to reproduce the FUSE performance issue locally
using a simple read-bench FUSE filesystem:

1. Set up the FUSE Filesystem:
git clone https://github.com/jacobsa/fuse.git jacobsa-fuse
cd jacobsa-fuse/samples/mount_readbenchfs
# Replace <mnt_dir> with your desired mount point
go run mount.go --mount_point <mnt_dir>

2. Run Fio Benchmark (iodepth 1):
fio  --name=randread --rw=randread --ioengine=io_uring --thread
--filename=<mnt_dir>/test --filesize=1G --time_based=1 --runtime=5s
--bs=4K --numjobs=1 --iodepth=1 --direct=1 --group_reporting=1

3. Run Fio Benchmark (iodepth 4):
fio --name=randread --rw=randread --ioengine=io_uring --thread
--filename=<mnt_dir>/test --filesize=1G --time_based=1 --runtime=5s
--bs=4K --numjobs=1 --iodepth=4 --direct=1 --group_reporting=1


Example Results on Kernel 6.14 (Regression Observed)

The following output shows the lack of scaling on my machine with Kernel 6.14:

Kernel:
Linux abhishek-west4a-2504 6.14.0-1019-gcp #20-Ubuntu SMP Wed Oct 15
00:41:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Iodepth = 1:
READ: bw=74.3MiB/s (77.9MB/s), ... io=372MiB (390MB), run=5001-5001msec

Iodepth = 4:
READ: bw=87.6MiB/s (91.9MB/s), ... io=438MiB (459MB), run=5000-5000msec

Thanks,
Abhishek


On Fri, Nov 28, 2025 at 4:35 AM Bernd Schubert <bernd@bsbernd.com> wrote:
>
> Hi Abhishek,
>
> On 11/27/25 14:37, Abhishek Gupta wrote:
> > Hi Bernd,
> >
> > Thanks for looking into this.
> > Please find below the fio output on 6.11 & 6.14 kernel versions.
> >
> >
> > On kernel 6.11
> >
> > ~/gcsfuse$ uname -a
> > Linux abhishek-c4-192-west4a 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP
> > Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
> >
> > iodepth = 1
> > :~/fio-fio-3.38$ ./fio --name=randread --rw=randread
> > --ioengine=io_uring --thread
> > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> > --iodepth=1 --group_reporting=1 --direct=1
> > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> > 4096B-4096B, ioengine=io_uring, iodepth=1
> > fio-3.38
> > Starting 1 thread
> > ...
> > Run status group 0 (all jobs):
> >    READ: bw=3311KiB/s (3391kB/s), 3311KiB/s-3311KiB/s
> > (3391kB/s-3391kB/s), io=48.5MiB (50.9MB), run=15001-15001msec
> >
> > iodepth=4
> > :~/fio-fio-3.38$ ./fio --name=randread --rw=randread
> > --ioengine=io_uring --thread
> > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> > --iodepth=4 --group_reporting=1 --direct=1
> > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> > 4096B-4096B, ioengine=io_uring, iodepth=4
> > fio-3.38
> > Starting 1 thread
> > ...
> > Run status group 0 (all jobs):
> >    READ: bw=11.0MiB/s (11.6MB/s), 11.0MiB/s-11.0MiB/s
> > (11.6MB/s-11.6MB/s), io=166MiB (174MB), run=15002-15002msec
> >
> >
> > On kernel 6.14
> >
> > :~$ uname -a
> > Linux abhishek-west4a-2504 6.14.0-1019-gcp #20-Ubuntu SMP Wed Oct 15
> > 00:41:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
> >
> > iodepth=1
> > :~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
> > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> > --iodepth=1 --group_reporting=1 --direct=1
> > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> > 4096B-4096B, ioengine=io_uring, iodepth=1
> > fio-3.38
> > Starting 1 thread
> > ...
> > Run status group 0 (all jobs):
> >    READ: bw=3576KiB/s (3662kB/s), 3576KiB/s-3576KiB/s
> > (3662kB/s-3662kB/s), io=52.4MiB (54.9MB), run=15001-15001msec
> >
> > iodepth=4
> > :~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
> > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> > --iodepth=4 --group_reporting=1 --direct=1
> > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> > 4096B-4096B, ioengine=io_uring, iodepth=4
> > fio-3.38
> > ...
> > Run status group 0 (all jobs):
> >    READ: bw=3863KiB/s (3956kB/s), 3863KiB/s-3863KiB/s
> > (3956kB/s-3956kB/s), io=56.6MiB (59.3MB), run=15001-15001msec
>
> assuming I would find some time over the weekend and with the fact that
> I don't know anything about google cloud, how can I reproduce this?
>
>
> Thanks,
> Bernd

Re: FUSE: [Regression] Fuse legacy path performance scaling lost in v6.14 vs v6.8/6.11 (iodepth scaling with io_uring)

[Bernd Schubert]

Hi Abhishek,

yes I was able to run it today, will send out a mail later. Sorry,
rather busy with other work.


Best,
Bernd

On 12/8/25 18:43, Abhishek Gupta wrote:
> Hi Bernd,
> 
> Were you able to reproduce the issue locally using the steps I provided?
> Please let me know if you require any further information or assistance.
> 
> Thanks,
> Abhishek
> 
> 
> On Tue, Dec 2, 2025 at 4:12 PM Abhishek Gupta <abhishekmgupta@google.com
> <mailto:abhishekmgupta@google.com>> wrote:
> 
>     Hi Bernd,
> 
>     Apologies for the delay in responding.
> 
>     Here are the steps to reproduce the FUSE performance issue locally
>     using a simple read-bench FUSE filesystem:
> 
>     1. Set up the FUSE Filesystem:
>     git clone https://github.com/jacobsa/fuse.git <https://github.com/
>     jacobsa/fuse.git> jacobsa-fuse
>     cd jacobsa-fuse/samples/mount_readbenchfs
>     # Replace <mnt_dir> with your desired mount point
>     go run mount.go --mount_point <mnt_dir>
> 
>     2. Run Fio Benchmark (iodepth 1):
>     fio  --name=randread --rw=randread --ioengine=io_uring --thread
>     --filename=<mnt_dir>/test --filesize=1G --time_based=1 --runtime=5s
>     --bs=4K --numjobs=1 --iodepth=1 --direct=1 --group_reporting=1
> 
>     3. Run Fio Benchmark (iodepth 4):
>     fio --name=randread --rw=randread --ioengine=io_uring --thread
>     --filename=<mnt_dir>/test --filesize=1G --time_based=1 --runtime=5s
>     --bs=4K --numjobs=1 --iodepth=4 --direct=1 --group_reporting=1
> 
> 
>     Example Results on Kernel 6.14 (Regression Observed)
> 
>     The following output shows the lack of scaling on my machine with
>     Kernel 6.14:
> 
>     Kernel:
>     Linux abhishek-west4a-2504 6.14.0-1019-gcp #20-Ubuntu SMP Wed Oct 15
>     00:41:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
> 
>     Iodepth = 1:
>     READ: bw=74.3MiB/s (77.9MB/s), ... io=372MiB (390MB), run=5001-5001msec
> 
>     Iodepth = 4:
>     READ: bw=87.6MiB/s (91.9MB/s), ... io=438MiB (459MB), run=5000-5000msec
> 
>     Thanks,
>     Abhishek
> 
> 
>     On Fri, Nov 28, 2025 at 4:35 AM Bernd Schubert <bernd@bsbernd.com
>     <mailto:bernd@bsbernd.com>> wrote:
>     >
>     > Hi Abhishek,
>     >
>     > On 11/27/25 14:37, Abhishek Gupta wrote:
>     > > Hi Bernd,
>     > >
>     > > Thanks for looking into this.
>     > > Please find below the fio output on 6.11 & 6.14 kernel versions.
>     > >
>     > >
>     > > On kernel 6.11
>     > >
>     > > ~/gcsfuse$ uname -a
>     > > Linux abhishek-c4-192-west4a 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP
>     > > Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
>     > >
>     > > iodepth = 1
>     > > :~/fio-fio-3.38$ ./fio --name=randread --rw=randread
>     > > --ioengine=io_uring --thread
>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
>     > > --iodepth=1 --group_reporting=1 --direct=1
>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
>     4096B-4096B, (T)
>     > > 4096B-4096B, ioengine=io_uring, iodepth=1
>     > > fio-3.38
>     > > Starting 1 thread
>     > > ...
>     > > Run status group 0 (all jobs):
>     > >    READ: bw=3311KiB/s (3391kB/s), 3311KiB/s-3311KiB/s
>     > > (3391kB/s-3391kB/s), io=48.5MiB (50.9MB), run=15001-15001msec
>     > >
>     > > iodepth=4
>     > > :~/fio-fio-3.38$ ./fio --name=randread --rw=randread
>     > > --ioengine=io_uring --thread
>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
>     > > --iodepth=4 --group_reporting=1 --direct=1
>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
>     4096B-4096B, (T)
>     > > 4096B-4096B, ioengine=io_uring, iodepth=4
>     > > fio-3.38
>     > > Starting 1 thread
>     > > ...
>     > > Run status group 0 (all jobs):
>     > >    READ: bw=11.0MiB/s (11.6MB/s), 11.0MiB/s-11.0MiB/s
>     > > (11.6MB/s-11.6MB/s), io=166MiB (174MB), run=15002-15002msec
>     > >
>     > >
>     > > On kernel 6.14
>     > >
>     > > :~$ uname -a
>     > > Linux abhishek-west4a-2504 6.14.0-1019-gcp #20-Ubuntu SMP Wed Oct 15
>     > > 00:41:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
>     > >
>     > > iodepth=1
>     > > :~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
>     > > --iodepth=1 --group_reporting=1 --direct=1
>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
>     4096B-4096B, (T)
>     > > 4096B-4096B, ioengine=io_uring, iodepth=1
>     > > fio-3.38
>     > > Starting 1 thread
>     > > ...
>     > > Run status group 0 (all jobs):
>     > >    READ: bw=3576KiB/s (3662kB/s), 3576KiB/s-3576KiB/s
>     > > (3662kB/s-3662kB/s), io=52.4MiB (54.9MB), run=15001-15001msec
>     > >
>     > > iodepth=4
>     > > :~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
>     > > --iodepth=4 --group_reporting=1 --direct=1
>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
>     4096B-4096B, (T)
>     > > 4096B-4096B, ioengine=io_uring, iodepth=4
>     > > fio-3.38
>     > > ...
>     > > Run status group 0 (all jobs):
>     > >    READ: bw=3863KiB/s (3956kB/s), 3863KiB/s-3863KiB/s
>     > > (3956kB/s-3956kB/s), io=56.6MiB (59.3MB), run=15001-15001msec
>     >
>     > assuming I would find some time over the weekend and with the fact
>     that
>     > I don't know anything about google cloud, how can I reproduce this?
>     >
>     >
>     > Thanks,
>     > Bernd
> 

Re: FUSE: [Regression] Fuse legacy path performance scaling lost in v6.14 vs v6.8/6.11 (iodepth scaling with io_uring)

[Bernd Schubert]

Hi Abishek,

really sorry for the delay. I can see the same as you do, no improvement
with --iodepth. Although increasing the number of fio threads/jobs helps.

Interesting is that this is not what I'm seeing with passthrough_hp,
at least I think so

I had run quite some tests here
https://lore.kernel.org/r/20251003-reduced-nr-ring-queues_3-v2-6-742ff1a8fc58@ddn.com
focused on io-uring, but I had also done some tests with legacy
fuse. I was hoping I would managed to re-run today before sending
the mail, but much too late right. Will try in the morning.



Thanks,
Bernd


On 12/8/25 18:52, Bernd Schubert wrote:
> Hi Abhishek,
> 
> yes I was able to run it today, will send out a mail later. Sorry,
> rather busy with other work.
> 
> 
> Best,
> Bernd
> 
> On 12/8/25 18:43, Abhishek Gupta wrote:
>> Hi Bernd,
>>
>> Were you able to reproduce the issue locally using the steps I provided?
>> Please let me know if you require any further information or assistance.
>>
>> Thanks,
>> Abhishek
>>
>>
>> On Tue, Dec 2, 2025 at 4:12 PM Abhishek Gupta <abhishekmgupta@google.com
>> <mailto:abhishekmgupta@google.com>> wrote:
>>
>>     Hi Bernd,
>>
>>     Apologies for the delay in responding.
>>
>>     Here are the steps to reproduce the FUSE performance issue locally
>>     using a simple read-bench FUSE filesystem:
>>
>>     1. Set up the FUSE Filesystem:
>>     git clone https://github.com/jacobsa/fuse.git <https://github.com/
>>     jacobsa/fuse.git> jacobsa-fuse
>>     cd jacobsa-fuse/samples/mount_readbenchfs
>>     # Replace <mnt_dir> with your desired mount point
>>     go run mount.go --mount_point <mnt_dir>
>>
>>     2. Run Fio Benchmark (iodepth 1):
>>     fio  --name=randread --rw=randread --ioengine=io_uring --thread
>>     --filename=<mnt_dir>/test --filesize=1G --time_based=1 --runtime=5s
>>     --bs=4K --numjobs=1 --iodepth=1 --direct=1 --group_reporting=1
>>
>>     3. Run Fio Benchmark (iodepth 4):
>>     fio --name=randread --rw=randread --ioengine=io_uring --thread
>>     --filename=<mnt_dir>/test --filesize=1G --time_based=1 --runtime=5s
>>     --bs=4K --numjobs=1 --iodepth=4 --direct=1 --group_reporting=1
>>
>>
>>     Example Results on Kernel 6.14 (Regression Observed)
>>
>>     The following output shows the lack of scaling on my machine with
>>     Kernel 6.14:
>>
>>     Kernel:
>>     Linux abhishek-west4a-2504 6.14.0-1019-gcp #20-Ubuntu SMP Wed Oct 15
>>     00:41:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
>>
>>     Iodepth = 1:
>>     READ: bw=74.3MiB/s (77.9MB/s), ... io=372MiB (390MB), run=5001-5001msec
>>
>>     Iodepth = 4:
>>     READ: bw=87.6MiB/s (91.9MB/s), ... io=438MiB (459MB), run=5000-5000msec
>>
>>     Thanks,
>>     Abhishek
>>
>>
>>     On Fri, Nov 28, 2025 at 4:35 AM Bernd Schubert <bernd@bsbernd.com
>>     <mailto:bernd@bsbernd.com>> wrote:
>>     >
>>     > Hi Abhishek,
>>     >
>>     > On 11/27/25 14:37, Abhishek Gupta wrote:
>>     > > Hi Bernd,
>>     > >
>>     > > Thanks for looking into this.
>>     > > Please find below the fio output on 6.11 & 6.14 kernel versions.
>>     > >
>>     > >
>>     > > On kernel 6.11
>>     > >
>>     > > ~/gcsfuse$ uname -a
>>     > > Linux abhishek-c4-192-west4a 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP
>>     > > Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
>>     > >
>>     > > iodepth = 1
>>     > > :~/fio-fio-3.38$ ./fio --name=randread --rw=randread
>>     > > --ioengine=io_uring --thread
>>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
>>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
>>     > > --iodepth=1 --group_reporting=1 --direct=1
>>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
>>     4096B-4096B, (T)
>>     > > 4096B-4096B, ioengine=io_uring, iodepth=1
>>     > > fio-3.38
>>     > > Starting 1 thread
>>     > > ...
>>     > > Run status group 0 (all jobs):
>>     > >    READ: bw=3311KiB/s (3391kB/s), 3311KiB/s-3311KiB/s
>>     > > (3391kB/s-3391kB/s), io=48.5MiB (50.9MB), run=15001-15001msec
>>     > >
>>     > > iodepth=4
>>     > > :~/fio-fio-3.38$ ./fio --name=randread --rw=randread
>>     > > --ioengine=io_uring --thread
>>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
>>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
>>     > > --iodepth=4 --group_reporting=1 --direct=1
>>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
>>     4096B-4096B, (T)
>>     > > 4096B-4096B, ioengine=io_uring, iodepth=4
>>     > > fio-3.38
>>     > > Starting 1 thread
>>     > > ...
>>     > > Run status group 0 (all jobs):
>>     > >    READ: bw=11.0MiB/s (11.6MB/s), 11.0MiB/s-11.0MiB/s
>>     > > (11.6MB/s-11.6MB/s), io=166MiB (174MB), run=15002-15002msec
>>     > >
>>     > >
>>     > > On kernel 6.14
>>     > >
>>     > > :~$ uname -a
>>     > > Linux abhishek-west4a-2504 6.14.0-1019-gcp #20-Ubuntu SMP Wed Oct 15
>>     > > 00:41:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
>>     > >
>>     > > iodepth=1
>>     > > :~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
>>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
>>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
>>     > > --iodepth=1 --group_reporting=1 --direct=1
>>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
>>     4096B-4096B, (T)
>>     > > 4096B-4096B, ioengine=io_uring, iodepth=1
>>     > > fio-3.38
>>     > > Starting 1 thread
>>     > > ...
>>     > > Run status group 0 (all jobs):
>>     > >    READ: bw=3576KiB/s (3662kB/s), 3576KiB/s-3576KiB/s
>>     > > (3662kB/s-3662kB/s), io=52.4MiB (54.9MB), run=15001-15001msec
>>     > >
>>     > > iodepth=4
>>     > > :~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
>>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
>>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
>>     > > --iodepth=4 --group_reporting=1 --direct=1
>>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
>>     4096B-4096B, (T)
>>     > > 4096B-4096B, ioengine=io_uring, iodepth=4
>>     > > fio-3.38
>>     > > ...
>>     > > Run status group 0 (all jobs):
>>     > >    READ: bw=3863KiB/s (3956kB/s), 3863KiB/s-3863KiB/s
>>     > > (3956kB/s-3956kB/s), io=56.6MiB (59.3MB), run=15001-15001msec
>>     >
>>     > assuming I would find some time over the weekend and with the fact
>>     that
>>     > I don't know anything about google cloud, how can I reproduce this?
>>     >
>>     >
>>     > Thanks,
>>     > Bernd
>>
> 

Re: FUSE: [Regression] Fuse legacy path performance scaling lost in v6.14 vs v6.8/6.11 (iodepth scaling with io_uring)

[Abhishek Gupta]

Hi Bernd,

No worries. Thanks for the update. I look forward to hearing your findings.

Thanks,
Abhishek


On Tue, Dec 9, 2025 at 4:27 AM Bernd Schubert <bernd@bsbernd.com> wrote:
>
> Hi Abishek,
>
> really sorry for the delay. I can see the same as you do, no improvement
> with --iodepth. Although increasing the number of fio threads/jobs helps.
>
> Interesting is that this is not what I'm seeing with passthrough_hp,
> at least I think so
>
> I had run quite some tests here
> https://lore.kernel.org/r/20251003-reduced-nr-ring-queues_3-v2-6-742ff1a8fc58@ddn.com
> focused on io-uring, but I had also done some tests with legacy
> fuse. I was hoping I would managed to re-run today before sending
> the mail, but much too late right. Will try in the morning.
>
>
>
> Thanks,
> Bernd
>
>
> On 12/8/25 18:52, Bernd Schubert wrote:
> > Hi Abhishek,
> >
> > yes I was able to run it today, will send out a mail later. Sorry,
> > rather busy with other work.
> >
> >
> > Best,
> > Bernd
> >
> > On 12/8/25 18:43, Abhishek Gupta wrote:
> >> Hi Bernd,
> >>
> >> Were you able to reproduce the issue locally using the steps I provided?
> >> Please let me know if you require any further information or assistance.
> >>
> >> Thanks,
> >> Abhishek
> >>
> >>
> >> On Tue, Dec 2, 2025 at 4:12 PM Abhishek Gupta <abhishekmgupta@google.com
> >> <mailto:abhishekmgupta@google.com>> wrote:
> >>
> >>     Hi Bernd,
> >>
> >>     Apologies for the delay in responding.
> >>
> >>     Here are the steps to reproduce the FUSE performance issue locally
> >>     using a simple read-bench FUSE filesystem:
> >>
> >>     1. Set up the FUSE Filesystem:
> >>     git clone https://github.com/jacobsa/fuse.git <https://github.com/
> >>     jacobsa/fuse.git> jacobsa-fuse
> >>     cd jacobsa-fuse/samples/mount_readbenchfs
> >>     # Replace <mnt_dir> with your desired mount point
> >>     go run mount.go --mount_point <mnt_dir>
> >>
> >>     2. Run Fio Benchmark (iodepth 1):
> >>     fio  --name=randread --rw=randread --ioengine=io_uring --thread
> >>     --filename=<mnt_dir>/test --filesize=1G --time_based=1 --runtime=5s
> >>     --bs=4K --numjobs=1 --iodepth=1 --direct=1 --group_reporting=1
> >>
> >>     3. Run Fio Benchmark (iodepth 4):
> >>     fio --name=randread --rw=randread --ioengine=io_uring --thread
> >>     --filename=<mnt_dir>/test --filesize=1G --time_based=1 --runtime=5s
> >>     --bs=4K --numjobs=1 --iodepth=4 --direct=1 --group_reporting=1
> >>
> >>
> >>     Example Results on Kernel 6.14 (Regression Observed)
> >>
> >>     The following output shows the lack of scaling on my machine with
> >>     Kernel 6.14:
> >>
> >>     Kernel:
> >>     Linux abhishek-west4a-2504 6.14.0-1019-gcp #20-Ubuntu SMP Wed Oct 15
> >>     00:41:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
> >>
> >>     Iodepth = 1:
> >>     READ: bw=74.3MiB/s (77.9MB/s), ... io=372MiB (390MB), run=5001-5001msec
> >>
> >>     Iodepth = 4:
> >>     READ: bw=87.6MiB/s (91.9MB/s), ... io=438MiB (459MB), run=5000-5000msec
> >>
> >>     Thanks,
> >>     Abhishek
> >>
> >>
> >>     On Fri, Nov 28, 2025 at 4:35 AM Bernd Schubert <bernd@bsbernd.com
> >>     <mailto:bernd@bsbernd.com>> wrote:
> >>     >
> >>     > Hi Abhishek,
> >>     >
> >>     > On 11/27/25 14:37, Abhishek Gupta wrote:
> >>     > > Hi Bernd,
> >>     > >
> >>     > > Thanks for looking into this.
> >>     > > Please find below the fio output on 6.11 & 6.14 kernel versions.
> >>     > >
> >>     > >
> >>     > > On kernel 6.11
> >>     > >
> >>     > > ~/gcsfuse$ uname -a
> >>     > > Linux abhishek-c4-192-west4a 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP
> >>     > > Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
> >>     > >
> >>     > > iodepth = 1
> >>     > > :~/fio-fio-3.38$ ./fio --name=randread --rw=randread
> >>     > > --ioengine=io_uring --thread
> >>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> >>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> >>     > > --iodepth=1 --group_reporting=1 --direct=1
> >>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
> >>     4096B-4096B, (T)
> >>     > > 4096B-4096B, ioengine=io_uring, iodepth=1
> >>     > > fio-3.38
> >>     > > Starting 1 thread
> >>     > > ...
> >>     > > Run status group 0 (all jobs):
> >>     > >    READ: bw=3311KiB/s (3391kB/s), 3311KiB/s-3311KiB/s
> >>     > > (3391kB/s-3391kB/s), io=48.5MiB (50.9MB), run=15001-15001msec
> >>     > >
> >>     > > iodepth=4
> >>     > > :~/fio-fio-3.38$ ./fio --name=randread --rw=randread
> >>     > > --ioengine=io_uring --thread
> >>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> >>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> >>     > > --iodepth=4 --group_reporting=1 --direct=1
> >>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
> >>     4096B-4096B, (T)
> >>     > > 4096B-4096B, ioengine=io_uring, iodepth=4
> >>     > > fio-3.38
> >>     > > Starting 1 thread
> >>     > > ...
> >>     > > Run status group 0 (all jobs):
> >>     > >    READ: bw=11.0MiB/s (11.6MB/s), 11.0MiB/s-11.0MiB/s
> >>     > > (11.6MB/s-11.6MB/s), io=166MiB (174MB), run=15002-15002msec
> >>     > >
> >>     > >
> >>     > > On kernel 6.14
> >>     > >
> >>     > > :~$ uname -a
> >>     > > Linux abhishek-west4a-2504 6.14.0-1019-gcp #20-Ubuntu SMP Wed Oct 15
> >>     > > 00:41:12 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
> >>     > >
> >>     > > iodepth=1
> >>     > > :~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
> >>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> >>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> >>     > > --iodepth=1 --group_reporting=1 --direct=1
> >>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
> >>     4096B-4096B, (T)
> >>     > > 4096B-4096B, ioengine=io_uring, iodepth=1
> >>     > > fio-3.38
> >>     > > Starting 1 thread
> >>     > > ...
> >>     > > Run status group 0 (all jobs):
> >>     > >    READ: bw=3576KiB/s (3662kB/s), 3576KiB/s-3576KiB/s
> >>     > > (3662kB/s-3662kB/s), io=52.4MiB (54.9MB), run=15001-15001msec
> >>     > >
> >>     > > iodepth=4
> >>     > > :~$ fio --name=randread --rw=randread --ioengine=io_uring --thread
> >>     > > --filename_format='/home/abhishekmgupta_google_com/bucket/$jobnum'
> >>     > > --filesize=1G --time_based=1 --runtime=15s --bs=4K --numjobs=1
> >>     > > --iodepth=4 --group_reporting=1 --direct=1
> >>     > > randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W)
> >>     4096B-4096B, (T)
> >>     > > 4096B-4096B, ioengine=io_uring, iodepth=4
> >>     > > fio-3.38
> >>     > > ...
> >>     > > Run status group 0 (all jobs):
> >>     > >    READ: bw=3863KiB/s (3956kB/s), 3863KiB/s-3863KiB/s
> >>     > > (3956kB/s-3956kB/s), io=56.6MiB (59.3MB), run=15001-15001msec
> >>     >
> >>     > assuming I would find some time over the weekend and with the fact
> >>     that
> >>     > I don't know anything about google cloud, how can I reproduce this?
> >>     >
> >>     >
> >>     > Thanks,
> >>     > Bernd
> >>
> >
>

Re: FUSE: [Regression] Fuse legacy path performance scaling lost in v6.14 vs v6.8/6.11 (iodepth scaling with io_uring)

[Joanne Koong]

On Tue, Dec 9, 2025 at 6:57 AM Bernd Schubert <bernd@bsbernd.com> wrote:
>
> Hi Abishek,
>
> really sorry for the delay. I can see the same as you do, no improvement
> with --iodepth. Although increasing the number of fio threads/jobs helps.
>
> Interesting is that this is not what I'm seeing with passthrough_hp,
> at least I think so

I'm not seeing this regression on passthrough_hp either. On my local
vm (on top of the fuse for-next tree) I'm seeing ~13 MiB/s for
iodepth=1 and ~70 MiB/s for iodepth=4.

Abhishek, are you able to git bisect this to the commit that causes
your regression?

Thanks,
Joanne

>
> I had run quite some tests here
> https://lore.kernel.org/r/20251003-reduced-nr-ring-queues_3-v2-6-742ff1a8fc58@ddn.com
> focused on io-uring, but I had also done some tests with legacy
> fuse. I was hoping I would managed to re-run today before sending
> the mail, but much too late right. Will try in the morning.
>
>
>
> Thanks,
> Bernd
>