* Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 [not found] ` <CAAiJnjor+=Zn62n09f-aJw2amX2wxQOb-2TB3rea9wDCU7ONoA@mail.gmail.com> @ 2025-05-04 21:50 ` Dave Chinner 2025-05-05 12:29 ` Laurence Oberman 0 siblings, 1 reply; 11+ messages in thread From: Dave Chinner @ 2025-05-04 21:50 UTC (permalink / raw) To: Anton Gavriliuk; +Cc: linux-nvme, linux-xfs, linux-block [cc linux-block] [original bug report: https://lore.kernel.org/linux-xfs/CAAiJnjoo0--yp47UKZhbu8sNSZN6DZ-QzmZBMmtr1oC=fOOgAQ@mail.gmail.com/ ] On Sun, May 04, 2025 at 10:22:58AM +0300, Anton Gavriliuk wrote: > > What's the comparitive performance of an identical read profile > > directly on the raw MD raid0 device? > > Rocky 9.5 (5.14.0-503.40.1.el9_5.x86_64) > > [root@localhost ~]# df -mh /mnt > Filesystem Size Used Avail Use% Mounted on > /dev/md127 35T 1.3T 34T 4% /mnt > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 --exitall > --group_reporting --ioengine=libaio --runtime=30 --time_based > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) > 256KiB-256KiB, ioengine=libaio, iodepth=64 > fio-3.39-44-g19d9 > Starting 1 process > Jobs: 1 (f=1): [R(1)][100.0%][r=81.4GiB/s][r=334k IOPS][eta 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=43189: Sun May 4 08:22:12 2025 > read: IOPS=363k, BW=88.5GiB/s (95.1GB/s)(2656GiB/30001msec) > slat (nsec): min=971, max=312380, avg=1817.92, stdev=1367.75 > clat (usec): min=78, max=1351, avg=174.46, stdev=28.86 > lat (usec): min=80, max=1352, avg=176.27, stdev=28.81 > > Fedora 42 (6.14.5-300.fc42.x86_64) > > [root@localhost anton]# df -mh /mnt > Filesystem Size Used Avail Use% Mounted on > /dev/md127 35T 1.3T 34T 4% /mnt > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 --exitall > --group_reporting --ioengine=libaio --runtime=30 --time_based > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) > 256KiB-256KiB, ioengine=libaio, iodepth=64 > fio-3.39-44-g19d9 > Starting 1 process > Jobs: 1 (f=1): [R(1)][100.0%][r=41.0GiB/s][r=168k IOPS][eta 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=5685: Sun May 4 10:14:00 2025 > read: IOPS=168k, BW=41.0GiB/s (44.1GB/s)(1231GiB/30001msec) > slat (usec): min=3, max=273, avg= 5.63, stdev= 1.48 > clat (usec): min=67, max=2800, avg=374.99, stdev=29.90 > lat (usec): min=72, max=2914, avg=380.62, stdev=30.22 So the MD block device shows the same read performance as the filesystem on top of it. That means this is a regression at the MD device layer or in the block/driver layers below it. i.e. it is not an XFS of filesystem issue at all. -Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 2025-05-04 21:50 ` Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 Dave Chinner @ 2025-05-05 12:29 ` Laurence Oberman 2025-05-05 13:21 ` Laurence Oberman 0 siblings, 1 reply; 11+ messages in thread From: Laurence Oberman @ 2025-05-05 12:29 UTC (permalink / raw) To: Dave Chinner, Anton Gavriliuk; +Cc: linux-nvme, linux-xfs, linux-block On Mon, 2025-05-05 at 07:50 +1000, Dave Chinner wrote: > [cc linux-block] > > [original bug report: > https://lore.kernel.org/linux-xfs/CAAiJnjoo0--yp47UKZhbu8sNSZN6DZ-QzmZBMmtr1oC=fOOgAQ@mail.gmail.com/ > ] > > On Sun, May 04, 2025 at 10:22:58AM +0300, Anton Gavriliuk wrote: > > > What's the comparitive performance of an identical read profile > > > directly on the raw MD raid0 device? > > > > Rocky 9.5 (5.14.0-503.40.1.el9_5.x86_64) > > > > [root@localhost ~]# df -mh /mnt > > Filesystem Size Used Avail Use% Mounted on > > /dev/md127 35T 1.3T 34T 4% /mnt > > > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 --exitall > > --group_reporting --ioengine=libaio --runtime=30 --time_based > > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) > > 256KiB-256KiB, ioengine=libaio, iodepth=64 > > fio-3.39-44-g19d9 > > Starting 1 process > > Jobs: 1 (f=1): [R(1)][100.0%][r=81.4GiB/s][r=334k IOPS][eta > > 00m:00s] > > test: (groupid=0, jobs=1): err= 0: pid=43189: Sun May 4 08:22:12 > > 2025 > > read: IOPS=363k, BW=88.5GiB/s (95.1GB/s)(2656GiB/30001msec) > > slat (nsec): min=971, max=312380, avg=1817.92, stdev=1367.75 > > clat (usec): min=78, max=1351, avg=174.46, stdev=28.86 > > lat (usec): min=80, max=1352, avg=176.27, stdev=28.81 > > > > Fedora 42 (6.14.5-300.fc42.x86_64) > > > > [root@localhost anton]# df -mh /mnt > > Filesystem Size Used Avail Use% Mounted on > > /dev/md127 35T 1.3T 34T 4% /mnt > > > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 --exitall > > --group_reporting --ioengine=libaio --runtime=30 --time_based > > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) > > 256KiB-256KiB, ioengine=libaio, iodepth=64 > > fio-3.39-44-g19d9 > > Starting 1 process > > Jobs: 1 (f=1): [R(1)][100.0%][r=41.0GiB/s][r=168k IOPS][eta > > 00m:00s] > > test: (groupid=0, jobs=1): err= 0: pid=5685: Sun May 4 10:14:00 > > 2025 > > read: IOPS=168k, BW=41.0GiB/s (44.1GB/s)(1231GiB/30001msec) > > slat (usec): min=3, max=273, avg= 5.63, stdev= 1.48 > > clat (usec): min=67, max=2800, avg=374.99, stdev=29.90 > > lat (usec): min=72, max=2914, avg=380.62, stdev=30.22 > > So the MD block device shows the same read performance as the > filesystem on top of it. That means this is a regression at the MD > device layer or in the block/driver layers below it. i.e. it is not > an XFS of filesystem issue at all. > > -Dave. I have a lab setup, let me see if I can also reproduce and then trace this to see where it is spending the time ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 2025-05-05 12:29 ` Laurence Oberman @ 2025-05-05 13:21 ` Laurence Oberman 2025-05-05 17:39 ` Laurence Oberman 2025-05-05 22:56 ` Dave Chinner 0 siblings, 2 replies; 11+ messages in thread From: Laurence Oberman @ 2025-05-05 13:21 UTC (permalink / raw) To: Dave Chinner, Anton Gavriliuk; +Cc: linux-nvme, linux-xfs, linux-block On Mon, 2025-05-05 at 08:29 -0400, Laurence Oberman wrote: > On Mon, 2025-05-05 at 07:50 +1000, Dave Chinner wrote: > > [cc linux-block] > > > > [original bug report: > > https://lore.kernel.org/linux-xfs/CAAiJnjoo0--yp47UKZhbu8sNSZN6DZ-QzmZBMmtr1oC=fOOgAQ@mail.gmail.com/ > > ] > > > > On Sun, May 04, 2025 at 10:22:58AM +0300, Anton Gavriliuk wrote: > > > > What's the comparitive performance of an identical read profile > > > > directly on the raw MD raid0 device? > > > > > > Rocky 9.5 (5.14.0-503.40.1.el9_5.x86_64) > > > > > > [root@localhost ~]# df -mh /mnt > > > Filesystem Size Used Avail Use% Mounted on > > > /dev/md127 35T 1.3T 34T 4% /mnt > > > > > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > > > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 -- > > > exitall > > > --group_reporting --ioengine=libaio --runtime=30 --time_based > > > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, > > > (T) > > > 256KiB-256KiB, ioengine=libaio, iodepth=64 > > > fio-3.39-44-g19d9 > > > Starting 1 process > > > Jobs: 1 (f=1): [R(1)][100.0%][r=81.4GiB/s][r=334k IOPS][eta > > > 00m:00s] > > > test: (groupid=0, jobs=1): err= 0: pid=43189: Sun May 4 08:22:12 > > > 2025 > > > read: IOPS=363k, BW=88.5GiB/s (95.1GB/s)(2656GiB/30001msec) > > > slat (nsec): min=971, max=312380, avg=1817.92, stdev=1367.75 > > > clat (usec): min=78, max=1351, avg=174.46, stdev=28.86 > > > lat (usec): min=80, max=1352, avg=176.27, stdev=28.81 > > > > > > Fedora 42 (6.14.5-300.fc42.x86_64) > > > > > > [root@localhost anton]# df -mh /mnt > > > Filesystem Size Used Avail Use% Mounted on > > > /dev/md127 35T 1.3T 34T 4% /mnt > > > > > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > > > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 -- > > > exitall > > > --group_reporting --ioengine=libaio --runtime=30 --time_based > > > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, > > > (T) > > > 256KiB-256KiB, ioengine=libaio, iodepth=64 > > > fio-3.39-44-g19d9 > > > Starting 1 process > > > Jobs: 1 (f=1): [R(1)][100.0%][r=41.0GiB/s][r=168k IOPS][eta > > > 00m:00s] > > > test: (groupid=0, jobs=1): err= 0: pid=5685: Sun May 4 10:14:00 > > > 2025 > > > read: IOPS=168k, BW=41.0GiB/s (44.1GB/s)(1231GiB/30001msec) > > > slat (usec): min=3, max=273, avg= 5.63, stdev= 1.48 > > > clat (usec): min=67, max=2800, avg=374.99, stdev=29.90 > > > lat (usec): min=72, max=2914, avg=380.62, stdev=30.22 > > > > So the MD block device shows the same read performance as the > > filesystem on top of it. That means this is a regression at the MD > > device layer or in the block/driver layers below it. i.e. it is not > > an XFS of filesystem issue at all. > > > > -Dave. > > I have a lab setup, let me see if I can also reproduce and then trace > this to see where it is spending the time > Not seeing 1/2 the bandwidth but also significantly slower on Fedora42 kernel. I will trace it 9.5 kernel - 5.14.0-503.40.1.el9_5.x86_64 Run status group 0 (all jobs): READ: bw=14.7GiB/s (15.8GB/s), 14.7GiB/s-14.7GiB/s (15.8GB/s- 15.8GB/s), io=441GiB (473GB), run=30003-30003msec Fedora42 kernel - 6.14.5-300.fc42.x86_64 Run status group 0 (all jobs): READ: bw=10.4GiB/s (11.2GB/s), 10.4GiB/s-10.4GiB/s (11.2GB/s- 11.2GB/s), io=313GiB (336GB), run=30001-30001msec ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 2025-05-05 13:21 ` Laurence Oberman @ 2025-05-05 17:39 ` Laurence Oberman 2025-05-22 15:07 ` Laurence Oberman 2025-05-05 22:56 ` Dave Chinner 1 sibling, 1 reply; 11+ messages in thread From: Laurence Oberman @ 2025-05-05 17:39 UTC (permalink / raw) To: Dave Chinner, Anton Gavriliuk; +Cc: linux-nvme, linux-xfs, linux-block On Mon, 2025-05-05 at 09:21 -0400, Laurence Oberman wrote: > On Mon, 2025-05-05 at 08:29 -0400, Laurence Oberman wrote: > > On Mon, 2025-05-05 at 07:50 +1000, Dave Chinner wrote: > > > [cc linux-block] > > > > > > [original bug report: > > > https://lore.kernel.org/linux-xfs/CAAiJnjoo0--yp47UKZhbu8sNSZN6DZ-QzmZBMmtr1oC=fOOgAQ@mail.gmail.com/ > > > ] > > > > > > On Sun, May 04, 2025 at 10:22:58AM +0300, Anton Gavriliuk wrote: > > > > > What's the comparitive performance of an identical read > > > > > profile > > > > > directly on the raw MD raid0 device? > > > > > > > > Rocky 9.5 (5.14.0-503.40.1.el9_5.x86_64) > > > > > > > > [root@localhost ~]# df -mh /mnt > > > > Filesystem Size Used Avail Use% Mounted on > > > > /dev/md127 35T 1.3T 34T 4% /mnt > > > > > > > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > > > > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 -- > > > > exitall > > > > --group_reporting --ioengine=libaio --runtime=30 --time_based > > > > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, > > > > (T) > > > > 256KiB-256KiB, ioengine=libaio, iodepth=64 > > > > fio-3.39-44-g19d9 > > > > Starting 1 process > > > > Jobs: 1 (f=1): [R(1)][100.0%][r=81.4GiB/s][r=334k IOPS][eta > > > > 00m:00s] > > > > test: (groupid=0, jobs=1): err= 0: pid=43189: Sun May 4 > > > > 08:22:12 > > > > 2025 > > > > read: IOPS=363k, BW=88.5GiB/s (95.1GB/s)(2656GiB/30001msec) > > > > slat (nsec): min=971, max=312380, avg=1817.92, > > > > stdev=1367.75 > > > > clat (usec): min=78, max=1351, avg=174.46, stdev=28.86 > > > > lat (usec): min=80, max=1352, avg=176.27, stdev=28.81 > > > > > > > > Fedora 42 (6.14.5-300.fc42.x86_64) > > > > > > > > [root@localhost anton]# df -mh /mnt > > > > Filesystem Size Used Avail Use% Mounted on > > > > /dev/md127 35T 1.3T 34T 4% /mnt > > > > > > > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > > > > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 -- > > > > exitall > > > > --group_reporting --ioengine=libaio --runtime=30 --time_based > > > > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, > > > > (T) > > > > 256KiB-256KiB, ioengine=libaio, iodepth=64 > > > > fio-3.39-44-g19d9 > > > > Starting 1 process > > > > Jobs: 1 (f=1): [R(1)][100.0%][r=41.0GiB/s][r=168k IOPS][eta > > > > 00m:00s] > > > > test: (groupid=0, jobs=1): err= 0: pid=5685: Sun May 4 > > > > 10:14:00 > > > > 2025 > > > > read: IOPS=168k, BW=41.0GiB/s (44.1GB/s)(1231GiB/30001msec) > > > > slat (usec): min=3, max=273, avg= 5.63, stdev= 1.48 > > > > clat (usec): min=67, max=2800, avg=374.99, stdev=29.90 > > > > lat (usec): min=72, max=2914, avg=380.62, stdev=30.22 > > > > > > So the MD block device shows the same read performance as the > > > filesystem on top of it. That means this is a regression at the > > > MD > > > device layer or in the block/driver layers below it. i.e. it is > > > not > > > an XFS of filesystem issue at all. > > > > > > -Dave. > > > > I have a lab setup, let me see if I can also reproduce and then > > trace > > this to see where it is spending the time > > > > > Not seeing 1/2 the bandwidth but also significantly slower on > Fedora42 > kernel. > I will trace it > > 9.5 kernel - 5.14.0-503.40.1.el9_5.x86_64 > > Run status group 0 (all jobs): > READ: bw=14.7GiB/s (15.8GB/s), 14.7GiB/s-14.7GiB/s (15.8GB/s- > 15.8GB/s), io=441GiB (473GB), run=30003-30003msec > > Fedora42 kernel - 6.14.5-300.fc42.x86_64 > > Run status group 0 (all jobs): > READ: bw=10.4GiB/s (11.2GB/s), 10.4GiB/s-10.4GiB/s (11.2GB/s- > 11.2GB/s), io=313GiB (336GB), run=30001-30001msec > > > > Fedora42 kernel issue While my difference is not as severe we do see a consistently lower performance on the Fedora kernel. (6.14.5-300.fc42.x86_64) When I remove the software raid and run against a single NVME we converge to be much closer. Also latest upstream does not show this regression either. Not sure yet what is in our Fedora kernel causing this. We will work it via the Bugzilla Regards Laurence TLDR Fedora Kernel ------------- root@penguin9 blktracefedora]# uname -a Linux penguin9.2 6.14.5-300.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Fri May 2 14:16:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 5 runs of the fio against /dev/md1 [root@penguin9 ~]# for i in 1 2 3 4 5 > do > ./run_fio.sh | grep -A1 "Run status group" > done Run status group 0 (all jobs): READ: bw=11.3GiB/s (12.2GB/s), 11.3GiB/s-11.3GiB/s (12.2GB/s- 12.2GB/s), io=679GiB (729GB), run=60001-60001msec Run status group 0 (all jobs): READ: bw=11.2GiB/s (12.0GB/s), 11.2GiB/s-11.2GiB/s (12.0GB/s- 12.0GB/s), io=669GiB (718GB), run=60001-60001msec Run status group 0 (all jobs): READ: bw=11.4GiB/s (12.2GB/s), 11.4GiB/s-11.4GiB/s (12.2GB/s- 12.2GB/s), io=682GiB (733GB), run=60001-60001msec Run status group 0 (all jobs): READ: bw=11.1GiB/s (11.9GB/s), 11.1GiB/s-11.1GiB/s (11.9GB/s- 11.9GB/s), io=664GiB (713GB), run=60001-60001msec Run status group 0 (all jobs): READ: bw=11.3GiB/s (12.1GB/s), 11.3GiB/s-11.3GiB/s (12.1GB/s- 12.1GB/s), io=678GiB (728GB), run=60001-60001msec RHEL9.5 ------------ Linux penguin9.2 5.14.0-503.40.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 24 08:27:29 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux [root@penguin9 ~]# for i in 1 2 3 4 5; do ./run_fio.sh | grep -A1 "Run status group"; done Run status group 0 (all jobs): READ: bw=14.9GiB/s (16.0GB/s), 14.9GiB/s-14.9GiB/s (16.0GB/s- 16.0GB/s), io=894GiB (960GB), run=60003-60003msec Run status group 0 (all jobs): READ: bw=14.6GiB/s (15.6GB/s), 14.6GiB/s-14.6GiB/s (15.6GB/s- 15.6GB/s), io=873GiB (938GB), run=60003-60003msec Run status group 0 (all jobs): READ: bw=14.9GiB/s (16.0GB/s), 14.9GiB/s-14.9GiB/s (16.0GB/s- 16.0GB/s), io=892GiB (958GB), run=60003-60003msec Run status group 0 (all jobs): READ: bw=14.5GiB/s (15.6GB/s), 14.5GiB/s-14.5GiB/s (15.6GB/s- 15.6GB/s), io=872GiB (936GB), run=60003-60003msec Run status group 0 (all jobs): READ: bw=14.7GiB/s (15.8GB/s), 14.7GiB/s-14.7GiB/s (15.8GB/s- 15.8GB/s), io=884GiB (950GB), run=60003-60003msec Remove software raid from the layers and test just on a single nvme ---------------------------------------------------------------------- fio --name=test --rw=read --bs=256k --filename=/dev/nvme23n1 --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting --ioengine=libaio --runtime=60 --time_based Linux penguin9.2 5.14.0-503.40.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 24 08:27:29 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux [root@penguin9 ~]# ./run_nvme_fio.sh Run status group 0 (all jobs): READ: bw=3207MiB/s (3363MB/s), 3207MiB/s-3207MiB/s (3363MB/s- 3363MB/s), io=188GiB (202GB), run=60005-60005msec Back to fedora kernel [root@penguin9 ~]# uname -a Linux penguin9.2 6.14.5-300.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Fri May 2 14:16:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Within the margin of error Run status group 0 (all jobs): READ: bw=3061MiB/s (3210MB/s), 3061MiB/s-3061MiB/s (3210MB/s- 3210MB/s), io=179GiB (193GB), run=60006-60006msec Try recent upstream kernel --------------------------- [root@penguin9 ~]# uname -a Linux penguin9.2 6.13.0-rc7+ #2 SMP PREEMPT_DYNAMIC Mon May 5 10:59:12 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux [root@penguin9 ~]# for i in 1 2 3 4 5; do ./run_fio.sh | grep -A1 "Run status group"; done Run status group 0 (all jobs): READ: bw=14.6GiB/s (15.7GB/s), 14.6GiB/s-14.6GiB/s (15.7GB/s- 15.7GB/s), io=876GiB (941GB), run=60003-60003msec Run status group 0 (all jobs): READ: bw=14.8GiB/s (15.9GB/s), 14.8GiB/s-14.8GiB/s (15.9GB/s- 15.9GB/s), io=891GiB (957GB), run=60003-60003msec Run status group 0 (all jobs): READ: bw=14.8GiB/s (15.9GB/s), 14.8GiB/s-14.8GiB/s (15.9GB/s- 15.9GB/s), io=890GiB (956GB), run=60003-60003msec Run status group 0 (all jobs): READ: bw=14.5GiB/s (15.6GB/s), 14.5GiB/s-14.5GiB/s (15.6GB/s- 15.6GB/s), io=871GiB (935GB), run=60003-60003msec Update to latest upstream ------------------------- [root@penguin9 ~]# uname -a Linux penguin9.2 6.15.0-rc5 #1 SMP PREEMPT_DYNAMIC Mon May 5 12:18:22 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux Single nvme device is once again fine Run status group 0 (all jobs): READ: bw=3061MiB/s (3210MB/s), 3061MiB/s-3061MiB/s (3210MB/s- 3210MB/s), io=179GiB (193GB), run=60006-60006msec [root@penguin9 ~]# for i in 1 2 3 4 5; do ./run_fio.sh | grep -A1 "Run status group"; done Run status group 0 (all jobs): READ: bw=14.7GiB/s (15.7GB/s), 14.7GiB/s-14.7GiB/s (15.7GB/s- 15.7GB/s), io=880GiB (945GB), run=60003-60003msec Run status group 0 (all jobs): READ: bw=18.1GiB/s (19.4GB/s), 18.1GiB/s-18.1GiB/s (19.4GB/s- 19.4GB/s), io=1087GiB (1167GB), run=60003-60003msec Run status group 0 (all jobs): READ: bw=18.0GiB/s (19.4GB/s), 18.0GiB/s-18.0GiB/s (19.4GB/s- 19.4GB/s), io=1082GiB (1162GB), run=60003-60003msec Run status group 0 (all jobs): READ: bw=18.2GiB/s (19.5GB/s), 18.2GiB/s-18.2GiB/s (19.5GB/s- 19.5GB/s), io=1090GiB (1170GB), run=60005-60005msec ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 2025-05-05 17:39 ` Laurence Oberman @ 2025-05-22 15:07 ` Laurence Oberman 2025-05-23 9:39 ` Anton Gavriliuk 0 siblings, 1 reply; 11+ messages in thread From: Laurence Oberman @ 2025-05-22 15:07 UTC (permalink / raw) To: Dave Chinner, Anton Gavriliuk; +Cc: linux-nvme, linux-xfs, linux-block On Mon, 2025-05-05 at 13:39 -0400, Laurence Oberman wrote: > On Mon, 2025-05-05 at 09:21 -0400, Laurence Oberman wrote: > > On Mon, 2025-05-05 at 08:29 -0400, Laurence Oberman wrote: > > > On Mon, 2025-05-05 at 07:50 +1000, Dave Chinner wrote: > > > > [cc linux-block] > > > > > > > > [original bug report: > > > > https://lore.kernel.org/linux-xfs/CAAiJnjoo0--yp47UKZhbu8sNSZN6DZ-QzmZBMmtr1oC=fOOgAQ@mail.gmail.com/ > > > > ] > > > > > > > > On Sun, May 04, 2025 at 10:22:58AM +0300, Anton Gavriliuk > > > > wrote: > > > > > > What's the comparitive performance of an identical read > > > > > > profile > > > > > > directly on the raw MD raid0 device? > > > > > > > > > > Rocky 9.5 (5.14.0-503.40.1.el9_5.x86_64) > > > > > > > > > > [root@localhost ~]# df -mh /mnt > > > > > Filesystem Size Used Avail Use% Mounted on > > > > > /dev/md127 35T 1.3T 34T 4% /mnt > > > > > > > > > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > > > > > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 -- > > > > > exitall > > > > > --group_reporting --ioengine=libaio --runtime=30 --time_based > > > > > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB- > > > > > 256KiB, > > > > > (T) > > > > > 256KiB-256KiB, ioengine=libaio, iodepth=64 > > > > > fio-3.39-44-g19d9 > > > > > Starting 1 process > > > > > Jobs: 1 (f=1): [R(1)][100.0%][r=81.4GiB/s][r=334k IOPS][eta > > > > > 00m:00s] > > > > > test: (groupid=0, jobs=1): err= 0: pid=43189: Sun May 4 > > > > > 08:22:12 > > > > > 2025 > > > > > read: IOPS=363k, BW=88.5GiB/s (95.1GB/s)(2656GiB/30001msec) > > > > > slat (nsec): min=971, max=312380, avg=1817.92, > > > > > stdev=1367.75 > > > > > clat (usec): min=78, max=1351, avg=174.46, stdev=28.86 > > > > > lat (usec): min=80, max=1352, avg=176.27, stdev=28.81 > > > > > > > > > > Fedora 42 (6.14.5-300.fc42.x86_64) > > > > > > > > > > [root@localhost anton]# df -mh /mnt > > > > > Filesystem Size Used Avail Use% Mounted on > > > > > /dev/md127 35T 1.3T 34T 4% /mnt > > > > > > > > > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > > > > > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 -- > > > > > exitall > > > > > --group_reporting --ioengine=libaio --runtime=30 --time_based > > > > > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB- > > > > > 256KiB, > > > > > (T) > > > > > 256KiB-256KiB, ioengine=libaio, iodepth=64 > > > > > fio-3.39-44-g19d9 > > > > > Starting 1 process > > > > > Jobs: 1 (f=1): [R(1)][100.0%][r=41.0GiB/s][r=168k IOPS][eta > > > > > 00m:00s] > > > > > test: (groupid=0, jobs=1): err= 0: pid=5685: Sun May 4 > > > > > 10:14:00 > > > > > 2025 > > > > > read: IOPS=168k, BW=41.0GiB/s (44.1GB/s)(1231GiB/30001msec) > > > > > slat (usec): min=3, max=273, avg= 5.63, stdev= 1.48 > > > > > clat (usec): min=67, max=2800, avg=374.99, stdev=29.90 > > > > > lat (usec): min=72, max=2914, avg=380.62, stdev=30.22 > > > > > > > > So the MD block device shows the same read performance as the > > > > filesystem on top of it. That means this is a regression at the > > > > MD > > > > device layer or in the block/driver layers below it. i.e. it is > > > > not > > > > an XFS of filesystem issue at all. > > > > > > > > -Dave. > > > > > > I have a lab setup, let me see if I can also reproduce and then > > > trace > > > this to see where it is spending the time > > > > > > > > > Not seeing 1/2 the bandwidth but also significantly slower on > > Fedora42 > > kernel. > > I will trace it > > > > 9.5 kernel - 5.14.0-503.40.1.el9_5.x86_64 > > > > Run status group 0 (all jobs): > > READ: bw=14.7GiB/s (15.8GB/s), 14.7GiB/s-14.7GiB/s (15.8GB/s- > > 15.8GB/s), io=441GiB (473GB), run=30003-30003msec > > > > Fedora42 kernel - 6.14.5-300.fc42.x86_64 > > > > Run status group 0 (all jobs): > > READ: bw=10.4GiB/s (11.2GB/s), 10.4GiB/s-10.4GiB/s (11.2GB/s- > > 11.2GB/s), io=313GiB (336GB), run=30001-30001msec > > > > > > > > > > Fedora42 kernel issue > > While my difference is not as severe we do see a consistently lower > performance on the Fedora > kernel. (6.14.5-300.fc42.x86_64) > > When I remove the software raid and run against a single NVME we > converge to be much closer. > Also latest upstream does not show this regression either. > > Not sure yet what is in our Fedora kernel causing this. > We will work it via the Bugzilla > > Regards > Laurence > > TLDR > > > Fedora Kernel > ------------- > root@penguin9 blktracefedora]# uname -a > Linux penguin9.2 6.14.5-300.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Fri > May > 2 14:16:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux > > 5 runs of the fio against /dev/md1 > > [root@penguin9 ~]# for i in 1 2 3 4 5 > > do > > ./run_fio.sh | grep -A1 "Run status group" > > done > Run status group 0 (all jobs): > READ: bw=11.3GiB/s (12.2GB/s), 11.3GiB/s-11.3GiB/s (12.2GB/s- > 12.2GB/s), io=679GiB (729GB), run=60001-60001msec > Run status group 0 (all jobs): > READ: bw=11.2GiB/s (12.0GB/s), 11.2GiB/s-11.2GiB/s (12.0GB/s- > 12.0GB/s), io=669GiB (718GB), run=60001-60001msec > Run status group 0 (all jobs): > READ: bw=11.4GiB/s (12.2GB/s), 11.4GiB/s-11.4GiB/s (12.2GB/s- > 12.2GB/s), io=682GiB (733GB), run=60001-60001msec > Run status group 0 (all jobs): > READ: bw=11.1GiB/s (11.9GB/s), 11.1GiB/s-11.1GiB/s (11.9GB/s- > 11.9GB/s), io=664GiB (713GB), run=60001-60001msec > Run status group 0 (all jobs): > READ: bw=11.3GiB/s (12.1GB/s), 11.3GiB/s-11.3GiB/s (12.1GB/s- > 12.1GB/s), io=678GiB (728GB), run=60001-60001msec > > RHEL9.5 > ------------ > Linux penguin9.2 5.14.0-503.40.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC > Thu Apr 24 08:27:29 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux > > [root@penguin9 ~]# for i in 1 2 3 4 5; do ./run_fio.sh | grep -A1 > "Run > status group"; done > Run status group 0 (all jobs): > READ: bw=14.9GiB/s (16.0GB/s), 14.9GiB/s-14.9GiB/s (16.0GB/s- > 16.0GB/s), io=894GiB (960GB), run=60003-60003msec > Run status group 0 (all jobs): > READ: bw=14.6GiB/s (15.6GB/s), 14.6GiB/s-14.6GiB/s (15.6GB/s- > 15.6GB/s), io=873GiB (938GB), run=60003-60003msec > Run status group 0 (all jobs): > READ: bw=14.9GiB/s (16.0GB/s), 14.9GiB/s-14.9GiB/s (16.0GB/s- > 16.0GB/s), io=892GiB (958GB), run=60003-60003msec > Run status group 0 (all jobs): > READ: bw=14.5GiB/s (15.6GB/s), 14.5GiB/s-14.5GiB/s (15.6GB/s- > 15.6GB/s), io=872GiB (936GB), run=60003-60003msec > Run status group 0 (all jobs): > READ: bw=14.7GiB/s (15.8GB/s), 14.7GiB/s-14.7GiB/s (15.8GB/s- > 15.8GB/s), io=884GiB (950GB), run=60003-60003msec > > > Remove software raid from the layers and test just on a single nvme > --------------------------------------------------------------------- > - > > fio --name=test --rw=read --bs=256k --filename=/dev/nvme23n1 -- > direct=1 > --numjobs=1 --iodepth=64 --exitall --group_reporting -- > ioengine=libaio > --runtime=60 --time_based > > Linux penguin9.2 5.14.0-503.40.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC > Thu Apr 24 08:27:29 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux > > [root@penguin9 ~]# ./run_nvme_fio.sh > > Run status group 0 (all jobs): > READ: bw=3207MiB/s (3363MB/s), 3207MiB/s-3207MiB/s (3363MB/s- > 3363MB/s), io=188GiB (202GB), run=60005-60005msec > > > Back to fedora kernel > > [root@penguin9 ~]# uname -a > Linux penguin9.2 6.14.5-300.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Fri > May > 2 14:16:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux > > Within the margin of error > > Run status group 0 (all jobs): > READ: bw=3061MiB/s (3210MB/s), 3061MiB/s-3061MiB/s (3210MB/s- > 3210MB/s), io=179GiB (193GB), run=60006-60006msec > > > Try recent upstream kernel > --------------------------- > [root@penguin9 ~]# uname -a > Linux penguin9.2 6.13.0-rc7+ #2 SMP PREEMPT_DYNAMIC Mon May 5 > 10:59:12 > EDT 2025 x86_64 x86_64 x86_64 GNU/Linux > > [root@penguin9 ~]# for i in 1 2 3 4 5; do ./run_fio.sh | grep -A1 > "Run > status group"; done > Run status group 0 (all jobs): > READ: bw=14.6GiB/s (15.7GB/s), 14.6GiB/s-14.6GiB/s (15.7GB/s- > 15.7GB/s), io=876GiB (941GB), run=60003-60003msec > Run status group 0 (all jobs): > READ: bw=14.8GiB/s (15.9GB/s), 14.8GiB/s-14.8GiB/s (15.9GB/s- > 15.9GB/s), io=891GiB (957GB), run=60003-60003msec > Run status group 0 (all jobs): > READ: bw=14.8GiB/s (15.9GB/s), 14.8GiB/s-14.8GiB/s (15.9GB/s- > 15.9GB/s), io=890GiB (956GB), run=60003-60003msec > Run status group 0 (all jobs): > READ: bw=14.5GiB/s (15.6GB/s), 14.5GiB/s-14.5GiB/s (15.6GB/s- > 15.6GB/s), io=871GiB (935GB), run=60003-60003msec > > > Update to latest upstream > ------------------------- > > [root@penguin9 ~]# uname -a > Linux penguin9.2 6.15.0-rc5 #1 SMP PREEMPT_DYNAMIC Mon May 5 > 12:18:22 > EDT 2025 x86_64 x86_64 x86_64 GNU/Linux > > Single nvme device is once again fine > > Run status group 0 (all jobs): > READ: bw=3061MiB/s (3210MB/s), 3061MiB/s-3061MiB/s (3210MB/s- > 3210MB/s), io=179GiB (193GB), run=60006-60006msec > > > [root@penguin9 ~]# for i in 1 2 3 4 5; do ./run_fio.sh | grep -A1 > "Run > status group"; done > Run status group 0 (all jobs): > READ: bw=14.7GiB/s (15.7GB/s), 14.7GiB/s-14.7GiB/s (15.7GB/s- > 15.7GB/s), io=880GiB (945GB), run=60003-60003msec > Run status group 0 (all jobs): > READ: bw=18.1GiB/s (19.4GB/s), 18.1GiB/s-18.1GiB/s (19.4GB/s- > 19.4GB/s), io=1087GiB (1167GB), run=60003-60003msec > Run status group 0 (all jobs): > READ: bw=18.0GiB/s (19.4GB/s), 18.0GiB/s-18.0GiB/s (19.4GB/s- > 19.4GB/s), io=1082GiB (1162GB), run=60003-60003msec > Run status group 0 (all jobs): > READ: bw=18.2GiB/s (19.5GB/s), 18.2GiB/s-18.2GiB/s (19.5GB/s- > 19.5GB/s), io=1090GiB (1170GB), run=60005-60005msec > > This fell of my radar, I aologize and I was on PTO last week. Here is the Fedora kernel to install as mentioned https://people.redhat.com/loberman/customer/.fedora/ tar hxvf fedora_kernel.tar.xz rpm -ivh --force --nodeps *.rpm ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 2025-05-22 15:07 ` Laurence Oberman @ 2025-05-23 9:39 ` Anton Gavriliuk 0 siblings, 0 replies; 11+ messages in thread From: Anton Gavriliuk @ 2025-05-23 9:39 UTC (permalink / raw) To: Laurence Oberman; +Cc: Dave Chinner, linux-nvme, linux-xfs, linux-block > This fell of my radar, I aologize and I was on PTO last week. > Here is the Fedora kernel to install as mentioned > https://people.redhat.com/loberman/customer/.fedora/ > tar hxvf fedora_kernel.tar.xz > rpm -ivh --force --nodeps *.rpm Rocky 9.5 kernel is still faster Fedora 42 kernel. [root@memverge4 ~]# uname -r 6.14.5-300.fc42.x86_64 [root@memverge4 ~]# [root@memverge4 ~]# cat /etc/*release NAME="Rocky Linux" VERSION="9.5 (Blue Onyx)" ID="rocky" ID_LIKE="rhel centos fedora" VERSION_ID="9.5" PLATFORM_ID="platform:el9" PRETTY_NAME="Rocky Linux 9.5 (Blue Onyx)" ANSI_COLOR="0;32" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:rocky:rocky:9::baseos" HOME_URL="https://rockylinux.org/" VENDOR_NAME="RESF" VENDOR_URL="https://resf.org/" BUG_REPORT_URL="https://bugs.rockylinux.org/" SUPPORT_END="2032-05-31" ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9" ROCKY_SUPPORT_PRODUCT_VERSION="9.5" REDHAT_SUPPORT_PRODUCT="Rocky Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.5" Rocky Linux release 9.5 (Blue Onyx) Rocky Linux release 9.5 (Blue Onyx) Rocky Linux release 9.5 (Blue Onyx) Block access - [root@memverge4 ~]# fio --name=test --rw=read --bs=256k --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting --ioengine=libaio --runtime=30 --time_based test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64 fio-3.40 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=34.7GiB/s][r=142k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=3566: Fri May 23 12:20:18 2025 read: IOPS=142k, BW=34.7GiB/s (37.2GB/s)(1040GiB/30001msec) slat (usec): min=3, max=1065, avg= 6.68, stdev= 2.19 clat (usec): min=75, max=2712, avg=443.75, stdev=36.12 lat (usec): min=83, max=2835, avg=450.43, stdev=36.49 File access - [root@memverge4 ~]# mount /dev/md127 /mnt [root@memverge4 ~]# fio --name=test --rw=read --bs=256k --filename=/mnt/testfile --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting --ioengine=libaio --runtime=30 --time_based test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64 fio-3.40 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=41.4GiB/s][r=169k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=3666: Fri May 23 12:21:33 2025 read: IOPS=172k, BW=42.1GiB/s (45.2GB/s)(1263GiB/30001msec) slat (usec): min=3, max=1054, avg= 5.46, stdev= 1.81 clat (usec): min=118, max=2500, avg=365.50, stdev=28.08 lat (usec): min=121, max=2794, avg=370.96, stdev=28.35 Back to latest 9.5 kernel (5.14.0-503.40.1.el9_5.x86_64) Block access - [root@memverge4 ~]# fio --name=test --rw=read --bs=256k --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting --ioengine=libaio --runtime=30 --time_based test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64 fio-3.40 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=70.8GiB/s][r=290k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=6121: Fri May 23 12:35:22 2025 read: IOPS=287k, BW=70.1GiB/s (75.3GB/s)(2104GiB/30001msec) slat (nsec): min=1492, max=165338, avg=3029.64, stdev=1544.70 clat (usec): min=71, max=1069, avg=219.56, stdev=21.22 lat (usec): min=74, max=1233, avg=222.59, stdev=21.34 File access - [root@memverge4 ~]# mount /dev/md127 /mnt [root@memverge4 ~]# fio --name=test --rw=read --bs=256k --filename=/mnt/testfile --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting --ioengine=libaio --runtime=30 --time_based test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64 fio-3.40 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=73.5GiB/s][r=301k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=6200: Fri May 23 12:36:47 2025 read: IOPS=301k, BW=73.4GiB/s (78.8GB/s)(2201GiB/30001msec) slat (nsec): min=1443, max=291427, avg=2951.98, stdev=1952.66 clat (usec): min=118, max=1449, avg=209.84, stdev=23.13 lat (usec): min=121, max=1562, avg=212.79, stdev=23.23 Anton чт, 22 мая 2025 г. в 18:08, Laurence Oberman <loberman@redhat.com>: > > On Mon, 2025-05-05 at 13:39 -0400, Laurence Oberman wrote: > > On Mon, 2025-05-05 at 09:21 -0400, Laurence Oberman wrote: > > > On Mon, 2025-05-05 at 08:29 -0400, Laurence Oberman wrote: > > > > On Mon, 2025-05-05 at 07:50 +1000, Dave Chinner wrote: > > > > > [cc linux-block] > > > > > > > > > > [original bug report: > > > > > https://lore.kernel.org/linux-xfs/CAAiJnjoo0--yp47UKZhbu8sNSZN6DZ-QzmZBMmtr1oC=fOOgAQ@mail.gmail.com/ > > > > > ] > > > > > > > > > > On Sun, May 04, 2025 at 10:22:58AM +0300, Anton Gavriliuk > > > > > wrote: > > > > > > > What's the comparitive performance of an identical read > > > > > > > profile > > > > > > > directly on the raw MD raid0 device? > > > > > > > > > > > > Rocky 9.5 (5.14.0-503.40.1.el9_5.x86_64) > > > > > > > > > > > > [root@localhost ~]# df -mh /mnt > > > > > > Filesystem Size Used Avail Use% Mounted on > > > > > > /dev/md127 35T 1.3T 34T 4% /mnt > > > > > > > > > > > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > > > > > > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 -- > > > > > > exitall > > > > > > --group_reporting --ioengine=libaio --runtime=30 --time_based > > > > > > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB- > > > > > > 256KiB, > > > > > > (T) > > > > > > 256KiB-256KiB, ioengine=libaio, iodepth=64 > > > > > > fio-3.39-44-g19d9 > > > > > > Starting 1 process > > > > > > Jobs: 1 (f=1): [R(1)][100.0%][r=81.4GiB/s][r=334k IOPS][eta > > > > > > 00m:00s] > > > > > > test: (groupid=0, jobs=1): err= 0: pid=43189: Sun May 4 > > > > > > 08:22:12 > > > > > > 2025 > > > > > > read: IOPS=363k, BW=88.5GiB/s (95.1GB/s)(2656GiB/30001msec) > > > > > > slat (nsec): min=971, max=312380, avg=1817.92, > > > > > > stdev=1367.75 > > > > > > clat (usec): min=78, max=1351, avg=174.46, stdev=28.86 > > > > > > lat (usec): min=80, max=1352, avg=176.27, stdev=28.81 > > > > > > > > > > > > Fedora 42 (6.14.5-300.fc42.x86_64) > > > > > > > > > > > > [root@localhost anton]# df -mh /mnt > > > > > > Filesystem Size Used Avail Use% Mounted on > > > > > > /dev/md127 35T 1.3T 34T 4% /mnt > > > > > > > > > > > > [root@localhost ~]# fio --name=test --rw=read --bs=256k > > > > > > --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 -- > > > > > > exitall > > > > > > --group_reporting --ioengine=libaio --runtime=30 --time_based > > > > > > test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB- > > > > > > 256KiB, > > > > > > (T) > > > > > > 256KiB-256KiB, ioengine=libaio, iodepth=64 > > > > > > fio-3.39-44-g19d9 > > > > > > Starting 1 process > > > > > > Jobs: 1 (f=1): [R(1)][100.0%][r=41.0GiB/s][r=168k IOPS][eta > > > > > > 00m:00s] > > > > > > test: (groupid=0, jobs=1): err= 0: pid=5685: Sun May 4 > > > > > > 10:14:00 > > > > > > 2025 > > > > > > read: IOPS=168k, BW=41.0GiB/s (44.1GB/s)(1231GiB/30001msec) > > > > > > slat (usec): min=3, max=273, avg= 5.63, stdev= 1.48 > > > > > > clat (usec): min=67, max=2800, avg=374.99, stdev=29.90 > > > > > > lat (usec): min=72, max=2914, avg=380.62, stdev=30.22 > > > > > > > > > > So the MD block device shows the same read performance as the > > > > > filesystem on top of it. That means this is a regression at the > > > > > MD > > > > > device layer or in the block/driver layers below it. i.e. it is > > > > > not > > > > > an XFS of filesystem issue at all. > > > > > > > > > > -Dave. > > > > > > > > I have a lab setup, let me see if I can also reproduce and then > > > > trace > > > > this to see where it is spending the time > > > > > > > > > > > > > Not seeing 1/2 the bandwidth but also significantly slower on > > > Fedora42 > > > kernel. > > > I will trace it > > > > > > 9.5 kernel - 5.14.0-503.40.1.el9_5.x86_64 > > > > > > Run status group 0 (all jobs): > > > READ: bw=14.7GiB/s (15.8GB/s), 14.7GiB/s-14.7GiB/s (15.8GB/s- > > > 15.8GB/s), io=441GiB (473GB), run=30003-30003msec > > > > > > Fedora42 kernel - 6.14.5-300.fc42.x86_64 > > > > > > Run status group 0 (all jobs): > > > READ: bw=10.4GiB/s (11.2GB/s), 10.4GiB/s-10.4GiB/s (11.2GB/s- > > > 11.2GB/s), io=313GiB (336GB), run=30001-30001msec > > > > > > > > > > > > > > > > Fedora42 kernel issue > > > > While my difference is not as severe we do see a consistently lower > > performance on the Fedora > > kernel. (6.14.5-300.fc42.x86_64) > > > > When I remove the software raid and run against a single NVME we > > converge to be much closer. > > Also latest upstream does not show this regression either. > > > > Not sure yet what is in our Fedora kernel causing this. > > We will work it via the Bugzilla > > > > Regards > > Laurence > > > > TLDR > > > > > > Fedora Kernel > > ------------- > > root@penguin9 blktracefedora]# uname -a > > Linux penguin9.2 6.14.5-300.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Fri > > May > > 2 14:16:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux > > > > 5 runs of the fio against /dev/md1 > > > > [root@penguin9 ~]# for i in 1 2 3 4 5 > > > do > > > ./run_fio.sh | grep -A1 "Run status group" > > > done > > Run status group 0 (all jobs): > > READ: bw=11.3GiB/s (12.2GB/s), 11.3GiB/s-11.3GiB/s (12.2GB/s- > > 12.2GB/s), io=679GiB (729GB), run=60001-60001msec > > Run status group 0 (all jobs): > > READ: bw=11.2GiB/s (12.0GB/s), 11.2GiB/s-11.2GiB/s (12.0GB/s- > > 12.0GB/s), io=669GiB (718GB), run=60001-60001msec > > Run status group 0 (all jobs): > > READ: bw=11.4GiB/s (12.2GB/s), 11.4GiB/s-11.4GiB/s (12.2GB/s- > > 12.2GB/s), io=682GiB (733GB), run=60001-60001msec > > Run status group 0 (all jobs): > > READ: bw=11.1GiB/s (11.9GB/s), 11.1GiB/s-11.1GiB/s (11.9GB/s- > > 11.9GB/s), io=664GiB (713GB), run=60001-60001msec > > Run status group 0 (all jobs): > > READ: bw=11.3GiB/s (12.1GB/s), 11.3GiB/s-11.3GiB/s (12.1GB/s- > > 12.1GB/s), io=678GiB (728GB), run=60001-60001msec > > > > RHEL9.5 > > ------------ > > Linux penguin9.2 5.14.0-503.40.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC > > Thu Apr 24 08:27:29 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux > > > > [root@penguin9 ~]# for i in 1 2 3 4 5; do ./run_fio.sh | grep -A1 > > "Run > > status group"; done > > Run status group 0 (all jobs): > > READ: bw=14.9GiB/s (16.0GB/s), 14.9GiB/s-14.9GiB/s (16.0GB/s- > > 16.0GB/s), io=894GiB (960GB), run=60003-60003msec > > Run status group 0 (all jobs): > > READ: bw=14.6GiB/s (15.6GB/s), 14.6GiB/s-14.6GiB/s (15.6GB/s- > > 15.6GB/s), io=873GiB (938GB), run=60003-60003msec > > Run status group 0 (all jobs): > > READ: bw=14.9GiB/s (16.0GB/s), 14.9GiB/s-14.9GiB/s (16.0GB/s- > > 16.0GB/s), io=892GiB (958GB), run=60003-60003msec > > Run status group 0 (all jobs): > > READ: bw=14.5GiB/s (15.6GB/s), 14.5GiB/s-14.5GiB/s (15.6GB/s- > > 15.6GB/s), io=872GiB (936GB), run=60003-60003msec > > Run status group 0 (all jobs): > > READ: bw=14.7GiB/s (15.8GB/s), 14.7GiB/s-14.7GiB/s (15.8GB/s- > > 15.8GB/s), io=884GiB (950GB), run=60003-60003msec > > > > > > Remove software raid from the layers and test just on a single nvme > > --------------------------------------------------------------------- > > - > > > > fio --name=test --rw=read --bs=256k --filename=/dev/nvme23n1 -- > > direct=1 > > --numjobs=1 --iodepth=64 --exitall --group_reporting -- > > ioengine=libaio > > --runtime=60 --time_based > > > > Linux penguin9.2 5.14.0-503.40.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC > > Thu Apr 24 08:27:29 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux > > > > [root@penguin9 ~]# ./run_nvme_fio.sh > > > > Run status group 0 (all jobs): > > READ: bw=3207MiB/s (3363MB/s), 3207MiB/s-3207MiB/s (3363MB/s- > > 3363MB/s), io=188GiB (202GB), run=60005-60005msec > > > > > > Back to fedora kernel > > > > [root@penguin9 ~]# uname -a > > Linux penguin9.2 6.14.5-300.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Fri > > May > > 2 14:16:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux > > > > Within the margin of error > > > > Run status group 0 (all jobs): > > READ: bw=3061MiB/s (3210MB/s), 3061MiB/s-3061MiB/s (3210MB/s- > > 3210MB/s), io=179GiB (193GB), run=60006-60006msec > > > > > > Try recent upstream kernel > > --------------------------- > > [root@penguin9 ~]# uname -a > > Linux penguin9.2 6.13.0-rc7+ #2 SMP PREEMPT_DYNAMIC Mon May 5 > > 10:59:12 > > EDT 2025 x86_64 x86_64 x86_64 GNU/Linux > > > > [root@penguin9 ~]# for i in 1 2 3 4 5; do ./run_fio.sh | grep -A1 > > "Run > > status group"; done > > Run status group 0 (all jobs): > > READ: bw=14.6GiB/s (15.7GB/s), 14.6GiB/s-14.6GiB/s (15.7GB/s- > > 15.7GB/s), io=876GiB (941GB), run=60003-60003msec > > Run status group 0 (all jobs): > > READ: bw=14.8GiB/s (15.9GB/s), 14.8GiB/s-14.8GiB/s (15.9GB/s- > > 15.9GB/s), io=891GiB (957GB), run=60003-60003msec > > Run status group 0 (all jobs): > > READ: bw=14.8GiB/s (15.9GB/s), 14.8GiB/s-14.8GiB/s (15.9GB/s- > > 15.9GB/s), io=890GiB (956GB), run=60003-60003msec > > Run status group 0 (all jobs): > > READ: bw=14.5GiB/s (15.6GB/s), 14.5GiB/s-14.5GiB/s (15.6GB/s- > > 15.6GB/s), io=871GiB (935GB), run=60003-60003msec > > > > > > Update to latest upstream > > ------------------------- > > > > [root@penguin9 ~]# uname -a > > Linux penguin9.2 6.15.0-rc5 #1 SMP PREEMPT_DYNAMIC Mon May 5 > > 12:18:22 > > EDT 2025 x86_64 x86_64 x86_64 GNU/Linux > > > > Single nvme device is once again fine > > > > Run status group 0 (all jobs): > > READ: bw=3061MiB/s (3210MB/s), 3061MiB/s-3061MiB/s (3210MB/s- > > 3210MB/s), io=179GiB (193GB), run=60006-60006msec > > > > > > [root@penguin9 ~]# for i in 1 2 3 4 5; do ./run_fio.sh | grep -A1 > > "Run > > status group"; done > > Run status group 0 (all jobs): > > READ: bw=14.7GiB/s (15.7GB/s), 14.7GiB/s-14.7GiB/s (15.7GB/s- > > 15.7GB/s), io=880GiB (945GB), run=60003-60003msec > > Run status group 0 (all jobs): > > READ: bw=18.1GiB/s (19.4GB/s), 18.1GiB/s-18.1GiB/s (19.4GB/s- > > 19.4GB/s), io=1087GiB (1167GB), run=60003-60003msec > > Run status group 0 (all jobs): > > READ: bw=18.0GiB/s (19.4GB/s), 18.0GiB/s-18.0GiB/s (19.4GB/s- > > 19.4GB/s), io=1082GiB (1162GB), run=60003-60003msec > > Run status group 0 (all jobs): > > READ: bw=18.2GiB/s (19.5GB/s), 18.2GiB/s-18.2GiB/s (19.5GB/s- > > 19.5GB/s), io=1090GiB (1170GB), run=60005-60005msec > > > > > > This fell of my radar, I aologize and I was on PTO last week. > Here is the Fedora kernel to install as mentioned > https://people.redhat.com/loberman/customer/.fedora/ > > tar hxvf fedora_kernel.tar.xz > rpm -ivh --force --nodeps *.rpm > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 2025-05-05 13:21 ` Laurence Oberman 2025-05-05 17:39 ` Laurence Oberman @ 2025-05-05 22:56 ` Dave Chinner 2025-05-06 11:03 ` Anton Gavriliuk 1 sibling, 1 reply; 11+ messages in thread From: Dave Chinner @ 2025-05-05 22:56 UTC (permalink / raw) To: Laurence Oberman; +Cc: Anton Gavriliuk, linux-nvme, linux-xfs, linux-block On Mon, May 05, 2025 at 09:21:19AM -0400, Laurence Oberman wrote: > On Mon, 2025-05-05 at 08:29 -0400, Laurence Oberman wrote: > > On Mon, 2025-05-05 at 07:50 +1000, Dave Chinner wrote: > > > So the MD block device shows the same read performance as the > > > filesystem on top of it. That means this is a regression at the MD > > > device layer or in the block/driver layers below it. i.e. it is not > > > an XFS of filesystem issue at all. > > > > > > -Dave. > > > > I have a lab setup, let me see if I can also reproduce and then trace > > this to see where it is spending the time > > > > > Not seeing 1/2 the bandwidth but also significantly slower on Fedora42 > kernel. > I will trace it > > 9.5 kernel - 5.14.0-503.40.1.el9_5.x86_64 > > Run status group 0 (all jobs): > READ: bw=14.7GiB/s (15.8GB/s), 14.7GiB/s-14.7GiB/s (15.8GB/s- > 15.8GB/s), io=441GiB (473GB), run=30003-30003msec > > Fedora42 kernel - 6.14.5-300.fc42.x86_64 > > Run status group 0 (all jobs): > READ: bw=10.4GiB/s (11.2GB/s), 10.4GiB/s-10.4GiB/s (11.2GB/s- > 11.2GB/s), io=313GiB (336GB), run=30001-30001msec So is this MD chunk size related? i.e. what is the chunk size the MD device? Is it smaller than the IO size (256kB) or larger? Does the regression go away if the chunk size matches the IO size, or if the IO size vs chunk size relationship is reversed? -Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 2025-05-05 22:56 ` Dave Chinner @ 2025-05-06 11:03 ` Anton Gavriliuk 2025-05-06 21:46 ` Dave Chinner 0 siblings, 1 reply; 11+ messages in thread From: Anton Gavriliuk @ 2025-05-06 11:03 UTC (permalink / raw) To: Dave Chinner; +Cc: Laurence Oberman, linux-nvme, linux-xfs, linux-block > So is this MD chunk size related? i.e. what is the chunk size > the MD device? Is it smaller than the IO size (256kB) or larger? > Does the regression go away if the chunk size matches the IO size, > or if the IO size vs chunk size relationship is reversed? According to the output below, the chunk size is 512K, [root@localhost anton]# mdadm -D /dev/md127 /dev/md127: Version : 1.2 Creation Time : Thu Apr 17 14:58:23 2025 Raid Level : raid0 Array Size : 37505814528 (34.93 TiB 38.41 TB) Raid Devices : 12 Total Devices : 12 Persistence : Superblock is persistent Update Time : Thu Apr 17 14:58:23 2025 State : clean Active Devices : 12 Working Devices : 12 Failed Devices : 0 Spare Devices : 0 Layout : original Chunk Size : 512K Consistency Policy : none Name : localhost.localdomain:127 (local to host localhost.localdomain) UUID : 2fadc96b:f37753af:f3b528a0:067c320d Events : 0 Number Major Minor RaidDevice State 0 259 15 0 active sync /dev/nvme7n1 1 259 27 1 active sync /dev/nvme0n1 2 259 10 2 active sync /dev/nvme1n1 3 259 28 3 active sync /dev/nvme2n1 4 259 13 4 active sync /dev/nvme8n1 5 259 22 5 active sync /dev/nvme5n1 6 259 26 6 active sync /dev/nvme3n1 7 259 16 7 active sync /dev/nvme4n1 8 259 24 8 active sync /dev/nvme9n1 9 259 14 9 active sync /dev/nvme10n1 10 259 25 10 active sync /dev/nvme11n1 11 259 12 11 active sync /dev/nvme12n1 [root@localhost anton]# uname -r 6.14.5-300.fc42.x86_64 [root@localhost anton]# cat /proc/mdstat Personalities : [raid0] md127 : active raid0 nvme4n1[7] nvme1n1[2] nvme12n1[11] nvme7n1[0] nvme9n1[8] nvme11n1[10] nvme2n1[3] nvme8n1[4] nvme0n1[1] nvme5n1[5] nvme3n1[6] nvme10n1[9] 37505814528 blocks super 1.2 512k chunks unused devices: <none> [root@localhost anton]# When I/O size is less 512K [root@localhost ~]# fio --name=test --rw=read --bs=256k --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting --ioengine=libaio --runtime=30 --time_based test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64 fio-3.39-44-g19d9 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=48.1GiB/s][r=197k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=14340: Tue May 6 13:59:23 2025 read: IOPS=197k, BW=48.0GiB/s (51.6GB/s)(1441GiB/30001msec) slat (usec): min=3, max=1041, avg= 4.74, stdev= 1.48 clat (usec): min=76, max=2042, avg=320.30, stdev=26.82 lat (usec): min=79, max=2160, avg=325.04, stdev=27.08 When I/O size is greater 512K [root@localhost ~]# fio --name=test --rw=read --bs=1024k --filename=/dev/md127 --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting --ioengine=libaio --runtime=30 --time_based test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64 fio-3.39-44-g19d9 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=63.7GiB/s][r=65.2k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=14395: Tue May 6 14:00:28 2025 read: IOPS=64.6k, BW=63.0GiB/s (67.7GB/s)(1891GiB/30001msec) slat (usec): min=9, max=1045, avg=15.12, stdev= 3.84 clat (usec): min=81, max=18494, avg=975.87, stdev=112.11 lat (usec): min=96, max=18758, avg=990.99, stdev=113.49 But still much worse than with 256k on Rocky 9.5 Anton вт, 6 мая 2025 г. в 01:56, Dave Chinner <david@fromorbit.com>: > > On Mon, May 05, 2025 at 09:21:19AM -0400, Laurence Oberman wrote: > > On Mon, 2025-05-05 at 08:29 -0400, Laurence Oberman wrote: > > > On Mon, 2025-05-05 at 07:50 +1000, Dave Chinner wrote: > > > > So the MD block device shows the same read performance as the > > > > filesystem on top of it. That means this is a regression at the MD > > > > device layer or in the block/driver layers below it. i.e. it is not > > > > an XFS of filesystem issue at all. > > > > > > > > -Dave. > > > > > > I have a lab setup, let me see if I can also reproduce and then trace > > > this to see where it is spending the time > > > > > > > > > Not seeing 1/2 the bandwidth but also significantly slower on Fedora42 > > kernel. > > I will trace it > > > > 9.5 kernel - 5.14.0-503.40.1.el9_5.x86_64 > > > > Run status group 0 (all jobs): > > READ: bw=14.7GiB/s (15.8GB/s), 14.7GiB/s-14.7GiB/s (15.8GB/s- > > 15.8GB/s), io=441GiB (473GB), run=30003-30003msec > > > > Fedora42 kernel - 6.14.5-300.fc42.x86_64 > > > > Run status group 0 (all jobs): > > READ: bw=10.4GiB/s (11.2GB/s), 10.4GiB/s-10.4GiB/s (11.2GB/s- > > 11.2GB/s), io=313GiB (336GB), run=30001-30001msec > > So is this MD chunk size related? i.e. what is the chunk size > the MD device? Is it smaller than the IO size (256kB) or larger? > Does the regression go away if the chunk size matches the IO size, > or if the IO size vs chunk size relationship is reversed? > > -Dave. > -- > Dave Chinner > david@fromorbit.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 2025-05-06 11:03 ` Anton Gavriliuk @ 2025-05-06 21:46 ` Dave Chinner 2025-05-07 12:26 ` Anton Gavriliuk 0 siblings, 1 reply; 11+ messages in thread From: Dave Chinner @ 2025-05-06 21:46 UTC (permalink / raw) To: Anton Gavriliuk; +Cc: Laurence Oberman, linux-nvme, linux-xfs, linux-block On Tue, May 06, 2025 at 02:03:37PM +0300, Anton Gavriliuk wrote: > > So is this MD chunk size related? i.e. what is the chunk size > > the MD device? Is it smaller than the IO size (256kB) or larger? > > Does the regression go away if the chunk size matches the IO size, > > or if the IO size vs chunk size relationship is reversed? > > According to the output below, the chunk size is 512K, Ok. `iostat -dxm 5` output during the fio run on both kernels will give us some indication of the differences in IO patterns, queue depths, etc. Silly question: if you use DM to create the same RAID 0 array with a dm table such as: 0 75011629056 striped 12 1024 /dev/nvme7n1 0 /dev/nvme0n1 0 .... /dev/nvme12n1 0 to create a similar 38TB raid 0 array, do you see the same perf degradation? -Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 2025-05-06 21:46 ` Dave Chinner @ 2025-05-07 12:26 ` Anton Gavriliuk 2025-05-07 21:59 ` Dave Chinner 0 siblings, 1 reply; 11+ messages in thread From: Anton Gavriliuk @ 2025-05-07 12:26 UTC (permalink / raw) To: Dave Chinner; +Cc: Laurence Oberman, linux-nvme, linux-xfs, linux-block [-- Attachment #1: Type: text/plain, Size: 2937 bytes --] > `iostat -dxm 5` output during the fio run on both kernels will give us some indication of the differences in IO patterns, queue depths, etc. iostat files attached. fedora 42 [root@localhost ~]# fio --name=test --rw=read --bs=256k --filename=/mnt/testfile --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting --ioengine=libaio --runtime=30 --time_based test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64 fio-3.39-44-g19d9 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=43.6GiB/s][r=179k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=18826: Wed May 7 13:44:38 2025 read: IOPS=178k, BW=43.4GiB/s (46.7GB/s)(1303GiB/30001msec) slat (usec): min=3, max=267, avg= 5.29, stdev= 1.62 clat (usec): min=147, max=2549, avg=354.18, stdev=28.87 lat (usec): min=150, max=2657, avg=359.47, stdev=29.15 rocky 9.5 [root@localhost ~]# fio --name=test --rw=read --bs=256k --filename=/mnt/testfile --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting --ioengine=libaio --runtime=30 --time_based test: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64 fio-3.39-44-g19d9 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=98.3GiB/s][r=403k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=10500: Wed May 7 15:16:39 2025 read: IOPS=403k, BW=98.4GiB/s (106GB/s)(2951GiB/30001msec) slat (nsec): min=1101, max=156185, avg=2087.89, stdev=1415.57 clat (usec): min=82, max=951, avg=156.56, stdev=20.19 lat (usec): min=83, max=1078, avg=158.65, stdev=20.25 > Silly question: if you use DM to create the same RAID 0 array with a dm table such as: > 0 75011629056 striped 12 1024 /dev/nvme7n1 0 /dev/nvme0n1 0 .... /dev/nvme12n1 0 > to create a similar 38TB raid 0 array, do you see the same perf degradation? Will check that tomorrow. Anton ср, 7 мая 2025 г. в 00:46, Dave Chinner <david@fromorbit.com>: > > On Tue, May 06, 2025 at 02:03:37PM +0300, Anton Gavriliuk wrote: > > > So is this MD chunk size related? i.e. what is the chunk size > > > the MD device? Is it smaller than the IO size (256kB) or larger? > > > Does the regression go away if the chunk size matches the IO size, > > > or if the IO size vs chunk size relationship is reversed? > > > > According to the output below, the chunk size is 512K, > > Ok. > > `iostat -dxm 5` output during the fio run on both kernels will give > us some indication of the differences in IO patterns, queue depths, > etc. > > Silly question: if you use DM to create the same RAID 0 array > with a dm table such as: > > 0 75011629056 striped 12 1024 /dev/nvme7n1 0 /dev/nvme0n1 0 .... /dev/nvme12n1 0 > > to create a similar 38TB raid 0 array, do you see the same perf > degradation? > > -Dave. > -- > Dave Chinner > david@fromorbit.com [-- Attachment #2: rocky_95_iostat_dxm_5 --] [-- Type: application/octet-stream, Size: 30896 bytes --] Linux 5.14.0-503.40.1.el9_5.x86_64 (localhost.localdomain) 05/07/2025 _x86_64_ (48 CPU) Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 9.50 0.54 0.00 0.00 0.12 58.52 2.46 0.13 0.00 0.00 0.62 53.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.88 dm-1 0.16 0.00 0.00 0.00 0.12 21.35 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.22 0.00 0.00 0.00 0.08 8.66 1.87 0.15 0.00 0.00 0.49 82.49 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 md127 56228.39 14018.86 0.00 0.00 0.12 255.30 0.01 0.00 0.00 0.00 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.56 14.54 nvme0c0n1 4685.58 1168.23 0.56 0.01 0.12 255.31 0.00 0.00 0.00 0.00 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme0n1 4685.58 1168.23 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme10c10n1 4685.79 1168.24 0.00 0.00 0.12 255.30 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme10n1 4685.80 1168.24 0.00 0.00 0.12 255.30 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme11c11n1 4685.59 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme11n1 4685.59 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme12c12n1 4685.61 1168.24 0.56 0.01 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme12n1 4685.63 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme1c1n1 4685.61 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme1n1 4685.63 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme2c2n1 4685.73 1168.24 0.00 0.00 0.12 255.30 0.00 0.00 0.00 0.00 0.00 288.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme2n1 4685.74 1168.24 0.00 0.00 0.12 255.30 0.00 0.00 0.00 0.00 0.00 288.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.55 14.53 nvme3c3n1 4685.63 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 224.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.55 14.53 nvme3n1 4685.61 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 224.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.55 14.53 nvme4c4n1 4685.63 1168.25 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme4n1 4685.63 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme5c5n1 13.57 0.60 1.51 10.00 0.22 44.99 4.91 0.33 1.55 23.97 4.20 69.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 1.00 nvme5n1 13.57 0.60 0.00 0.00 0.23 44.99 4.91 0.33 0.00 0.00 4.19 69.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.23 nvme6c6n1 4685.59 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme6n1 4685.57 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme7c7n1 4685.62 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme7n1 4685.63 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme8c8n1 4685.59 1168.24 0.55 0.01 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme8n1 4685.59 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.55 14.53 nvme9c9n1 4685.61 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 nvme9n1 4685.60 1168.24 0.00 0.00 0.12 255.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.54 14.53 Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 403871.40 100964.85 0.00 0.00 0.12 255.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 47.35 100.00 nvme0c0n1 33654.80 8413.70 4.20 0.01 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.96 nvme0n1 33655.20 8413.80 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.96 nvme10c10n1 33654.80 8413.70 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.90 99.92 nvme10n1 33654.40 8413.60 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.92 nvme11c11n1 33654.40 8413.60 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 100.00 nvme11n1 33654.80 8413.70 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 100.00 nvme12c12n1 33654.80 8413.70 4.00 0.01 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.96 nvme12n1 33654.00 8413.50 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.96 nvme1c1n1 33655.40 8413.85 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.92 nvme1n1 33654.40 8413.60 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.92 nvme2c2n1 33655.60 8413.90 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.96 nvme2n1 33654.60 8413.65 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 99.94 nvme3c3n1 33654.20 8413.55 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 99.94 nvme3n1 33655.00 8413.75 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 99.94 nvme4c4n1 33654.00 8413.50 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.94 nvme4n1 33654.20 8413.55 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.96 nvme5c5n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme5n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme6c6n1 33654.20 8413.55 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.94 nvme6n1 33654.80 8413.70 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.94 nvme7c7n1 33654.40 8413.60 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.98 nvme7n1 33653.80 8413.45 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 99.98 nvme8c8n1 33654.60 8413.65 3.80 0.01 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.95 99.98 nvme8n1 33654.60 8413.65 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.96 99.98 nvme9c9n1 33654.40 8413.65 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.90 99.98 nvme9n1 33654.80 8413.70 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.96 Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.00 0.00 0.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 404956.20 100416.80 0.00 0.00 0.12 253.92 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 47.36 100.00 nvme0c0n1 33745.00 8368.00 4.00 0.01 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.90 nvme0n1 33745.00 8368.00 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 99.90 nvme10c10n1 33745.60 8368.10 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.90 99.98 nvme10n1 33745.60 8368.10 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.98 nvme11c11n1 33745.00 8368.00 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.92 nvme11n1 33745.00 8368.00 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.92 nvme12c12n1 33745.40 8368.05 4.00 0.01 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.94 nvme12n1 33745.20 8368.00 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.94 nvme1c1n1 33745.20 8368.05 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.96 nvme1n1 33745.40 8368.10 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 99.96 nvme2c2n1 33745.20 8368.05 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.96 nvme2n1 33745.60 8368.15 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.96 nvme3c3n1 33745.20 8368.05 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.95 99.98 nvme3n1 33745.40 8368.10 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.97 99.96 nvme4c4n1 33745.20 8368.05 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.90 nvme4n1 33745.40 8368.10 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.90 nvme5c5n1 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.00 0.00 0.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme5n1 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.00 0.00 0.00 0.00 16.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme6c6n1 33745.00 8368.00 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.90 99.96 nvme6n1 33745.00 8368.00 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.90 99.96 nvme7c7n1 33745.40 8368.05 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.94 nvme7n1 33745.80 8368.15 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 99.94 nvme8c8n1 33745.00 8367.95 4.00 0.01 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.98 nvme8n1 33745.00 8367.95 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.98 nvme9c9n1 33745.40 8368.00 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.96 nvme9n1 33745.60 8368.10 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.95 99.96 Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 403128.60 100779.15 0.00 0.00 0.12 255.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 47.30 100.00 nvme0c0n1 33593.20 8398.30 4.00 0.01 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.98 nvme0n1 33593.20 8398.30 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.98 nvme10c10n1 33593.20 8398.30 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.90 99.94 nvme10n1 33593.20 8398.30 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.89 99.94 nvme11c11n1 33593.20 8398.30 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.94 nvme11n1 33593.00 8398.25 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 99.94 nvme12c12n1 33593.20 8398.30 4.00 0.01 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.90 100.00 nvme12n1 33593.60 8398.40 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 100.00 nvme1c1n1 33593.20 8398.30 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.98 nvme1n1 33593.00 8398.25 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.98 nvme2c2n1 33593.40 8398.35 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.94 nvme2n1 33593.00 8398.25 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.94 nvme3c3n1 33593.20 8398.30 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 100.00 nvme3n1 33593.00 8398.25 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.96 100.00 nvme4c4n1 33593.00 8398.25 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.92 nvme4n1 33592.80 8398.20 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.92 nvme5c5n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme5n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme6c6n1 33593.20 8398.30 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.90 99.96 nvme6n1 33593.00 8398.25 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.96 nvme7c7n1 33593.40 8398.35 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.92 nvme7n1 33593.40 8398.35 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 99.92 nvme8c8n1 33593.40 8398.35 4.00 0.01 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.95 100.00 nvme8n1 33593.20 8398.30 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.97 100.00 nvme9c9n1 33593.40 8398.35 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.90 99.96 nvme9n1 33593.00 8398.25 0.00 0.00 0.12 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.90 99.96 Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.40 0.00 0.00 0.00 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 405567.20 100569.55 0.00 0.00 0.12 253.92 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 47.51 100.00 nvme0c0n1 33796.20 8380.80 4.00 0.01 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.90 nvme0n1 33795.80 8380.70 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.90 nvme10c10n1 33796.40 8380.80 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.90 99.94 nvme10n1 33796.60 8380.85 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.94 nvme11c11n1 33796.00 8380.75 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.96 nvme11n1 33796.00 8380.75 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.96 99.96 nvme12c12n1 33796.80 8380.90 4.00 0.01 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.92 nvme12n1 33796.20 8380.75 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 99.92 nvme1c1n1 33796.20 8380.80 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.96 nvme1n1 33796.40 8380.85 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.96 nvme2c2n1 33796.00 8380.75 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.94 nvme2n1 33796.20 8380.80 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.95 99.92 nvme3c3n1 33796.20 8380.80 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.96 100.00 nvme3n1 33796.60 8380.90 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.97 100.00 nvme4c4n1 33796.60 8380.90 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.92 nvme4n1 33796.80 8380.95 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.92 nvme5c5n1 0.00 0.00 0.00 0.00 0.00 0.00 0.40 0.00 0.00 0.00 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme5n1 0.00 0.00 0.00 0.00 0.00 0.00 0.40 0.00 0.00 0.00 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme6c6n1 33795.80 8380.70 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.91 99.96 nvme6n1 33796.00 8380.75 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.92 99.96 nvme7c7n1 33796.20 8380.75 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 99.98 nvme7n1 33796.00 8380.70 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.97 99.96 nvme8c8n1 33795.80 8380.65 4.00 0.01 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 100.00 nvme8n1 33796.00 8380.70 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.94 100.00 nvme9c9n1 33796.20 8380.75 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.93 99.98 nvme9n1 33796.60 8380.85 0.00 0.00 0.12 253.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.97 99.98 [-- Attachment #3: fedora_42_iostat_dxm_5 --] [-- Type: application/octet-stream, Size: 23936 bytes --] Linux 6.14.5-300.fc42.x86_64 (localhost.localdomain) 05/07/2025 _x86_64_ (48 CPU) Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 0.06 0.00 0.00 0.00 0.14 25.06 0.09 0.00 0.00 0.00 0.69 30.39 0.00 0.05 0.00 0.00 7.40 283478.13 0.00 0.00 0.00 0.08 md127 236.92 62.75 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.70 206.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.13 nvme0c0n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 176.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme0n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.33 176.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme10c10n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme10n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme11c11n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme11n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme12c12n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme12n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme1c1n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme1n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 1.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme2c2n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme2n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 1.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme3c3n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme3n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme4c4n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme4n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme5c5n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme5n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme6c6n1 0.06 0.00 0.00 2.32 0.17 23.83 0.10 0.00 0.00 3.34 3.00 27.78 0.00 0.05 0.00 82.76 3.50 1443929.20 0.00 0.00 0.00 0.18 nvme6n1 0.06 0.00 0.00 0.00 0.15 23.83 0.10 0.00 0.00 0.00 2.93 27.78 0.00 0.05 0.00 0.00 3.50 1443929.20 0.00 0.00 0.00 0.04 nvme7c7n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 162.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme7n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.50 162.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme8c8n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 192.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme8n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 1.00 192.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme9c9n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 nvme9n1 19.74 5.23 0.00 0.00 0.10 271.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 zram0 0.00 0.00 0.00 0.00 0.00 21.18 0.00 0.00 0.00 0.00 0.00 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 177981.80 44494.15 0.00 0.00 0.10 255.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 18.63 100.00 nvme0c0n1 14831.60 3707.90 1.80 0.01 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 85.16 nvme0n1 14831.60 3707.90 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.54 84.14 nvme10c10n1 14831.20 3707.80 1.80 0.01 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.44 nvme10n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.56 nvme11c11n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.28 nvme11n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.08 nvme12c12n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.16 nvme12n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 83.14 nvme1c1n1 14831.60 3707.90 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.38 nvme1n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 83.62 nvme2c2n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.44 nvme2n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 83.62 nvme3c3n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.34 nvme3n1 14831.40 3707.85 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.56 nvme4c4n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.06 nvme4n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 83.20 nvme5c5n1 14831.20 3707.80 1.60 0.01 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.57 84.68 nvme5n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.54 83.96 nvme6c6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme7c7n1 14831.20 3707.80 0.00 0.00 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.16 nvme7n1 14831.00 3707.75 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.34 nvme8c8n1 14831.40 3707.85 0.00 0.00 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 85.84 nvme8n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 85.04 nvme9c9n1 14831.20 3707.80 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.30 nvme9n1 14831.60 3707.90 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.40 zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 0.00 0.00 0.00 0.00 0.00 0.00 8.80 0.12 0.00 0.00 0.02 13.55 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 md127 178035.00 44507.40 0.00 0.00 0.10 255.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 18.61 100.00 nvme0c0n1 14835.60 3708.90 1.80 0.01 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.40 nvme0n1 14835.60 3708.90 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.54 83.58 nvme10c10n1 14836.00 3709.00 1.80 0.01 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.02 nvme10n1 14835.60 3708.90 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.32 nvme11c11n1 14835.80 3708.95 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 83.52 nvme11n1 14836.00 3709.00 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 82.90 nvme12c12n1 14836.00 3709.00 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 83.74 nvme12n1 14835.60 3708.90 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 83.08 nvme1c1n1 14835.60 3708.90 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 83.60 nvme1n1 14835.80 3708.95 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 82.90 nvme2c2n1 14836.00 3709.00 0.00 0.00 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 83.76 nvme2n1 14836.00 3709.00 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 83.10 nvme3c3n1 14836.00 3709.00 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.00 nvme3n1 14835.80 3708.95 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.44 nvme4c4n1 14835.60 3708.90 0.00 0.00 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.57 85.04 nvme4n1 14835.60 3708.90 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.54 84.20 nvme5c5n1 14835.60 3708.90 1.80 0.01 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.02 nvme5n1 14836.00 3709.00 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.38 nvme6c6n1 0.00 0.00 0.00 0.00 0.00 0.00 8.80 0.12 0.00 0.00 0.07 13.55 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 8.80 0.12 0.00 0.00 0.02 13.55 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 nvme7c7n1 14835.80 3708.95 0.00 0.00 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 85.40 nvme7n1 14835.80 3708.95 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.72 nvme8c8n1 14835.60 3708.90 0.00 0.00 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.24 nvme8n1 14836.00 3709.00 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.56 nvme9c9n1 14836.00 3709.00 0.00 0.00 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.26 nvme9n1 14835.60 3708.90 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.30 zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 0.00 0.00 0.00 0.00 0.00 0.00 2.40 0.02 0.00 0.00 0.00 6.58 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md127 177958.00 44488.15 0.00 0.00 0.10 255.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 18.56 100.00 nvme0c0n1 14829.60 3707.40 1.80 0.01 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.04 nvme0n1 14829.60 3707.40 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 83.28 nvme10c10n1 14829.40 3707.35 1.80 0.01 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.16 nvme10n1 14829.60 3707.40 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 83.24 nvme11c11n1 14829.40 3707.35 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 83.76 nvme11n1 14829.40 3707.35 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 82.86 nvme12c12n1 14829.20 3707.30 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.42 nvme12n1 14829.60 3707.40 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.56 nvme1c1n1 14829.60 3707.40 0.00 0.00 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.82 nvme1n1 14829.40 3707.35 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 84.04 nvme2c2n1 14829.20 3707.30 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 83.26 nvme2n1 14829.20 3707.30 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.51 82.38 nvme3c3n1 14829.20 3707.30 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.56 nvme3n1 14829.20 3707.30 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.60 nvme4c4n1 14829.60 3707.40 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.55 84.36 nvme4n1 14829.60 3707.40 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.52 83.54 nvme5c5n1 14829.60 3707.40 1.80 0.01 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.10 nvme5n1 14829.20 3707.30 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.54 83.36 nvme6c6n1 0.00 0.00 0.00 0.00 0.00 0.00 2.40 0.02 0.00 0.00 0.08 6.58 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme6n1 0.00 0.00 0.00 0.00 0.00 0.00 2.40 0.02 0.00 0.00 0.00 6.58 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme7c7n1 14829.40 3707.35 0.00 0.00 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.10 nvme7n1 14829.60 3707.40 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.18 nvme8c8n1 14829.60 3707.40 0.00 0.00 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.24 nvme8n1 14829.20 3707.30 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.52 nvme9c9n1 14829.20 3707.30 0.00 0.00 0.11 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.56 84.50 nvme9n1 14829.60 3707.40 0.00 0.00 0.10 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.53 83.56 zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 2025-05-07 12:26 ` Anton Gavriliuk @ 2025-05-07 21:59 ` Dave Chinner 0 siblings, 0 replies; 11+ messages in thread From: Dave Chinner @ 2025-05-07 21:59 UTC (permalink / raw) To: Anton Gavriliuk; +Cc: Laurence Oberman, linux-nvme, linux-xfs, linux-block On Wed, May 07, 2025 at 03:26:08PM +0300, Anton Gavriliuk wrote: > > `iostat -dxm 5` output during the fio run on both kernels will give us some indication of the differences in IO patterns, queue depths, etc. > > iostat files attached. Yeah, that definitely looks like MD is the bottleneck. In both traces the NVMe drives are completing read IOs in about 110-120us. In fedora 42, the nvme drives are not at 100% utilisation so the md device is not feeding them fast enough. That can also be seen in that the rocky 9.5 kernel with a nvme device queue depth of about 4 IOs, whilst it is only 1.5 for the fedora 42 kernel. Given that nobody from the block/MD side of things has responded with any ideas yet, you might just have to bisect it to find out where things went wrong... -Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2025-05-23 9:39 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAAiJnjoo0--yp47UKZhbu8sNSZN6DZ-QzmZBMmtr1oC=fOOgAQ@mail.gmail.com>
[not found] ` <aBaVsli2AKbIa4We@dread.disaster.area>
[not found] ` <CAAiJnjor+=Zn62n09f-aJw2amX2wxQOb-2TB3rea9wDCU7ONoA@mail.gmail.com>
2025-05-04 21:50 ` Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5 Dave Chinner
2025-05-05 12:29 ` Laurence Oberman
2025-05-05 13:21 ` Laurence Oberman
2025-05-05 17:39 ` Laurence Oberman
2025-05-22 15:07 ` Laurence Oberman
2025-05-23 9:39 ` Anton Gavriliuk
2025-05-05 22:56 ` Dave Chinner
2025-05-06 11:03 ` Anton Gavriliuk
2025-05-06 21:46 ` Dave Chinner
2025-05-07 12:26 ` Anton Gavriliuk
2025-05-07 21:59 ` Dave Chinner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).