Re: Support for polled IO completions for md-raid devices

["Yu Kuai" <yukuai@fnnas.com>]

Hi,

在 2025/12/1 16:38, Anton Gavriliuk 写道:
> Hello
>
> It is not surprising that polled IO completions significantly reduce
> latency and increase IOPS.  Here is an example with a single CM7-V
> PCIe 5.0 NVMe SSD.
>
> Interrupts,
>
> [root@memverge4 ~]# fio --name=local_test --ioengine=io_uring
> --rw=randread --bs=4K --numjobs=1 --direct=1 --filename=/dev/nvme1n1
> --iodepth=64 --fixedbufs=1 --time_based=1 --runtime=20
> local_test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> (T) 4096B-4096B, ioengine=io_uring, iodepth=64
> fio-3.41-49-gde3d
> Starting 1 process
> Jobs: 1 (f=1): [r(1)][100.0%][r=2936MiB/s][r=752k IOPS][eta 00m:00s]
> local_test: (groupid=0, jobs=1): err= 0: pid=96203: Mon Dec  1 10:29:47 2025
>    read: IOPS=746k, BW=2915MiB/s (3056MB/s)(56.9GiB/20001msec)
>      slat (nsec): min=510, max=931130, avg=993.69, stdev=462.86
>      clat (usec): min=44, max=1056, avg=84.48, stdev=12.57
>       lat (usec): min=45, max=1057, avg=85.48, stdev=12.60
>
> Polling,
>
> [root@memverge4 ~]# fio --name=local_test --ioengine=io_uring
> --rw=randread --bs=4K --numjobs=1 --direct=1 --filename=/dev/nvme1n1
> --iodepth=64 --fixedbufs=1 --hipri=1 --time_based=1 --runtime=20
> local_test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> (T) 4096B-4096B, ioengine=io_uring, iodepth=64
> fio-3.41-49-gde3d
> Starting 1 process
> Jobs: 1 (f=1): [r(1)][100.0%][r=4618MiB/s][r=1182k IOPS][eta 00m:00s]
> local_test: (groupid=0, jobs=1): err= 0: pid=96543: Mon Dec  1 10:32:41 2025
>    read: IOPS=1174k, BW=4588MiB/s (4810MB/s)(89.6GiB/20001msec)
>      slat (nsec): min=510, max=909786, avg=663.38, stdev=410.86
>      clat (usec): min=34, max=1024, avg=53.69, stdev= 6.33
>       lat (usec): min=35, max=1024, avg=54.36, stdev= 6.34
>
> Dut it doesn't work for md-raid device,
>
> [root@memverge4 ~]# cat /proc/mdstat
> Personalities : [raid0]
> md127 : active raid0 nvme6n1[5] nvme0n1[3] nvme5n1[0] nvme2n1[1]
> nvme3n1[4] nvme1n1[2]
>        18752907264 blocks super 1.2 512k chunks
>
> unused devices: <none>
> [root@memverge4 ~]#
> [root@memverge4 ~]# fio --name=local_test --ioengine=io_uring
> --rw=randread --bs=4K --numjobs=1 --direct=1 --filename=/dev/md127
> --iodepth=64 --fixedbufs=1 --hipri=1 --time_based=1 --runtime=20
> local_test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B,
> (T) 4096B-4096B, ioengine=io_uring, iodepth=64
> fio-3.41-49-gde3d
> Starting 1 process
> fio: file /dev/md127 exceeds 32-bit tausworthe random generator.
> fio: Switching to tausworthe64. Use the random_generator= option to
> get rid of this warning.
> fio: io_u error on file /dev/md127: Operation not supported: read
> offset=18761783697408, buflen=4096
> fio: io_u error on file /dev/md127: Operation not supported: read
> offset=17577830526976, buflen=4096
> fio: io_u error on file /dev/md127: Operation not supported: read
> offset=16065491521536, buflen=4096
> fio: io_u error on file /dev/md127: Operation not supported: read
> offset=15054980657152, buflen=4096
> ...............................................
>
> I'm not a developer or programmer.
>
> May I ask as a feature request to add support for polling IO
> completions for md-raid devices ?

This will help improve performance if underlying disks are nvme, I
can add this to my todo list(with low priority), but I can't guarantee
when will this be done. :( Meanwhile, feel free to take a look if anyone
is interested.

>
> Anton
>
-- 
Thanks,
Kuai