xor_gen() md-raid6 performance improvement

xor_gen() md-raid6 performance improvement

[Anton Gavriliuk]

Hello

I would like to share xor_gen performance improvement for md-raid6 write.
Box - AMD Zen5 9455, 6 x PCIe 5.0 NVMe SSD CM7-V in raid6 (4+2),
Fedora Server 45.


[root@memverge3 ~]# uname -r
7.2.15-300.fc45.x86_64
[root@memverge3 ~]# dmesg | grep -i xor
[    0.471277] xor: automatically using best checksumming function   avx
[root@memverge3 ~]# fio --name=test --rw=write --bs=2048k
--filename=/dev/md111 --direct=1 --numjobs=1 --iodepth=64 --exitall
--group_reporting --ioengine=io_uring --runtime=600 --time_based=1
test: (g=0): rw=write, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB,
(T) 2048KiB-2048KiB, ioengine=io_uring, iodepth=64
fio-3.43-31-gbfffd
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=4712MiB/s][w=2356 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2351: Fri Nov  6 15:49:05 2026
  write: IOPS=2323, BW=4647MiB/s (4873MB/s)(2723GiB/600026msec)
    slat (usec): min=5, max=3469, avg=26.01, stdev=17.20
    clat (usec): min=2953, max=69868, avg=27516.08, stdev=3473.78
     lat (usec): min=2963, max=69930, avg=27542.09, stdev=3474.01


[root@memverge3 ~]# uname -r
7.4.0-rc1
[root@memverge3 ~]# dmesg | grep -i xor
[    0.553181] xor: automatically using best checksumming function   avx512
[root@memverge3 ~]# fio --name=test --rw=write --bs=2048k
--filename=/dev/md111 --direct=1 --numjobs=1 --iodepth=64 --exitall
--group_reporting --ioengine=io_uring --runtime=600 --time_based=1
test: (g=0): rw=write, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB,
(T) 2048KiB-2048KiB, ioengine=io_uring, iodepth=64
fio-3.43-31-gbfffd
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=7438MiB/s][w=3719 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=4837: Fri Nov  6 17:02:31 2026
  write: IOPS=3791, BW=7582MiB/s (7951MB/s)(4443GiB/600013msec)
    slat (usec): min=8, max=3073, avg=35.39, stdev=14.87
    clat (usec): min=2334, max=44420, avg=16845.30, stdev=3951.02
     lat (usec): min=2375, max=44455, avg=16880.69, stdev=3955.63


Nice results, but further improvements required to fully unlock PCI
5.0 drives potential.

Re: xor_gen() md-raid6 performance improvement

[Dragan Milivojević]

> [root@memverge3 ~]# fio --name=test --rw=write --bs=2048k
> --filename=/dev/md111 --direct=1 --numjobs=1 --iodepth=64 --exitall
> --group_reporting --ioengine=io_uring --runtime=600 --time_based=1
> test: (g=0): rw=write, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB,
> (T) 2048KiB-2048KiB, ioengine=io_uring, iodepth=64
> fio-3.43-31-gbfffd
> Starting 1 process
> Jobs: 1 (f=1): [W(1)][100.0%][w=7438MiB/s][w=3719 IOPS][eta 00m:00s]
> test: (groupid=0, jobs=1): err= 0: pid=4837: Fri Nov  6 17:02:31 2026
>   write: IOPS=3791, BW=7582MiB/s (7951MB/s)(4443GiB/600013msec)
>     slat (usec): min=8, max=3073, avg=35.39, stdev=14.87
>     clat (usec): min=2334, max=44420, avg=16845.30, stdev=3951.02
>      lat (usec): min=2375, max=44455, avg=16880.69, stdev=3955.63
>
>
Nice improvement
Run the same test using a single drive, in ideal case the performance should
be 4x of a single drive.

Re: xor_gen() md-raid6 performance improvement

[Miroslav Cimerman]

Hi Anton,

thanks for sharing the numbers. I recently submitted a raid6 patch that
makes read-modify-writes faster, at least on the Intel machines I have.
If you have time, could you please test it on your Zen 5 box?

  https://lore.kernel.org/all/20261109030805.25709-1-mc@raidlab.org/

It changes xor_syndrome(), which md uses for small read-modify-writes.
Your 2 MiB sequential test does not use it, as full-stripe writes
compute P and Q with gen_syndrome(). Random 4 KiB writes do use it,
i.e. --rw=randwrite --bs=4k:

  echo 2 > /sys/block/md111/md/rmw_level
  fio --name=test --rw=randwrite --bs=4k --filename=/dev/md111 \
      --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting \
      --ioengine=io_uring --size=100g --runtime=60 --time_based=1

rmw_level=2 is needed on a 4+2 array: a read-modify-write and
a reconstruct-write both need 3 reads there, and by default md picks
reconstruct-write on a tie (1 is the default).

One job at QD64 should be enough. md's single raid6 thread should be
the limit. If the md111_raid6 thread is not near 100% in "top -H", try
--numjobs=4 --iodepth=32.

If md111 has the new lockless bitmap and was created without
--assume-clean, md uses reconstruct-writes in regions not written since
the array was created. Adding --size=100g to fio keeps the writes in
the start of the array, which your sequential tests have probably
already written.

Also please make sure that no resync is in progress.

My numbers were measured without a write-intent bitmap, so the
difference may be smaller with one.

Please make sure that both runs, with and without the patch, use
avx512x4:

  dmesg | grep 'raid6: using algorithm' | tail -1

The kernel picks the implementation by gen_syndrome() speed when
raid6_pq is loaded, and the patch only changes the x4 versions (x1 and
x2 already use normal stores). If another one is picked, reloading
raid6_pq with md111 stopped runs the benchmark again, or I can send a
two-line test-only change that forces avx512x4.

Either way, sharing your numbers would help, and a reply with Tested-by
to the patch if it works for you would be very appreciated.

Thanks,
Miroslav

Re: xor_gen() md-raid6 performance improvement

[Anton Gavriliuk]

Ok, пока ждем, но одна новая ошибка при текущем прогоне уже есть.

Теперь, внимание, что я хочу сделать глобально - полностью замкнутый и
автономный цикл, а именно -

1.  Локальной LLM и агентом выполняется кросс-компиляция.  Если
завершилось успешно, то останавливаемся.
Если за 8 попыток кросс-компиляция успешно не завершилась, то
переходим к пункту 2.
2.  На оосновании новых ошибк добавленных
/mnt/llm_experience/experience_db.json, проводим полный цикл
дообучения модели, создания нового gguf файла и квантования до уровня
Q8_0.
3.  Повторяем запуск кросс-комиляции на доученной модели.  И снова,
если успех, то останавливаемся.  Если нет, то снова повторяем пункты 2
и 3 аж пока кросс-компиляция не завершится успешно.

Хочу что бы это выполнялось автоматически одним скриптом.  То есть, по
сути нужно последние успешные скрипты обьеденить в один мега-скрипт.








Hi Miroslav

I'm very interested in a high performance md-raid subsystem, to be
aligned with current PCIe 5.0 and coming 6.0 NVMe SSDs.
So I'm ready to assist you as much as required.

There is the single socket box - HPE DL345g11, 6 x PCIe 5.0 NVMe SSDs
(CM7-V 3.2 TB), 12 x 16 GB DDR5 DRAM 6400 MT/s, AMD Zen5 9455 with
re-installed (since my previous email) Fedora Server 45.

Yesterday NVMe SSDs drives were used for another activity, but now I'm
ready to switch back to test md-raid performance.

[root@family ~]# lsblk
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
zram0           251:0    0     8G  0 disk [SWAP]
nvme6n1         259:2    0 447.1G  0 disk
├─nvme6n1p1     259:4    0   600M  0 part /boot/efi
├─nvme6n1p2     259:5    0     2G  0 part /boot
└─nvme6n1p3     259:6    0 444.5G  0 part
  └─fedora-root 252:0    0 444.5G  0 lvm  /
nvme3n1         259:3    0   2.9T  0 disk
nvme1n1         259:8    0   2.9T  0 disk
nvme5n1         259:10   0   2.9T  0 disk
nvme4n1         259:12   0   2.9T  0 disk
nvme2n1         259:14   0   2.9T  0 disk
nvme0n1         259:16   0   2.9T  0 disk
[root@family ~]#
[root@family ~]# dmesg|grep -i raid6
[    0.492578] raid6: using avx512x2 recovery algorithm
[    0.605293] raid6: skipped pq benchmark and selected avx512x4
[root@family ~]#

Ok, I will create md-raid6 (4+2) with --bitmap=none and without
lockless.  But which chunk size should I select, default 512K ?

Which kernel should I select ?,

[root@family ~]# uname -r
7.3.3-300.fc45.x86_64

Or 7.4-rc2 ?

I'm not a developer, please provide detailed steps on how to apply the
patch you mentioned.

Anton

вт, 10 нояб. 2026 г. в 15:38, Miroslav Cimerman <mc@raidlab.org>:
>
> Hi Anton,
>
> thanks for sharing the numbers. I recently submitted a raid6 patch that
> makes read-modify-writes faster, at least on the Intel machines I have.
> If you have time, could you please test it on your Zen 5 box?
>
>   https://lore.kernel.org/all/20261109030805.25709-1-mc@raidlab.org/
>
> It changes xor_syndrome(), which md uses for small read-modify-writes.
> Your 2 MiB sequential test does not use it, as full-stripe writes
> compute P and Q with gen_syndrome(). Random 4 KiB writes do use it,
> i.e. --rw=randwrite --bs=4k:
>
>   echo 2 > /sys/block/md111/md/rmw_level
>   fio --name=test --rw=randwrite --bs=4k --filename=/dev/md111 \
>       --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting \
>       --ioengine=io_uring --size=100g --runtime=60 --time_based=1
>
> rmw_level=2 is needed on a 4+2 array: a read-modify-write and
> a reconstruct-write both need 3 reads there, and by default md picks
> reconstruct-write on a tie (1 is the default).
>
> One job at QD64 should be enough. md's single raid6 thread should be
> the limit. If the md111_raid6 thread is not near 100% in "top -H", try
> --numjobs=4 --iodepth=32.
>
> If md111 has the new lockless bitmap and was created without
> --assume-clean, md uses reconstruct-writes in regions not written since
> the array was created. Adding --size=100g to fio keeps the writes in
> the start of the array, which your sequential tests have probably
> already written.
>
> Also please make sure that no resync is in progress.
>
> My numbers were measured without a write-intent bitmap, so the
> difference may be smaller with one.
>
> Please make sure that both runs, with and without the patch, use
> avx512x4:
>
>   dmesg | grep 'raid6: using algorithm' | tail -1
>
> The kernel picks the implementation by gen_syndrome() speed when
> raid6_pq is loaded, and the patch only changes the x4 versions (x1 and
> x2 already use normal stores). If another one is picked, reloading
> raid6_pq with md111 stopped runs the benchmark again, or I can send a
> two-line test-only change that forces avx512x4.
>
> Either way, sharing your numbers would help, and a reply with Tested-by
> to the patch if it works for you would be very appreciated.
>
> Thanks,
> Miroslav

Re: xor_gen() md-raid6 performance improvement

[Anton Gavriliuk]

Hi Miroslav

I'm very interested in a high performance md-raid subsystem, to be
aligned with current PCIe 5.0 and coming 6.0 NVMe SSDs.
So I'm ready to assist you as much as required.

There is the single socket box - HPE DL345g11, 6 x PCIe 5.0 NVMe SSDs
(CM7-V 3.2 TB), 12 x 16 GB DDR5 DRAM 6400 MT/s, AMD Zen5 9455 with
re-installed (since my previous email) Fedora Server 45.

Yesterday NVMe SSDs drives were used for another activity, but now I'm
ready to switch back to test md-raid performance.

[root@family ~]# lsblk
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
zram0           251:0    0     8G  0 disk [SWAP]
nvme6n1         259:2    0 447.1G  0 disk
├─nvme6n1p1     259:4    0   600M  0 part /boot/efi
├─nvme6n1p2     259:5    0     2G  0 part /boot
└─nvme6n1p3     259:6    0 444.5G  0 part
  └─fedora-root 252:0    0 444.5G  0 lvm  /
nvme3n1         259:3    0   2.9T  0 disk
nvme1n1         259:8    0   2.9T  0 disk
nvme5n1         259:10   0   2.9T  0 disk
nvme4n1         259:12   0   2.9T  0 disk
nvme2n1         259:14   0   2.9T  0 disk
nvme0n1         259:16   0   2.9T  0 disk
[root@family ~]#
[root@family ~]# dmesg|grep -i raid6
[    0.492578] raid6: using avx512x2 recovery algorithm
[    0.605293] raid6: skipped pq benchmark and selected avx512x4
[root@family ~]#

Ok, I will create md-raid6 (4+2) with --bitmap=none and without
lockless.  But which chunk size should I select, default 512K ?

Which kernel should I select ?,

[root@family ~]# uname -r
7.3.3-300.fc45.x86_64

Or 7.4-rc2 ?

I'm not a developer, please provide detailed steps on how to apply the
patch you mentioned.

Anton

вт, 10 нояб. 2026 г. в 15:38, Miroslav Cimerman <mc@raidlab.org>:
>
> Hi Anton,
>
> thanks for sharing the numbers. I recently submitted a raid6 patch that
> makes read-modify-writes faster, at least on the Intel machines I have.
> If you have time, could you please test it on your Zen 5 box?
>
>   https://lore.kernel.org/all/20261109030805.25709-1-mc@raidlab.org/
>
> It changes xor_syndrome(), which md uses for small read-modify-writes.
> Your 2 MiB sequential test does not use it, as full-stripe writes
> compute P and Q with gen_syndrome(). Random 4 KiB writes do use it,
> i.e. --rw=randwrite --bs=4k:
>
>   echo 2 > /sys/block/md111/md/rmw_level
>   fio --name=test --rw=randwrite --bs=4k --filename=/dev/md111 \
>       --direct=1 --numjobs=1 --iodepth=64 --exitall --group_reporting \
>       --ioengine=io_uring --size=100g --runtime=60 --time_based=1
>
> rmw_level=2 is needed on a 4+2 array: a read-modify-write and
> a reconstruct-write both need 3 reads there, and by default md picks
> reconstruct-write on a tie (1 is the default).
>
> One job at QD64 should be enough. md's single raid6 thread should be
> the limit. If the md111_raid6 thread is not near 100% in "top -H", try
> --numjobs=4 --iodepth=32.
>
> If md111 has the new lockless bitmap and was created without
> --assume-clean, md uses reconstruct-writes in regions not written since
> the array was created. Adding --size=100g to fio keeps the writes in
> the start of the array, which your sequential tests have probably
> already written.
>
> Also please make sure that no resync is in progress.
>
> My numbers were measured without a write-intent bitmap, so the
> difference may be smaller with one.
>
> Please make sure that both runs, with and without the patch, use
> avx512x4:
>
>   dmesg | grep 'raid6: using algorithm' | tail -1
>
> The kernel picks the implementation by gen_syndrome() speed when
> raid6_pq is loaded, and the patch only changes the x4 versions (x1 and
> x2 already use normal stores). If another one is picked, reloading
> raid6_pq with md111 stopped runs the benchmark again, or I can send a
> two-line test-only change that forces avx512x4.
>
> Either way, sharing your numbers would help, and a reply with Tested-by
> to the patch if it works for you would be very appreciated.
>
> Thanks,
> Miroslav

Re: xor_gen() md-raid6 performance improvement

[Miroslav Cimerman]

Hi Anton,

Anton Gavriliuk wrote:

> I'm very interested in a high performance md-raid subsystem, to be
> aligned with current PCIe 5.0 and coming 6.0 NVMe SSDs.
> So I'm ready to assist you as much as required.

Thank you, your setup is great.

> But which chunk size should I select, default 512K ?

The default 512K chunk is fine.

> Which kernel should I select ?,
> [...]
> Or 7.4-rc2 ?

7.4-rc2 for both runs, once as it is and once with the patch

I sent you detailed instructions off-list, as they are quite long.

Thanks,
Miroslav