RAID 5, 10 modern post 2020 drives, slow speeds

RAID 5, 10 modern post 2020 drives, slow speeds

[David Hajes]

Hi guys,

I have issues with RAID5 running on post-2020 14TB drives.

I am getting max writting speeds of 220MBs.

If I understood properly docs, RAID5 with three drives should have 2x write/read speeds.

Single drive tests running 190-220MBs, no issues in SMART reports.

RAID5 with 4 drives...same speed.

RAID10 should have 2x write, 4x read...still running 220MBs max as RAID5

I have played with chunk size...default 512k-2MBs...no difference

"Read-ahead" set for md0 virtual disk

NCQ disabled - set 1 for all physical drives

I have basically tried every suggestion on famous ArchWiki.

I have tried SAS2 HBA...same results.

Initial resync drops to 130MBs

Debian 12 running on SuperMicro 10 series MB. 32GB RAM, cpu max 30% load during tests/real-life copy.

Is it possible this weird issue is linked to HDD timeout described there >>> https://archive.kernel.org/oldwiki/raid.wiki.kernel.org/index.php/Timeout_Mismatch.html

Any suggestion, please?

Array should be doing 260-560MBs with ease

Regards

Hajes

Re: RAID 5, 10 modern post 2020 drives, slow speeds

[Roman Mamedov]

Hello,

On Fri, 07 Mar 2025 18:36:13 +0000
David Hajes <d.hajes29a@pm.me> wrote:

> I have issues with RAID5 running on post-2020 14TB drives.
> 
> I am getting max writting speeds of 220MBs.

What about read speeds, do you get much more, or clamped in the same ballpark?

To not wait for a full resync just to check this (or various other settings),
you can create the array with --assume-clean.

In case reads are also limited to the same value, I'd suspect PCIe bandwidth
issues, such as the HBA getting choked by 2.5 GT/s x1 for whatever reason.
Check the bandwidth in "lspci -vvv".

> I have played with chunk size...default 512k-2MBs...no difference
> 
> "Read-ahead" set for md0 virtual disk
> 
> NCQ disabled - set 1 for all physical drives
> 
> I have basically tried every suggestion on famous ArchWiki.

Do you use the Write-Intent bitmap, and what is its chunk size?
Try without one briefly, to see if this was the issue.
For production use, increase the bitmap chunk size and see if that helps.

> Initial resync drops to 130MBs

Are your drives SMR or CMR? For SMR drives it is common to briefly write
quickly and then slow down as they need to do their housekeeping during the
same time as new writes. SMR are not recommended for RAID.

> Is it possible this weird issue is linked to HDD timeout described there >>> https://archive.kernel.org/oldwiki/raid.wiki.kernel.org/index.php/Timeout_Mismatch.html

No.

-- 
With respect,
Roman

Re: RAID 5, 10 modern post 2020 drives, slow speeds

[David Hajes]

Hi Roman,

Thanks for reply.

All drives tourist SATA3 CMR

Link speed for LSI SAS2308 8GTs/8x

Intel C224 chipset SATA controller

Write speed is always same, no matter RAI level, chipset C224 or SAS connection.

Read test for RAID10 is 414MBs

I was hoping for higher writing speeds. What is interesting RAID5 in default setting does 220MBs while RAID10 struggles at 170MBs.

There is something horribly wrong :o)

So bitmap seems to be on. Mdstat says "bitmap: 0/204 pages 65M chunk

--bitmap=none

Write speed 170MBs on RAID10 with chunk 1MB

Bitmap internal with chunk 128M write speed 170


-------- Original Message --------
On 07/03/2025 19:47, Roman Mamedov <rm@romanrm.net> wrote:

>  Hello,
>  
>  On Fri, 07 Mar 2025 18:36:13 +0000
>  David Hajes <d.hajes29a@pm.me> wrote:
>  
>  > I have issues with RAID5 running on post-2020 14TB drives.
>  >
>  > I am getting max writting speeds of 220MBs.
>  
>  What about read speeds, do you get much more, or clamped in the same ballpark?
>  
>  To not wait for a full resync just to check this (or various other settings),
>  you can create the array with --assume-clean.
>  
>  In case reads are also limited to the same value, I'd suspect PCIe bandwidth
>  issues, such as the HBA getting choked by 2.5 GT/s x1 for whatever reason.
>  Check the bandwidth in "lspci -vvv".
>  
>  > I have played with chunk size...default 512k-2MBs...no difference
>  >
>  > "Read-ahead" set for md0 virtual disk
>  >
>  > NCQ disabled - set 1 for all physical drives
>  >
>  > I have basically tried every suggestion on famous ArchWiki.
>  
>  Do you use the Write-Intent bitmap, and what is its chunk size?
>  Try without one briefly, to see if this was the issue.
>  For production use, increase the bitmap chunk size and see if that helps.
>  
>  > Initial resync drops to 130MBs
>  
>  Are your drives SMR or CMR? For SMR drives it is common to briefly write
>  quickly and then slow down as they need to do their housekeeping during the
>  same time as new writes. SMR are not recommended for RAID.
>  
>  > Is it possible this weird issue is linked to HDD timeout described there >>> https://archive.kernel.org/oldwiki/raid.wiki.kernel.org/index.php/Timeout_Mismatch.html
>  
>  No.
>  
>  --
>  With respect,
>  Roman
>

Re: RAID 5, 10 modern post 2020 drives, slow speeds

[Roger Heflin]

I put an external bitmap on a raid1 SSD and that seemed to speed up my
writes.  I am not sure if external bitmaps will continue to be
supported as I have seen notes that I don't exactly understand for
external bitmaps, and I have to reapply the external bitmap on each
reboot for my arrays which has some data loss risks in a crash case
with a dirty bitmap.

This is the command I used to set it up.
mdadm --grow --force --bitmap=/mdraid-bitmaps/md15-bitmap.img /dev/md15

On Fri, Mar 7, 2025 at 1:25 PM David Hajes <d.hajes29a@pm.me> wrote:
>
> Hi Roman,
>
> Thanks for reply.
>
> All drives tourist SATA3 CMR
>
> Link speed for LSI SAS2308 8GTs/8x
>
> Intel C224 chipset SATA controller
>
> Write speed is always same, no matter RAI level, chipset C224 or SAS connection.
>
> Read test for RAID10 is 414MBs
>
> I was hoping for higher writing speeds. What is interesting RAID5 in default setting does 220MBs while RAID10 struggles at 170MBs.
>
> There is something horribly wrong :o)
>
> So bitmap seems to be on. Mdstat says "bitmap: 0/204 pages 65M chunk
>
> --bitmap=none
>
> Write speed 170MBs on RAID10 with chunk 1MB
>
> Bitmap internal with chunk 128M write speed 170
>
>
> -------- Original Message --------
> On 07/03/2025 19:47, Roman Mamedov <rm@romanrm.net> wrote:
>
> >  Hello,
> >
> >  On Fri, 07 Mar 2025 18:36:13 +0000
> >  David Hajes <d.hajes29a@pm.me> wrote:
> >
> >  > I have issues with RAID5 running on post-2020 14TB drives.
> >  >
> >  > I am getting max writting speeds of 220MBs.
> >
> >  What about read speeds, do you get much more, or clamped in the same ballpark?
> >
> >  To not wait for a full resync just to check this (or various other settings),
> >  you can create the array with --assume-clean.
> >
> >  In case reads are also limited to the same value, I'd suspect PCIe bandwidth
> >  issues, such as the HBA getting choked by 2.5 GT/s x1 for whatever reason.
> >  Check the bandwidth in "lspci -vvv".
> >
> >  > I have played with chunk size...default 512k-2MBs...no difference
> >  >
> >  > "Read-ahead" set for md0 virtual disk
> >  >
> >  > NCQ disabled - set 1 for all physical drives
> >  >
> >  > I have basically tried every suggestion on famous ArchWiki.
> >
> >  Do you use the Write-Intent bitmap, and what is its chunk size?
> >  Try without one briefly, to see if this was the issue.
> >  For production use, increase the bitmap chunk size and see if that helps.
> >
> >  > Initial resync drops to 130MBs
> >
> >  Are your drives SMR or CMR? For SMR drives it is common to briefly write
> >  quickly and then slow down as they need to do their housekeeping during the
> >  same time as new writes. SMR are not recommended for RAID.
> >
> >  > Is it possible this weird issue is linked to HDD timeout described there >>> https://archive.kernel.org/oldwiki/raid.wiki.kernel.org/index.php/Timeout_Mismatch.html
> >
> >  No.
> >
> >  --
> >  With respect,
> >  Roman
> >
>

Re: RAID 5, 10 modern post 2020 drives, slow speeds

[Roman Mamedov]

On Fri, 7 Mar 2025 14:42:24 -0600
Roger Heflin <rogerheflin@gmail.com> wrote:

> I put an external bitmap on a raid1 SSD and that seemed to speed up my
> writes.  I am not sure if external bitmaps will continue to be
> supported as I have seen notes that I don't exactly understand for
> external bitmaps, and I have to reapply the external bitmap on each
> reboot for my arrays which has some data loss risks in a crash case
> with a dirty bitmap.
> 
> This is the command I used to set it up.
> mdadm --grow --force --bitmap=/mdraid-bitmaps/md15-bitmap.img /dev/md15

In this case the result cited seems to have shown the bitmap is not the issue.

I remember seeing patches or talks to remove external bitmap support, too.

In my experience the internal bitmap with a large enough chunk size does not
slow down the write speed that much. Try a chunk size of 256M. Not sure how
high it's worth going before the benefits diminish.

-- 
With respect,
Roman

Re: RAID 5, 10 modern post 2020 drives, slow speeds

[David Hajes]

In case someone wonders in future why SW RAID5 or 10 is slow.

Unless, two or more process do not write in parallel - ARRAY SPEED WILL ALWAYS BE SINGLE DRIVE LIMIT.

Basically, any single user operations of storing data on the array will became as writing to single drive.

In case of modern SATA HDDs, it would be 120-220MBs

Only the HW RAID controllers are allegedly capable to write on more than one drive parallel thus achiving the logical/envisioned/intuitive speed.

based on theory where at least two chunks are written at once to the two different drives thus doubling the writing speed as confussingly written in all RAID wikis.


-------- Original Message --------
On 07/03/2025 21:47, Roman Mamedov <rm@romanrm.net> wrote:

>  On Fri, 7 Mar 2025 14:42:24 -0600
>  Roger Heflin <rogerheflin@gmail.com> wrote:
>  
>  > I put an external bitmap on a raid1 SSD and that seemed to speed up my
>  > writes.  I am not sure if external bitmaps will continue to be
>  > supported as I have seen notes that I don't exactly understand for
>  > external bitmaps, and I have to reapply the external bitmap on each
>  > reboot for my arrays which has some data loss risks in a crash case
>  > with a dirty bitmap.
>  >
>  > This is the command I used to set it up.
>  > mdadm --grow --force --bitmap=/mdraid-bitmaps/md15-bitmap.img /dev/md15
>  
>  In this case the result cited seems to have shown the bitmap is not the issue.
>  
>  I remember seeing patches or talks to remove external bitmap support, too.
>  
>  In my experience the internal bitmap with a large enough chunk size does not
>  slow down the write speed that much. Try a chunk size of 256M. Not sure how
>  high it's worth going before the benefits diminish.
>  
>  --
>  With respect,
>  Roman
>

Re: RAID 5, 10 modern post 2020 drives, slow speeds

[David Hajes]

Update on issue. I came across the "mismatch_cnt" stat after init resync or just common check/scrub.

mismatch_cnt suppose to be 0. I had counter reaching millions. I haven't found definitive answer whether "mismatch_cnt" bad, and should be ignored.

Allegedly high mismatch_cnt count suggest HW  or SW issues. I run SMART, no system corruption.

I have tried to run "repair". Repair 400-500MBs, and mismatch_cnt is now 0.

Initial resync started at 500MBs, 20 min later dropped to 200 MBs. 3 days later speed was at 120MBs.

I will try some real tests to see if array is still fast in real life as well.

Hajes


-------- Original Message --------
On 08/03/2025 18:59, David Hajes <d.hajes29a@pm.me> wrote:

>  
>  In case someone wonders in future why SW RAID5 or 10 is slow.
>  
>  Unless, two or more process do not write in parallel - ARRAY SPEED WILL ALWAYS BE SINGLE DRIVE LIMIT.
>  
>  Basically, any single user operations of storing data on the array will became as writing to single drive.
>  
>  In case of modern SATA HDDs, it would be 120-220MBs
>  
>  Only the HW RAID controllers are allegedly capable to write on more than one drive parallel thus achiving the logical/envisioned/intuitive speed.
>  
>  based on theory where at least two chunks are written at once to the two different drives thus doubling the writing speed as confussingly written in all RAID wikis.
>  
>  
>  -------- Original Message --------
>  On 07/03/2025 21:47, Roman Mamedov <rm@romanrm.net> wrote:
>  
>  >  On Fri, 7 Mar 2025 14:42:24 -0600
>  >  Roger Heflin <rogerheflin@gmail.com> wrote:
>  >  
>  >  > I put an external bitmap on a raid1 SSD and that seemed to speed up my
>  >  > writes.  I am not sure if external bitmaps will continue to be
>  >  > supported as I have seen notes that I don't exactly understand for
>  >  > external bitmaps, and I have to reapply the external bitmap on each
>  >  > reboot for my arrays which has some data loss risks in a crash case
>  >  > with a dirty bitmap.
>  >  >
>  >  > This is the command I used to set it up.
>  >  > mdadm --grow --force --bitmap=/mdraid-bitmaps/md15-bitmap.img /dev/md15
>  >  
>  >  In this case the result cited seems to have shown the bitmap is not the issue.
>  >  
>  >  I remember seeing patches or talks to remove external bitmap support, too.
>  >  
>  >  In my experience the internal bitmap with a large enough chunk size does not
>  >  slow down the write speed that much. Try a chunk size of 256M. Not sure how
>  >  high it's worth going before the benefits diminish.
>  >  
>  >  --
>  >  With respect,
>  >  Roman
>  >

Fw: Re: RAID 5, 10 modern post 2020 drives, slow speeds

[David Hajes]

Of course, but not by 150%

4 drives RAID10 suppose to be running theoretically 300-500MBs...not 120MBs.

Most modern drives do 150-250MBs inside to outside tracks


-------- Original Message -------- 
On 13/03/2025 03:54, Dragan Milivojević wrote: 
Speed drops as you approach the end of the disk. 

On Wed, 12 Mar 2025 at 22:10, David Hajes <d.hajes29a@pm.me> wrote: 
Update on issue. I came across the "mismatch_cnt" stat after init resync or just common check/scrub. 

mismatch_cnt suppose to be 0. I had counter reaching millions. I haven't found definitive answer whether "mismatch_cnt" bad, and should be ignored. 

Allegedly high mismatch_cnt count suggest HW  or SW issues. I run SMART, no system corruption. 

I have tried to run "repair". Repair 400-500MBs, and mismatch_cnt is now 0. 

Initial resync started at 500MBs, 20 min later dropped to 200 MBs. 3 days later speed was at 120MBs. 

I will try some real tests to see if array is still fast in real life as well. 

Hajes 


-------- Original Message -------- 
On 08/03/2025 18:59, David Hajes <d.hajes29a@pm.me> wrote: 

>  
>  In case someone wonders in future why SW RAID5 or 10 is slow. 
>  
>  Unless, two or more process do not write in parallel - ARRAY SPEED WILL ALWAYS BE SINGLE DRIVE LIMIT. 
>  
>  Basically, any single user operations of storing data on the array will became as writing to single drive. 
>  
>  In case of modern SATA HDDs, it would be 120-220MBs 
>  
>  Only the HW RAID controllers are allegedly capable to write on more than one drive parallel thus achiving the logical/envisioned/intuitive speed. 
>  
>  based on theory where at least two chunks are written at once to the two different drives thus doubling the writing speed as confussingly written in all RAID wikis. 
>  
>  
>  -------- Original Message -------- 
>  On 07/03/2025 21:47, Roman Mamedov <rm@romanrm.net> wrote: 
>  
>  >  On Fri, 7 Mar 2025 14:42:24 -0600 
>  >  Roger Heflin <rogerheflin@gmail.com> wrote: 
>  >  
>  >  > I put an external bitmap on a raid1 SSD and that seemed to speed up my 
>  >  > writes.  I am not sure if external bitmaps will continue to be 
>  >  > supported as I have seen notes that I don't exactly understand for 
>  >  > external bitmaps, and I have to reapply the external bitmap on each 
>  >  > reboot for my arrays which has some data loss risks in a crash case 
>  >  > with a dirty bitmap. 
>  >  > 
>  >  > This is the command I used to set it up. 
>  >  > mdadm --grow --force --bitmap=/mdraid-bitmaps/md15-bitmap.img /dev/md15 
>  >  
>  >  In this case the result cited seems to have shown the bitmap is not the issue. 
>  >  
>  >  I remember seeing patches or talks to remove external bitmap support, too. 
>  >  
>  >  In my experience the internal bitmap with a large enough chunk size does not 
>  >  slow down the write speed that much. Try a chunk size of 256M. Not sure how 
>  >  high it's worth going before the benefits diminish. 
>  >  
>  >  -- 
>  >  With respect, 
>  >  Roman 
>  >  

Re: RAID 5, 10 modern post 2020 drives, slow speeds

[Xiao Ni]

Hi David

The io performance decreases when sync io happens. It's expected. How
about the performance without sync io? And what's your kernel version?

I read other emails in this thread too. Your sync speed also decreases
in three days. I can't understand this. What's the io environment
there? It's better to describe more about the situation too.

Regards
Xiao

On Sat, Mar 8, 2025 at 2:36 AM David Hajes <d.hajes29a@pm.me> wrote:
>
> Hi guys,
>
> I have issues with RAID5 running on post-2020 14TB drives.
>
> I am getting max writting speeds of 220MBs.
>
> If I understood properly docs, RAID5 with three drives should have 2x write/read speeds.
>
> Single drive tests running 190-220MBs, no issues in SMART reports.
>
> RAID5 with 4 drives...same speed.
>
> RAID10 should have 2x write, 4x read...still running 220MBs max as RAID5
>
> I have played with chunk size...default 512k-2MBs...no difference
>
> "Read-ahead" set for md0 virtual disk
>
> NCQ disabled - set 1 for all physical drives
>
> I have basically tried every suggestion on famous ArchWiki.
>
> I have tried SAS2 HBA...same results.
>
> Initial resync drops to 130MBs
>
> Debian 12 running on SuperMicro 10 series MB. 32GB RAM, cpu max 30% load during tests/real-life copy.
>
> Is it possible this weird issue is linked to HDD timeout described there >>> https://archive.kernel.org/oldwiki/raid.wiki.kernel.org/index.php/Timeout_Mismatch.html
>
> Any suggestion, please?
>
> Array should be doing 260-560MBs with ease
>
> Regards
>
> Hajes
>