RAID6 syncing at 24 MB/s?

RAID6 syncing at 24 MB/s?

[George Spelvin]

I'm putting together a 7x2TB RAID6 for a home media server, and after
encountering a fun kernel bug building the array (old kernel, a new one
fixed it), I fot things working, but now notice that each disk is syncing
at 23.8 MB/s.  hdparm -t reports they can do well over 100 MB/s each,
and the processor, while only an i3, is spending about 20% of one core
in the raid6 task.

Basicall, it was created with:
mdadm -C -l 6 -n 7 -c 128 -b internal /dev/md0 /dev/sd[b-h]
(The kernel bug was in 2.6.35, and was triggered by forgetting the
"/dev/md0"; one of the RAID ioctls it tries confused up the /dev/sd
driver quite thoroughly)

It's an Intel chipset motherboard with a jmicros 2xSATA+PATA chip,
and one port on a SiI3132 plug-in card.  And mostly cheap Caviar green
EARS drives (yes, I know about the lack of TLER).  Note that I didn't
partition the drives, so there's no risk of a 4K-unaligned partition.

A friend with his own Linux media server said he saw the exact same speed
building his RAID6 array.

The speed_limit_max is set to the default, the system is idle, and the
resultant md drive is not mounted.

Is there a web page somewhere with speed tuning suggestions?


Thank you!

Re: RAID6 syncing at 24 MB/s?

[Jack Wang]

change  speed_limit_max to a bigger value will help.

Jack

2012/4/18 George Spelvin <linux@horizon.com>:
> I'm putting together a 7x2TB RAID6 for a home media server, and after
> encountering a fun kernel bug building the array (old kernel, a new one
> fixed it), I fot things working, but now notice that each disk is syncing
> at 23.8 MB/s.  hdparm -t reports they can do well over 100 MB/s each,
> and the processor, while only an i3, is spending about 20% of one core
> in the raid6 task.
>
> Basicall, it was created with:
> mdadm -C -l 6 -n 7 -c 128 -b internal /dev/md0 /dev/sd[b-h]
> (The kernel bug was in 2.6.35, and was triggered by forgetting the
> "/dev/md0"; one of the RAID ioctls it tries confused up the /dev/sd
> driver quite thoroughly)
>
> It's an Intel chipset motherboard with a jmicros 2xSATA+PATA chip,
> and one port on a SiI3132 plug-in card.  And mostly cheap Caviar green
> EARS drives (yes, I know about the lack of TLER).  Note that I didn't
> partition the drives, so there's no risk of a 4K-unaligned partition.
>
> A friend with his own Linux media server said he saw the exact same speed
> building his RAID6 array.
>
> The speed_limit_max is set to the default, the system is idle, and the
> resultant md drive is not mounted.
>
> Is there a web page somewhere with speed tuning suggestions?
>
>
> Thank you!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID6 syncing at 24 MB/s?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1828 bytes --]

On 18 Apr 2012 00:35:40 -0400 "George Spelvin" <linux@horizon.com> wrote:

> I'm putting together a 7x2TB RAID6 for a home media server, and after
> encountering a fun kernel bug building the array (old kernel, a new one
> fixed it), I fot things working, but now notice that each disk is syncing
> at 23.8 MB/s.  hdparm -t reports they can do well over 100 MB/s each,
> and the processor, while only an i3, is spending about 20% of one core
> in the raid6 task.

Can you try an experiment for me (if you don't have anything useful on the
array yet)?

Stop the array.  Recreate it exactly the same way as before, and see if it
goes faster.  I think it might - at least until it got up to where it was up
to.

NeilBrown


> 
> Basicall, it was created with:
> mdadm -C -l 6 -n 7 -c 128 -b internal /dev/md0 /dev/sd[b-h]
> (The kernel bug was in 2.6.35, and was triggered by forgetting the
> "/dev/md0"; one of the RAID ioctls it tries confused up the /dev/sd
> driver quite thoroughly)
> 
> It's an Intel chipset motherboard with a jmicros 2xSATA+PATA chip,
> and one port on a SiI3132 plug-in card.  And mostly cheap Caviar green
> EARS drives (yes, I know about the lack of TLER).  Note that I didn't
> partition the drives, so there's no risk of a 4K-unaligned partition.
> 
> A friend with his own Linux media server said he saw the exact same speed
> building his RAID6 array.
> 
> The speed_limit_max is set to the default, the system is idle, and the
> resultant md drive is not mounted.
> 
> Is there a web page somewhere with speed tuning suggestions?
> 
> 
> Thank you!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID6 syncing at 24 MB/s?

[George Spelvin]

> Can you try an experiment for me (if you don't have anything useful on the
> array yet)?
> 
> Stop the array.  Recreate it exactly the same way as before, and see if it
> goes faster.  I think it might - at least until it got up to where it was up
> to.

Um, unfortunately, I do (I didn't actually make the posting until after
the sync finished), but it should actually be non-destructive to do that
to an existing array, right?

It's just like deleting and re-creating a partition; as long as you're
*damn* sure you kept the parameters the same, it's a no-op.  Right?

I still have the data on the backup drives, so it's "only" a day's
lost work if I fry it (I did some reorganization as well as simple bulk
copying), so it's not *deadly* dangerous...

Thank you for the response!

Re: RAID6 syncing at 24 MB/s?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1630 bytes --]

On 18 Apr 2012 01:22:35 -0400 "George Spelvin" <linux@horizon.com> wrote:

> > Can you try an experiment for me (if you don't have anything useful on the
> > array yet)?
> > 
> > Stop the array.  Recreate it exactly the same way as before, and see if it
> > goes faster.  I think it might - at least until it got up to where it was up
> > to.
> 
> Um, unfortunately, I do (I didn't actually make the posting until after
> the sync finished), but it should actually be non-destructive to do that
> to an existing array, right?
> 
> It's just like deleting and re-creating a partition; as long as you're
> *damn* sure you kept the parameters the same, it's a no-op.  Right?

Should be - yes.  But some times it is best not to play with fire.
Up to you....

Alternately, look at /proc/diskstats for the member devices.

The first 4 numbers relate to reads (iOs, Merges, Sectors, ticks) and the next
4 to writes.

You will probably see about  3 times as many reads as writes.

RAID6 resync reads the whole stripe, checks the parity blocks, and if they
are wrong it writes them out.

If most of them are good - you should see streaming writes at the full speed
of the disk/bus/slowest-part.

If lots are not correct (which can be expected on a new array), you get lot
of 
    read - seek-backwards - write

sequences, which slow things down a lot.

NeilBrown


> 
> I still have the data on the backup drives, so it's "only" a day's
> lost work if I fry it (I did some reorganization as well as simple bulk
> copying), so it's not *deadly* dangerous...
> 
> Thank you for the response!


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]