Linux MD RAID 1 read performance tunning

Linux MD RAID 1 read performance tunning

[Ciprian Dorin, Craciun]

    Hello all!

    I've created a 64G RAID 1 matrix from 3 real disks. (I intend to
use this as a target for backups.)
    Now while playing around with this array, I've observed that the
read performance is quite low because it always reads from the disk in
the first slot (which happens to be the slowest...)

    So my questions are:
    * is there any way to tell the MD driver to load-balance the reads
between the three disks?
    * or is there any way to tell the MD driver that I would prefer to
read from a certain device? (maybe use write-mostly?)

    Thanks,
    Ciprian.

Re: Linux MD RAID 1 read performance tunning

[Keld Jørn Simonsen]

On Tue, Dec 22, 2009 at 06:34:55PM +0200, Ciprian Dorin, Craciun wrote:
>     Hello all!
> 
>     I've created a 64G RAID 1 matrix from 3 real disks. (I intend to
> use this as a target for backups.)
>     Now while playing around with this array, I've observed that the
> read performance is quite low because it always reads from the disk in
> the first slot (which happens to be the slowest...)
> 
>     So my questions are:
>     * is there any way to tell the MD driver to load-balance the reads
> between the three disks?

It does not make sense to do distributed reading in raid1 for sequential
files. This is because it will not be faster to read from more drives,
as this will only make the reading from one drive skipping blocks on
that drive. In other words, in the time you use for skipping blocks on
one drive, you could just as well have read the blocks. So then better
just read all the blocks off one drive, and then do other possible IO
from other drives.

RAID10 is a modern form of raid1, which can do better distributed
reading, especially raid10,f2 is good for disks, while raid10,o2 may be
better for SSDs.

Best regards
Keld

Re: Linux MD RAID 1 read performance tunning

[Ciprian Dorin, Craciun]

2009/12/22 Keld Jørn Simonsen <keld@keldix.com>:
> On Tue, Dec 22, 2009 at 06:34:55PM +0200, Ciprian Dorin, Craciun wrote:
>>     Hello all!
>>
>>     I've created a 64G RAID 1 matrix from 3 real disks. (I intend to
>> use this as a target for backups.)
>>     Now while playing around with this array, I've observed that the
>> read performance is quite low because it always reads from the disk in
>> the first slot (which happens to be the slowest...)
>>
>>     So my questions are:
>>     * is there any way to tell the MD driver to load-balance the reads
>> between the three disks?
>
> It does not make sense to do distributed reading in raid1 for sequential
> files. This is because it will not be faster to read from more drives,
> as this will only make the reading from one drive skipping blocks on
> that drive. In other words, in the time you use for skipping blocks on
> one drive, you could just as well have read the blocks. So then better
> just read all the blocks off one drive, and then do other possible IO
> from other drives.

    Aha. It makes sens now. But, does it mean that if I have parallel
IO's (from different read operations) they are going to be distributed
between the disks?


> RAID10 is a modern form of raid1, which can do better distributed
> reading, especially raid10,f2 is good for disks, while raid10,o2 may be
> better for SSDs.
>
> Best regards
> Keld

    Unfortunately my setup forces me to use RAID1. (Because any of the
three drives should be usable for a full recovery of the backup data.)
(P.S.: Just to be clear, I'm not using RAID as a backup solution, but
rather a redundancy solution for the backup solution. :) )

    Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux MD RAID 1 read performance tunning

[Keld Jørn Simonsen]

On Tue, Dec 22, 2009 at 07:08:25PM +0200, Ciprian Dorin, Craciun wrote:
> 2009/12/22 Keld Jørn Simonsen <keld@keldix.com>:
> > On Tue, Dec 22, 2009 at 06:34:55PM +0200, Ciprian Dorin, Craciun wrote:
> >>     Hello all!
> >>
> >>     I've created a 64G RAID 1 matrix from 3 real disks. (I intend to
> >> use this as a target for backups.)
> >>     Now while playing around with this array, I've observed that the
> >> read performance is quite low because it always reads from the disk in
> >> the first slot (which happens to be the slowest...)
> >>
> >>     So my questions are:
> >>     * is there any way to tell the MD driver to load-balance the reads
> >> between the three disks?
> >
> > It does not make sense to do distributed reading in raid1 for sequential
> > files. This is because it will not be faster to read from more drives,
> > as this will only make the reading from one drive skipping blocks on
> > that drive. In other words, in the time you use for skipping blocks on
> > one drive, you could just as well have read the blocks. So then better
> > just read all the blocks off one drive, and then do other possible IO
> > from other drives.
> 
>     Aha. It makes sens now. But, does it mean that if I have parallel
> IO's (from different read operations) they are going to be distributed
> between the disks?

It should, but I am not fully sure it does.
But try it out with two concurrent reads of two big files, and then
watch it with iostat 

Best regards
keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux MD RAID 1 read performance tunning

[Goswin von Brederlow]

Keld Jørn Simonsen <keld@keldix.com> writes:

> On Tue, Dec 22, 2009 at 07:08:25PM +0200, Ciprian Dorin, Craciun wrote:
>> 2009/12/22 Keld Jørn Simonsen <keld@keldix.com>:
>> > On Tue, Dec 22, 2009 at 06:34:55PM +0200, Ciprian Dorin, Craciun wrote:
>> >>     Hello all!
>> >>
>> >>     I've created a 64G RAID 1 matrix from 3 real disks. (I intend to
>> >> use this as a target for backups.)
>> >>     Now while playing around with this array, I've observed that the
>> >> read performance is quite low because it always reads from the disk in
>> >> the first slot (which happens to be the slowest...)
>> >>
>> >>     So my questions are:
>> >>     * is there any way to tell the MD driver to load-balance the reads
>> >> between the three disks?
>> >
>> > It does not make sense to do distributed reading in raid1 for sequential
>> > files. This is because it will not be faster to read from more drives,
>> > as this will only make the reading from one drive skipping blocks on
>> > that drive. In other words, in the time you use for skipping blocks on
>> > one drive, you could just as well have read the blocks. So then better
>> > just read all the blocks off one drive, and then do other possible IO
>> > from other drives.
>> 
>>     Aha. It makes sens now. But, does it mean that if I have parallel
>> IO's (from different read operations) they are going to be distributed
>> between the disks?
>
> It should, but I am not fully sure it does.
> But try it out with two concurrent reads of two big files, and then
> watch it with iostat 
>
> Best regards
> keld

Actualy the kernel remembers the last read/write position for each
raid1 component and then uses the one which is nearest.

And when you read at 2/3 different positions at the same time then it
will use different components for each an use the same ones for
subsequent reads (as they will be nearer).

Try

dd if=/dev/md0 of=/dev/null bs=1M cunt=1024 skip=0 &
dd if=/dev/md0 of=/dev/null bs=1M cunt=1024 skip=1024 &
dd if=/dev/md0 of=/dev/null bs=1M cunt=1024 skip=2048

They should more or less get the same speed as a single dd.

MfG
        Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux MD RAID 1 read performance tunning

[Ciprian Dorin, Craciun]

On Thu, Dec 24, 2009 at 11:03 AM, Goswin von Brederlow
<goswin-v-b@web.de> wrote:
> Keld Jørn Simonsen <keld@keldix.com> writes:
>
>> On Tue, Dec 22, 2009 at 07:08:25PM +0200, Ciprian Dorin, Craciun wrote:
>>> 2009/12/22 Keld Jørn Simonsen <keld@keldix.com>:
>>> > On Tue, Dec 22, 2009 at 06:34:55PM +0200, Ciprian Dorin, Craciun wrote:
>>> >>     Hello all!
>>> >>
>>> >>     I've created a 64G RAID 1 matrix from 3 real disks. (I intend to
>>> >> use this as a target for backups.)
>>> >>     Now while playing around with this array, I've observed that the
>>> >> read performance is quite low because it always reads from the disk in
>>> >> the first slot (which happens to be the slowest...)
>>> >>
>>> >>     So my questions are:
>>> >>     * is there any way to tell the MD driver to load-balance the reads
>>> >> between the three disks?
>>> >
>>> > It does not make sense to do distributed reading in raid1 for sequential
>>> > files. This is because it will not be faster to read from more drives,
>>> > as this will only make the reading from one drive skipping blocks on
>>> > that drive. In other words, in the time you use for skipping blocks on
>>> > one drive, you could just as well have read the blocks. So then better
>>> > just read all the blocks off one drive, and then do other possible IO
>>> > from other drives.
>>>
>>>     Aha. It makes sens now. But, does it mean that if I have parallel
>>> IO's (from different read operations) they are going to be distributed
>>> between the disks?
>>
>> It should, but I am not fully sure it does.
>> But try it out with two concurrent reads of two big files, and then
>> watch it with iostat
>>
>> Best regards
>> keld
>
> Actualy the kernel remembers the last read/write position for each
> raid1 component and then uses the one which is nearest.
>
> And when you read at 2/3 different positions at the same time then it
> will use different components for each an use the same ones for
> subsequent reads (as they will be nearer).
>
> Try
>
> dd if=/dev/md0 of=/dev/null bs=1M cunt=1024 skip=0 &
> dd if=/dev/md0 of=/dev/null bs=1M cunt=1024 skip=1024 &
> dd if=/dev/md0 of=/dev/null bs=1M cunt=1024 skip=2048
>
> They should more or less get the same speed as a single dd.
>
> MfG
>        Goswin


    Thanks all for your feedback. (I haven't tried the proposed three
dd's in parallel, but I promise I'll try them the next time I assemble
my backup array.)

    But one observation though:
    * indeed my usage of the array was mono-process;
    * when reading from the array to construct the MD5 sums for the
files I've used only one process;
    * indeed the data was read from a single disk (at a time);
    * but now the interesting think comes: I think it favored one disk
(the same most of the time) over the others;

    Is this as expected?

    Thanks again,
    Ciprian.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux MD RAID 1 read performance tunning

[Goswin von Brederlow]

"Ciprian Dorin, Craciun" <ciprian.craciun@gmail.com> writes:

>     Thanks all for your feedback. (I haven't tried the proposed three
> dd's in parallel, but I promise I'll try them the next time I assemble
> my backup array.)
>
>     But one observation though:
>     * indeed my usage of the array was mono-process;
>     * when reading from the array to construct the MD5 sums for the
> files I've used only one process;
>     * indeed the data was read from a single disk (at a time);
>     * but now the interesting think comes: I think it favored one disk
> (the same most of the time) over the others;
>
>     Is this as expected?
>
>     Thanks again,
>     Ciprian.

I don't think it should favour any one disk in particular. But it
might be that if all disks are at the same place (e.g. you just wrote
something) it will always pick the first one. You have to check the
source for that or maybe Neil knows from memory.

But you can tune this:

       -W, --write-mostly
              subsequent devices listed in a --build, --create, or
              --add com- mand will be flagged as 'write-mostly'.  This
              is valid for RAID1 only and means that the 'md' driver
              will avoid reading from these devices if at all
              possible.  This can be useful if mirror- ing over a slow
              link.

       --write-behind=
              Specify that write-behind mode should be enabled (valid
              for RAID1 only).  If an argument is specified, it will
              set the maxi- mum number of outstanding writes allowed.
              The default value is 256.  A write-intent bitmap is
              required in order to use write- behind mode, and
              write-behind is only attempted on drives marked as
              write-mostly.

Since you mentioned that you have different speeds on the disks maybe
you should set this for the slow drive(s).

MfG
        Goswin