Unbalanced reads of RAID10

Unbalanced reads of RAID10

[Piergiorgio Sartor]

Hi all,

some time ago I was reporting about a strange issue.

I have a two HDDs system, with a small RAID1 (/boot)
and the rest as RAID10 f2 (with LVM on top).

It seems that /dev/sdb has more reads than /dev/sda.

I had a quick check, with "iostat", and it seems that
all small reads, somehow below 1~4KiB, are done from
/dev/sdb2, regardless.
Actually, it seems that only if there is a pending
(small) read, this will be scheduled to /dev/sda2,
but non-overlapping small reads seem to happen always
from /dev/sdb2.

This occurs with the RAID10, but it seems also with
the RAID1.

In normal operation, this does not seem to lead to
problems, but during the smart long test /dev/sdb
takes by far more time than /dev/sda, since each
small read stop the test, and small read occurs
whenever there is a small write from syslog or
similar.
Note that failing and removing /dev/sdb2 results
in much shorter time for the smart test, about
1hr30min vs. the 6~7hrs with the drive still
attached to RAID10.

Is there any way to tune which is the "preferred"
drive or the "preferred" policy in case of these
small (or big) reads?

Could this be due to HW configuration?
The two HDDs are numbered SATA1 and SATA2 in BIOS,
there are still SATA3 and SATA4 ports somehow
available (SATA3 has a DVD).

How are the reads scheduled withing the RAID10 software?

Thanks,

bye,

-- 

piergiorgio

Re: Unbalanced reads of RAID10

[Keld Jørn Simonsen]

On Sun, Aug 23, 2009 at 04:24:39PM +0200, Piergiorgio Sartor wrote:
> Hi all,
> 
> some time ago I was reporting about a strange issue.
> 
> I have a two HDDs system, with a small RAID1 (/boot)
> and the rest as RAID10 f2 (with LVM on top).
> 
> It seems that /dev/sdb has more reads than /dev/sda.
> 
> I had a quick check, with "iostat", and it seems that
> all small reads, somehow below 1~4KiB, are done from
> /dev/sdb2, regardless.
> Actually, it seems that only if there is a pending
> (small) read, this will be scheduled to /dev/sda2,
> but non-overlapping small reads seem to happen always
> from /dev/sdb2.
> 
> This occurs with the RAID10, but it seems also with
> the RAID1.

Hmm, have you done testing separately on each array?

> In normal operation, this does not seem to lead to
> problems, but during the smart long test /dev/sdb
> takes by far more time than /dev/sda, since each
> small read stop the test, and small read occurs
> whenever there is a small write from syslog or
> similar.
> Note that failing and removing /dev/sdb2 results
> in much shorter time for the smart test, about
> 1hr30min vs. the 6~7hrs with the drive still
> attached to RAID10.
> 
> Is there any way to tune which is the "preferred"
> drive or the "preferred" policy in case of these
> small (or big) reads?

What level of the kernel are you running?

> Could this be due to HW configuration?
> The two HDDs are numbered SATA1 and SATA2 in BIOS,
> there are still SATA3 and SATA4 ports somehow
> available (SATA3 has a DVD).
> 
> How are the reads scheduled withing the RAID10 software?

there was a change of this about 2.6.25 which forced reads to always be
from the faster inner part of the disks, and that should even out reads.

Anyway I have seen some strange test results on reading small blocks
with raid10,f2 , but maybe this was pre 2.6.25. raid1 reads not behaving
as expected has been reported before.

best regards
keld

Re: Unbalanced reads of RAID10

[Bill Davidsen]

Keld Jørn Simonsen wrote:
> On Sun, Aug 23, 2009 at 04:24:39PM +0200, Piergiorgio Sartor wrote:
>   
>> Hi all,
>>
>> some time ago I was reporting about a strange issue.
>>
>> I have a two HDDs system, with a small RAID1 (/boot)
>> and the rest as RAID10 f2 (with LVM on top).
>>
>> It seems that /dev/sdb has more reads than /dev/sda.
>>
>> I had a quick check, with "iostat", and it seems that
>> all small reads, somehow below 1~4KiB, are done from
>> /dev/sdb2, regardless.
>> Actually, it seems that only if there is a pending
>> (small) read, this will be scheduled to /dev/sda2,
>> but non-overlapping small reads seem to happen always
>> from /dev/sdb2.
>>
>> This occurs with the RAID10, but it seems also with
>> the RAID1.
>>     
>
> Hmm, have you done testing separately on each array?
>
>   
>> In normal operation, this does not seem to lead to
>> problems, but during the smart long test /dev/sdb
>> takes by far more time than /dev/sda, since each
>> small read stop the test, and small read occurs
>> whenever there is a small write from syslog or
>> similar.
>> Note that failing and removing /dev/sdb2 results
>> in much shorter time for the smart test, about
>> 1hr30min vs. the 6~7hrs with the drive still
>> attached to RAID10.
>>
>> Is there any way to tune which is the "preferred"
>> drive or the "preferred" policy in case of these
>> small (or big) reads?
>>     
>
> What level of the kernel are you running?
>
>   
>> Could this be due to HW configuration?
>> The two HDDs are numbered SATA1 and SATA2 in BIOS,
>> there are still SATA3 and SATA4 ports somehow
>> available (SATA3 has a DVD).
>>
>> How are the reads scheduled withing the RAID10 software?
>>     
>
> there was a change of this about 2.6.25 which forced reads to always be
> from the faster inner part of the disks, and that should even out reads.
>   

??? what faster inner part is that? The linear velocity and 
bytes/cylinder are higher as the diameter increases.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc

"Now we have another quarterback besides Kurt Warner telling us during postgame
interviews that he owes every great thing that happens to him on a football
field to his faith in Jesus. I knew there had to be a reason why the Almighty
included a mute button on my remote."
			-- Arthur Troyer on Tim Tebow (Sports Illustrated)


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unbalanced reads of RAID10

[Keld Jørn Simonsen]

On Wed, Sep 02, 2009 at 12:13:15PM -0400, Bill Davidsen wrote:
> Keld Jørn Simonsen wrote:
>> On Sun, Aug 23, 2009 at 04:24:39PM +0200, Piergiorgio Sartor wrote:
>>   
>>> Hi all,
>>>
>>> some time ago I was reporting about a strange issue.
>>>
>>> I have a two HDDs system, with a small RAID1 (/boot)
>>> and the rest as RAID10 f2 (with LVM on top).
>>>
>>> It seems that /dev/sdb has more reads than /dev/sda.
>>>
>>> I had a quick check, with "iostat", and it seems that
>>> all small reads, somehow below 1~4KiB, are done from
>>> /dev/sdb2, regardless.
>>> Actually, it seems that only if there is a pending
>>> (small) read, this will be scheduled to /dev/sda2,
>>> but non-overlapping small reads seem to happen always
>>> from /dev/sdb2.
>>>
>>> This occurs with the RAID10, but it seems also with
>>> the RAID1.
>>>     
>>
>> Hmm, have you done testing separately on each array?
>>
>>   
>>> In normal operation, this does not seem to lead to
>>> problems, but during the smart long test /dev/sdb
>>> takes by far more time than /dev/sda, since each
>>> small read stop the test, and small read occurs
>>> whenever there is a small write from syslog or
>>> similar.
>>> Note that failing and removing /dev/sdb2 results
>>> in much shorter time for the smart test, about
>>> 1hr30min vs. the 6~7hrs with the drive still
>>> attached to RAID10.
>>>
>>> Is there any way to tune which is the "preferred"
>>> drive or the "preferred" policy in case of these
>>> small (or big) reads?
>>>     
>>
>> What level of the kernel are you running?
>>
>>   
>>> Could this be due to HW configuration?
>>> The two HDDs are numbered SATA1 and SATA2 in BIOS,
>>> there are still SATA3 and SATA4 ports somehow
>>> available (SATA3 has a DVD).
>>>
>>> How are the reads scheduled withing the RAID10 software?
>>>     
>>
>> there was a change of this about 2.6.25 which forced reads to always be
>> from the faster inner part of the disks, and that should even out reads.
>>   
>
> ??? what faster inner part is that? The linear velocity and  
> bytes/cylinder are higher as the diameter increases.

I meant the faster lower numbered sectors, which are the *outer*
sectors on the physical disk.

Best regards
Keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html