striping of a 4 drive raid10

striping of a 4 drive raid10

[Keld Jørn Simonsen]

Hi

I have tried to make a striping raid out of my new 4 x 1 TB
SATA-2 disks. I tried raid10,f2 in several ways:

1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0
of md0+md1

2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2
of md0+md1

3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of 
md0 =md1 =128 KB,  md2 = raid0 of md0+md1 chunksize = 256 KB

4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize
of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB

5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1

My new disks give a transfer rate of about 80 MB/s, so I expected
to have something like 320 MB/s for the whole raid, but I did not get
more than about 180 MB/s.

I think it may be something with the layout, that in effect 
the drives should be something like:

          sda1             sdb1    sdc1      sdd1
           0                1       2         3
           4                5       6         7

And this was not really doable for the combination of raids,
because thet combinations give different block layouts.

How can it be done? Do we need a new raid type?

Best regards
keld

Re: striping of a 4 drive raid10

[Peter Grandi]

>>> On Sun, 27 Jan 2008 20:33:45 +0100, Keld Jørn Simonsen
>>> <keld@dkuug.dk> said:

keld> Hi I have tried to make a striping raid out of my new 4 x
keld> 1 TB SATA-2 disks. I tried raid10,f2 in several ways:

keld> 1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0
keld>    of md0+md1
keld> 2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2
keld>    of md0+md1
keld> 3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of 
keld>    md0 =md1 =128 KB,  md2 = raid0 of md0+md1 chunksize = 256 KB
keld> 4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize
keld>    of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB

These stacked RAID levels don't make a lot of sense.

keld> 5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1

This also does not make a lot of sense. Why have four mirrors
instead of two?

Instead, try 'md0 = raid10,f2' for example. The first mirror of
will be striped across the outer half of all four drives, and
the second mirrors will be rotated in the inner half of each
drive.

Which of course means that reads will be quite quick, but writes
and degraded operation will be slower.

Consider this post for more details:

  http://www.spinics.net/lists/raid/msg18130.html

[ ... ]
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: striping of a 4 drive raid10

[Neil Brown]

On Sunday January 27, keld@dkuug.dk wrote:
> Hi
> 
> I have tried to make a striping raid out of my new 4 x 1 TB
> SATA-2 disks. I tried raid10,f2 in several ways:
> 
> 1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0
> of md0+md1
> 
> 2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2
> of md0+md1
> 
> 3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of 
> md0 =md1 =128 KB,  md2 = raid0 of md0+md1 chunksize = 256 KB
> 
> 4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize
> of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB
> 
> 5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1

Try
  6: md0 = raid10,f2 of sda1+sdb1+sdc1+sdd1

Also try raid10,o2 with a largeish chunksize (256KB is probably big
enough).

NeilBrown


> 
> My new disks give a transfer rate of about 80 MB/s, so I expected
> to have something like 320 MB/s for the whole raid, but I did not get
> more than about 180 MB/s.
> 
> I think it may be something with the layout, that in effect 
> the drives should be something like:
> 
>           sda1             sdb1    sdc1      sdd1
>            0                1       2         3
>            4                5       6         7
> 
> And this was not really doable for the combination of raids,
> because thet combinations give different block layouts.
> 
> How can it be done? Do we need a new raid type?
> 
> Best regards
> keld
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: striping of a 4 drive raid10

[Keld Jørn Simonsen]

On Mon, Jan 28, 2008 at 07:13:30AM +1100, Neil Brown wrote:
> On Sunday January 27, keld@dkuug.dk wrote:
> > Hi
> > 
> > I have tried to make a striping raid out of my new 4 x 1 TB
> > SATA-2 disks. I tried raid10,f2 in several ways:
> > 
> > 1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0
> > of md0+md1
> > 
> > 2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2
> > of md0+md1
> > 
> > 3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of 
> > md0 =md1 =128 KB,  md2 = raid0 of md0+md1 chunksize = 256 KB
> > 
> > 4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize
> > of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB
> > 
> > 5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1
> 
> Try
>   6: md0 = raid10,f2 of sda1+sdb1+sdc1+sdd1

That I already tried, (and I wrongly stated that I used f4 in stead of
f2). I had two times a thruput of about 300 MB/s but since then I could
not reproduce the behaviour. Are there errors on this that has been
corrected in newer kernels?


> Also try raid10,o2 with a largeish chunksize (256KB is probably big
> enough).

I tried that too, but my mdadm did not allow me to use the o flag.

My kernel is 2.6.12  and mdadm is v1.12.0 - 14 June 2005.
can I upgrade the mdadm alone to a newer version, and then which is
recommendable?

best regards
keld

Re: striping of a 4 drive raid10

[Keld Jørn Simonsen]

On Sun, Jan 27, 2008 at 08:11:35PM +0000, Peter Grandi wrote:
> >>> On Sun, 27 Jan 2008 20:33:45 +0100, Keld Jørn Simonsen
> >>> <keld@dkuug.dk> said:
> 
> keld> Hi I have tried to make a striping raid out of my new 4 x
> keld> 1 TB SATA-2 disks. I tried raid10,f2 in several ways:
> 
> keld> 1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0
> keld>    of md0+md1
> keld> 2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2
> keld>    of md0+md1
> keld> 3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of 
> keld>    md0 =md1 =128 KB,  md2 = raid0 of md0+md1 chunksize = 256 KB
> keld> 4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize
> keld>    of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB
> 
> These stacked RAID levels don't make a lot of sense.
> 
> keld> 5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1
> 
> This also does not make a lot of sense. Why have four mirrors
> instead of two?

My error, I did mean f2.

Anyway 4 mirrors would make the disk 2 times faster than 2 disks, and given disk
prices these days this could make a lot of sense.

> Instead, try 'md0 = raid10,f2' for example. The first mirror of
> will be striped across the outer half of all four drives, and
> the second mirrors will be rotated in the inner half of each
> drive.
> 
> Which of course means that reads will be quite quick, but writes
> and degraded operation will be slower.
> 
> Consider this post for more details:
> 
>   http://www.spinics.net/lists/raid/msg18130.html

Thanks for the reference.

There is also more in the original article on possible layouts of what
is now known as raid10,f2

http://marc.info/?l=linux-raid&m=107427614604701&w=2

including performance enhancements due to use of the faster outer
sectors, and smaller average seek times because you can seek on only
half the disk.

best regards
keld
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: striping of a 4 drive raid10

[Neil Brown]

On Sunday January 27, keld@dkuug.dk wrote:
> On Mon, Jan 28, 2008 at 07:13:30AM +1100, Neil Brown wrote:
> > On Sunday January 27, keld@dkuug.dk wrote:
> > > Hi
> > > 
> > > I have tried to make a striping raid out of my new 4 x 1 TB
> > > SATA-2 disks. I tried raid10,f2 in several ways:
> > > 
> > > 1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0
> > > of md0+md1
> > > 
> > > 2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2
> > > of md0+md1
> > > 
> > > 3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of 
> > > md0 =md1 =128 KB,  md2 = raid0 of md0+md1 chunksize = 256 KB
> > > 
> > > 4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize
> > > of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB
> > > 
> > > 5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1
> > 
> > Try
> >   6: md0 = raid10,f2 of sda1+sdb1+sdc1+sdd1
> 
> That I already tried, (and I wrongly stated that I used f4 in stead of
> f2). I had two times a thruput of about 300 MB/s but since then I could
> not reproduce the behaviour. Are there errors on this that has been
> corrected in newer kernels?

No, I don't think any performance related changes have been made to
raid10 lately.

You could try increasing the read-ahead size.  For a 4-drive raid10 it
defaults to 4 times the read-ahead setting of a single drive, but
increasing substantially (e.g. 64 times) seem to increase the speed of
"dd" reading a gigabyte.
Whether that will actually affect your target workload is a different question.

> 
> 
> > Also try raid10,o2 with a largeish chunksize (256KB is probably big
> > enough).
> 
> I tried that too, but my mdadm did not allow me to use the o flag.
> 
> My kernel is 2.6.12  and mdadm is v1.12.0 - 14 June 2005.
> can I upgrade the mdadm alone to a newer version, and then which is
> recommendable?

You would need a newer kernel and a newer mdadm to get raid10 - offset
mode.

NeilBrown

Re: striping of a 4 drive raid10

[Bill Davidsen]

Keld Jørn Simonsen wrote:
> On Mon, Jan 28, 2008 at 07:13:30AM +1100, Neil Brown wrote:
>   
>> On Sunday January 27, keld@dkuug.dk wrote:
>>     
>>> Hi
>>>
>>> I have tried to make a striping raid out of my new 4 x 1 TB
>>> SATA-2 disks. I tried raid10,f2 in several ways:
>>>
>>> 1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0
>>> of md0+md1
>>>
>>> 2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2
>>> of md0+md1
>>>
>>> 3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize of 
>>> md0 =md1 =128 KB,  md2 = raid0 of md0+md1 chunksize = 256 KB
>>>
>>> 4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize
>>> of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB
>>>
>>> 5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1
>>>       
>> Try
>>   6: md0 = raid10,f2 of sda1+sdb1+sdc1+sdd1
>>     
>
> That I already tried, (and I wrongly stated that I used f4 in stead of
> f2). I had two times a thruput of about 300 MB/s but since then I could
> not reproduce the behaviour. Are there errors on this that has been
> corrected in newer kernels?
>
>
>   
>> Also try raid10,o2 with a largeish chunksize (256KB is probably big
>> enough).
>>     
>
> I tried that too, but my mdadm did not allow me to use the o flag.
>
> My kernel is 2.6.12  and mdadm is v1.12.0 - 14 June 2005.
> can I upgrade the mdadm alone to a newer version, and then which is
> recommendable?
>   

I doubt that updating the mdadm is going to help, the kernel is old and 
lacks a number of improvements in the last few years. I don't think you 
will see any major improvements without a kernel upgrade.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 



-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: striping of a 4 drive raid10

[Keld Jørn Simonsen]

On Mon, Jan 28, 2008 at 01:32:48PM -0500, Bill Davidsen wrote:
> Neil Brown wrote:
> >On Sunday January 27, keld@dkuug.dk wrote:
> >  
> >>Hi
> >>
> >>I have tried to make a striping raid out of my new 4 x 1 TB
> >>SATA-2 disks. I tried raid10,f2 in several ways:
> >>
> >>1: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, md2 = raid0
> >>of md0+md1
> >>
> >>2: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, md2 = raid01,f2
> >>of md0+md1
> >>
> >>3: md0 = raid10,f2 of sda1+sdb1, md1= raid10,f2 of sdc1+sdd1, chunksize 
> >>of md0 =md1 =128 KB,  md2 = raid0 of md0+md1 chunksize = 256 KB
> >>
> >>4: md0 = raid0 of sda1+sdb1, md1= raid0 of sdc1+sdd1, chunksize
> >>of md0 = md1 = 128 KB, md2 = raid01,f2 of md0+md1 chunksize = 256 KB
> >>
> >>5: md0= raid10,f4 of sda1+sdb1+sdc1+sdd1
> >>    
> >
> >Try
> >  6: md0 = raid10,f2 of sda1+sdb1+sdc1+sdd1
> >
> >Also try raid10,o2 with a largeish chunksize (256KB is probably big
> >enough).
> >  
> 
> Looking at the issues raised, there might be some benefit from having 
> the mirror chunks on the slower inner tracks of a raid10, and to read 
> from the outer tracks if the drives with the data on the outer tracks 
> are idle. This would appear to offer a transfer rate benefit overall.

Hmm, how do I do this? I think this is normal behaviour of a raid10,f2.
Is that so?

So you mean I should rather use f2 than o2? Or should I configure the f2
in some way?

My hdparm -t gives:

/dev/sda5:
 Timing buffered beginning disk reads:   82 MB in  1.00 seconds = 81.686 MB/sec
 Timing buffered ending    disk reads:   42 MB in  1.03 seconds = 40.625 MB/sec
 Average seek time 13.714 msec, min=4.641, max=23.921
 Average track-to-track time 28.151 msec, min=26.729, max=28.730

So, yes, there is a reason to use the faster outer tracks - and have the 
faster access time that f2 gives . How does o2 behave here? Does it read
and search on the whole disk?


As to your other comments in another mail, I could of cause install
a newer kernel and mdadm, but then I would loose the support of my
supported and paid system. And Neil said that there have been no
performance fixes for f2 since the kernel I use (2.6.12).
I thought that o2 support was included since 2.6.10 - but apparantly not
so. 

Best regards
keld