stripe_cache_size and performance

stripe_cache_size and performance

[Jon Nelson]

I've been futzing with stripe_cache_size on a 3x component raid5, 
using 2.6.18.8-0.3-default on x86_64 (openSUSE 10.2).

With the value set at 4096 I get pretty great write numbers.
2048 and on down the write numbers slowly drop.

However, at values above 512 the 'check' performance is terrible. By 
'check' performance I mean the value displayed by /proc/mdstat after
I issue:

echo check > /sys/block/md0/md/sync_action

When I say "terrible" I mean < 3MB/s.
When I use 384, the performance goes to ~70MB/s
512.. 72-73MB/s
640.. 73-74MB/s

768.. 3300 K/s. Wow!

Can somebody 'splain to me what is going on?

--
Jon Nelson <jnelson-linux-raid@jamponi.net>

Re: stripe_cache_size and performance

[Raz]

What is your raid configuration ?
Please note that the stripe_cache_size is acting as a bottle neck in some cases.



On 6/21/07, Jon Nelson <jnelson-linux-raid@jamponi.net> wrote:
>
> I've been futzing with stripe_cache_size on a 3x component raid5,
> using 2.6.18.8-0.3-default on x86_64 (openSUSE 10.2).
>
> With the value set at 4096 I get pretty great write numbers.
> 2048 and on down the write numbers slowly drop.
>
> However, at values above 512 the 'check' performance is terrible. By
> 'check' performance I mean the value displayed by /proc/mdstat after
> I issue:
>
> echo check > /sys/block/md0/md/sync_action
>
> When I say "terrible" I mean < 3MB/s.
> When I use 384, the performance goes to ~70MB/s
> 512.. 72-73MB/s
> 640.. 73-74MB/s
>
> 768.. 3300 K/s. Wow!
>
> Can somebody 'splain to me what is going on?
>
> --
> Jon Nelson <jnelson-linux-raid@jamponi.net>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Raz

Re: stripe_cache_size and performance

[Jon Nelson]

On Thu, 21 Jun 2007, Raz wrote:

> What is your raid configuration ?
> Please note that the stripe_cache_size is acting as a bottle neck in some
> cases.

Well, it's 3x SATA drives in raid5. 320G drives each, and I'm using a 
314G partition from each disk (the rest of the space is quiescent).

> On 6/21/07, Jon Nelson <jnelson-linux-raid@jamponi.net> wrote:
> >
> > I've been futzing with stripe_cache_size on a 3x component raid5,
> > using 2.6.18.8-0.3-default on x86_64 (openSUSE 10.2).
> >
> > With the value set at 4096 I get pretty great write numbers.
> > 2048 and on down the write numbers slowly drop.
> >
> > However, at values above 512 the 'check' performance is terrible. By
> > 'check' performance I mean the value displayed by /proc/mdstat after
> > I issue:
> >
> > echo check > /sys/block/md0/md/sync_action
> >
> > When I say "terrible" I mean < 3MB/s.
> > When I use 384, the performance goes to ~70MB/s
> > 512.. 72-73MB/s
> > 640.. 73-74MB/s
> >
> > 768.. 3300 K/s. Wow!
> >
> > Can somebody 'splain to me what is going on?
> >
> > --
> > Jon Nelson <jnelson-linux-raid@jamponi.net>
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> 
> -- 
> Raz
> 
> 
> 

--
Jon Nelson <jnelson-linux-raid@jamponi.net>

Re: stripe_cache_size and performance

[Raz]

On 6/22/07, Jon Nelson <jnelson-linux-raid@jamponi.net> wrote:
> On Thu, 21 Jun 2007, Raz wrote:
>
> > What is your raid configuration ?
> > Please note that the stripe_cache_size is acting as a bottle neck in some
> > cases.
>
> Well, it's 3x SATA drives in raid5. 320G drives each, and I'm using a
> 314G partition from each disk (the rest of the space is quiescent).
what is the chunk size ?. Please note that a raid performance is dependent
on the access pattern to the disks. What is this raid aimed to do ?
more reads ? more writes ? If you can control the access pattern ,
you might be able to increase performance.
> > On 6/21/07, Jon Nelson <jnelson-linux-raid@jamponi.net> wrote:
> > >
> > > I've been futzing with stripe_cache_size on a 3x component raid5,
> > > using 2.6.18.8-0.3-default on x86_64 (openSUSE 10.2).
> > >
> > > With the value set at 4096 I get pretty great write numbers.
> > > 2048 and on down the write numbers slowly drop.
> > >
> > > However, at values above 512 the 'check' performance is terrible. By
> > > 'check' performance I mean the value displayed by /proc/mdstat after
> > > I issue:
> > >
> > > echo check > /sys/block/md0/md/sync_action
> > >
> > > When I say "terrible" I mean < 3MB/s.
> > > When I use 384, the performance goes to ~70MB/s
> > > 512.. 72-73MB/s
> > > 640.. 73-74MB/s
> > >
> > > 768.. 3300 K/s. Wow!
> > >
> > > Can somebody 'splain to me what is going on?
> > >
> > > --
> > > Jon Nelson <jnelson-linux-raid@jamponi.net>
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >
> >
> >
> > --
> > Raz
> >
> >
> >
>
> --
> Jon Nelson <jnelson-linux-raid@jamponi.net>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Raz

Re: stripe_cache_size and performance

[Justin Piszcz]

I have found a 16MB stripe_cache_size results in optimal performance after 
testing many many values :)

On Fri, 22 Jun 2007, Raz wrote:

> On 6/22/07, Jon Nelson <jnelson-linux-raid@jamponi.net> wrote:
>> On Thu, 21 Jun 2007, Raz wrote:
>> 
>> > What is your raid configuration ?
>> > Please note that the stripe_cache_size is acting as a bottle neck in some
>> > cases.
>> 
>> Well, it's 3x SATA drives in raid5. 320G drives each, and I'm using a
>> 314G partition from each disk (the rest of the space is quiescent).
> what is the chunk size ?. Please note that a raid performance is dependent
> on the access pattern to the disks. What is this raid aimed to do ?
> more reads ? more writes ? If you can control the access pattern ,
> you might be able to increase performance.
>> > On 6/21/07, Jon Nelson <jnelson-linux-raid@jamponi.net> wrote:
>> > >
>> > > I've been futzing with stripe_cache_size on a 3x component raid5,
>> > > using 2.6.18.8-0.3-default on x86_64 (openSUSE 10.2).
>> > >
>> > > With the value set at 4096 I get pretty great write numbers.
>> > > 2048 and on down the write numbers slowly drop.
>> > >
>> > > However, at values above 512 the 'check' performance is terrible. By
>> > > 'check' performance I mean the value displayed by /proc/mdstat after
>> > > I issue:
>> > >
>> > > echo check > /sys/block/md0/md/sync_action
>> > >
>> > > When I say "terrible" I mean < 3MB/s.
>> > > When I use 384, the performance goes to ~70MB/s
>> > > 512.. 72-73MB/s
>> > > 640.. 73-74MB/s
>> > >
>> > > 768.. 3300 K/s. Wow!
>> > >
>> > > Can somebody 'splain to me what is going on?
>> > >
>> > > --
>> > > Jon Nelson <jnelson-linux-raid@jamponi.net>
>> > > -
>> > > To unsubscribe from this list: send the line "unsubscribe linux-raid" 
>> in
>> > > the body of a message to majordomo@vger.kernel.org
>> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> > >
>> >
>> >
>> > --
>> > Raz
>> >
>> >
>> >
>> 
>> --
>> Jon Nelson <jnelson-linux-raid@jamponi.net>
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
>
>
> -- 
> Raz
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: stripe_cache_size and performance

[Bill Davidsen]

Justin Piszcz wrote:
> I have found a 16MB stripe_cache_size results in optimal performance 
> after testing many many values :)

We have discussed this before, my experience has been that after 8 x 
stripe size the performance gains hit diminishing returns, particularly 
for typical write instead of big aligned blocks, possibly with O_DIRECT. 
I would suggest that as a target even on a low memory machine.

Do your tests show similar? I was only able to test three and four drive 
setups using dedicated drives.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: stripe_cache_size and performance

[Justin Piszcz]

On Mon, 25 Jun 2007, Bill Davidsen wrote:

> Justin Piszcz wrote:
>> I have found a 16MB stripe_cache_size results in optimal performance after 
>> testing many many values :)
>
> We have discussed this before, my experience has been that after 8 x stripe 
> size the performance gains hit diminishing returns, particularly for typical 
> write instead of big aligned blocks, possibly with O_DIRECT. I would suggest 
> that as a target even on a low memory machine.
>
> Do your tests show similar? I was only able to test three and four drive 
> setups using dedicated drives.
>
> -- 
> bill davidsen <davidsen@tmr.com>
> CTO TMR Associates, Inc
> Doing interesting things with small computers since 1979
>

I will re-benchmark with my current setup, however:

Each of these are averaged over three runs:

128k_stripe: 69.2MB/s
256k_stripe: 105.3MB/s
512k_stripe: 142.0MB/s
1024k_stripe: 144.6MB/s
2048k_stripe: 208.3MB/s
4096k_stripe: 223.6MB/s
8192k_stripe: 226.0MB/s
16384k_stripe: 215.0MB/s

This was with 6 sata disks.

Justin.

Re: stripe_cache_size and performance

[Justin Piszcz]

On Mon, 25 Jun 2007, Justin Piszcz wrote:

>
>
> On Mon, 25 Jun 2007, Bill Davidsen wrote:
>
>> Justin Piszcz wrote:
>>> I have found a 16MB stripe_cache_size results in optimal performance after 
>>> testing many many values :)
>> 
>> We have discussed this before, my experience has been that after 8 x stripe 
>> size the performance gains hit diminishing returns, particularly for 
>> typical write instead of big aligned blocks, possibly with O_DIRECT. I 
>> would suggest that as a target even on a low memory machine.
>> 
>> Do your tests show similar? I was only able to test three and four drive 
>> setups using dedicated drives.
>> 
>> -- 
>> bill davidsen <davidsen@tmr.com>
>> CTO TMR Associates, Inc
>> Doing interesting things with small computers since 1979
>> 
>
> I will re-benchmark with my current setup, however:
>
> Each of these are averaged over three runs:
>
> 128k_stripe: 69.2MB/s
> 256k_stripe: 105.3MB/s
> 512k_stripe: 142.0MB/s
> 1024k_stripe: 144.6MB/s
> 2048k_stripe: 208.3MB/s
> 4096k_stripe: 223.6MB/s
> 8192k_stripe: 226.0MB/s
> 16384k_stripe: 215.0MB/s
>
> This was with 6 sata disks.
>
> Justin.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

I set it to 32 and my machine has several hung processes in D state, not 
good.  I will start with 128k and up.

Justin.

Re: stripe_cache_size and performance

[Justin Piszcz]

It was going with 32k just REALLY slow, will use 128k+

  1073737728 2007-06-25 13:07 Bonnie.5178.000


On Mon, 25 Jun 2007, Justin Piszcz wrote:

>
>
> On Mon, 25 Jun 2007, Justin Piszcz wrote:
>
>> 
>> 
>> On Mon, 25 Jun 2007, Bill Davidsen wrote:
>> 
>>> Justin Piszcz wrote:
>>>> I have found a 16MB stripe_cache_size results in optimal performance 
>>>> after testing many many values :)
>>> 
>>> We have discussed this before, my experience has been that after 8 x 
>>> stripe size the performance gains hit diminishing returns, particularly 
>>> for typical write instead of big aligned blocks, possibly with O_DIRECT. I 
>>> would suggest that as a target even on a low memory machine.
>>> 
>>> Do your tests show similar? I was only able to test three and four drive 
>>> setups using dedicated drives.
>>> 
>>> -- 
>>> bill davidsen <davidsen@tmr.com>
>>> CTO TMR Associates, Inc
>>> Doing interesting things with small computers since 1979
>>> 
>> 
>> I will re-benchmark with my current setup, however:
>> 
>> Each of these are averaged over three runs:
>> 
>> 128k_stripe: 69.2MB/s
>> 256k_stripe: 105.3MB/s
>> 512k_stripe: 142.0MB/s
>> 1024k_stripe: 144.6MB/s
>> 2048k_stripe: 208.3MB/s
>> 4096k_stripe: 223.6MB/s
>> 8192k_stripe: 226.0MB/s
>> 16384k_stripe: 215.0MB/s
>> 
>> This was with 6 sata disks.
>> 
>> Justin.
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
>
> I set it to 32 and my machine has several hung processes in D state, not 
> good.  I will start with 128k and up.
>
> Justin.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: stripe_cache_size and performance

[Justin Piszcz]

On Mon, 25 Jun 2007, Bill Davidsen wrote:

> Justin Piszcz wrote:
>> I have found a 16MB stripe_cache_size results in optimal performance after 
>> testing many many values :)
>
> We have discussed this before, my experience has been that after 8 x stripe 
> size the performance gains hit diminishing returns, particularly for typical 
> write instead of big aligned blocks, possibly with O_DIRECT. I would suggest 
> that as a target even on a low memory machine.
>
> Do your tests show similar? I was only able to test three and four drive 
> setups using dedicated drives.
>
> -- 
> bill davidsen <davidsen@tmr.com>
> CTO TMR Associates, Inc
> Doing interesting things with small computers since 1979
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Running test with 10 RAPTOR 150 hard drives, expect it to take awhile 
until I get the results, avg them etc. :)

128k,256k,512k,1024k,2048k,4096k,8192k,16384k

Justin.

Re: stripe_cache_size and performance [BUG with =64kb]

[Justin Piszcz]

On Mon, 25 Jun 2007, Justin Piszcz wrote:

>
>
> On Mon, 25 Jun 2007, Bill Davidsen wrote:
>
>> Justin Piszcz wrote:
>>> I have found a 16MB stripe_cache_size results in optimal performance after 
>>> testing many many values :)
>> 
>> We have discussed this before, my experience has been that after 8 x stripe 
>> size the performance gains hit diminishing returns, particularly for 
>> typical write instead of big aligned blocks, possibly with O_DIRECT. I 
>> would suggest that as a target even on a low memory machine.
>> 
>> Do your tests show similar? I was only able to test three and four drive 
>> setups using dedicated drives.
>> 
>> -- 
>> bill davidsen <davidsen@tmr.com>
>> CTO TMR Associates, Inc
>> Doing interesting things with small computers since 1979
>> 
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
>
> Running test with 10 RAPTOR 150 hard drives, expect it to take awhile until I 
> get the results, avg them etc. :)
>
> 128k,256k,512k,1024k,2048k,4096k,8192k,16384k
>
> Justin.
>

Definitely a kernel bug, I set it to 64kb and it stayed in D-state until I 
ran alt-sysrq-b.

Pretty nasty!

Justin.

Re: stripe_cache_size and performance [BUG with =64kb]

[Justin Piszcz]

On Mon, 25 Jun 2007, Justin Piszcz wrote:

>
>
> On Mon, 25 Jun 2007, Justin Piszcz wrote:
>
>> 
>> 
>> On Mon, 25 Jun 2007, Bill Davidsen wrote:
>> 
>>> Justin Piszcz wrote:
>>>> I have found a 16MB stripe_cache_size results in optimal performance 
>>>> after testing many many values :)
>>> 
>>> We have discussed this before, my experience has been that after 8 x 
>>> stripe size the performance gains hit diminishing returns, particularly 
>>> for typical write instead of big aligned blocks, possibly with O_DIRECT. I 
>>> would suggest that as a target even on a low memory machine.
>>> 
>>> Do your tests show similar? I was only able to test three and four drive 
>>> setups using dedicated drives.
>>> 
>>> -- 
>>> bill davidsen <davidsen@tmr.com>
>>> CTO TMR Associates, Inc
>>> Doing interesting things with small computers since 1979
>>> 
>>> -
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> 
>> 
>> Running test with 10 RAPTOR 150 hard drives, expect it to take awhile until 
>> I get the results, avg them etc. :)
>> 
>> 128k,256k,512k,1024k,2048k,4096k,8192k,16384k
>> 
>> Justin.
>> 
>
> Definitely a kernel bug, I set it to 64kb and it stayed in D-state until I 
> ran alt-sysrq-b.
>
> Pretty nasty!
>
> Justin.
>

Ack, help??

[   64.032895] Starting XFS recovery on filesystem: md3 (logdev: internal)
[   66.210602] XFS: xlog_recover_process_data: bad clientid
[   66.210656] XFS: log mount/recovery failed: error 5
[   66.210709] XFS: log mount failed

After I ran 64kb it killed my RAID!

Re: stripe_cache_size and performance [BUG with =64kb]

[Justin Piszcz]

On Mon, 25 Jun 2007, Justin Piszcz wrote:

>
>
> On Mon, 25 Jun 2007, Justin Piszcz wrote:
>
>> 
>> 
>> On Mon, 25 Jun 2007, Justin Piszcz wrote:
>> 
>>> 
>>> 
>>> On Mon, 25 Jun 2007, Bill Davidsen wrote:
>>> 
>>>> Justin Piszcz wrote:
>>>>> I have found a 16MB stripe_cache_size results in optimal performance 
>>>>> after testing many many values :)
>>>> 
>>>> We have discussed this before, my experience has been that after 8 x 
>>>> stripe size the performance gains hit diminishing returns, particularly 
>>>> for typical write instead of big aligned blocks, possibly with O_DIRECT. 
>>>> I would suggest that as a target even on a low memory machine.
>>>> 
>>>> Do your tests show similar? I was only able to test three and four drive 
>>>> setups using dedicated drives.
>>>> 
>>>> -- 
>>>> bill davidsen <davidsen@tmr.com>
>>>> CTO TMR Associates, Inc
>>>> Doing interesting things with small computers since 1979
>>>> 
>>>> -
>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> 
>>> 
>>> Running test with 10 RAPTOR 150 hard drives, expect it to take awhile 
>>> until I get the results, avg them etc. :)
>>> 
>>> 128k,256k,512k,1024k,2048k,4096k,8192k,16384k
>>> 
>>> Justin.
>>> 
>> 
>> Definitely a kernel bug, I set it to 64kb and it stayed in D-state until I 
>> ran alt-sysrq-b.
>> 
>> Pretty nasty!
>> 
>> Justin.
>> 
>
> Ack, help??
>
> [   64.032895] Starting XFS recovery on filesystem: md3 (logdev: internal)
> [   66.210602] XFS: xlog_recover_process_data: bad clientid
> [   66.210656] XFS: log mount/recovery failed: error 5
> [   66.210709] XFS: log mount failed
>
> After I ran 64kb it killed my RAID!
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Yeah, I won't be trying that anymore :P

p34:/r1# find lost+found/|wc
     157     157    3369
p34:/r1# du -sh lost+found/
166G    lost+found/
p34:/r1#

Re: stripe_cache_size and performance [BUG with =64kb]

[Justin Piszcz]

On Mon, 25 Jun 2007, Justin Piszcz wrote:

>
>
> On Mon, 25 Jun 2007, Justin Piszcz wrote:
>
>> 
>> 
>> On Mon, 25 Jun 2007, Justin Piszcz wrote:
>> 
>>> 
>>> 
>>> On Mon, 25 Jun 2007, Justin Piszcz wrote:
>>> 
>>>> 
>>>> 
>>>> On Mon, 25 Jun 2007, Bill Davidsen wrote:
>>>> 
>>>>> Justin Piszcz wrote:
>>>>>> I have found a 16MB stripe_cache_size results in optimal performance 
>>>>>> after testing many many values :)
>>>>> 
>>>>> We have discussed this before, my experience has been that after 8 x 
>>>>> stripe size the performance gains hit diminishing returns, particularly 
>>>>> for typical write instead of big aligned blocks, possibly with O_DIRECT. 
>>>>> I would suggest that as a target even on a low memory machine.
>>>>> 
>>>>> Do your tests show similar? I was only able to test three and four drive 
>>>>> setups using dedicated drives.
>>>>> 
>>>>> -- 
>>>>> bill davidsen <davidsen@tmr.com>
>>>>> CTO TMR Associates, Inc
>>>>> Doing interesting things with small computers since 1979
>>>>> 
>>>>> -
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>> 
>>>> 
>>>> Running test with 10 RAPTOR 150 hard drives, expect it to take awhile 
>>>> until I get the results, avg them etc. :)
>>>> 
>>>> 128k,256k,512k,1024k,2048k,4096k,8192k,16384k
>>>> 
>>>> Justin.
>>>> 
>>> 
>>> Definitely a kernel bug, I set it to 64kb and it stayed in D-state until I 
>>> ran alt-sysrq-b.
>>> 
>>> Pretty nasty!
>>> 
>>> Justin.
>>> 
>> 
>> Ack, help??
>> 
>> [   64.032895] Starting XFS recovery on filesystem: md3 (logdev: internal)
>> [   66.210602] XFS: xlog_recover_process_data: bad clientid
>> [   66.210656] XFS: log mount/recovery failed: error 5
>> [   66.210709] XFS: log mount failed
>> 
>> After I ran 64kb it killed my RAID!
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
>
> Yeah, I won't be trying that anymore :P
>
> p34:/r1# find lost+found/|wc
>    157     157    3369
> p34:/r1# du -sh lost+found/
> 166G    lost+found/
> p34:/r1#
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

16MB is best performance per previous benchmarks.

Re: stripe_cache_size and performance

[Jon Nelson]

On Thu, 21 Jun 2007, Jon Nelson wrote:

> On Thu, 21 Jun 2007, Raz wrote:
> 
> > What is your raid configuration ?
> > Please note that the stripe_cache_size is acting as a bottle neck in some
> > cases.

Well, that's kind of the point of my email. I'll try to restate things, 
as my question appears to have gotten lost.

1. I have a 3x component raid5, ~314G per component. Each component 
happens to be the 4th partition of a 320G SATA drive. Each drive 
can sustain approx. 70MB/s reads/writes. Except for the first 
drive, none of the other partitions are used for anything else at this 
time. The system is nominally quiescent during these tests.

2. The kernel is 2.6.18.8-0.3-default on x86_64 (openSUSE 10.2).

3. My best sustained write performance comes with a stripe_cache_size of 
4096. Larger than that seems to reduce performance, although only very 
slightly.

4. At values below 4096, the absolute write performance is less than the 
best, but only marginally. 

5. HOWEVER, at any value *above* 512 the 'check' performance is REALLY 
BAD. By 'check' performance I mean the value displayed by /proc/mdstat 
after I issue:

echo check > /sys/block/md0/md/sync_action

When I say "REALLY BAD" I mean < 3MB/s. 

6. Here is a short incomplete table of stripe_cache_size to 'check' 
performance:

384.... 72-73MB/s
512.... 72-73MB/s
640.... 73-74MB/s
768.....3-3.4MB/s

And the performance stays "bad" as I increase the stripe_cache_size.

7. And now, the question: the best absolute 'write' performance comes 
with a stripe_cache_size value of 4096 (for my setup). However, any 
value of stripe_cache_size above 384 really, really hurts 'check' (and 
rebuild, one can assume) performance.  Why?

--
Jon Nelson <jnelson-linux-raid@jamponi.net>

Re: stripe_cache_size and performance

[Dan Williams]

> 7. And now, the question: the best absolute 'write' performance comes
> with a stripe_cache_size value of 4096 (for my setup). However, any
> value of stripe_cache_size above 384 really, really hurts 'check' (and
> rebuild, one can assume) performance.  Why?
>
Question:
After performance goes "bad" does it go back up if you reduce the size
back down to 384?

> --
> Jon Nelson <jnelson-linux-raid@jamponi.net>

Dan

Re: stripe_cache_size and performance

[Jon Nelson]

On Mon, 25 Jun 2007, Dan Williams wrote:

> > 7. And now, the question: the best absolute 'write' performance comes
> > with a stripe_cache_size value of 4096 (for my setup). However, any
> > value of stripe_cache_size above 384 really, really hurts 'check' (and
> > rebuild, one can assume) performance.  Why?
> >
> Question:
> After performance goes "bad" does it go back up if you reduce the size
> back down to 384?

Yes, and almost instantly.

--
Jon Nelson <jnelson-linux-raid@jamponi.net>

Re: stripe_cache_size and performance

[Justin Piszcz]

On Mon, 25 Jun 2007, Jon Nelson wrote:

> On Thu, 21 Jun 2007, Jon Nelson wrote:
>
>> On Thu, 21 Jun 2007, Raz wrote:
>>
>>> What is your raid configuration ?
>>> Please note that the stripe_cache_size is acting as a bottle neck in some
>>> cases.
>
> Well, that's kind of the point of my email. I'll try to restate things,
> as my question appears to have gotten lost.
>
> 1. I have a 3x component raid5, ~314G per component. Each component
> happens to be the 4th partition of a 320G SATA drive. Each drive
> can sustain approx. 70MB/s reads/writes. Except for the first
> drive, none of the other partitions are used for anything else at this
> time. The system is nominally quiescent during these tests.
>
> 2. The kernel is 2.6.18.8-0.3-default on x86_64 (openSUSE 10.2).
>
> 3. My best sustained write performance comes with a stripe_cache_size of
> 4096. Larger than that seems to reduce performance, although only very
> slightly.
>
> 4. At values below 4096, the absolute write performance is less than the
> best, but only marginally.
>
> 5. HOWEVER, at any value *above* 512 the 'check' performance is REALLY
> BAD. By 'check' performance I mean the value displayed by /proc/mdstat
> after I issue:
>
> echo check > /sys/block/md0/md/sync_action
>
> When I say "REALLY BAD" I mean < 3MB/s.
>
> 6. Here is a short incomplete table of stripe_cache_size to 'check'
> performance:
>
> 384.... 72-73MB/s
> 512.... 72-73MB/s
> 640.... 73-74MB/s
> 768.....3-3.4MB/s
>
> And the performance stays "bad" as I increase the stripe_cache_size.
>
> 7. And now, the question: the best absolute 'write' performance comes
> with a stripe_cache_size value of 4096 (for my setup). However, any
> value of stripe_cache_size above 384 really, really hurts 'check' (and
> rebuild, one can assume) performance.  Why?
>
> --
> Jon Nelson <jnelson-linux-raid@jamponi.net>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Neil has a patch for the bad speed.

In the mean time, do this (or better to set it to 30, for instance):

# Set minimum and maximum raid rebuild speed to 60MB/s.
echo "Setting minimum and maximum resync speed to 60 MiB/s..."
echo 60000 > /sys/block/md0/md/sync_speed_min
echo 60000 > /sys/block/md0/md/sync_speed_max
echo 60000 > /sys/block/md1/md/sync_speed_min
echo 60000 > /sys/block/md1/md/sync_speed_max
echo 60000 > /sys/block/md2/md/sync_speed_min
echo 60000 > /sys/block/md2/md/sync_speed_max
echo 60000 > /sys/block/md3/md/sync_speed_min
echo 60000 > /sys/block/md3/md/sync_speed_max

Re: stripe_cache_size and performance

[Jon Nelson]

On Mon, 25 Jun 2007, Justin Piszcz wrote:

> Neil has a patch for the bad speed.

What does the patch do?

> In the mean time, do this (or better to set it to 30, for instance):
> 
> # Set minimum and maximum raid rebuild speed to 60MB/s.
> echo "Setting minimum and maximum resync speed to 60 MiB/s..."
> echo 60000 > /sys/block/md0/md/sync_speed_min
> echo 60000 > /sys/block/md0/md/sync_speed_max

sync_speed_max defaults to 200000 on this box already.
I tried a binary search of values between the default (1000) 
and 60000 which resulted in some pretty weird behavior:

at values below 26000 the rate (also confirmed via dstat output) stayed 
low.  2-3MB/s.  At 26000 and up, the value jumped more or less instantly 
to 70-74MB/s. What makes 26000 special? If I set the value to 20000 why 
do I still get 2-3MB/s actual?

--
Jon Nelson <jnelson-linux-raid@jamponi.net>

Re: stripe_cache_size and performance

[Justin Piszcz]

On Tue, 26 Jun 2007, Jon Nelson wrote:

> On Mon, 25 Jun 2007, Justin Piszcz wrote:
>
>> Neil has a patch for the bad speed.
>
> What does the patch do?
>
>> In the mean time, do this (or better to set it to 30, for instance):
>>
>> # Set minimum and maximum raid rebuild speed to 60MB/s.
>> echo "Setting minimum and maximum resync speed to 60 MiB/s..."
>> echo 60000 > /sys/block/md0/md/sync_speed_min
>> echo 60000 > /sys/block/md0/md/sync_speed_max
>
> sync_speed_max defaults to 200000 on this box already.
> I tried a binary search of values between the default (1000)
> and 60000 which resulted in some pretty weird behavior:
>
> at values below 26000 the rate (also confirmed via dstat output) stayed
> low.  2-3MB/s.  At 26000 and up, the value jumped more or less instantly
> to 70-74MB/s. What makes 26000 special? If I set the value to 20000 why
> do I still get 2-3MB/s actual?
>
> --
> Jon Nelson <jnelson-linux-raid@jamponi.net>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

You want to use sync_speed_min.

Justin.

Re: stripe_cache_size and performance

[Jon Nelson]

On Tue, 26 Jun 2007, Justin Piszcz wrote:

> 
> 
> On Tue, 26 Jun 2007, Jon Nelson wrote:
> 
> > On Mon, 25 Jun 2007, Justin Piszcz wrote:
> >
> > > Neil has a patch for the bad speed.
> >
> > What does the patch do?
> >
> > > In the mean time, do this (or better to set it to 30, for instance):
> > >
> > > # Set minimum and maximum raid rebuild speed to 60MB/s.
> > > echo "Setting minimum and maximum resync speed to 60 MiB/s..."
> > > echo 60000 > /sys/block/md0/md/sync_speed_min
> > > echo 60000 > /sys/block/md0/md/sync_speed_max
> >
> > sync_speed_max defaults to 200000 on this box already.
> > I tried a binary search of values between the default (1000)
> > and 60000 which resulted in some pretty weird behavior:
> >
> > at values below 26000 the rate (also confirmed via dstat output) stayed
> > low.  2-3MB/s.  At 26000 and up, the value jumped more or less instantly
> > to 70-74MB/s. What makes 26000 special? If I set the value to 20000 why
> > do I still get 2-3MB/s actual?


> You want to use sync_speed_min.

I forgot to say I changed the values of sync_speed_min.


--
Jon Nelson <jnelson-linux-raid@jamponi.net>

Re: stripe_cache_size and performance

[Justin Piszcz]

On Tue, 26 Jun 2007, Jon Nelson wrote:

> On Tue, 26 Jun 2007, Justin Piszcz wrote:
>
>>
>>
>> On Tue, 26 Jun 2007, Jon Nelson wrote:
>>
>>> On Mon, 25 Jun 2007, Justin Piszcz wrote:
>>>
>>>> Neil has a patch for the bad speed.
>>>
>>> What does the patch do?
>>>
>>>> In the mean time, do this (or better to set it to 30, for instance):
>>>>
>>>> # Set minimum and maximum raid rebuild speed to 60MB/s.
>>>> echo "Setting minimum and maximum resync speed to 60 MiB/s..."
>>>> echo 60000 > /sys/block/md0/md/sync_speed_min
>>>> echo 60000 > /sys/block/md0/md/sync_speed_max
>>>
>>> sync_speed_max defaults to 200000 on this box already.
>>> I tried a binary search of values between the default (1000)
>>> and 60000 which resulted in some pretty weird behavior:
>>>
>>> at values below 26000 the rate (also confirmed via dstat output) stayed
>>> low.  2-3MB/s.  At 26000 and up, the value jumped more or less instantly
>>> to 70-74MB/s. What makes 26000 special? If I set the value to 20000 why
>>> do I still get 2-3MB/s actual?
>
>
>> You want to use sync_speed_min.
>
> I forgot to say I changed the values of sync_speed_min.
>
>
> --
> Jon Nelson <jnelson-linux-raid@jamponi.net>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Sounds quite strange, what chunk size are you using for your RAID?

Re: stripe_cache_size and performance

[Jon Nelson]

On Tue, 26 Jun 2007, Justin Piszcz wrote:

> 
> 
> On Tue, 26 Jun 2007, Jon Nelson wrote:
> 
> > On Tue, 26 Jun 2007, Justin Piszcz wrote:
> >
> > >
> > >
> > > On Tue, 26 Jun 2007, Jon Nelson wrote:
> > >
> > > > On Mon, 25 Jun 2007, Justin Piszcz wrote:
> > > >
> > > > > Neil has a patch for the bad speed.
> > > >
> > > > What does the patch do?

I repeat: what does the patch do (or is this no longer applicable)?


> > > > sync_speed_max defaults to 200000 on this box already.

Altering sync_speed_min...

> > > > I tried a binary search of values between the default (1000)
> > > > and 60000 which resulted in some pretty weird behavior:
> > > >
> > > > at values below 26000 the rate (also confirmed via dstat output) stayed
> > > > low.  2-3MB/s.  At 26000 and up, the value jumped more or less instantly
> > > > to 70-74MB/s. What makes 26000 special? If I set the value to 20000 why
> > > > do I still get 2-3MB/s actual?

> Sounds quite strange, what chunk size are you using for your RAID?

The default: 64

md0 : active raid5 sdc4[2] sda4[0] sdb4[1]
      613409664 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]


--
Jon Nelson <jnelson-linux-raid@jamponi.net>

Re: stripe_cache_size and performance

[Justin Piszcz]

I repeat: what does the patch do (or is this no longer applicable)?

This was for if your stripe_cache_size was above a certain number, it 
would run at 1-3MB/s rebuild speed.  You can always force with the min 
parameter.  Forcing it you should get good speed, faster than 1-3MB/s 
anyway :)

Justin.

On Tue, 26 Jun 2007, Jon Nelson wrote:

> On Tue, 26 Jun 2007, Justin Piszcz wrote:
>
>>
>>
>> On Tue, 26 Jun 2007, Jon Nelson wrote:
>>
>>> On Tue, 26 Jun 2007, Justin Piszcz wrote:
>>>
>>>>
>>>>
>>>> On Tue, 26 Jun 2007, Jon Nelson wrote:
>>>>
>>>>> On Mon, 25 Jun 2007, Justin Piszcz wrote:
>>>>>
>>>>>> Neil has a patch for the bad speed.
>>>>>
>>>>> What does the patch do?
>
> I repeat: what does the patch do (or is this no longer applicable)?
>
>
>>>>> sync_speed_max defaults to 200000 on this box already.
>
> Altering sync_speed_min...
>
>>>>> I tried a binary search of values between the default (1000)
>>>>> and 60000 which resulted in some pretty weird behavior:
>>>>>
>>>>> at values below 26000 the rate (also confirmed via dstat output) stayed
>>>>> low.  2-3MB/s.  At 26000 and up, the value jumped more or less instantly
>>>>> to 70-74MB/s. What makes 26000 special? If I set the value to 20000 why
>>>>> do I still get 2-3MB/s actual?
>
>> Sounds quite strange, what chunk size are you using for your RAID?
>
> The default: 64
>
> md0 : active raid5 sdc4[2] sda4[0] sdb4[1]
>      613409664 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
>
>
> --
> Jon Nelson <jnelson-linux-raid@jamponi.net>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>