poor performance with raid1 / raid0

poor performance with raid1 / raid0

[Per Andreas Buer]

Hi.

I have two 70GB drives and a 140GB disk set up in RAID1 across the 140GB
drive and a RAID0 built over the 70GB disks. Both the RAID0 and the big
disk are quite fast (transfers ~ 110MB/s and 70MB/s) - but the mirror is
dead slow.

Resync lokes like this: 

md1 : active raid1 md11[2] sdd1[0]
      497856 blocks [2/1] [U_]
      [=========>...........]  recovery = 48.0% (240188/497856)
      finish=15.6min speed=272K/sec

If this configuration unwise in some respect?

Plain 2.4.20-kernel with some fixes (ext3 and ptrace bugs). Adaptec
7899P (rev 08).~

# cat /proc/scsi/scsi 
Attached devices: 
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: IBM      Model: DDYS-T18350M     Rev: SA2A
  Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 01 Lun: 00
  Vendor: FUJITSU  Model: MAN3735MC        Rev: 0109
  Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 02 Lun: 00
  Vendor: FUJITSU  Model: MAN3735MC        Rev: 0109
  Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 03 Lun: 00
  Vendor: SEAGATE  Model: ST3146807LC      Rev: 0004
  Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 06 Lun: 00
  Vendor: SUPER    Model: GEM354 REV001    Rev: 1.04
  Type:   Processor                        ANSI SCSI revision: 02


-- 
There are only 10 different kinds of people in the world, 
those who understand binary, and those who don't.

Re: poor performance with raid1 / raid0

[Andrew Rechenberg]

Check /proc/sys/dev/raid/speed_limit_[min|max].  If they are set too low
then your re-mirror will behave as such.  To change just echo a new
value:

echo 30000 > /proc/sys/dev/raid/speed_limit_min
echo 50000 > /proc/sys/dev/raid/speed_limit_max

Max is 100000 I believe.

Hope this helps,
Andy.

On Wed, 2003-06-18 at 08:32, Per Andreas Buer wrote:
> Hi.
> 
> I have two 70GB drives and a 140GB disk set up in RAID1 across the 140GB
> drive and a RAID0 built over the 70GB disks. Both the RAID0 and the big
> disk are quite fast (transfers ~ 110MB/s and 70MB/s) - but the mirror is
> dead slow.
> 
> Resync lokes like this: 
> 
> md1 : active raid1 md11[2] sdd1[0]
>       497856 blocks [2/1] [U_]
>       [=========>...........]  recovery = 48.0% (240188/497856)
>       finish=15.6min speed=272K/sec
> 
> If this configuration unwise in some respect?
> 
> Plain 2.4.20-kernel with some fixes (ext3 and ptrace bugs). Adaptec
> 7899P (rev 08).~
> 
> # cat /proc/scsi/scsi 
> Attached devices: 
> Host: scsi0 Channel: 00 Id: 00 Lun: 00
>   Vendor: IBM      Model: DDYS-T18350M     Rev: SA2A
>   Type:   Direct-Access                    ANSI SCSI revision: 03
> Host: scsi0 Channel: 00 Id: 01 Lun: 00
>   Vendor: FUJITSU  Model: MAN3735MC        Rev: 0109
>   Type:   Direct-Access                    ANSI SCSI revision: 03
> Host: scsi0 Channel: 00 Id: 02 Lun: 00
>   Vendor: FUJITSU  Model: MAN3735MC        Rev: 0109
>   Type:   Direct-Access                    ANSI SCSI revision: 03
> Host: scsi0 Channel: 00 Id: 03 Lun: 00
>   Vendor: SEAGATE  Model: ST3146807LC      Rev: 0004
>   Type:   Direct-Access                    ANSI SCSI revision: 03
> Host: scsi0 Channel: 00 Id: 06 Lun: 00
>   Vendor: SUPER    Model: GEM354 REV001    Rev: 1.04
>   Type:   Processor                        ANSI SCSI revision: 02
-- 
Andrew Rechenberg <arechenberg@shermfin.com>
Infrastructure Team, Sherman Financial Group

Re: poor performance with raid1 / raid0

[Per Andreas Buer]

Andrew Rechenberg <arechenberg@shermfin.com> writes:

> Check /proc/sys/dev/raid/speed_limit_[min|max].  If they are set too low
> then your re-mirror will behave as such.  To change just echo a new
> value:
> 
> echo 30000 > /proc/sys/dev/raid/speed_limit_min
> echo 50000 > /proc/sys/dev/raid/speed_limit_max
> 

I did so - and now:

md6 : active raid1 md16[2] sdd6[0]
      117185984 blocks [2/1] [U_]
      [>....................]  recovery =  0.1% (229120/117185984)
finish=59.5min speed=32731K/sec

:D

Thanks.

-- 
There are only 10 different kinds of people in the world, 
those who understand binary, and those who don't.

Re: poor performance with raid1 / raid0

[Paul Clements]

Per Andreas Buer wrote:
> 
> Andrew Rechenberg <arechenberg@shermfin.com> writes:
> 
> > Check /proc/sys/dev/raid/speed_limit_[min|max].  If they are set too low
> > then your re-mirror will behave as such.  To change just echo a new
> > value:
> >
> > echo 30000 > /proc/sys/dev/raid/speed_limit_min
> > echo 50000 > /proc/sys/dev/raid/speed_limit_max
> >
> 
> I did so - and now:
> 
> md6 : active raid1 md16[2] sdd6[0]
>       117185984 blocks [2/1] [U_]
>       [>....................]  recovery =  0.1% (229120/117185984)
> finish=59.5min speed=32731K/sec

That's probably because you just lowered the values. The maximum is
normally 100000. To check what the default values are, reboot and do:

cat /proc/sys/dev/raid/speed_limit_*

Then echo a _larger_ number into those files.

There is no absolute maximum value (except maybe 2^32), so don't be
afraid to raise the values to 10 or 100 times what they were. That
should ensure that your resync speed is close to its theoretical
maximum.

--
Paul

Re: poor performance with raid1 / raid0

[Farkas Levente]

Paul Clements wrote:
> Per Andreas Buer wrote:
> 
>>Andrew Rechenberg <arechenberg@shermfin.com> writes:
>>
>>
>>>Check /proc/sys/dev/raid/speed_limit_[min|max].  If they are set too low
>>>then your re-mirror will behave as such.  To change just echo a new
>>>value:
>>>
>>>echo 30000 > /proc/sys/dev/raid/speed_limit_min
>>>echo 50000 > /proc/sys/dev/raid/speed_limit_max
>>>
>>
>>I did so - and now:
>>
>>md6 : active raid1 md16[2] sdd6[0]
>>      117185984 blocks [2/1] [U_]
>>      [>....................]  recovery =  0.1% (229120/117185984)
>>finish=59.5min speed=32731K/sec
> 
> 
> That's probably because you just lowered the values. The maximum is
> normally 100000. To check what the default values are, reboot and do:
> 
> cat /proc/sys/dev/raid/speed_limit_*
> 
> Then echo a _larger_ number into those files.
> 
> There is no absolute maximum value (except maybe 2^32), so don't be
> afraid to raise the values to 10 or 100 times what they were. That
> should ensure that your resync speed is close to its theoretical
> maximum.

can someone tell me what these numbers means?
of course my goal (as everyones) to achieve the maximum possible 
performance.
can I do any harm if I set these numbers to high?
it no, why not set it bu default?

thanks in advance.

-- 
   Levente                               "Si vis pacem para bellum!"

Re: poor performance with raid1 / raid0

[Paul Clements]

Farkas Levente wrote:
> 
> Paul Clements wrote:
> > Per Andreas Buer wrote:
> >
> >>Andrew Rechenberg <arechenberg@shermfin.com> writes:
> >>
> >>
> >>>Check /proc/sys/dev/raid/speed_limit_[min|max].  If they are set too low
> >>>then your re-mirror will behave as such.  To change just echo a new
> >>>value:
> >>>
> >>>echo 30000 > /proc/sys/dev/raid/speed_limit_min
> >>>echo 50000 > /proc/sys/dev/raid/speed_limit_max
> >>>
> >>
> >>I did so - and now:
> >>
> >>md6 : active raid1 md16[2] sdd6[0]
> >>      117185984 blocks [2/1] [U_]
> >>      [>....................]  recovery =  0.1% (229120/117185984)
> >>finish=59.5min speed=32731K/sec
> >
> >
> > That's probably because you just lowered the values. The maximum is
> > normally 100000. To check what the default values are, reboot and do:
> >
> > cat /proc/sys/dev/raid/speed_limit_*
> >
> > Then echo a _larger_ number into those files.
> >
> > There is no absolute maximum value (except maybe 2^32), so don't be
> > afraid to raise the values to 10 or 100 times what they were. That
> > should ensure that your resync speed is close to its theoretical
> > maximum.
> 
> can someone tell me what these numbers means?

I think I posted something a week or two ago that explained what these
are...


> of course my goal (as everyones) to achieve the maximum possible
> performance.

Well, this is a trade-off, that's precisely why these values are
tunables. 

These tunables govern the _resync_ speed of a raid1/5 array. They do not
control the regular I/O throughput. So what you generally want to do is
to set these values as high as possible without there being an impact on
normal I/O (or for that matter, system responsiveness on the whole)
during a resync.


> can I do any harm if I set these numbers to high?

Yes, the system might become sluggish or your regular I/O throughput to
the device might become very slow during resyncs if these values are too
high.

--
Paul

Re: poor performance with raid1 / raid0

[Ross Vandegrift]

On Wed, Jun 18, 2003 at 04:52:12PM +0200, Farkas Levente wrote:
> can someone tell me what these numbers means?

They're a max speed in K/sec.

> of course my goal (as everyones) to achieve the maximum possible 
> performance.
> can I do any harm if I set these numbers to high?

Nope.

> it no, why not set it bu default?

By default, the RAID code uses max bandwidth and backs off when it sees
that the device is busy.  However, you're running RAID1 over RAID0 - so
to the RAID1 layer, the lower layer is busy - namely, it's being used by
the RAID0 code.  You can avoid this problem by running RAID0 over RAID1
instead.  But since your disk configuration makes this impossible (you'd
need matching disks for the mirrors), just modify your boot scripts to max
out speed_limit*.

-- 
Ross Vandegrift
ross@willow.seitz.com

A Pope has a Water Cannon.                               It is a Water Cannon.
He fires Holy-Water from it.                        It is a Holy-Water Cannon.
He Blesses it.                                 It is a Holy Holy-Water Cannon.
He Blesses the Hell out of it.          It is a Wholly Holy Holy-Water Cannon.
He has it pierced.                It is a Holey Wholly Holy Holy-Water Cannon.
He makes it official.       It is a Canon Holey Wholly Holy Holy-Water Cannon.
Batman and Robin arrive.                                       He shoots them.