Linux i/o tweaking

Linux i/o tweaking

[Roy Sigurd Karlsbakk]

Hi all

After three days at Compaq's lab in Oslo, testing their medium-level
servers and storage systems with Linux, I've come to some sort of
conclusions, although these may be wrong. I also have come over a few
problems that I couln't find a good solution to.

 * When running RAID from a Compaq Smart 5302/64 controller, software
RAID-5 is (slightly - ~15%) faster (on JBOD - each disk is configured as
a RAID-0 device with max - 256kB - stripe size) than the
hardware/controller based RAID-5. Both CPUs (1266MHz/512kB cache) are
maxed out by reading from software RAID-5 (???), giving me >= 107MB/s on
two SCSI-3 buses with six disks on each bus.

 * Even though I can get up to 25 MB/s from each disk, I can't get more
than 107 MB/s on the whole bunch (12 drives). It doesn't help much to do
RAID-0 either. Don't understand anything ...

Thanks for all help.

roy

--
Roy Sigurd Karlsbakk, MCSE, MCNE, CLS, LCA

Computers are like air conditioners.
They stop working when you open Windows.

Re: Linux i/o tweaking

[Jens Axboe]

On Thu, Nov 15 2001, Roy Sigurd Karlsbakk wrote:
> Hi all
> 
> After three days at Compaq's lab in Oslo, testing their medium-level
> servers and storage systems with Linux, I've come to some sort of
> conclusions, although these may be wrong. I also have come over a few
> problems that I couln't find a good solution to.
> 
>  * When running RAID from a Compaq Smart 5302/64 controller, software
> RAID-5 is (slightly - ~15%) faster (on JBOD - each disk is configured as
> a RAID-0 device with max - 256kB - stripe size) than the
> hardware/controller based RAID-5. Both CPUs (1266MHz/512kB cache) are
> maxed out by reading from software RAID-5 (???), giving me >= 107MB/s on
> two SCSI-3 buses with six disks on each bus.
> 
>  * Even though I can get up to 25 MB/s from each disk, I can't get more
> than 107 MB/s on the whole bunch (12 drives). It doesn't help much to do
> RAID-0 either. Don't understand anything ...

Could you please try and profile where the time is spent? Boot with
profile=2, and then do

# readprofile -r
# do I/O testing
# readprofile | sort -nr

-- 
Jens Axboe

Re: Linux i/o tweaking

[Roy Sigurd Karlsbakk]

> Could you please try and profile where the time is spent? Boot with
> profile=2, and then do
>
> # readprofile -r
> # do I/O testing
> # readprofile | sort -nr

I will.

However ... Is it normal for a server to max out 2xPIII 1266MHz CPUs by
reading from software RAID-5???

--
Roy Sigurd Karlsbakk, MCSE, MCNE, CLS, LCA

Computers are like air conditioners.
They stop working when you open Windows.

Re: Linux i/o tweaking

[Jakob Østergaard]

On Thu, Nov 15, 2001 at 05:32:29PM +0100, Roy Sigurd Karlsbakk wrote:
> > Could you please try and profile where the time is spent? Boot with
> > profile=2, and then do
> >
> > # readprofile -r
> > # do I/O testing
> > # readprofile | sort -nr
> 
> I will.
> 
> However ... Is it normal for a server to max out 2xPIII 1266MHz CPUs by
> reading from software RAID-5???

Certainly not.

Well, if you down-scale the experiment it's not.  Reading 10.7 MB/sec
will not consume 10% of your two processors.

But queue systems are evil   ;)   I look forward to seeing the profile.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: Linux i/o tweaking

[Roy Sigurd Karlsbakk]

> Certainly not.
>
> Well, if you down-scale the experiment it's not.  Reading 10.7 MB/sec
> will not consume 10% of your two processors.
>
> But queue systems are evil   ;)   I look forward to seeing the profile.

I'll post the pre- and post-profile here tomorrow...

roy
--
Roy Sigurd Karlsbakk, MCSE, MCNE, CLS, LCA

Computers are like air conditioners.
They stop working when you open Windows.

Re: Linux i/o tweaking

[Joel Jaeggli]

I guess the question I'd ask would be what kinda number do you expect from 
the raid 5 stripe. 107MB/s sounds like a reasonable number from a two 
channel u160 controller... 

my previous best is around 89MB/s with 4 cheetah 15K 18gb drives raid/0

On Thu, 15 Nov 2001, Roy Sigurd Karlsbakk 
wrote:

> Hi all
> 
> After three days at Compaq's lab in Oslo, testing their medium-level
> servers and storage systems with Linux, I've come to some sort of
> conclusions, although these may be wrong. I also have come over a few
> problems that I couln't find a good solution to.
> 
>  * When running RAID from a Compaq Smart 5302/64 controller, software
> RAID-5 is (slightly - ~15%) faster (on JBOD - each disk is configured as
> a RAID-0 device with max - 256kB - stripe size) than the
> hardware/controller based RAID-5. Both CPUs (1266MHz/512kB cache) are
> maxed out by reading from software RAID-5 (???), giving me >= 107MB/s on
> two SCSI-3 buses with six disks on each bus.
> 
>  * Even though I can get up to 25 MB/s from each disk, I can't get more
> than 107 MB/s on the whole bunch (12 drives). It doesn't help much to do
> RAID-0 either. Don't understand anything ...
> 
> Thanks for all help.
> 
> roy
> 
> --
> Roy Sigurd Karlsbakk, MCSE, MCNE, CLS, LCA
> 
> Computers are like air conditioners.
> They stop working when you open Windows.
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
-------------------------------------------------------------------------- 
Joel Jaeggli				       joelja@darkwing.uoregon.edu    
Academic User Services			     consult@gladstone.uoregon.edu
     PGP Key Fingerprint: 1DE9 8FCA 51FB 4195 B42A 9C32 A30D 121E
--------------------------------------------------------------------------
It is clear that the arm of criticism cannot replace the criticism of
arms.  Karl Marx -- Introduction to the critique of Hegel's Philosophy of
the right, 1843.

Re: Linux i/o tweaking

[Roy Sigurd Karlsbakk]

> I guess the question I'd ask would be what kinda number do you expect from
> the raid 5 stripe. 107MB/s sounds like a reasonable number from a two
> channel u160 controller...

I really don't know. My 'logical' mind told me I could get close-to
speed-per-disk * number-of-disks, but it might be wrong.

> my previous best is around 89MB/s with 4 cheetah 15K 18gb drives raid/0

I had 5 drives per SCSI bus and the controller was sitting alone on a
66MHz/64bit PCI hose.  I really can't see where the bottleneck is!

roy

--
Roy Sigurd Karlsbakk, MCSE, MCNE, CLS, LCA

Computers are like air conditioners.
They stop working when you open Windows.

Re: Linux i/o tweaking

[Lionel Bouton]

Roy Sigurd Karlsbakk wrote:

>>I guess the question I'd ask would be what kinda number do you expect from
>>the raid 5 stripe. 107MB/s sounds like a reasonable number from a two
>>channel u160 controller...
>>
>
>I really don't know. My 'logical' mind told me I could get close-to
>speed-per-disk * number-of-disks, but it might be wrong.
>
>
>>my previous best is around 89MB/s with 4 cheetah 15K 18gb drives raid/0
>>
>
>I had 5 drives per SCSI bus and the controller was sitting alone on a
>66MHz/64bit PCI hose.  I really can't see where the bottleneck is!
>
# Hdparm
Do you use hdparm for your test ?

What gives hdparm -T (timing buffer-cache reading) ?
I believe it's kind of a hard ceiling...

<jealousy>
I use single channel PC66 sdram on a BX chipset here and have only 60 
MB/s from buffer-cache on a SMP 2.4.13-ac2 kernel. Seeing a complain for 
107 MB/s from a disk subsystem makes me feel somehow frustrated :-)
</jealousy>

# Hardware
Are all drives recognised as 160 MB/s capable devices ? SCSI is so 
touchy, you never know...
Look in /var/log/dmesg for the scsi adapter logs.

# Software
What's your kernel version. Any patch applied ?

# Looking for a clue
You may try to profile the kernel on arrays with different numbers of 
disks in raid 0 (to keep cpu out of equation at first).
It could help to find the bandwidth or the number of disks where the 
actual speedup begins to fall behind the theoritical speedup and then 
have the profiles at hand.

-- 
 Lionel Bouton

-
"I wanted to be free, so I opensourced my whole DNA code" Gyver, 1999.