raid 1 vs raid 0+1

raid 1 vs raid 0+1

[Antonello Piemonte]

Hello

I have a server wich I would like to set up with
mirroring for some data protections (against disk failure).
the machine is supposed to deal mostly with writing of
lots of small files (2kb perhaps 4kb each) to disk (well, array)
and the goal would be to be able to write at least few hundreds 
files per seconds (!).

the question is: for performance, is it better a raid 1 or a 
raid 0+1 configuration? is the above load (number of files written
per second) a realistic goal to attain with a SCSI based uniprocessor PIII 
800MHZ with ext3 file system (this I will tackle separately, perhaps
will use ext2 to increase performance) and 1 Gig of RAM? 

thanks in advance for any opinion/comment!
regards
antonello

Re: raid 1 vs raid 0+1

[Jakob Oestergaard]

On Tue, Oct 29, 2002 at 01:29:23PM +0100, Antonello Piemonte wrote:
> Hello
> 
> I have a server wich I would like to set up with
> mirroring for some data protections (against disk failure).
> the machine is supposed to deal mostly with writing of
> lots of small files (2kb perhaps 4kb each) to disk (well, array)
> and the goal would be to be able to write at least few hundreds 
> files per seconds (!).
> 
> the question is: for performance, is it better a raid 1 or a 
> raid 0+1 configuration? is the above load (number of files written
> per second) a realistic goal to attain with a SCSI based uniprocessor PIII 
> 800MHZ with ext3 file system (this I will tackle separately, perhaps
> will use ext2 to increase performance) and 1 Gig of RAM? 

On a dual PIII-550 with 512 MB of memory, ext3, and a RAID-0+1 (four 40G
7200rpm IBM IDE Deathstar disks, 64k chunk-size on the RAID-0), I get:

$ time for i in {0,1,2,3,4,5,6,7,8,9}; do mkdir $i; for j in
{0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9}; do dd if=/dev/zero of=$j/$i
bs=1k count=4; done done

real    0m5.024s

So, 5 seconds for writing one thousand 4 kb files *sequentially*.

Note that I only put 100 files in each directory - if I put 1000 files
in one directory, performance would degrade (more significantly when the
number is, say, 10000).

You may want to experiment with ext3 journalling options - you may see
better performance on data=journal mode, if you write the files in
bursts (with some longer pauses in between).   I cannot give you any
certain advice here, other than to experiment.

RAID-0 will probably allow you to scale better, but I'm really not sure
how the performance on this rather perculiar workload changes as you add
disks - in any case it will be *highly* filesystem dependent.  You
should definitely also try out ReiserFS and eventually JFS, XFS, and
perhaps even one of the FAT variants (yes, FAT is actually *very* fast
for some very particular workloads, because it is so primitive (eg. it
gets less in the way) - at least this used to be true, but I do not know
if it is still so, and I'm not sure about your workload either).

So in short; Everything I said here except for the 5 second benchmark is
guessing...  Now you go measure  ;)

Please, if you do decide to measure, do post a summary here to the list.
I'm sure people will find it interesting, and it will appear in the
archives for the next person with the same problem to find.

Cheers,

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid 1 vs raid 0+1

[Stephen Lee]

On Tue, 2002-10-29 at 05:14, Jakob Oestergaard wrote:
> On Tue, Oct 29, 2002 at 01:29:23PM +0100, Antonello Piemonte wrote:
> > the question is: for performance, is it better a raid 1 or a 
> > raid 0+1 configuration? is the above load (number of files written
> > per second) a realistic goal to attain with a SCSI based uniprocessor PIII 
> > 800MHZ with ext3 file system (this I will tackle separately, perhaps
> > will use ext2 to increase performance) and 1 Gig of RAM? 
> 
> On a dual PIII-550 with 512 MB of memory, ext3, and a RAID-0+1 (four 40G
> 7200rpm IBM IDE Deathstar disks, 64k chunk-size on the RAID-0), I get:
> 
> $ time for i in {0,1,2,3,4,5,6,7,8,9}; do mkdir $i; for j in
> {0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9}; do dd if=/dev/zero of=$j/$i
> bs=1k count=4; done done
> 
> real    0m5.024s
> 

My systems are very different than what you describe above but here are
some data points for reference. BTW, should that not be "of=$i/$j"?  

For a Dual P3/1.4Gig with 3x10000rpm SCSI drives running software raid5
and ext3 on a 2.4.18 (Redhat 7.3 2.4.18-17.7) kernel:

real	0m2.217s

For a P3/1Gig with 2x10000rpm SCSI drive running software raid1 and ext3
on a stock 2.4.18 kernel:

real	0m3.349s

Stephen

Re: raid 1 vs raid 0+1

[Antonello Piemonte]

On Tuesday 29 October 2002 14:14, Jakob Oestergaard wrote:
> On a dual PIII-550 with 512 MB of memory, ext3, and a RAID-0+1 (four 40G
> 7200rpm IBM IDE Deathstar disks, 64k chunk-size on the RAID-0), I get:
>
> $ time for i in {0,1,2,3,4,5,6,7,8,9}; do mkdir $i; for j in
> {0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9}; do dd if=/dev/zero of=$j/$i
> bs=1k count=4; done done
>
> real    0m5.024s
>
> So, 5 seconds for writing one thousand 4 kb files *sequentially*.
>
> Note that I only put 100 files in each directory - if I put 1000 files
> in one directory, performance would degrade (more significantly when the
> number is, say, 10000).

Dear Jakob,

for a start I have set my "baseline" measurement: using a 10000 rpm IBM 
Ultrastar with Adaptec 29160 Ultra160 SCSI adapter
with 512Mb RAM and PIII-800 I can create 1000 4kb files
in one directory (this case is what is going to happen in the real
application) in abou 8secs using this:
--
#!/bin/sh

i=1
while [ $i -le 1000 ]
do
        echo $i
        dd if=/dev/zero of=file_$i bs=1k count=4
        i=`expr $i + 1`
done
exit 0
--

$ time sh write.sh
...
 real    0m8.388s
user    0m4.037s
sys     0m3.834s

writing 10.000 files gives instead
$ time sh write.sh
real    1m46.499s
user    0m40.096s
sys     1m0.400s

so a 10-folds increase of number of files gives about
a 12-fold increase in "real" time ... 
I will play next with journaling mode (using data=ordered now)
and post outcome here: thanks for your comments so far !
antonello

Re: raid 1 vs raid 0+1

[Adam Luter]

On Tue, Oct 29, 2002 at 07:16:26AM -0800, Stephen Lee wrote:
> > real    0m5.024s
> > 
> 
> My systems are very different than what you describe above but here are
> some data points for reference. BTW, should that not be "of=$i/$j"?  
> 
> For a Dual P3/1.4Gig with 3x10000rpm SCSI drives running software raid5
> and ext3 on a 2.4.18 (Redhat 7.3 2.4.18-17.7) kernel:
> 
> real	0m2.217s
> 
> For a P3/1Gig with 2x10000rpm SCSI drive running software raid1 and ext3
> on a stock 2.4.18 kernel:
> 
> real	0m3.349s

#!/bin/sh
time for i in {0,1,2,3,4,5,6,7,8,9}
do
mkdir $i
 for j in {0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9}
 do
  dd if=/dev/zero of=$i/$j bs=1k count=4 2>/dev/null
 done
done

If everyone else is going to start squirting testosterone around, then
I'll add my share:

The above code, made an average time of about 3.7s .  This is with
raid-5 (soft) 2.4.20pre8 with 4 IDE drives (Ibm 120 gig 7200rpm)
running at/on a Promise ATA/100 card (so 2 drives per bus).  Also, the
filesystem is reiserfs, and running at medium-light load.

Changing the above code to:

#!/bin/sh
time for i in {0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9}
do
 dd if=/dev/zero of=$i bs=1k count=4 2>/dev/null
done

(so that it writes them all to one directory), made the same time,
3.7s on average.

In fact, adding another 0..9 onto that loop only changed the average
time to 44 seconds.

For comparison the same tests on my system drive, a Maxtor IDE 40 gig
7200rpm drive connected to the onboard IDE controller (and also
reiserfs), resulted in 3.3 seconds, and 35 seconds respectively.

My conclusions are:  IDE is damn fast and reiser kicks ass with small
files.

Of course alternate conclusions are that my raid-5 array suffers from
using two channels per drive (I know -- but when I put two promise
cards they or linux has DMA problems) and that I'm still behind the
pack when you throw 10krpm at me :) (2.2 seconds is nice!).

But, I'll never be rich enough for SCSI.

Back to the grind...

-Gryn (Adam Luter)

Re: raid 1 vs raid 0+1

[Adam Luter]

On Tue, Oct 29, 2002 at 04:27:32PM +0100, Antonello Piemonte wrote:
> for a start I have set my "baseline" measurement: using a 10000 rpm IBM 
> Ultrastar with Adaptec 29160 Ultra160 SCSI adapter
> with 512Mb RAM and PIII-800 I can create 1000 4kb files
> in one directory (this case is what is going to happen in the real
> application) in abou 8secs using this:
> --
> #!/bin/sh
> 
> i=1
> while [ $i -le 1000 ]
> do
>         echo $i
>         dd if=/dev/zero of=file_$i bs=1k count=4
>         i=`expr $i + 1`
> done
> exit 0
> --
> 
> $ time sh write.sh
> ...
>  real    0m8.388s
> user    0m4.037s
> sys     0m3.834s
> 
> writing 10.000 files gives instead
> $ time sh write.sh
> real    1m46.499s
> user    0m40.096s
> sys     1m0.400s
> 
> so a 10-folds increase of number of files gives about
> a 12-fold increase in "real" time ... 
> I will play next with journaling mode (using data=ordered now)
> and post outcome here: thanks for your comments so far !
> antonello

Get rid of while loop in favor of the for loop, your spending alot of
time with the shell!  (It's bad enough this test involves spawning
10,000 copies of dd!).  Also don't do an echo, and pipe dd's output to
dev/null.  You'll increase your numbers -alot- this way.  (Especially
if you are in a fb-console or xterm, I/O to the screen costs -alot-
then).

-Gryn (Adam Luter)

Re: raid 1 vs raid 0+1

[Stephen Lee]

On Tue, 2002-10-29 at 07:39, Adam Luter wrote:
> On Tue, Oct 29, 2002 at 07:16:26AM -0800, Stephen Lee wrote:
> Of course alternate conclusions are that my raid-5 array suffers from
> using two channels per drive (I know -- but when I put two promise
> cards they or linux has DMA problems) and that I'm still behind the
> pack when you throw 10krpm at me :) (2.2 seconds is nice!).

It's around 2.0 seconds if I direct stdout to /dev/null ;-)

I also have a system using a Promise raid controller with 2x7200rpm ide
drives in a raid1 configuration (AMD K7 1.4Gig, ext2, RH 2.4.2 kernel,
Promises proprietary driver) and the same test produced a speed of 2.6
seconds. 

Stephen

Re: raid 1 vs raid 0+1

[Roy Sigurd Karlsbakk]

<snip>
> Note that I only put 100 files in each directory - if I put 1000 files
> in one directory, performance would degrade (more significantly when the
> number is, say, 10000).
</snip>

this is not, however, the fact with other filesystems such as JFS, XFS and 
ReiserFS

-- 
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356

Computers are like air conditioners.
They stop working when you open Windows.

Re: raid 1 vs raid 0+1

[Trent Piepho]

> If everyone else is going to start squirting testosterone around, then
> I'll add my share:
> 
> The above code, made an average time of about 3.7s .  This is with
> raid-5 (soft) 2.4.20pre8 with 4 IDE drives (Ibm 120 gig 7200rpm)
> running at/on a Promise ATA/100 card (so 2 drives per bus).  Also, the
> filesystem is reiserfs, and running at medium-light load.

Oh me too!


dual P3 1Ghz, 1GB ram, 2.2.20RAID, ext2, Mylex extremeraid 2000 hardware
RAID5, six IBM 36LZX 10k RPM 36GB U160 SCSI drives.

1.89 seconds

same system, but mylex hardware RAID0 instead of raid5, still 1.89 seconds.

same system, but with a 3ware 7850 with six maxtor 160GB drives....  1.89
seconds.

same system, but with two IBM 18LZX 18GB 10k RPM SCSI drives on an onboard
adaptec U160 controller and linux software RAID0...  1.88 seconds.

It seems disk+controller doesn't matter one bit for this test.

Now a new system:

dual AthlonXP1800, 1GB ram, 2.4.16, ext3, 3ware 7410 RAID0, two maxtor 160GB
drives...  2.38 seconds

Conclusion:  either ext3 or the 2.4 kernel are slow, as the second machine as
has PC2100 RAM vs PC133 and 1.5Ghz athlons vs 1Ghz pentiumIIIs.  It should
be faster, not slower.