Re: [reiserfs-list] ReiserFS on RAID5 Linux-2.4 - speed problem

Re: [reiserfs-list] ReiserFS on RAID5 Linux-2.4 - speed problem

[Edward Shushkin]

Rene Rebe wrote:
> 
> (Repost - since my first mail has still not arrived ...)
> 
> Hi - I need some speed tips for using ReiserFS on an RAID5 Linux-soft-raid device.
> 
> Config: AMD K6-2 350
>         3x IBM 40GB IDE discs (connected as master each to a single IDE channel)
>         Linux-2.4.1x (currently .16)
>         the discs are combined via Linux-software-RAID5
> 
> I run ReiserFS on a RAID5 (of 3 IDE disks) using the latest 2.4.(e.g.16)
> kernel for weeks. It works well except that is is painfully slow. In todays tests
> I got arround 3MB/s from ReiserFS (dd if=/home/database/some-video.avi ...) - wich
> matches exatly what I get over NFS for normal development usage, too. The IDE or
> RAID setup doesn't seem to be a problem because I get 33MB/s when I read the
> /dev/md/0 directly (dd if=/dev/md/0 ...)!

Thats odd!
I don't have three various ide-controllers to reproduce your configuration, 
but for the raid5 of 3 various scsi disks i have that the times of dd readings for 
if= /dev/sda, /dev/md0, /mnt/file_on_reiserfs_on_md0 relates approximately as 
1.5 : 1 : 1   
So the performance doesn't decrease..
What is about your speed for (dd if=/dev/hdx...)??
Can you measure this for ext2 on md0?

Thanks,
Edward.

Re: [reiserfs-list] ReiserFS on RAID5 Linux-2.4 - speed problem

[Rene Rebe]

Thanks for the reply!

On Thu, 29 Nov 2001 17:11:51 +0000
Edward Shushkin <edward@namesys.com> wrote:

> Thats odd!
> I don't have three various ide-controllers to reproduce your configuration,

You only need one controller in additoin to your on-board one ... 2x 2 channels
-> 4 IDE Master possible ...

> but for the raid5 of 3 various scsi disks i have that the times of dd readings for 
> if= /dev/sda, /dev/md0, /mnt/file_on_reiserfs_on_md0 relates approximately as 
> 1.5 : 1 : 1   
> So the performance doesn't decrease..
> What is about your speed for (dd if=/dev/hdx...)??

Hardware: AMD K6-2 350Mhz; 128MB RAM; Aladin5 based Gigabyte board; additinal
Promise ULTRA ATA-100 controller; vanilla 2.4.16 kernel; [glibc-2.1.3]

I tuned the kernel a bit by:
echo 1023 > /proc/sys/vm/max-readahead
(without this some short test were much worser ...)

	RAID-disk I	RAID-disk II	RAID-disk III
	host2 ch0 ma	host2 ch1 ma	host0 ch1 ma
1. hdparm -tT
	60.66 MB/sec	60.66 MB/sec	60.66 MB/sec
	20.13 MB/sec	23.62 MB/sec	15.92 MB/sec (UDMA mode II only)

2. dd if=x of=/dev/null bs=1024 count=100000 (avg. of 3 measurements)
	30.52 MB/sec	30.54 MB/sec	30.6 MB/sec (??? UDMA2 ... ?)
# ok too much is cached by LInux ... again with 400 MB:

3. dd if=x of=/dev/null bs=1024 count=400000 (avg. of 3 measurements)
	30.92 MB/sec	28.32 MB/sec	29.32 MB/sec

4. time dd if=/dev/md/0 of=/dev/null bs=1024 count=400000 (avg. of 3 measurements)
	34.567 MB/sec

4. time dd if=/home/everybody/video/movies/Matrix\ 1-2.avi of=/dev/null bs=1024 count=200000
(home is the ReiserFS on RAID5 filesystem ...) (avg. of 3 measurements)
	8.785 MB/sec

5. time dd if=/home/everybody/video/movies/Matrix\ 1-2.avi of=/dev/null bs=1024 count=400000
(home is the ReiserFS on RAID5 filesystem ...) (avg. of 3 measurements)
	9.306 MB/ses

So I have ide/discX md/0 file-from-reiserfs:
	3.2 : 3.71 : 1

Over NFS I get 7.1 MB/sec - this is MUCH better than yesterday - so the "echo 1023 >
 /proc/sys/vm/max-readahead" must have done it ... - but more would still be better ;-)
(I'll write a benchmark script and redo the test without this tuning tonight ...)

Notes:
1. The uncontinous results may be caused by a compile workstation that currently compiles
my distribution and accesed some source from time to time (not that much ...).

2. I allready "tuned" the IDE by the additional IDE controller. At the beginning
all discs were connected to the on-board Ali - which gave much worser results. So I added
the Promisse one so that each disc is a Master without Slave devices (I thought the
slowdown may be caused by the concurrend IDE-bus accesses ...)

Question: Are there some block-sizes / allignments issues I should take care when using
ReiserFS on RAID5 ?? - The current setup is a straightforward first-try with default
parameters for all tools (mkraid / mkreiserfs ... - I found no hints on the web).

> Can you measure this for ext2 on md0?

No - sorry. It is a "production" machine. I could backup all the stuff and reinitialize
RAID and reiserfs in another way when I know what to do better (except using SCSI ;-)
or what I did wrong - but I can not play arround with the data all the time ...

> Thanks,
> Edward.


k33p h4ck1n6
  René

-- 
René Rebe (Registered Linux user: #127875 <http://counter.li.org>)

eMail:    rene.rebe@gmx.net
          rene@rocklinux.org

Homepage: http://www.tfh-berlin.de/~s712059/index.html

Anyone sending unwanted advertising e-mail to this address will be
charged $25 for network traffic and computing time. By extracting my
address from this message or its header, you agree to these terms.

RE: [reiserfs-list] ReiserFS on RAID5 Linux-2.4 - speed problem

[Torrey Hoffman]

René Rebe wrote:
[...]
> I run ReiserFS on a RAID5 (of 3 IDE disks) using the latest 
> 2.4.(e.g.16)
> kernel for weeks. It works well except that is is painfully 
> slow. 
[...]

You are not the only one with these problems...

I am convinced there is a serious, negative performance 
interaction between software RAID 5 and ReiserFS, at least
on IDE.  Performance is less than 25% of what it should be.

---------------
Concise summary: 

dbench of reiserfs on a normal drive partition is slightly
* faster * than timing a raw "dd" from that partition.

But, dbench of reiserfs on a reiserfs on software RAID5
is ** only 1/4 as fast ** as a raw "dd" from the RAID5, 

And, to make it worse, software RAID 5 is significantly 
slower than I think it should be.  As a result, my nice
RAID 5 is far, far slower than an ordinary partition.
These are not just dbench numbers - all sorts of testing
shows the problem.

---------------
Here are the details:

I have four Maxtor 60 GB IDE disks, each is master on a single
cable, using two Promise Ultra TX2/100 cards.  The host is 
a dual P3-800 with 512 MB of RAM.  I bought this hardware
specifically to act as a fast, cheap, reliable server...  
I have no complaints except the speed.  

The raid disks are hde, hdg, hdi, hdk, as /dev/md0, not
partitioned, /dev/md0 is a single 180 GB reiserfs and is 
mounted on /home.  Chunk size is 1024.

Here's some low level numbers:

Single drive from the RAID:
time dd if=/dev/hdg of=/dev/null bs=1M count=1024
- elapsed time 0:37 = 27.65 MB/sec

Raw RAID5:
time dd if=/dev/md0 of=/dev/null bs=1M count=1024
- elapsed time 0:25 = 40.96 MB/sec

File on Reiserfs on RAID5:
time dd if=/home/test_file_1GB of=/dev/null bs=1M count=1024
- elapsed time 0:33 = 31.03 MB/sec

Old home partition on hda9:
time dd if=/dev/hda9 of=/dev/null bs=1M count=1024
- elapsed time 0:41 = 24 MB/sec

dbench tests:

dbench 32 on reiserfs on md0  : 10.7027 MB/sec
dbench 32 on reiserfs on hda9 : 27.7925 MB/sec

Note that when I first set this hardware up, I tried RAID0
on the same hardware and saw dbench 32 numbers of 76 MB/sec.

So, I think the dbench number for reiserfs on md0 should be
** at least ** five times faster than it is.  Are my 
expectations too high or is there a real problem here?

(tested on 2.4.15-pre5, this problem has been around for
all 2.4 kernels I've tried.)

Happy to help test and debug... please send suggestions.

Torrey Hoffman

Re: [reiserfs-list] ReiserFS on RAID5 Linux-2.4 - speed problem

[Rene Rebe]

On Thu, 29 Nov 2001 13:36:08 -0800
Torrey Hoffman <torrey.hoffman@myrio.com> wrote:

> René Rebe wrote:
> [...]
> > I run ReiserFS on a RAID5 (of 3 IDE disks) using the latest 
> > 2.4.(e.g.16)
> > kernel for weeks. It works well except that is is painfully 
> > slow. 
> [...]
> 
> You are not the only one with these problems...

Have you tryed:

echo 1023 > /proc/sys/vm/max-readahead

(power of 2 -1) found in some -ac or 2.4.16 kernel. I was amazed that
it might accelerated my setup by a factor of 2 ? (I still have
to verify this ...)

[...]

> Happy to help test and debug... please send suggestions.
> 
> Torrey Hoffman
> 



k33p h4ck1n6
  René

-- 
René Rebe (Registered Linux user: #127875 <http://counter.li.org>)

eMail:    rene.rebe@gmx.net
          rene@rocklinux.org

Homepage: http://www.tfh-berlin.de/~s712059/index.html

Anyone sending unwanted advertising e-mail to this address will be
charged $25 for network traffic and computing time. By extracting my
address from this message or its header, you agree to these terms.

RE: [reiserfs-list] ReiserFS on RAID5 Linux-2.4 - speed problem

[Torrey Hoffman]

René Rebe wrote:
> Have you tryed:
> 
> echo 1023 > /proc/sys/vm/max-readahead
[...]

Well, I gave that a try.  

note:
/dev/md0 is a 180 GB reiserfs on software RAID 5.
/dev/hda9 is an 18 GB reiserfs on a normal partition.
System is a dual PIII-800 with 512 MB RAM.
Kernel is 2.4.16 + Andrew Morton's Low Latency and I/O 
scheduling patches.

before:
cat /proc/sys/vm/max-readahead:	31
dbench 32 on /dev/md0  :	8.11 MB/sec
dbench 32 on /dev/hda9 :	24.85 MB/sec

after:
echo 1023 > /proc/sys/vm/max-readahead
dbench 32 on /dev/md0  :	8.047 MB/sec
dbench 32 on /dev/hda9 :	24.65 MB/sec

So that change actually made things a little worse,
at least on this kernel.  :-(

Torrey