Large Partition Performance Test

Large Partition Performance Test

[Bill Rees]

I upgraded my kernel to 2.4.19-pre6 with the speedup
patch installed and still see similiar results as
before.  As the amount of data on the partition
increases, my write performance slows.  This time, I
simply modified a previously posted script to mimic my
production application.  The test is still running and
will take quite awhile to fill the disk.  Each line of
the output represents 1.68gb of disk space.

Test system:
------------

Dual Athlon MP 1600+
512 gb system memory
3ware 7450 disk switch configured for RAID 5 (6x160
Maxtor drives) formatted to 800gb
custom distribution of Linux using aformentioned
kernel

Results:
--------

day/hour 00 00 seconds = 259
day/hour 00 01 seconds = 278
day/hour 00 02 seconds = 290
day/hour 00 03 seconds = 302
day/hour 00 04 seconds = 314
day/hour 00 05 seconds = 328
day/hour 00 06 seconds = 339
day/hour 00 07 seconds = 352
day/hour 00 08 seconds = 365
day/hour 00 09 seconds = 375
day/hour 00 10 seconds = 386
day/hour 00 11 seconds = 393
day/hour 00 12 seconds = 405
day/hour 00 13 seconds = 416
day/hour 00 14 seconds = 425
day/hour 00 15 seconds = 434
day/hour 00 16 seconds = 446
day/hour 00 17 seconds = 457
day/hour 00 18 seconds = 468

Script:
-------

#!/bin/sh
block_size=1024
blocks_count=15
for d1 in 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14
15 16 17 18 19 20 21 22 23 24 25; do
    for h1 in 00 01 02 03 04 05 06 07 08 09 10 11 12
13 14 15 16 17 18 19 20 21 22 23; do
        echo -n "day/hour $d1 $h1 seconds = "
        begin_time=`date +%s`
        for m1 in 00 01 02 03 04 05 06 07 08 09 10 11
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 46 47 48
49 50 51 52 53 54 55 56 57 58 59 ; do
            mkdir -p /video00/$d1/$h1$m1
            for s1 in 00 01 02 03 04 05 06 07 08 09 10
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
47 48 49 50 51 52 53 54 55 56 57 58 59; do
                for f1 in 00 01 02 03 04 05 06 07 08
09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
27 28 29; do
                    dd if=/dev/zero
of=/video00/$d1/$h1$m1/$s1$f1 \
                        bs=$block_size
count=$blocks_count \
                        >/dev/null 2>&1
                done
            done
        done
        end_time=`date +%s`
        echo $(($end_time - $begin_time))
    done
done

__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/

Re: Large Partition Performance Test

[Hans Reiser]

Bill Rees wrote:

>I upgraded my kernel to 2.4.19-pre6 with the speedup
>patch installed and still see similiar results as
>before.  As the amount of data on the partition
>increases, my write performance slows.  This time, I
>simply modified a previously posted script to mimic my
>production application.  The test is still running and
>will take quite awhile to fill the disk.  Each line of
>the output represents 1.68gb of disk space.
>
>Test system:
>------------
>
>Dual Athlon MP 1600+
>512 gb system memory
>3ware 7450 disk switch configured for RAID 5 (6x160
>Maxtor drives) formatted to 800gb
>custom distribution of Linux using aformentioned
>kernel
>
>Results:
>--------
>
>day/hour 00 00 seconds = 259
>day/hour 00 01 seconds = 278
>day/hour 00 02 seconds = 290
>day/hour 00 03 seconds = 302
>day/hour 00 04 seconds = 314
>day/hour 00 05 seconds = 328
>day/hour 00 06 seconds = 339
>day/hour 00 07 seconds = 352
>day/hour 00 08 seconds = 365
>day/hour 00 09 seconds = 375
>day/hour 00 10 seconds = 386
>day/hour 00 11 seconds = 393
>day/hour 00 12 seconds = 405
>day/hour 00 13 seconds = 416
>day/hour 00 14 seconds = 425
>day/hour 00 15 seconds = 434
>day/hour 00 16 seconds = 446
>day/hour 00 17 seconds = 457
>day/hour 00 18 seconds = 468
>
>Script:
>-------
>
>#!/bin/sh
>block_size=1024
>blocks_count=15
>for d1 in 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14
>15 16 17 18 19 20 21 22 23 24 25; do
>    for h1 in 00 01 02 03 04 05 06 07 08 09 10 11 12
>13 14 15 16 17 18 19 20 21 22 23; do
>        echo -n "day/hour $d1 $h1 seconds = "
>        begin_time=`date +%s`
>        for m1 in 00 01 02 03 04 05 06 07 08 09 10 11
>12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
>30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 46 47 48
>49 50 51 52 53 54 55 56 57 58 59 ; do
>            mkdir -p /video00/$d1/$h1$m1
>            for s1 in 00 01 02 03 04 05 06 07 08 09 10
>11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
>29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
>47 48 49 50 51 52 53 54 55 56 57 58 59; do
>                for f1 in 00 01 02 03 04 05 06 07 08
>09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
>27 28 29; do
>                    dd if=/dev/zero
>of=/video00/$d1/$h1$m1/$s1$f1 \
>                        bs=$block_size
>count=$blocks_count \
>                        >/dev/null 2>&1
>                done
>            done
>        done
>        end_time=`date +%s`
>        echo $(($end_time - $begin_time))
>    done
>done
>
>__________________________________________________
>Do You Yahoo!?
>Yahoo! Tax Center - online filing with TurboTax
>http://taxes.yahoo.com/
>
>
Interesting, please investigate Edward and Oleg....

Hans

Large Partition Performance Test

[Ivan Ivanov]

May be you have bad or very long disk cables.
I remember that some years ago I had the similar problem with one
machine.
I used Norton Disk Doctor for DOS to check the IDE drive - read/write
test.
The check speed that disk doctor shows in the begining of the drive was
more then
2 times higher then the check speed at the end of the disk.
When I replaced IDE cable with a shorter one the differense was much
lower - about 20%.


And if it is posible for you to try the same test on a smaller part of
the disks (about 20Gb for each drive).
I made my  test on 1.2GB partition in the middle of a 6GB hdd,  so the
influence of the phisical
disk geometry is lower.

Regards
Ivan

Re: Large Partition Performance Test

[Oleg Drokin]

Hello!

On Thu, Apr 11, 2002 at 11:14:59AM -0700, Bill Rees wrote:

> I upgraded my kernel to 2.4.19-pre6 with the speedup
> patch installed and still see similiar results as
> before.  As the amount of data on the partition

Good.

> Dual Athlon MP 1600+
> 512 gb system memory

HUH???

> Results:
> --------
> day/hour 00 00 seconds = 259

4 minutes 20 seconds to write 1.6G of data even in the beginning?!
This sounds too slow to me. Though RAID5 certainly can slow things down.

We will perform some tests tomorrow with your test script.

Thank you for the report.

Bye,
    Oleg

Re: Large Partition Performance Test

[Bill Rees]

Correction, 512mb of RAM, not 512gb! :)


--- Bill Rees <mtfbwy_1968@yahoo.com> wrote:
> I upgraded my kernel to 2.4.19-pre6 with the speedup
> patch installed and still see similiar results as
> before.  As the amount of data on the partition
> increases, my write performance slows.  This time, I
> simply modified a previously posted script to mimic
> my
> production application.  The test is still running
> and
> will take quite awhile to fill the disk.  Each line
> of
> the output represents 1.68gb of disk space.
> 
> Test system:
> ------------
> 
> Dual Athlon MP 1600+
> 512 gb system memory
> 3ware 7450 disk switch configured for RAID 5 (6x160
> Maxtor drives) formatted to 800gb
> custom distribution of Linux using aformentioned
> kernel
> 
> Results:
> --------
> 
> day/hour 00 00 seconds = 259
> day/hour 00 01 seconds = 278
> day/hour 00 02 seconds = 290
> day/hour 00 03 seconds = 302
> day/hour 00 04 seconds = 314
> day/hour 00 05 seconds = 328
> day/hour 00 06 seconds = 339
> day/hour 00 07 seconds = 352
> day/hour 00 08 seconds = 365
> day/hour 00 09 seconds = 375
> day/hour 00 10 seconds = 386
> day/hour 00 11 seconds = 393
> day/hour 00 12 seconds = 405
> day/hour 00 13 seconds = 416
> day/hour 00 14 seconds = 425
> day/hour 00 15 seconds = 434
> day/hour 00 16 seconds = 446
> day/hour 00 17 seconds = 457
> day/hour 00 18 seconds = 468
> 
> Script:
> -------
> 
> #!/bin/sh
> block_size=1024
> blocks_count=15
> for d1 in 00 01 02 03 04 05 06 07 08 09 10 11 12 13
> 14
> 15 16 17 18 19 20 21 22 23 24 25; do
>     for h1 in 00 01 02 03 04 05 06 07 08 09 10 11 12
> 13 14 15 16 17 18 19 20 21 22 23; do
>         echo -n "day/hour $d1 $h1 seconds = "
>         begin_time=`date +%s`
>         for m1 in 00 01 02 03 04 05 06 07 08 09 10
> 11
> 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
> 29
> 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 46 47
> 48
> 49 50 51 52 53 54 55 56 57 58 59 ; do
>             mkdir -p /video00/$d1/$h1$m1
>             for s1 in 00 01 02 03 04 05 06 07 08 09
> 10
> 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
> 28
> 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
> 46
> 47 48 49 50 51 52 53 54 55 56 57 58 59; do
>                 for f1 in 00 01 02 03 04 05 06 07 08
> 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
> 26
> 27 28 29; do
>                     dd if=/dev/zero
> of=/video00/$d1/$h1$m1/$s1$f1 \
>                         bs=$block_size
> count=$blocks_count \
>                         >/dev/null 2>&1
>                 done
>             done
>         done
>         end_time=`date +%s`
>         echo $(($end_time - $begin_time))
>     done
> done
> 
> __________________________________________________
> Do You Yahoo!?
> Yahoo! Tax Center - online filing with TurboTax
> http://taxes.yahoo.com/


=====
Bill Rees - bill@billrees.com - http://billrees.com

__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/

Re: Large Partition Performance Test

[Dirk Mueller]

On Don, 11 Apr 2002, Oleg Drokin wrote:

> 4 minutes 20 seconds to write 1.6G of data even in the beginning?!
> This sounds too slow to me. Though RAID5 certainly can slow things down.

Its not the RAID,I can reproduce it: 

day/hour 00 00 seconds = 691
day/hour 00 01 seconds = 720

this seems to be highly CPU bound, because it hardly ever accessed the 
harddisk. it spends 60% in kernel space cpu though. 


Dirk

Re: Large Partition Performance Test

[Chris Mason]

On Thu, 2002-04-11 at 16:28, Dirk Mueller wrote:
> On Don, 11 Apr 2002, Oleg Drokin wrote:
> 
> > 4 minutes 20 seconds to write 1.6G of data even in the beginning?!
> > This sounds too slow to me. Though RAID5 certainly can slow things down.
> 
> Its not the RAID,I can reproduce it: 
> 
> day/hour 00 00 seconds = 691
> day/hour 00 01 seconds = 720
> 
> this seems to be highly CPU bound, because it hardly ever accessed the 
> harddisk. it spends 60% in kernel space cpu though. 

Then I would expect the new allocator in busy_skip mode to fix it.  But,
having profiling on would tell us for sure.

-chris

Re: Large Partition Performance Test

[Oleg Drokin]

Hello!

On Thu, Apr 11, 2002 at 11:14:59AM -0700, Bill Rees wrote:
> I upgraded my kernel to 2.4.19-pre6 with the speedup
> patch installed and still see similiar results as
> before.  As the amount of data on the partition
> increases, my write performance slows.  This time, I
> simply modified a previously posted script to mimic my
> production application.  The test is still running and
> will take quite awhile to fill the disk.  Each line of
> the output represents 1.68gb of disk space.

Ok, I see the problem now. You create a lot of small files, and when a lot
of disk space is used, a lot of unnecesary blockfinding is performed on
each file creation.
if you mount your partition with -o no_unhashed_relocation option,
effect of that will be much lighter. With this option that's what I see:
day/hour 00 00 seconds = 285
day/hour 00 01 seconds = 288
day/hour 00 02 seconds = 290
day/hour 00 03 seconds = 289
day/hour 00 04 seconds = 290
day/hour 00 05 seconds = 292
day/hour 00 06 seconds = 292
day/hour 00 07 seconds = 292


Also new block allocator that I posted to the list recently should not excibit
this behavior at all in any of modes. (I am just going to verify this theory ;)
)

Bye,
    Oleg

Re: Large Partition Performance Test

[Oleg Drokin]

Hello!

On Thu, Apr 11, 2002 at 06:11:23PM -0400, Chris Mason wrote:

> Then I would expect the new allocator in busy_skip mode to fix it.  But,
> having profiling on would tell us for sure.

Even withoyt skip_busy mode it will work much better.
The problem is on each file creation we start to look for a free block almost
from the beginning of disk. using no_unhashed_relocation mount option,
we make initial block allocation to be more random and cause less overhead
(verified).
And in this test we create a lot of 15k files.

Bye,
    Oleg

Re: Large Partition Performance Test

[Chris Mason]

On Fri, 2002-04-12 at 05:41, Oleg Drokin wrote:
> Hello!
> 
> On Thu, Apr 11, 2002 at 06:11:23PM -0400, Chris Mason wrote:
> 
> > Then I would expect the new allocator in busy_skip mode to fix it.  But,
> > having profiling on would tell us for sure.
> 
> Even withoyt skip_busy mode it will work much better.
> The problem is on each file creation we start to look for a free block almost
> from the beginning of disk. using no_unhashed_relocation mount option,
> we make initial block allocation to be more random and cause less overhead
> (verified).
> And in this test we create a lot of 15k files.

Ok, I'll buy that.  The only way to know for sure is to give it a try or
profile the run, but I think it is very likely to be allocator related.

-chris

Re: Large Partition Performance Test

[Bill Rees]

Oleg,

That seems to have done the trick.  Which is the most
stable release that you would recommend that would
include this fix?

thanks!
Bill

--- Oleg Drokin <green@namesys.com> wrote:
> Hello!
> 
> On Thu, Apr 11, 2002 at 11:14:59AM -0700, Bill Rees
> wrote:
> > I upgraded my kernel to 2.4.19-pre6 with the
> speedup
> > patch installed and still see similiar results as
> > before.  As the amount of data on the partition
> > increases, my write performance slows.  This time,
> I
> > simply modified a previously posted script to
> mimic my
> > production application.  The test is still running
> and
> > will take quite awhile to fill the disk.  Each
> line of
> > the output represents 1.68gb of disk space.
> 
> Ok, I see the problem now. You create a lot of small
> files, and when a lot
> of disk space is used, a lot of unnecesary
> blockfinding is performed on
> each file creation.
> if you mount your partition with -o
> no_unhashed_relocation option,
> effect of that will be much lighter. With this
> option that's what I see:
> day/hour 00 00 seconds = 285
> day/hour 00 01 seconds = 288
> day/hour 00 02 seconds = 290
> day/hour 00 03 seconds = 289
> day/hour 00 04 seconds = 290
> day/hour 00 05 seconds = 292
> day/hour 00 06 seconds = 292
> day/hour 00 07 seconds = 292
> 
> 
> Also new block allocator that I posted to the list
> recently should not excibit
> this behavior at all in any of modes. (I am just
> going to verify this theory ;)
> )
> 
> Bye,
>     Oleg


=====
Bill Rees - bill@billrees.com - http://billrees.com

__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/

Re: Large Partition Performance Test

[Oleg Drokin]

Hello!

On Fri, Apr 12, 2002 at 06:21:33AM -0700, Bill Rees wrote:

> That seems to have done the trick.  Which is the most
> stable release that you would recommend that would
> include this fix?

Huh? Fix? This mount options is in kernel for ages! ( cannot say for
how long exactly, but certainly more than for half of year).
Too bad is not documented well.
We now recommend 2.4.18 kernel that contains all critical fixes.
When 2.4.19 will eb out, it will become out recommended version, if
nothign bad won't happen with it.

We also have this new block allocator alrogithm, I recently posted to the list,
which should also have positive impact on workload like yours .
But we cannot recommend it for productional use yet.

Bye,
    Oleg

Re: Large Partition Performance Test

[Oleg Drokin]

Hello!

On Fri, Apr 12, 2002 at 09:21:22AM -0400, Chris Mason wrote:
> > Even withoyt skip_busy mode it will work much better.
> > The problem is on each file creation we start to look for a free block almost
> > from the beginning of disk. using no_unhashed_relocation mount option,
> > we make initial block allocation to be more random and cause less overhead
> > (verified).
> > And in this test we create a lot of 15k files.
> Ok, I'll buy that.  The only way to know for sure is to give it a try or

OF course you'll buy that, because my suggested workaround worked ;)

> profile the run, but I think it is very likely to be allocator related.

First I decided I need a profile run, but once I realised how small created
files are, and looked on the on-disk block distribution chart.
Then I remembered on what basis we choose 1st block of a file to be allocated,
and I realised I do not need profile run ;)

In fact I am interested in similar task performance of ext3, since I've yet
to try it.

Bye,
    Oleg