Huge disk performance degradation in 2.4.9

Huge disk performance degradation in 2.4.9

[Wolly]

Sorry if this is already known or fixed. 
I did not find something related in the last 2.4.10-pre changelog. 

I recently upgraded from 2.4.6 to 2.4.9 and since then noticed that my 
hd (old noisy thing in an old P100 box) repeatedly 
sounds strange (read on, please!) when dealing with lots of small files. 
(No physical problem; it's a kernel flushing issue.)

This turned out to be the sign of a quite huge performance loss anywhere 
between 2.4.7 and 2.4.9 (sorry, using a 56k Modem, I cannot test them 
all and it's a shame that I already deleted 2.4.6 sources). 

I verified this on a PII-350 (440BX, 196Mb) using 600 files (instead of 300). 
I only tested things on reiserfs (v3.6. using 3.5 partitions) becuase I 
don't have ext2 around any longer. 

For testing purposes, I used: 
sync ; cat /proc/version ; sleep 1 ; /usr/bin/time sh -c 'declare -i cnt=0 ; 
while test $cnt -lt 300 ; do echo -en "\b\b\b\b\b\b\b$cnt " ; 
dd if=/dev/zero of=file-$cnt bs=1 count=16 >/dev/null 2>&1 ; 
cnt=$cnt+1 ; done ; echo' ; rm file-*

This creates lots of 16-byte files using 16 write() calls [second test 
with 160kb files using 16 write() calls] and prints out the time 
(On P-100; 72Mb, 1Gb hd; besides kernel: equal setup for 
all tests; machine idle; all tests ran several times; all kernels compiled 
with gcc version 2.95.2 19991024 (release)): 

       |   (dd bs=1 count=16)    | (dd bs=10240 count=16)
kernel | user system elapsed CPU | user system elapsed CPU
2.4.9  | 3.12   5.10   32.46 25% | 3.84  14.31   73.09 24%
2.4.6  | 3.28   4.17    8.17 91% | 3.96  14.76   25.76 72%
2.2.19 | 2.82   3.75    7.12 92% | 4.42  12.90   19.26 89%
(user, system, elapsed time in seconds)

Look at the elapsed times! 2.4.9 takes >=3 times as long as 2.4.6 
(and 2.2.19 performs even better). 
This is a huge performance issue and I actually notice it using when 
squid or doing a CVS checkout with lots of small files. 

When listening to the hd you note a difference. While 2.4.6 does 
a clustered write call from time to time, 2.4.9 starts to burst out in 
accessing the hd (always moving the hd's head) and does not 
finish until the test is over (same with my PII-350 and IBM 12Gb) 

Anyone reproducing this? Reiserfs issue or cache/buffer/flush issue? 

Regards, 
Wolly

Re: Huge disk performance degradation in 2.4.9

[Steve Kieu]

> 
> Anyone reproducing this? Reiserfs issue or
> cache/buffer/flush issue? 

I can not reproduced it witn ext2 2.4.10pre12aa1

I noticed the performance is much improved (to 2.4.9
and < )

I am gonna try it with ext3 later...

Cheers,

 
> Regards, 
> Wolly
> -
> To unsubscribe from this list: send the line
> "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at 
> http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/ 

=====
S.KIEU

http://travel.yahoo.com.au - Yahoo! Travel
- Got Itchy feet? Get inspired!

Re: Huge disk performance degradation STILL IN 2.4.10

[Wolly]

Hi kernel hackers, 

As soon as 2.4.10 was out, I got the patch and tested it again. 
The problem is still there and did not get better at all. 

Please have a look at that: 

How to reproduce: Get the slowest computer you have and 
execute the file creation script as mentioned in my previous e-mail 
(appended below). 

The faster the computer and the more RAM, the more files you have 
to create. Try increasing their number from 300 to 600 or 1000. 
Try running it several times; turn off swap. 

The symptomes are as follows: The file counter raises quickly until 
at some point it begins to raise slowly only (2-4 files/sec on a P100 
with 5 year old hardware). The hd is then accessed all the time and 
the hd's head is moved around continuously making things really slow. 

If you still cannot reproduce it, make sure you have some applications 
running (X, windowmanager, xterm) and try executing `sync' in 
another XTerm while the file creation script is running. 

When I tried, sync did not return until all files were created and 
the creation was awefully slow (steady hd accessed & head move). 

>Is there nobody out there who can try & reproduce this?!

To me, it seems this is either a cache/buffer/flush issue or a 
reiserfs journal congestion issue. 

Regards, 
Wolly

On Friday 21 September 2001 19:23, I wrote:
> I recently upgraded from 2.4.6 to 2.4.9 and since then noticed that my
> hd (old noisy thing in an old P100 box) repeatedly
> sounds strange (read on, please!) when dealing with lots of small files.
> (No physical problem; it's a kernel flushing issue.)
>
> This turned out to be the sign of a quite huge performance loss anywhere
> between 2.4.7 and 2.4.9 (sorry, using a 56k Modem, I cannot test them
> all and it's a shame that I already deleted 2.4.6 sources).
>
> I verified this on a PII-350 (440BX, 196Mb) using 600 files (instead of
> 300). I only tested things on reiserfs (v3.6. using 3.5 partitions) becuase
> I don't have ext2 around any longer.
>
> For testing purposes, I used:
> sync ; cat /proc/version ; sleep 1 ; /usr/bin/time sh -c 'declare -i cnt=0
> ; while test $cnt -lt 300 ; do echo -en "\b\b\b\b\b\b\b$cnt " ;
> dd if=/dev/zero of=file-$cnt bs=1 count=16 >/dev/null 2>&1 ;
> cnt=$cnt+1 ; done ; echo' ; rm file-*
>
> This creates lots of 16-byte files using 16 write() calls [second test
> with 160kb files using 16 write() calls] and prints out the time
> (On P-100; 72Mb, 1Gb hd; besides kernel: equal setup for
> all tests; machine idle; all tests ran several times; all kernels compiled
>
> with gcc version 2.95.2 19991024 (release)):
>        |   (dd bs=1 count=16)    | (dd bs=10240 count=16)
>
> kernel | user system elapsed CPU | user system elapsed CPU
> 2.4.9  | 3.12   5.10   32.46 25% | 3.84  14.31   73.09 24%
> 2.4.6  | 3.28   4.17    8.17 91% | 3.96  14.76   25.76 72%
> 2.2.19 | 2.82   3.75    7.12 92% | 4.42  12.90   19.26 89%
> (user, system, elapsed time in seconds)
>
> Look at the elapsed times! 2.4.9 takes >=3 times as long as 2.4.6
> (and 2.2.19 performs even better).
> This is a huge performance issue and I actually notice it using when
> squid or doing a CVS checkout with lots of small files.
>
> When listening to the hd you note a difference. While 2.4.6 does
> a clustered write call from time to time, 2.4.9 starts to burst out in
> accessing the hd (always moving the hd's head) and does not
> finish until the test is over (same with my PII-350 and IBM 12Gb)
>
> Anyone reproducing this? Reiserfs issue or cache/buffer/flush issue?
>
> Regards,
> Wolly

Re: Huge disk performance degradation STILL IN 2.4.10

[Chris Mason]

On Monday, September 24, 2001 01:19:18 AM +0200 Wolly <wwolly@gmx.net>
wrote:

> Hi kernel hackers, 
> 
> As soon as 2.4.10 was out, I got the patch and tested it again. 
> The problem is still there and did not get better at all. 
> 

Could you please mount the FS with -o notail and try again?  I think your
test is hitting a worst case in the 2.4.x reiserfs tail code.

-chris

Re: Huge disk performance degradation STILL IN 2.4.10

[Wolly]

On Monday 24 September 2001 01:55, Chris Mason wrote:
> On Monday, September 24, 2001 01:19:18 AM +0200 Wolly <wwolly@gmx.net>
> wrote:
> > Hi kernel hackers,
> >
> > As soon as 2.4.10 was out, I got the patch and tested it again.
> > The problem is still there and did not get better at all.
>
> Could you please mount the FS with -o notail and try again?  I think your
> test is hitting a worst case in the 2.4.x reiserfs tail code.
>
Okay, I plugged some old hd into my computer and formatted one half with 
ext2, the other half with reiserfs. 
It seems to be definitely a reiserfs issue because I cannot trigger the 
performance loss (permament hd head positioning) with ext2. However, 
passing -o notail when mounting does not help. (-o notail is accepted and 
`bash# mount' displays `/dev/hdxy on /mnt type reiserfs (rw,notail)')

Could you reproduce this? Did you even try?
And why does this not show up on kernels <=2.4.6 and shows up 
with linux>=2.4.9 (2.4.7, 2.4.8: unknown)?

-Wolly

Re: Huge disk performance degradation STILL IN 2.4.10

[Andre Pang]

On Mon, Sep 24, 2001 at 05:05:13PM +0200, Wolly wrote:

> Okay, I plugged some old hd into my computer and formatted one
> half with ext2, the other half with reiserfs.  It seems to be
> definitely a reiserfs issue because I cannot trigger the
> performance loss (permament hd head positioning) with ext2.

This might sound pedantic, but is ext2 on the first half of your
hard disk and reiserfs on the second half?  If so, that could skew
your results.  Try putting the reiserfs in the first half of the
disk and see if that makes a difference.

See http://www.coker.com.au/bonnie++/zcav/ for why.


-- 
#ozone/algorithm <ozone@algorithm.com.au>          - trust.in.love.to.save

Re: Huge disk performance degradation STILL IN 2.4.10

[Wolly]

On Monday 24 September 2001 17:19, Andre Pang wrote:
> On Mon, Sep 24, 2001 at 05:05:13PM +0200, Wolly wrote:
> > Okay, I plugged some old hd into my computer and formatted one
> > half with ext2, the other half with reiserfs.  It seems to be
> > definitely a reiserfs issue because I cannot trigger the
> > performance loss (permament hd head positioning) with ext2.
>
> This might sound pedantic, but is ext2 on the first half of your
> hard disk and reiserfs on the second half?  If so, that could skew
> your results.  Try putting the reiserfs in the first half of the
> disk and see if that makes a difference.
>
Well, actually this lead me to another idea: I created a disk image 
on my reiserfs home partition (the one I usually use to trigger the 
problem; note however that I can also reproduce it on my PII-350). 

Firstly, I formatted the image with ext2 and run the file creation script. 
No problems, it works fine. 

Then, I formatted the image with reiserfs and tried the file creation 
script. No problems either.. Strange. 

I doubt it's a fragmentation issue (as kernel-2.4.6 does not have the 
problem and the image file would be fragmented as well). 

Could this be some sort of reiserfs flushing issue? 
<speculation> I mean reiserfs wants to flush the newly created files 
too frequently. If reiserfs is on a disk partition, then this results in 
the performance degradation, if it is on an image, all requests to write 
to the image are cached again (by the reiserfs one level below which 
works fine because there is just manipulation of one large file) and the 
problem does not show up. </speculation>

Okay, as just a few people on the list seem to worry about the issue, 
I spent an hour online and downloaded kernel 2.4.6 + patches for 
2.4.[789] again to see exactly where the problem begins. 

Wolly

Re: Huge disk performance degradation STILL IN 2.4.10

[Tom Vier]

i noticed it in 2.4.7-ac3 -> ac4, iirc.

On Tue, Sep 25, 2001 at 12:00:00PM +0200, Wolly wrote:
> Okay, as just a few people on the list seem to worry about the issue, 
> I spent an hour online and downloaded kernel 2.4.6 + patches for 
> 2.4.[789] again to see exactly where the problem begins. 

-- 
Tom Vier <tmv5@home.com>
DSA Key id 0x27371A2C