I/O causes performance problem with 2.4.8-ac3

I/O causes performance problem with 2.4.8-ac3

[Chris Schanzle]

This probably belongs in the "use-once" thread...

I ran into a significant (lack of) performance situation with 2.4.8-ac3 
that does not exist with 2.4.8.  Perhaps someone can shed some light on 
what happened and how to avoid it in the future.

The system is a new dual PIII 866MHz VALinux server, 2 GB of RAM, with two 
eight-disk JBODs connected through a Mylex ExtremeRAID 2000 (i.e., not a 
pig).  I am doing testing of various kernels and filesystems (ext3, 
reiserfs) to see which will be most appropriate for production use (mostly 
as an NFS server).  Using SMP-enabled kernels.

The system had been up for several hours, I had netscape going, a few 
emacs windows, untarred and compiled a bunch of linux source trees, built 
a few RPMS, did some NFS write testing, etc.  In other words, system had 
cached a bunch of buffers.

Performance was excellent until I decided to "dd bs=1024k </dev/cdrom 
 >somefile" a 600+MB cdrom while a kernel build was going on.  It took 
nearly 7 minutes to complete the dd and near the end, the cdrom drive 
light was only occasionally flickering activity (not "on" as it was at the 
start), keystrokes were delayed, refreshes were sluggish.  "top" showed 
the loadavg was hovering around 4-5, and the top-runnng processes were 
kswapd, kreclaimd, and kjournald and they were taking 30-60% of the 
processor time.  Top also showed 3.7 MB free, 425 MB buff, and about 1.5 
GB cached.  I think the first was a "dd" to an ext3 fs, then I tried 
dumping to a reiserfs, with similar performance problems.

So this morning, I built a 2.4.8 SMP kernel with only the ext3 patches 
applied and have been unable to replicate the problem.  The "dd" completed 
in under 3 minutes, while free memory dropped to lowish-levels, 
(fluctuates between 5-20 MB), and system response and loadavg were 
"reasonable."  It appears when kswapd kicks in, I get another 5-10 MB free.

So, what is one in my situation to gather from this?

--Chris

Re: I/O causes performance problem with 2.4.8-ac3

[Helge Hafting]

Chris Schanzle wrote:
> 
> This probably belongs in the "use-once" thread...
> 
> I ran into a significant (lack of) performance situation with 2.4.8-ac3
> that does not exist with 2.4.8.  Perhaps someone can shed some light on
> what happened and how to avoid it in the future.
[...]
> In other words, system had
> cached a bunch of buffers.
> 
> Performance was excellent until I decided to "dd bs=1024k </dev/cdrom
>  >somefile" a 600+MB cdrom while a kernel build was going on.  It took
[...]
Such a big copy operation is exactly what "use-once" does well.
2.4.8 has use-once, 2.4.8ac3 don't have use-once. 

One can construct scenarios where use-once performs worse too,
I believe this is why Alan Cox didn't want it yet.  

A big copy without use-once will push everything else out of
cache, and push a lot of programs into swap in order to cache
a lot of the big copy.  That's bad if the big
copy is done once and you don't really need the stuff again.
Then you want the "other" stuff to stay in cache instead.

Use-once may perform worse if stuff falls out of cache
and has to be re-read from disk.

Helge Hafting

Re: I/O causes performance problem with 2.4.8-ac3

[Chris Mason]

On Thursday 16 August 2001 01:54 pm, Chris Schanzle wrote:
> This probably belongs in the "use-once" thread...
>
> I ran into a significant (lack of) performance situation with 2.4.8-ac3
> that does not exist with 2.4.8.  Perhaps someone can shed some light on
> what happened and how to avoid it in the future.
>
>
> Performance was excellent until I decided to "dd bs=1024k </dev/cdrom
>
>  >somefile" a 600+MB cdrom while a kernel build was going on.  It took
>
> nearly 7 minutes to complete the dd and near the end, the cdrom drive
> light was only occasionally flickering activity (not "on" as it was at the
> start), keystrokes were delayed, refreshes were sluggish.  

Most of the current use once code is for the page cache, dd if=/dev/cdrom 
will read from the buffer cache, where the pages will take longer to age out. 
 One of the patches between 2.4.8pre7 and 2.4.8final seems to have helped a 
lot here, but I'm not sure which one it was (check out the 'still buffer 
cache problems' thread).  I'm also not sure if the speedup was intentional, 
so it might go away during future tweaks.

Rik also posted a patch in 2.4.8-ac5 that deactivates buffer cache pages 
after a threshold, which should help as well (and benefit reiserfs/ext3 in 
general).

-chris