I/O causes performance problem with 2.4.8-ac3

[Chris Schanzle <chris.schanzle@jhuapl.edu>]

This probably belongs in the "use-once" thread...

I ran into a significant (lack of) performance situation with 2.4.8-ac3 
that does not exist with 2.4.8.  Perhaps someone can shed some light on 
what happened and how to avoid it in the future.

The system is a new dual PIII 866MHz VALinux server, 2 GB of RAM, with two 
eight-disk JBODs connected through a Mylex ExtremeRAID 2000 (i.e., not a 
pig).  I am doing testing of various kernels and filesystems (ext3, 
reiserfs) to see which will be most appropriate for production use (mostly 
as an NFS server).  Using SMP-enabled kernels.

The system had been up for several hours, I had netscape going, a few 
emacs windows, untarred and compiled a bunch of linux source trees, built 
a few RPMS, did some NFS write testing, etc.  In other words, system had 
cached a bunch of buffers.

Performance was excellent until I decided to "dd bs=1024k </dev/cdrom 
 >somefile" a 600+MB cdrom while a kernel build was going on.  It took 
nearly 7 minutes to complete the dd and near the end, the cdrom drive 
light was only occasionally flickering activity (not "on" as it was at the 
start), keystrokes were delayed, refreshes were sluggish.  "top" showed 
the loadavg was hovering around 4-5, and the top-runnng processes were 
kswapd, kreclaimd, and kjournald and they were taking 30-60% of the 
processor time.  Top also showed 3.7 MB free, 425 MB buff, and about 1.5 
GB cached.  I think the first was a "dd" to an ext3 fs, then I tried 
dumping to a reiserfs, with similar performance problems.

So this morning, I built a 2.4.8 SMP kernel with only the ext3 patches 
applied and have been unable to replicate the problem.  The "dd" completed 
in under 3 minutes, while free memory dropped to lowish-levels, 
(fluctuates between 5-20 MB), and system response and loadavg were 
"reasonable."  It appears when kswapd kicks in, I get another 5-10 MB free.

So, what is one in my situation to gather from this?

--Chris