Weird performance decrease

[Sascha Nitsch <sgi@linuxhowtos.org>]

Hi,

I'm observing a rather strange behaviour of the filesystem cache algorithm.

I have a server running the following app scenario:

A filesystem tree with a depth of 7 directories and 4 character directory 
names.
In the deepest directories are files.
filesize from 100 bytes to 5kb.
Filesystem is XFS.

The app creates dirs in the tree and reads/writes files into the deepest dirs 
in the tree.

CPU: Dual Xeon 3.0 Ghz w/HT 512KB cache each, 2GB RAM, SCSI-HDD 15k RPM

The first while, all is fine and extremely fast. After a while the buffer size 
is about 3.5 MB
and cache size about 618 MB.
Until that moment ~445000 directories and ~106000 files have been created

Thats where the weird behaviour starts.

The buffer size drops to ~200 kb and cache size starts decreasing fast.
This results in a drastic performace drop in my app.
(avg. read/write times increase from 0.3ms to 4ms)
not a constant increase, a jumping increase. During the next while it 
constantly gets slower (19ms and more).

After running a while (with still reducing cache size) the buffer size stays 
at
~700kb and cache about 400 MB. Performane is terrible. Way slower than 
starting up with no cache.

restarting the app makes no change, neither remounting the partition.

cmd to create the fs:
mkfs.xfs -b size=512 -i maxpct=0 -l version=2 -n size=16k /dev/sdc
mounting with
mount /dev/sdc /data

I'm open for suggestion on mkfs calls, mount options and kernel tuning via 
procfs.
I have a testcase to reproduce the problem. It happens after ~45 minutes.

xfs_info /data/
meta-data=/data                  isize=256    agcount=16, agsize=8960921 blks
         =                       sectsz=512
data     =                       bsize=512    blocks=143374736, imaxpct=0
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=16384
log      =internal               bsize=512    blocks=65536, version=2
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

kernel:
a 2.6.9-34.0.2.ELsmp #1 SMP Mon Jul 17 21:41:41 CDT 2006 i686 i686 i386 
GNU/Linux

filesystem usage is < 1%