iowait stability regressed from kernel 2.6.22 under heavy load of multi-thread aio writing/reading

["rae l" <crquan@gmail.com>]

I found a problem with the vanilla kernel from 2.6.22 (include 23, 24):

the situation is to test a customized linux distribution under heavy IO load:
1. the client process initiates tens of POSIX threads (using
libpthread), each thread uses aio_write or aio_read(using librt)
operating
   on one small file, then close it and write another small file;
   the whole objective is to get a maxium throughput of small files,
by generating heavy aio stress on the system;

I have tested the vanilla kernel 2.6.22/23/24.y, all these kernels
have the common problem:
1. I use top, vmstat, and iostat to monitor the system performance, I
found that the iowait time of CPU is high at most time,
   above 60%, and not stable while the client process running,
although the throughput is stable, the CPU iowait time not stable;
   As a result of instability processes felt too long and unacceptable delay.
2. in seldom testing cases, the system even stopped: with bi/bo are 0
and iowait 100%, all writing processes are blocking with
   uninterruptible state (D state in ps output), these cases are all
system running after several days writing, but cannot guarantee
   to reproduce;

First I suspect the filesystem or the storage medium have problems, I
tried ext2/ext3/reiserfs/xfs, for different filesytems and
SATA/SAS, IDE disks, and several commercial hard RAID card for
different storage medium, but the bad result remained;

Then I tested 2.6.21.7 kernel, this results a stable iowait CPU time
(below 10%) but not so good throughput of small files;

Now I think the improvements of IO effciency in the development of
2.6.22 also caused the instability of iowait time, right? but how
to find out? Simple bisecting seems not work.

by the way, a question is how to guarantee a kernel not regress under
heavy multi-thread aio writing load? ltp project seems not give the
answer:
http://ltp.sourceforge.net/tooltable.php

--
Denis