Re: problem with iowait (disk I/O) in 2.4.21 kernel

[Nick Piggin <piggin@cyberone.com.au>]

Suman Puthana wrote:

>Hello,
>
>Is anybody aware of any changes in the 2.4.21 kernel which affects disk I/O
>badly?  We are noticing that the cpu taken up by iowait's is causing
>significant degradation in disk performance.
>
>We have an application in which there are multiple processes writing to disk
>simultaneously, each process writing anywhere from 16 - 64 K Bytes at a time
>and sleeping for about 50 ms depending on how long the write() call takes to
>finish. It is important that the write() finishes within 50 ms for this
>application. The file's are opened with the open() call with the standard
>O_RDWR and O_CREAT flags.
>
>On the 2.4.18 kernel, 25 of these processes take up about 15% of CPU on a
>dual Xeon Pentium 4 2.4 GHz processor with 1 MB of memory for a total
>throughput of about 7 MBps which is very good.  The write() calls typically
>take less than 5 ms to complete and we sleep for the remaining 45 ms or so.
>
>But on the same system when we installed the 2.4.21 kernel, these 25
>processes take anywhere between 40 - 70 % of the CPU depending on how  much
>CPU the iowait is taking up. The iowait part of it varies all the way from
>10 - 60 %. (The iowait CPU is within 1% on the 2.4.18 kernel.).  What is
>even worse - sometimes the write() calls take anywhere between 1 - 2
>seconds( yes seconds!) to return, which degrades the performance of our
>application server pretty badly.
>
> It happens on all kinds of storage devices so it's definitely not a
>hardware problem.(We tested on the standard IDE disk all the way upto the
>EMC clariion storage system).
>
>Is this iowait something which was introduced in the 2.4.21 kernel core code
>or in the file system driver code? It definitely happens with jfs, reiserfs
>and also on ext3 though not as badly . Is there a way to turn it off or to
>make it take up lesser CPU?
>
>Any insight or pointers regarding this problem would be greatly
>appreciated.  Thank you for your support.
>

Does this happen with 2.4.23? If yes, then can you find the last kernel
with good performance (is it 2.4.18, 19 or 20)?

Can you make the source code of this program available or make available
a minimal program with which you can reproduce the bad behaviour?

Thanks,
Nick