problem with iowait (disk I/O) in 2.4.21 kernel

problem with iowait (disk I/O) in 2.4.21 kernel

[Suman Puthana]

Hello,

Is anybody aware of any changes in the 2.4.21 kernel which affects disk I/O
badly?  We are noticing that the cpu taken up by iowait's is causing
significant degradation in disk performance.

We have an application in which there are multiple processes writing to disk
simultaneously, each process writing anywhere from 16 - 64 K Bytes at a time
and sleeping for about 50 ms depending on how long the write() call takes to
finish. It is important that the write() finishes within 50 ms for this
application. The file's are opened with the open() call with the standard
O_RDWR and O_CREAT flags.

On the 2.4.18 kernel, 25 of these processes take up about 15% of CPU on a
dual Xeon Pentium 4 2.4 GHz processor with 1 MB of memory for a total
throughput of about 7 MBps which is very good.  The write() calls typically
take less than 5 ms to complete and we sleep for the remaining 45 ms or so.

But on the same system when we installed the 2.4.21 kernel, these 25
processes take anywhere between 40 - 70 % of the CPU depending on how  much
CPU the iowait is taking up. The iowait part of it varies all the way from
10 - 60 %. (The iowait CPU is within 1% on the 2.4.18 kernel.).  What is
even worse - sometimes the write() calls take anywhere between 1 - 2
seconds( yes seconds!) to return, which degrades the performance of our
application server pretty badly.

 It happens on all kinds of storage devices so it's definitely not a
hardware problem.(We tested on the standard IDE disk all the way upto the
EMC clariion storage system).

Is this iowait something which was introduced in the 2.4.21 kernel core code
or in the file system driver code? It definitely happens with jfs, reiserfs
and also on ext3 though not as badly . Is there a way to turn it off or to
make it take up lesser CPU?

Any insight or pointers regarding this problem would be greatly
appreciated.  Thank you for your support.

- Suman Puthana

PS - Something else of interest here is that when the iowait part of the CPU
increases, iostat shows a significantly  higher throughput than what we are
actually trying to write to disk. (When sampled over 30 seconds, when we are
trying to write 7 MBps, iostat actually shows 11-12 MBps when iowait goes
upto 30-40 %). It's almost like there's some internal copies going on.

Re: problem with iowait (disk I/O) in 2.4.21 kernel

[Nick Piggin]

Suman Puthana wrote:

>Hello,
>
>Is anybody aware of any changes in the 2.4.21 kernel which affects disk I/O
>badly?  We are noticing that the cpu taken up by iowait's is causing
>significant degradation in disk performance.
>
>We have an application in which there are multiple processes writing to disk
>simultaneously, each process writing anywhere from 16 - 64 K Bytes at a time
>and sleeping for about 50 ms depending on how long the write() call takes to
>finish. It is important that the write() finishes within 50 ms for this
>application. The file's are opened with the open() call with the standard
>O_RDWR and O_CREAT flags.
>
>On the 2.4.18 kernel, 25 of these processes take up about 15% of CPU on a
>dual Xeon Pentium 4 2.4 GHz processor with 1 MB of memory for a total
>throughput of about 7 MBps which is very good.  The write() calls typically
>take less than 5 ms to complete and we sleep for the remaining 45 ms or so.
>
>But on the same system when we installed the 2.4.21 kernel, these 25
>processes take anywhere between 40 - 70 % of the CPU depending on how  much
>CPU the iowait is taking up. The iowait part of it varies all the way from
>10 - 60 %. (The iowait CPU is within 1% on the 2.4.18 kernel.).  What is
>even worse - sometimes the write() calls take anywhere between 1 - 2
>seconds( yes seconds!) to return, which degrades the performance of our
>application server pretty badly.
>
> It happens on all kinds of storage devices so it's definitely not a
>hardware problem.(We tested on the standard IDE disk all the way upto the
>EMC clariion storage system).
>
>Is this iowait something which was introduced in the 2.4.21 kernel core code
>or in the file system driver code? It definitely happens with jfs, reiserfs
>and also on ext3 though not as badly . Is there a way to turn it off or to
>make it take up lesser CPU?
>
>Any insight or pointers regarding this problem would be greatly
>appreciated.  Thank you for your support.
>

Does this happen with 2.4.23? If yes, then can you find the last kernel
with good performance (is it 2.4.18, 19 or 20)?

Can you make the source code of this program available or make available
a minimal program with which you can reproduce the bad behaviour?

Thanks,
Nick