i/o wait eating all of CPU on 2.6.1

i/o wait eating all of CPU on 2.6.1

[Jaakko Helminen]

I have two servers, both of which have more than 300 gigabytes of hard drive
space and those files are made available to the network with samba, nfs and
http and it worked fine with 2.6.0 but when I upgraded to 2.6.1 I noticed
that everything was VERY slow, from a machine that is connected to the other
server with a 100M link, 57kB/s tops. i/o wait eats up all of the cpu.
On the other hand, Apache (and everything else) works very fast when I only
send /dev/zero to a client, since that doesn't need disk operations.

I don't notice anything suspicious in dmesg but since this happens on two
machines and has only happened when upgraded to 2.6.1, it's most likely
because of 2.6.1. I'm downgrading to 2.6.0 (with mremap-patch) today if I
don't figure out what is wrong. Any ideas?

And since I'm not subscribed to Linux Kernel Mailing List, please forward
any replies to me.


-Jaakko Helminen

Re: i/o wait eating all of CPU on 2.6.1

[Paolo Ornati]

On Sunday 25 January 2004 15:30, Jaakko Helminen wrote:
> I have two servers, both of which have more than 300 gigabytes of hard
> drive space and those files are made available to the network with samba,
> nfs and http and it worked fine with 2.6.0 but when I upgraded to 2.6.1 I
> noticed that everything was VERY slow, from a machine that is connected
> to the other server with a 100M link, 57kB/s tops. i/o wait eats up all
> of the cpu. On the other hand, Apache (and everything else) works very
> fast when I only send /dev/zero to a client, since that doesn't need disk
> operations.
>
> I don't notice anything suspicious in dmesg but since this happens on two
> machines and has only happened when upgraded to 2.6.1, it's most likely
> because of 2.6.1. I'm downgrading to 2.6.0 (with mremap-patch) today if I
> don't figure out what is wrong. Any ideas?

Is DMA enabled with 2.6.1 on these two machines?

-- 
	Paolo Ornati
	Linux v2.6.2-rc1-mm3

Re: i/o wait eating all of CPU on 2.6.1

[Jaakko Helminen]

On Sun, Jan 25, 2004 at 03:37:50PM +0100, Paolo Ornati wrote:
> 
> Is DMA enabled with 2.6.1 on these two machines?
> 
> -- 
> 	Paolo Ornati
> 	Linux v2.6.2-rc1-mm3
> 

Oh yes, I forgot to mention it, DMA is enabled on both machines.

-Jaakko Helminen

Re: i/o wait eating all of CPU on 2.6.1

[Ed Tomlinson]

Hi,

A couple of things.  IO wait does not 'eat cpu'.  Its a wait state, recorded
when the box has nothing to do and IO is pending.  That being said I remember
reading here about nfs bugs in 2.6.1 that slow it down.  A search should
find some patches for you to try.

Ed

On January 25, 2004 09:30 am, Jaakko Helminen wrote:
> I have two servers, both of which have more than 300 gigabytes of hard
> drive space and those files are made available to the network with samba,
> nfs and http and it worked fine with 2.6.0 but when I upgraded to 2.6.1 I
> noticed that everything was VERY slow, from a machine that is connected to
> the other server with a 100M link, 57kB/s tops. i/o wait eats up all of the
> cpu. On the other hand, Apache (and everything else) works very fast when I
> only send /dev/zero to a client, since that doesn't need disk operations.
>
> I don't notice anything suspicious in dmesg but since this happens on two
> machines and has only happened when upgraded to 2.6.1, it's most likely
> because of 2.6.1. I'm downgrading to 2.6.0 (with mremap-patch) today if I
> don't figure out what is wrong. Any ideas?
>
> And since I'm not subscribed to Linux Kernel Mailing List, please forward
> any replies to me.
>
>
> -Jaakko Helminen
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: i/o wait eating all of CPU on 2.6.1

[bill davidsen]

In article <20040125143042.GA20274@ihme.org>,
Jaakko Helminen  <haukkari@ihme.org> wrote:
| 
| I have two servers, both of which have more than 300 gigabytes of hard drive
| space and those files are made available to the network with samba, nfs and
| http and it worked fine with 2.6.0 but when I upgraded to 2.6.1 I noticed
| that everything was VERY slow, from a machine that is connected to the other
| server with a 100M link, 57kB/s tops. i/o wait eats up all of the cpu.
| On the other hand, Apache (and everything else) works very fast when I only
| send /dev/zero to a client, since that doesn't need disk operations.

Actually, io-wait is an indication that the CPU is idle with io
outstanding, and doesn't eat the machine any more than idle time. It is
an indication that the io is not keeping up, of course.

The fact that sending /dev/zero is fast sort of eliminates the reported
problems with NFS and setting the read/write size to 8k on the client.
Just for grins, hopefully you have followed the "NFS is slow" thread,
but I saw people having the issue with 2.6.0, so that's not likely to be
be involved.

I have some preliminary numbers which indicate 2.6.2-rc2 is faster at
some disk operations than 2.6.1, but since that's one benchmark on one
machine all I can say is that if you want to move forward, that kernel
is working well for me. Of course if there's nothing in 2.6.1 you
really want or need, falling back is a sure cure.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.