What causes iowait other than waiting for i/o?

What causes iowait other than waiting for i/o?

[Bill Davidsen]

I recently noted that my system was spending a lot of time in i/o wait 
when doing some tasks which I thought didn't involve i/o, as noted by 
the lack of disk light activity most of the time. I thought of network, 
certainly the NIC had no activity for this job. So I set up a little 
loop to capture all disk i/o and network activity (including loopback). 
That was no obvious help, and the program doesn't use pipes.

At this point I'm really curious, does someone have a good clue?

Note: I don't think this is a bug or performance issue, unless the 
kernel is doing something and charging time to iowait instead of system 
I don't see anything to fix, but I would like to understand.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: What causes iowait other than waiting for i/o?

[Jesper Juhl]

On 28/05/07, Bill Davidsen <davidsen@tmr.com> wrote:
> I recently noted that my system was spending a lot of time in i/o wait
> when doing some tasks which I thought didn't involve i/o, as noted by
> the lack of disk light activity most of the time. I thought of network,
> certainly the NIC had no activity for this job. So I set up a little
> loop to capture all disk i/o and network activity (including loopback).
> That was no obvious help, and the program doesn't use pipes.
>
> At this point I'm really curious, does someone have a good clue?
>
> Note: I don't think this is a bug or performance issue, unless the
> kernel is doing something and charging time to iowait instead of system
> I don't see anything to fix, but I would like to understand.
>

Nothing /should/ cause time to be attributed to i/o wait other than
waiting for i/o. But the accounting isn't completely accurate. When
the kernel records what the has happen (governed by when the timer
interrupt triggers as far as I remember), many different processes may
have run and done may different things, but only what the last one did
is recorded. So if you are unlucky you may have the system doing all
sorts of activities, but only recording i/o wait if the last process
to run before the kernel looked was doing i/o wait...  The numbers are
not exactely accurate...


-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: What causes iowait other than waiting for i/o?

[Satyam Sharma]

Hi Bill,

On 5/29/07, Bill Davidsen <davidsen@tmr.com> wrote:
> I recently noted that my system was spending a lot of time in i/o wait
> when doing some tasks which I thought didn't involve i/o, as noted by
> the lack of disk light activity most of the time. I thought of network,
> certainly the NIC had no activity for this job. So I set up a little
> loop to capture all disk i/o and network activity (including loopback).
> That was no obvious help, and the program doesn't use pipes.
>
> At this point I'm really curious, does someone have a good clue?
>
> Note: I don't think this is a bug or performance issue, unless the
> kernel is doing something and charging time to iowait instead of system
> I don't see anything to fix, but I would like to understand.

What tool / kernel instrumentation / mechanism are you using to
determine that some task(s) are indeed blocked waiting for i/o? Perhaps
some userspace process accounting tools could be "broken" in the sense
that they generalize all uninterruptible sleep as waiting for i/o ...

Satyam

Re: What causes iowait other than waiting for i/o?

[Bill Davidsen]

Satyam Sharma wrote:
> Hi Bill,
>
> On 5/29/07, Bill Davidsen <davidsen@tmr.com> wrote:
>> I recently noted that my system was spending a lot of time in i/o wait
>> when doing some tasks which I thought didn't involve i/o, as noted by
>> the lack of disk light activity most of the time. I thought of network,
>> certainly the NIC had no activity for this job. So I set up a little
>> loop to capture all disk i/o and network activity (including loopback).
>> That was no obvious help, and the program doesn't use pipes.
>>
>> At this point I'm really curious, does someone have a good clue?
>>
>> Note: I don't think this is a bug or performance issue, unless the
>> kernel is doing something and charging time to iowait instead of system
>> I don't see anything to fix, but I would like to understand.
>
> What tool / kernel instrumentation / mechanism are you using to
> determine that some task(s) are indeed blocked waiting for i/o? Perhaps
> some userspace process accounting tools could be "broken" in the sense
> that they generalize all uninterruptible sleep as waiting for i/o ...

I wouldn't expect /proc/stat and similar to be broken in that way, but 
If no one has a better idea I guess I will assume there's a check needed 
of where time is added to iowait. I was hoping to avoid a full kernel 
search. Never thought of /proc data as a user space tool, but I guess.

-- 
Bill Davidsen <davidsen@tmr.com>
  "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: What causes iowait other than waiting for i/o?

[Rik van Riel]

Bill Davidsen wrote:
> I recently noted that my system was spending a lot of time in i/o wait 
> when doing some tasks which I thought didn't involve i/o, as noted by 
> the lack of disk light activity most of the time. I thought of network, 
> certainly the NIC had no activity for this job. So I set up a little 
> loop to capture all disk i/o and network activity (including loopback). 
> That was no obvious help, and the program doesn't use pipes.
> 
> At this point I'm really curious, does someone have a good clue?
> 
> Note: I don't think this is a bug or performance issue, unless the 
> kernel is doing something and charging time to iowait instead of system 
> I don't see anything to fix, but I would like to understand.

All filesystem IO and direct disk IO can cause iowait.

This includes NFS activity.

-- 
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

Re: What causes iowait other than waiting for i/o?

[Bill Davidsen]

Rik van Riel wrote:
> Bill Davidsen wrote:
>> I recently noted that my system was spending a lot of time in i/o 
>> wait when doing some tasks which I thought didn't involve i/o, as 
>> noted by the lack of disk light activity most of the time. I thought 
>> of network, certainly the NIC had no activity for this job. So I set 
>> up a little loop to capture all disk i/o and network activity 
>> (including loopback). That was no obvious help, and the program 
>> doesn't use pipes.
>>
>> At this point I'm really curious, does someone have a good clue?
>>
>> Note: I don't think this is a bug or performance issue, unless the 
>> kernel is doing something and charging time to iowait instead of 
>> system I don't see anything to fix, but I would like to understand.
>
> All filesystem IO and direct disk IO can cause iowait.
>
> This includes NFS activity.
>
If I didn't note it before, I'm read the the data from /proc, cpustats, 
net/dev, and diskstats. I assume that all i/o would show up in one of 
those places. NFS isn't involved, although this machine is a fileserver 
as a side job the modules weren't even loaded during testing.

A puzzlement for future consideration. If I get a chance later this week 
I'll make a pretty graphic of all the stuff going on when the iowait 
spiked, ctx rate, inq rate, hell the last time I even grabbed the CPU 
temp to see if it told me anything (didn't, thermal throttling NOT).

Thanks for the feedback, I think that lets out the obvious stuff.

-- 
Bill Davidsen <davidsen@tmr.com>
  "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot