Re: high system cpu load during intense disk i/o

Andrew Morton
2007-08-06T17:33:36Z

On Mon, 06 Aug 2007 16:20:30 +0200 Dimitrios Apostolou <jimis@gmx.net> wrote:

> Andrew Morton wrote:
> > On Sun, 5 Aug 2007 19:03:12 +0300 Dimitrios Apostolou <jimis@gmx.net> wrote:
> >
> >> was my report so complicated?
> >
> > We're bad.
> >
> > Seems that your context switch rate when running two instances of
> > badblocks against two different disks went batshit insane. It doesn't
> > happen here.
> >
> > Please capture the `vmstat 1' output while running the problematic
> > workload.
> >
> > The oom-killing could have been unrelated to the CPU load problem. iirc
> > badblocks uses a lot of memory, so it might have been genuine. Keep an eye
> > on the /proc/meminfo output and send the kernel dmesg output from the
> > oom-killing event.
>
> Please see the attached files. Unfortunately I don't see any useful info
> in them:
> *_before: before running any badblocks process
> *_while: while running badblocks process, but without any cron job
> having kicked in
> *_bad: 5 minutes later that some cron jobs kicked in
>
> About the OOM killer, indeed I believe that it is unrelated. It started
> killing after about 2 days, that hundreds of processes were stuck as
> running and taking up memory, so I suppose the 256 MB RAM were truly
> filled. I just mentioned it because its behaviour is completely
> non-helpful. It doesn't touch the badblocks process, it rarely touches
> the stuck as running cron jobs, but it kills other irrelevant processes.
> If you still want the killing logs, tell me and I'll search for them.

ah. Your context-switch rate during the dual-badblocks run is not high at
all.

I suspect I was fooled by the oprofile output, which showed tremendous
amounts of load in schedule() and switch_to(). The percentages which
opreport shows are the percentage of non-halted CPU time. So if you have a
function in the kernel which is using 1% of the total CPU, and the CPU is
halted for 95% of the time, it appears that the function is taking 20% of
CPU!

The fix for that is to boot with the "idle=poll" boot parameter, to make
the CPU spin when it has nothing else to do.

I'm suspecting that your machine is just stuck in D state waiting for disk.
Did we have a sysrq-T trace?