Re: Idle loadavg of ~1, maybe MD related

[Andrew Morton <akpm@linux-foundation.org>]

On Sun, 30 Dec 2007 02:06:14 -0800 "Robin H. Johnson" <robbat2@gentoo.org> wrote:

> (Please CC me, I'm subbed to LKML).
> 
> My G5, while running practically nothing (just sshd and some to watch the
> load), has a weird cycle of load averages. I think it might be related to MD,
> simply because that's the only thing that is clocking up cputime.

Usually this is caused by a process which is stuck in "D" state
(uninterruptible sleep) when it shouldn't be.

Check this by running `ps aux' or whatever.

> A full cycle lasts approximately 27 minutes.
> Mins	Load
> 0-2		0.0-0.15 (stable, 0 level)
> 3-5		0.50, 0.80, 0.95 (fast increase)
> 6-21	0.95-1.10 (stable, 1 level)
> 22-24   0.9, 0.8, 0.1 (fast decrease, to 0 level)
> 25-27   0.2, 0.3, 0.15 (local maxima peak)
> 
> Here's a graph of it, spanning 230 minutes:
> http://dev.gentoo.org/~robbat2/20071230-g5-loadavg-bug.png
> 
> Processed data for the graph here:
> http://dev.gentoo.org/~robbat2/20071230-g5-loadavg-bug.txt
> 
> For the entire 230 minute period, there was _no_ disk I/O.
> Not recorded by iostat, nor generated.
> 
> # while true ; do uptime ; iostat -t 60 2  -N -d | tail -n15 ; done >/dev/shm/foo
> 
> Example of single output pass for the above loop:
>  00:59:37 up  8:32,  2 users,  load average: 0.02, 0.47, 0.66
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               0.00         0.00         0.00          0          0
> sdb               0.00         0.00         0.00          0          0
> md1               0.00         0.00         0.00          0          0
> md0               0.00         0.00         0.00          0          0
> md2               0.00         0.00         0.00          0          0
> md3               0.00         0.00         0.00          0          0
> vg-usr            0.00         0.00         0.00          0          0
> vg-var            0.00         0.00         0.00          0          0
> vg-tmp            0.00         0.00         0.00          0          0
> vg-opt            0.00         0.00         0.00          0          0
> vg-home           0.00         0.00         0.00          0          0
> vg-usr_src        0.00         0.00         0.00          0          0
> vg-usr_portage     0.00         0.00         0.00          0          0
> 
> This is basically 1842c7f2 from Linus's tree, my own stuff is config'd out with
> =n for the moment. And the problem does still occur in the main tree.
> 
> Snippet from the head of 'top', sorting by cputime.
> 
> top - 01:59:08 up  9:32,  2 users,  load average: 1.04, 0.87, 0.70
> Tasks:  74 total,   1 running,  73 sleeping,   0 stopped,   0 zombie
> Cpu0  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu1  :  0.0%us,  0.5%sy,  0.0%ni, 99.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu2  :  0.0%us,  0.2%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Mem:  12074480k total,   292520k used, 11781960k free,    76812k buffers
> Swap:  8388536k total,        0k used,  8388536k free,   144276k cached
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  4635 root      15  -5     0    0    0 S    0  0.0   8:14.33 md3_raid1
>  3121 root      15  -5     0    0    0 S    1  0.0   3:15.12 md2_raid1
>  3098 root      15  -5     0    0    0 S    0  0.0   3:07.83 md1_raid1
>  3076 root      15  -5     0    0    0 S    0  0.0   0:45.82 md0_raid1
>   829 root      15  -5     0    0    0 D    0  0.0   0:01.85 kwindfarm
>    13 root      15  -5     0    0    0 S    0  0.0   0:01.41 ksoftirqd/3
>    18 root      15  -5     0    0    0 S    0  0.0   0:01.39 events/3
>    10 root      15  -5     0    0    0 S    0  0.0   0:00.94 ksoftirqd/2
>     1 root      20   0  1900  652  576 S    0  0.0   0:00.89 init
> 32086 root      20   0  9336 2696 2076 S    0  0.0   0:00.87 sshd

>From that I'd suspect that kwindfarm is being a bad citizen.

If a process is consistently stuck in D state, run

	echo w > /proc/sysrq-trigger

then record the resulting dmesg output so we can see where it got stuck.