Re: iotop: khugepaged at 99.99% (2.6.38.3)

[Gilles Hamel <hamelg@laposte.net>]

On 12/05/2011 16:03, Andrea Arcangeli wrote:
> On Wed, May 11, 2011 at 10:53:18AM +0000, Ulrich Keller wrote:
>> I am seeing exactly the same symptoms on my Lenovo T60 Core2 duo, 3GB RAM,
>> running Arch Linux i686 with Kernel 2.6.38.6. When I've heavily used Firefox for
>> a while, or used R with high memory usage (>1 GB), individual applications
>> become unresponsive, new processes fail to start and after a while the whole
>> system freezes. When it happens, iotop shows khugepaged and sometimes firefox at
>> 99.99%.
> SYSRQ+T run multiple times during the hang and /proc/zoneinfo as well
> run multiple times during the hang is the best info we can have for
> now, /proc/zoneinfo is the most interesting as it will show us the
> values that the too_many_isolated loop is checking to decide if to
> continue looping.
Me too :(
Since running 2.6.38, it was happening only 3 times on the same process 
(convert from the ImageMagick toolkit). The last time I'm running 2.6.38.7.
This process is launched every 15 minutes by crond, like this :

*/15 * * * * convert -delay 50 
http://www.meteo60.org/radars/radar-nord-picardie-idf{-90,-90,-90,-75,-60,-45,-30,-15,,,}.png 
-loop 0 $HOME/temp/radar-pluie.gif >/dev/null 2>&1
*/15 * * * * convert -delay 50 
http://www.sat24.com/image.ashx\?ok=1\&country=fr\&type=slide\&time=\&index={9,9,9,8,7,6,5,4,3,2,1,1,1}\&sat=vis 
-loop 0 $HOME/temp/radar-nuage.gif >/dev/null 2>&1

When it's happen, I'm using firefox. Here, the whole system continues 
functioning normally a moment, then the X server hangs. Only these 3 
tasks was stuck at 99% io busy in iotop :

$ iotop -ob
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
    26 be/7 root        0.00 B/s    0.00 B/s  0.00 % 96.84 % [khugepaged]
22839 be/4 hamelg      0.00 B/s    0.00 B/s  0.00 % 96.84 % convert 
-delay 50 http://www.meteo60....
22841 be/4 hamelg      0.00 B/s    0.00 B/s  0.00 % 96.84 % convert 
-delay 50 http://www.sat24....
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
22839 be/4 hamelg      0.00 B/s    0.00 B/s  0.00 % 99.99 % convert 
-delay 50 http://www.meteo60...
    26 be/7 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [khugepaged]
22841 be/4 hamelg      0.00 B/s    0.00 B/s  0.00 % 99.99 % convert 
-delay 50 http://www.sat24...
...

Before rebooting, I followed your hints. You'll find the output of 
multiple SYSRQ+T, /proc/zoneinfo and ps axu and my config.gz :

http://gilles.hamel.free.fr/config.gz
http://gilles.hamel.free.fr/typescript
http://gilles.hamel.free.fr/sysrq+t.txt

I hope these additional clues will help you to hunt this bug.

Regards