2.6.27 inotify causes unkillable processes with 100% CPU usage

2.6.27 inotify causes unkillable processes with 100% CPU usage

[Timo Sirainen]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

There appears to be a bug in 2.6.27 and inotify where it causes  
processes in running state to eat 100% CPU and they can't be killed  
with kill -9, requiring a reboot to get rid of them.

3 people (Cc'd) so far have reported this to me, all of them last  
week. The problem shows up when running my Dovecot IMAP server. Its  
imap processes start hanging, apparently it can happen after running  
for only a couple of minutes:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
6717 arno      20   0  2964 1608 1192 R  100  0.2   1158:05 imap

Only the imap processes hang (Dovecot has a few others). The only  
thing special about it is that it uses inotify. After disabling  
Dovecot's inotify support at least one of the people (Arno) reported  
that he can't reprocuce the bug anymore. He also couldn't reproduce  
the bug with 2.6.26 kernel.

I don't think Dovecot does anything special with its inotify code. It  
just listens on two directories. You can find the code here: http://hg.dovecot.org/dovecot-1.1/file/f9a000ee400d/src/lib/ioloop-notify-inotify.c

Kernels causing the hang:

2.6.27-9.slh.1-sidux-686
Arch Linux, kernel 2.6.27.8
Gentoo with 2.6.27-gentoo-r5, amd64, 2 dual core opterons

Not causing the hang:

2.6.26-6.slh.1-sidux-686


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: 2.6.27 inotify causes unkillable processes with 100% CPU usage

[Al Viro]

On Mon, Dec 15, 2008 at 01:31:29PM +0200, Timo Sirainen wrote:
> There appears to be a bug in 2.6.27 and inotify where it causes processes 
> in running state to eat 100% CPU and they can't be killed with kill -9, 
> requiring a reboot to get rid of them.

> Kernels causing the hang:
>
> 2.6.27-9.slh.1-sidux-686
> Arch Linux, kernel 2.6.27.8
> Gentoo with 2.6.27-gentoo-r5, amd64, 2 dual core opterons
>
> Not causing the hang:
>
> 2.6.26-6.slh.1-sidux-686

Could you try to bisect that?  Particulary interesting commits:

8f7b0ba1c853919b85b54774775f567f30006107
6ff2d39b91aec3dcae951afa982059e3dd9b49dc
711a49a07f84f914aac26a52143f6e7526571143

Re: 2.6.27 inotify causes unkillable processes with 100% CPU usage

[Holger Hoffstaette]

On Mon, 15 Dec 2008 12:05:12 +0000, Al Viro wrote:

> On Mon, Dec 15, 2008 at 01:31:29PM +0200, Timo Sirainen wrote:
>> There appears to be a bug in 2.6.27 and inotify where it causes
>> processes in running state to eat 100% CPU and they can't be killed with
>> kill -9, requiring a reboot to get rid of them.
> 
>> Kernels causing the hang:
>>
>> 2.6.27-9.slh.1-sidux-686
>> Arch Linux, kernel 2.6.27.8
>> Gentoo with 2.6.27-gentoo-r5, amd64, 2 dual core opterons
>>
>> Not causing the hang:
>>
>> 2.6.26-6.slh.1-sidux-686
> 
> Could you try to bisect that?  Particulary interesting commits:

Please see this posting, which also contains a pointer to the
necessary patch:
http://lkml.indiana.edu/hypermail/linux/kernel/0812.1/02305.html

According to GregKH it will be in -stable.10, but in the meantime you can
also apply the  patch from http://tinyurl.com/5gpqkj on top stable .8
(which introduced the regression) or .9.

At this point it is not clear whether the changes to inotify caused this
or just fell victim to the parallel regression in idr.

-h