[3.2.2] oom + no-killable-processes.

[3.2.2] oom + no-killable-processes.

[Paweł Sikora]

Hi all,
 
i've found on my 8-core opterons following oops (oom+nokillable): http://imgbin.org/index.php?page=image&id=6945
usualy the kernel just kills processes from userspace but this time something has gone wrong...
sysrq was dead, so no more stacktraces :/

Re: [3.2.2] oom + no-killable-processes.

[Greg KH]

On Wed, Feb 29, 2012 at 08:28:23PM +0100, Paweł Sikora wrote:
> Hi all,
> 
> i've found on my 8-core opterons following oops (oom+nokillable): http://imgbin.org/index.php?page=image&id=6945
> usualy the kernel just kills processes from userspace but this time something has gone wrong...
> sysrq was dead, so no more stacktraces :/

Is this something new with 3.2.2?  If so, can you bisect it down to the
patch that causes the problem?  Does 3.2.0 have this issue?  3.2.8?

thanks,

greg k-h

Re: [3.2.2] oom + no-killable-processes.

[Paweł Sikora]

On Wednesday 29 of February 2012 12:09:10 Greg KH wrote:
> On Wed, Feb 29, 2012 at 08:28:23PM +0100, Paweł Sikora wrote:
> > Hi all,
> > 
> > i've found on my 8-core opterons following oops (oom+nokillable): http://imgbin.org/index.php?page=image&id=6945
> > usualy the kernel just kills processes from userspace but this time something has gone wrong...
> > sysrq was dead, so no more stacktraces :/
> 
> Is this something new with 3.2.2?

i see such nokillable oom first time. the 3.1.x stable has been working for few months.
as you can see on the oops timestamp the 3.2.2 failed after ~27.5d so it isn't an immediate
crash which can be easily bisected :/ currently i'm stressing the 3.2.7...

Re: [3.2.2] oom + no-killable-processes.

[David Rientjes]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1407 bytes --]

On Wed, 29 Feb 2012, Paweł Sikora wrote:

> > > Hi all,
> > > 
> > > i've found on my 8-core opterons following oops (oom+nokillable): http://imgbin.org/index.php?page=image&id=6945
> > > usualy the kernel just kills processes from userspace but this time something has gone wrong...
> > > sysrq was dead, so no more stacktraces :/
> > 
> > Is this something new with 3.2.2?
> 
> i see such nokillable oom first time. the 3.1.x stable has been working for few months.
> as you can see on the oops timestamp the 3.2.2 failed after ~27.5d so it isn't an immediate
> crash which can be easily bisected :/ currently i'm stressing the 3.2.7...
> 

The trace you posted isn't very useful because all the important 
information has already scrolled off the screen.

This only happens if all eligible threads are oom disabled.  If you're not 
using cpusets or mempolicies or the memory controller, check 
/proc/pid/oom_score for all pids.  Those with a score of '0' are 
unkillable.  If there are threads that should be eligible, check 
/proc/pid/oom_score_adj for that pid.  If it is -1000, then userspace has 
probably disabled oom killing for it unnecessarily.

If you're using cpusets or the memory controller, then the system will 
panic if the oom occurred because all cpuset mems or the memcg limit is 
exhausted and there are also no threads in that cpuset or memcg that are 
killable (same process above).