Re: Misleading OOM messages

[Dave Hansen <dave@linux.vnet.ibm.com>]

On Thu, 2009-05-14 at 23:34 +0200, Pavel Machek wrote:
> On Thu 2009-05-14 17:30:02, Christoph Lameter wrote:
> > On Thu, 14 May 2009, Dave Hansen wrote: 
> > > -	printk(KERN_ERR "%s: kill process %d (%s) score %li or a child\n",
> > > +	printk(KERN_ERR "No available memory %s: "
> > > +			"kill process %d (%s) score %li or a child\n",
> > >  					message, task_pid_nr(p), p->comm, points);
> > 
> > "No available memory" still suggests that plugging in more memory is the
> > right solution.
> 
> And... on correctly working kernel, it is, right?
> 
> If you have no swap space and too many applications, you plug more
> memory. (Or invent some swap).
> 
> If you misconfigured cgroups, you give more memory to them.
> 
> If your applications mlocked 900MB and you have 1GB, you need to plug
> more memory.
> 
> So... when is plugging more memory _not_ valid answer? AFAICT it is
> when it is some kernel problem, resulting in memory not being
> reclaimed fast enough....

I recently had a problem (~2.6.27) where the user did an mlock() of ~95%
of memory then started doing ftp tests.  The machine also had 64k base
pages.  We let you dirty ~30% of memory, so they were able to dirty 6x
more memory than what we even had to work with.  We OOMed pretty fast
every time.

Now, that situation never gets better when you add more memory.  It only
gets worse because that "30% of memory number" takes longer and longer
to write out to the disk.

This is actually a pretty common scenario for the HPC and database
folks.  They go sucking up and locking as much memory as they can get
their hands on.  Adding memory never helps them because they'll use up
whatever is there.

-- Dave