tmpfs and the OOM killer

tmpfs and the OOM killer

[Pedro]

  After suffering some days from a not|mis configured tmpfs,

  As the OOM killer is not Posix,

  Better than to kill processes would be to resize tmpfs, to use tmpfs empty 
space.

  I'm using kernel 2.6.20.4. If someone ask I'll send a test application.

Re: tmpfs and the OOM killer

[Willy Tarreau]

On Wed, Apr 11, 2007 at 02:23:31AM -0300, Pedro wrote:
>   After suffering some days from a not|mis configured tmpfs,
> 
>   As the OOM killer is not Posix,
> 
>   Better than to kill processes would be to resize tmpfs, to use tmpfs empty 
> space.

Will not work, because tmpfs does not use any memory for unused space. If
you don't believe me, simply create a large file on your tmpfs, then check
free memory, then remove the file and check free memory again.

So your problem is not caused by the empty space on tmpfs, but either by
too much space used on tmpfs or by your application using too much memory.

>   I'm using kernel 2.6.20.4. If someone ask I'll send a test application.

Not needed, the one-liner "main(){while(malloc(4096));}" is enough to
trigger an OOM.

If you cannot control your application's memory usage, you'll have to finely
tune the overcommit_ratio.

Regards,
Willy

RE: tmpfs and the OOM killer

[Mouawad, Tony]

Can someone describe the process of finding the best value to tune the
overcommit_ratio to?

-----Original Message-----
From: linux-kernel-owner@vger.kernel.org
[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Willy Tarreau
Sent: Wednesday, April 11, 2007 3:48 PM
To: Pedro
Cc: linux-kernel@vger.kernel.org
Subject: Re: tmpfs and the OOM killer

On Wed, Apr 11, 2007 at 02:23:31AM -0300, Pedro wrote:
>   After suffering some days from a not|mis configured tmpfs,
> 
>   As the OOM killer is not Posix,
> 
>   Better than to kill processes would be to resize tmpfs, to use tmpfs
empty 
> space.

Will not work, because tmpfs does not use any memory for unused space.
If
you don't believe me, simply create a large file on your tmpfs, then
check
free memory, then remove the file and check free memory again.

So your problem is not caused by the empty space on tmpfs, but either by
too much space used on tmpfs or by your application using too much
memory.

>   I'm using kernel 2.6.20.4. If someone ask I'll send a test
application.

Not needed, the one-liner "main(){while(malloc(4096));}" is enough to
trigger an OOM.

If you cannot control your application's memory usage, you'll have to
finely
tune the overcommit_ratio.

Regards,
Willy

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: tmpfs and the OOM killer

[Pedro]

On Wednesday 11 April 2007 16:48, Willy Tarreau wrote:
> On Wed, Apr 11, 2007 at 02:23:31AM -0300, Pedro wrote:
> >   After suffering some days from a not|mis configured tmpfs,
> >
> >   As the OOM killer is not Posix,
> >
> >   Better than to kill processes would be to resize tmpfs, to use tmpfs
> > empty space.
>
> Will not work, because tmpfs does not use any memory for unused space. If
> you don't believe me, simply create a large file on your tmpfs, then check
> free memory, then remove the file and check free memory again.
>
> So your problem is not caused by the empty space on tmpfs, but either by
> too much space used on tmpfs or by your application using too much memory.
>
...
>
> If you cannot control your application's memory usage, you'll have to
> finely tune the overcommit_ratio.
>
> Regards,
> Willy

  You are right.

  But now I have two questions:

  1) Why is tmpfs total space fixed if at the check moment does not exist 
sufficient memory?

  2) How should an application be written to not be killed by OOM?

Re: tmpfs and the OOM killer

[Alan Cox]

>   2) How should an application be written to not be killed by OOM?

OOM isn't an application matter. The kernel has to choose between
allowing overcommit on the basis it might run out of memory and have to
kill stuff, or that it won't in which case an applicatio which correctly
handles malloc() and similar failures will not be killed (unless it is
out of space on a stack grow which is a C language flaw as you can't
catch that event in C)

It's configured by /proc/sys/vm/overcommit_memory

0 - try and spot obviously dumb allocations
1 - anything goes
2 - strictly control resource commit

Re: tmpfs and the OOM killer

[Pedro]

On Wednesday 11 April 2007 19:39, Alan Cox wrote:
> >   2) How should an application be written to not be killed by OOM?
>
> OOM isn't an application matter. The kernel has to choose between
> allowing overcommit on the basis it might run out of memory and have to
> kill stuff, or that it won't in which case an applicatio which correctly
> handles malloc() and similar failures will not be killed (unless it is
> out of space on a stack grow which is a C language flaw as you can't
> catch that event in C)
>
> It's configured by /proc/sys/vm/overcommit_memory
>
> 0 - try and spot obviously dumb allocations
> 1 - anything goes
> 2 - strictly control resource commit

  I deduce that a fail-safe application must scanf overcommit_memory, warn 
the user and waitpid.

Re: tmpfs and the OOM killer

[Alan Cox]

> > 0 - try and spot obviously dumb allocations
> > 1 - anything goes
> > 2 - strictly control resource commit
> 
>   I deduce that a fail-safe application must scanf overcommit_memory, warn 
> the user and waitpid.

If you are building a fail safe system you need to look a bit beyond out
of memory handling settings - power supplies, failover, fault tolerance
requirements, error detection (eg ECC ram), raid arrays over
non-electrically conductive links etc.

Alan

Re: tmpfs and the OOM killer

[Pedro]

I wrote:
> >   I deduce that a fail-safe application must scanf overcommit_memory,
> > warn the user and waitpid.

Alan Cox wrote:
> If you are building a fail safe system you need to look a bit beyond out
> of memory handling settings - power supplies, failover, fault tolerance
> requirements, error detection (eg ECC ram), raid arrays over
> non-electrically conductive links etc.

  If you lived here, you'ld be worried about seismic-electric impact on your 
HDD and data ;) So SIGKILL is not the hardest problem to be solved.

  Maybe it is time to write an avoidoomd.

Re: tmpfs and the OOM killer

[Theodore Tso]

On Thu, Apr 12, 2007 at 02:19:02AM -0300, Pedro wrote:
> > OOM isn't an application matter. The kernel has to choose between
> > allowing overcommit on the basis it might run out of memory and have to
> > kill stuff, or that it won't in which case an applicatio which correctly
> > handles malloc() and similar failures will not be killed (unless it is
> > out of space on a stack grow which is a C language flaw as you can't
> > catch that event in C)
> >
> > It's configured by /proc/sys/vm/overcommit_memory
> >
> > 0 - try and spot obviously dumb allocations
> > 1 - anything goes
> > 2 - strictly control resource commit
> 
>   I deduce that a fail-safe application must scanf overcommit_memory, warn 
> the user and waitpid.

If a fail-safe applicaion is running on a system which is that close
to the edge in terms of available physical memory and swap, it's not
likely going to be in deep trouble anyway.  Even if you disable the
OOM killer, now random malloc()'s will start returning NULL because
your system doesn't have enough memory.  Do you have intelligent error
handling and recovery mechanisms for every single malloc() failure?
Also, the machine will likely be thrashing so badly that any service
level performance guarantees that the application might have will
probably be totally trashed.

						- Ted

Re: tmpfs and the OOM killer

[Pedro]

On Thursday 12 April 2007 08:25, Theodore Tso wrote:
> likely going to be in deep trouble anyway.  Even if you disable the
> OOM killer, now random malloc()'s will start returning NULL because
> your system doesn't have enough memory.  Do you have intelligent error
> handling and recovery mechanisms for every single malloc() failure?

  When malloc return NULL, the process may tell the user ENOMEM.
  When OOM kill the process, the user claim the program sometimes die.

Re: tmpfs and the OOM killer

[Willy Tarreau]

On Thu, Apr 12, 2007 at 11:35:32AM -0300, Pedro wrote:
> On Thursday 12 April 2007 08:25, Theodore Tso wrote:
> > likely going to be in deep trouble anyway.  Even if you disable the
> > OOM killer, now random malloc()'s will start returning NULL because
> > your system doesn't have enough memory.  Do you have intelligent error
> > handling and recovery mechanisms for every single malloc() failure?
> 
>   When malloc return NULL, the process may tell the user ENOMEM.
>   When OOM kill the process, the user claim the program sometimes die.

Then use overcommit=2. The default overcommit mode is a convenience provided
to allow poorly designed applications run even when they pretend they need
gigs of RAM when they only use a few tens of megs.

If your application correctly handles malloc(), simply switch overcommit
to 2 and let the system refuse to allocate memory when none is available,
then your application will be aware of this by the NULL result to malloc()
calls. It is a normal behaviour.

I do have appliances which run perfectly controlled software with
overcommit_mode=2 and overcommit_ratio around 70% and without any
swap, and they work like a charm. It just requires some finer grained
tuning on your side. I don't see what the problem is here. You know
the app, you know how much RAM you want to allocate to it, you know
how much you want to keep free. Then say this to the system.

BTW, ulimit -v is your friend here too, and does not require to be root.

Regards,
Willy

RE: tmpfs and the OOM killer

[Mouawad, Tony]

I have noticed that with overcommit_memory=2 and overcommit_ratio=100,
my system cannot leverage as much ram as it could if it was configured
for overcommit_memory=0.

Is this because when overcommit_memory=2, anything that mallocs memory
but doesn't touch that memory is counted as used memory?  I see a value
in /proc/meminfo called Commited_AS: and it seems to reflect what has
been malloced in the system but not necessarily touched.  Is this true?

Cheers,
Tony


-----Original Message-----
From: linux-kernel-owner@vger.kernel.org
[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Willy Tarreau
Sent: Thursday, April 12, 2007 4:32 PM
To: Pedro
Cc: linux-kernel@vger.kernel.org
Subject: Re: tmpfs and the OOM killer

On Thu, Apr 12, 2007 at 11:35:32AM -0300, Pedro wrote:
> On Thursday 12 April 2007 08:25, Theodore Tso wrote:
> > likely going to be in deep trouble anyway.  Even if you disable the
> > OOM killer, now random malloc()'s will start returning NULL because
> > your system doesn't have enough memory.  Do you have intelligent
error
> > handling and recovery mechanisms for every single malloc() failure?
> 
>   When malloc return NULL, the process may tell the user ENOMEM.
>   When OOM kill the process, the user claim the program sometimes die.

Then use overcommit=2. The default overcommit mode is a convenience
provided
to allow poorly designed applications run even when they pretend they
need
gigs of RAM when they only use a few tens of megs.

If your application correctly handles malloc(), simply switch overcommit
to 2 and let the system refuse to allocate memory when none is
available,
then your application will be aware of this by the NULL result to
malloc()
calls. It is a normal behaviour.

I do have appliances which run perfectly controlled software with
overcommit_mode=2 and overcommit_ratio around 70% and without any
swap, and they work like a charm. It just requires some finer grained
tuning on your side. I don't see what the problem is here. You know
the app, you know how much RAM you want to allocate to it, you know
how much you want to keep free. Then say this to the system.

BTW, ulimit -v is your friend here too, and does not require to be root.

Regards,
Willy

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: tmpfs and the OOM killer

[Willy Tarreau]

On Thu, Apr 12, 2007 at 04:56:24PM -0400, Mouawad, Tony wrote:
> I have noticed that with overcommit_memory=2 and overcommit_ratio=100,
> my system cannot leverage as much ram as it could if it was configured
> for overcommit_memory=0.
> 
> Is this because when overcommit_memory=2, anything that mallocs memory
> but doesn't touch that memory is counted as used memory?

Most probably, yes. This proves that your system may endup doing OOM
in mode 0 if one of your applications suddenly decided to use all the
memory it has allocated.

> I see a value
> in /proc/meminfo called Commited_AS: and it seems to reflect what has
> been malloced in the system but not necessarily touched.  Is this true?

Yes (at least I do think so).

Regards,
Willy

Re: tmpfs and the OOM killer

[Jan Engelhardt]

On Apr 11 2007 21:48, Willy Tarreau wrote:
>On Wed, Apr 11, 2007 at 02:23:31AM -0300, Pedro wrote:
>>   After suffering some days from a not|mis configured tmpfs,
>> 
>>   As the OOM killer is not Posix,
>> 
>>   Better than to kill processes would be to resize tmpfs, to use tmpfs empty 
>> space.
>
>Will not work, because tmpfs does not use any memory for unused space. If
>you don't believe me, simply create a large file on your tmpfs, then check
>free memory, then remove the file and check free memory again.
>
>So your problem is not caused by the empty space on tmpfs, but either by
>too much space used on tmpfs or by your application using too much memory.
>
>>   I'm using kernel 2.6.20.4. If someone ask I'll send a test application.
>
>Not needed, the one-liner "main(){while(malloc(4096));}" is enough to
>trigger an OOM.

No, that won't do anything, malloc happily returns NULL after a few seconds.

int main()
{
	while(1) {
		char *p = malloc(4096);
		*p = 1;
	}
}

This is more likely to trigger OOM, because it actually dirties the page.


Jan
-- 

Re: tmpfs and the OOM killer

[Willy Tarreau]

On Thu, Apr 12, 2007 at 10:13:05AM +0200, Jan Engelhardt wrote:
> 
> On Apr 11 2007 21:48, Willy Tarreau wrote:
> >On Wed, Apr 11, 2007 at 02:23:31AM -0300, Pedro wrote:
> >>   After suffering some days from a not|mis configured tmpfs,
> >> 
> >>   As the OOM killer is not Posix,
> >> 
> >>   Better than to kill processes would be to resize tmpfs, to use tmpfs empty 
> >> space.
> >
> >Will not work, because tmpfs does not use any memory for unused space. If
> >you don't believe me, simply create a large file on your tmpfs, then check
> >free memory, then remove the file and check free memory again.
> >
> >So your problem is not caused by the empty space on tmpfs, but either by
> >too much space used on tmpfs or by your application using too much memory.
> >
> >>   I'm using kernel 2.6.20.4. If someone ask I'll send a test application.
> >
> >Not needed, the one-liner "main(){while(malloc(4096));}" is enough to
> >trigger an OOM.
> 
> No, that won't do anything, malloc happily returns NULL after a few seconds.
> 
> int main()
> {
> 	while(1) {
> 		char *p = malloc(4096);
> 		*p = 1;
> 	}
> }
> 
> This is more likely to trigger OOM, because it actually dirties the page.

Yes, you're right, and that's indeed what my "freemem" program does ;-)

Willy

Re: tmpfs and the OOM killer

[Al Boldi]

Pedro wrote:
> On Wednesday 11 April 2007 16:48, Willy Tarreau wrote:
> > On Wed, Apr 11, 2007 at 02:23:31AM -0300, Pedro wrote:
> > >
> > >   As the OOM killer is not Posix,
> >
> > If you cannot control your application's memory usage, you'll have to
> > finely tune the overcommit_ratio.
>
>   2) How should an application be written to not be killed by OOM?


Try this:

# echo -17 > /proc/<pid>/oom_adj

Or this:

# echo 2 > /proc/sys/vm/overcommit_memory
# echo 95 > /proc/sys/vm/overcommit_ratio

Or this:

# ulimit -v [max vm]


Thanks, and good luck with the OOM killer!


--
Al

Re: tmpfs and the OOM killer

[Pedro]

On Thursday 12 April 2007 02:04, Al Boldi wrote:
> > Pedro wrote:
> >   2) How should an application be written to not be killed by OOM?
>
> Try this:
>
> # echo -17 > /proc/<pid>/oom_adj

  I should know that to run a fail-safe application is a superuser privilege.

  Sorry from wasting your time.