OOM policy, overcommit control, and soft limits

OOM policy, overcommit control, and soft limits

[Chris Frey]

Hi,

The kernel provides things like ulimit, overcommit_memory, and the OOM
killer notifications, so in theory memory management should not be a
problem, but from time to time, I have a real need to regain control of
my system when it runs away on me.

I like how mode 2 of overcommit_memory uses the ratio as a boundary limit.
Ideally I would like something like this as a soft limit, so once the
system gets that full, I get a warning.

Here's my ideal OOM flow:

	1) set my soft limit to 90% of RAM

	2) any malloc that hits this limit first runs through a notification
		hook, that talks to a userspace daemon if present,
		or just denies the malloc if not

	3) the daemon can decide whether to allow the allocation, going
		beyond the soft limit

	4) the daemon can make these decisions automatically based on
		policy (i.e. X always gets the green light), or if we
		want to get fancy it can talk to some pre-allocated
		GUI to present the decision to the user...
		(i.e. Allocate / Deny / Stop / Kill)

	5) if the user foolishly keeps allocating, then the current
		OOM killer comes into play

I'm sure someone has thought of this before me.  Does anything remotely
similar to this already exist?  I've googled for OOM policy, but so far
all I've seen is Rusty Lynch's patch from 2003, and really, I want this
behaviour to happen when there is still a bit of memory left, so things
can be dealt with before they are OOM-level dire.

Thanks in advance,
- Chris

Re: OOM policy, overcommit control, and soft limits

[Alan Cox]

> I'm sure someone has thought of this before me.  Does anything remotely
> similar to this already exist?  I've googled for OOM policy, but so far
> all I've seen is Rusty Lynch's patch from 2003, and really, I want this
> behaviour to happen when there is still a bit of memory left, so things
> can be dealt with before they are OOM-level dire.

Not that I am aware of. However you can set priorities on OOM killing
order and there is a patch from the XO people which seems to have
vanished into the darkness which added a nice memory pressure notifier so
apps could respond to memory shortages intelligently (eg by cleaning up)

Re: OOM policy, overcommit control, and soft limits

[Andrea Righi]

Chris Frey wrote:
> I'm sure someone has thought of this before me.  Does anything remotely
> similar to this already exist?  I've googled for OOM policy, but so far
> all I've seen is Rusty Lynch's patch from 2003, and really, I want this
> behaviour to happen when there is still a bit of memory left, so things
> can be dealt with before they are OOM-level dire.

Have you seen the OOM killer policy implemented in memory the resource
controller?
http://kernelnewbies.org/Linux_2_6_25#head-450b26e12955b8035a05cf07b3f31c501ee4bfab

BTW read the TODO comment in this commit log... ;-)
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c7ba5c9e8176704bfac0729875fa62798037584d

Maybe a possible solution could be to just run critical and non-critical
applications in 2 different cgroups, using different memory policies.
Anyway, userspace OOM handling would surely permit to implement more
interesting features.

-Andrea

Re: OOM policy, overcommit control, and soft limits

[Alan Jenkins]

Chris Frey wrote:
> Hi,
>
> The kernel provides things like ulimit, overcommit_memory, and the OOM
> killer notifications, so in theory memory management should not be a
> problem, but from time to time, I have a real need to regain control of
> my system when it runs away on me.
>
> I like how mode 2 of overcommit_memory uses the ratio as a boundary limit.
> Ideally I would like something like this as a soft limit, so once the
> system gets that full, I get a warning.
>
> Here's my ideal OOM flow:
>
> 	1) set my soft limit to 90% of RAM
>
> 	2) any malloc that hits this limit first runs through a notification
> 		hook, that talks to a userspace daemon if present,
> 		or just denies the malloc if not
>
> 	3) the daemon can decide whether to allow the allocation, going
> 		beyond the soft limit
>
> 	4) the daemon can make these decisions automatically based on
> 		policy (i.e. X always gets the green light), or if we
> 		want to get fancy it can talk to some pre-allocated
> 		GUI to present the decision to the user...
> 		(i.e. Allocate / Deny / Stop / Kill)
>
> 	5) if the user foolishly keeps allocating, then the current
> 		OOM killer comes into play
>
> I'm sure someone has thought of this before me.  Does anything remotely
> similar to this already exist?  I've googled for OOM policy, but so far
> all I've seen is Rusty Lynch's patch from 2003, and really, I want this
> behaviour to happen when there is still a bit of memory left, so things
> can be dealt with before they are OOM-level dire.
>
> Thanks in advance,
> - Chris

I'm not sure how helpful this is.  It sounds like you may be missing
something, but I'm not competent to explain it.  But here are some of
my thoughts.

1. If your system is really running away, maybe killing the processes
responsible is a good idea.

2. Why do you want to set an _overcommit_ soft-limit at 90% of RAM?
Even without swap, that sounds very restrictrive.

If I naively add up the top 10 memory hogs on my 512M system, they're
using a total of over 4G of virtual memory.  (And I have less than
512M swap).  Some of that will be shared memory or backed by disk
files - but I don't think that it can account for all 3G of the
overcommit.  Shared memory tends to be code only, and my entire "disk"
is only 4G.

In other words, I reckon I have on the order of a gigabyte of virtual
address space, which has been malloc'ed or equivalent, but is not used
and therefore requires no memory resource (ram or swap).

3. Personally, without knowing whether it's already done, I think a
significant solution for the desktop is for browsers, mail clients and
similar to place strict limits on their anonymous memory consumption,
and use mmaped files for the caches / object stores which can grow so
large.  Similarly, the GIMP uses a "tile cache" file (which perhaps
shows its age).  If applications are storing caches on disk, they
should make this clear to the kernel - using mmapped files for cache
effectively turns them into swap files, letting the kernel page them
out to disk without using up the kernel swap file.  Exposing the cache
memory usage on disk also makes it slightly more accessible, e.g. to
disk quotas, or users with "du".

Re: OOM policy, overcommit control, and soft limits

[Alan Cox]

> In other words, I reckon I have on the order of a gigabyte of virtual
> address space, which has been malloc'ed or equivalent, but is not used
> and therefore requires no memory resource (ram or swap).

No need to reckon. The committed_as in the proc file should give a rough
value.

Alan

Re: OOM policy, overcommit control, and soft limits

[Alan Jenkins]

Alan Cox wrote:
>> In other words, I reckon I have on the order of a gigabyte of virtual
>> address space, which has been malloc'ed or equivalent, but is not used
>> and therefore requires no memory resource (ram or swap).
>>     
>
> No need to reckon. The committed_as in the proc file should give a rough
> value.
>
> Alan
>   
Thanks for the education.  I shall read up on the other numbers in 
/proc/meminfo as well.

In that case I was overly pessimistic.  I was only committed around the 
512M mark.  It jumps up to 750M if I open Amarok and Firefox though.  At 
times I've run more - I would guess I can contrive combinations which go 
above 1000M.


I've not had an OOM event on this machine.  I have had runaway 
development-related loads, causing thrashing (hitting swap) out of 
control, but I can't really comment.  I don't remember what caused it 
exactly.  Plus I'm swapping to a Flash drive; a Flash specific IO 
scheduler might have coped better (than noop) and made it easier to recover.

Alan