OOM policy, overcommit control, and soft limits

[Chris Frey <cdfrey@foursquare.net>]

Hi,

The kernel provides things like ulimit, overcommit_memory, and the OOM
killer notifications, so in theory memory management should not be a
problem, but from time to time, I have a real need to regain control of
my system when it runs away on me.

I like how mode 2 of overcommit_memory uses the ratio as a boundary limit.
Ideally I would like something like this as a soft limit, so once the
system gets that full, I get a warning.

Here's my ideal OOM flow:

	1) set my soft limit to 90% of RAM

	2) any malloc that hits this limit first runs through a notification
		hook, that talks to a userspace daemon if present,
		or just denies the malloc if not

	3) the daemon can decide whether to allow the allocation, going
		beyond the soft limit

	4) the daemon can make these decisions automatically based on
		policy (i.e. X always gets the green light), or if we
		want to get fancy it can talk to some pre-allocated
		GUI to present the decision to the user...
		(i.e. Allocate / Deny / Stop / Kill)

	5) if the user foolishly keeps allocating, then the current
		OOM killer comes into play

I'm sure someone has thought of this before me.  Does anything remotely
similar to this already exist?  I've googled for OOM policy, but so far
all I've seen is Rusty Lynch's patch from 2003, and really, I want this
behaviour to happen when there is still a bit of memory left, so things
can be dealt with before they are OOM-level dire.

Thanks in advance,
- Chris