[PATCH] OOM handling

[PATCH] OOM handling

[Martin Dalecki]

[-- Attachment #1: Type: text/plain, Size: 2087 bytes --]

Martin Dalecki wrote:
> 
> I have a constructive proposal:
> 
> It would make much sense to make the oom killer
> leave not just root processes alone but processes belonging to a UID
> lower
> then a certain value as well (500). This would be:
> 
> 1. Easly managable by the admin. Just let oracle/www and analogous users
>    have a UID lower then let's say 500.
> 
> 2. In full compliance with the port trick done by TCP/IP (ports < 1024
> vers other)
> 
> 3. It wouldn't need any addition of new interface (no jebanoje gawno in
> /proc in addition()
> 
> 4. Really simple to implement/document understand.
> 
> 5. Be the same way as Solaris does similiar things.
> 
> ...
> 
> Damn: I will let my chess club alone toady and will just code it down
> NOW.
> 
> Spec:
> 
> 1. Processes with a UID < 100 are immune to OOM killers.
> 2. Processes with a UID >= 100 && < 500 are hard for the OOM killer to
> take on.
> 3. Processes with a UID >= 500 are easy targets.
> 
> Let me introduce a new terminology in full analogy to "fire walls"
> routers and therabouts:
> 
> Processes of category 1. are called captains (oficerzy)
> Processes of category 2. are called corporals (porucznicy)
> Processes of category 2. are called privates (¿o³nierze)

OK I just did it. as I already told I have "stress tested it" by 
installing the Orcale insternet application server suide
on a hoplessly underequipped box ("only" 128MByte RMA).
The assorted patch is attached. 

Since I'm one day late up to my promise to provide this
patch it's actually fascinating that already 4 people (in esp. not
newbees requesting a new /proc entry for everything)
for reassurance that I will indeed implement it... Well 
this kind of "high" and "eager" feadback seems for me to indicate 
that there is very serious desire for it. And then of course I
just have to ask our people working with DB's here at work as well :-).

Ah... and of course I think this patch can already go directly 
into the official kernel. The quality of code should permit 
it. I would esp. request Rik van Riel to have a closer look
at it...

[-- Attachment #2: oom.diff --]
[-- Type: text/plain, Size: 11110 bytes --]

diff -urN linux/mm/oom_kill.c linux-new/mm/oom_kill.c
--- linux/mm/oom_kill.c	Tue Nov 14 19:56:46 2000
+++ linux-new/mm/oom_kill.c	Sun Mar 25 17:17:34 2001
@@ -1,18 +1,64 @@
 /*
  *  linux/mm/oom_kill.c
- * 
+ *
  *  Copyright (C)  1998,2000  Rik van Riel
  *	Thanks go out to Claus Fischer for some serious inspiration and
  *	for goading me into coding this file...
  *
- *  The routines in this file are used to kill a process when
- *  we're seriously out of memory. This gets called from kswapd()
- *  in linux/mm/vmscan.c when we really run out of memory.
- *
- *  Since we won't call these routines often (on a well-configured
- *  machine) this file will double as a 'coding guide' and a signpost
- *  for newbie kernel hackers. It features several pointers to major
- *  kernel subsystems and hints as to where to find out what things do.
+ *  Sat Mar 24 22:07:15 CET 2001 Marcin Dalecki <dalecki@evision-ventures.com>:
+ *
+ *	Replaced the original algorith with something reasonably, predictable
+ *	and managable. I will call this "Stalins Eviction".
+ */
+
+/*
+ *  The routines in this file are used to kill a process when the system is
+ *  entierly out of memmory (both: RAM and swap).  This gets called from
+ *  kswapd() in linux/mm/vmscan.c when we are in total starvation due to the
+ *  fact, that the only thing the system is busy at, is to try to allocate some
+ *  physical memmory page, where there are no pages anymore left. In such it
+ *  does make perfect sense to kill some offending process, just to make the
+ *  system go on and survive.
+ *
+ *  IT IS A LAST RESORT!
+ *
+ *  ALLERT: In contrast to popular beleve the invention of the mechanism
+ *  presented here IS IMPORTANT for system security reasons. It is preventing
+ *  one border corner of an easy DNS attack in case the sysadmin didn't take
+ *  other measures, which he either overworked or incompetent as he is usually
+ *  doesn't.
+ *
+ *  Basically the eviction goes on as follows:
+ *
+ *  1. Normal interactive user processes are the first candidates for a shoot.
+ *  We consider all users with a UID >= 500 as normal interactive users.
+ *
+ *  2. If there are no processes started by a normal interactive user, we aim
+ *  at the processes from nonessential processes (for the "live" of the system
+ *  as a whole).  We consider users with a UID >= 100 and < 500 as essential
+ *  service user.
+ *
+ *  3. If this still isn't the case we start to shut down the system components
+ *  peace by peace... (UID < 100).
+ *
+ *  In fact the heuristics used to determine, at which of the process classes
+ *  to aim first, are a bit more sophisticated, If you wan't those details
+ *  please read the code below. It does (hopefully so) speak for itself.
+ *
+ *  As an example: If you are running a big Linux box, which is mainly deployed
+ *  as an oracle server, but where normal interactive human users can log on as
+ *  well, then you should run oracle server with a UID < 500 and >= 100. Then
+ *  dumb ass loosers starting 100 netscape and 500 emacs sessions, won't be
+ *  able anylonger to kill the essential oracle service.
+ *
+ *  The introduction of this additional UID semantics shouldn't affect any
+ *  present systems. (Read: It won't make anything worser in comparision to
+ *  previous versions of the Linux kernel.) However every single distributor of
+ *  "enterprise grade" applications for Linux SHOULD take a note on this.
+ *
+ *  regards:
+ *
+ *		Marcin Dalecki
  */
 
 #include <linux/mm.h>
@@ -23,125 +69,141 @@
 
 /* #define DEBUG */
 
-/**
- * int_sqrt - oom_kill.c internal function, rough approximation to sqrt
- * @x: integer of which to calculate the sqrt
- * 
- * A very rough approximation to the sqrt() function.
- */
-static unsigned int int_sqrt(unsigned int x)
-{
-	unsigned int out = x;
-	while (x & ~(unsigned int)1) x >>=2, out >>=1;
-	if (x) out -= out >> 2;
-	return (out ? out : 1);
-}	
-
-/**
- * oom_badness - calculate a numeric value for how bad this task has been
- * @p: task struct of which task we should calculate
- *
- * The formula used is relatively simple and documented inline in the
- * function. The main rationale is that we want to select a good task
- * to kill when we run out of memory.
- *
- * Good in this context means that:
- * 1) we lose the minimum amount of work done
- * 2) we recover a large amount of memory
- * 3) we don't kill anything innocent of eating tons of memory
- * 4) we want to kill the minimum amount of processes (one)
- * 5) we try to kill the process the user expects us to kill, this
- *    algorithm has been meticulously tuned to meet the priniciple
- *    of least surprise ... (be careful when you change it)
- */
+#define CPU_FACTOR 32
+#define AGE_FACTOR 256
 
-static int badness(struct task_struct *p)
+enum uid_class {
+	normal,
+	service,
+	system,
+	immune
+};
+
+static int determine_uid_class(struct task_struct *p)
 {
-	int points, cpu_time, run_time;
+	int uid;
+	int uid_class = system;
 
-	if (!p->mm)
-		return 0;
-	/*
-	 * The memory size of the process is the basis for the badness.
+	/* This makes processes started by for example suexec be better killing
+	 * candidates then root's processes themself.
 	 */
-	points = p->mm->total_vm;
+	uid = p->uid;
+	if (p->euid > p->uid)
+		uid = p->euid;
 
-	/*
-	 * CPU time is in seconds and run time is in minutes. There is no
-	 * particular reason for this other than that it turned out to work
-	 * very well in practice. This is not safe against jiffie wraps
-	 * but we don't care _that_ much...
+	/* This is implementing the intendid semantics of different user id
+	 * value ranges.
 	 */
-	cpu_time = (p->times.tms_utime + p->times.tms_stime) >> (SHIFT_HZ + 3);
-	run_time = (jiffies - p->start_time) >> (SHIFT_HZ + 10);
+	if (uid < 100)
+		uid_class = system;
+	else if (uid < 500)
+		uid_class = service;
+	else
+		uid_class = normal;
 
-	points /= int_sqrt(cpu_time);
-	points /= int_sqrt(int_sqrt(run_time));
-
-	/*
-	 * Niced processes are most likely less important, so double
-	 * their badness points.
-	 */
-	if (p->nice > 0)
-		points *= 2;
 
-	/*
-	 * Superuser processes are usually more important, so we make it
+	/* Superuser processes are usually more important, so we make it
 	 * less likely that we kill those.
 	 */
-	if (cap_t(p->cap_effective) & CAP_TO_MASK(CAP_SYS_ADMIN) ||
-				p->uid == 0 || p->euid == 0)
-		points /= 4;
+	if (cap_t(p->cap_effective) & CAP_TO_MASK(CAP_SYS_ADMIN))
+		uid_class = system;
 
-	/*
-	 * We don't want to kill a process with direct hardware access.
+	/* We don't want to kill a process with direct hardware access.
 	 * Not only could that mess up the hardware, but usually users
 	 * tend to only have this flag set on applications they think
 	 * of as important.
 	 */
 	if (cap_t(p->cap_effective) & CAP_TO_MASK(CAP_SYS_RAWIO))
-		points /= 4;
-#ifdef DEBUG
-	printk(KERN_DEBUG "OOMkill: task %d (%s) got %d points\n",
-	p->pid, p->comm, points);
-#endif
-	return points;
+		uid_class = system;
+
+	return uid_class;
+}
+
+static int calculate_penalty(struct task_struct *p)
+{
+	int cpu_penalty = 0;
+	int age_penalty = 0;
+
+
+	/* Now we calculate the penalty due to the cpu usage.  NOTE: This is
+	 * not safe against jiffie wraps.
+	 */
+	{
+		int run_time = (jiffies - p->start_time) >> (SHIFT_HZ + 10);
+
+		if (run_time > 0) {
+			cpu_penalty = (CPU_FACTOR * run_time) /
+				((p->times.tms_utime + p->times.tms_stime) >> (SHIFT_HZ + 3) + run_time);
+		} else
+			cpu_penalty = CPU_FACTOR;
+	}
+
+	/* Let's make older processes more important then newer ones.
+	 * This is not safe against jiffie wraps, delibrately so.
+	 */
+	if (p->start_time > 0)
+		age_penalty = AGE_FACTOR * p->start_time / jiffies;
+	else
+		age_penalty = 0;
+
+	/* OK this should be sufficient, we don't want to make things more
+	 * complicated then needed. In esp. since there is no easy and portable
+	 * way to determine the total amount of memmory pages present, we don't
+	 * take this into account here.
+	 *
+	 * Let us worry about more detailed heuristics here, only if there will
+	 * be still many people reporting serious problems on linux-kernel.
+	 */
+
+	return cpu_penalty + age_penalty;
 }
 
 /*
- * Simple selection loop. We chose the process with the highest
- * number of 'points'. We need the locks to make sure that the
- * list of task structs doesn't change while we look the other way.
- *
- * (not docbooked, we don't want this one cluttering up the manual)
+ * Simple selection loop. We chose the process with the highest penalty.
  */
-static struct task_struct * select_bad_process(void)
+static struct task_struct * select_process(void)
 {
-	int maxpoints = 0;
-	struct task_struct *p = NULL;
-	struct task_struct *chosen = NULL;
-
-	read_lock(&tasklist_lock);
-	for_each_task(p) {
-		if (p->pid) {
-			int points = badness(p);
-			if (points > maxpoints) {
-				chosen = p;
-				maxpoints = points;
+	enum uid_class i;
+	struct task_struct *choice = NULL;
+
+	for (i = normal; i != immune; ++i) {
+		int maxpenalty = 0;
+		struct task_struct *p = NULL;
+
+		/* The locks make sure that the list of task structs doesn't
+		 * change while we look at it.
+		 */
+
+		read_lock(&tasklist_lock);
+		for_each_task(p) {
+			if (!p->mm)
+				continue;
+
+			if (i != determine_uid_class(p))
+				continue;
+
+			if (p->pid) {
+				int penalty = calculate_penalty(p);
+
+				if (penalty > maxpenalty) {
+					choice = p;
+					maxpenalty = penalty;
+				}
 			}
 		}
+		read_unlock(&tasklist_lock);
+
+		if (choice != NULL)
+			break;
 	}
-	read_unlock(&tasklist_lock);
-	return chosen;
+
+	return choice;
 }
 
-/**
- * oom_kill - kill the "best" process when we run out of memory
- *
+/*
  * If we run out of memory, we have the choice between either
  * killing a random task (bad), letting the system crash (worse)
- * OR try to be smart about which process to kill. Note that we
- * don't have to be perfect here, we just have to be good.
+ * OR try to be smart about which process to kill.
  *
  * We must be careful though to never send SIGKILL a process with
  * CAP_SYS_RAW_IO set, send SIGTERM instead (but it's unlikely that
@@ -149,14 +211,12 @@
  */
 void oom_kill(void)
 {
+	struct task_struct *p = select_process();
 
-	struct task_struct *p = select_bad_process();
-
-	/* Found nothing?!?! Either we hang forever, or we panic. */
 	if (p == NULL)
 		panic("Out of memory and no killable processes...\n");
 
-	printk(KERN_ERR "Out of Memory: Killed process %d (%s).\n", p->pid, p->comm);
+	printk(KERN_ERR "Out of memory: killed process %d (%s).\n", p->pid, p->comm);
 
 	/*
 	 * We give our sacrificial lamb high priority and access to
@@ -180,14 +240,14 @@
 	 */
 	current->policy |= SCHED_YIELD;
 	schedule();
+
 	return;
 }
 
-/**
- * out_of_memory - is the system out of memory?
+/** out_of_memory - is the system out of memory?
  *
- * Returns 0 if there is still enough memory left,
- * 1 when we are out of memory (otherwise).
+ * Returns 0 if there is still enough memory left, 1 when we are out of memory
+ * (otherwise).
  */
 int out_of_memory(void)
 {

Re: [PATCH] OOM handling

[Rik van Riel]

On Sun, 25 Mar 2001, Martin Dalecki wrote:

> Ah... and of course I think this patch can already go directly 
> into the official kernel. The quality of code should permit 
> it. I would esp. request Rik van Riel to have a closer look
> at it...

- the algorithms are just as much black magic as the old ones
- it hasn't been tested in any other workload than your Oracle
  server (at least, not that I've heard of)
- the comments are just too rude  ;)
  (though fun)
- the AGE_FACTOR calculation will overflow after the system has
  an uptime of just _3_ days 
- your code might be good for server loads, but for normal
  users it will kill what amounts to a random process ... this
  is horribly wrong for desktop systems

In short, I like some of your ideas, but I really fail to see why
this version of the code would be any better than what we're having
now. In fact, since there seem to be about 1000x more desktop boxes
than Oracle boxes (probably even more), I'd say that the current
algorithm in the kernel is better (since it's right for more systems).

Now if you can make something which preserves the heuristics which
serve us so well on desktop boxes and add something that makes it
also work on your Oracle servers, then I'd be interested.

Alternatively, I also wouldn't mind a completely new algorithm, as
long as it turns out to work well on desktop boxes too. But remember
that we cannot tell this without first testing the thing on a few
dozen (hundreds?) of machines with different workloads...

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

Re: [PATCH] OOM handling

[Martin Dalecki]

Rik van Riel wrote:
> 
> On Sun, 25 Mar 2001, Martin Dalecki wrote:
> 
> > Ah... and of course I think this patch can already go directly
> > into the official kernel. The quality of code should permit
> > it. I would esp. request Rik van Riel to have a closer look
> > at it...
> 
> - the algorithms are just as much black magic as the old ones
> - it hasn't been tested in any other workload than your Oracle
>   server (at least, not that I've heard of)

No that's not true! Read the code please. The result is a simple
wighted sum without artificial unit.

> - the comments are just too rude  ;)
>   (though fun)

That's only a matter for the "smooth" anglosaxons. Different
cultures have different measures on this. I don't feel the need
to adjust myself to the american cultural obstructivity.
I esp. to the habit of don't saying clearly what one means if one
want's to criticize something.

> - the AGE_FACTOR calculation will overflow after the system has
>   an uptime of just _3_ days
> - your code might be good for server loads, but for normal
>   users it will kill what amounts to a random process ... this
>   is horribly wrong for desktop systems

No that isn't true.  I esp. the behaviour will be predictable.

> In short, I like some of your ideas, but I really fail to see why
> this version of the code would be any better than what we're having
> now. In fact, since there seem to be about 1000x more desktop boxes
> than Oracle boxes (probably even more), I'd say that the current
> algorithm in the kernel is better (since it's right for more systems).

You misunderstood me compleatly. I wasn't using an running oracle
db as a test case. I was using the INSTALLATION process.
Since you apparently don't know about oracle I will tell you:
It involves a lot of different applications. Infact TONS of:
Java, shell, compiler, linker, apache, perl and whatanot.

> Now if you can make something which preserves the heuristics which
> serve us so well on desktop boxes and add something that makes it
> also work on your Oracle servers, then I'd be interested.

I would like to state: The current heuristics DON'T serve us well 
on desktop boxes...

> Alternatively, I also wouldn't mind a completely new algorithm, as
> long as it turns out to work well on desktop boxes too. But remember

I was testing on a NOTEBOOK.

> that we cannot tell this without first testing the thing on a few
> dozen (hundreds?) of machines with different workloads...

That's true for sure.

Re: [PATCH] OOM handling

[Jonathan Morton]

>- the AGE_FACTOR calculation will overflow after the system has
>  an uptime of just _3_ days

Tsk tsk tsk...

>Now if you can make something which preserves the heuristics which
>serve us so well on desktop boxes and add something that makes it
>also work on your Oracle servers, then I'd be interested.

What do people think of my "adjustments" to the existing algorithm?  Mostly
it gives extra longevity to low-UID and long-running processes, which to my
mind makes sense for both server and desktop boxen.

Taking for example an 80Mb process under my adjustments, it is reduced to
under the badness of a new shell process after less than a week's uptime
(compared to several months), especially if it is run as low-UID.  Small,
short-lived interactive processes still don't get *too* adversely affected,
but a memory hog with only a few hours' uptime will still get killed with
high probability (pretty much what we want).

I didn't quite understand Martin's comments about "not normalised" -
presumably this is some mathematical argument, but what does this actually
mean?

--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     chromi@cyberspace.org  (not for attachments)
big-mail: chromatix@penguinpowered.com
uni-mail: j.d.morton@lancaster.ac.uk

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS
PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*)
-----END GEEK CODE BLOCK-----

Re: [PATCH] OOM handling

[Martin Dalecki]

Jonathan Morton wrote:
> 
> >- the AGE_FACTOR calculation will overflow after the system has
> >  an uptime of just _3_ days
> 
> Tsk tsk tsk...
> 
> >Now if you can make something which preserves the heuristics which
> >serve us so well on desktop boxes and add something that makes it
> >also work on your Oracle servers, then I'd be interested.
> 
> What do people think of my "adjustments" to the existing algorithm?  Mostly
> it gives extra longevity to low-UID and long-running processes, which to my
> mind makes sense for both server and desktop boxen.
> 
> Taking for example an 80Mb process under my adjustments, it is reduced to
> under the badness of a new shell process after less than a week's uptime
> (compared to several months), especially if it is run as low-UID.  Small,
> short-lived interactive processes still don't get *too* adversely affected,
> but a memory hog with only a few hours' uptime will still get killed with
> high probability (pretty much what we want).
> 
> I didn't quite understand Martin's comments about "not normalised" -
> presumably this is some mathematical argument, but what does this actually
> mean?

Not mathematics. It's from physics. Very trivial physics, basic scool
indeed.
If you try to calculate some weightning
factors which involve different units (in this case mostly seconds and
bits)
then you will have to make sure tha those units get factorized out.
Rik is just throwing the absolute values together...

Trivial example:
"How long does it take to travel from A to B?"
"It takes about 1000sec."
"How long does it take to travel from C to D?"
"It takes about  100sec."
"Ah, so it's 10 times longer from A to B then from C to D".

Write it down - you just divide the seconds out.

In case of varying intervalls you have to normalize
measures by max/min values. Since for example the
amount of RAM in a box can vary as well. Otherwise
your algorithms will behave very differently on boxes
with low RAM in comparision to boxes with huge amounts of
it. That's what one says if he talks about an
algorithm "scalling well".

Re: [PATCH] OOM handling

[Jeff Garzik]

Martin Dalecki wrote:
> Rik van Riel wrote:
> > - the comments are just too rude  ;)
> >   (though fun)
> 
> That's only a matter for the "smooth" anglosaxons. Different
> cultures have different measures on this. I don't feel the need
> to adjust myself to the american cultural obstructivity.
> I esp. to the habit of don't saying clearly what one means if one
> want's to criticize something.

Rik should know that lkml and the kernel sources are in no way
politically correct...  Fuck 'em... :)

	Jeff


-- 
Jeff Garzik       | May you have warm words on a cold evening,
Building 1024     | a full moon on a dark night,
MandrakeSoft      | and a smooth road all the way to your door.

Re: [PATCH] OOM handling

[Jonathan Morton]

>> I didn't quite understand Martin's comments about "not normalised" -
>> presumably this is some mathematical argument, but what does this actually
>> mean?
>
>Not mathematics. It's from physics. Very trivial physics, basic scool
>indeed.
>If you try to calculate some weightning
>factors which involve different units (in this case mostly seconds and
>bits)
>then you will have to make sure tha those units get factorized out.
>Rik is just throwing the absolute values together...

Understood - my Physics courses covered this as well, but not using the
word "normalise".

--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     chromi@cyberspace.org  (not for attachments)
big-mail: chromatix@penguinpowered.com
uni-mail: j.d.morton@lancaster.ac.uk

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS
PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*)
-----END GEEK CODE BLOCK-----

Re: [PATCH] OOM handling

[Rik van Riel]

On Sun, 25 Mar 2001, Martin Dalecki wrote:
> Rik van Riel wrote:

> > - the AGE_FACTOR calculation will overflow after the system has
> >   an uptime of just _3_ days
> 
> I esp. the behaviour will be predictable.

Ummmm ?

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

Re: [PATCH] OOM handling

[Matthew Chappee]

> OK I just did it. as I already told I have "stress tested it" by 

> Since I'm one day late up to my promise to provide this
> patch it's actually fascinating that already 4 people (in esp. not
> newbees requesting a new /proc entry for everything)
> for reassurance that I will indeed implement it... Well 
> this kind of "high" and "eager" feadback seems for me to indicate  that
> there is very serious desire for it. And then of course I
> just have to ask our people working with DB's here at work as well :-).


I'm one of the four that contacted you. :-)  I'm certainly not a newbie and it appears that you nailed the reason that I'm interested.  I'm an Oracle DBA that runs a fairly large database(s) on Linux.  A patch like this is important.  Case in point:

We do not have loads of money, so we have to double up our servers.  A database server can also be an app server, or a web server, etc.  Now, let's say that Joe Surfer has 10 netscape sessions open on my database server (hey, talk to my boss, it's not my fault). He's grabbing Pr0n/MP3s/whatever as fast as our 'T' will allow.  One of the websites that he visits has some nasty java that bloats his browser to the point of OOM.  Something has to die in order for the machine to stay alive.  Remember the 100 sided die from D&D?  Roll it and kill -9?  Hopefully not, I should be able to tell the OOM_Killer to wipe out this user's stuff first, based on the prowess of his UID.

The point being, my database shouldn't be selected for termination.  Nobody ever got fired for kill -9'ing netscape, but Oracle is a different story.  I urge you, consider the patch.

> Ah... and of course I think this patch can already go directly 
> into the official kernel. The quality of code should permit 
> it. I would esp. request Rik van Riel to have a closer look
> at it...

Whoa, easy there trigger.  I'd rather have a wacked out OOM_Killer than a barely-tested alternative.

Matthew

Re: [PATCH] OOM handling

[Ingo Oeser]

On Sun, Mar 25, 2001 at 09:13:20PM -0500, Matthew Chappee wrote:
> The point being, my database shouldn't be selected for
> termination.  Nobody ever got fired for kill -9'ing netscape,
> but Oracle is a different story.  I urge you, consider the
> patch.

No, you got fired for not setting ulimits. Your boss is right
then!

ulimit -d 65536
ulimit -v 81920

and my netscape is very happy most of the time.

And my system is not disturbed.

64MB RAM + 256MB swap.

In a school I had the same setup on a 256MB server (256MB swap)
serving apps (StarOffice and Netscape) to  ~16 X clients.

I never had OOM there.

I think this is the amount of memory an oracle server at least
have to have, right?

What are your ulimits? What are your amounts of RAM+SWAP?

Regards

Ingo Oeser
-- 
10.+11.03.2001 - 3. Chemnitzer LinuxTag <http://www.tu-chemnitz.de/linux/tag>
         <<<<<<<<<<<<     been there and had much fun   >>>>>>>>>>>>

Re: [PATCH] OOM handling

[Jasper Spaans]

On Mon, Mar 26, 2001 at 01:33:05PM +0200, Ingo Oeser wrote:
> > The point being, my database shouldn't be selected for
> > termination.  Nobody ever got fired for kill -9'ing netscape,
> > but Oracle is a different story.  I urge you, consider the
> > patch.
> 
> No, you got fired for not setting ulimits. Your boss is right
> then!
> 
> ulimit -d 65536
> ulimit -v 81920

Ehm, right.

Running netscape (or any other memory hog which doesn't belong on a server)
on a production server seems reason enough for a little talk with your boss.

On the other hand, if no other apps are running on your box, and Oracle gets
killed due to OOM, you probably have underestimated your hardware needs, or
Oracle has gone haywire, which is a good reason for killing it.

Thus, nothing seems wrong with the current kill algorithm to me...

Just my two cents,
-- 
  Q_.           Jasper Spaans <j@sp3r.net>
 `~\            http://jsp.ds9a.nl/
Mr /\           Tel/Fax: +31-20-8749842
Zap             Move all .sig for great justice!

Re: [PATCH] OOM handling

[Michael Peddemors]

Uh... and aside from init, mission critical stuff... crond should never
get killed, it runs mission critical cleanup tasks..
If crond dies, might as well make the machine die in a lot of cases.. I
hate to miss my nightly database exports...

Getting to look more and more like we need some way to configure certain
tasks at the admin level to never die..


-- 
"Catch the Magic of Linux..."
--------------------------------------------------------
Michael Peddemors - Senior Consultant
LinuxAdministration - Internet Services
NetworkServices - Programming - Security
WizardInternet Services http://www.wizard.ca
Linux Support Specialist - http://www.linuxmagic.com
--------------------------------------------------------
(604)589-0037 Beautiful British Columbia, Canada

Re: [PATCH] OOM handling

[Kevin Buhr]

Jonathan Morton <chromi@cyberspace.org> writes:
>
> Understood - my Physics courses covered this as well, but not using the
> word "normalise".

Be that as it may, Martin's comments about normalizing are nonsense.
Rik's killer (at least in 2.4.3-pre7) produces a badness value that's
a product of badness factors of various units.  It then uses these
products only for relative comparisons, choosing the process with
maximum badness product to kill.  No normalization is necessary, nor
would it have any effect.

The reason a 256 Meg process on a 1 Gig machine was being killed had
nothing to do with normalization---it was a bug where the OOM killer
was being called long before we were reduced to last resorts.

Kevin <buhr@stat.wisc.edu>

Re: [PATCH] OOM handling

[Jonathan Morton]

>> Understood - my Physics courses covered this as well, but not using the
>> word "normalise".
>
>Be that as it may, Martin's comments about normalizing are nonsense.
>Rik's killer (at least in 2.4.3-pre7) produces a badness value that's
>a product of badness factors of various units.  It then uses these
>products only for relative comparisons, choosing the process with
>maximum badness product to kill.  No normalization is necessary, nor
>would it have any effect.
>
>The reason a 256 Meg process on a 1 Gig machine was being killed had
>nothing to do with normalization---it was a bug where the OOM killer
>was being called long before we were reduced to last resorts.

Of course, I realised that.  Actually, what the code does is take an
initial badness factor (the memory usage), then divide it using goodness
factors (some based on time, some purely arbitrary), both of which can be
considered dimensionless.  Also, at the end, the absolute value is not
considered - we simply look at the biggest one and kill it.  All
"denormalisation" does is scale all the values, it doesn't affect which one
actually turns out biggest.


--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     chromi@cyberspace.org  (not for attachments)
big-mail: chromatix@penguinpowered.com
uni-mail: j.d.morton@lancaster.ac.uk

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS
PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*)
-----END GEEK CODE BLOCK-----

Re: [PATCH] OOM handling

[Jonathan Morton]

>> >> Of course, I realised that.  Actually, what the code does is take an
>> >> initial badness factor (the memory usage), then divide it using goodness
>> >> factors (some based on time, some purely arbitrary), both of which can be
>> >> considered dimensionless.  Also, at the end, the absolute value is not
>> >> considered - we simply look at the biggest one and kill it.  All
>> >> "denormalisation" does is scale all the values, it doesn't affect
>>which one
>> >> actually turns out biggest.
>> >
>> >So you should realize as well that the actual code implementing this
>> >all is by no means numerically stable...
>>
>> It probably isn't, no.  I'll take another look at it and do some dry runs
>> sometime, and see whether they come out as I expect.
>
>Well the output depends heavly on the actual memsize of the process,
>which IMHO isn't a good value for choosing killing candidates...
>Second there is the problem that it's not possible to wight
>the goodness values against each other. The unit
>remaining is Bit/sqr(seconds). Try to get a grasp on this.
>Please have a look at my patch. The function I'm using
>there is a simply wighted sum of two process parameters.

I just ran the following test case through my (Saturday) version of the code:

80MB Oracle process
1 hour CPU time
1 week uptime
UID = 50

The result was less than 1, which means Oracle (or virtually any other
process with an hour of CPU time and a week's uptime) would not get killed.

You're perfectly right about the numerical stability argument, though.
Integers are notoriously granular, so maybe an increase in resolution is
justified.  There's also an issue where an almost-new process (with
run_time under 1024 seconds) would be given infinitely large badness - that
needs fixing.  Jiffie wrap is worth taking account for, too.  The comments
accompanying the code are completely wrong - cpu_time is in units of 8
seconds, and run_time is in units of 1024 seconds, NOT seconds and minutes
as described.

HOWEVER, I just took a look at your patch from Sunday.  I have very serious
concerns about this, which I will try to explain below:

First, your code uses a hard and arbitrary priority level.  This is
arranged such that if the "bad process" (which I use as a euphemism to
indicate a runaway memory hog) is in any class other than "normal", all
"normal" processes MUST exit before the "bad process" will even be
considered.  As a test case:

Suppose you're running Sendmail as uid 25, which puts it in the "system"
class.  This is a multiuser system and there are a lot of interactive,
unprivileged users present.  You are also running RPC services as "service"
class, using UIDs between 100-500.  Now suppose that Sendmail springs a big
memory leak and swamps the available memory, causing OOM - Sendmail is now
the "bad process" I mentioned earlier.  The sysadmin isn't watching the
system closely enough to kill Sendmail manually, and in any case the system
is thrashing so hard he wouldn't be able to log in quickly.

With your code, all the interactive users would be systematically thrown
off the system (losing all their work - SIGKILL is not kind) and the RPC
services would be shut down.  Depending on the relative ages of Sendmail
and other system services, other essential system daemons may also be shut
down (since your code does not take memory usage into account).  Finally,
Sendmail itself is killed and the problem goes away.

In the same scenario, my version of the code would probably kill Sendmail
relatively early in the sequence, since it is the one hogging all the RAM.
A few of the larger interactive process might get killed, depending on
relative ages.  The major flaw in my code is that a sufficiently long-lived
process becomes virtually immortal, even if it happens to spring a serious
leak after this time - the flaw in yours is that system processes have *too
high* priority relative to others, *right from the beginning*.  Both
problems need addressing if either of our algorithms can be considered
acceptable.

Oh and BTW, I think Bit/sqr(seconds) is a perfectly acceptable unit for
"badness".  Think about it - it increases with pigginess and decreases with
longevity.  I really don't see a problem with it per se.

I'm going to be travelling tomorrow, so I've moved my VM work onto my
PowerBook and will consider OOM-kill-selection algorithms and
memory-accounting while I fly.  See you on the other side of the ocean, and
hopefully the fresh Canadian air will help me think about this clearly.

--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     chromi@cyberspace.org  (not for attachments)
big-mail: chromatix@penguinpowered.com
uni-mail: j.d.morton@lancaster.ac.uk

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS
PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*)
-----END GEEK CODE BLOCK-----

RE: [PATCH] OOM handling

[Michel Wilson]

> relative ages.  The major flaw in my code is that a sufficiently
> long-lived
> process becomes virtually immortal, even if it happens to spring a serious
> leak after this time - the flaw in yours is that system processes

I think this could easily be fixed if you'd 'chop off' the runtime at a
certain point:

if(runtime > something_big)
	runtime = something_big;

This would of course need some tuning. The only thing i don't like about
this is that it's a kind of 'magical value', but i suppose it's not a very
good idea to make this configurable, right?

Michel Wilson.

Re: [PATCH] OOM handling

[Martin Dalecki]

Jonathan Morton wrote:

> 
> Oh and BTW, I think Bit/sqr(seconds) is a perfectly acceptable unit for
> "badness".  Think about it - it increases with pigginess and decreases with
> longevity.  I really don't see a problem with it per se.

Right it's not a problem pre se, but as you already explained
the problem is in the weightinig of different factors.
It's a matter of principle.

Re: [PATCH] OOM handling

[Martin Dalecki]

Michel Wilson wrote:
> 
> > relative ages.  The major flaw in my code is that a sufficiently
> > long-lived
> > process becomes virtually immortal, even if it happens to spring a serious
> > leak after this time - the flaw in yours is that system processes
> 
> I think this could easily be fixed if you'd 'chop off' the runtime at a
> certain point:
> 
> if(runtime > something_big)
>         runtime = something_big;
> 
> This would of course need some tuning. The only thing i don't like about
> this is that it's a kind of 'magical value', but i suppose it's not a very
> good idea to make this configurable, right?

Then after some time runtime becomes allmost irrelevant.
You are basically for what I call normalization by the total 
system uptime.

RE: [PATCH] OOM handling

[Jonathan Morton]

>> relative ages.  The major flaw in my code is that a sufficiently
>> long-lived
>> process becomes virtually immortal, even if it happens to spring a serious
>> leak after this time - the flaw in yours is that system processes
>
>I think this could easily be fixed if you'd 'chop off' the runtime at a
>certain point:
>
>if(runtime > something_big)
>	runtime = something_big;
>
>This would of course need some tuning. The only thing i don't like about
>this is that it's a kind of 'magical value', but i suppose it's not a very
>good idea to make this configurable, right?

Configurable is good, but right now I'm considering alternative (but
reasonably similar) algorithms.  If I can come up with something that works
reasonably well under all the scenarios I can think up - which is quite a
range - then configurable options may not be necessary.  In any case, other
work I'm doing should make OOM a thing of the past on most systems, since
malloc() and other memory-reservation calls will normally fail before OOM
happens.

It might just happen that totally different algorithms apply best to
different usage patterns, and I can put in some logic to try and detect
these patterns as needed, selecting the most appropriate algorithm.  An
embedded system is very different from a large batch-computation system,
and likewise for an Internet server, multiuser host, or single-user
workstation.  Internet servers come in different sizes, too - the 486 NAT
and web proxy differs considerably from the dedicated mail/web/database
server.

What would really help me is if a number of people with boxen under each of
the above loads could send me a "snapshot" of their system, under normal
load, containing the following info:

- General usage pattern description, in plain English
- Physical and swap memory: total sizes and current utilisation, in MB
- System uptime in days
- Summary of processes running at that instant, including for each process:
	- Approximate UID range
	- SIZE (not RSS, I want total size)
	- CPU time (with separate user and system totals if possible)
	- run time
Generalisations would probably be helpful - I don't expect to receive a
list of 500 emacs and bash processes, but indications of the distribution
of the above values for sensible groupings of processes would be valuable.
Of course, if you group processes, include information on how many process
you're grouping.  :)

For your security and protection, it would probably not be wise to indicate
the hostname or IP address(es) of the systems you profile in this manner.
You may, however, wish to invent codenames for the machines in case it
becomes necessary to refer to specific cases.  Profiles can be sent to me
at <chromi@cyberspace.org>, please include the string [SNAPSHOT] in the
subject for easy identification.

--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     chromi@cyberspace.org  (not for attachments)
big-mail: chromatix@penguinpowered.com
uni-mail: j.d.morton@lancaster.ac.uk

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS
PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*)
-----END GEEK CODE BLOCK-----

RE: [PATCH] OOM handling

[Rik van Riel]

On Tue, 27 Mar 2001, Michel Wilson wrote:

> > relative ages.  The major flaw in my code is that a sufficiently
> > long-lived
> > process becomes virtually immortal, even if it happens to spring a serious
> > leak after this time - the flaw in yours is that system processes
>
> I think this could easily be fixed if you'd 'chop off' the runtime at a
> certain point:
>
> if(runtime > something_big)
> 	runtime = something_big;
>
> This would of course need some tuning. The only thing i don't
> like about this is that it's a kind of 'magical value',

This is the reason I used the sqrt approximation in my
OOM killer ;)

Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml

Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/