Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Rik van Riel]

On Mon, 25 Oct 2004, Rik van Riel wrote:
> On Mon, 25 Oct 2004, Andrew Morton wrote:
> > Rik van Riel <riel@redhat.com> wrote:
> > >
> > > -		if (referenced && page_mapping_inuse(page))
> > > +		if (referenced && sc->priority && page_mapping_inuse(page))
> > 
> > Makes heaps of sense, but I'd like to exactly understand why people are
> > getting oomings before doing something like this.  I think we're still
> > waiting for a testcase?
> 
> I'm now running Yum on a (virtual) system with 96MB RAM and
> 100MB swap.  This used to get an OOM kill very quickly, but
> still seems to be running now, after 20 minutes.

It completed, without being OOM killed like before.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Marcelo Tosatti]

On Mon, Oct 25, 2004 at 06:33:35PM -0400, Rik van Riel wrote:
> On Mon, 25 Oct 2004, Rik van Riel wrote:
> > On Mon, 25 Oct 2004, Andrew Morton wrote:
> > > Rik van Riel <riel@redhat.com> wrote:
> > > >
> > > > -		if (referenced && page_mapping_inuse(page))
> > > > +		if (referenced && sc->priority && page_mapping_inuse(page))
> > > 
> > > Makes heaps of sense, but I'd like to exactly understand why people are
> > > getting oomings before doing something like this.  I think we're still
> > > waiting for a testcase?
> > 
> > I'm now running Yum on a (virtual) system with 96MB RAM and
> > 100MB swap.  This used to get an OOM kill very quickly, but
> > still seems to be running now, after 20 minutes.
> 
> It completed, without being OOM killed like before.

Barry,

Can you please test Rik's patch with your spurious OOM kill testcase?

===== mm/vmscan.c 1.231 vs edited =====
--- 1.231/mm/vmscan.c	Sun Oct 17 01:07:24 2004
+++ edited/mm/vmscan.c	Mon Oct 25 17:38:56 2004
@@ -379,7 +379,7 @@
 
 		referenced = page_referenced(page, 1);
 		/* In active use or really unfreeable?  Activate it. */
-		if (referenced && page_mapping_inuse(page))
+		if (referenced && sc->priority && page_mapping_inuse(page))
 			goto activate_locked;
 
 #ifdef CONFIG_SWAP
@@ -715,7 +715,7 @@
 		if (page_mapped(page)) {
 			if (!reclaim_mapped ||
 			    (total_swap_pages == 0 && PageAnon(page)) ||
-			    page_referenced(page, 0)) {
+			    (page_referenced(page, 0) && sc->priority)) {
 				list_add(&page->lru, &l_active);
 				continue;
 			}

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Chris Ross]

Marcelo Tosatti escreveu:
> Can you please test Rik's patch with your spurious OOM kill testcase?

Do you have a particular test case in mind? Is it accessible to the rest 
of us? If you send it to me I will run it on my 64MB P2 machine, which 
makes a very good test rig for the oom_killer because it is normally 
plagued by it.

I have already run Rik's patch to great success using my test case of 
compiling umlsim. Without the patch this fails every time at the linking 
the UML kernel stage.

Regards,
Chris R.

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Marcelo Tosatti]

On Thu, Oct 28, 2004 at 05:27:18PM +0200, Chris Ross wrote:
> 
> 
> Marcelo Tosatti escreveu:
> >Can you please test Rik's patch with your spurious OOM kill testcase?
> 
> Do you have a particular test case in mind? 

Anonymous memory intensive loads. It was easy to trigger the 
problem with "fillmem" from Quintela's memtest suite.

> Is it accessible to the rest 
> of us? If you send it to me I will run it on my 64MB P2 machine, which 
> makes a very good test rig for the oom_killer because it is normally 
> plagued by it.
> 
> I have already run Rik's patch to great success using my test case of 
> compiling umlsim. Without the patch this fails every time at the linking 
> the UML kernel stage.

Cool!

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Chris Ross]

Marcelo & co.,

Testing again: on plain 2.6.10-rc1-mm2 (i.e. without Rik's patch) 
building umlsim fails on my 64MB P2 350MHz Gentoo box exactly as before.

Regards,
Chris R.

Oct 29 15:25:19 sleepy oom-killer: gfp_mask=0xd0
Oct 29 15:25:19 sleepy DMA per-cpu:
Oct 29 15:25:19 sleepy cpu 0 hot: low 2, high 6, batch 1
Oct 29 15:25:19 sleepy cpu 0 cold: low 0, high 2, batch 1
Oct 29 15:25:19 sleepy Normal per-cpu:
Oct 29 15:25:19 sleepy cpu 0 hot: low 4, high 12, batch 2
Oct 29 15:25:19 sleepy cpu 0 cold: low 0, high 4, batch 2
Oct 29 15:25:19 sleepy HighMem per-cpu: empty
Oct 29 15:25:19 sleepy
Oct 29 15:25:19 sleepy Free pages:         244kB (0kB HighMem)
Oct 29 15:25:19 sleepy Active:12269 inactive:596 dirty:0 writeback:0 
unstable:0 free:61 slab:1117 mapped:12368 pagetables:140
Oct 29 15:25:19 sleepy DMA free:60kB min:60kB low:120kB high:180kB 
active:12304kB inactive:0kB present:16384kB pages_scanned:15252 
all_unreclaimable? yes
Oct 29 15:25:19 sleepy protections[]: 0 0 0
Oct 29 15:25:19 sleepy Normal free:184kB min:188kB low:376kB high:564kB 
active:36772kB inactive:2384kB present:49144kB pages_scanned:41571 
all_unreclaimable
? yes
Oct 29 15:25:19 sleepy protections[]: 0 0 0
Oct 29 15:25:19 sleepy HighMem free:0kB min:128kB low:256kB high:384kB 
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Oct 29 15:25:19 sleepy protections[]: 0 0 0
Oct 29 15:25:19 sleepy DMA: 4294567735*4kB 4294792863*8kB 
4294895642*16kB 4294943555*32kB 4294962724*64kB 4294966891*128kB 
4294967255*256kB 4294967283*512kB
  4294967290*1024kB 4294967293*2048kB 4294967294*4096kB = 4289685332kB
Oct 29 15:25:19 sleepy Normal: 4293893066*4kB 4294583823*8kB 
4294849819*16kB 4294950038*32kB 4294966291*64kB 4294966753*128kB 
4294967182*256kB 4294967238*51
2kB 4294967265*1024kB 4294967278*2048kB 4294967281*4096kB = 4284847952kB
Oct 29 15:25:19 sleepy HighMem: empty
Oct 29 15:25:19 sleepy Swap cache: add 9372, delete 7530, find 
1491/1835, race 0+0
Oct 29 15:25:19 sleepy Out of Memory: Killed process 12157 (ld).

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Chris Ross]

Chris Ross escreveu:
> Testing again: on plain 2.6.10-rc1-mm2 (i.e. without Rik's patch) 
> building umlsim fails on my 64MB P2 350MHz Gentoo box exactly as before.

To confirm, 2.6.10-rc1-mm2 with Rik's patch compiles umlsim-65 
(http://umlsim.sourceforge.net/umlsim-65.tar.gz) just fine.

Regards,
Chris R.

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Hiroyuki KAMEZAWA]

Hi,
 > Oct 29 15:25:19 sleepy protections[]: 0 0 0
 > Oct 29 15:25:19 sleepy DMA: 4294567735*4kB 4294792863*8kB
 > 4294895642*16kB 4294943555*32kB 4294962724*64kB 4294966891*128kB
 > 4294967255*256kB 4294967283*512kB
 >  4294967290*1024kB 4294967293*2048kB 4294967294*4096kB = 4289685332kB
 > Oct 29 15:25:19 sleepy Normal: 4293893066*4kB 4294583823*8kB
 > 4294849819*16kB 4294950038*32kB 4294966291*64kB 4294966753*128kB
 > 4294967182*256kB 4294967238*51
 > 2kB 4294967265*1024kB 4294967278*2048kB 4294967281*4096kB = 4284847952kB
 > Oct 29 15:25:19 sleepy HighMem: empty
 > Oct 29 15:25:19 sleepy Swap cache: add 9372, delete 7530, find

This looks odd.

How about this fix ?
I don't know why this is missng ....

Kame <kamezawa.hiroyu@jp.fujitsu.com>
--



---

  linux-2.6.10-rc1-mm2-kamezawa/mm/page_alloc.c |    4 +++-
  1 files changed, 3 insertions(+), 1 deletion(-)

diff -puN mm/page_alloc.c~clean-up mm/page_alloc.c
--- linux-2.6.10-rc1-mm2/mm/page_alloc.c~clean-up	2004-10-30 17:07:01.918419104 +0900
+++ linux-2.6.10-rc1-mm2-kamezawa/mm/page_alloc.c	2004-10-30 17:08:25.904651256 +0900
@@ -261,7 +261,9 @@ static inline void __free_pages_bulk (st
  	}
  	coalesced = base + page_idx;
  	set_page_order(coalesced, order);
-	list_add(&coalesced->lru, &zone->free_area[order].free_list);
+	area = zone->free_area + order;
+	list_add(&coalesced->lru, &area->free_list);
+	area->nr_free++;
  }

  static inline void free_pages_check(const char *function, struct page *page)

_

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Chris Ross]

Hiroyuki KAMEZAWA escreveu:
> How about this fix ?
> I don't know why this is missng ....

Instead of, or as well as Rik's fix?

Regards,
Chris R.

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Hiroyuki KAMEZAWA]

Chris Ross wrote:

> 
> 
> Hiroyuki KAMEZAWA escreveu:
> 
>> How about this fix ?
>> I don't know why this is missng ....
> 
> 
> Instead of, or as well as Rik's fix?
> 

Both of Rik's and this will be needed, I think.

> Regards,
> Chris R.
> 
zone->free_area[order]->nr_free is corrupted, this patch fix it.

It looks there is no area->nr_free++ code during freeing pages, now.


Kame <kamezawa.hiroyu@jp.fujitsu.com>

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Chris Ross]

Hiroyuki KAMEZAWA escreveu:
> zone->free_area[order]->nr_free is corrupted, this patch fix it.
> 
> It looks there is no area->nr_free++ code during freeing pages, now.

It's corrupt because area is out of scope at that point - it's declared 
within the for loop above.

Should I move your fix into the loop or move the declaration of area to 
function scope?

Regards,
Chris R.

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Hiroyuki KAMEZAWA]

Chris Ross wrote:

> 
> 
> Hiroyuki KAMEZAWA escreveu:
> 
>> zone->free_area[order]->nr_free is corrupted, this patch fix it.
>>
>> It looks there is no area->nr_free++ code during freeing pages, now.
> 
> 
> It's corrupt because area is out of scope at that point - it's declared 
> within the for loop above.
> 
> Should I move your fix into the loop or move the declaration of area to 
> function scope?
> 
Oh, Okay, my patch was wrong ;(.
Very sorry for wrong hack.
This one will be Okay.

Sorry,
Kame <kamezawa.hiroyu@jp.fujitsu.com>

-

  linux-2.6.10-rc1-mm2-kamezawa/mm/page_alloc.c |    1 +
  1 files changed, 1 insertion(+)

diff -puN mm/page_alloc.c~cleanup2 mm/page_alloc.c
--- linux-2.6.10-rc1-mm2/mm/page_alloc.c~cleanup2	2004-10-30 18:40:19.024529640 +0900
+++ linux-2.6.10-rc1-mm2-kamezawa/mm/page_alloc.c	2004-10-30 18:40:40.225306632 +0900
@@ -262,6 +262,7 @@ static inline void __free_pages_bulk (st
  	coalesced = base + page_idx;
  	set_page_order(coalesced, order);
  	list_add(&coalesced->lru, &zone->free_area[order].free_list);
+	zone->free_area[order].nr_free++;
  }

  static inline void free_pages_check(const char *function, struct page *page)

_

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Thomas Gleixner]

On Sat, 2004-10-30 at 18:53 +0900, Hiroyuki KAMEZAWA wrote:
> > Should I move your fix into the loop or move the declaration of area to 
> > function scope?
> > 
> Oh, Okay, my patch was wrong ;(.
> Very sorry for wrong hack.
> This one will be Okay.

It fixes at least the corrupted output of show_free_areas().
DMA: 4294966389*4kB 4294966983*8kB 4294967156*16kB .....
Normal: 4294954991*4kB 4294962949*8kB 4294965607*16kB ....

now it's
DMA: 248*4kB 63*8kB 7*16kB 1*32kB 0*64kB 0*128kB ...
Normal: 204*4kB 416*8kB 157*16kB 20*32kB 3*64kB ...

Good catch.

But it still does not fix the random madness of oom-killer. Once it is
triggered it keeps going even if there is 50MB free memory available.

tglx

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Thomas Gleixner]

[-- Attachment #1: Type: text/plain, Size: 951 bytes --]

> But it still does not fix the random madness of oom-killer. Once it is
> triggered it keeps going even if there is 50MB free memory available.
> 

Additionally to Rik's and Kame's fixes, I changed the criterias in
oom_kill a bit to

- take processes which fork a lot of children into account instead of
killing stuff like portmap and sshd just because they haven't used a lot
of CPU time since they started.

- prevent oom-killer to continue to kill processes, even if memory is
available

I was facing the above problems on a small embedded system, where a
forking server started to flood the machine with child processes.
oom-killer killed portmap and sshd instead of the real culprit and took
away the opportunity to log into the machine remotely.

The problem can be simulated with hackbench on a small UP system.

I don't know, if these changes have any negative impacts on the
testcases which were used in the original design. Rik ??

tglx




[-- Attachment #2: oom_kill.c.diff --]
[-- Type: text/x-patch, Size: 4147 bytes --]

--- 2.6.10-rc1-mm2.orig/mm/oom_kill.c	2004-10-31 11:13:32.000000000 +0100
+++ 2.6.10-rc1-mm2/mm/oom_kill.c	2004-10-31 13:00:06.000000000 +0100
@@ -45,8 +45,10 @@
 static unsigned long badness(struct task_struct *p, unsigned long uptime)
 {
 	unsigned long points, cpu_time, run_time, s;
+        struct list_head *tsk;
 
-	if (!p->mm)
+	/* Ignore mm-less tasks and init */
+	if (!p->mm || p->pid == 1)
 		return 0;
 
 	if (p->flags & PF_MEMDIE)
@@ -57,6 +59,19 @@
 	points = p->mm->total_vm;
 
 	/*
+	 * Processes which fork a lot of child processes are likely 
+	 * a good choice. We add the vmsize of the childs if they
+	 * have an own mm. This prevents forking servers to flood the
+	 * machine with an endless amount of childs
+	 */
+	list_for_each(tsk, &p->children) {
+		struct task_struct *chld;
+		chld = list_entry(tsk, struct task_struct, sibling);
+		if (chld->mm != p->mm && chld->mm)
+			points += chld->mm->total_vm;
+	}
+
+	/*
 	 * CPU time is in tens of seconds and run time is in thousands
          * of seconds. There is no particular reason for this other than
          * that it turned out to work very well in practice.
@@ -176,6 +191,27 @@
 	return mm;
 }
 
+static struct mm_struct *oom_kill_process(task_t *p)
+{
+	struct mm_struct *mm;
+	struct task_struct *g, *q;
+
+	mm = oom_kill_task(p);
+	if (!mm)
+		return NULL;
+	/*
+	 * kill all processes that share the ->mm (i.e. all threads),
+	 * but are in a different thread group
+	 */
+	do_each_thread(g, q)
+		if (q->mm == mm && q->tgid != p->tgid)
+			__oom_kill_task(q);
+
+	while_each_thread(g, q);
+	if (!p->mm)
+		printk(KERN_INFO "Fixed up OOM kill of mm-less task\n");
+	return mm;
+}
 
 /**
  * oom_kill - kill the "best" process when we run out of memory
@@ -188,7 +224,9 @@
 void oom_kill(void)
 {
 	struct mm_struct *mm;
-	struct task_struct *g, *p, *q;
+	struct task_struct *c, *p;
+	struct list_head *tsk;
+	int mmcnt = 0;
 	
 	read_lock(&tasklist_lock);
 retry:
@@ -200,21 +238,25 @@
 		panic("Out of memory and no killable processes...\n");
 	}
 
-	mm = oom_kill_task(p);
-	if (!mm)
-		goto retry;
 	/*
-	 * kill all processes that share the ->mm (i.e. all threads),
-	 * but are in a different thread group
+	 * Kill the child processes first
 	 */
-	do_each_thread(g, q)
-		if (q->mm == mm && q->tgid != p->tgid)
-			__oom_kill_task(q);
-	while_each_thread(g, q);
-	if (!p->mm)
-		printk(KERN_INFO "Fixed up OOM kill of mm-less task\n");
+	list_for_each(tsk, &p->children) {
+		c = list_entry(tsk, struct task_struct, sibling);
+		if (c->mm == p->mm)
+			continue;
+		mm = oom_kill_process(c);
+		if (mm) {
+			mmcnt ++;
+			mmput(mm);
+		}
+	}
+	mm = oom_kill_process(p);
+	if (!mmcnt && !mm)
+		goto retry;
+	if (mm)
+		mmput(mm);
 	read_unlock(&tasklist_lock);
-	mmput(mm);
 	return;
 }
 
@@ -224,14 +266,22 @@
 void out_of_memory(int gfp_mask)
 {
 	/*
-	 * oom_lock protects out_of_memory()'s static variables.
-	 * It's a global lock; this is not performance-critical.
+	 * inprogress protects out_of_memory()'s static variables.
+	 * We don't use a spin_lock here, as spinlocks are
+	 * nops on UP systems.
 	 */
-	static spinlock_t oom_lock = SPIN_LOCK_UNLOCKED;
+	static unsigned long inprogress;
 	static unsigned long first, last, count, lastkill;
+	static unsigned int  freepages = 1000000;
 	unsigned long now, since;
 
-	spin_lock(&oom_lock);
+	if (test_and_set_bit(0, &inprogress))
+		return;
+	
+	/* Check, if memory was freed since the last oom kill */
+	if (freepages < nr_free_pages())
+		goto out_unlock;
+
 	now = jiffies;
 	since = now - last;
 	last = now;
@@ -274,9 +324,8 @@
 
 	printk("oom-killer: gfp_mask=0x%x\n", gfp_mask);
 	show_free_areas();
-
-	/* oom_kill() sleeps */
-	spin_unlock(&oom_lock);
+	/* Store free pages  * 2 for the check above */
+	freepages = (nr_free_pages() << 1);
 	oom_kill();
 	/*
 	 * Make kswapd go out of the way, so "p" has a good chance of
@@ -284,7 +333,6 @@
 	 * for more memory.
 	 */
 	yield();
-	spin_lock(&oom_lock);
 
 reset:
 	/*
@@ -294,7 +342,7 @@
 	if (time_after(now, first))
 		first = now;
 	count = 0;
-
+	
 out_unlock:
-	spin_unlock(&oom_lock);
+	clear_bit(0, &inprogress);
 }

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Chris Ross]

Hiroyuki KAMEZAWA escreveu:
> Oh, Okay, my patch was wrong ;(.
> Very sorry for wrong hack.
> This one will be Okay.

That works, now my oom report looks like this...

Oct 30 17:32:22 sleepy oom-killer: gfp_mask=0x1d2
Oct 30 17:32:22 sleepy DMA per-cpu:
Oct 30 17:32:22 sleepy cpu 0 hot: low 2, high 6, batch 1
Oct 30 17:32:22 sleepy cpu 0 cold: low 0, high 2, batch 1
Oct 30 17:32:22 sleepy Normal per-cpu:
Oct 30 17:32:22 sleepy cpu 0 hot: low 4, high 12, batch 2
Oct 30 17:32:22 sleepy cpu 0 cold: low 0, high 4, batch 2
Oct 30 17:32:22 sleepy HighMem per-cpu: empty
Oct 30 17:32:22 sleepy
Oct 30 17:32:22 sleepy Free pages:         332kB (0kB HighMem)
Oct 30 17:32:22 sleepy Active:11887 inactive:517 dirty:0 writeback:0 
unstable:0 free:83 slab:1347 mapped:11930 pagetables:247
Oct 30 17:32:22 sleepy DMA free:60kB min:60kB low:120kB high:180kB 
active:11256kB inactive:436kB present:16384kB pages_scanned:11686 
all_unreclaimable? yes
Oct 30 17:32:22 sleepy protections[]: 0 0 0
Oct 30 17:32:22 sleepy Normal free:272kB min:188kB low:376kB high:564kB 
active:36292kB inactive:1632kB present:49144kB pages_scanned:6922 
all_unreclaimable? no
Oct 30 17:32:22 sleepy protections[]: 0 0 0
Oct 30 17:32:22 sleepy HighMem free:0kB min:128kB low:256kB high:384kB 
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Oct 30 17:32:22 sleepy protections[]: 0 0 0
Oct 30 17:32:22 sleepy DMA: 1*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60kB
Oct 30 17:32:22 sleepy Normal: 0*4kB 12*8kB 1*16kB 1*32kB 0*64kB 1*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 272kB
Oct 30 17:32:22 sleepy HighMem: empty
Oct 30 17:32:22 sleepy Swap cache: add 136776, delete 129314, find 
37853/51620, race 0+0
Oct 30 17:32:22 sleepy Out of Memory: Killed process 12395 (ld).

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Thomas Gleixner]

[-- Attachment #1: Type: text/plain, Size: 1720 bytes --]

Marcelo,

On Thu, 2004-10-28 at 10:06 -0200, Marcelo Tosatti wrote:
> Can you please test Rik's patch with your spurious OOM kill testcase?

I have similar problems with hackbench on a PIII/Celeron 300MHz, 128MB
RAM. 

Also Rik's patch does not help. I have the impression that it makes
almost no difference whether swap is on or not. With swap on, oom-killer
even kills rpc.mountd when it prints out that 56MB memory are available.

It happens on 2.6.8.1 too. On 2.6.7 I can run hackbench 40 without any
problem.

Full logs are attached.

tglx

Running hackbench 40, with swap on, I get 

oom-killer: gfp_mask=0xd0
Free pages:         356kB (0kB HighMem)
Out of Memory: Killed process 1030 (portmap).

oom-killer: gfp_mask=0xd0
Free pages:        6684kB (0kB HighMem)
Out of Memory: Killed process 1182 (atd).

oom-killer: gfp_mask=0xd0
Free pages:       17076kB (0kB HighMem)
Out of Memory: Killed process 1173 (sshd).

oom-killer: gfp_mask=0xd0
Free pages:       20160kB (0kB HighMem)
Out of Memory: Killed process 1191 (bash).
 - That's the shell on which hackbench was started

oom-killer: gfp_mask=0xd0
Free pages:       56544kB (0kB HighMem)
Out of Memory: Killed process 1149 (rpc.mountd).


Switching swap off, I get 

oom-killer: gfp_mask=0xd0
Free pages:         404kB (0kB HighMem)
Out of Memory: Killed process 1031 (portmap).

oom-killer: gfp_mask=0xd0
Free pages:         356kB (0kB HighMem)
Out of Memory: Killed process 1169 (atd).

oom-killer: gfp_mask=0xd0
Free pages:         792kB (0kB HighMem)
Out of Memory: Killed process 1160 (sshd).

oom-killer: gfp_mask=0xd0
Free pages:        2340kB (0kB HighMem)
Out of Memory: Killed process 1178 (bash).
- That's the shell on which hackbench was started


[-- Attachment #2: hb-swapon.log --]
[-- Type: text/x-log, Size: 5052 bytes --]

oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 12, high 36, batch 6
cpu 0 cold: low 0, high 12, batch 6
HighMem per-cpu: empty

Free pages:         356kB (0kB HighMem)
Active:0 inactive:0 dirty:0 writeback:2 unstable:0 free:89 slab:14778 mapped:9244 pagetables:6472
DMA free:44kB min:44kB low:88kB high:132kB active:0kB inactive:0kB present:16384kB
protections[]: 0 0 0
Normal free:312kB min:312kB low:624kB high:936kB active:0kB inactive:0kB present:114676kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44kB
Normal: 0*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 312kB
HighMem: empty
Swap cache: add 992, delete 727, find 10/11, race 0+0
Out of Memory: Killed process 1030 (portmap).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 12, high 36, batch 6
cpu 0 cold: low 0, high 12, batch 6
HighMem per-cpu: empty

Free pages:        6684kB (0kB HighMem)
Active:7 inactive:15 dirty:0 writeback:0 unstable:0 free:1671 slab:14067 mapped:8332 pagetables:6448
DMA free:2292kB min:44kB low:88kB high:132kB active:0kB inactive:24kB present:16384kB
protections[]: 0 0 0
Normal free:4392kB min:312kB low:624kB high:936kB active:28kB inactive:36kB present:114676kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 277*4kB 80*8kB 30*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2292kB
Normal: 666*4kB 120*8kB 30*16kB 1*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4392kB
HighMem: empty
Swap cache: add 1871, delete 1435, find 33/44, race 0+0
Out of Memory: Killed process 1182 (atd).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 12, high 36, batch 6
cpu 0 cold: low 0, high 12, batch 6
HighMem per-cpu: empty

Free pages:       17076kB (0kB HighMem)
Active:0 inactive:6 dirty:0 writeback:1 unstable:0 free:4269 slab:12021 mapped:7389 pagetables:6316
DMA free:4332kB min:44kB low:88kB high:132kB active:0kB inactive:24kB present:16384kB
protections[]: 0 0 0
Normal free:12744kB min:312kB low:624kB high:936kB active:0kB inactive:0kB present:114676kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 481*4kB 199*8kB 47*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4332kB
Normal: 1646*4kB 482*8kB 114*16kB 7*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 12744kB
HighMem: empty
Swap cache: add 2737, delete 1849, find 95/118, race 0+0
Out of Memory: Killed process 1173 (sshd).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 12, high 36, batch 6
cpu 0 cold: low 0, high 12, batch 6
HighMem per-cpu: empty

Free pages:       20160kB (0kB HighMem)
Active:0 inactive:7 dirty:0 writeback:2 unstable:0 free:5040 slab:11694 mapped:6115 pagetables:6252
DMA free:4704kB min:44kB low:88kB high:132kB active:0kB inactive:24kB present:16384kB
protections[]: 0 0 0
Normal free:15456kB min:312kB low:624kB high:936kB active:0kB inactive:4kB present:114676kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 474*4kB 217*8kB 61*16kB 3*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4704kB
Normal: 1576*4kB 674*8kB 187*16kB 16*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15456kB
HighMem: empty
Swap cache: add 4460, delete 2529, find 279/365, race 0+0
Out of Memory: Killed process 1191 (bash).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 12, high 36, batch 6
cpu 0 cold: low 0, high 12, batch 6
HighMem per-cpu: empty

Free pages:       56544kB (0kB HighMem)
Active:6 inactive:11 dirty:0 writeback:219 unstable:0 free:14136 slab:10701 mapped:1064 pagetables:2804
DMA free:4752kB min:44kB low:88kB high:132kB active:0kB inactive:24kB present:16384kB
protections[]: 0 0 0
Normal free:51792kB min:312kB low:624kB high:936kB active:24kB inactive:20kB present:114676kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 474*4kB 219*8kB 63*16kB 3*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4752kB
Normal: 2542*4kB 2035*8kB 1038*16kB 263*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 51792kB
HighMem: empty
Swap cache: add 7575, delete 6377, find 368/540, race 0+0
Out of Memory: Killed process 1149 (rpc.mountd).

[-- Attachment #3: hb-swoff.log --]
[-- Type: text/x-log, Size: 3978 bytes --]

oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 12, high 36, batch 6
cpu 0 cold: low 0, high 12, batch 6
HighMem per-cpu: empty

Free pages:         404kB (0kB HighMem)
Active:6603 inactive:9 dirty:0 writeback:0 unstable:0 free:101 slab:13757 mapped:10233 pagetables:6472
DMA free:44kB min:44kB low:88kB high:132kB active:1560kB inactive:0kB present:16384kB
protections[]: 0 0 0
Normal free:360kB min:312kB low:624kB high:936kB active:24852kB inactive:36kB present:114676kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44kB
Normal: 18*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 360kB
HighMem: empty
Swap cache: add 0, delete 0, find 0/0, race 0+0
Out of Memory: Killed process 1031 (portmap).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 12, high 36, batch 6
cpu 0 cold: low 0, high 12, batch 6
HighMem per-cpu: empty

Free pages:         356kB (0kB HighMem)
Active:5728 inactive:0 dirty:0 writeback:0 unstable:0 free:89 slab:14301 mapped:10171 pagetables:6472
DMA free:44kB min:44kB low:88kB high:132kB active:792kB inactive:0kB present:16384kB
protections[]: 0 0 0
Normal free:312kB min:312kB low:624kB high:936kB active:22120kB inactive:0kB present:114676kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44kB
Normal: 0*4kB 1*8kB 1*16kB 1*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 312kB
HighMem: empty
Swap cache: add 0, delete 0, find 0/0, race 0+0
Out of Memory: Killed process 1169 (atd).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 12, high 36, batch 6
cpu 0 cold: low 0, high 12, batch 6
HighMem per-cpu: empty

Free pages:         792kB (0kB HighMem)
Active:6588 inactive:1 dirty:0 writeback:0 unstable:0 free:198 slab:14647 mapped:9883 pagetables:6304
DMA free:48kB min:44kB low:88kB high:132kB active:1700kB inactive:4kB present:16384kB
protections[]: 0 0 0
Normal free:744kB min:312kB low:624kB high:936kB active:24652kB inactive:0kB present:114676kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 2*4kB 1*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
Normal: 88*4kB 3*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 744kB
HighMem: empty
Swap cache: add 0, delete 0, find 0/0, race 0+0
Out of Memory: Killed process 1160 (sshd).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 12, high 36, batch 6
cpu 0 cold: low 0, high 12, batch 6
HighMem per-cpu: empty

Free pages:        2340kB (0kB HighMem)
Active:7217 inactive:7 dirty:0 writeback:0 unstable:0 free:585 slab:15072 mapped:9371 pagetables:5968
DMA free:1436kB min:44kB low:88kB high:132kB active:1832kB inactive:4kB present:16384kB
protections[]: 0 0 0
Normal free:904kB min:312kB low:624kB high:936kB active:27036kB inactive:24kB present:114676kB
protections[]: 0 0 0
HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
protections[]: 0 0 0
DMA: 205*4kB 65*8kB 4*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1436kB
Normal: 52*4kB 9*8kB 5*16kB 7*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 904kB
HighMem: empty
Swap cache: add 0, delete 0, find 0/0, race 0+0
Out of Memory: Killed process 1178 (bash).

Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Javier Marcet]

[-- Attachment #1: Type: text/plain, Size: 1030 bytes --]

I've been following quite closely the development of 2.6.9, testing
every -rc release and a lot of -bk's.

Upon changing from 2.6.9-rc2 to 2.6.9-rc3 I began experiencing random
oom kills whenever a high memory i/o load took place.
This happened with plenty of free memory, and with whatever values I
used for vm.overcommit_ratio and vm.overcommit_memory
Doubling the physical RAM didn't change the situation either.

Having traced the problem to 2.6.9-rc3, I took a look at the differences
in memory handling between 2.6.9-rc2 and 2.6.9-rc3 and with the attached
patch I have no more oom kills. Not a single one.

I'm not saying everything within the patch is needed, not even that it's
the right thing to change. Nonetheless, 2.6.9 vanilla was unusable,
while this avoids those memory leaks.

Please, review and see what's wrong there :)


-- 
A mother takes twenty years to make a man of her boy, and another woman
makes a fool of him in twenty minutes.
a		-- Frost

Javier Marcet <javier@marcet.info>

[-- Attachment #2: revert_mem_to_2.6.8.1.patch --]
[-- Type: text/plain, Size: 1248 bytes --]

--- linux-2.6.9-rc3/mm/highmem.c	2004-10-23 09:31:45.314459000 +0200
+++ linux-2.6.9-rc2/mm/highmem.c	2004-10-23 09:29:30.451961216 +0200
@@ -300,7 +300,6 @@
 		 */
 		vfrom = page_address(fromvec->bv_page) + tovec->bv_offset;
 
-		flush_dcache_page(tovec->bv_page);
 		bounce_copy_vec(tovec, vfrom);
 	}
 }
@@ -407,7 +406,6 @@
 		if (rw == WRITE) {
 			char *vto, *vfrom;
 
-			flush_dcache_page(from->bv_page);
 			vto = page_address(to->bv_page) + to->bv_offset;
 			vfrom = kmap(from->bv_page) + from->bv_offset;
 			memcpy(vto, vfrom, to->bv_len);
--- linux-2.6.9-rc3/mm/mempolicy.c	2004-10-23 09:31:45.321457936 +0200
+++ linux-2.6.9-rc2/mm/mempolicy.c	2004-10-23 09:29:30.474957720 +0200
@@ -132,7 +132,6 @@
 	unsigned long nlongs;
 	unsigned long endmask;
 
-	--maxnode;
 	bitmap_zero(nodes, MAX_NUMNODES);
 	if (maxnode == 0 || !nmask)
 		return 0;
@@ -146,8 +145,6 @@
 	/* When the user specified more nodes than supported just check
 	   if the non supported part is all zero. */
 	if (nlongs > BITS_TO_LONGS(MAX_NUMNODES)) {
-		if (nlongs > PAGE_SIZE/sizeof(long))
-			return -EINVAL;
 		for (k = BITS_TO_LONGS(MAX_NUMNODES); k < nlongs; k++) {
 			unsigned long t;
 			if (get_user(t,  nmask + k))

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Andrew Morton]

Javier Marcet <javier@marcet.info> wrote:
>
> I've been following quite closely the development of 2.6.9, testing
>  every -rc release and a lot of -bk's.
> 
>  Upon changing from 2.6.9-rc2 to 2.6.9-rc3 I began experiencing random
>  oom kills whenever a high memory i/o load took place.

Do you have swap online?

What sort of machine is it, and how much memory has it?

>  This happened with plenty of free memory, and with whatever values I
>  used for vm.overcommit_ratio and vm.overcommit_memory
>  Doubling the physical RAM didn't change the situation either.
> 
>  Having traced the problem to 2.6.9-rc3, I took a look at the differences
>  in memory handling between 2.6.9-rc2 and 2.6.9-rc3 and with the attached
>  patch I have no more oom kills. Not a single one.
> 
>  I'm not saying everything within the patch is needed, not even that it's
>  the right thing to change. Nonetheless, 2.6.9 vanilla was unusable,
>  while this avoids those memory leaks.

That patch only affects NUMA machines?

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 471 bytes --]

Andrew Morton wrote:
> Javier Marcet <javier@marcet.info> wrote:
> 
>>I've been following quite closely the development of 2.6.9, testing
>> every -rc release and a lot of -bk's.
>>
>> Upon changing from 2.6.9-rc2 to 2.6.9-rc3 I began experiencing random
>> oom kills whenever a high memory i/o load took place.
> 
> 
> Do you have swap online?

When he first reported it he said no swap.

> What sort of machine is it, and how much memory has it?

Ditto - 1Gb ram.

Con

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Rik van Riel]

On Sat, 23 Oct 2004, Javier Marcet wrote:

> I'm not saying everything within the patch is needed, not even that it's
> the right thing to change.

I suspect the following (still untested) patch might be
needed, too.  Basically when the VM gets tight, it could
still ignore swappable pages with the referenced bit set.
Both really referenced pages and pages from the process
that currently has the swap token.

Forcefully deactivating a few pages when we run with
priority 0 might get rid of the false OOM kills.

I'm about to test this on a very small system here, and
will let you know how things go.


===== mm/vmscan.c 1.231 vs edited =====
--- 1.231/mm/vmscan.c	Sun Oct 17 01:07:24 2004
+++ edited/mm/vmscan.c	Mon Oct 25 17:38:56 2004
@@ -379,7 +379,7 @@
 
 		referenced = page_referenced(page, 1);
 		/* In active use or really unfreeable?  Activate it. */
-		if (referenced && page_mapping_inuse(page))
+		if (referenced && sc->priority && page_mapping_inuse(page))
 			goto activate_locked;
 
 #ifdef CONFIG_SWAP
@@ -715,7 +715,7 @@
 		if (page_mapped(page)) {
 			if (!reclaim_mapped ||
 			    (total_swap_pages == 0 && PageAnon(page)) ||
-			    page_referenced(page, 0)) {
+			    (page_referenced(page, 0) && sc->priority)) {
 				list_add(&page->lru, &l_active);
 				continue;
 			}

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Andrew Morton]

Rik van Riel <riel@redhat.com> wrote:
>
> -		if (referenced && page_mapping_inuse(page))
> +		if (referenced && sc->priority && page_mapping_inuse(page))

Makes heaps of sense, but I'd like to exactly understand why people are
getting oomings before doing something like this.  I think we're still
waiting for a testcase?

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Nick Piggin]

Andrew Morton wrote:
> Rik van Riel <riel@redhat.com> wrote:
> 
>>-		if (referenced && page_mapping_inuse(page))
>>+		if (referenced && sc->priority && page_mapping_inuse(page))
> 
> 
> Makes heaps of sense, but I'd like to exactly understand why people are
> getting oomings before doing something like this.  I think we're still
> waiting for a testcase?

I have found that quite often it is because all_unreclaimable gets set,
scanning slows down, and the OOM killer goes off.

Rik, I wonder if you can put some printk's where all_unreclaimable
is being set to 1, and see if there is any correlation to OOMs?


Aside from that, the patch does make sense, but might be too aggressive.
In heavy swapping loads, the "zero priority" scan might make up a
significant proportion of the scanning done so you'll want to be
careful about regressions there.

Re: Mem issues in 2.6.9 (ever since 2.6.9-rc3) and possible cause

[Thomas Gleixner]

On Sat, 2004-10-23 at 14:59 +0200, Javier Marcet wrote:
> I've been following quite closely the development of 2.6.9, testing
> every -rc release and a lot of -bk's.
> 
> Upon changing from 2.6.9-rc2 to 2.6.9-rc3 I began experiencing random
> oom kills whenever a high memory i/o load took place.
> This happened with plenty of free memory, and with whatever values I
> used for vm.overcommit_ratio and vm.overcommit_memory
> Doubling the physical RAM didn't change the situation either.
> 
> Having traced the problem to 2.6.9-rc3, I took a look at the differences
> in memory handling between 2.6.9-rc2 and 2.6.9-rc3 and with the attached
> patch I have no more oom kills. Not a single one.
> 
> I'm not saying everything within the patch is needed, not even that it's
> the right thing to change. Nonetheless, 2.6.9 vanilla was unusable,
> while this avoids those memory leaks.
> 
> Please, review and see what's wrong there :)

The changes in mempolicy.c are unrelated, except you have a NUMA enabled
machine.

The flush_dcache_page() is only relevant for non x86, as they result in
a NOP there.

tglx