[mmotm][BUG] free is bigger than presnet Re: mmotm 2009-08-27-16-51 uploaded

[mmotm][BUG] free is bigger than presnet Re: mmotm 2009-08-27-16-51 uploaded

[KAMEZAWA Hiroyuki]

On Thu, 27 Aug 2009 16:55:42 -0700
akpm@linux-foundation.org wrote:

> The mm-of-the-moment snapshot 2009-08-27-16-51 has been uploaded to
> 
>    http://userweb.kernel.org/~akpm/mmotm/
> 
> and will soon be available at
> 
>    git://git.zen-sources.org/zen/mmotm.git
> 
> It contains the following patches against 2.6.31-rc7:
> 

I'm not digggin so much but /proc/meminfo corrupted.

[kamezawa@bluextal cgroup]$ cat /proc/meminfo
MemTotal:       24421124 kB
MemFree:        38314388 kB

Wow ;)
On x86-64 8cpu box + 24G memory.
(config is NUMA but the system itself is not NUMA.)

At boot time and for a while, It seems to be no trouble.

I used this.

== malloc.c

#include <stdio.h>

#define MSIZE (1024 * 1024 * 8)

int main(int argc, char *argv[])
{
        char **c;
        long size;
        int array_size, i;

        size = atoi(argv[1]);
        size *= 1024 * 1024;

        array_size =  size/MSIZE + 1;
        c= malloc(sizeof(void *) * array_size);

        for (i = 0; i < array_size; i++) {
                c[i] = malloc(MSIZE);
                memset(c[i], 0, MSIZE);
        }
        while (1) {
                for (i = 0; i < array_size; i++) {
                        memset(c[i], 0, MSIZE);
                        sleep(2);
                }
                sleep(10);
        }
        return;
}
==
# malloc 23000
# malloc 1000
and run hackbench 20.

OOM Kill message says free exceeds present ;(
==
Sep  1 18:01:17 localhost kernel: [ 3012.503440] active_anon:5461242 inactive_anon:473226 isolated_anon:384
Sep  1 18:01:17 localhost kernel: [ 3012.503440]  active_file:133 inactive_file:664 isolated_file:0
Sep  1 18:01:17 localhost kernel: [ 3012.503440]  unevictable:0 dirty:0 writeback:73 unstable:0 buffer:283
Sep  1 18:01:17 localhost kernel: [ 3012.503440]  free:9454041 slab_reclaimable:5144 slab_unreclaimable:10564
Sep  1 18:01:17 localhost kernel: [ 3012.503440]  mapped:7019 shmem:0 pagetables:22572 bounce:0
Sep  1 18:01:17 localhost kernel: [ 3012.503440] Node 0 DMA free:15788kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15016kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Sep  1 18:01:17 localhost kernel: [ 3012.503440] lowmem_reserve[]: 0 2951 23909 23909
Sep  1 18:01:17 localhost kernel: [ 3012.503440] Node 0 DMA32 free:11729908kB min:2440kB low:3048kB high:3660kB active_anon:1866292kB inactive_anon:466548kB active_file:140kB inactive_file:1588kB unevictable:0kB isolated(anon):256kB isolated(file):0kB present:3022624kB mlocked:0kB dirty:0kB writeback:292kB mapped:8kB shmem:0kB slab_reclaimable:2724kB slab_unreclaimable:10124kB kernel_stack:4504kB pagetables:21536kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:227 all_unreclaimable? no
Sep  1 18:01:17 localhost kernel: [ 3012.503440] lowmem_reserve[]: 0 0 20958 20958
Sep  1 18:01:17 localhost kernel: [ 3012.503440] Node 0 Normal free:26071392kB min:17340kB low:21672kB high:26008kB active_anon:19978676kB inactive_anon:1426356kB active_file:392kB inactive_file:1068kB unevictable:0kB isolated(anon):1280kB isolated(file):0kB present:21460992kB mlocked:0kB dirty:0kB writeback:0kB mapped:28068kB shmem:0kB slab_reclaimable:17852kB slab_unreclaimable:32132kB kernel_stack:3672kB pagetables:68752kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3840 all_unreclaimable? no

==

I'll dig more but does anyone have hints ?


Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [mmotm][BUG] free is bigger than presnet Re: mmotm 2009-08-27-16-51 uploaded

[Hugh Dickins]

On Tue, 1 Sep 2009, KAMEZAWA Hiroyuki wrote:
> 
> I'm not digggin so much but /proc/meminfo corrupted.
> 
> [kamezawa@bluextal cgroup]$ cat /proc/meminfo
> MemTotal:       24421124 kB
> MemFree:        38314388 kB

If that's without my fix to shrink_active_list(), I'd try again with.
Hugh

[PATCH mmotm] vmscan move pgdeactivate modification to shrink_active_list fix

mmotm 2009-08-27-16-51 lets the OOM killer loose on my loads even
quicker than last time: one bug fixed but another bug introduced.
vmscan-move-pgdeactivate-modification-to-shrink_active_list.patch
forgot to add NR_LRU_BASE to lru index to make zone_page_state index.

Signed-off-by: Hugh Dickins <hugh.dickins@tiscali.co.uk>
---

 mm/vmscan.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

--- mmotm/mm/vmscan.c	2009-08-28 10:07:57.000000000 +0100
+++ linux/mm/vmscan.c	2009-08-28 18:30:33.000000000 +0100
@@ -1381,8 +1381,10 @@ static void shrink_active_list(unsigned
 	reclaim_stat->recent_rotated[file] += nr_rotated;
 	__count_vm_events(PGDEACTIVATE, nr_deactivated);
 	__mod_zone_page_state(zone, NR_ISOLATED_ANON + file, -nr_taken);
-	__mod_zone_page_state(zone, LRU_ACTIVE + file * LRU_FILE, nr_rotated);
-	__mod_zone_page_state(zone, LRU_BASE + file * LRU_FILE, nr_deactivated);
+	__mod_zone_page_state(zone, NR_ACTIVE_ANON + file * LRU_FILE,
+							nr_rotated);
+	__mod_zone_page_state(zone, NR_INACTIVE_ANON + file * LRU_FILE,
+							nr_deactivated);
 	spin_unlock_irq(&zone->lru_lock);
 }
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [mmotm][BUG] free is bigger than presnet Re: mmotm 2009-08-27-16-51 uploaded

[KAMEZAWA Hiroyuki]

On Tue, 1 Sep 2009 10:33:31 +0100 (BST)
Hugh Dickins <hugh.dickins@tiscali.co.uk> wrote:

> On Tue, 1 Sep 2009, KAMEZAWA Hiroyuki wrote:
> > 
> > I'm not digggin so much but /proc/meminfo corrupted.
> > 
> > [kamezawa@bluextal cgroup]$ cat /proc/meminfo
> > MemTotal:       24421124 kB
> > MemFree:        38314388 kB
> 
> If that's without my fix to shrink_active_list(), I'd try again with.
> Hugh
> 
Thank you very much. I missed this patch.
It's fixed.

Regards,
-Kame


> [PATCH mmotm] vmscan move pgdeactivate modification to shrink_active_list fix
> 
> mmotm 2009-08-27-16-51 lets the OOM killer loose on my loads even
> quicker than last time: one bug fixed but another bug introduced.
> vmscan-move-pgdeactivate-modification-to-shrink_active_list.patch
> forgot to add NR_LRU_BASE to lru index to make zone_page_state index.
> 
> Signed-off-by: Hugh Dickins <hugh.dickins@tiscali.co.uk>
> ---
> 
>  mm/vmscan.c |    6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> --- mmotm/mm/vmscan.c	2009-08-28 10:07:57.000000000 +0100
> +++ linux/mm/vmscan.c	2009-08-28 18:30:33.000000000 +0100
> @@ -1381,8 +1381,10 @@ static void shrink_active_list(unsigned
>  	reclaim_stat->recent_rotated[file] += nr_rotated;
>  	__count_vm_events(PGDEACTIVATE, nr_deactivated);
>  	__mod_zone_page_state(zone, NR_ISOLATED_ANON + file, -nr_taken);
> -	__mod_zone_page_state(zone, LRU_ACTIVE + file * LRU_FILE, nr_rotated);
> -	__mod_zone_page_state(zone, LRU_BASE + file * LRU_FILE, nr_deactivated);
> +	__mod_zone_page_state(zone, NR_ACTIVE_ANON + file * LRU_FILE,
> +							nr_rotated);
> +	__mod_zone_page_state(zone, NR_INACTIVE_ANON + file * LRU_FILE,
> +							nr_deactivated);
>  	spin_unlock_irq(&zone->lru_lock);
>  }
>  
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [mmotm][BUG] free is bigger than presnet Re: mmotm 2009-08-27-16-51 uploaded

[KAMEZAWA Hiroyuki]

On Tue, 1 Sep 2009 18:50:13 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> On Tue, 1 Sep 2009 10:33:31 +0100 (BST)
> Hugh Dickins <hugh.dickins@tiscali.co.uk> wrote:
> 
> > On Tue, 1 Sep 2009, KAMEZAWA Hiroyuki wrote:
> > > 
> > > I'm not digggin so much but /proc/meminfo corrupted.
> > > 
> > > [kamezawa@bluextal cgroup]$ cat /proc/meminfo
> > > MemTotal:       24421124 kB
> > > MemFree:        38314388 kB
> > 
> > If that's without my fix to shrink_active_list(), I'd try again with.
> > Hugh
> > 
> Thank you very much. I missed this patch.
> It's fixed.

Sorry again, at continuing tests...thre are still..

MemTotal:       24421124 kB
MemFree:        25158956 kB
Buffers:            2264 kB
Cached:            34936 kB
SwapCached:         5140 kB

I wonder I miss something..

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [mmotm][BUG] free is bigger than presnet Re: mmotm 2009-08-27-16-51 uploaded

[Hugh Dickins]

On Tue, 1 Sep 2009, KAMEZAWA Hiroyuki wrote:
> 
> Sorry again, at continuing tests...thre are still..
> 
> MemTotal:       24421124 kB
> MemFree:        25158956 kB
> Buffers:            2264 kB
> Cached:            34936 kB
> SwapCached:         5140 kB
> 
> I wonder I miss something..

I've not been looking at /proc/meminfo: I'll do some stuff and see
if it goes wrong for me too, will let you know if so.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [mmotm][BUG] free is bigger than presnet Re: mmotm 2009-08-27-16-51 uploaded

[Hugh Dickins]

On Tue, 1 Sep 2009, Hugh Dickins wrote:
> On Tue, 1 Sep 2009, KAMEZAWA Hiroyuki wrote:
> > 
> > Sorry again, at continuing tests...thre are still..
> > 
> > MemTotal:       24421124 kB
> > MemFree:        25158956 kB
> > Buffers:            2264 kB
> > Cached:            34936 kB
> > SwapCached:         5140 kB
> > 
> > I wonder I miss something..
> 
> I've not been looking at /proc/meminfo: I'll do some stuff and see
> if it goes wrong for me too, will let you know if so.

Well, I've not yet noticed unbelievable MemFree, but my Active(anon)
(and Active) is bigger than my MemTotal and rising each iteration.

Probably not directly related to your case, and probably related to
my tmpfs or loop use: but I'd better pursue the anomaly I can so
easily reproduce, than worry about the anomaly you can reproduce.

Good luck with yours!
Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [mmotm][BUG] free is bigger than presnet Re: mmotm 2009-08-27-16-51 uploaded

[KAMEZAWA Hiroyuki]

Hugh Dickins wrote:
> On Tue, 1 Sep 2009, Hugh Dickins wrote:
>> On Tue, 1 Sep 2009, KAMEZAWA Hiroyuki wrote:
>> >
>> > Sorry again, at continuing tests...thre are still..
>> >
>> > MemTotal:       24421124 kB
>> > MemFree:        25158956 kB
>> > Buffers:            2264 kB
>> > Cached:            34936 kB
>> > SwapCached:         5140 kB
>> >
>> > I wonder I miss something..
>>
>> I've not been looking at /proc/meminfo: I'll do some stuff and see
>> if it goes wrong for me too, will let you know if so.
>
> Well, I've not yet noticed unbelievable MemFree, but my Active(anon)
> (and Active) is bigger than my MemTotal and rising each iteration.
>
> Probably not directly related to your case, and probably related to
> my tmpfs or loop use: but I'd better pursue the anomaly I can so
> easily reproduce, than worry about the anomaly you can reproduce.
>
I'll dig more. (After dinner, I doubt myself ;)

> Good luck with yours!
you too.

-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [mmotm][BUG] free is bigger than presnet Re: mmotm 2009-08-27-16-51 uploaded

[KAMEZAWA Hiroyuki]

On Tue, 1 Sep 2009 19:10:18 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> On Tue, 1 Sep 2009 18:50:13 +0900
> KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> 
> > On Tue, 1 Sep 2009 10:33:31 +0100 (BST)
> > Hugh Dickins <hugh.dickins@tiscali.co.uk> wrote:
> > 
> > > On Tue, 1 Sep 2009, KAMEZAWA Hiroyuki wrote:
> > > > 
> > > > I'm not digggin so much but /proc/meminfo corrupted.
> > > > 
> > > > [kamezawa@bluextal cgroup]$ cat /proc/meminfo
> > > > MemTotal:       24421124 kB
> > > > MemFree:        38314388 kB
> > > 
> > > If that's without my fix to shrink_active_list(), I'd try again with.
> > > Hugh
> > > 
> > Thank you very much. I missed this patch.
> > It's fixed.
> 
> Sorry again, at continuing tests...thre are still..
> 
> MemTotal:       24421124 kB
> MemFree:        25158956 kB
> Buffers:            2264 kB
> Cached:            34936 kB
> SwapCached:         5140 kB
> 
> I wonder I miss something..
> 
My mistake, sorry.

Thanks,
-Kame


> Thanks,
> -Kame
> 
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [mmotm][BUG] free is bigger than presnet Re: mmotm 2009-08-27-16-51 uploaded

[Johannes Weiner]

On Tue, Sep 01, 2009 at 06:00:32PM +0900, KAMEZAWA Hiroyuki wrote:
> On Thu, 27 Aug 2009 16:55:42 -0700
> akpm@linux-foundation.org wrote:
> 
> > The mm-of-the-moment snapshot 2009-08-27-16-51 has been uploaded to
> > 
> >    http://userweb.kernel.org/~akpm/mmotm/
> > 
> > and will soon be available at
> > 
> >    git://git.zen-sources.org/zen/mmotm.git
> > 
> > It contains the following patches against 2.6.31-rc7:
> > 
> 
> I'm not digggin so much but /proc/meminfo corrupted.
> 
> [kamezawa@bluextal cgroup]$ cat /proc/meminfo
> MemTotal:       24421124 kB
> MemFree:        38314388 kB

The bug is that every anon deactivation increases the 'free pages'
counter.  This should fix it:

http://marc.info/?l=linux-kernel&m=125148840818965&w=2

	Hannes

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [mmotm][BUG] lockdep warning block I/O (Was Re: mmotm 2009-08-27-16-51 uploaded

[Andrew Morton]

On Tue, 1 Sep 2009 18:07:17 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> 
> Here is mmont-Aug27's lockdep wanring. This was printed out when oom-kill happens.
> I'm sorry if already fixed.

My life's project is to hunt down the guy who invented mail client
wordwrapping, set him on fire then dance on his ashes.

> =
> Sep  1 18:01:16 localhost kernel: [ 3012.503035] ======================================================
> Sep  1 18:01:16 localhost kernel: [ 3012.503039] [ INFO: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected ]
> Sep  1 18:01:16 localhost kernel: [ 3012.503042] 2.6.31-rc7-mm1 #3
> Sep  1 18:01:16 localhost kernel: [ 3012.503049] ------------------------------------------------------
> Sep  1 18:01:16 localhost kernel: [ 3012.503052] kblockd/7/350 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
> Sep  1 18:01:16 localhost kernel: [ 3012.503058]  (bdev_lock){+.+...}, at: [<ffffffff811458c7>] nr_blockdev_pages+0x1
> 7/0x80
> Sep  1 18:01:16 localhost kernel: [ 3012.503069]
> Sep  1 18:01:16 localhost kernel: [ 3012.503070] and this task is already holding:
> Sep  1 18:01:16 localhost kernel: [ 3012.503075]  (&q->__queue_lock){..-.-.}, at: [<ffffffff811e9ff8>] cfq_kick_queue
> +0x28/0x50
> Sep  1 18:01:16 localhost kernel: [ 3012.503083] which would create a new lock dependency:
> Sep  1 18:01:16 localhost kernel: [ 3012.503087]  (&q->__queue_lock){..-.-.} -> (bdev_lock){+.+...}
> Sep  1 18:01:16 localhost kernel: [ 3012.503100]

I'd say the core problem here is that __alloc_pages_slowpath() is
calling show_mem().  Because show_mem() is a "high level" function which
takes "high level" locks.  ie: bdev_lock.

It's inappropriate that alloc_pages() is assuming that it is safe to
call show_mem() from all contexts in which alloc_pages() might be
called.

That show_mem() call has been there since 2005, so I don't know what
caused this to be revealed now.

It's not at all a serious bug and the chances of us deadlocking the
kernel here are close to zero.  An appropriate fix would be to replace
that show_mem() call with something which can be safely called from all
contexts in which the page allocator can be called.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [mmotm][BUG] lockdep warning block I/O (Was Re: mmotm 2009-08-27-16-51 uploaded

[KAMEZAWA Hiroyuki]

On Thu, 3 Sep 2009 14:28:36 -0700
Andrew Morton <akpm@linux-foundation.org> wrote:

> On Tue, 1 Sep 2009 18:07:17 +0900
> KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> 
> > 
> > Here is mmont-Aug27's lockdep wanring. This was printed out when oom-kill happens.
> > I'm sorry if already fixed.
> 
> My life's project is to hunt down the guy who invented mail client
> wordwrapping, set him on fire then dance on his ashes.
> 
Hmm, I should write a script to cut "Sep 1 ,,,,, : [.....]"...



> > =
> > Sep  1 18:01:16 localhost kernel: [ 3012.503035] ======================================================
> > Sep  1 18:01:16 localhost kernel: [ 3012.503039] [ INFO: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected ]
> > Sep  1 18:01:16 localhost kernel: [ 3012.503042] 2.6.31-rc7-mm1 #3
> > Sep  1 18:01:16 localhost kernel: [ 3012.503049] ------------------------------------------------------
> > Sep  1 18:01:16 localhost kernel: [ 3012.503052] kblockd/7/350 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
> > Sep  1 18:01:16 localhost kernel: [ 3012.503058]  (bdev_lock){+.+...}, at: [<ffffffff811458c7>] nr_blockdev_pages+0x1
> > 7/0x80
> > Sep  1 18:01:16 localhost kernel: [ 3012.503069]
> > Sep  1 18:01:16 localhost kernel: [ 3012.503070] and this task is already holding:
> > Sep  1 18:01:16 localhost kernel: [ 3012.503075]  (&q->__queue_lock){..-.-.}, at: [<ffffffff811e9ff8>] cfq_kick_queue
> > +0x28/0x50
> > Sep  1 18:01:16 localhost kernel: [ 3012.503083] which would create a new lock dependency:
> > Sep  1 18:01:16 localhost kernel: [ 3012.503087]  (&q->__queue_lock){..-.-.} -> (bdev_lock){+.+...}
> > Sep  1 18:01:16 localhost kernel: [ 3012.503100]
> 
> I'd say the core problem here is that __alloc_pages_slowpath() is
> calling show_mem().  Because show_mem() is a "high level" function which
> takes "high level" locks.  ie: bdev_lock.
> 
> It's inappropriate that alloc_pages() is assuming that it is safe to
> call show_mem() from all contexts in which alloc_pages() might be
> called.
> 
> That show_mem() call has been there since 2005, so I don't know what
> caused this to be revealed now.
> 
> It's not at all a serious bug and the chances of us deadlocking the
> kernel here are close to zero.  An appropriate fix would be to replace
> that show_mem() call with something which can be safely called from all
> contexts in which the page allocator can be called.
> 
ok, I'll study this path.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>