2.6.23-rc6-mm1 panic (memory controller issue ?)

2.6.23-rc6-mm1 panic (memory controller issue ?)

[Badari Pulavarty]

Hi Balbir,

I get following panic from SLUB, while doing simple fsx tests.
I haven't used any container/memory controller stuff except 
that I configured them in :(

Looks like slub doesn't like one of the flags passed in ?

Known issue ? Ideas ?

Thanks,
Badari

CONFIG_CONTAINERS=y
CONFIG_CONTAINER_DEBUG=y
CONFIG_CONTAINER_NS=y
CONFIG_CONTAINER_CPUACCT=y
CONFIG_CONTAINER_MEM_CONT=y
CONFIG_ACPI_CONTAINER=m


elm3b29 login: ------------[ cut here ]------------
kernel BUG at mm/slub.c:1093!
invalid opcode: 0000 [1] SMP
last sysfs file: /power/state
CPU 3
Modules linked in:
Pid: 3885, comm: fsx-linux Not tainted 2.6.23-rc6-mm1 #2
RIP: 0010:[<ffffffff8029a458>]  [<ffffffff8029a458>] new_slab
+0x238/0x260
RSP: 0018:ffff81010140faf8  EFLAGS: 00010202
RAX: 0000000000000305 RBX: 0000000000000000 RCX: 00000000ffffffff
RDX: 00000000ffffffff RSI: 00000000001280d2 RDI: ffffffff806f3240
RBP: ffff81010140fb28 R08: 0000000000000040 R09: 0000000000000000
R10: 00000000000160c9 R11: 0000000000000002 R12: ffff8101c00146c0
R13: ffffffff806f3240 R14: ffffffff806f3240 R15: 00000000ffffffff
FS:  00007f7668f546d0(0000) GS:ffff8101c0729400(0000)
knlGS:0000000055749930
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f7668f6c000 CR3: 00000001821c1000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process fsx-linux (pid: 3885, threadinfo ffff81010140e000, task
ffff81010158aca0)
last branch before last exception/interrupt
 from  [<ffffffff8029a23c>] new_slab+0x1c/0x260
 to  [<ffffffff8029a458>] new_slab+0x238/0x260
Stack:  ffff8101c0729290 0000000000000000 ffff8101c00146c0
ffff8101df8618c0
 ffffffff806f3240 00000000ffffffff ffff81010140fb78 ffffffff8029a620
 001200d200000000 ffffffff8029eb10 001280d200000002 0000000000000000
Call Trace:
 [<ffffffff8029a620>] __slab_alloc+0x1a0/0x450
 [<ffffffff8029eb10>] mem_container_charge+0x90/0x2a0
 [<ffffffff8029ad0b>] kmem_cache_alloc+0x7b/0xa0
 [<ffffffff8029eb10>] mem_container_charge+0x90/0x2a0
 [<ffffffff80277c3e>] __alloc_pages+0x6e/0x360
 [<ffffffff8029ed4b>] mem_container_cache_charge+0x2b/0x40
 [<ffffffff8027176e>] add_to_page_cache+0x3e/0x120
 [<ffffffff80271869>] add_to_page_cache_lru+0x19/0x40
 [<ffffffff80271f2c>] find_or_create_page+0x5c/0xa0
 [<ffffffff803295c2>] ext3_truncate+0x342/0x990
 [<ffffffff8029eac2>] mem_container_charge+0x42/0x2a0
 [<ffffffff8027143d>] unlock_page+0x2d/0x40
 [<ffffffff80280e7f>] __do_fault+0x10f/0x3f0
 [<ffffffff8027f295>] __dec_zone_page_state+0x25/0x30
 [<ffffffff8028a146>] page_remove_rmap+0x46/0x140
 [<ffffffff80284b80>] vmtruncate+0xb0/0x110
 [<ffffffff802b81c0>] inode_setattr+0x30/0x180
 [<ffffffff8032ac9c>] ext3_setattr+0x12c/0x240
 [<ffffffff802b8690>] notify_change+0x380/0x3e0
 [<ffffffff802a0343>] do_truncate+0x63/0x90
 [<ffffffff802a1dc1>] generic_file_llseek+0x61/0xc0
 [<ffffffff802a0446>] sys_ftruncate+0xd6/0x120
 [<ffffffff8020c34e>] system_call+0x7e/0x83


Code: 0f 0b eb fe 66 66 66 90 41 8b 4d 14 ba 00 10 00 00 be 5a 00
RIP  [<ffffffff8029a458>] new_slab+0x238/0x260
 RSP <ffff81010140faf8>

Re: 2.6.23-rc6-mm1 panic (memory controller issue ?)

[Badari Pulavarty]

On Tue, 2007-09-18 at 15:21 -0700, Badari Pulavarty wrote:
> Hi Balbir,
> 
> I get following panic from SLUB, while doing simple fsx tests.
> I haven't used any container/memory controller stuff except 
> that I configured them in :(
> 
> Looks like slub doesn't like one of the flags passed in ?
> 
> Known issue ? Ideas ?
> 

I think, I found the issue. I am still running tests to
verify. Does this sound correct ?

Thanks,
Badari

Need to strip __GFP_HIGHMEM flag while passing to mem_container_cache_charge().

Signed-off-by: Badari Pulavarty <pbadari@us.ibm.com>
 mm/filemap.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Index: linux-2.6.23-rc6/mm/filemap.c
===================================================================
--- linux-2.6.23-rc6.orig/mm/filemap.c	2007-09-18 12:43:54.000000000 -0700
+++ linux-2.6.23-rc6/mm/filemap.c	2007-09-18 19:14:44.000000000 -0700
@@ -441,7 +441,8 @@ int filemap_write_and_wait_range(struct 
 int add_to_page_cache(struct page *page, struct address_space *mapping,
 		pgoff_t offset, gfp_t gfp_mask)
 {
-	int error = mem_container_cache_charge(page, current->mm, gfp_mask);
+	int error = mem_container_cache_charge(page, current->mm,
+			gfp_mask & ~__GFP_HIGHMEM);
 	if (error)
 		goto out;
 

Re: 2.6.23-rc6-mm1 panic (memory controller issue ?)

[Balbir Singh]

Badari Pulavarty wrote:
> On Tue, 2007-09-18 at 15:21 -0700, Badari Pulavarty wrote:
>> Hi Balbir,
>>
>> I get following panic from SLUB, while doing simple fsx tests.
>> I haven't used any container/memory controller stuff except 
>> that I configured them in :(
>>
>> Looks like slub doesn't like one of the flags passed in ?
>>
>> Known issue ? Ideas ?
>>
> 
> I think, I found the issue. I am still running tests to
> verify. Does this sound correct ?
> 
> Thanks,
> Badari
> 
> Need to strip __GFP_HIGHMEM flag while passing to mem_container_cache_charge().
> 
> Signed-off-by: Badari Pulavarty <pbadari@us.ibm.com>
>  mm/filemap.c |    3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> Index: linux-2.6.23-rc6/mm/filemap.c
> ===================================================================
> --- linux-2.6.23-rc6.orig/mm/filemap.c	2007-09-18 12:43:54.000000000 -0700
> +++ linux-2.6.23-rc6/mm/filemap.c	2007-09-18 19:14:44.000000000 -0700
> @@ -441,7 +441,8 @@ int filemap_write_and_wait_range(struct 
>  int add_to_page_cache(struct page *page, struct address_space *mapping,
>  		pgoff_t offset, gfp_t gfp_mask)
>  {
> -	int error = mem_container_cache_charge(page, current->mm, gfp_mask);
> +	int error = mem_container_cache_charge(page, current->mm,
> +			gfp_mask & ~__GFP_HIGHMEM);
>  	if (error)
>  		goto out;
> 
> 
> 

Hi, Badari,

The fix looks correct, radix_tree_preload() does the same thing in
add_to_page_cache(). Thanks for identifying the fix

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: 2.6.23-rc6-mm1 panic (memory controller issue ?)

[Christoph Lameter]

On Tue, 18 Sep 2007, Badari Pulavarty wrote:

> I think, I found the issue. I am still running tests to
> verify. Does this sound correct ?

Use GFP_LEVEL_MASK (or the equivalent in mm) to mask out the not allowed 
bits. The patch below only addresses issues with the __GFP_HIGHMEM bit. 
There may be others set. See mm/vmalloc.c for examples.

Re: 2.6.23-rc6-mm1 panic (memory controller issue ?)

[Christoph Lameter]

On Wed, 19 Sep 2007, Balbir Singh wrote:

> The fix looks correct, radix_tree_preload() does the same thing in
> add_to_page_cache(). Thanks for identifying the fix

Hmmm.... Radix tree preload can only take a limited set of flags?

Re: 2.6.23-rc6-mm1 panic (memory controller issue ?)

[Balbir Singh]

Christoph Lameter wrote:
> On Wed, 19 Sep 2007, Balbir Singh wrote:
> 
>> The fix looks correct, radix_tree_preload() does the same thing in
>> add_to_page_cache(). Thanks for identifying the fix
> 
> Hmmm.... Radix tree preload can only take a limited set of flags?
> 

Yes, the whole code is very interesting. From add_to_page_cache()
we call radix_tree_preload with __GFP_HIGHMEM cleared, but
from __add_to_swap_cache(), we don't make any changes to the
gfp_mask. radix_tree_preload() calls kmem_cache_alloc() and in slub
there is a check

        BUG_ON(flags & GFP_SLAB_BUG_MASK);

So, I guess all our allocations should check against __GFP_DMA and
__GFP_HIGHMEM. I'll review the code, test it and send a fix.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: 2.6.23-rc6-mm1 panic (memory controller issue ?)

[Christoph Lameter]

On Wed, 19 Sep 2007, Balbir Singh wrote:

> Yes, the whole code is very interesting. From add_to_page_cache()
> we call radix_tree_preload with __GFP_HIGHMEM cleared, but
> from __add_to_swap_cache(), we don't make any changes to the
> gfp_mask. radix_tree_preload() calls kmem_cache_alloc() and in slub
> there is a check
> 
>         BUG_ON(flags & GFP_SLAB_BUG_MASK);
> 
> So, I guess all our allocations should check against __GFP_DMA and
> __GFP_HIGHMEM. I'll review the code, test it and send a fix.

You need to use the proper mask from include/linux/gfp.h. Masking 
individual bits will create problems when we create new bits.

Re: 2.6.23-rc6-mm1 panic (memory controller issue ?)

[Balbir Singh]

Christoph Lameter wrote:
> On Wed, 19 Sep 2007, Balbir Singh wrote:
> 
>> Yes, the whole code is very interesting. From add_to_page_cache()
>> we call radix_tree_preload with __GFP_HIGHMEM cleared, but
>> from __add_to_swap_cache(), we don't make any changes to the
>> gfp_mask. radix_tree_preload() calls kmem_cache_alloc() and in slub
>> there is a check
>>
>>         BUG_ON(flags & GFP_SLAB_BUG_MASK);
>>
>> So, I guess all our allocations should check against __GFP_DMA and
>> __GFP_HIGHMEM. I'll review the code, test it and send a fix.
> 
> You need to use the proper mask from include/linux/gfp.h. Masking 
> individual bits will create problems when we create new bits.
> 

I agree 100%, that's why I want to review the code. I want to use
a mask that clears the GFP_SLAB_BUG_MASK bits and review it. I want
to check against other call sites that use gfp_mask as well.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL