From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by e1.ny.us.ibm.com (8.13.8/8.13.8) with ESMTP id l8DJYjYg030832 for ; Thu, 13 Sep 2007 15:34:45 -0400 Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v8.5) with ESMTP id l8DJYjqi563994 for ; Thu, 13 Sep 2007 15:34:45 -0400 Received: from d01av02.pok.ibm.com (loopback [127.0.0.1]) by d01av02.pok.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l8DJYi36001047 for ; Thu, 13 Sep 2007 15:34:45 -0400 Subject: 2.6.23-rc4-mm1 memory controller BUG_ON() From: Dave Hansen Content-Type: text/plain Date: Thu, 13 Sep 2007 12:34:43 -0700 Message-Id: <1189712083.17236.1626.camel@localhost> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: Balbir Singh Cc: Andrew Morton , linux-mm List-ID: Looks like somebody is holding a lock while trying to do a mem_container_charge(), and the mem_container_charge() call is doing an allocation. Naughty. I'm digging into it a bit more, but thought I'd report it, first. .config: http://sr71.net/~dave/linux/memory-controller-bug.config BUG: sleeping function called from invalid context at /home/dave/work/linux/2.6/23/rc4/mm1/lxc/mm/slab.c:3052 in_atomic():1, irqs_disabled():0 [] show_trace_log_lvl+0x19/0x2e [] show_trace+0x12/0x14 [] dump_stack+0x13/0x15 [] __might_sleep+0xe4/0xea [] kmem_cache_alloc+0x25/0xae [] mem_container_charge+0xc9/0x2cd [] mem_container_cache_charge+0x22/0x28 [] add_to_page_cache+0x35/0xd7 [] add_to_page_cache_lru+0x15/0x29 [] find_or_create_page+0x75/0x93 [] grow_dev_page+0x32/0x125 [] grow_buffers+0xb1/0xd4 [] __getblk_slow+0xb7/0xcf [] __getblk+0x44/0x4f [] ext3_getblk+0xca/0x19c [] ext3_find_entry+0x127/0x325 [] ext3_lookup+0x2c/0xe1 [] real_lookup+0x54/0xc5 [] do_lookup+0x59/0xa0 [] __link_path_walk+0x220/0xa4f [] link_path_walk+0x41/0xa5 [] path_walk+0x18/0x1a [] do_path_lookup+0x165/0x182 [] __path_lookup_intent_open+0x44/0x75 [] path_lookup_open+0x21/0x27 [] open_namei+0x7f/0x4c4 [] do_filp_open+0x26/0x3b [] do_sys_open+0x43/0xc7 [] sys_open+0x1a/0x1c [] init_post+0x45/0xe7 [] kernel_init+0x8a/0x8e [] kernel_thread_helper+0x7/0x10 ======================= -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from d03relay02.boulder.ibm.com (d03relay02.boulder.ibm.com [9.17.195.227]) by e31.co.us.ibm.com (8.13.8/8.13.8) with ESMTP id l8DJphnA001142 for ; Thu, 13 Sep 2007 15:51:43 -0400 Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by d03relay02.boulder.ibm.com (8.13.8/8.13.8/NCO v8.5) with ESMTP id l8DJphw3417454 for ; Thu, 13 Sep 2007 13:51:43 -0600 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l8DJphGV008226 for ; Thu, 13 Sep 2007 13:51:43 -0600 Subject: Re: 2.6.23-rc4-mm1 memory controller BUG_ON() From: Dave Hansen In-Reply-To: <1189712083.17236.1626.camel@localhost> References: <1189712083.17236.1626.camel@localhost> Content-Type: text/plain Date: Thu, 13 Sep 2007 12:51:41 -0700 Message-Id: <1189713102.17236.1647.camel@localhost> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: Balbir Singh Cc: Andrew Morton , linux-mm List-ID: On Thu, 2007-09-13 at 12:34 -0700, Dave Hansen wrote: > Looks like somebody is holding a lock while trying to do a > mem_container_charge(), and the mem_container_charge() call is doing an > allocation. Naughty. > > I'm digging into it a bit more, but thought I'd report it, first. > > .config: http://sr71.net/~dave/linux/memory-controller-bug.config I'm now thinking this is because the add_to_page_cache() functions have a gfp_mask passed in, and the mem_container_charge() functions don't take that mask. So, even if the add_to_page_cache() user specified ! __GFP_WAIT, the mem_container_charge() function can sleep on its kmalloc. I'll try passing gfp_flags through to it and see what happens. -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from d23relay03.au.ibm.com (d23relay03.au.ibm.com [202.81.18.234]) by e23smtp04.au.ibm.com (8.13.1/8.13.1) with ESMTP id l8DKORrq001213 for ; Fri, 14 Sep 2007 06:24:27 +1000 Received: from d23av02.au.ibm.com (d23av02.au.ibm.com [9.190.235.138]) by d23relay03.au.ibm.com (8.13.8/8.13.8/NCO v8.5) with ESMTP id l8DKLv3U1814754 for ; Fri, 14 Sep 2007 06:21:57 +1000 Received: from d23av02.au.ibm.com (loopback [127.0.0.1]) by d23av02.au.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l8DKLu2E019888 for ; Fri, 14 Sep 2007 06:21:56 +1000 Message-ID: <46E99BDE.9000602@linux.vnet.ibm.com> Date: Fri, 14 Sep 2007 01:51:50 +0530 From: Balbir Singh Reply-To: balbir@linux.vnet.ibm.com MIME-Version: 1.0 Subject: Re: 2.6.23-rc4-mm1 memory controller BUG_ON() References: <1189712083.17236.1626.camel@localhost> In-Reply-To: <1189712083.17236.1626.camel@localhost> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: Dave Hansen Cc: Balbir Singh , Andrew Morton , linux-mm List-ID: Dave Hansen wrote: > Looks like somebody is holding a lock while trying to do a > mem_container_charge(), and the mem_container_charge() call is doing an > allocation. Naughty. > > I'm digging into it a bit more, but thought I'd report it, first. > Hi, Dave, Thanks for reporting this. I sent out a patch to fix this problem (suggested by Nick Piggin). The patch is available at http://lkml.org/lkml/2007/9/12/113 Could you try the patch and check if the problem goes away? Any pointers on how to reproduce the problem would be extremely useful. -- Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from d23relay03.au.ibm.com (d23relay03.au.ibm.com [202.81.18.234]) by e23smtp04.au.ibm.com (8.13.1/8.13.1) with ESMTP id l8DKPLa8001571 for ; Fri, 14 Sep 2007 06:25:21 +1000 Received: from d23av03.au.ibm.com (d23av03.au.ibm.com [9.190.234.97]) by d23relay03.au.ibm.com (8.13.8/8.13.8/NCO v8.5) with ESMTP id l8DKPMgl4173902 for ; Fri, 14 Sep 2007 06:25:22 +1000 Received: from d23av03.au.ibm.com (loopback [127.0.0.1]) by d23av03.au.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l8DKP5fZ022951 for ; Fri, 14 Sep 2007 06:25:06 +1000 Message-ID: <46E99C92.9060800@linux.vnet.ibm.com> Date: Fri, 14 Sep 2007 01:54:50 +0530 From: Balbir Singh Reply-To: balbir@linux.vnet.ibm.com MIME-Version: 1.0 Subject: Re: 2.6.23-rc4-mm1 memory controller BUG_ON() References: <1189712083.17236.1626.camel@localhost> <1189713102.17236.1647.camel@localhost> In-Reply-To: <1189713102.17236.1647.camel@localhost> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: Dave Hansen Cc: Balbir Singh , Andrew Morton , linux-mm List-ID: Dave Hansen wrote: > On Thu, 2007-09-13 at 12:34 -0700, Dave Hansen wrote: >> Looks like somebody is holding a lock while trying to do a >> mem_container_charge(), and the mem_container_charge() call is doing an >> allocation. Naughty. >> >> I'm digging into it a bit more, but thought I'd report it, first. >> >> .config: http://sr71.net/~dave/linux/memory-controller-bug.config > > I'm now thinking this is because the add_to_page_cache() functions have > a gfp_mask passed in, and the mem_container_charge() functions don't > take that mask. So, even if the add_to_page_cache() user specified ! > __GFP_WAIT, the mem_container_charge() function can sleep on its > kmalloc. > > I'll try passing gfp_flags through to it and see what happens. > Please see my patch at http://lkml.org/lkml/2007/9/12/113 and some more details as a reply to your earlier email. -- Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by e1.ny.us.ibm.com (8.13.8/8.13.8) with ESMTP id l8DKsBTV008666 for ; Thu, 13 Sep 2007 16:54:11 -0400 Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v8.5) with ESMTP id l8DKsBou560538 for ; Thu, 13 Sep 2007 16:54:11 -0400 Received: from d01av02.pok.ibm.com (loopback [127.0.0.1]) by d01av02.pok.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l8DKsA97001134 for ; Thu, 13 Sep 2007 16:54:10 -0400 Subject: Re: 2.6.23-rc4-mm1 memory controller BUG_ON() From: Dave Hansen In-Reply-To: <46E99BDE.9000602@linux.vnet.ibm.com> References: <1189712083.17236.1626.camel@localhost> <46E99BDE.9000602@linux.vnet.ibm.com> Content-Type: text/plain Date: Thu, 13 Sep 2007 13:54:09 -0700 Message-Id: <1189716849.17236.1712.camel@localhost> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: balbir@linux.vnet.ibm.com Cc: Balbir Singh , Andrew Morton , linux-mm List-ID: On Fri, 2007-09-14 at 01:51 +0530, Balbir Singh wrote: > Dave Hansen wrote: > > Looks like somebody is holding a lock while trying to do a > > mem_container_charge(), and the mem_container_charge() call is doing > an > > allocation. Naughty. > > > > I'm digging into it a bit more, but thought I'd report it, first. > > > > Hi, Dave, > > Thanks for reporting this. I sent out a patch to fix this problem > (suggested by Nick Piggin). The patch is available at > > http://lkml.org/lkml/2007/9/12/113 > > Could you try the patch and check if the problem goes away? Balbir and I had a chat about this on IRC. Those patches don't seem to fix it. But, I'm getting Balbir hooked up with the kvm instance that I ran this in along with my .config. -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from d23relay03.au.ibm.com (d23relay03.au.ibm.com [202.81.18.234]) by e23smtp05.au.ibm.com (8.13.1/8.13.1) with ESMTP id l8E93BWq012030 for ; Fri, 14 Sep 2007 19:03:11 +1000 Received: from d23av03.au.ibm.com (d23av03.au.ibm.com [9.190.234.97]) by d23relay03.au.ibm.com (8.13.8/8.13.8/NCO v8.5) with ESMTP id l8E93Ag44735148 for ; Fri, 14 Sep 2007 19:03:10 +1000 Received: from d23av03.au.ibm.com (loopback [127.0.0.1]) by d23av03.au.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l8E92sLU031835 for ; Fri, 14 Sep 2007 19:02:54 +1000 Date: Fri, 14 Sep 2007 14:19:02 +0530 From: Balbir Singh Subject: Re: 2.6.23-rc4-mm1 memory controller BUG_ON() Message-ID: <20070914084902.GA27180@linux.vnet.ibm.com> Reply-To: balbir@linux.vnet.ibm.com References: <1189712083.17236.1626.camel@localhost> <46E99BDE.9000602@linux.vnet.ibm.com> <1189716849.17236.1712.camel@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1189716849.17236.1712.camel@localhost> Sender: owner-linux-mm@kvack.org Return-Path: To: Dave Hansen Cc: Balbir Singh , Andrew Morton , linux-mm List-ID: On Thu, Sep 13, 2007 at 01:54:09PM -0700, Dave Hansen wrote: > On Fri, 2007-09-14 at 01:51 +0530, Balbir Singh wrote: > > Dave Hansen wrote: > > > Looks like somebody is holding a lock while trying to do a > > > mem_container_charge(), and the mem_container_charge() call is doing > > an > > > allocation. Naughty. > > > > > > I'm digging into it a bit more, but thought I'd report it, first. > > > > > > > Hi, Dave, > > > > Thanks for reporting this. I sent out a patch to fix this problem > > (suggested by Nick Piggin). The patch is available at > > > > http://lkml.org/lkml/2007/9/12/113 > > > > Could you try the patch and check if the problem goes away? > > Balbir and I had a chat about this on IRC. Those patches don't seem to > fix it. But, I'm getting Balbir hooked up with the kvm instance that I > ran this in along with my .config. > Hi, Dave, Here's a fix for the problem, it works nicely on the qemu setup that you helped me with. I am also testing on another box. Could you please verify if the patch helps? Description of the patch ------------------------ Move mem_controller_cache_charge() above radix_tree_preload(). radix_tree_preload() disables preemption, even though the gfp_mask passed contains __GFP_WAIT, we cannot really do __GFP_WAIT allocations, thus we hit a BUG_ON() in kmem_cache_alloc(). This patch moves mem_controller_cache_charge() to above radix_tree_preload() for cache charging. Signed-off-by: Balbir Singh --- mm/filemap.c | 13 ++++++------- mm/swap_state.c | 13 +++++++------ 2 files changed, 13 insertions(+), 13 deletions(-) diff -puN mm/filemap.c~memory-controller-make-charging-gfpmask-aware-fixes mm/filemap.c --- linux-2.6.23-rc4/mm/filemap.c~memory-controller-make-charging-gfpmask-aware-fixes 2007-09-14 13:20:44.000000000 +0530 +++ linux-2.6.23-rc4-balbir/mm/filemap.c 2007-09-14 13:23:50.000000000 +0530 @@ -441,14 +441,12 @@ int filemap_write_and_wait_range(struct int add_to_page_cache(struct page *page, struct address_space *mapping, pgoff_t offset, gfp_t gfp_mask) { - int error = radix_tree_preload(gfp_mask & ~__GFP_HIGHMEM); + int error = mem_container_cache_charge(page, current->mm, gfp_mask); + if (error) + goto out; + error = radix_tree_preload(gfp_mask & ~__GFP_HIGHMEM); if (error == 0) { - - error = mem_container_cache_charge(page, current->mm, gfp_mask); - if (error) - goto out; - write_lock_irq(&mapping->tree_lock); error = radix_tree_insert(&mapping->page_tree, offset, page); if (!error) { @@ -463,7 +461,8 @@ int add_to_page_cache(struct page *page, write_unlock_irq(&mapping->tree_lock); radix_tree_preload_end(); - } + } else + mem_container_uncharge_page(page); out: return error; } diff -puN mm/swap_state.c~memory-controller-make-charging-gfpmask-aware-fixes mm/swap_state.c --- linux-2.6.23-rc4/mm/swap_state.c~memory-controller-make-charging-gfpmask-aware-fixes 2007-09-14 13:20:44.000000000 +0530 +++ linux-2.6.23-rc4-balbir/mm/swap_state.c 2007-09-14 13:26:14.000000000 +0530 @@ -78,13 +78,13 @@ static int __add_to_swap_cache(struct pa BUG_ON(!PageLocked(page)); BUG_ON(PageSwapCache(page)); BUG_ON(PagePrivate(page)); - error = radix_tree_preload(gfp_mask); - if (!error) { - error = mem_container_cache_charge(page, current->mm, gfp_mask); - if (error) - goto out; + error = mem_container_cache_charge(page, current->mm, gfp_mask); + if (error) + goto out; + error = radix_tree_preload(gfp_mask); + if (!error) { write_lock_irq(&swapper_space.tree_lock); error = radix_tree_insert(&swapper_space.page_tree, entry.val, page); @@ -99,7 +99,8 @@ static int __add_to_swap_cache(struct pa write_unlock_irq(&swapper_space.tree_lock); radix_tree_preload_end(); - } + } else + mem_container_uncharge_page(page); out: return error; } _ -- Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org