From: Pekka Enberg <penberg@kernel.org>
To: Mikael Abrahamsson <swmike@swm.pp.se>
Cc: Stan Hoeppner <stan@hardwarefreak.com>,
Christoph Lameter <cl@linux.com>,
Linux Kernel List <linux-kernel@vger.kernel.org>,
linux-mm@kvack.org, Mel Gorman <mel@csn.ul.ie>,
Linux Netdev List <netdev@vger.kernel.org>
Subject: Re: 2.6.34.1 page allocation failure
Date: Sun, 29 Aug 2010 13:49:01 +0300 [thread overview]
Message-ID: <4C7A3B1D.7050500@kernel.org> (raw)
In-Reply-To: <alpine.DEB.1.10.1008242114120.8562@uplift.swm.pp.se>
On 24.8.2010 22.21, Mikael Abrahamsson wrote:
> On Tue, 24 Aug 2010, Pekka Enberg wrote:
>
>> It looks to me as if tcp_create_openreq_child() is able to cope with
>> the situation so the warning could be harmless. If that's the case,
>> we should probably stick a __GFP_NOWARN there.
>
> What about my situation? (a complete dmesg can be had at
> <http://swm.pp.se/dmesg.100809-2.txt.gz>)
This looks like something the kernel can't really recover from.
> [87578.494471] swapper: page allocation failure. order:0, mode:0x4020
> [87578.494476] Pid: 0, comm: swapper Not tainted 2.6.32-24-generic
> #39-Ubuntu
> [87578.494480] Call Trace:
> [87578.494483] <IRQ> [<ffffffff810fad0e>]
> __alloc_pages_slowpath+0x56e/0x580
> [87578.494499] [<ffffffff810fae7e>] __alloc_pages_nodemask+0x15e/0x1a0
> [87578.494506] [<ffffffff8112dba7>] alloc_pages_current+0x87/0xd0
> [87578.494511] [<ffffffff81133b17>] new_slab+0x2f7/0x310
> [87578.494516] [<ffffffff811363c1>] __slab_alloc+0x201/0x2d0
> [87578.494522] [<ffffffff81455fe6>] ? __netdev_alloc_skb+0x36/0x60
> [87578.494528] [<ffffffff81137408>]
> __kmalloc_node_track_caller+0xb8/0x180
> [87578.494532] [<ffffffff81455fe6>] ? __netdev_alloc_skb+0x36/0x60
> [87578.494536] [<ffffffff81455ca0>] __alloc_skb+0x80/0x190
> [87578.494540] [<ffffffff81455fe6>] __netdev_alloc_skb+0x36/0x60
> [87578.494564] [<ffffffffa008f5c7>] rtl8169_rx_interrupt+0x247/0x5b0
> [r8169]
> [87578.494572] [<ffffffffa008faad>] rtl8169_poll+0x3d/0x270 [r8169]
> [87578.494580] [<ffffffff810397a9>] ? default_spin_lock_flags+0x9/0x10
> [87578.494586] [<ffffffff8146029f>] net_rx_action+0x10f/0x250
> [87578.494594] [<ffffffffa008d54e>] ? rtl8169_interrupt+0xde/0x1e0
> [r8169]
> [87578.494600] [<ffffffff8106e467>] __do_softirq+0xb7/0x1e0
> [87578.494605] [<ffffffff810c52c0>] ? handle_IRQ_event+0x60/0x170
> [87578.494610] [<ffffffff810142ec>] call_softirq+0x1c/0x30
> [87578.494614] [<ffffffff81015cb5>] do_softirq+0x65/0xa0
> [87578.494618] [<ffffffff8106e305>] irq_exit+0x85/0x90
> [87578.494623] [<ffffffff81549515>] do_IRQ+0x75/0xf0
> [87578.494627] [<ffffffff81013b13>] ret_from_intr+0x0/0x11
> [87578.494629] <EOI> [<ffffffff8130f7cb>] ? acpi_idle_enter_c1+0xa3/0xc1
> [87578.494639] [<ffffffff8130f7aa>] ? acpi_idle_enter_c1+0x82/0xc1
> [87578.494646] [<ffffffff8143a5a7>] ? cpuidle_idle_call+0xa7/0x140
> [87578.494652] [<ffffffff81011e73>] ? cpu_idle+0xb3/0x110
> [87578.494657] [<ffffffff8153e27e>] ? start_secondary+0xa8/0xaa
> [87578.494660] Mem-Info:
> [87578.494662] Node 0 DMA per-cpu:
> [87578.494666] CPU 0: hi: 0, btch: 1 usd: 0
> [87578.494669] CPU 1: hi: 0, btch: 1 usd: 0
> [87578.494672] CPU 2: hi: 0, btch: 1 usd: 0
> [87578.494674] CPU 3: hi: 0, btch: 1 usd: 0
> [87578.494677] Node 0 DMA32 per-cpu:
> [87578.494680] CPU 0: hi: 186, btch: 31 usd: 173
> [87578.494683] CPU 1: hi: 186, btch: 31 usd: 87
> [87578.494686] CPU 2: hi: 186, btch: 31 usd: 168
> [87578.494689] CPU 3: hi: 186, btch: 31 usd: 63
> [87578.494691] Node 0 Normal per-cpu:
> [87578.494695] CPU 0: hi: 186, btch: 31 usd: 177
> [87578.494698] CPU 1: hi: 186, btch: 31 usd: 176
> [87578.494700] CPU 2: hi: 186, btch: 31 usd: 82
> [87578.494703] CPU 3: hi: 186, btch: 31 usd: 191
> [87578.494710] active_anon:22970 inactive_anon:6433 isolated_anon:0
> [87578.494711] active_file:916528 inactive_file:914736 isolated_file:0
> [87578.494713] unevictable:0 dirty:135959 writeback:24423 unstable:0
> [87578.494714] free:9990 slab_reclaimable:59767 slab_unreclaimable:11135
> [87578.494716] mapped:119343 shmem:985 pagetables:2113 bounce:0
> [87578.494719] Node 0 DMA free:15860kB min:20kB low:24kB high:28kB
> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15272kB
> mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
> pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
> pages_scanned:0 all_unreclaimable? yes
> [87578.494733] lowmem_reserve[]: 0 2866 7852 7852
> [87578.494738] Node 0 DMA32 free:21420kB min:4136kB low:5168kB
> high:6204kB active_anon:4056kB inactive_anon:5856kB
> active_file:1322360kB inactive_file:1320432kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:2935456kB mlocked:0kB
> dirty:190824kB writeback:31900kB mapped:157676kB shmem:0kB
> slab_reclaimable:107316kB slab_unreclaimable:15480kB kernel_stack:56kB
> pagetables:764kB unstable:0kB bounce:0kB writeback_tmp:0kB
> pages_scanned:0 all_unreclaimable? no
> [87578.494754] lowmem_reserve[]: 0 0 4986 4986
> [87578.494759] Node 0 Normal free:2680kB min:7192kB low:8988kB
> high:10788kB active_anon:87824kB inactive_anon:19876kB
> active_file:2343752kB inactive_file:2338512kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:5105664kB mlocked:0kB
> dirty:353012kB writeback:65792kB mapped:319696kB shmem:3940kB
> slab_reclaimable:131752kB slab_unreclaimable:29060kB
> kernel_stack:2160kB pagetables:7688kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> [87578.494775] lowmem_reserve[]: 0 0 0 0
> [87578.494779] Node 0 DMA: 3*4kB 3*8kB 3*16kB 1*32kB 2*64kB 2*128kB
> 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15860kB
> [87578.494792] Node 0 DMA32: 789*4kB 765*8kB 589*16kB 1*32kB 1*64kB
> 4*128kB 4*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 21356kB
> [87578.494805] Node 0 Normal: 374*4kB 4*8kB 20*16kB 1*32kB 0*64kB
> 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2648kB
You seem to have 4K pages available still. I wonder why the page
allocator isn't giving them to SLUB?
> [87578.494818] 1832322 total pagecache pages
> [87578.494820] 0 pages in swap cache
> [87578.494823] Swap cache stats: add 0, delete 0, find 0/0
> [87578.494825] Free swap = 0kB
> [87578.494827] Total swap = 0kB
> [87578.531041] 2064368 pages RAM
> [87578.531044] 66019 pages reserved
> [87578.531046] 1501227 pages shared
> [87578.531048] 619257 pages non-shared
> [87578.531053] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
> [87578.531057] cache: kmalloc-4096, object size: 4096, buffer size:
> 4096, default order: 3, min order: 0
> [87578.531061] node 0: slabs: 1322, objs: 4129, free: 0
>
> This actually made the machine go offline for hours before it for some
> reason came back. The second time this happened it did not come back
> (waited 8 hours).
Do you see these out-of-memory problems with 2.6.35?
WARNING: multiple messages have this Message-ID (diff)
From: Pekka Enberg <penberg@kernel.org>
To: Mikael Abrahamsson <swmike@swm.pp.se>
Cc: Stan Hoeppner <stan@hardwarefreak.com>,
Christoph Lameter <cl@linux.com>,
Linux Kernel List <linux-kernel@vger.kernel.org>,
linux-mm@kvack.org, Mel Gorman <mel@csn.ul.ie>,
Linux Netdev List <netdev@vger.kernel.org>
Subject: Re: 2.6.34.1 page allocation failure
Date: Sun, 29 Aug 2010 13:49:01 +0300 [thread overview]
Message-ID: <4C7A3B1D.7050500@kernel.org> (raw)
In-Reply-To: <alpine.DEB.1.10.1008242114120.8562@uplift.swm.pp.se>
On 24.8.2010 22.21, Mikael Abrahamsson wrote:
> On Tue, 24 Aug 2010, Pekka Enberg wrote:
>
>> It looks to me as if tcp_create_openreq_child() is able to cope with
>> the situation so the warning could be harmless. If that's the case,
>> we should probably stick a __GFP_NOWARN there.
>
> What about my situation? (a complete dmesg can be had at
> <http://swm.pp.se/dmesg.100809-2.txt.gz>)
This looks like something the kernel can't really recover from.
> [87578.494471] swapper: page allocation failure. order:0, mode:0x4020
> [87578.494476] Pid: 0, comm: swapper Not tainted 2.6.32-24-generic
> #39-Ubuntu
> [87578.494480] Call Trace:
> [87578.494483] <IRQ> [<ffffffff810fad0e>]
> __alloc_pages_slowpath+0x56e/0x580
> [87578.494499] [<ffffffff810fae7e>] __alloc_pages_nodemask+0x15e/0x1a0
> [87578.494506] [<ffffffff8112dba7>] alloc_pages_current+0x87/0xd0
> [87578.494511] [<ffffffff81133b17>] new_slab+0x2f7/0x310
> [87578.494516] [<ffffffff811363c1>] __slab_alloc+0x201/0x2d0
> [87578.494522] [<ffffffff81455fe6>] ? __netdev_alloc_skb+0x36/0x60
> [87578.494528] [<ffffffff81137408>]
> __kmalloc_node_track_caller+0xb8/0x180
> [87578.494532] [<ffffffff81455fe6>] ? __netdev_alloc_skb+0x36/0x60
> [87578.494536] [<ffffffff81455ca0>] __alloc_skb+0x80/0x190
> [87578.494540] [<ffffffff81455fe6>] __netdev_alloc_skb+0x36/0x60
> [87578.494564] [<ffffffffa008f5c7>] rtl8169_rx_interrupt+0x247/0x5b0
> [r8169]
> [87578.494572] [<ffffffffa008faad>] rtl8169_poll+0x3d/0x270 [r8169]
> [87578.494580] [<ffffffff810397a9>] ? default_spin_lock_flags+0x9/0x10
> [87578.494586] [<ffffffff8146029f>] net_rx_action+0x10f/0x250
> [87578.494594] [<ffffffffa008d54e>] ? rtl8169_interrupt+0xde/0x1e0
> [r8169]
> [87578.494600] [<ffffffff8106e467>] __do_softirq+0xb7/0x1e0
> [87578.494605] [<ffffffff810c52c0>] ? handle_IRQ_event+0x60/0x170
> [87578.494610] [<ffffffff810142ec>] call_softirq+0x1c/0x30
> [87578.494614] [<ffffffff81015cb5>] do_softirq+0x65/0xa0
> [87578.494618] [<ffffffff8106e305>] irq_exit+0x85/0x90
> [87578.494623] [<ffffffff81549515>] do_IRQ+0x75/0xf0
> [87578.494627] [<ffffffff81013b13>] ret_from_intr+0x0/0x11
> [87578.494629] <EOI> [<ffffffff8130f7cb>] ? acpi_idle_enter_c1+0xa3/0xc1
> [87578.494639] [<ffffffff8130f7aa>] ? acpi_idle_enter_c1+0x82/0xc1
> [87578.494646] [<ffffffff8143a5a7>] ? cpuidle_idle_call+0xa7/0x140
> [87578.494652] [<ffffffff81011e73>] ? cpu_idle+0xb3/0x110
> [87578.494657] [<ffffffff8153e27e>] ? start_secondary+0xa8/0xaa
> [87578.494660] Mem-Info:
> [87578.494662] Node 0 DMA per-cpu:
> [87578.494666] CPU 0: hi: 0, btch: 1 usd: 0
> [87578.494669] CPU 1: hi: 0, btch: 1 usd: 0
> [87578.494672] CPU 2: hi: 0, btch: 1 usd: 0
> [87578.494674] CPU 3: hi: 0, btch: 1 usd: 0
> [87578.494677] Node 0 DMA32 per-cpu:
> [87578.494680] CPU 0: hi: 186, btch: 31 usd: 173
> [87578.494683] CPU 1: hi: 186, btch: 31 usd: 87
> [87578.494686] CPU 2: hi: 186, btch: 31 usd: 168
> [87578.494689] CPU 3: hi: 186, btch: 31 usd: 63
> [87578.494691] Node 0 Normal per-cpu:
> [87578.494695] CPU 0: hi: 186, btch: 31 usd: 177
> [87578.494698] CPU 1: hi: 186, btch: 31 usd: 176
> [87578.494700] CPU 2: hi: 186, btch: 31 usd: 82
> [87578.494703] CPU 3: hi: 186, btch: 31 usd: 191
> [87578.494710] active_anon:22970 inactive_anon:6433 isolated_anon:0
> [87578.494711] active_file:916528 inactive_file:914736 isolated_file:0
> [87578.494713] unevictable:0 dirty:135959 writeback:24423 unstable:0
> [87578.494714] free:9990 slab_reclaimable:59767 slab_unreclaimable:11135
> [87578.494716] mapped:119343 shmem:985 pagetables:2113 bounce:0
> [87578.494719] Node 0 DMA free:15860kB min:20kB low:24kB high:28kB
> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15272kB
> mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
> pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
> pages_scanned:0 all_unreclaimable? yes
> [87578.494733] lowmem_reserve[]: 0 2866 7852 7852
> [87578.494738] Node 0 DMA32 free:21420kB min:4136kB low:5168kB
> high:6204kB active_anon:4056kB inactive_anon:5856kB
> active_file:1322360kB inactive_file:1320432kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:2935456kB mlocked:0kB
> dirty:190824kB writeback:31900kB mapped:157676kB shmem:0kB
> slab_reclaimable:107316kB slab_unreclaimable:15480kB kernel_stack:56kB
> pagetables:764kB unstable:0kB bounce:0kB writeback_tmp:0kB
> pages_scanned:0 all_unreclaimable? no
> [87578.494754] lowmem_reserve[]: 0 0 4986 4986
> [87578.494759] Node 0 Normal free:2680kB min:7192kB low:8988kB
> high:10788kB active_anon:87824kB inactive_anon:19876kB
> active_file:2343752kB inactive_file:2338512kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:5105664kB mlocked:0kB
> dirty:353012kB writeback:65792kB mapped:319696kB shmem:3940kB
> slab_reclaimable:131752kB slab_unreclaimable:29060kB
> kernel_stack:2160kB pagetables:7688kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> [87578.494775] lowmem_reserve[]: 0 0 0 0
> [87578.494779] Node 0 DMA: 3*4kB 3*8kB 3*16kB 1*32kB 2*64kB 2*128kB
> 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15860kB
> [87578.494792] Node 0 DMA32: 789*4kB 765*8kB 589*16kB 1*32kB 1*64kB
> 4*128kB 4*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 21356kB
> [87578.494805] Node 0 Normal: 374*4kB 4*8kB 20*16kB 1*32kB 0*64kB
> 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2648kB
You seem to have 4K pages available still. I wonder why the page
allocator isn't giving them to SLUB?
> [87578.494818] 1832322 total pagecache pages
> [87578.494820] 0 pages in swap cache
> [87578.494823] Swap cache stats: add 0, delete 0, find 0/0
> [87578.494825] Free swap = 0kB
> [87578.494827] Total swap = 0kB
> [87578.531041] 2064368 pages RAM
> [87578.531044] 66019 pages reserved
> [87578.531046] 1501227 pages shared
> [87578.531048] 619257 pages non-shared
> [87578.531053] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
> [87578.531057] cache: kmalloc-4096, object size: 4096, buffer size:
> 4096, default order: 3, min order: 0
> [87578.531061] node 0: slabs: 1322, objs: 4129, free: 0
>
> This actually made the machine go offline for hours before it for some
> reason came back. The second time this happened it did not come back
> (waited 8 hours).
Do you see these out-of-memory problems with 2.6.35?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2010-08-29 10:49 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-08-22 6:13 2.6.34.1 page allocation failure Stan Hoeppner
2010-08-22 6:47 ` Mikael Abrahamsson
2010-08-22 6:47 ` Mikael Abrahamsson
2010-08-22 19:51 ` Pekka Enberg
2010-08-22 19:51 ` Pekka Enberg
2010-08-22 22:40 ` Christoph Lameter
2010-08-22 22:40 ` Christoph Lameter
2010-08-23 9:37 ` Pekka Enberg
2010-08-23 9:37 ` Pekka Enberg
2010-08-23 22:35 ` Stan Hoeppner
2010-08-23 22:35 ` Stan Hoeppner
2010-08-24 17:13 ` Christoph Lameter
2010-08-24 17:13 ` Christoph Lameter
2010-08-24 18:03 ` Pekka Enberg
2010-08-24 18:03 ` Pekka Enberg
2010-08-24 19:08 ` Stan Hoeppner
2010-08-24 19:08 ` Stan Hoeppner
2010-08-24 19:21 ` Mikael Abrahamsson
2010-08-24 19:21 ` Mikael Abrahamsson
2010-08-29 10:49 ` Pekka Enberg [this message]
2010-08-29 10:49 ` Pekka Enberg
2010-08-29 12:38 ` Mikael Abrahamsson
2010-08-29 12:38 ` Mikael Abrahamsson
2010-08-29 13:17 ` Pekka Enberg
2010-08-29 13:17 ` Pekka Enberg
2010-08-29 15:37 ` Mikael Abrahamsson
2010-08-29 15:37 ` Mikael Abrahamsson
2010-08-31 20:28 ` Mikael Abrahamsson
2010-08-31 20:28 ` Mikael Abrahamsson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4C7A3B1D.7050500@kernel.org \
--to=penberg@kernel.org \
--cc=cl@linux.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mel@csn.ul.ie \
--cc=netdev@vger.kernel.org \
--cc=stan@hardwarefreak.com \
--cc=swmike@swm.pp.se \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.