From: Daniel J Blueman <daniel@numascale.com>
To: Waiman Long <waiman.long@hp.com>, Mel Gorman <mgorman@suse.de>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Nathan Zimmer <nzimmer@sgi.com>,
Dave Hansen <dave.hansen@intel.com>,
Scott Norton <scott.norton@hp.com>, Linux-MM <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 0/13] Parallel struct page initialisation v4
Date: Sun, 03 May 2015 00:05:49 +0800 [thread overview]
Message-ID: <1430582749.21217.0@cpanel21.proisp.no> (raw)
In-Reply-To: <1430556732.28355.0@cpanel21.proisp.no>
On Sat, May 2, 2015 at 4:52 PM, Daniel J Blueman <daniel@numascale.com>
wrote:
> On Sat, May 2, 2015 at 8:09 AM, Waiman Long <waiman.long@hp.com>
> wrote:
>> On 05/01/2015 06:02 PM, Waiman Long wrote:
>>>
>>> Bad news!
>>>
>>> I tried your patch on a 24-TB DragonHawk and got an out of memory
>>> panic. The kernel log messages were:
>>> :
>>> [ 80.126186] CPU 474: hi: 186, btch: 31 usd: 0
>>> [ 80.131457] CPU 475: hi: 186, btch: 31 usd: 0
>>> [ 80.136726] CPU 476: hi: 186, btch: 31 usd: 0
>>> [ 80.141997] CPU 477: hi: 186, btch: 31 usd: 0
>>> [ 80.147267] CPU 478: hi: 186, btch: 31 usd: 0
>>> [ 80.152538] CPU 479: hi: 186, btch: 31 usd: 0
>>> [ 80.157813] active_anon:0 inactive_anon:0 isolated_anon:0
>>> [ 80.157813] active_file:0 inactive_file:0 isolated_file:0
>>> [ 80.157813] unevictable:0 dirty:0 writeback:0 unstable:0
>>> [ 80.157813] free:209 slab_reclaimable:7 slab_unreclaimable:42986
>>> [ 80.157813] mapped:0 shmem:0 pagetables:0 bounce:0
>>> [ 80.157813] free_cma:0
>>> [ 80.190428] Node 0 DMA free:568kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:15988kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB
>>> mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:14928kB kernel_stack:400kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.233475] lowmem_reserve[]: 0 0 0 0
>>> [ 80.237542] Node 0 DMA32 free:20kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1961924kB managed:1333604kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB
>>> slab_unreclaimable:101664kB kernel_stack:50176kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.281456] lowmem_reserve[]: 0 0 0 0
>>> [ 80.285527] Node 0 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1608515580kB managed:2097148kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:4kB
>>> slab_unreclaimable:948kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.328958] lowmem_reserve[]: 0 0 0 0
>>> [ 80.333031] Node 1 Normal free:248kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612732kB managed:2228220kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB
>>> slab_unreclaimable:46240kB kernel_stack:3232kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.377256] lowmem_reserve[]: 0 0 0 0
>>> [ 80.381325] Node 2 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:612kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.424764] lowmem_reserve[]: 0 0 0 0
>>> [ 80.428842] Node 3 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:600kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.472293] lowmem_reserve[]: 0 0 0 0
>>> [ 80.476360] Node 4 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:620kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.519803] lowmem_reserve[]: 0 0 0 0
>>> [ 80.523875] Node 5 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:584kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.567312] lowmem_reserve[]: 0 0 0 0
>>> [ 80.571379] Node 6 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:556kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.614814] lowmem_reserve[]: 0 0 0 0
>>> [ 80.618881] Node 7 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:556kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.662316] lowmem_reserve[]: 0 0 0 0
>>> [ 80.666390] Node 8 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.709827] lowmem_reserve[]: 0 0 0 0
>>> [ 80.713898] Node 9 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.757336] lowmem_reserve[]: 0 0 0 0
>>> [ 80.761407] Node 10 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:564kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.804941] lowmem_reserve[]: 0 0 0 0
>>> [ 80.809015] Node 11 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.852548] lowmem_reserve[]: 0 0 0 0
>>> [ 80.856620] Node 12 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:616kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.900158] lowmem_reserve[]: 0 0 0 0
>>> [ 80.904236] Node 13 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:592kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.947765] lowmem_reserve[]: 0 0 0 0
>>> [ 80.951847] Node 14 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:600kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 80.995380] lowmem_reserve[]: 0 0 0 0
>>> [ 80.999448] Node 15 Normal free:0kB min:0kB low:0kB high:0kB
>>> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
>>> unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB
>>> writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> slab_unreclaimable:548kB kernel_stack:0kB pagetables:0kB
>>> unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0 all_unreclaimable? yes
>>> [ 81.042974] lowmem_reserve[]: 0 0 0 0
>>> [ 81.047044] Node 0 DMA: 132*4kB (U) 5*8kB (U) 0*16kB 0*32kB
>>> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 568kB
>>> [ 81.059632] Node 0 DMA32: 5*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
>>> [ 81.071733] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.083443] Node 1 Normal: 52*4kB (U) 5*8kB (U) 0*16kB 0*32kB
>>> 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 248kB
>>> [ 81.096227] Node 2 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.107935] Node 3 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.119643] Node 4 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.131347] Node 5 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.143056] Node 6 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.154767] Node 7 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.166473] Node 8 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.178179] Node 9 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.189893] Node 10 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.201695] Node 11 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.213496] Node 12 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.225324] Node 13 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.237130] Node 14 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.248926] Node 15 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB
>>> 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
>>> [ 81.260726] 0 total pagecache pages
>>> [ 81.264565] 0 pages in swap cache
>>> [ 81.268212] Swap cache stats: add 0, delete 0, find 0/0
>>> [ 81.273962] Free swap = 0kB
>>> [ 81.277125] Total swap = 0kB
>>> [ 81.280341] 6442421132 pages RAM
>>> [ 81.283888] 0 pages HighMem/MovableOnly
>>> [ 81.288109] 6433662383 pages reserved
>>> [ 81.292135] 0 pages hwpoisoned
>>> [ 81.295491] [ pid ] uid tgid total_vm rss nr_ptes
>>> nr_pmds swapents oom_score_adj name
>>> [ 81.305245] Kernel panic - not syncing: Out of memory and no
>>> killable processes...
>>> [ 81.305245]
>>> [ 81.315200] CPU: 240 PID: 1 Comm: swapper/0 Not tainted
>>> 4.0.1-pmm-bigsmp #1
>>> [ 81.322856] Hardware name: HP Superdome2 16s x86, BIOS Bundle:
>>> 006.000.042 SFW: 015.099.000 04/01/2015
>>> [ 81.333096] 0000000000000000 ffff8800044c79c8 ffffffff8151b0c9
>>> ffff8800044c7a48
>>> [ 81.341262] ffffffff8151ae1e 0000000000000008 ffff8800044c7a58
>>> ffff8800044c79f8
>>> [ 81.349428] ffffffff810785c3 ffffffff81a13480 0000000000000000
>>> ffff8800001001d0
>>> [ 81.357595] Call Trace:
>>> [ 81.360287] [<ffffffff8151b0c9>] dump_stack+0x68/0x77
>>> [ 81.365942] [<ffffffff8151ae1e>] panic+0xb9/0x219
>>> [ 81.371213] [<ffffffff810785c3>] ?
>>> __blocking_notifier_call_chain+0x63/0x80
>>> [ 81.378971] [<ffffffff811384ce>] __out_of_memory+0x34e/0x350
>>> [ 81.385292] [<ffffffff811385ee>] out_of_memory+0x5e/0x90
>>> [ 81.391230] [<ffffffff8113ce9e>]
>>> __alloc_pages_slowpath+0x6be/0x740
>>> [ 81.398219] [<ffffffff8113d15c>]
>>> __alloc_pages_nodemask+0x23c/0x250
>>> [ 81.405212] [<ffffffff81186346>] kmem_getpages+0x56/0x110
>>> [ 81.411246] [<ffffffff81187f44>] fallback_alloc+0x164/0x200
>>> [ 81.417474] [<ffffffff81187cfd>] ____cache_alloc_node+0x8d/0x170
>>> [ 81.424179] [<ffffffff811887bb>]
>>> kmem_cache_alloc_trace+0x17b/0x240
>>> [ 81.431169] [<ffffffff813d5f3a>] init_memory_block+0x3a/0x110
>>> [ 81.437586] [<ffffffff81b5f687>] memory_dev_init+0xd7/0x13d
>>> [ 81.443810] [<ffffffff81b5f2af>] driver_init+0x2f/0x37
>>> [ 81.449556] [<ffffffff81b1599b>] do_basic_setup+0x29/0xd5
>>> [ 81.455597] [<ffffffff81b372c4>] ? sched_init_smp+0x140/0x147
>>> [ 81.462015] [<ffffffff81b15c55>]
>>> kernel_init_freeable+0x20e/0x297
>>> [ 81.468815] [<ffffffff81512ea0>] ? rest_init+0x80/0x80
>>> [ 81.474565] [<ffffffff81512ea9>] kernel_init+0x9/0xf0
>>> [ 81.480216] [<ffffffff8151f788>] ret_from_fork+0x58/0x90
>>> [ 81.486156] [<ffffffff81512ea0>] ? rest_init+0x80/0x80
>>> [ 81.492350] ---[ end Kernel panic - not syncing: Out of memory
>>> and no killable processes...
>>> [ 81.492350]
>>>
>>> -Longman
>>
>> I increased the pre-initialized memory per node in
>> update_defer_init() of mm/page_alloc.c from 2G to 4G. Now I am able
>> to boot the 24-TB machine without error. The 12-TB has 0.75TB/node,
>> while the 24-TB machine has 1.5TB/node. I would suggest something
>> like pre-initializing 1G per 0.25TB/node. In this way, it will scale
>> properly with the memory size.
>>
>> Before the patch, the boot time from elilo prompt to ssh login was
>> 694s. After the patch, the boot up time was 346s, a saving of 348s
>> (about 50%).
>
> I second scaling the up-front init with the zone size. The 7TB system
> I was booting has only 32GB per NUMA node, which at 1GB per 0.25TB
> would work out at 128MB up-front init per-NUMA-node, which worked
> nice and booted faster yet.
>
> Even booting with 64MB per NUMA node worked great, so there is
> adequate margin for the 8 cores, just I guess we'd need to enforce a
> minimum of eg 64MB or so.
Varying the synchronous per-NUMA-node initialisation (with non-temporal
patch, but that just removes a constant from PMD init), from kernel
load to login prompt on this 7TB, 1728-core system takes:
512MB 699.2s
256MB 680.3s
128MB 661.7s
64MB 663.6s
32MB 667.8s
So, in this case 128MB per NUMA node gives more locality than 64MB, so
should be a good minimum, and matches Waiman's scaling suggestion.
Thanks,
Daniel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2015-05-02 16:06 UTC|newest]
Thread overview: 81+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-04-28 14:36 [PATCH 0/13] Parallel struct page initialisation v4 Mel Gorman
2015-04-28 14:36 ` [PATCH 01/13] memblock: Introduce a for_each_reserved_mem_region iterator Mel Gorman
2015-04-28 14:36 ` [PATCH 02/13] mm: meminit: Move page initialization into a separate function Mel Gorman
2015-04-28 14:37 ` [PATCH 03/13] mm: meminit: Only set page reserved in the memblock region Mel Gorman
2015-05-22 20:31 ` Tony Luck
2015-05-26 10:22 ` Mel Gorman
2015-04-28 14:37 ` [PATCH 04/13] mm: page_alloc: Pass PFN to __free_pages_bootmem Mel Gorman
2015-05-01 9:20 ` [PATCH] mm: page_alloc: pass PFN to __free_pages_bootmem -fix Mel Gorman
2015-04-28 14:37 ` [PATCH 05/13] mm: meminit: Make __early_pfn_to_nid SMP-safe and introduce meminit_pfn_in_nid Mel Gorman
2015-04-28 14:37 ` [PATCH 06/13] mm: meminit: Inline some helper functions Mel Gorman
2015-04-30 21:53 ` Andrew Morton
2015-04-30 21:55 ` Andrew Morton
2015-05-04 8:33 ` Michal Hocko
2015-05-04 8:38 ` Michal Hocko
2015-04-28 14:37 ` [PATCH 07/13] mm: meminit: Initialise a subset of struct pages if CONFIG_DEFERRED_STRUCT_PAGE_INIT is set Mel Gorman
2015-04-29 21:19 ` Andrew Morton
2015-04-30 8:45 ` Mel Gorman
2015-05-01 9:21 ` [PATCH] mm: meminit: Initialise a subset of struct pages if CONFIG_DEFERRED_STRUCT_PAGE_INIT is set -fix Mel Gorman
2015-07-14 15:54 ` 4.2-rc2: hitting "file-max limit 8192 reached" Dave Hansen
2015-07-14 16:15 ` Andrew Morton
2015-07-15 10:45 ` Mel Gorman
2015-04-28 14:37 ` [PATCH 08/13] mm: meminit: Initialise remaining struct pages in parallel with kswapd Mel Gorman
2015-04-28 14:37 ` [PATCH 09/13] mm: meminit: Minimise number of pfn->page lookups during initialisation Mel Gorman
2015-04-28 14:37 ` [PATCH 10/13] x86: mm: Enable deferred struct page initialisation on x86-64 Mel Gorman
2015-04-28 14:37 ` [PATCH 11/13] mm: meminit: Free pages in large chunks where possible Mel Gorman
2015-04-28 14:37 ` [PATCH 12/13] mm: meminit: Reduce number of times pageblocks are set during struct page init Mel Gorman
2015-05-01 9:23 ` [PATCH] mm: meminit: Reduce number of times pageblocks are set during struct page init -fix Mel Gorman
2015-04-28 14:37 ` [PATCH 13/13] mm: meminit: Remove mminit_verify_page_links Mel Gorman
2015-04-28 16:06 ` [PATCH 0/13] Parallel struct page initialisation v4 Pekka Enberg
2015-04-28 18:38 ` nzimmer
2015-04-30 16:10 ` Daniel J Blueman
2015-04-30 17:12 ` nzimmer
2015-04-30 17:28 ` Mel Gorman
2015-05-02 11:52 ` Elliott, Robert (Server Storage)
2015-04-29 1:16 ` Waiman Long
2015-05-01 22:02 ` Waiman Long
2015-05-02 0:09 ` Waiman Long
2015-05-02 8:52 ` Daniel J Blueman
2015-05-02 16:05 ` Daniel J Blueman [this message]
2015-05-04 21:30 ` Andrew Morton
2015-05-05 3:32 ` Waiman Long
2015-05-05 10:45 ` Mel Gorman
2015-05-05 13:55 ` Waiman Long
2015-05-05 14:31 ` Mel Gorman
2015-05-05 15:01 ` Waiman Long
2015-05-06 3:39 ` Waiman Long
2015-05-06 0:55 ` Waiman Long
2015-05-05 20:02 ` Andrew Morton
2015-05-05 22:13 ` Mel Gorman
2015-05-05 22:25 ` Andrew Morton
2015-05-06 7:12 ` Mel Gorman
2015-05-06 10:22 ` Mel Gorman
2015-05-06 12:05 ` Mel Gorman
2015-05-06 17:58 ` Waiman Long
2015-05-07 2:37 ` Waiman Long
2015-05-07 7:21 ` Mel Gorman
2015-05-06 1:21 ` Waiman Long
2015-05-06 2:01 ` Andrew Morton
2015-05-07 7:25 ` [PATCH] mm: meminit: Finish initialisation of struct pages before basic setup Mel Gorman
2015-05-07 22:09 ` Andrew Morton
2015-05-07 22:52 ` Mel Gorman
2015-05-07 23:02 ` Andrew Morton
2015-05-13 15:53 ` nzimmer
2015-05-13 16:31 ` Mel Gorman
2015-05-14 10:03 ` Daniel J Blueman
2015-05-14 15:47 ` nzimmer
2015-05-19 18:31 ` nzimmer
2015-05-19 19:06 ` Mel Gorman
2015-05-22 6:30 ` Daniel J Blueman
2015-05-22 9:33 ` Mel Gorman
2015-05-22 17:14 ` Waiman Long
2015-05-22 21:43 ` Davidlohr Bueso
2015-05-23 3:49 ` Daniel J Blueman
2015-06-24 22:50 ` Nathan Zimmer
2015-06-25 20:48 ` Mel Gorman
2015-06-25 20:57 ` Mel Gorman
2015-06-25 21:37 ` Nathan Zimmer
2015-06-25 21:34 ` Nathan Zimmer
2015-06-26 10:16 ` Mel Gorman
2015-07-06 17:45 ` Daniel J Blueman
2015-07-09 17:49 ` Nathan Zimmer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1430582749.21217.0@cpanel21.proisp.no \
--to=daniel@numascale.com \
--cc=akpm@linux-foundation.org \
--cc=dave.hansen@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=nzimmer@sgi.com \
--cc=scott.norton@hp.com \
--cc=waiman.long@hp.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).