All of lore.kernel.org
 help / color / mirror / Atom feed
From: Dave Hansen <dave.hansen@intel.com>
To: Nathan Zimmer <nzimmer@sgi.com>
Cc: hpa@zytor.com, mingo@kernel.org, linux-kernel@vger.kernel.org,
	linux-mm@kvack.org, holt@sgi.com, rob@landley.net,
	travis@sgi.com, daniel@numascale-asia.com,
	akpm@linux-foundation.org, gregkh@linuxfoundation.org,
	yinghai@kernel.org, mgorman@suse.de
Subject: Re: [RFC v3 0/5] Transparent on-demand struct page initialization embedded in the buddy allocator
Date: Fri, 16 Aug 2013 09:36:39 -0700	[thread overview]
Message-ID: <520E5517.9070606@intel.com> (raw)
In-Reply-To: <1376344480-156708-1-git-send-email-nzimmer@sgi.com>

Hey Nathan,

Could you post your boot timing patches?  My machines are much smaller
than yours, but I'm curious how things behave here as well.

I did some very imprecise timings (strace -t on a telnet attached to the
serial console).  The 'struct page' initializations take about a minute
of boot time for me to do 1TB across 8 NUMA nodes (this is a glueless
QPI system[1]).  My _quick_ calculations look like it's 2x as fast to
initialize node0's memory vs. the other nodes, and boot time is
increased by a second for about every 30G of memory we add.

So even with nothing else fancy, we could get some serious improvements
from just doing the initialization locally.

[1] We call anything using pure QPI without any other circuitry for the
NUMA interconnects to be "glueless"

	

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

WARNING: multiple messages have this Message-ID (diff)
From: Dave Hansen <dave.hansen@intel.com>
To: Nathan Zimmer <nzimmer@sgi.com>
Cc: hpa@zytor.com, mingo@kernel.org, linux-kernel@vger.kernel.org,
	linux-mm@kvack.org, holt@sgi.com, rob@landley.net,
	travis@sgi.com, daniel@numascale-asia.com,
	akpm@linux-foundation.org, gregkh@linuxfoundation.org,
	yinghai@kernel.org, mgorman@suse.de
Subject: Re: [RFC v3 0/5] Transparent on-demand struct page initialization embedded in the buddy allocator
Date: Fri, 16 Aug 2013 09:36:39 -0700	[thread overview]
Message-ID: <520E5517.9070606@intel.com> (raw)
In-Reply-To: <1376344480-156708-1-git-send-email-nzimmer@sgi.com>

Hey Nathan,

Could you post your boot timing patches?  My machines are much smaller
than yours, but I'm curious how things behave here as well.

I did some very imprecise timings (strace -t on a telnet attached to the
serial console).  The 'struct page' initializations take about a minute
of boot time for me to do 1TB across 8 NUMA nodes (this is a glueless
QPI system[1]).  My _quick_ calculations look like it's 2x as fast to
initialize node0's memory vs. the other nodes, and boot time is
increased by a second for about every 30G of memory we add.

So even with nothing else fancy, we could get some serious improvements
from just doing the initialization locally.

[1] We call anything using pure QPI without any other circuitry for the
NUMA interconnects to be "glueless"

	

  parent reply	other threads:[~2013-08-16 16:36 UTC|newest]

Thread overview: 153+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-07-12  2:03 [RFC 0/4] Transparent on-demand struct page initialization embedded in the buddy allocator Robin Holt
2013-07-12  2:03 ` Robin Holt
2013-07-12  2:03 ` [RFC 1/4] memblock: Introduce a for_each_reserved_mem_region iterator Robin Holt
2013-07-12  2:03   ` Robin Holt
2013-07-12  2:03 ` [RFC 2/4] Have __free_pages_memory() free in larger chunks Robin Holt
2013-07-12  2:03   ` Robin Holt
2013-07-12  7:45   ` Robin Holt
2013-07-12  7:45     ` Robin Holt
2013-07-13  3:08     ` Yinghai Lu
2013-07-13  3:08       ` Yinghai Lu
2013-07-16 13:02   ` Sam Ben
2013-07-16 13:02     ` Sam Ben
2013-07-23 15:32     ` Johannes Weiner
2013-07-23 15:32       ` Johannes Weiner
2013-07-12  2:03 ` [RFC 3/4] Seperate page initialization into a separate function Robin Holt
2013-07-12  2:03   ` Robin Holt
2013-07-13  3:06   ` Yinghai Lu
2013-07-13  3:06     ` Yinghai Lu
2013-07-15  3:19     ` Robin Holt
2013-07-15  3:19       ` Robin Holt
2013-07-12  2:03 ` [RFC 4/4] Sparse initialization of struct page array Robin Holt
2013-07-12  2:03   ` Robin Holt
2013-07-13  4:19   ` Yinghai Lu
2013-07-13  4:19     ` Yinghai Lu
2013-07-13  4:39     ` H. Peter Anvin
2013-07-13  4:39       ` H. Peter Anvin
2013-07-13  5:31       ` Yinghai Lu
2013-07-13  5:31         ` Yinghai Lu
2013-07-13  5:38         ` H. Peter Anvin
2013-07-13  5:38           ` H. Peter Anvin
2013-07-15 14:08         ` Nathan Zimmer
2013-07-15 14:08           ` Nathan Zimmer
2013-07-15 17:45     ` Nathan Zimmer
2013-07-15 17:45       ` Nathan Zimmer
2013-07-15 17:54       ` H. Peter Anvin
2013-07-15 17:54         ` H. Peter Anvin
2013-07-15 18:26         ` Robin Holt
2013-07-15 18:26           ` Robin Holt
2013-07-15 18:29           ` H. Peter Anvin
2013-07-15 18:29             ` H. Peter Anvin
2013-07-23  8:32             ` Ingo Molnar
2013-07-23  8:32               ` Ingo Molnar
2013-07-23 11:09               ` Robin Holt
2013-07-23 11:09                 ` Robin Holt
2013-07-23 11:15                 ` Robin Holt
2013-07-23 11:15                   ` Robin Holt
2013-07-23 11:41                   ` Robin Holt
2013-07-23 11:41                     ` Robin Holt
2013-07-23 11:50                     ` Robin Holt
2013-07-23 11:50                       ` Robin Holt
2013-07-16 10:26     ` Robin Holt
2013-07-16 10:26       ` Robin Holt
2013-07-25  2:25     ` Robin Holt
2013-07-25  2:25       ` Robin Holt
2013-07-25 12:50       ` Yinghai Lu
2013-07-25 12:50         ` Yinghai Lu
2013-07-25 13:42         ` Robin Holt
2013-07-25 13:42           ` Robin Holt
2013-07-25 13:52           ` Yinghai Lu
2013-07-25 13:52             ` Yinghai Lu
2013-07-15 21:30   ` Andrew Morton
2013-07-15 21:30     ` Andrew Morton
2013-07-16 10:38     ` Robin Holt
2013-07-16 10:38       ` Robin Holt
2013-07-12  8:27 ` [RFC 0/4] Transparent on-demand struct page initialization embedded in the buddy allocator Ingo Molnar
2013-07-12  8:27   ` Ingo Molnar
2013-07-12  8:47   ` boot tracing Borislav Petkov
2013-07-12  8:47     ` Borislav Petkov
2013-07-12  8:53     ` Ingo Molnar
2013-07-12  8:53       ` Ingo Molnar
2013-07-15  1:38       ` Sam Ben
2013-07-15  1:38         ` Sam Ben
2013-07-23  8:18         ` Ingo Molnar
2013-07-23  8:18           ` Ingo Molnar
2013-07-12  9:19   ` [RFC 0/4] Transparent on-demand struct page initialization embedded in the buddy allocator Robert Richter
2013-07-12  9:19     ` Robert Richter
2013-07-15 15:16   ` Robin Holt
2013-07-15 15:16     ` Robin Holt
2013-07-16  8:55   ` Joonsoo Kim
2013-07-16  8:55     ` Joonsoo Kim
2013-07-16  9:08     ` Borislav Petkov
2013-07-16  9:08       ` Borislav Petkov
2013-07-23  8:20       ` Ingo Molnar
2013-07-23  8:20         ` Ingo Molnar
2013-07-15 15:00 ` Robin Holt
2013-07-15 15:00   ` Robin Holt
2013-07-17  5:17 ` Sam Ben
2013-07-17  5:17   ` Sam Ben
2013-07-17  9:30   ` Robin Holt
2013-07-17  9:30     ` Robin Holt
2013-07-19 23:51     ` Yinghai Lu
2013-07-22  6:13       ` Robin Holt
2013-07-22  6:13         ` Robin Holt
2013-08-02 17:44 ` [RFC v2 0/5] " Nathan Zimmer
2013-08-02 17:44   ` Nathan Zimmer
2013-08-02 17:44   ` [RFC v2 1/5] memblock: Introduce a for_each_reserved_mem_region iterator Nathan Zimmer
2013-08-02 17:44     ` Nathan Zimmer
2013-08-02 17:44   ` [RFC v2 2/5] Have __free_pages_memory() free in larger chunks Nathan Zimmer
2013-08-02 17:44     ` Nathan Zimmer
2013-08-02 17:44   ` [RFC v2 3/5] Move page initialization into a separate function Nathan Zimmer
2013-08-02 17:44     ` Nathan Zimmer
2013-08-02 17:44   ` [RFC v2 4/5] Only set page reserved in the memblock region Nathan Zimmer
2013-08-02 17:44     ` Nathan Zimmer
2013-08-03 20:04     ` Nathan Zimmer
2013-08-03 20:04       ` Nathan Zimmer
2013-08-02 17:44   ` [RFC v2 5/5] Sparse initialization of struct page array Nathan Zimmer
2013-08-02 17:44     ` Nathan Zimmer
2013-08-05  9:58   ` [RFC v2 0/5] Transparent on-demand struct page initialization embedded in the buddy allocator Ingo Molnar
2013-08-05  9:58     ` Ingo Molnar
2013-08-12 21:54   ` [RFC v3 " Nathan Zimmer
2013-08-12 21:54     ` Nathan Zimmer
2013-08-12 21:54     ` [RFC v3 1/5] memblock: Introduce a for_each_reserved_mem_region iterator Nathan Zimmer
2013-08-12 21:54       ` Nathan Zimmer
2013-08-12 21:54     ` [RFC v3 2/5] Have __free_pages_memory() free in larger chunks Nathan Zimmer
2013-08-12 21:54       ` Nathan Zimmer
2013-08-12 21:54     ` [RFC v3 3/5] Move page initialization into a separate function Nathan Zimmer
2013-08-12 21:54       ` Nathan Zimmer
2013-08-12 21:54     ` [RFC v3 4/5] Only set page reserved in the memblock region Nathan Zimmer
2013-08-12 21:54       ` Nathan Zimmer
2013-08-12 21:54     ` [RFC v3 5/5] Sparse initialization of struct page array Nathan Zimmer
2013-08-12 21:54       ` Nathan Zimmer
2013-08-13 10:58     ` [RFC v3 0/5] Transparent on-demand struct page initialization embedded in the buddy allocator Ingo Molnar
2013-08-13 10:58       ` Ingo Molnar
2013-08-13 17:09     ` Linus Torvalds
2013-08-13 17:09       ` Linus Torvalds
2013-08-13 17:23       ` H. Peter Anvin
2013-08-13 17:23         ` H. Peter Anvin
2013-08-13 17:33       ` Mike Travis
2013-08-13 17:33         ` Mike Travis
2013-08-13 17:51         ` Linus Torvalds
2013-08-13 17:51           ` Linus Torvalds
2013-08-13 18:04           ` Mike Travis
2013-08-13 18:04             ` Mike Travis
2013-08-13 19:06             ` Mike Travis
2013-08-13 19:06               ` Mike Travis
2013-08-13 20:24               ` Yinghai Lu
2013-08-13 20:24                 ` Yinghai Lu
2013-08-13 20:37                 ` Mike Travis
2013-08-13 20:37                   ` Mike Travis
2013-08-13 21:35             ` Nathan Zimmer
2013-08-13 21:35               ` Nathan Zimmer
2013-08-13 23:10           ` Nathan Zimmer
2013-08-13 23:10             ` Nathan Zimmer
2013-08-13 23:55             ` Linus Torvalds
2013-08-13 23:55               ` Linus Torvalds
2013-08-14 11:27               ` Ingo Molnar
2013-08-14 11:27                 ` Ingo Molnar
2013-08-14 11:05           ` Ingo Molnar
2013-08-14 11:05             ` Ingo Molnar
2013-08-14 22:15             ` Nathan Zimmer
2013-08-14 22:15               ` Nathan Zimmer
2013-08-16 16:36     ` Dave Hansen [this message]
2013-08-16 16:36       ` Dave Hansen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=520E5517.9070606@intel.com \
    --to=dave.hansen@intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=daniel@numascale-asia.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=holt@sgi.com \
    --cc=hpa@zytor.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mgorman@suse.de \
    --cc=mingo@kernel.org \
    --cc=nzimmer@sgi.com \
    --cc=rob@landley.net \
    --cc=travis@sgi.com \
    --cc=yinghai@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.