Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis <travis@sgi.com>]

On 6/21/2013 11:36 AM, Yinghai Lu wrote:
> On Fri, Jun 21, 2013 at 9:25 AM, Nathan Zimmer <nzimmer@sgi.com> wrote:
>> This rfc patch set delays initializing large sections of memory until we have
>> started cpus.  This has the effect of reducing startup times on large memory
>> systems.  On 16TB it can take over an hour to boot and most of that time
>> is spent initializing memory.
> 
> One hour on system with 16T ram? BIOS or OS?

The BIOS is about 20 minutes *before* the 1+ hour.  (When we started this
UV project way back when, 8TB took over 4 hours before we threw in the towel.)
> 
> I use wall clock to check bootime on one system with 3T and 16 pcie cards,
> Linus only takes about 3m and 30 seconds from bootloader.

I can send some stats on where various delays are but most of it was in
memory initialization.  On average UV nodes carry 128 or 256G per node,
so 12 nodes would take about 3 or 4 minutes, perhaps more.
> 
> wonder if you boot delay is with so many cpu get onlined in serialized mode.

Nope.  If that was the case, delaying memory but initializing all the
cpus would not affect the time.
> 
> so can you try boot your system with "maxcpus=128" to get the boot time with
> wall clock ?

We could try if you really think it will provide any useful info.  Not sure
exactly how having memory on nodes with no active cpus will react.
> 
> Thanks
> 
> Yinghai
>