Re: [RFC PATCH 0/14] Parallel memory initialisation

[Daniel J Blueman <daniel@numascale.com>]

[-- Attachment #1: Type: text/plain, Size: 3242 bytes --]

On Monday, April 13, 2015 at 6:20:05 PM UTC+8, Mel Gorman wrote:
 > Memory initialisation had been identified as one of the reasons why 
large
 > machines take a long time to boot. Patches were posted a long time 
ago
 > that attempted to move deferred initialisation into the page 
allocator
 > paths. This was rejected on the grounds it should not be necessary 
to hurt
 > the fast paths to parallelise initialisation. This series reuses 
much of
 > the work from that time but defers the initialisation of memory to 
kswapd
 > so that one thread per node initialises memory local to that node. 
The
 > issue is that on the machines I tested with, memory initialisation 
was not
 > a major contributor to boot times. I'm posting the RFC to both 
review the
 > series and see if it actually helps users of very large machines.
 >
 > After applying the series and setting the appropriate Kconfig 
variable I
 > see this in the boot log on a 64G machine
 >
 > [    7.383764] kswapd 0 initialised deferred memory in 188ms
 > [    7.404253] kswapd 1 initialised deferred memory in 208ms
 > [    7.411044] kswapd 3 initialised deferred memory in 216ms
 > [    7.411551] kswapd 2 initialised deferred memory in 216ms
 >
 > On a 1TB machine, I see
 >
 > [   11.913324] kswapd 0 initialised deferred memory in 1168ms
 > [   12.220011] kswapd 2 initialised deferred memory in 1476ms
 > [   12.245369] kswapd 3 initialised deferred memory in 1500ms
 > [   12.271680] kswapd 1 initialised deferred memory in 1528ms
 >
 > Once booted the machine appears to work as normal. Boot times were 
measured
 > from the time shutdown was called until ssh was available again.  In 
the
 > 64G case, the boot time savings are negligible. On the 1TB machine, 
the
 > savings were 10 seconds (about 8% improvement on kernel times but 
1-2%
 > overall as POST takes so long).
 >
 > It would be nice if the people that have access to really large 
machines
 > would test this series and report back if the complexity is 
justified.

Nice work!

On an older Numascale system with 1TB memory and 256 cores/32 NUMA 
nodes, platform init takes 52s (cold boot), firmware takes 84s 
(includes one warm reboot), stock linux 4.0 then takes 732s to boot [1] 
(due to the 700ns roundtrip, RMW cache-coherent cycles due to the 
temporal writes for pagetable init and per-core store queue limits), so 
there is huge potential.

Alas I ran into crashing during list manipulation [2] which list 
debugging detects [3]; I had started adding some debug [4], but need to 
look a bit deeper into it. I annotated the time of the output from cold 
power on.

Thanks,
  Daniel

[1] https://resources.numascale.com/telemetry/defermem/console-stock.txt
[2] 
https://resources.numascale.com/telemetry/defermem/console-patched.txt
[3] 
https://resources.numascale.com/telemetry/defermem/console-patched-debug.txt

-- [4]

static void free_pcppages_bulk(struct zone *zone, int count,
					struct per_cpu_pages *pcp)
...
		pr_err("migrate_type=%d\n", migratetype);

		/* This is the only non-empty list. Free them all. */
		if (batch_free == MIGRATE_PCPTYPES)
			batch_free = to_free;

		do {
			int mt;	/* migratetype of the to-be-freed page */

			pr_err("list_empty=%d\n", list_empty(list));

[-- Attachment #2: Type: text/html, Size: 4893 bytes --]