Re: [PATCH part2 v2 0/8] Arrange hotpluggable memory as ZONE_MOVABLE

[Tejun Heo <tj@kernel.org>]

Hello, Yinghai.

On Mon, Oct 14, 2013 at 12:34:49PM -0700, Yinghai Lu wrote:
> The points for parsing SRAT early instead of Yanfei/Tang v7:
>
> 1. We just reached one unified path to setup page tables for 32bit,
> 64bit and xen or non xen after several years. We should not have add
> another path for system
> that support hotplug.

The separate code path we're talking about is tiny.  It's just an
extra function for page table allocation and another for memblock
allocation which is symmetric to the existing one.  Sure, there are
benefits to not diverging code paths but these are fairly trivial in
terms of maintenance overhead and test coverage.

> 2. also we should avoid adding "movable_nodes" command line.

Can we?  What about the pgdat?  We're allocating them off-node with
movable_nodes which can't be the default behavior.

> 3. debug mapping 4k, and it is working all the way, why breaking it even for
> memory hotplug path?

If it comes for free, sure, no reason to break it.  On the other hand,
if maintaining it fully with a niche feature costs overhead, it's
somethinig to be traded-off.  It's not like using 4k page mapping with
bottom-up allocation will be immediately broken either.  It might
affect devices which can't DMA to higher addresses on gigantic
machines under debug configs.  It's quite a corner case.

> 4. numa_meminfo now is static structure.
> we have no reason that we can not parse SRAT etc to fill that struct.

Sure, there's no reason we can't.  The whole point is that the
benefits arent' strong enough.  We don't do things just because we
can.

> 5. for device tree, i assume that we could do same like srat parsing to find out
> numa to fill the numa_meminfo early. or with help of BRK.

Digesting device tree involves a lot more complexity.  The whole
reason why things like SRAT are broken into tables in the first place.
We'll be basically pulling in huge chunk of ACPICA into early boot.
Again, justfications.  The *only* thing which may benefit from that
are debug setups.  We'll have to pull in a lot of complexity before
page table setup and modify page table allocation to be
memory-device-specific just for debug configs, which is not a good
trade-off.  Benefit / cost ratio doesn't make any sense.

> 6. in the long run, We should rework our NUMA booting:
> a. boot system with boot numa nodes early only.
> b. in later init stage or user space, init other nodes
> RAM/CPU/PCI...in parallel.
> that will reduce boot time for 8 sockets/32 sockets dramatically.
> 
> We will need to parse srat table early so could avoid init memory for
> non-boot nodes.

Among the six you listed, this one sounds somewhat valid but still
assuming huge page, what difference does it make?  We're just talking
about page table alloc / init and ACPI init.  If you wanna speed up
huge NUMA machine booting and chop down memory init per-NUMA, sure,
move those pieces to later stages.  You can init the amount necessary
during early boot and then bring up the rest later on.  I don't see
why that'd require parsing SRAT.  In fact, I think there'll be more
cases where you want to actively ignore NUMA mapping during early
boot.  What if the system maps low memory to a non-boot numa node?

Optimizing NUMA boot just requires moving the heavy lifting to
appropriate NUMA nodes.  It doesn't require that early boot phase
should strictly follow NUMA node boundaries.

Thanks.

-- 
tejun