Re: [RFC 0/2] Delay initializing of large sections of memory

[Mike Travis <travis@sgi.com>]

On 6/21/2013 5:23 PM, Yinghai Lu wrote:
> On Fri, Jun 21, 2013 at 2:30 PM, Mike Travis <travis@sgi.com> wrote:
>>
>>
>> On 6/21/2013 11:50 AM, Greg KH wrote:
>>> On Fri, Jun 21, 2013 at 11:44:22AM -0700, Yinghai Lu wrote:
>>>> On Fri, Jun 21, 2013 at 10:03 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>>>>> On 06/21/2013 09:51 AM, Greg KH wrote:
>>>>>
>>>>> I suspect the cutoff for this should be a lot lower than 8 TB even, more
>>>>> like 128 GB or so.  The only concern is to not set the cutoff so low
>>>>> that we can end up running out of memory or with suboptimal NUMA
>>>>> placement just because of this.
>>>>
>>>> I would suggest another way:
>>>> only boot the system with boot node (include cpu, ram and pci root buses).
>>>> then after boot, could add other nodes.
>>>
>>> What exactly do you mean by "after boot"?  Often, the boot process of
>>> userspace needs those additional cpus and ram in order to initialize
>>> everything (like the pci devices) properly.
>>
>> Exactly.  That's why I left both low and high memory on each node.
> 
> looks like you assume every node have same ram, and before booting you
> you need to know memory layout to append the boot command line.
> 
> We have patchset that moving srat table parse early.
> git://git.kernel.org/pub/scm/linux/kernel/git/yinghai/linux-yinghai.git
> for-x86-mm
> https://git.kernel.org/cgit/linux/kernel/git/yinghai/linux-yinghai.git/log/?h=for-x86-mm
> 
> on top that, we could make your patch pass more simple command like
> 1/2^n of every node, and only need to pass n instead.

The two params that I couldn't figure out how to provide except via kernel
param option was the memory block size (128M or 2G) and the physical
address space per node.  The other 3 params can be automatically
setup by a script when the total system size is known.  As soon as we
verify on the 32TB system and surmise what will be needed for 64TB,
then those 3 params can probably disappear.




> 
> Thanks
> 
> Yinghai
>