Re: [PATCH 0/3] bootmem2 III

[Johannes Weiner <hannes@saeurebad.de>]

Hi Andy,

Andy Whitcroft <apw@shadowen.org> writes:

> On Tue, May 13, 2008 at 02:40:44PM +0200, Johannes Weiner wrote:
>> Hi,
>> 
>> Andi Kleen <andi@firstfloor.org> writes:
>> 
>> > Johannes Weiner wrote:
>> >
>> >>> On Fri, May 09, 2008 at 05:17:13PM +0200, Johannes Weiner wrote:
>> >>>> here is bootmem2, a memory block-oriented boot time allocator.
>> >>>>
>> >>>> Recent NUMA topologies broke the current bootmem's assumption that
>> >>>> memory nodes provide non-overlapping and contiguous ranges of pages.
>> >>> I'm still not sure that's a really good rationale for bootmem2.
>> >>> e.g. the non continuous nodes are really special cases and there tends
>> >>> to be enough memory at the beginning which is enough for boot time
>> >>> use, so for those systems it would be quite reasonably to only 
>> >>> put the continuous starts of the nodes into bootmem.
>> >> 
>> >> Hm, that would put the logic into arch-code.  I have no strong opinion
>> >> about it.
>> >
>> > In fact I suspect the current code will already work like that
>> > implicitely. The aliasing is only a problem for the new "arbitary node
>> > free_bootmem" right?
>> 
>> And that alloc_bootmem_node() can not garuantee node-locality which is
>> the much worse part, I think.
>> 
>> >>> That said the bootmem code has gotten a little crufty and a clean
>> >>> rewrite might be a good idea.
>> >> 
>> >> I agree completely.
>> >
>> > The trouble is just that bootmem is used in early boot and early boot is
>> > very subtle and getting it working over all architectures could be a
>> > challenge. Not wanting to discourage you, but it's not exactly the
>> > easiest part of the kernel to hack on.
>> 
>> Bootmem seemed pretty self-contained to me, at least in the beginning.
>> The bad thing is that I can test only the most simple configuration with
>> it.
>> 
>> I was wondering yesterday if it would be feasible to enforce
>> contiguousness for nodes.  So that arch-code does not create one pgdat
>> for each node but one for each contiguous block.  I have not yet looked
>
> That re-introduces the concept that a node is not a unit of numa locality,
> but one of memory contiguity.  The kernel pretty much assumes that a node
> exhibits memory locality.

Okay.

>> deeper into it, but I suspect that other mm code has similar problems
>> with nodes spanning other nodes.
>
> One thing we do know is that we already have systems in the wild with
> overlapping nodes.  PowerPC systems sometimes exhibit this behaviour, the
> ones I have seen have node 1 embedded within node 0.  x86_64 also enables
> this support.  This necessitated checks when initially freeing memory
> into the allocator to make sure it ended up freed into the right node.
> For non-sparsemem configurations these systems have some wasted mem_map,
> but otherwise it does work.
>
> Check out NODES_SPAN_OTHER_NODES for the code to avoid miss-placing
> memory.

Will have a better look at all this.  Thanks for the comment.

	Hannes