Re: [Patch] numa:x86_64: Cacheline aliasing makes for_each_populated_zone extremely expensive -V2.

["H. Peter Anvin" <hpa@zytor.com>]

On 08/20/2010 08:03 AM, Robin Holt wrote:
> 
> In short, without the cpu information, I think we are heading back to
> as much of a kludge as I had originally submitted.  We could assume
> the number of sets will always be less than some large value like 16MB,
> but that runs the risk of wasting a large amount of memory.
> 
> Alternatively, we could base the color value upon something very concrete.
> For this particular allocation, we have an array of structures whose
> elements are 1792 bytes long (28 cache lines).  If I specify an offset
> of 29, it merely means the first element of my newly allocated array
> is now going to collide with the first allocation's second element.
> I really see no advantage to further allocating space.  The advantage
> to this method is it entirely removes the processor configuration from
> the question.  It allows me to keep the offset calculation from polluting
> the e820 allocator as well.  Basically, the change remains localized.
> 

That's pretty much the idea, really.  However, I don't think you can
localize the change without making invalid assumptions of the behavior
of lower primitives, which given that changes are in progress will cause
serious problems.

The issue here is that the e820 allocator (which is about to be axed,
but its successor will need to support similar operations) has a few
parameters that it takes in: (start, end, size, alignment).  It will
return a block at address (addr) fulfilling the requirements:

	start <= addr
	addr+size <= end
	(addr % alignment) == 0

However, for coloring (which is what you're doing here, coloring doesn't
have to be precise) what you really want is for the last constraint to
read like:

	start <= addr
	addr+size <= end
	(addr % alignment) == offset

You can leave your alignment some arbitrarily large value (in the case
of your 1792-byte structure, you can make the observation that
1792*4096 < 8 MiB) and the alignment is simply 1792*(node number).  This
will over-color massively, of course, *but you're not allocating memory
you don't need* and so it doesn't really matter.

However, this does mean that there is a need to be able to pass the
offset parameter down to the allocator.  Not doing that will either mean
wasting huge amount of memory or relying on internal behavior of the
allocator which is already scheduled to change.

Does this make sense?

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.