Re: [PATCH] x86: Consider multiple nodes in a single socket to be "sane"

[Peter Zijlstra <peterz@infradead.org>]

On Tue, Sep 16, 2014 at 01:17:44AM -0700, Dave Hansen wrote:
> On 09/15/2014 08:29 PM, Peter Zijlstra wrote:

> > What is that cluster-on-die thing? I've heard it before but never could
> > find anything on it.
> 
> You split each socket in to halves.  Each half gets slightly lower
> latency to the memory on its half's memory controller.  But, at an
> increased latency cost to go to every other bit of memory on the system.
>  I've got some data squirreled away on exactly what the latencies are,
> but it really is a substantial increase for memory, especially for the
> memory that is still on package but is now "off-node".

So it really splits the local memory DIMMs in two as well, ok.

> I went back and re-read the text around there.  I don't see any mention
> of nodes or NUMA.  It's just talking about the topology within the CPU
> from what I can tell.
> 
> Good thing it's just in "Documentation/ABI/testing/" I guess.
> 
> Either way, are you saying that a core's "physical_package_id" should
> depend on what BIOS options get set, and that different cores in the
> same physical package should have different "physical_package_id"s?

Yah, I more or less changed my mind halfway through the reply. I suppose
it makes sense to have this mask as defined. It just doesn't match which
how the mask is used in the scheduler (which is the main in-kernel user
or the thing).

> > The idea is that core_siblings (or rather cpu_core_mask) is a mask of
> > all cores on a node. 
> 
> Why would we need that in the CPU topology information if we can get at
> it easily here?
> 
> 	/sys/devices/system/cpu/cpu0/node0/cpulist

Good point, the 'problem' is that we currently have the static order of
the masks, if we were to flip the MC and NUMA masks we need a condition
to do that on and make sure everything is aware of that.

CoD not being detectable sucks arse for sure :/

Also, I think we want to rename the mask to not be MC but PKG or so.

> s390 has this concept of a "book" although it's not obvious at all where
> this fits in to the hierarchy to me.  The help text and comments are
> pretty funny:
> 
> 	Book scheduler support improves the CPU scheduler's decision
> 	making when dealing with machines that have several books.
> 
> Wow, really?  Book scheduling helps with books?  I would have never
> guessed.  Wish I knew what a book was. :)  Google helped a bit:
> 
> 	http://sysmagazine.com/posts/150430/
> 
> but it's still not clear.  I think a book is below a node?
> 
> 	SMT <= LLC <= MC <= BOOK <= NODE
> 
> Or is a book a collection of nodes?
> 
> 	SMT <= LLC <= MC <= NODE <= BOOK

s390 is 'special' you should know that :-) Since s390 is a virtualized
platform they cannot do NUMA since the vCPU can move around freely. So
what they do to 'hide' stuff is big L4 caches, which they call BOOKS.

> > The scheduler assumes: SMT <= LLC <= MC <= NODE and if setting the MC
> > mask to cover multiple nodes works, its by accident.
> 
> This makes a lot of sense to enforce inside the scheduler.  But, do we
> really need to expose all the naming all the way out to userspace?

Dunno.. lets start out by not doing that (the safe option).