Re: [PATCH] x86: Consider multiple nodes in a single socket to be "sane"

Peter Zijlstra
2014-09-16T17:59:03Z

On Tue, Sep 16, 2014 at 01:17:44AM -0700, Dave Hansen wrote:
> On 09/15/2014 08:29 PM, Peter Zijlstra wrote:

> > What is that cluster-on-die thing? I've heard it before but never could
> > find anything on it.
>
> You split each socket in to halves. Each half gets slightly lower
> latency to the memory on its half's memory controller. But, at an
> increased latency cost to go to every other bit of memory on the system.
> I've got some data squirreled away on exactly what the latencies are,
> but it really is a substantial increase for memory, especially for the
> memory that is still on package but is now "off-node".

So it really splits the local memory DIMMs in two as well, ok.

> I went back and re-read the text around there. I don't see any mention
> of nodes or NUMA. It's just talking about the topology within the CPU
> from what I can tell.
>
> Good thing it's just in "Documentation/ABI/testing/" I guess.
>
> Either way, are you saying that a core's "physical_package_id" should
> depend on what BIOS options get set, and that different cores in the
> same physical package should have different "physical_package_id"s?

Yah, I more or less changed my mind halfway through the reply. I suppose
it makes sense to have this mask as defined. It just doesn't match which
how the mask is used in the scheduler (which is the main in-kernel user
or the thing).

> > The idea is that core_siblings (or rather cpu_core_mask) is a mask of
> > all cores on a node.
>
> Why would we need that in the CPU topology information if we can get at
> it easily here?
>
> /sys/devices/system/cpu/cpu0/node0/cpulist

Good point, the 'problem' is that we currently have the static order of
the masks, if we were to flip the MC and NUMA masks we need a condition
to do that on and make sure everything is aware of that.

CoD not being detectable sucks arse for sure :/

Also, I think we want to rename the mask to not be MC but PKG or so.

> s390 has this concept of a "book" although it's not obvious at all where
> this fits in to the hierarchy to me. The help text and comments are
> pretty funny:
>
> Book scheduler support improves the CPU scheduler's decision
> making when dealing with machines that have several books.
>
> Wow, really? Book scheduling helps with books? I would have never
> guessed. Wish I knew what a book was. :) Google helped a bit:
>
> http://sysmagazine.com/posts/150430/
>
> but it's still not clear. I think a book is below a node?
>
> SMT <= LLC <= MC <= BOOK <= NODE
>
> Or is a book a collection of nodes?
>
> SMT <= LLC <= MC <= NODE <= BOOK

s390 is 'special' you should know that :-) Since s390 is a virtualized
platform they cannot do NUMA since the vCPU can move around freely. So
what they do to 'hide' stuff is big L4 caches, which they call BOOKS.

> > The scheduler assumes: SMT <= LLC <= MC <= NODE and if setting the MC
> > mask to cover multiple nodes works, its by accident.
>
> This makes a lot of sense to enforce inside the scheduler. But, do we
> really need to expose all the naming all the way out to userspace?

Dunno.. lets start out by not doing that (the safe option).