Re: [sched] fix sched_domains hotplug bootstrap ordering vs. cpu_online_map issue

[James Bottomley <James.Bottomley@SteelEye.com>]

On Sun, 2004-09-05 at 07:46, Ingo Molnar wrote:
> > cpu_online_map is not set up at the time of sched domain
> > initialization when hotplug cpu paths are used for SMP booting. At
> > this phase of bootstrapping, cpu_possible_map can be used by the
> > various architectures using cpu hotplugging for SMP bootstrap, but the
> > manipulations of cpu_online_map done on behalf of NUMA architectures,
> > done indirectly via node_to_cpumask(), can't, because cpu_online_map
> > starts depopulated and hasn't yet been populated. On true NUMA
> > architectures this is a distinct cpumask_t from cpu_online_map and so
> > the unpatched code works on NUMA; on non-NUMA architectures the
> > definition of node_to_cpumask() this way breaks and would require an
> > invasive sweeping of users of node_to_cpumask() to change it to e.g.
> > cpu_possible_map, as cpu_possible_map is not suitable for use at
> > runtime as a substitute for cpu_online_map.
> > 
> > Signed-off-by: William Irwin <wli@holomorphy.com>
> 
> Signed-off-by: Ingo Molnar <mingo@elte.hu>

Well this patch got in, which is what I want, since it allows the
non-NUMA machines to work with hotplug CPUs again.  However, is anyone
actually looking to fix this for real?

The fundamental problem is that NUMA or the scheduler (or both) are
broken with regard to hotplug.

The origin of the breakage is the differences between cpu_possible_map
and cpu_online_map.  In hotplug CPU, there are two ways to do
initialisations: you can initialise from cpu_online_map, but then you
*must* have a cpu hotplug notify listener to add data structures for the
extra CPUs as they come on-line, or you can initialise from
cpu_possible_map and not bother with a notifier.  The disadvantage of
the latter is that cpu_possible_map may be vastly larger than
cpu_online_map ever gets to, thus wasting valuable kernel memory.

The scheduler code is schizophrenic in this regard in that it does both:
it initialises static data structures from cpu_possible_map, but it also
has a hotplug cpu listener for starting things like the migration
threads.

I suspect the NUMA people would like us all to go to the former method
(initialise only from cpu_online_map and have a proper hotplug listener)
since their possible maps are pretty huge.  However, which is it to be:
fix NUMA (to have two cpu_to_node() maps for the possible and online
cpus per node) or fix the scheduler to do initialisation correctly?

Perhaps this should be phased: change NUMA first temporarily for phase
one and then fix the scheduler (and everyone else initialising from
cpu_possible_map) in the second.

James