Re: [PATCH 0/4] arch/CPU hotplug: Add missing CPU Hotplug bits to fix nasty issues

[Peter Zijlstra <a.p.zijlstra@chello.nl>]

On Thu, 2012-03-22 at 16:58 +0530, Srivatsa S. Bhat wrote:
> Commit 5fbd036b552f633abb394a319f7c62a5c86a9cd7 (sched: Cleanup cpu_active
> madness) introduced some changes that made the scheduler rely on the
> CPU_STARTING notifier. And hence those architectures which forgot to
> send out the CPU_STARTING notification will almost surely get into trouble.
> (Xen is one example[1]). 

The requirement is that CPU_STARTING is ran before set_cpu_online() or
the open-coded equivalent -- eg. blackfin gets this wrong.

However, it is also required that all this happens while local IRQs are
still disabled, since the moment we enable IRQs interrupts can happen
and interrupts can cause wakeups, and wakeups need to have this state
set up -- blackfin, cris, m32r, mips, sh, sparc64, sparc32, um?, x86 get
this wrong.

Furthermore, all archs that use CONFIG_USE_GENERIC_SMP_HELPERS should
hold ipi_call_lock() over setting the cpu online -- alpha, arm, m32r,
mips, sh, sparc32 seem wrong.

Furthermore, I was pondering the scenario where a 3rd cpu IPIs the newly
booting cpu, I suspect we need a smp_wmb() after setting cpu_active and
a rmb in select_fallback_rq() before reading active.

All in all its a complete friggin trainwreck.