Re: [PATCH] cpu-hotplug: Register update_sched_domains() notifier with higher prio

[Andrew Morton <akpm@linux-foundation.org>]

On Mon, 10 Mar 2008 18:43:55 +0530 Gautham R Shenoy <ego@in.ibm.com> wrote:

> cpu-hotplug: Register update_sched_domains() notifier with higher prio
> From: Gautham R Shenoy <ego@in.ibm.com>
> 
> The current -rt wake_up logic uses the rq->rd->online_map which is
> updated on cpu-hotplug events by update_sched_domains(). Currently
> update_sched_domains() is registered with a priority 0. It is
> preferable that update_sched_domains() be the first notifier called on
> a CPU_DEAD or CPU_ONLINE events inorder to ensure that the
> rq->rd->online_map is in sync with the cpu_online_map.
> 
> This way on a CPU_DOWN_PREPARE,
> we would destroy the sched_domains and reattach all the rq's to the
> def_root_domain. This will be followed by a CPU_DOWN_PREPARE in
> migration_call() which will clear the bit of the cpu going offline
> from the corresponding rq.
> This will ensure that no task will be woken up on the cpu which is
> going to be offlined between CPU_DOWN_PREPARE and CPU_DEAD.
> 
> Ditto for the CPU_ONLINE path.
> 
> Tested on a 4-way Intel Xeon Box with cpu-hotplug tests running in parallel
> with kernbench.
> 
> Signed-off-by: Gautham R Shenoy <ego@in.ibm.com>
> ---
> 
>  kernel/sched.c |   12 ++++++++++--
>  1 files changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/sched.c b/kernel/sched.c
> index 69ecb1a..374c46f 100644
> --- a/kernel/sched.c
> +++ b/kernel/sched.c
> @@ -7079,8 +7079,16 @@ void __init sched_init_smp(void)
>  	if (cpus_empty(non_isolated_cpus))
>  		cpu_set(smp_processor_id(), non_isolated_cpus);
>  	put_online_cpus();
> -	/* XXX: Theoretical race here - CPU may be hotplugged now */
> -	hotcpu_notifier(update_sched_domains, 0);
> +	/*
> +	 * XXX: Theoretical race here - CPU may be hotplugged now
> +	 *
> +	 * We register the notifier with priority 11, which means that
> +	 * update_sched_domains() will be called just before migration_call().
> +	 *
> +	 * This is necessary to ensure that the rt wake up logic works fine
> +	 * and the rq->rd->online_map remains in sync with the cpu_online_map.
> +	 */
> +	hotcpu_notifier(update_sched_domains, 11);
>  
>  	/* Move init over to a non-isolated CPU */
>  	if (set_cpus_allowed(current, non_isolated_cpus) < 0)

This fixes my poweroff-doesn't-work post-2.6.24 regression.