From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1758117AbbJIQwf (ORCPT <rfc822;w@1wt.eu>);
	Fri, 9 Oct 2015 12:52:35 -0400
Received: from mx1.redhat.com ([209.132.183.28]:47291 "EHLO mx1.redhat.com"
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
	id S1751892AbbJIQwd (ORCPT <rfc822;linux-kernel@vger.kernel.org>);
	Fri, 9 Oct 2015 12:52:33 -0400
Date: Fri, 9 Oct 2015 18:49:14 +0200
From: Oleg Nesterov <oleg@redhat.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: heiko.carstens@de.ibm.com, Tejun Heo <tj@kernel.org>,
        Ingo Molnar <mingo@kernel.org>, Rik van Riel <riel@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>, linux-kernel@vger.kernel.org,
        Vitaly Kuznetsov <vkuznets@redhat.com>
Subject: Re: [PATCH 3/3] sched: start stopper early
Message-ID: <20151009164914.GA11947@redhat.com>
References: <20151009160031.GA5408@redhat.com> <20151009160054.GA10176@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20151009160054.GA10176@redhat.com>
User-Agent: Mutt/1.5.18 (2008-05-17)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 10/09, Oleg Nesterov wrote:
>
> From: Peter Zijlstra <peterz@infradead.org>

Peter, I tried to compromise you.

>  	case CPU_ONLINE:
> +		stop_machine_unpark(cpu);
>  		/*
>  		 * At this point a starting CPU has marked itself as online via
>  		 * set_cpu_online(). But it might not yet have marked itself
> @@ -5337,7 +5340,7 @@ static int sched_cpu_active(struct notifier_block *nfb,
>  		 * Thus, fall-through and help the starting CPU along.
>  		 */
>  	case CPU_DOWN_FAILED:
> -		set_cpu_active((long)hcpu, true);
> +		set_cpu_active(cpu, true);

On a second thought, we can't do this (and your initial change has
the same problem).

We can not wakeup it before set_cpu_active(). This can lead to the
same problem fixed by dd9d3843755da95f6 "sched: Fix cpu_active_mask/
cpu_online_mask race". The stopper thread can hit
BUG_ON(td->cpu != smp_processor_id()) in smpboot_thread_fn().

Easy to fix, CPU_ONLINE should do set_cpu_active() itself and not
fall through to CPU_DOWN_FAILED,

	case CPU_ONLINE:
		set_cpu_active(cpu, true);
		stop_machine_unpark(cpu);
		break;

But. This is another proof that stop_two_cpus() must not rely on
cpu_active().

Right?

Oleg.