[PATCH 1/2] Revert "cpu hotplug: adjust root-domain->online span in response to hotplug

[PATCH 1/2] Revert "cpu hotplug: adjust root-domain->online span in response to hotplug

[Gregory Haskins]

event"

This reverts commit 393d94d98b19089ec172566e23557997931b137e.

I submitted 393d94d98b1 to -mm and sched-devel, and then quickly realized
it was flawed.  However, the patch was already "in flight" and made it
to Linus' tree and is in .25-rc5.  So this patch reverts the
original/flawed fix, and the patch that follows has the updated version.

Signed-off-by: Gregory Haskins <ghaskins@novell.com>
Cc: Gautham R Shenoy <ego@in.ibm.com>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
Cc: Andrew Morton <akpm@linux-foundation.org>
---

 kernel/sched.c |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/kernel/sched.c b/kernel/sched.c
index b02e4fc..52b9867 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -5813,13 +5813,6 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)
 		/* Must be high prio: stop_machine expects to yield to it. */
 		rq = task_rq_lock(p, &flags);
 		__setscheduler(rq, p, SCHED_FIFO, MAX_RT_PRIO-1);
-
-		/* Update our root-domain */
-		if (rq->rd) {
-			BUG_ON(!cpu_isset(cpu, rq->rd->span));
-			cpu_set(cpu, rq->rd->online);
-		}
-
 		task_rq_unlock(rq, &flags);
 		cpu_rq(cpu)->migration_thread = p;
 		break;
@@ -5828,6 +5821,15 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)
 	case CPU_ONLINE_FROZEN:
 		/* Strictly unnecessary, as first user will wake it. */
 		wake_up_process(cpu_rq(cpu)->migration_thread);
+
+		/* Update our root-domain */
+		rq = cpu_rq(cpu);
+		spin_lock_irqsave(&rq->lock, flags);
+		if (rq->rd) {
+			BUG_ON(!cpu_isset(cpu, rq->rd->span));
+			cpu_set(cpu, rq->rd->online);
+		}
+		spin_unlock_irqrestore(&rq->lock, flags);
 		break;
 
 #ifdef CONFIG_HOTPLUG_CPU
@@ -6103,6 +6105,8 @@ static void rq_attach_root(struct rq *rq, struct root_domain *rd)
 	rq->rd = rd;
 
 	cpu_set(rq->cpu, rd->span);
+	if (cpu_isset(rq->cpu, cpu_online_map))
+		cpu_set(rq->cpu, rd->online);
 
 	for (class = sched_class_highest; class; class = class->next) {
 		if (class->join_domain)

[PATCH 2/2] keep rd->online and cpu_online_map in sync

[Gregory Haskins]

It is possible to allow the root-domain cache of online cpus to
become out of sync with the global cpu_online_map.  This is because we
currently trigger removal of cpus too early in the notifier chain.
Other DOWN_PREPARE handlers may in fact run and reconfigure the
root-domain topology, thereby stomping on our own offline handling.

The end result is that rd->online may become out of sync with
cpu_online_map, which results in potential task misrouting.

So change the offline handling to be more tightly coupled with the
global offline process by triggering on CPU_DYING intead of
CPU_DOWN_PREPARE.

Signed-off-by: Gregory Haskins <ghaskins@novell.com>
Cc: Gautham R Shenoy <ego@in.ibm.com>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
Cc: Andrew Morton <akpm@linux-foundation.org>
---

 kernel/sched.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/kernel/sched.c b/kernel/sched.c
index 52b9867..a616fa1 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -5881,7 +5881,7 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)
 		spin_unlock_irq(&rq->lock);
 		break;
 
-	case CPU_DOWN_PREPARE:
+	case CPU_DYING:
 		/* Update our root-domain */
 		rq = cpu_rq(cpu);
 		spin_lock_irqsave(&rq->lock, flags);

Re: [PATCH 1/2] Revert "cpu hotplug: adjust root-domain->online span in response to hotplug

[Ingo Molnar]

* Gregory Haskins <ghaskins@novell.com> wrote:

> This reverts commit 393d94d98b19089ec172566e23557997931b137e.
> 
> I submitted 393d94d98b1 to -mm and sched-devel, and then quickly 
> realized it was flawed.  However, the patch was already "in flight" 
> and made it to Linus' tree and is in .25-rc5.  So this patch reverts 
> the original/flawed fix, and the patch that follows has the updated 
> version.

yeah, but the patch fixed Andrew's regression and passed my testing so i 
wanted to have _something_ in -rc5, even if we found the right fix.

i've applied your revert and the real fix now.

	Ingo

Re: [PATCH 1/2] Revert "cpu hotplug: adjust root-domain->onlinespan in response to hotplug

[Gregory Haskins]

>>> On Tue, Mar 11, 2008 at  4:52 AM, in message <20080311085201.GD25110@elte.hu>,
Ingo Molnar <mingo@elte.hu> wrote: 

> * Gregory Haskins <ghaskins@novell.com> wrote:
> 
>> This reverts commit 393d94d98b19089ec172566e23557997931b137e.
>> 
>> I submitted 393d94d98b1 to -mm and sched-devel, and then quickly 
>> realized it was flawed.  However, the patch was already "in flight" 
>> and made it to Linus' tree and is in .25-rc5.  So this patch reverts 
>> the original/flawed fix, and the patch that follows has the updated 
>> version.
> 
> yeah, but the patch fixed Andrew's regression and passed my testing so i 
> wanted to have _something_ in -rc5, even if we found the right fix.
> 
> i've applied your revert and the real fix now.


Hi Ingo,
  Understood, and no problem.  I just wanted to make sure I accurately described what I was doing ;)

Also, for the second patch, make sure you pick up "v2" which has both DYING/DYING_FROZEN.  The hotplug gurus indicated that I need both to work properly.

Thanks!

-Greg

Re: [PATCH 1/2] Revert "cpu hotplug: adjust root-domain->onlinespan in response to hotplug

[Ingo Molnar]

* Gregory Haskins <ghaskins@novell.com> wrote:

> Also, for the second patch, make sure you pick up "v2" which has both 
> DYING/DYING_FROZEN.  The hotplug gurus indicated that I need both to 
> work properly.

yes, it is v2 that i have.

	Ingo