Re: [PATCH v2] NOHZ: fix nohz on cpu unplug

[Suresh Siddha <suresh.b.siddha@intel.com>]

On Mon, 2009-02-02 at 04:23 -0800, Christian Borntraeger wrote:
> After some testing, I think this patch is better:
> [PATCH] NOHZ: fix nohz on cpu unplug
> 
> From: Christian Borntraeger <borntraeger@de.ibm.com>
> 
> After using cpu unplug I have seen one cpu with full ticks, even
> on a idle systems. It turns out that nohz.cpu_mask is not updated on
> cpu unplug. 
> In select_nohz_load_balancer we check if the system is completely
> idle to turn of load balancing. We compare cpu_online_map with
> nohz.cpu_mask.
> Since cpu_online_map is updated on cpu unplug, but nohz.cpu_mask is
> not, the check fails and the scheduler believes that we need an 
> "idle load balancer" even on a fully idle system. Since the ilb 
> cpu does not deactivate the timer tick this breaks NOHZ.
> 
> This patch clear the nohz bits in the migration_call, a function that
> is already called by the cpu hotplug notifier.
> 
> Opinions?

Hi,

select_nohz_load_balancer() already takes care of one cpu offline
scenario. It missed one more scenario leading to the issue you
explained. We should take care of this in select_nohz_load_balancer().

Does this patch fix the issue? Thanks.

---
diff --git a/kernel/sched.c b/kernel/sched.c
index 96439a4..8a0419b 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -4026,19 +4026,24 @@ int select_nohz_load_balancer(int stop_tick)
 	int cpu = smp_processor_id();
 
 	if (stop_tick) {
-		cpumask_set_cpu(cpu, nohz.cpu_mask);
 		cpu_rq(cpu)->in_nohz_recently = 1;
 
-		/*
-		 * If we are going offline and still the leader, give up!
-		 */
-		if (!cpu_active(cpu) &&
-		    atomic_read(&nohz.load_balancer) == cpu) {
+		if (!cpu_active(cpu) {
+			if (atomic_read(&nohz.load_balancer) != cpu)
+				return 0;
+
+			/*
+			 * If we are going offline and still the leader,
+			 * give up!
+			 */
 			if (atomic_cmpxchg(&nohz.load_balancer, cpu, -1) != cpu)
 				BUG();
+
 			return 0;
 		}
 
+		cpumask_set_cpu(cpu, nohz.cpu_mask);
+
 		/* time for ilb owner also to sleep */
 		if (cpumask_weight(nohz.cpu_mask) == num_online_cpus()) {
 			if (atomic_read(&nohz.load_balancer) == cpu)