[PATCH 2.6.13-rc6] cpu_exclusive sched domains on partial nodes temp fix

[Paul Jackson <pj@sgi.com>]

If Dinakar, Hawkes and Nick concur (and no one else complains too
loud) then the following should go into 2.6.13, to avoid the potential
kernel oops that Hawkes reported in Dinakar's feature to allow user
control of dynamic sched domain placement using cpu_exclusive cpusets.

This patch keeps the kernel/cpuset.c routine update_cpu_domains()
from invoking the sched.c routine partition_sched_domains() if the
cpuset in question doesn't fall on node boundaries.

I have boot tested this on an SN2, and with the help of a couple of
ad hoc printk's, determined that it does indeed avoid calling the
partition_sched_domains() routine on partial nodes.

I did not directly verify that this avoids setting up bogus sched
domains or avoids the oops that Hawkes saw.

Obviously, if the above named parties decide to take some other path,
then this patch should be discarded.  I submit this patch under the
expectation that Hawkes and others fixes to support sched domains not
on node boundaries will go into *-mm and 2.6.14.  Do not include the
following patch in *-mm or 2.6.14 versions which have the real sched
domain fixes.

This patch imposes a silent artificial constraint on which cpusets
can be used to define dynamic sched domains.

This patch should allow proceeding with this new feature in 2.6.13 for
the configurations in which it is useful (node alligned sched domains)
while avoiding trying to setup sched domains in the less useful cases
that can cause the kernel corruption and oops.

Signed-off-by: Paul Jackson <pj@sgi.com>

Index: linux-2.6.13-cpuset-mempolicy-migrate/kernel/cpuset.c
===================================================================
--- linux-2.6.13-cpuset-mempolicy-migrate.orig/kernel/cpuset.c
+++ linux-2.6.13-cpuset-mempolicy-migrate/kernel/cpuset.c
@@ -636,6 +636,23 @@ static void update_cpu_domains(struct cp
 		return;
 
 	/*
+	 * Hack to avoid 2.6.13 partial node dynamic sched domain bug.
+	 * Require the 'cpu_exclusive' cpuset to include all (or none)
+	 * of the CPUs on each node, or return w/o changing sched domains.
+	 * Remove this hack when dynamic sched domains fixed.
+	 */
+	{
+		int i, j;
+
+		for_each_cpu_mask(i, cur->cpus_allowed) {
+			for_each_cpu_mask(j, node_to_cpumask(cpu_to_node(i))) {
+				if (!cpu_isset(j, cur->cpus_allowed))
+					return;
+			}
+		}
+	}
+
+	/*
 	 * Get all cpus from parent's cpus_allowed not part of exclusive
 	 * children
 	 */

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373