[PATCH 2.6.13-rc6] cpu_exclusive sched domains on partial nodes temp fix

[PATCH 2.6.13-rc6] cpu_exclusive sched domains on partial nodes temp fix

[Paul Jackson]

If Dinakar, Hawkes and Nick concur (and no one else complains too
loud) then the following should go into 2.6.13, to avoid the potential
kernel oops that Hawkes reported in Dinakar's feature to allow user
control of dynamic sched domain placement using cpu_exclusive cpusets.

This patch keeps the kernel/cpuset.c routine update_cpu_domains()
from invoking the sched.c routine partition_sched_domains() if the
cpuset in question doesn't fall on node boundaries.

I have boot tested this on an SN2, and with the help of a couple of
ad hoc printk's, determined that it does indeed avoid calling the
partition_sched_domains() routine on partial nodes.

I did not directly verify that this avoids setting up bogus sched
domains or avoids the oops that Hawkes saw.

Obviously, if the above named parties decide to take some other path,
then this patch should be discarded.  I submit this patch under the
expectation that Hawkes and others fixes to support sched domains not
on node boundaries will go into *-mm and 2.6.14.  Do not include the
following patch in *-mm or 2.6.14 versions which have the real sched
domain fixes.

This patch imposes a silent artificial constraint on which cpusets
can be used to define dynamic sched domains.

This patch should allow proceeding with this new feature in 2.6.13 for
the configurations in which it is useful (node alligned sched domains)
while avoiding trying to setup sched domains in the less useful cases
that can cause the kernel corruption and oops.

Signed-off-by: Paul Jackson <pj@sgi.com>

Index: linux-2.6.13-cpuset-mempolicy-migrate/kernel/cpuset.c
===================================================================
--- linux-2.6.13-cpuset-mempolicy-migrate.orig/kernel/cpuset.c
+++ linux-2.6.13-cpuset-mempolicy-migrate/kernel/cpuset.c
@@ -636,6 +636,23 @@ static void update_cpu_domains(struct cp
 		return;
 
 	/*
+	 * Hack to avoid 2.6.13 partial node dynamic sched domain bug.
+	 * Require the 'cpu_exclusive' cpuset to include all (or none)
+	 * of the CPUs on each node, or return w/o changing sched domains.
+	 * Remove this hack when dynamic sched domains fixed.
+	 */
+	{
+		int i, j;
+
+		for_each_cpu_mask(i, cur->cpus_allowed) {
+			for_each_cpu_mask(j, node_to_cpumask(cpu_to_node(i))) {
+				if (!cpu_isset(j, cur->cpus_allowed))
+					return;
+			}
+		}
+	}
+
+	/*
 	 * Get all cpus from parent's cpus_allowed not part of exclusive
 	 * children
 	 */

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: [PATCH 2.6.13-rc6] cpu_exclusive sched domains on partial nodes temp fix

[Ingo Molnar]

* Paul Jackson <pj@sgi.com> wrote:

>  	/*
> +	 * Hack to avoid 2.6.13 partial node dynamic sched domain bug.
> +	 * Require the 'cpu_exclusive' cpuset to include all (or none)
> +	 * of the CPUs on each node, or return w/o changing sched domains.
> +	 * Remove this hack when dynamic sched domains fixed.
> +	 */
> +	{
> +		int i, j;
> +
> +		for_each_cpu_mask(i, cur->cpus_allowed) {
> +			for_each_cpu_mask(j, node_to_cpumask(cpu_to_node(i))) {
> +				if (!cpu_isset(j, cur->cpus_allowed))
> +					return;
> +			}
> +		}
> +	}
> +

certainly looks acceptable from a scheduler POV.

Acked-by: Ingo Molnar <mingo@elte.hu>

	Ingo

Re: [PATCH 2.6.13-rc6] cpu_exclusive sched domains on partial nodes temp fix

[Dinakar Guniguntala]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On Tue, Aug 23, 2005 at 01:04:27AM -0700, Paul Jackson wrote:
> If Dinakar, Hawkes and Nick concur (and no one else complains too
> loud) then the following should go into 2.6.13, to avoid the potential
> kernel oops that Hawkes reported in Dinakar's feature to allow user
> control of dynamic sched domain placement using cpu_exclusive cpusets.

I agree this is the way to go for 2.6.13 before we fix things the
right way for 2.6.14. Thanks for the patch Paul.

> This patch should allow proceeding with this new feature in 2.6.13 for
> the configurations in which it is useful (node alligned sched domains)
> while avoiding trying to setup sched domains in the less useful cases
> that can cause the kernel corruption and oops.
> 

Dunno if it is something in my setup (4 CPU Power5 box with NUMA enabled)
but this patch causes some hard hangs when I run the attached script.
The same script runs for much longer with Ingo's changes but panics
as I had described earlier. I am still debugging what causes this.

	-Dinakar



[-- Attachment #2: sd-stress.tar.gz --]
[-- Type: application/x-gzip, Size: 1075 bytes --]

Re: [PATCH 2.6.13-rc6] cpu_exclusive sched domains on partial nodes temp fix

[John Hawkes]

> + for_each_cpu_mask(i, cur->cpus_allowed) {
> + for_each_cpu_mask(j, node_to_cpumask(cpu_to_node(i))) {
> + if (!cpu_isset(j, cur->cpus_allowed))
> + return;

Looks good to me.
Feel free to add:
Acked-by: John Hawkes <hawkes@sgi.com>


John Hawkes