[PATCH] top level scheduler domain for ia64

[PATCH] top level scheduler domain for ia64

[Jesse Barnes]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

Some have noticed that the overlapping sched domains code doesn't quite work 
as intended (it results in disjoint domains on some machines), and that a top 
level, machine spanning domain is needed.  This patch from John Hawkes adds 
it to the ia64 code.  This allows processes to run on all CPUs in large 
systems, though balancing is limited.  It should go to Linus soon now 
otherwise large systems will only have ~16p (depending on topology) usable by 
the scheduler.  I sanity checked it on a small system after rediffing John's 
original, and he's done some testing on very large systems.

Nick, can you buy off on the sched.c change?  Alternatively, do you want to 
send that fix separately John?

Signed-off-by: Jesse Barnes <jbarnes@sgi.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>

Thanks,
Jesse

[-- Attachment #2: sched-domains-top-level-3.patch --]
[-- Type: text/plain, Size: 3203 bytes --]

===== arch/ia64/kernel/domain.c 1.3 vs edited =====
--- 1.3/arch/ia64/kernel/domain.c	2004-10-18 22:26:51 -07:00
+++ edited/arch/ia64/kernel/domain.c	2004-10-19 14:18:07 -07:00
@@ -119,6 +119,14 @@
  */
 static DEFINE_PER_CPU(struct sched_domain, node_domains);
 static struct sched_group *sched_group_nodes[MAX_NUMNODES];
+
+static DEFINE_PER_CPU(struct sched_domain, allnodes_domains);
+static struct sched_group sched_group_allnodes[MAX_NUMNODES];
+
+static int __devinit cpu_to_allnodes_group(int cpu)
+{
+	return cpu_to_node(cpu);
+}
 #endif
 
 /*
@@ -149,9 +157,21 @@
 		cpus_and(nodemask, nodemask, cpu_default_map);
 
 #ifdef CONFIG_NUMA
+		if (num_online_cpus()
+				> SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
+			sd = &per_cpu(allnodes_domains, i);
+			*sd = SD_ALLNODES_INIT;
+			sd->span = cpu_default_map;
+			group = cpu_to_allnodes_group(i);
+			sd->groups = &sched_group_allnodes[group];
+			p = sd;
+		} else
+			p = NULL;
+
 		sd = &per_cpu(node_domains, i);
 		*sd = SD_NODE_INIT;
 		sd->span = sched_domain_node_span(node);
+		sd->parent = p;
 		cpus_and(sd->span, sd->span, cpu_default_map);
 #endif
 
@@ -201,6 +221,9 @@
 	}
 
 #ifdef CONFIG_NUMA
+	init_sched_build_groups(sched_group_allnodes, cpu_default_map,
+				&cpu_to_allnodes_group);
+
 	for (i = 0; i < MAX_NUMNODES; i++) {
 		/* Set up node groups */
 		struct sched_group *sg, *prev;
@@ -282,6 +305,15 @@
 		power = SCHED_LOAD_SCALE + SCHED_LOAD_SCALE *
 				(cpus_weight(sd->groups->cpumask)-1) / 10;
 		sd->groups->cpu_power = power;
+
+#ifdef CONFIG_NUMA
+		sd = &per_cpu(allnodes_domains, i);
+		if (sd->groups) {
+			power = SCHED_LOAD_SCALE + SCHED_LOAD_SCALE *
+				(cpus_weight(sd->groups->cpumask)-1) / 10;
+			sd->groups->cpu_power = power;
+		}
+#endif
 	}
 
 #ifdef CONFIG_NUMA
===== include/asm-ia64/topology.h 1.12 vs edited =====
--- 1.12/include/asm-ia64/topology.h	2004-10-18 22:26:52 -07:00
+++ edited/include/asm-ia64/topology.h	2004-10-19 14:18:06 -07:00
@@ -58,7 +58,26 @@
 				| SD_BALANCE_EXEC	\
 				| SD_WAKE_BALANCE,	\
 	.last_balance		= jiffies,		\
-	.balance_interval	= 10,			\
+	.balance_interval	= 1,			\
+	.nr_balance_failed	= 0,			\
+}
+
+/* sched_domains SD_ALLNODES_INIT for IA64 NUMA machines */
+#define SD_ALLNODES_INIT (struct sched_domain) {	\
+	.span			= CPU_MASK_NONE,	\
+	.parent			= NULL,			\
+	.groups			= NULL,			\
+	.min_interval		= 80,			\
+	.max_interval		= 320,			\
+	.busy_factor		= 320,			\
+	.imbalance_pct		= 125,			\
+	.cache_hot_time		= (10*1000000),		\
+	.cache_nice_tries	= 1,			\
+	.per_cpu_gain		= 100,			\
+	.flags			= SD_LOAD_BALANCE	\
+				| SD_BALANCE_EXEC,	\
+	.last_balance		= jiffies,		\
+	.balance_interval	= 100*(63+num_online_cpus())/64,   \
 	.nr_balance_failed	= 0,			\
 }
 
===== kernel/sched.c 1.367 vs edited =====
--- 1.367/kernel/sched.c	2004-10-18 22:26:52 -07:00
+++ edited/kernel/sched.c	2004-10-19 14:18:06 -07:00
@@ -4378,11 +4378,10 @@
 			printk("domain %d: ", level);
 
 			if (!(sd->flags & SD_LOAD_BALANCE)) {
-				printk("does not balance");
+				printk("does not load-balance");
 				if (sd->parent)
 					printk(" ERROR !SD_LOAD_BALANCE domain has parent");
 				printk("\n");
-				break;
 			}
 
 			printk("span %s\n", str);

Re: [PATCH] top level scheduler domain for ia64

[Nick Piggin]

Jesse Barnes wrote:
> Some have noticed that the overlapping sched domains code doesn't quite work 
> as intended (it results in disjoint domains on some machines), and that a top 
> level, machine spanning domain is needed.  This patch from John Hawkes adds 
> it to the ia64 code.  This allows processes to run on all CPUs in large 
> systems, though balancing is limited.  It should go to Linus soon now 
> otherwise large systems will only have ~16p (depending on topology) usable by 
> the scheduler.  I sanity checked it on a small system after rediffing John's 
> original, and he's done some testing on very large systems.
> 
> Nick, can you buy off on the sched.c change?  Alternatively, do you want to 
> send that fix separately John?

No, that looks good. It is a pretty trivial change so I think its harmless
to go in with this patch.

I'd consider increasing balancing rates for the SD_NODE_INIT domains after
this patch goes in too.

RE: [PATCH] top level scheduler domain for ia64

[Luck, Tony]

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

>Some have noticed that the overlapping sched domains code 
>doesn't quite work as intended (it results in disjoint domains
>on some machines), and that a top level, machine spanning domain
>is needed.

Why is the solution to jam this into the ia64 specific code?  From
this description it sounds like a generic scheduler problem, so
the solution ought to be up in some generic code.

+	.min_interval		= 80,			\
+	.max_interval		= 320,			\
+	.busy_factor		= 320,			\
+	.imbalance_pct		= 125,			\
+	.cache_hot_time		= (10*1000000),		\
+	.balance_interval	= 100*(63+num_online_cpus())/64,   \

That's a lot of magic numbers and formulae ... are they right?
How would a user know if they are right.

>Nick, can you buy off on the sched.c change?  Alternatively, 
>do you want to send that fix separately John?

I saw the ACK from Nick ... but kernel/sched.c changes will have
to go through Andrew, not me.  Are the ia64 and generic parts
separable?  If the sched.c change goes in, do other architectures
need to have some equivalent change?

-Tony

[-- Attachment #2: sched-domains-top-level-3.patch --]
[-- Type: application/octet-stream, Size: 3316 bytes --]

===== arch/ia64/kernel/domain.c 1.3 vs edited =====
--- 1.3/arch/ia64/kernel/domain.c	2004-10-18 22:26:51 -07:00
+++ edited/arch/ia64/kernel/domain.c	2004-10-19 14:18:07 -07:00
@@ -119,6 +119,14 @@
  */
 static DEFINE_PER_CPU(struct sched_domain, node_domains);
 static struct sched_group *sched_group_nodes[MAX_NUMNODES];
+
+static DEFINE_PER_CPU(struct sched_domain, allnodes_domains);
+static struct sched_group sched_group_allnodes[MAX_NUMNODES];
+
+static int __devinit cpu_to_allnodes_group(int cpu)
+{
+	return cpu_to_node(cpu);
+}
 #endif
 
 /*
@@ -149,9 +157,21 @@
 		cpus_and(nodemask, nodemask, cpu_default_map);
 
 #ifdef CONFIG_NUMA
+		if (num_online_cpus()
+				> SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
+			sd = &per_cpu(allnodes_domains, i);
+			*sd = SD_ALLNODES_INIT;
+			sd->span = cpu_default_map;
+			group = cpu_to_allnodes_group(i);
+			sd->groups = &sched_group_allnodes[group];
+			p = sd;
+		} else
+			p = NULL;
+
 		sd = &per_cpu(node_domains, i);
 		*sd = SD_NODE_INIT;
 		sd->span = sched_domain_node_span(node);
+		sd->parent = p;
 		cpus_and(sd->span, sd->span, cpu_default_map);
 #endif
 
@@ -201,6 +221,9 @@
 	}
 
 #ifdef CONFIG_NUMA
+	init_sched_build_groups(sched_group_allnodes, cpu_default_map,
+				&cpu_to_allnodes_group);
+
 	for (i = 0; i < MAX_NUMNODES; i++) {
 		/* Set up node groups */
 		struct sched_group *sg, *prev;
@@ -282,6 +305,15 @@
 		power = SCHED_LOAD_SCALE + SCHED_LOAD_SCALE *
 				(cpus_weight(sd->groups->cpumask)-1) / 10;
 		sd->groups->cpu_power = power;
+
+#ifdef CONFIG_NUMA
+		sd = &per_cpu(allnodes_domains, i);
+		if (sd->groups) {
+			power = SCHED_LOAD_SCALE + SCHED_LOAD_SCALE *
+				(cpus_weight(sd->groups->cpumask)-1) / 10;
+			sd->groups->cpu_power = power;
+		}
+#endif
 	}
 
 #ifdef CONFIG_NUMA
===== include/asm-ia64/topology.h 1.12 vs edited =====
--- 1.12/include/asm-ia64/topology.h	2004-10-18 22:26:52 -07:00
+++ edited/include/asm-ia64/topology.h	2004-10-19 14:18:06 -07:00
@@ -58,7 +58,26 @@
 				| SD_BALANCE_EXEC	\
 				| SD_WAKE_BALANCE,	\
 	.last_balance		= jiffies,		\
-	.balance_interval	= 10,			\
+	.balance_interval	= 1,			\
+	.nr_balance_failed	= 0,			\
+}
+
+/* sched_domains SD_ALLNODES_INIT for IA64 NUMA machines */
+#define SD_ALLNODES_INIT (struct sched_domain) {	\
+	.span			= CPU_MASK_NONE,	\
+	.parent			= NULL,			\
+	.groups			= NULL,			\
+	.min_interval		= 80,			\
+	.max_interval		= 320,			\
+	.busy_factor		= 320,			\
+	.imbalance_pct		= 125,			\
+	.cache_hot_time		= (10*1000000),		\
+	.cache_nice_tries	= 1,			\
+	.per_cpu_gain		= 100,			\
+	.flags			= SD_LOAD_BALANCE	\
+				| SD_BALANCE_EXEC,	\
+	.last_balance		= jiffies,		\
+	.balance_interval	= 100*(63+num_online_cpus())/64,   \
 	.nr_balance_failed	= 0,			\
 }
 
===== kernel/sched.c 1.367 vs edited =====
--- 1.367/kernel/sched.c	2004-10-18 22:26:52 -07:00
+++ edited/kernel/sched.c	2004-10-19 14:18:06 -07:00
@@ -4378,11 +4378,10 @@
 			printk("domain %d: ", level);
 
 			if (!(sd->flags & SD_LOAD_BALANCE)) {
-				printk("does not balance");
+				printk("does not load-balance");
 				if (sd->parent)
 					printk(" ERROR !SD_LOAD_BALANCE domain has parent");
 				printk("\n");
-				break;
 			}
 
 			printk("span %s\n", str);

Re: [PATCH] top level scheduler domain for ia64

[Nick Piggin]

Luck, Tony wrote:
>>Some have noticed that the overlapping sched domains code 
>>doesn't quite work as intended (it results in disjoint domains
>>on some machines), and that a top level, machine spanning domain
>>is needed.
> 
> 
> Why is the solution to jam this into the ia64 specific code?  From
> this description it sounds like a generic scheduler problem, so
> the solution ought to be up in some generic code.
> 

So far, ia64 is the only one that uses these overlapping domains.
The code to build them is complex enough that we didn't want to
put it into the generic code at the moment (but it is fairly well
contained, all just in arch/ia64/kernel/domain.c and asm/topology.h).

> +	.min_interval		= 80,			\
> +	.max_interval		= 320,			\
> +	.busy_factor		= 320,			\
> +	.imbalance_pct		= 125,			\
> +	.cache_hot_time		= (10*1000000),		\
> +	.balance_interval	= 100*(63+num_online_cpus())/64,   \
> 
> That's a lot of magic numbers and formulae ... are they right?
> How would a user know if they are right.
> 

To be honest you really wouldn't. It would take a lot of careful
testing on numerous workloads and systems. I believe SGI is
starting to do a bit of testing... I don't have the resources to
do many "real world" tests.

At this stage I wouldn't let them worry you too much :P
Hopefully they'll gradually improve.

> 
>>Nick, can you buy off on the sched.c change?  Alternatively, 
>>do you want to send that fix separately John?
> 
> 
> I saw the ACK from Nick ... but kernel/sched.c changes will have
> to go through Andrew, not me.  Are the ia64 and generic parts
> separable?  If the sched.c change goes in, do other architectures
> need to have some equivalent change?
> 

No, it should be ok as is. The parts should be ok to seperate if
you would prefer it that way.

Re: [PATCH] top level scheduler domain for ia64

[Jesse Barnes]

On Wednesday, October 20, 2004 10:48 am, Luck, Tony wrote:
> >Some have noticed that the overlapping sched domains code
> >doesn't quite work as intended (it results in disjoint domains
> >on some machines), and that a top level, machine spanning domain
> >is needed.
>
> Why is the solution to jam this into the ia64 specific code?  From
> this description it sounds like a generic scheduler problem, so
> the solution ought to be up in some generic code.

It used to be in generic code, but now it's arch specific, so each arch builds 
its own scheduling domains.  This patch adds some NUMA specific scheduling 
domain code to build a top level domain on boxes with lots of nodes.  It 
won't affect non-NUMA or small NUMA boxes.

> + .min_interval  = 80,   \
> + .max_interval  = 320,   \
> + .busy_factor  = 320,   \
> + .imbalance_pct  = 125,   \
> + .cache_hot_time  = (10*1000000),  \
> + .balance_interval = 100*(63+num_online_cpus())/64,   \
>
> That's a lot of magic numbers and formulae ... are they right?
> How would a user know if they are right.

John has run several tests on large systems to come up with something 
reasonable, but no doubt they could use more tweaking.  John or Nick, care to 
comment?

> >Nick, can you buy off on the sched.c change?  Alternatively,
> >do you want to send that fix separately John?
>
> I saw the ACK from Nick ... but kernel/sched.c changes will have
> to go through Andrew, not me.  Are the ia64 and generic parts
> separable?  If the sched.c change goes in, do other architectures
> need to have some equivalent change?

Ok, I'll send out the sched.c bit separately then as it's standalone.

Thanks,
Jesse

Re: [PATCH] top level scheduler domain for ia64

[Xavier Bru]

Hello Nick & all,

Nick Piggin wrote:

> Luck, Tony wrote:
>
>
>> +    .min_interval        = 80,            \
>> +    .max_interval        = 320,            \
>> +    .busy_factor        = 320,            \
>> +    .imbalance_pct        = 125,            \
>> +    .cache_hot_time        = (10*1000000),        \
>> +    .balance_interval    = 100*(63+num_online_cpus())/64,   \
>>
>> That's a lot of magic numbers and formulae ... are they right?
>> How would a user know if they are right.
>>
>
> To be honest you really wouldn't. It would take a lot of careful
> testing on numerous workloads and systems. I believe SGI is
> starting to do a bit of testing... I don't have the resources to
> do many "real world" tests.
>
> At this stage I wouldn't let them worry you too much :P
> Hopefully they'll gradually improve.

Why should'nt we use the node_distance() function to build in an 
independant way the Numa hierarchy and compute the right parameters for 
each level ?


-- 

 Sincères salutations.
_____________________________________________________________________
 
Xavier BRU                 BULL ISD/R&D/INTEL office:     FREC B1-422
tel : +33 (0)4 76 29 77 45                    http://www-frec.bull.fr
fax : +33 (0)4 76 29 77 70                 mailto:Xavier.Bru@bull.net
addr: BULL, 1 rue de Provence, BP 208, 38432 Echirolles Cedex, FRANCE
_____________________________________________________________________

Re: [PATCH] top level scheduler domain for ia64

[Nick Piggin]

Xavier Bru wrote:
> Hello Nick & all,
> 
> Nick Piggin wrote:
> 
>> Luck, Tony wrote:
>>
>>
>>> +    .min_interval        = 80,            \
>>> +    .max_interval        = 320,            \
>>> +    .busy_factor        = 320,            \
>>> +    .imbalance_pct        = 125,            \
>>> +    .cache_hot_time        = (10*1000000),        \
>>> +    .balance_interval    = 100*(63+num_online_cpus())/64,   \
>>>
>>> That's a lot of magic numbers and formulae ... are they right?
>>> How would a user know if they are right.
>>>
>>
>> To be honest you really wouldn't. It would take a lot of careful
>> testing on numerous workloads and systems. I believe SGI is
>> starting to do a bit of testing... I don't have the resources to
>> do many "real world" tests.
>>
>> At this stage I wouldn't let them worry you too much :P
>> Hopefully they'll gradually improve.
> 
> 
> Why should'nt we use the node_distance() function to build in an 
> independant way the Numa hierarchy and compute the right parameters for 
> each level ?
> 
> 

Hi Xavier,
That would probably be a good idea where possible, although for many
architectures this sort of information won't be available. It may be
that we ultimately will want to represent the NUMA topology with
node_distance being the first class function/measure (I personally
think sched-domains should be extended into the memory topology). At
the present time though, it would be a backward step to force everyone
to build a node_distance table.

Two things to note - first, even if node_distance does return something
meaningful, it still has to be input into a larger field of parameters,
so there will still be some heuristics/fudging/tuning going on.

Second, we can do runtime probing to gain more information. For example,
Ingo has a patch in the works that will compute real cache transfer
times between any two CPUs, which looks promising. We can query the
number of online CPUs when deciding on balancing rates, etc etc.

So in short, we basically want as much info as we can possibly gather...
and that is the easy part :( People need to do tests on their real life
workloads with real systems to convert this information into useful
parameters.

Anyway, the scheduler isn't _quite_ at the point where you want to be
doing serious fine tuning with it yet; we've got to get a few more
things to go in (eg. Ingo's patch, improvements from John Hawkes, some
performance patches from me, etc).

Re: [PATCH] top level scheduler domain for ia64

[Takayoshi Kochi]

[-- Attachment #1: Type: Text/Plain, Size: 4360 bytes --]

Hi Jesse,

From: Jesse Barnes <jbarnes@engr.sgi.com>
Subject: [PATCH] top level scheduler domain for ia64
Date: Tue, 19 Oct 2004 14:27:27 -0700

> Some have noticed that the overlapping sched domains code doesn't quite work 
> as intended (it results in disjoint domains on some machines), and that a top
> level, machine spanning domain is needed.  This patch from John Hawkes adds 
> it to the ia64 code.  This allows processes to run on all CPUs in large 
> systems, though balancing is limited.  It should go to Linus soon now 
> otherwise large systems will only have ~16p (depending on topology) usable by 
> the scheduler.  I sanity checked it on a small system after rediffing John's 
> original, and he's done some testing on very large systems.

Our 32way machine still isn't configured well with the overwrapping
domain partitioning.  CPUs 0-15 belongs to domain (0 1 2 3 4 5 6)
and CPUs 16-31 belongs to domain (0 1 4 5 6 7), which is assymmetric
and at least does not reflect the real connection.

dmesg of the machine is attached.

The following patch makes the ia64 domain partitioning (maybe Altix
specific ;) optional and makes the magic number (6) configurable.

What do you think of this?

Signed-off-by: Takayoshi Kochi <t-kochi@bq.jp.nec.com>

Index: 269-bk/include/asm-ia64/processor.h
===================================================================
--- 269-bk/include/asm-ia64/processor.h	(revision 48)
+++ 269-bk/include/asm-ia64/processor.h	(working copy)
@@ -20,8 +20,10 @@
 #include <asm/ptrace.h>
 #include <asm/ustack.h>
 
+#ifdef CONFIG_IA64_SCHED_SPAN
 /* Our arch specific arch_init_sched_domain is in arch/ia64/kernel/domain.c */
 #define ARCH_HAS_SCHED_DOMAIN
+#endif
 
 #define IA64_NUM_DBG_REGS	8
 /*
Index: 269-bk/arch/ia64/Kconfig
===================================================================
--- 269-bk/arch/ia64/Kconfig	(revision 48)
+++ 269-bk/arch/ia64/Kconfig	(working copy)
@@ -188,6 +188,24 @@
 	  or have huge holes in the physical address space for other reasons.
 	  See <file:Documentation/vm/numa> for more.
 
+config IA64_SCHED_SPAN
+	bool "Bounded spanning of scheduling domains for large NUMA"
+	depends on NUMA
+	default y if IA64_SGI_SN2
+	help
+	  Say Y to support bounded spanning of scheduling domains for large
+	  NUMA systems, which will bound the number of nodes to span
+	  for a top-level scheduling domain.  If not set, the top-level
+	  domain spans over all nodes.
+
+config IA64_SD_NODES_PER_DOMAIN
+	int "Nodes to span"
+	depends on IA64_SCHED_SPAN
+	default "6"
+	help
+	  This is the number of nodes to span for a top-level scheduling
+	  domain.
+
 config IA64_CYCLONE
 	bool "Cyclone (EXA) Time Source support"
 	help
Index: 269-bk/arch/ia64/kernel/domain.c
===================================================================
--- 269-bk/arch/ia64/kernel/domain.c	(revision 49)
+++ 269-bk/arch/ia64/kernel/domain.c	(working copy)
@@ -12,8 +12,6 @@
 #include <linux/init.h>
 #include <linux/topology.h>
 
-#define SD_NODES_PER_DOMAIN 6
-
 #ifdef CONFIG_NUMA
 /**
  * find_next_best_node - find the next node to include in a sched_domain
@@ -77,7 +75,7 @@
 	cpus_or(span, span, nodemask);
 	set_bit(node, used_nodes);
 
-	for (i = 1; i < SD_NODES_PER_DOMAIN; i++) {
+	for (i = 1; i < CONFIG_IA64_SD_NODES_PER_DOMAIN; i++) {
 		int next_node = find_next_best_node(node, used_nodes);
 		nodemask = node_to_cpumask(next_node);
 		cpus_or(span, span, nodemask);
@@ -158,7 +156,7 @@
 
 #ifdef CONFIG_NUMA
 		if (num_online_cpus()
-				> SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
+		    > CONFIG_IA64_SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
 			sd = &per_cpu(allnodes_domains, i);
 			*sd = SD_ALLNODES_INIT;
 			sd->span = cpu_default_map;
Index: 269-bk/arch/ia64/kernel/Makefile
===================================================================
--- 269-bk/arch/ia64/kernel/Makefile	(revision 48)
+++ 269-bk/arch/ia64/kernel/Makefile	(working copy)
@@ -14,7 +14,8 @@
 obj-$(CONFIG_IA64_PALINFO)	+= palinfo.o
 obj-$(CONFIG_IOSAPIC)		+= iosapic.o
 obj-$(CONFIG_MODULES)		+= module.o
-obj-$(CONFIG_SMP)		+= smp.o smpboot.o domain.o
+obj-$(CONFIG_SMP)		+= smp.o smpboot.o
+obj-$(CONFIG_IA64_SCHED_SPAN)	+= domain.o
 obj-$(CONFIG_PERFMON)		+= perfmon_default_smpl.o
 obj-$(CONFIG_IA64_CYCLONE)	+= cyclone.o
 obj-$(CONFIG_IA64_MCA_RECOVERY)	+= mca_recovery.o


---
Takayoshi Kochi

[-- Attachment #2: dmesg-domain --]
[-- Type: Text/Plain, Size: 9023 bytes --]

Brought up 32 CPUs
Total of 32 processors activated (47813.16 BogoMIPS).
CPU0:
 domain 0: span 0000000f
  groups: 00000001 00000002 00000004 00000008
  domain 1: span 00ffffff
   groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000
   domain 2: span ffffffff
    groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000
CPU1:
 domain 0: span 0000000f
  groups: 00000002 00000004 00000008 00000001
  domain 1: span 00ffffff
   groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000
   domain 2: span ffffffff
    groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000
CPU2:
 domain 0: span 0000000f
  groups: 00000004 00000008 00000001 00000002
  domain 1: span 00ffffff
   groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000
   domain 2: span ffffffff
    groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000
CPU3:
 domain 0: span 0000000f
  groups: 00000008 00000001 00000002 00000004
  domain 1: span 00ffffff
   groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000
   domain 2: span ffffffff
    groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000
CPU4:
 domain 0: span 000000f0
  groups: 00000010 00000020 00000040 00000080
  domain 1: span 00ffffff
   groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0000000f
   domain 2: span ffffffff
    groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f
CPU5:
 domain 0: span 000000f0
  groups: 00000020 00000040 00000080 00000010
  domain 1: span 00ffffff
   groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0000000f
   domain 2: span ffffffff
    groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f
CPU6:
 domain 0: span 000000f0
  groups: 00000040 00000080 00000010 00000020
  domain 1: span 00ffffff
   groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0000000f
   domain 2: span ffffffff
    groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f
CPU7:
 domain 0: span 000000f0
  groups: 00000080 00000010 00000020 00000040
  domain 1: span 00ffffff
   groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0000000f
   domain 2: span ffffffff
    groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f
CPU8:
 domain 0: span 00000f00
  groups: 00000100 00000200 00000400 00000800
  domain 1: span 00ffffff
   groups: 00000f00 0000f000 000f0000 00f00000 0000000f 000000f0
   domain 2: span ffffffff
    groups: 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0
CPU9:
 domain 0: span 00000f00
  groups: 00000200 00000400 00000800 00000100
  domain 1: span 00ffffff
   groups: 00000f00 0000f000 000f0000 00f00000 0000000f 000000f0
   domain 2: span ffffffff
    groups: 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0
CPU10:
 domain 0: span 00000f00
  groups: 00000400 00000800 00000100 00000200
  domain 1: span 00ffffff
   groups: 00000f00 0000f000 000f0000 00f00000 0000000f 000000f0
   domain 2: span ffffffff
    groups: 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0
CPU11:
 domain 0: span 00000f00
  groups: 00000800 00000100 00000200 00000400
  domain 1: span 00ffffff
   groups: 00000f00 0000f000 000f0000 00f00000 0000000f 000000f0
   domain 2: span ffffffff
    groups: 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0
CPU12:
 domain 0: span 0000f000
  groups: 00001000 00002000 00004000 00008000
  domain 1: span 00ffffff
   groups: 0000f000 000f0000 00f00000 0000000f 000000f0 00000f00
   domain 2: span ffffffff
    groups: 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00
CPU13:
 domain 0: span 0000f000
  groups: 00002000 00004000 00008000 00001000
  domain 1: span 00ffffff
   groups: 0000f000 000f0000 00f00000 0000000f 000000f0 00000f00
   domain 2: span ffffffff
    groups: 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00
CPU14:
 domain 0: span 0000f000
  groups: 00004000 00008000 00001000 00002000
  domain 1: span 00ffffff
   groups: 0000f000 000f0000 00f00000 0000000f 000000f0 00000f00
   domain 2: span ffffffff
    groups: 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00
CPU15:
 domain 0: span 0000f000
  groups: 00008000 00001000 00002000 00004000
  domain 1: span 00ffffff
   groups: 0000f000 000f0000 00f00000 0000000f 000000f0 00000f00
   domain 2: span ffffffff
    groups: 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00
CPU16:
 domain 0: span 000f0000
  groups: 00010000 00020000 00040000 00080000
  domain 1: span ffff00ff
   groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0
   domain 2: span ffffffff
    groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000
CPU17:
 domain 0: span 000f0000
  groups: 00020000 00040000 00080000 00010000
  domain 1: span ffff00ff
   groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0
   domain 2: span ffffffff
    groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000
CPU18:
 domain 0: span 000f0000
  groups: 00040000 00080000 00010000 00020000
  domain 1: span ffff00ff
   groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0
   domain 2: span ffffffff
    groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000
CPU19:
 domain 0: span 000f0000
  groups: 00080000 00010000 00020000 00040000
  domain 1: span ffff00ff
   groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0
   domain 2: span ffffffff
    groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000
CPU20:
 domain 0: span 00f00000
  groups: 00100000 00200000 00400000 00800000
  domain 1: span ffff00ff
   groups: 00f00000 0f000000 f0000000 0000000f 000000f0 000f0000
   domain 2: span ffffffff
    groups: 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000
CPU21:
 domain 0: span 00f00000
  groups: 00200000 00400000 00800000 00100000
  domain 1: span ffff00ff
   groups: 00f00000 0f000000 f0000000 0000000f 000000f0 000f0000
   domain 2: span ffffffff
    groups: 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000
CPU22:
 domain 0: span 00f00000
  groups: 00400000 00800000 00100000 00200000
  domain 1: span ffff00ff
   groups: 00f00000 0f000000 f0000000 0000000f 000000f0 000f0000
   domain 2: span ffffffff
    groups: 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000
CPU23:
 domain 0: span 00f00000
  groups: 00800000 00100000 00200000 00400000
  domain 1: span ffff00ff
   groups: 00f00000 0f000000 f0000000 0000000f 000000f0 000f0000
   domain 2: span ffffffff
    groups: 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000
CPU24:
 domain 0: span 0f000000
  groups: 01000000 02000000 04000000 08000000
  domain 1: span ffff00ff
   groups: 0f000000 f0000000 0000000f 000000f0 000f0000 00f00000
   domain 2: span ffffffff
    groups: 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000
CPU25:
 domain 0: span 0f000000
  groups: 02000000 04000000 08000000 01000000
  domain 1: span ffff00ff
   groups: 0f000000 f0000000 0000000f 000000f0 000f0000 00f00000
   domain 2: span ffffffff
    groups: 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000
CPU26:
 domain 0: span 0f000000
  groups: 04000000 08000000 01000000 02000000
  domain 1: span ffff00ff
   groups: 0f000000 f0000000 0000000f 000000f0 000f0000 00f00000
   domain 2: span ffffffff
    groups: 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000
CPU27:
 domain 0: span 0f000000
  groups: 08000000 01000000 02000000 04000000
  domain 1: span ffff00ff
   groups: 0f000000 f0000000 0000000f 000000f0 000f0000 00f00000
   domain 2: span ffffffff
    groups: 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000
CPU28:
 domain 0: span f0000000
  groups: 10000000 20000000 40000000 80000000
  domain 1: span ffff00ff
   groups: f0000000 0000000f 000000f0 000f0000 00f00000 0f000000
   domain 2: span ffffffff
    groups: f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000
CPU29:
 domain 0: span f0000000
  groups: 20000000 40000000 80000000 10000000
  domain 1: span ffff00ff
   groups: f0000000 0000000f 000000f0 000f0000 00f00000 0f000000
   domain 2: span ffffffff
    groups: f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000
CPU30:
 domain 0: span f0000000
  groups: 40000000 80000000 10000000 20000000
  domain 1: span ffff00ff
   groups: f0000000 0000000f 000000f0 000f0000 00f00000 0f000000
   domain 2: span ffffffff
    groups: f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000
CPU31:
 domain 0: span f0000000
  groups: 80000000 10000000 20000000 40000000
  domain 1: span ffff00ff
   groups: f0000000 0000000f 000000f0 000f0000 00f00000 0f000000
   domain 2: span ffffffff
    groups: f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000

Re: [PATCH] top level scheduler domain for ia64

[Jesse Barnes]

On Thursday, October 28, 2004 2:29 am, Takayoshi Kochi wrote:
> Our 32way machine still isn't configured well with the overwrapping
> domain partitioning.  CPUs 0-15 belongs to domain (0 1 2 3 4 5 6)
> and CPUs 16-31 belongs to domain (0 1 4 5 6 7), which is assymmetric
> and at least does not reflect the real connection.

Hm, I was afraid of that since the top level domain is only built if the 
system has more CPUs than SD_NODES_PER_DOMAIN * cpus_per_node.  We could 
change that to be a simple if (numnodes > SD_NODES_PER_DOMAIN) instead.

> dmesg of the machine is attached.
>
> The following patch makes the ia64 domain partitioning (maybe Altix
> specific ;) optional and makes the magic number (6) configurable.
>
> What do you think of this?

Maybe a boot parameter would be better for configuring SD_NODES_PER_DOMAIN?  
That would allow a single kernel binary to be configured to run well on many 
types of NUMA systems.  It would also mean you could play with very large 
node domains, with and w/o a top level domain.

Jesse

Re: [PATCH] top level scheduler domain for ia64

[Takayoshi Kochi]

Hi Jesse,

From: Jesse Barnes <jbarnes@engr.sgi.com>
Subject: Re: [PATCH] top level scheduler domain for ia64
Date: Thu, 28 Oct 2004 08:26:00 -0700

> > Our 32way machine still isn't configured well with the overwrapping
> > domain partitioning.  CPUs 0-15 belongs to domain (0 1 2 3 4 5 6)
> > and CPUs 16-31 belongs to domain (0 1 4 5 6 7), which is assymmetric
> > and at least does not reflect the real connection.
> 
> Hm, I was afraid of that since the top level domain is only built if the 
> system has more CPUs than SD_NODES_PER_DOMAIN * cpus_per_node.  We could 
> change that to be a simple if (numnodes > SD_NODES_PER_DOMAIN) instead.

But I don't think the domain partitioning thing will optimize anything
in all the cases that (numnodes > SD_NODES_PER_DOMAIN) is true.

> > dmesg of the machine is attached.
> >
> > The following patch makes the ia64 domain partitioning (maybe Altix
> > specific ;) optional and makes the magic number (6) configurable.
> >
> > What do you think of this?
> 
> Maybe a boot parameter would be better for configuring SD_NODES_PER_DOMAIN?  
> That would allow a single kernel binary to be configured to run well on many 
> types of NUMA systems.  It would also mean you could play with very large 
> node domains, with and w/o a top level domain.

It sounds good for -mm, but not for Linus?
That would be a pain for any ordinary user to test many parameters
for each configuration (and the number makes very little difference,
which would make it more difficult to find the best number).

What I'd like to do now is to get rid of subarch specific optimization
out of generic kernels.

Do you have any objection for the patch to be included in the mainline?

---
Takayoshi Kochi

Re: [PATCH] top level scheduler domain for ia64

[Jesse Barnes]

On Sunday, October 31, 2004 10:35 pm, Takayoshi Kochi wrote:
> > Maybe a boot parameter would be better for configuring
> > SD_NODES_PER_DOMAIN? That would allow a single kernel binary to be
> > configured to run well on many types of NUMA systems.  It would also mean
> > you could play with very large node domains, with and w/o a top level
> > domain.
>
> It sounds good for -mm, but not for Linus?
> That would be a pain for any ordinary user to test many parameters
> for each configuration (and the number makes very little difference,
> which would make it more difficult to find the best number).
>
> What I'd like to do now is to get rid of subarch specific optimization
> out of generic kernels.
>
> Do you have any objection for the patch to be included in the mainline?

I don't want it included since it hurts generic kernels on large systems (i.e. 
either they wouldn't boot or wouldn't see all the CPUs).  It should be 
possible to use a runtime check (whether a boot parameter or just a 
comparison on numnodes) instead to avoid non-overlapping domains.

If I understand you right, you don't want a top level domain for your 32 way 
systems, but you *do* want the node domains to span the whole thing.  Is that 
right?

If so, you could do something like this I think?

 if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
  SD_NODES_PER_DOMAIN = numnodes;
  build_node_domains(); /* each one spans the system */
 } else {
  SD_NODES_PER_DOMAIN = 4; /* or whatever */
  build_node_domains(); /* only spans nearby nodes */
  build_top_level_domain(); /* whole system, infrequently balanced */
 }

Would that address your concerns?

Jesse

Re: [PATCH] top level scheduler domain for ia64

[Matthew Wilcox]

On Mon, Nov 01, 2004 at 09:07:32AM -0800, Jesse Barnes wrote:
> If I understand you right, you don't want a top level domain for your 32 way 
> systems, but you *do* want the node domains to span the whole thing.  Is that 
> right?
> 
> If so, you could do something like this I think?
> 
>  if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
>   SD_NODES_PER_DOMAIN = numnodes;
>   build_node_domains(); /* each one spans the system */
>  } else {
>   SD_NODES_PER_DOMAIN = 4; /* or whatever */
>   build_node_domains(); /* only spans nearby nodes */
>   build_top_level_domain(); /* whole system, infrequently balanced */
>  }
> 
> Would that address your concerns?

Doesn't sound like a great idea.  HP's already shipping 128-way Superdome
IA-64 systems, and they'll want to be set up rather differently from the
Altix systems.  I think this code needs to be autotuning so it doesn't
need to be touched whenever a vendor releases a new configuration (I
think I heard that NASA's Altixes had a custom CPU brick with twice the
CPUs in it?)

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: [PATCH] top level scheduler domain for ia64

[Jesse Barnes]

On Monday, November 1, 2004 9:16 am, Matthew Wilcox wrote:
> On Mon, Nov 01, 2004 at 09:07:32AM -0800, Jesse Barnes wrote:
> > If I understand you right, you don't want a top level domain for your 32
> > way systems, but you *do* want the node domains to span the whole thing. 
> > Is that right?
> >
> > If so, you could do something like this I think?
> >
> >  if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
> >   SD_NODES_PER_DOMAIN = numnodes;
> >   build_node_domains(); /* each one spans the system */
> >  } else {
> >   SD_NODES_PER_DOMAIN = 4; /* or whatever */
> >   build_node_domains(); /* only spans nearby nodes */
> >   build_top_level_domain(); /* whole system, infrequently balanced */
> >  }
> >
> > Would that address your concerns?
>
> Doesn't sound like a great idea.  HP's already shipping 128-way Superdome
> IA-64 systems, and they'll want to be set up rather differently from the
> Altix systems.  I think this code needs to be autotuning so it doesn't
> need to be touched whenever a vendor releases a new configuration (I
> think I heard that NASA's Altixes had a custom CPU brick with twice the
> CPUs in it?)

Yeah, but still the same number of CPUs/FSB.  I agree that autotuning would be 
best (the above is a crude example of that).  Any suggestions?

Jesse

RE: [PATCH] top level scheduler domain for ia64

[Luck, Tony]

>> >  if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
>> >   SD_NODES_PER_DOMAIN = numnodes;
>> >   build_node_domains(); /* each one spans the system */
>> >  } else {
>> >   SD_NODES_PER_DOMAIN = 4; /* or whatever */

Ugh!  The magic '4' is going to be different for different machines,
so there is no value that will make all of the people happy all of
the time.

>Yeah, but still the same number of CPUs/FSB.  I agree that 
>autotuning would be 
>best (the above is a crude example of that).  Any suggestions?

Can't you use the SLIT table to build a hierachy of scheduler
domains that matches the actual hardware configuration, instead
of using compiled in constants or even boot-time command line
parameters?

-Tony

Re: [PATCH] top level scheduler domain for ia64

[Jesse Barnes]

On Monday, November 1, 2004 10:53 am, Luck, Tony wrote:
> >> >  if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
> >> >   SD_NODES_PER_DOMAIN = numnodes;
> >> >   build_node_domains(); /* each one spans the system */
> >> >  } else {
> >> >   SD_NODES_PER_DOMAIN = 4; /* or whatever */
>
> Ugh!  The magic '4' is going to be different for different machines,
> so there is no value that will make all of the people happy all of
> the time.

Right.

> >Yeah, but still the same number of CPUs/FSB.  I agree that
> >autotuning would be
> >best (the above is a crude example of that).  Any suggestions?
>
> Can't you use the SLIT table to build a hierachy of scheduler
> domains that matches the actual hardware configuration, instead
> of using compiled in constants or even boot-time command line
> parameters?

Maybe, but that will get complex very quickly I think.  Right now we have 
three domains on ia64, the cpu domain, the node domain, which contains 
several nodes worth of CPUs, and the top level domain, which spans the whole 
machine.

So there are two questions, how big should the node domain be and should it 
span the whole machine (avoiding the need for a top level domain)?  Obviously 
the answer is pretty machine specific, and I'm not sure the SLIT helps us 
much since its values are arbitrary distance values, not anything concrete.

Jesse

RE: [PATCH] top level scheduler domain for ia64

[Luck, Tony]

>Maybe, but that will get complex very quickly I think.  Right now we have 
>three domains on ia64, the cpu domain, the node domain, which contains 
>several nodes worth of CPUs, and the top level domain, which spans the whole 
>machine.
>
>So there are two questions, how big should the node domain be and should it 
>span the whole machine (avoiding the need for a top level domain)?  Obviously 
>the answer is pretty machine specific, and I'm not sure the SLIT helps us 
>much since its values are arbitrary distance values, not anything concrete.

I'd pose a broader question ... are the manufacturers of big machines happy
with three domains?  Perhaps it makes sense to allow for more levels that
match the physical parameters of the machine.  E.g. the NEC box has 4 cpus
per-node, and 4 nodes in a "super-node", and 2 "super-nodes" in a machine.
It would make sense to me if there was a scheduler domain level that would
handle balancing between the nodes in a super-node in addition to the top-level
domain to handle balancing between super-nodes.

While the values in SLIT can be somewhat abstract, they could be used to derive
the whole node, super-node, hyper-node, ultra-node, marketting-zeta-node
structure to build as many levels as make sense into the scheduler.

Or am I over-engineering?

-Tony

Re: [PATCH] top level scheduler domain for ia64

[Jesse Barnes]

On Monday, November 1, 2004 11:45 am, Luck, Tony wrote:
> I'd pose a broader question ... are the manufacturers of big machines happy
> with three domains?  Perhaps it makes sense to allow for more levels that
> match the physical parameters of the machine.  E.g. the NEC box has 4 cpus
> per-node, and 4 nodes in a "super-node", and 2 "super-nodes" in a machine.
> It would make sense to me if there was a scheduler domain level that would
> handle balancing between the nodes in a super-node in addition to the
> top-level domain to handle balancing between super-nodes.
>
> While the values in SLIT can be somewhat abstract, they could be used to
> derive the whole node, super-node, hyper-node, ultra-node,
> marketting-zeta-node structure to build as many levels as make sense into
> the scheduler.
>
> Or am I over-engineering?

Just guessing, but I think that might be overkill.  We'd have to collect data 
to know for sure though (which means someone has to implement it :).

Jesse

RE: [PATCH] top level scheduler domain for ia64

[Zou, Nanhai]

I have a patch in the latest mm tree which allow user to adjust 
scheduler domain parameters via sysctl.
However I have not include SPAN in the controllable parameters list.
Would including SPAN in sysctl parameter list helpful to this issue?

Zou Nan hai


-----Original Message-----
From: linux-ia64-owner@vger.kernel.org
[mailto:linux-ia64-owner@vger.kernel.org] On Behalf Of Jesse Barnes
Sent: Tuesday, November 02, 2004 3:02 AM
To: Luck, Tony
Cc: Matthew Wilcox; Takayoshi Kochi; linux-ia64@vger.kernel.org;
hawkes@sgi.com; nickpiggin@yahoo.com.au
Subject: Re: [PATCH] top level scheduler domain for ia64

On Monday, November 1, 2004 10:53 am, Luck, Tony wrote:
> >> >  if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
> >> >   SD_NODES_PER_DOMAIN = numnodes;
> >> >   build_node_domains(); /* each one spans the system */
> >> >  } else {
> >> >   SD_NODES_PER_DOMAIN = 4; /* or whatever */
>
> Ugh!  The magic '4' is going to be different for different machines,
> so there is no value that will make all of the people happy all of
> the time.

Right.

> >Yeah, but still the same number of CPUs/FSB.  I agree that
> >autotuning would be
> >best (the above is a crude example of that).  Any suggestions?
>
> Can't you use the SLIT table to build a hierachy of scheduler
> domains that matches the actual hardware configuration, instead
> of using compiled in constants or even boot-time command line
> parameters?

Maybe, but that will get complex very quickly I think.  Right now we
have 
three domains on ia64, the cpu domain, the node domain, which contains 
several nodes worth of CPUs, and the top level domain, which spans the
whole 
machine.

So there are two questions, how big should the node domain be and should
it 
span the whole machine (avoiding the need for a top level domain)?
Obviously 
the answer is pretty machine specific, and I'm not sure the SLIT helps
us 
much since its values are arbitrary distance values, not anything
concrete.

Jesse
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] top level scheduler domain for ia64

[Takayoshi Kochi]

Hi,

From: "Luck, Tony" <tony.luck@intel.com>
Subject: RE: [PATCH] top level scheduler domain for ia64
Date: Mon, 1 Nov 2004 11:45:44 -0800

> >Maybe, but that will get complex very quickly I think.  Right now we have 
> >three domains on ia64, the cpu domain, the node domain, which contains 
> >several nodes worth of CPUs, and the top level domain, which spans the whole 
> >machine.
> >
> >So there are two questions, how big should the node domain be and should it 
> >span the whole machine (avoiding the need for a top level domain)?  Obviously 
> >the answer is pretty machine specific, and I'm not sure the SLIT helps us 
> >much since its values are arbitrary distance values, not anything concrete.

Right, but if SLIT is implemented in real firmware (not a fake),
the numbers should have some meanings to describe how the machine
is configured.

For machines which have tree-like hierarchy NUMA, the SLIT table
can help enough to construct scheduling domains (like NEC's, and
I suppose Superdome is similar) and fits well for the sched-domain
scheduler.

The problem is that there can be other forms of topology.
It may be hard to derive its actual connection only with SLIT.
Quoting a SLIT example from the ACPI spec 3.0...

domain [0] [1] [2] [3]
   [0]  10  15  20  18
   [1]  15  10  16  24
   [2]  20  16  10  12
   [3]  18  24  12  10

This is an example which is not real, but it's possible
from the viewpoint of the ACPI spec;)
For this case, it's very hard to say it's worth making other 3
levels of scheduling domains above the cpu domain.

The userspace adjustment which Nanhai proposed does not work
for booting the 512-way Altix anyway (if we remove the node
domain from the generic kernel).

I think the short-term solution is to make the boot-time parameter
like Jesse said to control the creation and span size of the
node domain and see what is necessary next.

---
Takayoshi Kochi

Re: [PATCH] top level scheduler domain for ia64

[Gerald Pfeifer]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1050 bytes --]

On Mon, 1 Nov 2004, Matthew Wilcox wrote:
> Doesn't sound like a great idea.  HP's already shipping 128-way Superdome
> IA-64 systems, and they'll want to be set up rather differently from the
> Altix systems.  I think this code needs to be autotuning so it doesn't
> need to be touched whenever a vendor releases a new configuration (I
> think I heard that NASA's Altixes had a custom CPU brick with twice the
> CPUs in it?)

Yup.  Though that new brick is not specific to NASA's new Columbia system.

http://www.sgi.com/company_info/newsroom/press_releases/2004/october/altix_bx2.html

| SGI Doubles Density of High-End Altix Server
|
| New Altix 3700 Bx2 Ratchets Up Performance and Cost-Efficiency with 
| Twice the Bandwidth, Half the Footprint of Flagship System 
|
| MOUNTAIN VIEW, Calif., (October 28, 2004) [...] Silicon Graphics today 
| unveiled a new version of its SGI® Altix® 3700 system that delivers 
| twice the bandwidth and processor density of its flagship high-end 
| model.

Gerald (not working for SGI ;-)

Re: [PATCH] top level scheduler domain for ia64

[Takayoshi Kochi]

From: Takayoshi Kochi <t-kochi@bq.jp.nec.com>
Subject: Re: [PATCH] top level scheduler domain for ia64
Date: Tue, 02 Nov 2004 16:36:36 +0900 (JST)

> I think the short-term solution is to make the boot-time parameter
> like Jesse said to control the creation and span size of the
> node domain and see what is necessary next.

This patch adds a new kernel parameter "nodes_per_domain"
which specifies how many nodes are included in a node 
scheduling domain.

Signed-off-by: Takayoshi Kochi <t-kochi@bq.jp.nec.com>

Index: 269-bk/arch/ia64/kernel/domain.c
=================================--- 269-bk/arch/ia64/kernel/domain.c	(revision 49)
+++ 269-bk/arch/ia64/kernel/domain.c	(working copy)
@@ -12,9 +12,9 @@
 #include <linux/init.h>
 #include <linux/topology.h>
 
-#define SD_NODES_PER_DOMAIN 6
+#ifdef CONFIG_NUMA
+static int sd_nodes_per_domain = 6;
 
-#ifdef CONFIG_NUMA
 /**
  * find_next_best_node - find the next node to include in a sched_domain
  * @node: node whose sched_domain we're building
@@ -77,7 +77,7 @@
 	cpus_or(span, span, nodemask);
 	set_bit(node, used_nodes);
 
-	for (i = 1; i < SD_NODES_PER_DOMAIN; i++) {
+	for (i = 1; i < sd_nodes_per_domain; i++) {
 		int next_node = find_next_best_node(node, used_nodes);
 		nodemask = node_to_cpumask(next_node);
 		cpus_or(span, span, nodemask);
@@ -158,7 +158,7 @@
 
 #ifdef CONFIG_NUMA
 		if (num_online_cpus()
-				> SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
+				> sd_nodes_per_domain*cpus_weight(nodemask)) {
 			sd = &per_cpu(allnodes_domains, i);
 			*sd = SD_ALLNODES_INIT;
 			sd->span = cpu_default_map;
@@ -379,3 +379,18 @@
 #endif
 }
 
+#ifdef CONFIG_NUMA
+static int __init
+set_nodes_per_domain (char *str)
+{
+	int tmp;
+
+	get_option(&str, &tmp);
+	if (tmp > 1 && tmp <= MAX_NUMNODES)
+		sd_nodes_per_domain = tmp;
+
+	return 1;
+}
+
+__setup("nodes_per_domain=", set_nodes_per_domain);
+#endif


---
Takayoshi Kochi

RE: [PATCH] top level scheduler domain for ia64

[Luck, Tony]

>> I think the short-term solution is to make the boot-time parameter
>> like Jesse said to control the creation and span size of the
>> node domain and see what is necessary next.
>
>This patch adds a new kernel parameter "nodes_per_domain"
>which specifies how many nodes are included in a node 
>scheduling domain.

Can you add a short write-up for Documentation/kernel-parameters.txt
to explain the effect, and how a user should pick what value to use
(examples would be good ... if SGI want to throw in some
values for small/large Altix boxes, that would be good too).

Is '6' a reasonable default if the parameter is not set?

-Tony

Re: [PATCH] top level scheduler domain for ia64

[Takayoshi Kochi]

Hi,

From: "Luck, Tony" <tony.luck@intel.com>
Subject: RE: [PATCH] top level scheduler domain for ia64
Date: Tue, 2 Nov 2004 13:31:16 -0800

> >> I think the short-term solution is to make the boot-time parameter
> >> like Jesse said to control the creation and span size of the
> >> node domain and see what is necessary next.
> >
> >This patch adds a new kernel parameter "nodes_per_domain"
> >which specifies how many nodes are included in a node 
> >scheduling domain.
> 
> Can you add a short write-up for Documentation/kernel-parameters.txt
> to explain the effect, and how a user should pick what value to use
> (examples would be good ... if SGI want to throw in some
> values for small/large Altix boxes, that would be good too).

The diff is attached.
Though I'm not sure how to pick the best number...
Any ideas from SGI?

> Is '6' a reasonable default if the parameter is not set?

Maybe for Altix.  I guess the only user of this domain split
is Altix, so it's ok ;)


The patch below is a documentation for 'nodes_per_domain=' option.

Signed-off-by: Takayoshi Kochi <t-kochi@bq.jp.nec.com>

=== my-bk/Documentation/kernel-parameters.txt 1.57 vs edited ==--- 1.57/Documentation/kernel-parameters.txt	2004-10-30 18:31:21 +09:00
+++ edited/my-bk/Documentation/kernel-parameters.txt	2004-11-03 15:07:20 +09:00
@@ -475,6 +475,15 @@
 
 	hugepages=	[HW,IA-32,IA-64] Maximal number of HugeTLB pages.
 
+	nodes_per_domain=n
+			[IA-64] specifies number of nodes in a scheduling
+			domain.  If the number of nodes in a sysmtem is
+			large enough, the node scheduling domain (usually
+			spans the whole system) is split into <n> nodes
+			for each.  Default <n> is 6.
+			If you have a large NUMA machine, consider using
+			this option to reduce load balancing overhead.
+
 	noirqbalance	[IA-32,SMP,KNL] Disable kernel irq balancing
 
 	i8042.direct	[HW] Put keyboard port into non-translated mode


---
Takayoshi Kochi

Re: [PATCH] top level scheduler domain for ia64

[Jesse Barnes]

On Tuesday, November 2, 2004 10:15 pm, Takayoshi Kochi wrote:
> Hi,
>
> From: "Luck, Tony" <tony.luck@intel.com>
> Subject: RE: [PATCH] top level scheduler domain for ia64
> Date: Tue, 2 Nov 2004 13:31:16 -0800
>
> > >> I think the short-term solution is to make the boot-time parameter
> > >> like Jesse said to control the creation and span size of the
> > >> node domain and see what is necessary next.
> > >
> > >This patch adds a new kernel parameter "nodes_per_domain"
> > >which specifies how many nodes are included in a node
> > >scheduling domain.
> >
> > Can you add a short write-up for Documentation/kernel-parameters.txt
> > to explain the effect, and how a user should pick what value to use
> > (examples would be good ... if SGI want to throw in some
> > values for small/large Altix boxes, that would be good too).
>
> The diff is attached.
> Though I'm not sure how to pick the best number...
> Any ideas from SGI?
>
> > Is '6' a reasonable default if the parameter is not set?
>
> Maybe for Altix.  I guess the only user of this domain split
> is Altix, so it's ok ;)

Yep, looks good to me.  Thanks.

Jesse

RE: [PATCH] top level scheduler domain for ia64

[Luck, Tony]

>> > Is '6' a reasonable default if the parameter is not set?
>>
>> Maybe for Altix.  I guess the only user of this domain split
>> is Altix, so it's ok ;)
>
>Yep, looks good to me.  Thanks.

So tell me who wants a value other than 6, and what is the value
that they want?  I was hoping for some more specific advice in 
the kernel-parameters.txt file.  What's a good choice for a 16-node
machine?  For a 128-node machine?

-Tony

Re: [PATCH] top level scheduler domain for ia64

[Jesse Barnes]

On Wednesday, November 3, 2004 8:57 am, Luck, Tony wrote:
> >> > Is '6' a reasonable default if the parameter is not set?
> >>
> >> Maybe for Altix.  I guess the only user of this domain split
> >> is Altix, so it's ok ;)
> >
> >Yep, looks good to me.  Thanks.
>
> So tell me who wants a value other than 6, and what is the value
> that they want?  I was hoping for some more specific advice in
> the kernel-parameters.txt file.  What's a good choice for a 16-node
> machine?  For a 128-node machine?

I don't know.  The original value I chose was 4, but I think Nick changed it 
to 6 to get more overlap.  Any ideas John?

Jesse

Re: [PATCH] top level scheduler domain for ia64

[John Hawkes]

From: "Jesse Barnes" <jbarnes@engr.sgi.com>
> > So tell me who wants a value other than 6, and what is the value
> > that they want?  I was hoping for some more specific advice in
> > the kernel-parameters.txt file.  What's a good choice for a 16-node
> > machine?  For a 128-node machine?
>
> I don't know.  The original value I chose was 4, but I think Nick changed it
> to 6 to get more overlap.  Any ideas John?

I haven't been able to get enough reliably reproducible performance numbers
using the recent -mm kernels to allow me to experiment with different
nodes-per-domain values.  Using 6 and having an occasionally load-balancing
umbrella covering all the CPUs in the system seems to behave reasonably for
64p and above.  I just haven't tried other values, other than once trying 32
and without that all-CPU umbrella, and that performance was bad.

John Hawkes