* [PATCH] random ia64 sched-domains values
@ 2005-02-25 8:14 Nick Piggin
2005-02-25 18:46 ` Chen, Kenneth W
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Nick Piggin @ 2005-02-25 8:14 UTC (permalink / raw)
To: linux-ia64
[-- Attachment #1: Type: text/plain, Size: 1041 bytes --]
Hi,
These are some values I just found from thin-air that should
hopefully make ia64 work after the latest sched patches.
Though it did look like there may have been some problems with
the old values, which I tried to fix up too:
- SD_ALLNODES_INIT had .balance_interval set a specific way -
this field is internal to the scheduler and will end up
sitting between min_interval and max_interval and not do what
was intended, so tweak max_interval instead.
- SD_ALLNODES_INIT had SD_BALANCE_EXEC set. The scheduler
doesn't cope _really_ well with doing balance-on-exec over the
top level of a multi-level NUMA setup like this. It will tend
to fill up the first CPUs in each high level group (cluster of
6 nodes) rather than spreading the load on a per-node basis.
This could be improved if we really need it, but for now remove
the flag from the top level.
- Increase SD_NODES_PER_DOMAIN in the low level NUMA domain so we
still push tasks out a reasonable distance. Set SD_BALANCE_FORK
in this domain too.
Not compiled or tested, sorry.
[-- Attachment #2: sched-tune-ia64.patch --]
[-- Type: text/plain, Size: 3409 bytes --]
---
linux-2.6-npiggin/arch/ia64/kernel/domain.c | 2
linux-2.6-npiggin/include/asm-ia64/topology.h | 62 ++++++++++++++++++++------
2 files changed, 49 insertions(+), 15 deletions(-)
diff -puN include/asm-ia64/topology.h~sched-tune-ia64 include/asm-ia64/topology.h
--- linux-2.6/include/asm-ia64/topology.h~sched-tune-ia64 2005-02-25 18:18:43.000000000 +1100
+++ linux-2.6-npiggin/include/asm-ia64/topology.h 2005-02-25 18:56:50.000000000 +1100
@@ -42,25 +42,55 @@
void build_cpu_to_node_map(void);
+#define SD_CPU_INIT (struct sched_domain) { \
+ .span = CPU_MASK_NONE, \
+ .parent = NULL, \
+ .groups = NULL, \
+ .min_interval = 1, \
+ .max_interval = 4, \
+ .busy_factor = 64, \
+ .imbalance_pct = 125, \
+ .cache_hot_time = (10*1000000), \
+ .cache_nice_tries = 2, \
+ .per_cpu_gain = 100, \
+ .cache_nice_tries = 2, \
+ .busy_idx = 2, \
+ .idle_idx = 1, \
+ .newidle_idx = 2, \
+ .wake_idx = 1, \
+ .forkexec_idx = 1, \
+ .flags = SD_LOAD_BALANCE \
+ | SD_BALANCE_NEWIDLE \
+ | SD_BALANCE_EXEC \
+ | SD_WAKE_AFFINE, \
+ .last_balance = jiffies, \
+ .balance_interval = 1, \
+ .nr_balance_failed = 0, \
+}
+
/* sched_domains SD_NODE_INIT for IA64 NUMA machines */
#define SD_NODE_INIT (struct sched_domain) { \
.span = CPU_MASK_NONE, \
.parent = NULL, \
.groups = NULL, \
- .min_interval = 80, \
- .max_interval = 320, \
- .busy_factor = 320, \
+ .min_interval = 8, \
+ .max_interval = 8*(min(num_online_cpus(), 32)), \
+ .busy_factor = 64, \
.imbalance_pct = 125, \
.cache_hot_time = (10*1000000), \
- .cache_nice_tries = 1, \
+ .cache_nice_tries = 2, \
+ .busy_idx = 3, \
+ .idle_idx = 2, \
+ .newidle_idx = 0, /* unused */ \
+ .wake_idx = 1, \
+ .forkexec_idx = 1, \
.per_cpu_gain = 100, \
.flags = SD_LOAD_BALANCE \
| SD_BALANCE_EXEC \
- | SD_BALANCE_NEWIDLE \
- | SD_WAKE_IDLE \
+ | SD_BALANCE_FORK \
| SD_WAKE_BALANCE, \
.last_balance = jiffies, \
- .balance_interval = 1, \
+ .balance_interval = 64, \
.nr_balance_failed = 0, \
}
@@ -69,17 +99,21 @@ void build_cpu_to_node_map(void);
.span = CPU_MASK_NONE, \
.parent = NULL, \
.groups = NULL, \
- .min_interval = 80, \
- .max_interval = 320, \
- .busy_factor = 320, \
- .imbalance_pct = 125, \
+ .min_interval = 64, \
+ .max_interval = 64*num_online_cpus(), \
+ .busy_factor = 128, \
+ .imbalance_pct = 133, \
.cache_hot_time = (10*1000000), \
.cache_nice_tries = 1, \
+ .busy_idx = 3, \
+ .idle_idx = 3, \
+ .newidle_idx = 0, /* unused */ \
+ .wake_idx = 0, /* unused */ \
+ .forkexec_idx = 0, /* unused */ \
.per_cpu_gain = 100, \
- .flags = SD_LOAD_BALANCE \
- | SD_BALANCE_EXEC, \
+ .flags = SD_LOAD_BALANCE, \
.last_balance = jiffies, \
- .balance_interval = 100*(63+num_online_cpus())/64, \
+ .balance_interval = 64, \
.nr_balance_failed = 0, \
}
diff -puN arch/ia64/kernel/domain.c~sched-tune-ia64 arch/ia64/kernel/domain.c
--- linux-2.6/arch/ia64/kernel/domain.c~sched-tune-ia64 2005-02-25 18:18:48.000000000 +1100
+++ linux-2.6-npiggin/arch/ia64/kernel/domain.c 2005-02-25 18:49:43.000000000 +1100
@@ -14,7 +14,7 @@
#include <linux/topology.h>
#include <linux/nodemask.h>
-#define SD_NODES_PER_DOMAIN 6
+#define SD_NODES_PER_DOMAIN 16
#ifdef CONFIG_NUMA
/**
_
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH] random ia64 sched-domains values
2005-02-25 8:14 [PATCH] random ia64 sched-domains values Nick Piggin
@ 2005-02-25 18:46 ` Chen, Kenneth W
2005-02-25 23:41 ` Nick Piggin
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Chen, Kenneth W @ 2005-02-25 18:46 UTC (permalink / raw)
To: linux-ia64
Nick Piggin wrote on Friday, February 25, 2005 12:15 AM
> These are some values I just found from thin-air that should
> hopefully make ia64 work after the latest sched patches.
>
> + .cache_hot_time = (10*1000000), \
In 2.6.9, cache_hot_time gets the value from cache_decay_tick, since
when this become a constant again?
- Ken
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] random ia64 sched-domains values
2005-02-25 8:14 [PATCH] random ia64 sched-domains values Nick Piggin
2005-02-25 18:46 ` Chen, Kenneth W
@ 2005-02-25 23:41 ` Nick Piggin
2005-02-26 0:25 ` Chen, Kenneth W
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Nick Piggin @ 2005-02-25 23:41 UTC (permalink / raw)
To: linux-ia64
Chen, Kenneth W wrote:
> Nick Piggin wrote on Friday, February 25, 2005 12:15 AM
>
>>These are some values I just found from thin-air that should
>>hopefully make ia64 work after the latest sched patches.
>>
>>+ .cache_hot_time = (10*1000000), \
>
>
> In 2.6.9, cache_hot_time gets the value from cache_decay_tick, since
> when this become a constant again?
>
Not sure. cache_decay_ticks isn't set to anything meaningful on
ia64 anyway. Nor i386 for that matter.
Nick
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH] random ia64 sched-domains values
2005-02-25 8:14 [PATCH] random ia64 sched-domains values Nick Piggin
2005-02-25 18:46 ` Chen, Kenneth W
2005-02-25 23:41 ` Nick Piggin
@ 2005-02-26 0:25 ` Chen, Kenneth W
2005-02-26 0:40 ` Chen, Kenneth W
2005-02-26 0:48 ` Nick Piggin
4 siblings, 0 replies; 6+ messages in thread
From: Chen, Kenneth W @ 2005-02-26 0:25 UTC (permalink / raw)
To: linux-ia64
Nick Piggin wrote on Friday, February 25, 2005 3:42 PM
> >>These are some values I just found from thin-air that should
> >>hopefully make ia64 work after the latest sched patches.
> >>
> >>+ .cache_hot_time = (10*1000000), \
> >
> >
> > In 2.6.9, cache_hot_time gets the value from cache_decay_tick, since
> > when this become a constant again?
> >
>
> Not sure. cache_decay_ticks isn't set to anything meaningful on
> ia64 anyway. Nor i386 for that matter.
This parameter is known to be sensitive for one db workload. It was
discussed about 4 month back. The conclusion was that there is NO one
magic constant that works for every workloads on every platforms. Until
a new auto tune algorithm is written/tested, it was decided that we use
cache_decay_ticks as a boot time tunable for cache_hot_time.
I guess it somehow got lost soon after 2.6.9. This needs to be reinstated.
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
--- linux-2.6.10/include/linux/topology.h Fri Jan 7 21:44:04 2005
+++ linux-2.6.10/include/linux/topology.h Fri Feb 25 16:02:44 2005
@@ -109,7 +109,7 @@
.max_interval = 4, \
.busy_factor = 64, \
.imbalance_pct = 125, \
- .cache_hot_time = (5*1000000/2), \
+ .cache_hot_time = cache_decay_ticks*1000000 ? : (5*1000000/2),\
.cache_nice_tries = 1, \
.per_cpu_gain = 100, \
.flags = SD_LOAD_BALANCE \
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH] random ia64 sched-domains values
2005-02-25 8:14 [PATCH] random ia64 sched-domains values Nick Piggin
` (2 preceding siblings ...)
2005-02-26 0:25 ` Chen, Kenneth W
@ 2005-02-26 0:40 ` Chen, Kenneth W
2005-02-26 0:48 ` Nick Piggin
4 siblings, 0 replies; 6+ messages in thread
From: Chen, Kenneth W @ 2005-02-26 0:40 UTC (permalink / raw)
To: linux-ia64
Nick Piggin wrote on Friday, February 25, 2005 12:15 AM
> These are some values I just found from thin-air that should
> hopefully make ia64 work after the latest sched patches.
>
> +#define SD_CPU_INIT (struct sched_domain) { \
> ...
> + .max_interval = 4, \
> + .busy_factor = 64, \
> + .imbalance_pct = 125, \
For one db workload, performance increases with the following
values compare to the values proposed above.
.max_interval = 100
.busy_factor = 32
.imbalance_pct = 100
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] random ia64 sched-domains values
2005-02-25 8:14 [PATCH] random ia64 sched-domains values Nick Piggin
` (3 preceding siblings ...)
2005-02-26 0:40 ` Chen, Kenneth W
@ 2005-02-26 0:48 ` Nick Piggin
4 siblings, 0 replies; 6+ messages in thread
From: Nick Piggin @ 2005-02-26 0:48 UTC (permalink / raw)
To: linux-ia64
Chen, Kenneth W wrote:
> Nick Piggin wrote on Friday, February 25, 2005 12:15 AM
>
>>These are some values I just found from thin-air that should
>>hopefully make ia64 work after the latest sched patches.
>>
>>+#define SD_CPU_INIT (struct sched_domain) { \
>>...
>>+ .max_interval = 4, \
>>+ .busy_factor = 64, \
>>+ .imbalance_pct = 125, \
>
>
> For one db workload, performance increases with the following
> values compare to the values proposed above.
>
Sorry if I wasn't clear earlier - this patch is not meant to
go into 2.6. It is for Andrew to get ia64 working after my
last round of scheduler patches.
There are pretty significant changes, so the optimal values
for various things will have changed quite a bit.
> .max_interval = 100
> .busy_factor = 32
> .imbalance_pct = 100
>
I can imagine max_interval could probably be increased.
Setting imbalance_pct to 100 is interesting, I wonder why this
helps performance... It would be good to get schedstats output.
Nick
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2005-02-26 0:48 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-02-25 8:14 [PATCH] random ia64 sched-domains values Nick Piggin
2005-02-25 18:46 ` Chen, Kenneth W
2005-02-25 23:41 ` Nick Piggin
2005-02-26 0:25 ` Chen, Kenneth W
2005-02-26 0:40 ` Chen, Kenneth W
2005-02-26 0:48 ` Nick Piggin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox