Performance regression in 2.6.22-git1 (new sched code?)

Performance regression in 2.6.22-git1 (new sched code?)

[Martin Bligh]

~ 1% on 4-way x86_64

http://test.kernel.org/perf/kernbench.elm3b6.png

~ 4% on 16-way NUMA-Q (i386)

http://test.kernel.org/perf/kernbench.moe.png

~ 1.5% on 4-way i386

http://test.kernel.org/perf/kernbench.elm3b132.png

There's readprofiles and stuff available from here:
http://test.kernel.org/functional/index.html
if you drill down under the tests marked "regression", eg.
http://test.kernel.org/abat/98796/002.kernbench.test/profiling/profile.text

M.

Re: Performance regression in 2.6.22-git1 (new sched code?)

[Andy Whitcroft]

Martin Bligh wrote:
> ~ 1% on 4-way x86_64
> 
> http://test.kernel.org/perf/kernbench.elm3b6.png
> 
> ~ 4% on 16-way NUMA-Q (i386)
> 
> http://test.kernel.org/perf/kernbench.moe.png
> 
> ~ 1.5% on 4-way i386
> 
> http://test.kernel.org/perf/kernbench.elm3b132.png
> 
> There's readprofiles and stuff available from here:
> http://test.kernel.org/functional/index.html
> if you drill down under the tests marked "regression", eg.
> http://test.kernel.org/abat/98796/002.kernbench.test/profiling/profile.text

Ok, I've done a crude comparison of a bunch of the jobs we run between
2.6.22 and 2.6.22-git1 kernbench runs, extracting the wallclock, user
and system times for each.  First column is 2.6.22, second is
2.6.22-git1, third is the delta.  +ve bad, -ve good.  Note that I have
hand correlated these jobs so error is possible, but it matches Martin's
figures I belive.

bl6-13
	User    85.14  88.53   3.99
	System  34.16  34.71   1.60
	Total  119.30 123.24   3.30
elm3b132
	User   2397.82 2431.95   1.42
	System 207.41 206.78  -0.30
	Total  2605.23 2638.73   1.29
elm3b6
	User   352.18 357.18   1.42
	System  37.06  36.61  -1.22
	Total  389.24 393.79   1.17
oia5-hs20
	User   645.02 653.47   1.31
	System  49.30  52.45   6.37
	Total  694.33 705.92   1.67
elm3b239
	User   878.21 950.80   8.27
	System 416.06 348.76 -16.18
	Total  1294.27 1299.56   0.41

-apw

Re: Performance regression in 2.6.22-git1 (new sched code?)

[Ingo Molnar]

* Martin Bligh <mbligh@mbligh.org> wrote:

> ~ 1% on 4-way x86_64
> 
> http://test.kernel.org/perf/kernbench.elm3b6.png
> 
> ~ 4% on 16-way NUMA-Q (i386)
> 
> http://test.kernel.org/perf/kernbench.moe.png
> 
> ~ 1.5% on 4-way i386
> 
> http://test.kernel.org/perf/kernbench.elm3b132.png

thx! I'll check this tomorrow, meanwhile here are a few quick ideas.

1) does reverting this patch improve performance:

   commit 9c4801cebc2add1fe514bc8eb201b16372eee11a
   Author: Ingo Molnar <mingo@elte.hu>
   Date:   Mon Jul 9 18:52:01 2007 +0200

       sched: more agressive idle balancing

2) does changing CONFIG_HZ=250 to CONFIG_HZ=100 improve the numbers? 
   CFS has some internal tuning that depends on HZ - a higher HZ is a 
   sign that the user wants more finegrained scheduling. So for maximum 
   server throughput, use CONFIG_HZ=100.

3) could you turn off CONFIG_SCHED_DEBUG? The cost is small but perhaps 
   measurable.

(these 3 suggestions could be tested together - they should have 
cumulative effects.)

	Ingo

Re: Performance regression in 2.6.22-git1 (new sched code?)

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > http://test.kernel.org/perf/kernbench.elm3b132.png
> 
> thx! I'll check this tomorrow, meanwhile here are a few quick ideas.

ok, i've re-tested this, and could you please try the fix attached 
below, combined with a CONFIG_HZ=100 run and with SCHED_DEBUG disabled? 
I think a fair bit of the overhead comes from granularity being capped 
to 10 msecs instead of the intended 100 msecs.

	Ingo

--------------------------------------------->
Subject: [patch] sched: allow larger granularity
From: Ingo Molnar <mingo@elte.hu>

allow granularity up to 100 msecs, instead of 10 msecs.
(needed on larger boxes)

Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 kernel/sched.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: linux/kernel/sched.c
===================================================================
--- linux.orig/kernel/sched.c
+++ linux/kernel/sched.c
@@ -4772,7 +4772,7 @@ cpumask_t nohz_cpu_mask = CPU_MASK_NONE;
 static inline void sched_init_granularity(void)
 {
 	unsigned int factor = 1 + ilog2(num_online_cpus());
-	const unsigned long gran_limit = 10000000;
+	const unsigned long gran_limit = 100000000;
 
 	sysctl_sched_granularity *= factor;
 	if (sysctl_sched_granularity > gran_limit)