* [patch] Over schedule issue fixing
@ 2010-06-17 6:08 Alex,Shi
2010-06-18 4:25 ` Alex,Shi
2010-06-18 10:18 ` [tip:sched/urgent] sched: Fix over-scheduling bug tip-bot for Alex,Shi
0 siblings, 2 replies; 4+ messages in thread
From: Alex,Shi @ 2010-06-17 6:08 UTC (permalink / raw)
To: linux-kernel; +Cc: yanmin.zhang, tim.c.chen
commit e709715915d69b6a929d77e7652c9c3fea61c317 introduced an imbalance
schedule issue. If we do not use CGROUP, function update_h_load won't
want to update h_load. When the system has a large number of tasks far
more than logical CPU number, the incorrect cfs_rq[cpu]->h_load value
will cause load_balance() to pull too many tasks to local CPU from the
busiest CPU. So the busiest CPU keeps being in a round robin. That will
hurt performance.
The issue was found originally by a scientific calculation workload that
developed by Yanmin. with the commit, the workload performance drops
about 40% from this commit. We can be reproduced by a short program as
following.
# gcc -o sl sched-loop.c -lpthread
# ./sl -n 100 -t 100 &
# cat /proc/sched_debug &> sd1
# grep -A 1 cpu# sd1
sd1:cpu#0, 2533.008 MHz
sd1- .nr_running : 2
--
sd1:cpu#1, 2533.008 MHz
sd1- .nr_running : 1
--
sd1:cpu#2, 2533.008 MHz
sd1- .nr_running : 11
--
sd1:cpu#3, 2533.008 MHz
sd1- .nr_running : 12
--
sd1:cpu#4, 2533.008 MHz
sd1- .nr_running : 6
--
sd1:cpu#5, 2533.008 MHz
sd1- .nr_running : 11
--
sd1:cpu#6, 2533.008 MHz
sd1- .nr_running : 10
--
sd1:cpu#7, 2533.008 MHz
sd1- .nr_running : 12
--
sd1:cpu#8, 2533.008 MHz
sd1- .nr_running : 11
--
sd1:cpu#9, 2533.008 MHz
sd1- .nr_running : 12
--
sd1:cpu#10, 2533.008 MHz
sd1- .nr_running : 1
--
sd1:cpu#11, 2533.008 MHz
sd1- .nr_running : 1
--
sd1:cpu#12, 2533.008 MHz
sd1- .nr_running : 6
--
sd1:cpu#13, 2533.008 MHz
sd1- .nr_running : 2
--
sd1:cpu#14, 2533.008 MHz
sd1- .nr_running : 2
--
sd1:cpu#15, 2533.008 MHz
sd1- .nr_running : 1
After apply the fixing patch, cfs_rq get balance.
sd1:cpu#0, 2533.479 MHz
sd1- .nr_running : 7
--
sd1:cpu#1, 2533.479 MHz
sd1- .nr_running : 7
--
sd1:cpu#2, 2533.479 MHz
sd1- .nr_running : 6
--
sd1:cpu#3, 2533.479 MHz
sd1- .nr_running : 7
--
sd1:cpu#4, 2533.479 MHz
sd1- .nr_running : 6
--
sd1:cpu#5, 2533.479 MHz
sd1- .nr_running : 7
--
sd1:cpu#6, 2533.479 MHz
sd1- .nr_running : 6
--
sd1:cpu#7, 2533.479 MHz
sd1- .nr_running : 7
--
sd1:cpu#8, 2533.479 MHz
sd1- .nr_running : 6
--
sd1:cpu#9, 2533.479 MHz
sd1- .nr_running : 6
--
sd1:cpu#10, 2533.479 MHz
sd1- .nr_running : 6
--
sd1:cpu#11, 2533.479 MHz
sd1- .nr_running : 6
--
sd1:cpu#12, 2533.479 MHz
sd1- .nr_running : 6
--
sd1:cpu#13, 2533.479 MHz
sd1- .nr_running : 6
--
sd1:cpu#14, 2533.479 MHz
sd1- .nr_running : 6
--
sd1:cpu#15, 2533.479 MHz
sd1- .nr_running : 6
---
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
volatile int * exiting;
void *idle_loop(){
volatile int calc01 = 100;
while(*exiting !=1)
calc01++;
}
int main(int argc, char *argv[]){
int i, t, c, er=0, num=8;
static char optstr[] = "n:t:";
pthread_t ptid[1024];
while ((c = getopt(argc, argv, optstr)) != EOF)
switch (c) {
case 'n':
num = atoi(optarg);
break;
case 't':
t = atoi(optarg);
break;
case '?':
er = 1;
break;
}
if (er) {
printf("usage: %s %s\n", argv[0], optstr);
exit(1);
}
exiting = malloc(sizeof(int));
*exiting = 0;
for(i=0; i<num ; i++)
pthread_create(&ptid[i], NULL, idle_loop, NULL);
sleep(t);
*exiting = 1;
for (i=0; i<num; i++)
pthread_join(ptid[i], NULL);
exit(0);
}
Reviewed-by: Yanmin zhang <yanmin.zhang@intel.com>
Signed-off-by: Alex Shi <alex.shi@intel.com>
diff --git a/kernel/sched.c b/kernel/sched.c
index f8b8996..a18bf93 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -1660,9 +1660,6 @@ static void update_shares(struct sched_domain *sd)
static void update_h_load(long cpu)
{
- if (root_task_group_empty())
- return;
-
walk_tg_tree(tg_load_down, tg_nop, (void *)cpu);
}
^ permalink raw reply related [flat|nested] 4+ messages in thread* Re: [patch] Over schedule issue fixing
2010-06-17 6:08 [patch] Over schedule issue fixing Alex,Shi
@ 2010-06-18 4:25 ` Alex,Shi
2010-06-18 7:16 ` Peter Zijlstra
2010-06-18 10:18 ` [tip:sched/urgent] sched: Fix over-scheduling bug tip-bot for Alex,Shi
1 sibling, 1 reply; 4+ messages in thread
From: Alex,Shi @ 2010-06-18 4:25 UTC (permalink / raw)
To: linux-kernel, suresh.b.siddha, a.p.zijlstra; +Cc: yanmin.zhang, tim.c.chen
Add Suresh and Peter into thread.
Would you like to give some comments of this issue?
Thanks!
Alex
On Thu, 2010-06-17 at 14:08 +0800, Alex,Shi wrote:
> commit e709715915d69b6a929d77e7652c9c3fea61c317 introduced an imbalance
> schedule issue. If we do not use CGROUP, function update_h_load won't
> want to update h_load. When the system has a large number of tasks far
> more than logical CPU number, the incorrect cfs_rq[cpu]->h_load value
> will cause load_balance() to pull too many tasks to local CPU from the
> busiest CPU. So the busiest CPU keeps being in a round robin. That will
> hurt performance.
> The issue was found originally by a scientific calculation workload that
> developed by Yanmin. with the commit, the workload performance drops
> about 40% from this commit. We can be reproduced by a short program as
> following.
>
> # gcc -o sl sched-loop.c -lpthread
> # ./sl -n 100 -t 100 &
> # cat /proc/sched_debug &> sd1
> # grep -A 1 cpu# sd1
> sd1:cpu#0, 2533.008 MHz
> sd1- .nr_running : 2
> --
> sd1:cpu#1, 2533.008 MHz
> sd1- .nr_running : 1
> --
> sd1:cpu#2, 2533.008 MHz
> sd1- .nr_running : 11
> --
> sd1:cpu#3, 2533.008 MHz
> sd1- .nr_running : 12
> --
> sd1:cpu#4, 2533.008 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#5, 2533.008 MHz
> sd1- .nr_running : 11
> --
> sd1:cpu#6, 2533.008 MHz
> sd1- .nr_running : 10
> --
> sd1:cpu#7, 2533.008 MHz
> sd1- .nr_running : 12
> --
> sd1:cpu#8, 2533.008 MHz
> sd1- .nr_running : 11
> --
> sd1:cpu#9, 2533.008 MHz
> sd1- .nr_running : 12
> --
> sd1:cpu#10, 2533.008 MHz
> sd1- .nr_running : 1
> --
> sd1:cpu#11, 2533.008 MHz
> sd1- .nr_running : 1
> --
> sd1:cpu#12, 2533.008 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#13, 2533.008 MHz
> sd1- .nr_running : 2
> --
> sd1:cpu#14, 2533.008 MHz
> sd1- .nr_running : 2
> --
> sd1:cpu#15, 2533.008 MHz
> sd1- .nr_running : 1
>
> After apply the fixing patch, cfs_rq get balance.
>
> sd1:cpu#0, 2533.479 MHz
> sd1- .nr_running : 7
> --
> sd1:cpu#1, 2533.479 MHz
> sd1- .nr_running : 7
> --
> sd1:cpu#2, 2533.479 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#3, 2533.479 MHz
> sd1- .nr_running : 7
> --
> sd1:cpu#4, 2533.479 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#5, 2533.479 MHz
> sd1- .nr_running : 7
> --
> sd1:cpu#6, 2533.479 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#7, 2533.479 MHz
> sd1- .nr_running : 7
> --
> sd1:cpu#8, 2533.479 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#9, 2533.479 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#10, 2533.479 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#11, 2533.479 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#12, 2533.479 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#13, 2533.479 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#14, 2533.479 MHz
> sd1- .nr_running : 6
> --
> sd1:cpu#15, 2533.479 MHz
> sd1- .nr_running : 6
>
> ---
> #include <stdlib.h>
> #include <stdio.h>
> #include <unistd.h>
> #include <pthread.h>
>
> volatile int * exiting;
>
> void *idle_loop(){
> volatile int calc01 = 100;
> while(*exiting !=1)
> calc01++;
> }
> int main(int argc, char *argv[]){
> int i, t, c, er=0, num=8;
> static char optstr[] = "n:t:";
> pthread_t ptid[1024];
>
> while ((c = getopt(argc, argv, optstr)) != EOF)
> switch (c) {
> case 'n':
> num = atoi(optarg);
> break;
> case 't':
> t = atoi(optarg);
> break;
> case '?':
> er = 1;
> break;
> }
>
> if (er) {
> printf("usage: %s %s\n", argv[0], optstr);
> exit(1);
> }
> exiting = malloc(sizeof(int));
>
> *exiting = 0;
> for(i=0; i<num ; i++)
> pthread_create(&ptid[i], NULL, idle_loop, NULL);
>
> sleep(t);
> *exiting = 1;
>
> for (i=0; i<num; i++)
> pthread_join(ptid[i], NULL);
> exit(0);
>
> }
>
> Reviewed-by: Yanmin zhang <yanmin.zhang@intel.com>
> Signed-off-by: Alex Shi <alex.shi@intel.com>
>
> diff --git a/kernel/sched.c b/kernel/sched.c
> index f8b8996..a18bf93 100644
> --- a/kernel/sched.c
> +++ b/kernel/sched.c
> @@ -1660,9 +1660,6 @@ static void update_shares(struct sched_domain *sd)
>
> static void update_h_load(long cpu)
> {
> - if (root_task_group_empty())
> - return;
> -
> walk_tg_tree(tg_load_down, tg_nop, (void *)cpu);
> }
>
>
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [patch] Over schedule issue fixing
2010-06-18 4:25 ` Alex,Shi
@ 2010-06-18 7:16 ` Peter Zijlstra
0 siblings, 0 replies; 4+ messages in thread
From: Peter Zijlstra @ 2010-06-18 7:16 UTC (permalink / raw)
To: Alex,Shi; +Cc: linux-kernel, suresh.b.siddha, yanmin.zhang, tim.c.chen
On Fri, 2010-06-18 at 12:25 +0800, Alex,Shi wrote:
> Add Suresh and Peter into thread.
> Would you like to give some comments of this issue?
I took it, I looked at curing the issue differently but they all ended
up being more work.
Thanks
^ permalink raw reply [flat|nested] 4+ messages in thread
* [tip:sched/urgent] sched: Fix over-scheduling bug
2010-06-17 6:08 [patch] Over schedule issue fixing Alex,Shi
2010-06-18 4:25 ` Alex,Shi
@ 2010-06-18 10:18 ` tip-bot for Alex,Shi
1 sibling, 0 replies; 4+ messages in thread
From: tip-bot for Alex,Shi @ 2010-06-18 10:18 UTC (permalink / raw)
To: linux-tip-commits
Cc: linux-kernel, hpa, mingo, a.p.zijlstra, yanmin.zhang, alex.shi,
tglx, mingo
Commit-ID: 3c93717cfa51316e4dbb471e7c0f9d243359d5f8
Gitweb: http://git.kernel.org/tip/3c93717cfa51316e4dbb471e7c0f9d243359d5f8
Author: Alex,Shi <alex.shi@intel.com>
AuthorDate: Thu, 17 Jun 2010 14:08:13 +0800
Committer: Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 18 Jun 2010 10:45:25 +0200
sched: Fix over-scheduling bug
Commit e70971591 ("sched: Optimize unused cgroup configuration") introduced
an imbalanced scheduling bug.
If we do not use CGROUP, function update_h_load won't update h_load. When the
system has a large number of tasks far more than logical CPU number, the
incorrect cfs_rq[cpu]->h_load value will cause load_balance() to pull too
many tasks to the local CPU from the busiest CPU. So the busiest CPU keeps
going in a round robin. That will hurt performance.
The issue was found originally by a scientific calculation workload that
developed by Yanmin. With that commit, the workload performance drops
about 40%.
CPU before after
00 : 2 : 7
01 : 1 : 7
02 : 11 : 6
03 : 12 : 7
04 : 6 : 6
05 : 11 : 7
06 : 10 : 6
07 : 12 : 7
08 : 11 : 6
09 : 12 : 6
10 : 1 : 6
11 : 1 : 6
12 : 6 : 6
13 : 2 : 6
14 : 2 : 6
15 : 1 : 6
Reviewed-by: Yanmin zhang <yanmin.zhang@intel.com>
Signed-off-by: Alex Shi <alex.shi@intel.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <1276754893.9452.5442.camel@debian>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
kernel/sched.c | 3 ---
1 files changed, 0 insertions(+), 3 deletions(-)
diff --git a/kernel/sched.c b/kernel/sched.c
index 2aaceeb..6c9e7c8 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -1657,9 +1657,6 @@ static void update_shares(struct sched_domain *sd)
static void update_h_load(long cpu)
{
- if (root_task_group_empty())
- return;
-
walk_tg_tree(tg_load_down, tg_nop, (void *)cpu);
}
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2010-06-18 10:19 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-06-17 6:08 [patch] Over schedule issue fixing Alex,Shi
2010-06-18 4:25 ` Alex,Shi
2010-06-18 7:16 ` Peter Zijlstra
2010-06-18 10:18 ` [tip:sched/urgent] sched: Fix over-scheduling bug tip-bot for Alex,Shi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox