Re: netperf ~50% regression with 2.6.33-rc1, bisect to 1b9508f

[Lin Ming <ming.m.lin@intel.com>]

On Mon, 2010-01-25 at 19:35 +0800, Mike Galbraith wrote:
> On Mon, 2010-01-25 at 18:03 +0800, Lin Ming wrote:
> 
> > With above commit, the idle balance was rate limited, so CPU 15(server,
> > waiting data from client) is idle at most time.
> > 
> > CPU0(client) executes as below,
> > 
> > try_to_wake_up
> >    check_preempt_curr_idle
> >       resched_task
> >          smp_send_reschedule
> > 
> > This causes a lot of rescheduling IPI.
> > 
> > This commit can't be reverted due to conflict, so I just add below code
> > to disable "Rate-limit newidle" and the performance was recovered.
> > 
> > diff --git a/kernel/sched.c b/kernel/sched.c
> > index 18cceee..588fdef 100644
> > --- a/kernel/sched.c
> > +++ b/kernel/sched.c
> > @@ -4421,9 +4421,6 @@ static void idle_balance(int this_cpu, struct rq *this_rq)
> >  
> >  	this_rq->idle_stamp = this_rq->clock;
> >  
> > -	if (this_rq->avg_idle < sysctl_sched_migration_cost)
> > -		return;
> > -
> >  	for_each_domain(this_cpu, sd) {
> >  		unsigned long interval;
> >  
> 
> Heh, so you should see the same thing with newidle disabled, as it was
> in .31 and many kernels prior.  Do you?

Weird.
2.6.31 does not have so many reschedule IPI.

This Nehalem machine has 3 domain levels,
$ grep . cpu0/domain*/name
cpu0/domain0/name:SIBLING
cpu0/domain1/name:MC
cpu0/domain2/name:NODE

For 2.6.31, SD_BALANCE_NEWIDLE is only set on SIBLING level.
For 2.6.32-rc1, SD_BALANCE_NEWIDLE is set on all 3 levels.

I can see many reschedule IPI in 2.6.32-rc1 if SD_BALANCE_NEWIDLE is
cleared for all 3 levels.
But for 2.6.31, I didn't see so many IPI even SD_BALANCE_NEWIDLE is
cleared on SIBLING level.

So it seems something happens between 2.6.31 and 2.6.32-rc1.
I'll bisect ...

Lin Ming