Re: sched: how to pick runqueue when checking task hot?

Re: sched: how to pick runqueue when checking task hot?

[Peter Zijlstra]

[-- Attachment #1: Type: text/plain, Size: 1799 bytes --]

On Fri, Jun 06, 2014 at 02:59:56PM +0800, zhdxzx@sina.com wrote:
> Hi all
> 
> When computing cache hot, question-1: should we check the number of running tasks
> against the migration target runqueue, as shown by the following diff?
> 
> It looks that we dont migrate task if it is buddy and the target cpu is not idle.
> 
> 
> --- a/kernel/sched/fair.c	Fri Jun  6 12:37:37 2014
> +++ b/kernel/sched/fair.c	Fri Jun  6 13:55:35 2014
> @@ -5051,7 +5050,7 @@ task_hot(struct task_struct *p, u64 now)
>  	/*
>  	 * Buddy candidates are cache hot:
>  	 */
> -	if (sched_feat(CACHE_HOT_BUDDY) && this_rq()->nr_running &&
> +	if (sched_feat(CACHE_HOT_BUDDY) && env->dst_rq->nr_running &&
>  			(&p->se == cfs_rq_of(&p->se)->next ||
>  			 &p->se == cfs_rq_of(&p->se)->last))
>  		return 1;


That is indeed the effective result for normal balancing, seeing how it
pulls to the dst rq, and this_rq would be dst.

This is of course false for a number of cases these days, most obviously
the nohz idle balancing.

> But based on the comment, question-2: should we check running tasks
> against the runqueue of the given task?

> --- a/kernel/sched/fair.c	Fri Jun  6 12:37:37 2014
> +++ b/kernel/sched/fair.c	Fri Jun  6 14:32:34 2014
> @@ -5051,7 +5051,7 @@ task_hot(struct task_struct *p, u64 now)
>  	/*
>  	 * Buddy candidates are cache hot:
>  	 */
> -	if (sched_feat(CACHE_HOT_BUDDY) && this_rq()->nr_running &&
> +	if (sched_feat(CACHE_HOT_BUDDY) && task_rq(p)->nr_running &&
>  			(&p->se == cfs_rq_of(&p->se)->next ||
>  			 &p->se == cfs_rq_of(&p->se)->last))
>  		return 1;

That does appear to make more sense indeed, seeing how buddies are pairs
of tasks, so protecting a lone task doesn't make sense.


Mike, how did you intend this code to work? 

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: sched: how to pick runqueue when checking task hot?

[Mike Galbraith]

On Fri, 2014-06-06 at 12:15 +0200, Peter Zijlstra wrote:

> > --- a/kernel/sched/fair.c	Fri Jun  6 12:37:37 2014
> > +++ b/kernel/sched/fair.c	Fri Jun  6 14:32:34 2014
> > @@ -5051,7 +5051,7 @@ task_hot(struct task_struct *p, u64 now)
> >  	/*
> >  	 * Buddy candidates are cache hot:
> >  	 */
> > -	if (sched_feat(CACHE_HOT_BUDDY) && this_rq()->nr_running &&
> > +	if (sched_feat(CACHE_HOT_BUDDY) && task_rq(p)->nr_running &&
> >  			(&p->se == cfs_rq_of(&p->se)->next ||
> >  			 &p->se == cfs_rq_of(&p->se)->last))
> >  		return 1;
> 
> That does appear to make more sense indeed, seeing how buddies are pairs
> of tasks, so protecting a lone task doesn't make sense.
> 
> 
> Mike, how did you intend this code to work?

IIRC, this_rq()->nr_running was to say if we're idle, we don't care that
it's last/next, pull it.  Not sure I'm the one who did that, but could
be, I didn't look.

-Mike

Re: sched: how to pick runqueue when checking task hot?

[Peter Zijlstra]

[-- Attachment #1: Type: text/plain, Size: 1531 bytes --]

On Fri, Jun 06, 2014 at 01:16:23PM +0200, Mike Galbraith wrote:
> On Fri, 2014-06-06 at 12:15 +0200, Peter Zijlstra wrote:
> 
> > > --- a/kernel/sched/fair.c	Fri Jun  6 12:37:37 2014
> > > +++ b/kernel/sched/fair.c	Fri Jun  6 14:32:34 2014
> > > @@ -5051,7 +5051,7 @@ task_hot(struct task_struct *p, u64 now)
> > >  	/*
> > >  	 * Buddy candidates are cache hot:
> > >  	 */
> > > -	if (sched_feat(CACHE_HOT_BUDDY) && this_rq()->nr_running &&
> > > +	if (sched_feat(CACHE_HOT_BUDDY) && task_rq(p)->nr_running &&
> > >  			(&p->se == cfs_rq_of(&p->se)->next ||
> > >  			 &p->se == cfs_rq_of(&p->se)->last))
> > >  		return 1;
> > 
> > That does appear to make more sense indeed, seeing how buddies are pairs
> > of tasks, so protecting a lone task doesn't make sense.
> > 
> > 
> > Mike, how did you intend this code to work?
> 
> IIRC, this_rq()->nr_running was to say if we're idle, we don't care that
> it's last/next, pull it.  Not sure I'm the one who did that, but could
> be, I didn't look.
> 

commit f685ceacab07d3f6c236f04803e2f2f0dbcc5afb
Author: Mike Galbraith <efault@gmx.de>
Date:   Fri Oct 23 23:09:22 2009 +0200

    sched: Strengthen buddies and mitigate buddy induced latencies

...

-       if (sched_feat(CACHE_HOT_BUDDY) &&
+       if (sched_feat(CACHE_HOT_BUDDY) && this_rq()->nr_running &&
                        (&p->se == cfs_rq_of(&p->se)->next ||
                         &p->se == cfs_rq_of(&p->se)->last))

Yeah, was you ;-)

OK, so we want dst_rq. Thanks!


[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: sched: how to pick runqueue when checking task hot?

[Mike Galbraith]

On Fri, 2014-06-06 at 13:34 +0200, Peter Zijlstra wrote: 
> On Fri, Jun 06, 2014 at 01:16:23PM +0200, Mike Galbraith wrote:
> > On Fri, 2014-06-06 at 12:15 +0200, Peter Zijlstra wrote:
> > 
> > > > --- a/kernel/sched/fair.c	Fri Jun  6 12:37:37 2014
> > > > +++ b/kernel/sched/fair.c	Fri Jun  6 14:32:34 2014
> > > > @@ -5051,7 +5051,7 @@ task_hot(struct task_struct *p, u64 now)
> > > >  	/*
> > > >  	 * Buddy candidates are cache hot:
> > > >  	 */
> > > > -	if (sched_feat(CACHE_HOT_BUDDY) && this_rq()->nr_running &&
> > > > +	if (sched_feat(CACHE_HOT_BUDDY) && task_rq(p)->nr_running &&
> > > >  			(&p->se == cfs_rq_of(&p->se)->next ||
> > > >  			 &p->se == cfs_rq_of(&p->se)->last))
> > > >  		return 1;
> > > 
> > > That does appear to make more sense indeed, seeing how buddies are pairs
> > > of tasks, so protecting a lone task doesn't make sense.
> > > 
> > > 
> > > Mike, how did you intend this code to work?
> > 
> > IIRC, this_rq()->nr_running was to say if we're idle, we don't care that
> > it's last/next, pull it.  Not sure I'm the one who did that, but could
> > be, I didn't look.
> > 
> 
> commit f685ceacab07d3f6c236f04803e2f2f0dbcc5afb
> Author: Mike Galbraith <efault@gmx.de>
> Date:   Fri Oct 23 23:09:22 2009 +0200
> 
>     sched: Strengthen buddies and mitigate buddy induced latencies
> 
> ...
> 
> -       if (sched_feat(CACHE_HOT_BUDDY) &&
> +       if (sched_feat(CACHE_HOT_BUDDY) && this_rq()->nr_running &&
>                         (&p->se == cfs_rq_of(&p->se)->next ||
>                          &p->se == cfs_rq_of(&p->se)->last))
> 
> Yeah, was you ;-)

    Last hunk prevents buddies from stymieing BALANCE_NEWIDLE via
    CACHE_HOT_BUDDY.

Last hunk, first hunk, whatever, that's what it was for :)

-Mike