Re: sched: Improve load balancing in the presence of idle CPUs

[Preeti U Murthy <preeti@linux.vnet.ibm.com>]

Hi Jason,

On 03/31/2015 12:25 AM, Jason Low wrote:
> Hi Preeti,
> 
> I noticed that another commit 4a725627f21d converted the check in
> nohz_kick_needed() from idle_cpu() to rq->idle_balance, causing a
> potentially outdated value to be used if this cpu is able to pull tasks
> using rebalance_domains(), and nohz_kick_needed() directly returning
> false.

I see that rebalance_domains() will be run at the end of the scheduler
tick interrupt handling. trigger_load_balance() only sets the softirq,
it does not call rebalance_domains() immediately. So the call graph
would be:

rq->idle_balance = idle_cpu()
|____trigger_load_balance()
     |_____raise SCHED_SOFTIRQ - we are handling interrupt,hence defer
           |____nohz_kick_needed()
               |____rebalance_domains() run through the softirqd.

Correct me if I am wrong but since we do not pull any load between the
rq->idle_balance update and nohz_kick_needed(), we are safe in reading
rq->idle_balance in nohz_kick_needed().

> 
> Would this patch also help address some of the issue you are seeing?
> 
> Signed-off-by: Jason Low <jason.low2@hp.com>
> ---
>  kernel/sched/fair.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index fdae26e..ba8ec1a 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -7644,7 +7644,7 @@ static void nohz_idle_balance(struct rq *this_rq, enum cpu_idle_type idle)
>  		 * balancing owner will pick it up.
>  		 */
>  		if (need_resched())
> -			break;
> +			goto end;

Why is this hunk needed?

Regards
Preeti U Murthy
> 
>  		rq = cpu_rq(balance_cpu);
> 
> @@ -7687,7 +7687,7 @@ static inline bool nohz_kick_needed(struct rq *rq)
>  	int nr_busy, cpu = rq->cpu;
>  	bool kick = false;
> 
> -	if (unlikely(rq->idle_balance))
> +	if (unlikely(idle_cpu(cpu)))
>  		return false;
> 
>         /*
>