Re: [PATCH 2/4] sched/fair: Introduce idle enter/exit balance callbacks

[Jiri Olsa <jolsa@redhat.com>]

On Mon, Jun 20, 2016 at 04:30:03PM +0200, Peter Zijlstra wrote:
> On Mon, Jun 20, 2016 at 02:15:12PM +0200, Jiri Olsa wrote:
> > Introducing idle enter/exit balance callbacks to keep
> > balance.idle_cpus_mask cpumask of current idle cpus
> > in system.
> > 
> > It's used only when REBALANCE_AFFINITY feature is
> > switched on. The code functionality of this feature
> > is introduced in following patch.
> > 
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  kernel/sched/fair.c  | 32 ++++++++++++++++++++++++++++++++
> >  kernel/sched/idle.c  |  2 ++
> >  kernel/sched/sched.h |  3 +++
> >  3 files changed, 37 insertions(+)
> > 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index f19c9435c64d..78c4127f2f3a 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -7802,6 +7802,37 @@ static inline int on_null_domain(struct rq *rq)
> >  	return unlikely(!rcu_dereference_sched(rq->sd));
> >  }
> >  
> > +static struct {
> > +	cpumask_var_t	idle_cpus_mask;
> > +	atomic_t	nr_cpus;
> > +} balance ____cacheline_aligned;
> 
> How is this different from the nohz idle cpu mask?

well, the nohz idle cpu mask is deep in the nohz code:

tick_irq_exit
  if ((idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
    if (!in_interrupt())
      tick_nohz_irq_exit
        if (ts->inidle) {
          __tick_nohz_idle_enter(ts)
            tick_nohz_idle_enter
             __tick_nohz_idle_enter
               if (can_stop_idle_tick(cpu, ts)) {
                 tick_nohz_stop_sched_tick
                   if (ts->tick_stopped)
                     nohz_balance_enter_idle(cpu) {
                       set idle mask for cpu
                     }
        } else {
                tick_nohz_full_update_tick(ts);
                 if (can_stop_full_tick(ts))
                    tick_nohz_stop_sched_tick
                      if (ts->tick_stopped)
                        nohz_balance_enter_idle(cpu) {
                          set idle mask for cpu
                        }
	}

cpu_idle_loop
  tick_nohz_idle_enter
     __tick_nohz_idle_enter(ts)
       tick_nohz_idle_enter
        __tick_nohz_idle_enter
          if (can_stop_idle_tick(cpu, ts)) {
            tick_nohz_stop_sched_tick
              if (ts->tick_stopped)
                nohz_balance_enter_idle(cpu) {
                  set idle mask for cpu
                }


sched_cpu_dying
  nohz_balance_exit_idle(cpu) {
    unset idle mask for cpu
  }

trigger_load_balance
  nohz_kick_needed
    nohz_balance_exit_idle(cpu) {
      unset idle mask for cpu
    }

... I might have missed some of the paths


so it might not be that easy to switch it to use the easy
change I added as part of this RFC, but AFAIU it should be
the same idle mask, but this approach might be too naive
and miss some idle enter/exit paths.. CC-ing Frederic

thanks,
jirka