Re: [patch 3/6] sched, nohz: sched group, domain aware nohz idle load balancing

[Peter Zijlstra <peterz@infradead.org>]

On Mon, 2011-11-28 at 15:58 -0800, Suresh Siddha wrote:
> On Thu, 2011-11-24 at 03:53 -0800, Peter Zijlstra wrote:
> > On Fri, 2011-11-18 at 15:03 -0800, Suresh Siddha wrote:
> > > Make nohz idle load balancing more scalabale by using the nr_busy_cpus in
> > > the struct sched_group_power.
> > > 
> > > Idle load balance is kicked on one of the idle cpu's when there is atleast
> > > one idle cpu and
> > > 
> > >  - a busy rq having more than one task or
> > > 
> > >  - a busy scheduler group having multiple busy cpus that exceed the sched group
> > >    power or
> > > 
> > >  - for the SD_ASYM_PACKING domain, if the lower numbered cpu's in that
> > >    domain are idle compared to the busy ones.
> > > 
> > > This will help in kicking the idle load balancing request only when
> > > there is a real imbalance. And once it is mostly balanced, these kicks will
> > > be minimized.
> > > 
> > > These changes helped improve the workload that is context switch intensive
> > > between number of task pairs by 2x on a 8 socket NHM-EX based system.
> > 
> > OK, but the nohz idle balance will still iterate the whole machine
> > instead of smaller parts, right?
> 
> In the current series, yes. one idle cpu spending a bit more time doing
> idle load balancing might be better compared to waking up multiple idle
> cpu's from deep c-states.
> 
> But if needed, we can easily partition the nohz idle load balancer load
> to multiple idle cpu's. But we need a balance between the right
> partition size vs how many idle cpu's we need to bring out of tickless
> mode to do this idle load balancing.
> 
> Current proposed series already has the infrastructure to identify
> which scheduler domain has the imbalance. Perhaps we can use that to do
> the nohz idle load balancing only for that domain.

Yeah, trouble with that is if tehre's inter-domain balance things, like
for example balance for power, where you want to idle a domain. If you
only ever look within the one domain, you'll never see the possibility
to move your last one task to this other domain.

Pesky stuff that.