Re: [RESEND RFC PATCH V3] sched: Improve scalability of select_idle_sibling using SMT balance

[Peter Zijlstra <peterz@infradead.org>]

On Fri, Feb 02, 2018 at 11:53:40AM -0500, Steven Sistare wrote:
> >> +static int select_idle_smt(struct task_struct *p, struct sched_group *sg)
> >>  {
> >> +	int i, rand_index, rand_cpu;
> >> +	int this_cpu = smp_processor_id();
> >>  
> >> +	rand_index = CPU_PSEUDO_RANDOM(this_cpu) % sg->group_weight;
> >> +	rand_cpu = sg->cp_array[rand_index];
> > 
> > Right, so yuck.. I know why you need that, but that extra array and
> > dereference is the reason I never went there.
> > 
> > How much difference does it really make vs the 'normal' wrapping search
> > from last CPU ?
> > 
> > This really should be a separate patch with separate performance numbers
> > on.
> 
> For the benefit of other readers, if we always search and choose starting from
> the first CPU in a core, then later searches will often need to traverse the first
> N busy CPU's to find the first idle CPU.  Choosing a random starting point avoids
> such bias.  It is probably a win for processors with 4 to 8 CPUs per core, and
> a slight but hopefully negligible loss for 2 CPUs per core, and I agree we need
> to see performance data for this as a separate patch to decide.  We have SPARC
> systems with 8 CPUs per core.

Which is why the current code already doesn't start from the first cpu
in the mask. We start at whatever CPU the task ran last on, which is
effectively 'random' if the system is busy.

So how is a per-cpu rotor better than that?