Re: [PATCH v3 05/22] sched: remove domain iterations in fork/exec/wake

[Morten Rasmussen <Morten.Rasmussen@arm.com>]

On Sat, Jan 05, 2013 at 08:37:34AM +0000, Alex Shi wrote:
> Guess the search cpu from bottom to up in domain tree come from
> commit 3dbd5342074a1e sched: multilevel sbe sbf, the purpose is
> balancing over tasks on all level domains.
> 
> This balancing cost much if there has many domain/groups in a large
> system. And force spreading task among different domains may cause
> performance issue due to bad locality.
> 
> If we remove this code, we will get quick fork/exec/wake, plus better
> balancing among whole system, that also reduce migrations in future
> load balancing.
> 
> This patch increases 10+% performance of hackbench on my 4 sockets
> NHM and SNB machines.
> 
> Signed-off-by: Alex Shi <alex.shi@intel.com>
> ---
>  kernel/sched/fair.c | 20 +-------------------
>  1 file changed, 1 insertion(+), 19 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index ecfbf8e..895a3f4 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -3364,15 +3364,9 @@ select_task_rq_fair(struct task_struct *p, int sd_flag, int wake_flags)
>  		goto unlock;
>  	}
>  
> -	while (sd) {
> +	if (sd) {
>  		int load_idx = sd->forkexec_idx;
>  		struct sched_group *group;
> -		int weight;
> -
> -		if (!(sd->flags & sd_flag)) {
> -			sd = sd->child;
> -			continue;
> -		}
>  
>  		if (sd_flag & SD_BALANCE_WAKE)
>  			load_idx = sd->wake_idx;
> @@ -3382,18 +3376,6 @@ select_task_rq_fair(struct task_struct *p, int sd_flag, int wake_flags)
>  			goto unlock;
>  
>  		new_cpu = find_idlest_cpu(group, p, cpu);
> -
> -		/* Now try balancing at a lower domain level of new_cpu */
> -		cpu = new_cpu;
> -		weight = sd->span_weight;
> -		sd = NULL;
> -		for_each_domain(cpu, tmp) {
> -			if (weight <= tmp->span_weight)
> -				break;
> -			if (tmp->flags & sd_flag)
> -				sd = tmp;
> -		}
> -		/* while loop will break here if sd == NULL */

I agree that this should be a major optimization. I just can't figure
out why the existing recursive search for an idle cpu switches to the
new cpu near the end and then starts a search for an idle cpu in the new
cpu's domain. Is this to handle some exotic sched domain configurations?
If so, they probably wouldn't work with your optimizations.

Morten

>  	}
>  unlock:
>  	rcu_read_unlock();
> -- 
> 1.7.12
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>