Re: [PATCH RESEND] sched: prefer an idle cpu vs an idle sibling for BALANCE_WAKE

[Josef Bacik <jbacik@fb.com>]

On 06/11/2015 10:35 PM, Mike Galbraith wrote:
> On Thu, 2015-05-28 at 13:05 +0200, Peter Zijlstra wrote:
>
>> @@ -5022,22 +5026,28 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int sd_flag, int wake_f
>>   		 * If both cpu and prev_cpu are part of this domain,
>>   		 * cpu is a valid SD_WAKE_AFFINE target.
>>   		 */
>> -		if (want_affine && (tmp->flags & SD_WAKE_AFFINE) &&
>> -		    cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) {
>> +		if (want_affine && !affine_sd &&
>> +		    (tmp->flags & SD_WAKE_AFFINE) &&
>> +		    cpumask_test_cpu(prev_cpu, sched_domain_span(tmp)))
>>   			affine_sd = tmp;
>> -			break;
>> -		}
>>
>>   		if (tmp->flags & sd_flag)
>>   			sd = tmp;
>> +		else if (!want_affine || (want_affine && affine_sd))
>> +			break;
>>   	}
>
> Hm, new_cpu == cpu.
>
>> -	if (affine_sd && cpu != prev_cpu && wake_affine(affine_sd, p, sync))
>> +	if (affine_sd && cpu != prev_cpu && wake_affine(affine_sd, p, sync)) {
>>   		prev_cpu = cpu;
>> +		sd = NULL; /* WAKE_AFFINE trumps BALANCE_WAKE */
>> +	}
>
> If branch above is not taken, new_cpu remains cpu.
>
>>   	if (sd_flag & SD_BALANCE_WAKE) {
>> -		new_cpu = select_idle_sibling(p, prev_cpu);
>> -		goto unlock;
>> +		int tmp = select_idle_sibling(p, prev_cpu);
>> +		if (tmp >= 0) {
>> +			new_cpu = tmp;
>> +			goto unlock;
>> +		}
>>   	}
>
> If select_idle_sibling() returns -1, new_cpu remains cpu.
>
>>
>>   	while (sd) {
>
> If sd == NULL, we fall through and try to pull wakee despite nacked-by
> tsk_cpus_allowed() or wake_affine().
>

So maybe add a check in the if (sd_flag & SD_BALANCE_WAKE) for something 
like this

if (tmp >= 0) {
	new_cpu = tmp;
	goto unlock;
} else if (!want_affine) {
	new_cpu = prev_cpu;
}

so we can make sure we're not being pushed onto a cpu that we aren't 
allowed on?  Thanks,

Josef