Re: [QUERY] Confusing usage of rq->nr_running in load balancing

[Preeti U Murthy <preeti@linux.vnet.ibm.com>]

On 09/05/2014 05:57 PM, Vincent Guittot wrote:
> On 5 September 2014 14:19, Preeti U Murthy <preeti@linux.vnet.ibm.com> wrote:
>> Hi Vincent,
>>
>> On 09/03/2014 10:28 PM, Vincent Guittot wrote:
>>> On 3 September 2014 14:21, Preeti U Murthy <preeti@linux.vnet.ibm.com> wrote:
>>>> Hi,
>>>
>>> Hi Preeti,
>>>
>>>>
>>>> There are places in kernel/sched/fair.c in the load balancing part where
>>>> rq->nr_running is used as against cfs_rq->nr_running. At least I could
>>>> not make out why the former was used in the following scenarios.
>>>> It looks to me that it can very well lead to incorrect load balancing.
>>>> Also I did not pay attention to the numa balancing part of the code
>>>> while skimming through this file to catch this scenario. There are a
>>>> couple of places there too which need to be scrutinized.
>>>>
>>>> 1. load_balance(): The check (busiest->nr_running > 1)
>>>> The load balancing would be futile if there are tasks of other
>>>> scheduling classes, wouldn't it?
>>>
>>> agree with you
>>>
>>>>
>>>> 2. active_load_balance_cpu_stop(): A similar check and a similar
>>>> consequence as 1 here.
>>>
>>> agree with you
>>>
>>>>
>>>> 3. nohz_kick_needed() : We check for more than one task on the runqueue
>>>> and hence trigger load balancing even if there are rt-tasks.
>>>
>>> I can see one potentiel reason why rq->nr_running is interesting that
>>> is the group capacity might have changed because of non cfs tasks
>>> since last load balance. So we need to monitor the change of the
>>> groups' capacity to ensure that the average load of each group is
>>> still in the same level
>>>
>>>>
>>>> 4. cpu_avg_load_per_task(): This stands out among the rest as an
>>>> incorrect usage of rq->nr_running in place of cfs_rq->nr_running. We
>>>> divide the load associated with the cfs_rq by the number of tasks on the
>>>> rq. This will make the cfs_rq load look smaller.
>>>
>>> This one is solved in the consolidation of cpu_capacity patchset
>>
>> Sorry, but I don't see where in your patchset you have addressed this
>> issue. Can you please point out the patch?
> 
> In [PATCH v5 03/12] sched: fix avg_load computation:
> 
> static unsigned long cpu_avg_load_per_task(int cpu)
>  {
>         struct rq *rq = cpu_rq(cpu);
> -       unsigned long nr_running = ACCESS_ONCE(rq->nr_running);
> +       unsigned long nr_running = ACCESS_ONCE(rq->cfs.h_nr_running);
>         unsigned long load_avg = rq->cfs.runnable_load_avg;
> 
> Are you referring to another problem than the one above ?

No Vincent, this is one. Thanks for pointing it out.

Regards
Preeti U Murthy
> 
> Regards
> Vincent
> 
>>
>> Regards
>> Preeti U Murthy
>>
>