Re: [PATCH] sched/fair: vruntime should normalize when switching from fair

[Dietmar Eggemann <dietmar.eggemann@arm.com>]

Hi,

On 09/26/2018 11:50 AM, Wanpeng Li wrote:
> Hi Dietmar,
> On Tue, 28 Aug 2018 at 22:55, Dietmar Eggemann <dietmar.eggemann@arm.com> wrote:
>>
>> On 08/27/2018 12:14 PM, Peter Zijlstra wrote:
>>> On Fri, Aug 24, 2018 at 02:24:48PM -0700, Steve Muckle wrote:
>>>> On 08/24/2018 02:47 AM, Peter Zijlstra wrote:
>>>>>>> On 08/17/2018 11:27 AM, Steve Muckle wrote:

[...]

>>>>>>>> - later, when the prio is deboosted and the task is moved back
>>>>>>>>       to the fair class, the fair rq's min_vruntime is added to
>>>>>>>>       the task's vruntime, even though it wasn't subtracted earlier.
> 
> Could you point out when the fair rq's min_vruntime is added to the
> task's vruntime in your *later* scenario? attach_task_cfs_rq will not
> do that the same reason as detach_task_cfs_rq. fair task's
> sched_remote_wakeup is false which results in vruntime will not be
> renormalized in enqueue_entity.

The cfs_rq->min_vruntime is still added to the se->vruntime in
enqueue_task_fair().

It's just that without this patch, which adds the '&&
p->sched_remote_wakeup' bit to the condition under which
vruntime_normalized() returns true, detach_task_cfs_rq() won't go into the
'if (!vruntime_normalized(p))' path and not subtract cfs_rq->min_vruntime
from se->vruntime. 

Since 'task_cpu(p) equal cpu' in try_to_wake_up() for the fair task,
WF_MIGRATED is not set and set_task_cpu() -> migrate_task_rq_fair()
is not called which could subtract cfs_rq->min_vruntime from
se->vruntime as well.

My former example with a different set of trace events:

fair_task-3580  [004]    35.389346: sched_stat_runtime:   comm=fair_task pid=3580 runtime=45312 [ns] vruntime=46922871 [ns] <-- se->vruntime=46.922.871
...
  rt_task-3579  [000]    35.391573: sched_waking:         comm=fair_task pid=3580 prio=120 target_cpu=004
...
  rt_task-3579  [000]    35.391627: sched_pi_setprio:     comm=fair_task pid=3580 oldprio=120 newprio=19
...
  rt_task-3579  [000]    35.391661: bprint:               detach_task_cfs_rq: task=fair_task pid=3580 cpu=4 vruntime_normalized=1
  rt_task-3579  [000]    35.391706: sched_switch:         rt_task:3579 [19] D ==> swapper/0:0 [120]
   <idle>-0     [004]    35.391834: sched_wakeup:         fair_task:3580 [19] success=1 CPU:004
   <idle>-0     [004]    35.391840: sched_switch:         swapper/4:0 [120] S ==> fair_task:3580 [19]
fair_task-3580  [004]    35.391853: sched_pi_setprio:     comm=fair_task pid=3580 oldprio=19 newprio=120
...
fair_task-3580  [004]    35.391863: bprint:               enqueue_task_fair: task=fair_task pid=3580 curr=0 se->vruntime=93845742 cpu=4 cfs_rq->min_vruntime=46922871
...
fair_task-3580  [004]    35.391877: sched_waking:         comm=rt_task pid=3579 prio=19 target_cpu=000
...
fair_task-3580  [004]    35.391885: sched_stat_runtime:   comm=fair_task pid=3580 runtime=31250 [ns] vruntime=93876992 [ns] <-- se->vruntime=93.876.992