Re: [PATCH 2/2] sched: update runqueue clock before migrations away

[Chris Redpath <chris.redpath@arm.com>]

On 10/12/13 15:14, Peter Zijlstra wrote:
> On Tue, Dec 10, 2013 at 01:24:21PM +0000, Chris Redpath wrote:
>> What happens is that if you have a task which sleeps for a while and wakes
>> on a different CPU and the previous CPU hasn't had a tick for a while, then
>> that sleep time is lost.
>
> /me more confused now.
>
> Where does update_cfs_rq_blocked_load() account sleep time? Its only
> concerned with the blocked decay.
>
> Or is it the decay_count <= 0 case in enqueue_entity_load_avg() that's
> screwing you over?
>
Sorry, I don't appear to have explained myself well.

We don't get a negative decay_count when we synchronize the entity in
this situation because the decay_counter of the runqueue has not been
updated since the last update. __synchronize_entity_decay will return 0,
and hence the entity load average does not decay and neither does
blocked load.

It doesn't seem to matter much that the blocked load doesn't decay
because we are about to remove the load from the now-unblocked entity
anyway, but it impacts the entity load average because that sleep time
has now disappeared.

The call to update_rq_clock() is needed to get rq->clock_task updated,
and then I use update_cfs_rq_blocked_load() to get the decay_counter
updated.

It has the side-effect of updating the blocked_load_avg to stay
consistent but the effect I am looking for is the decay_counter update.

> That's guestimating the last_runnable_update based on decay_count, and
> per the previous the decay count can get slightly out of sync.

The guesstimation works fine, the issue is only that we can't tell at
this point how much time that entity was asleep when the CPU it ran on
has no tick and since it is too expensive to synchronize the clocks,
there isn't (currently) a way to find out without updating the rq we
came from.

I can't see anything handy lying around in struct rq, but is there a
copy of the jiffies held anywhere per-cpu presuming it would stop being
updated when the tick stops? If not, I could store one somewhere as part
of turning the tick off and then we could use the difference between
that and the current jiffies count to estimate the amount of time in
limbo. That would almost certainly be accurate enough for me - a few ms
won't hurt but when we lose seconds it does.

>
> If so, we should look to cure the issue enqueue_entity_load_avg() is
> trying to work around, namely the fact that we cannot assume synced
> clocks, nor can we read remote clocks atomically.
>

That would be nice, but IMHO the estimation error is small and
self-consistent so the optimization appears good.

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782