Re: [PATCH v2 0/5] sched: Add on_rq states and remove several double rq locks

[Kirill Tkhai <tkhai@yandex.ru>]

On 26.07.2014 23:39, Oleg Nesterov wrote:
> Hi Kirill,
> 
> I'll try to read this series later, just one silly question for now.
> 
> On 07/26, Kirill Tkhai wrote:
>>
>> Patch [2/5] is main in the series. It introduces new state: ONRQ_MIGRATING
>> and teaches scheduler to understand it (we need a little changes predominantly
>> in try_to_wake_up()). This will be used in the following way:
>>
>>         (we are changing task's rq)
>>
>>         raw_spin_lock(&src_rq->lock);
>>         dequeue_task(src_rq, p, 0);
>>         p->on_rq = ONRQ_MIGRATING;
>>         set_task_cpu(p, dst_cpu);
>>         raw_spin_unlock(&src_rq->lock);
>>
>>         raw_spin_lock(&dst_rq->lock);
>>         p->on_rq = ONRQ_QUEUED;
>>         enqueue_task(dst_rq, p, 0);
>>         raw_spin_unlock(&dst_rq->lock);
> 
> Hmm. And what if the code above doesn't hold p->pi_lock (4/5) and, say,
> __sched_setscheduler() does fair_sched_class->rt_sched_class transition
> in between?
> 
> ONRQ_MIGRATING helps to avoid the wrong dequeue + enqueue, but I am not
> sure about check_class_changed().
> 
> Say, switched_from_fair() will use dst_rq even if p was never queued on
> this rq... This only affects the .decay_count logic, perhaps this is fine,
> I simply do not know what this code does.

You're right. We have to check for "task_migrating" in switched_from_fair().
One more place is switched_from_dl().

> What about switched_to_rt() ? we lose the push_rt_task() logic... Hmm,
> which I can't understand too ;)
> 
> And we also lose ENQUEUE_HEAD in this case, but this looks fine.
> 
> In short: could you confirm there are no problems here?

This will be the reason of some RT/DL imbalance. We need a method how to
avoid this.

Maybe, it would be good to call something like check_class_changed()
at the end of migration process. We just need to save task's class
before migration and compare with the class after migration (for [3/5],
__migrate_task()). For [4/5] and [5/5] the class is always fair_sched_class.

Thanks for the comments. I'll think how to fix this in a good way,
and update the series.

Kirill