Re: [PATCH v4 3/3] sched: optimize migration by forcing rmb() and updating to be called once

[Peter Zijlstra <peterz@infradead.org>]

On Wed, Nov 11, 2015 at 08:51:47AM +0900, Byungchul Park wrote:
> On Tue, Nov 10, 2015 at 01:16:47PM +0100, Peter Zijlstra wrote:
> > So the problem is that as soon as that ->cpu store comes through, the
> > other rq->lock can happen, even though we might still hold a rq->lock
> > thinking we're serialized.
> > 
> > Take for instance move_queued_tasks(), it does:
> > 
> > 	dequeue_task(rq, p, 0);
> > 	p->on_rq = TASK_ON_RQ_MIGRATING;
> > 	set_task_cpu(p, new_cpu) {
> > 	  __set_task_cpu();
> > 
> > ^^^ here holding rq->lock is insufficient and the below:
> > 
> > 	  p->sched_class->migrate_task_rq()
> 
> Thank you for explaning in detail, but this's why i asked you.

> Yes, rq->lock is insufficient in this place as you said, but
> should migrate_task_rq() be serialized by rq->lock? I might have
> agreed with you if the migrate_task_rq() should be serialized by
> rq->lock, but I think it's not the case. I think it would be of
> if task->pi_lock can work correcly within *if statement* in 
> set_task_cpu(). Wrong?

So currently, set_task_cpu() is serialized by:

 - p->pi_lock; on wakeup
 - rq->lock; otherwise

(see the #ifdef CONFIG_LOCKDEP comment in set_task_cpu())

This means that sched_class::migrate_task() cannot indeed rely on
rq->lock for full serialization, however it still means that
task_rq_lock() will fully serialize against the thing.

By changing this, it no longer will.

Even without that; I think such a change, if correct, is very fragile
and prone to creating problems later on, and sets bad precedent.