Re: [PATCH 2/4] sched: Document Program-Order guarantees

[Peter Zijlstra <peterz@infradead.org>]

On Fri, Nov 20, 2015 at 10:08:50PM +0800, Boqun Feng wrote:
> Hi Peter,
> 
> On Fri, Nov 20, 2015 at 11:02:30AM +0100, Peter Zijlstra wrote:
> [snip]
> > + *  BLOCKING -- aka. SLEEP + WAKEUP
> > + *
> > + * For blocking we (obviously) need to provide the same guarantee as for
> > + * migration. However the means are completely different as there is no lock
> > + * chain to provide order. Instead we do:
> > + *
> > + *   1) smp_store_release(X->on_cpu, 0)
> > + *   2) smp_cond_acquire(!X->on_cpu)
> > + *
> > + * Example:
> > + *
> > + *   CPU0 (schedule)  CPU1 (try_to_wake_up) CPU2 (schedule)
> > + *
> > + *   LOCK rq(0)->lock LOCK X->pi_lock
> > + *   dequeue X
> > + *   sched-out X
> > + *   smp_store_release(X->on_cpu, 0);
> > + *
> > + *                    smp_cond_acquire(!X->on_cpu);
> > + *                    X->state = WAKING
> > + *                    set_task_cpu(X,2)
> > + *
> > + *                    LOCK rq(2)->lock
> > + *                    enqueue X
> > + *                    X->state = RUNNING
> > + *                    UNLOCK rq(2)->lock
> > + *
> > + *                                          LOCK rq(2)->lock // orders against CPU1
> > + *                                          sched-out Z
> > + *                                          sched-in X
> > + *                                          UNLOCK rq(1)->lock
> > + *
> > + *                    UNLOCK X->pi_lock
> > + *   UNLOCK rq(0)->lock
> > + *
> > + *
> > + * However; for wakeups there is a second guarantee we must provide, namely we
> > + * must observe the state that lead to our wakeup. That is, not only must our
> > + * task observe its own prior state, it must also observe the stores prior to
> > + * its wakeup.
> > + *
> > + * This means that any means of doing remote wakeups must order the CPU doing
> > + * the wakeup against the CPU the task is going to end up running on. This,
> > + * however, is already required for the regular Program-Order guarantee above,
> > + * since the waking CPU is the one issueing the ACQUIRE (2).
> > + *
> 
> Hope I'm the only one who got confused about the "2" in "ACQUIRE (2)",
> what does that refer? "2) smp_cond_acquire(!X->on_cpu)"?

Yes, exactly that. Would an unadorned 2 be clearer?