Re: [PATCH 2/4] sched: Document Program-Order guarantees

Peter Zijlstra
2015-11-20T14:18:21Z

On Fri, Nov 20, 2015 at 10:08:50PM +0800, Boqun Feng wrote:
> Hi Peter,
>
> On Fri, Nov 20, 2015 at 11:02:30AM +0100, Peter Zijlstra wrote:
> [snip]
> > + * BLOCKING -- aka. SLEEP + WAKEUP
> > + *
> > + * For blocking we (obviously) need to provide the same guarantee as for
> > + * migration. However the means are completely different as there is no lock
> > + * chain to provide order. Instead we do:
> > + *
> > + * 1) smp_store_release(X->on_cpu, 0)
> > + * 2) smp_cond_acquire(!X->on_cpu)
> > + *
> > + * Example:
> > + *
> > + * CPU0 (schedule) CPU1 (try_to_wake_up) CPU2 (schedule)
> > + *
> > + * LOCK rq(0)->lock LOCK X->pi_lock
> > + * dequeue X
> > + * sched-out X
> > + * smp_store_release(X->on_cpu, 0);
> > + *
> > + * smp_cond_acquire(!X->on_cpu);
> > + * X->state = WAKING
> > + * set_task_cpu(X,2)
> > + *
> > + * LOCK rq(2)->lock
> > + * enqueue X
> > + * X->state = RUNNING
> > + * UNLOCK rq(2)->lock
> > + *
> > + * LOCK rq(2)->lock // orders against CPU1
> > + * sched-out Z
> > + * sched-in X
> > + * UNLOCK rq(1)->lock
> > + *
> > + * UNLOCK X->pi_lock
> > + * UNLOCK rq(0)->lock
> > + *
> > + *
> > + * However; for wakeups there is a second guarantee we must provide, namely we
> > + * must observe the state that lead to our wakeup. That is, not only must our
> > + * task observe its own prior state, it must also observe the stores prior to
> > + * its wakeup.
> > + *
> > + * This means that any means of doing remote wakeups must order the CPU doing
> > + * the wakeup against the CPU the task is going to end up running on. This,
> > + * however, is already required for the regular Program-Order guarantee above,
> > + * since the waking CPU is the one issueing the ACQUIRE (2).
> > + *
>
> Hope I'm the only one who got confused about the "2" in "ACQUIRE (2)",
> what does that refer? "2) smp_cond_acquire(!X->on_cpu)"?

Yes, exactly that. Would an unadorned 2 be clearer?