Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Wed, Oct 07, 2015 at 01:01:20PM +0200, Peter Zijlstra wrote:
> On Wed, Oct 07, 2015 at 08:42:05AM +0000, Mathieu Desnoyers wrote:
> > ----- On Oct 7, 2015, at 3:51 AM, Peter Zijlstra peterz@infradead.org wrote:
> > 
> > > On Tue, Oct 06, 2015 at 01:58:50PM -0700, Paul E. McKenney wrote:
> > >> On Tue, Oct 06, 2015 at 10:29:37PM +0200, Peter Zijlstra wrote:
> > >> > On Tue, Oct 06, 2015 at 09:29:21AM -0700, Paul E. McKenney wrote:
> > >> > > +static void __maybe_unused rcu_report_exp_rnp(struct rcu_state *rsp,
> > >> > > +					      struct rcu_node *rnp, bool wake)
> > >> > > +{
> > >> > > +	unsigned long flags;
> > >> > > +	unsigned long mask;
> > >> > > +
> > >> > > +	raw_spin_lock_irqsave(&rnp->lock, flags);
> > >> > 
> > >> > Normally we require a comment with barriers, explaining the order and
> > >> > the pairing etc.. :-)
> > >> > 
> > >> > > +	smp_mb__after_unlock_lock();
> > >> 
> > >> Hmmmm...  That is not good.
> > >> 
> > >> Worse yet, I am missing comments on most of the pre-existing barriers
> > >> of this form.
> > > 
> > > Yes I noticed.. :/
> > > 
> > >> The purpose is to enforce the heavy-weight grace-period memory-ordering
> > >> guarantees documented in the synchronize_sched() header comment and
> > >> elsewhere.
> > > 
> > >> They pair with anything you might use to check for violation
> > >> of these guarantees, or, simiarly, any ordering that you might use when
> > >> relying on these guarantees.
> > > 
> > > I'm sure you know what that means, but I've no clue ;-) That is, I
> > > wouldn't know where to start looking in the RCU implementation to verify
> > > the barrier is either needed or sufficient. Unless you mean _everywhere_
> > > :-)
> > 
> > One example is the new membarrier system call. It relies on synchronize_sched()
> > to enforce this:
> 
> That again doesn't explain which UNLOCKs with non-matching lock values
> it pairs with and what particular ordering is important here.
> 
> I'm fully well aware of what sync_sched() guarantees and how one can use
> it, that is not the issue, what I'm saying is that a generic description
> of sync_sched() doesn't help in figuring out WTH that barrier is for and
> which other code I should also inspect.

Unfortunately, the answer is "pretty much all of it".  :-(

The enforced ordering relies on pretty much every acquisition/release of
an rcu_node structure's ->lock and all the dyntick-idle stuff, plus some
explicit barriers and a few smp_load_acquire()s and smp_store_release()s.

							Thanx, Paul