Re: [PATCH RFC v2 rcu] Fix get_state_synchronize_rcu_full() GP-start detection

[Frederic Weisbecker <frederic@kernel.org>]

Le Fri, Jan 24, 2025 at 11:40:54AM -0800, Paul E. McKenney a écrit :
> > > > I'm wondering, what prevents us from removing rcu_state.gp_seq and rely only on
> > > > the root node for the global state ?
> > > 
> > > One scenario comes to mind immediately.  There may be others.
> > > 
> > > Suppose we were running with default configuration on a system with
> > > "only" eight CPUs.  Then there is only the one rcu_node structure,
> > > which is both root and leaf.  Without rcu_state.gp_seq, there
> > > would be no way to communicate the beginning of the grace period to
> > > get_state_synchronize_rcu_full() without also allowing quiescent states
> > > to be reported.  There would thus be no time in which to check for newly
> > > onlined/offlined CPUs.
> > 
> > Heh, that makes sense! Though perhaps that qsmaskinit[next] handling part
> > could be done before rcu_seq_start()?
> 
> If we do that, aren't we vulnerable to a CPU coming online just after
> we handled qsmaskinit{,next} and just before we do rcu_seq_start()?

But then that CPU is guaranteed to see the pre-GP accesses because
rcutree_report_cpu_starting() holds the rnp lock.

Hmm but it won't see the starting gp_seq and it might therefore be one GP
too early while accelerating its own callbacks (mis-attributing them the
current GP in progress instead of a subsequent one) even though other CPUs
may have reported their own QS already.

Because I only now realize this necessary order while starting a grace period:

1) rcu_seq_start(rcu_state.gp_seq)
2) go through ->qsmaskinit[next] _while holding leaf rnp locks_
3) go through the whole tree breadth first to reflect rcu_state.gp_seq to
   rnp->gp_seq. CPUs may start reporting QS concurrently once the first leaf
   node is reached

And the step 2) with its fully ordered locking on leaves which release the write
to rcu_state.gp_seq is what makes sure that if any CPU from step 3) has already
reported a QS, it is guaranteed that any call to rcu_seq_snap() under any other
leaf rnp locking will return the sequence of the subsequent GP number and not the
current one in progress.

This is what makes RCU an entertaining companion. You (think you) understand
it in the evening and then you forget it all over again in the next morning.
(And in lunch time you stare at things...).

Thanks.