Re: Normal RCU grace period can be stalled for long because need-resched flags not set?

["Paul E. McKenney" <paulmck@linux.ibm.com>]

On Sat, Jul 06, 2019 at 07:03:31PM -0400, Joel Fernandes wrote:
> On Sat, Jul 06, 2019 at 11:21:17AM -0700, Paul E. McKenney wrote:
> > On Sat, Jul 06, 2019 at 08:02:30AM -0400, Joel Fernandes wrote:
> > > On Thu, Jul 04, 2019 at 03:17:02PM -0700, Paul E. McKenney wrote:
> > > > On Thu, Jul 04, 2019 at 02:50:55PM -0400, Joel Fernandes wrote:
> > > [snip] 
> > > > > I did have a request, could you help me understand why is the grace period
> > > > > duration double that of my busy wait time? You mentioned this has something
> > > > > to do with the thread not waking up before another GP is started. But I did
> > > > > not follow this. Thanks a lot!!
> > > > 
> > > > Let's look at a normal grace period, and assume the ->gp_seq grace-period
> > > > sequence counter is initially 0x1, so that a grace period is already
> > > > in progress (keep in mind that the low-order two bits of ->gp_seq are
> > > > the phase within the grace period and the rest of the bits are the
> > > > grace-period number, so we are in phase 1 of the grace period following
> > > > grace period #0).  This grace period was started via synchronize_rcu()
> > > > by Task A.
> > > > 
> > > > Then we have the following sequence of events:
> > > > 
> > > > o	Task B does call_rcu(), but is too late to use the already-started
> > > > 	grace period, so it needs another one.	The ->gp_seq_needed
> > > > 	counter thus becomes 0x8.
> > > > 
> > > > o	The current grace period completes, so that the ->gp_seq
> > > > 	counter becomes 0x4.  Callback invocation commences.
> > > > 
> > > > o	The grace-period kthread notices that ->gp_seq_needed is greater
> > > > 	than ->gp_seq, so it starts a new grace period.  The ->gp_seq
> > > > 	counter therefore becomes 0x5.
> > > > 
> > > > o	The callback for Task A's synchronize_rcu() is invoked, awakening
> > > > 	Task A.  This happens almost immediately after the new grace
> > > > 	period starts, but does definitely happen -after- that grace
> > > > 	period starts.
> > > 
> > > Yes, but at this point, we are still at the 1GP mark. But the distribution's
> > > median below shows a strong correlation with 2 preempt-disable durations and
> > > grace-period completion. For example, if the duration of the preempt
> > > disable/enable loop is 50ms, it strongly shows the writer's median time
> > > difference before and after the synchronize_rcu as 100ms, not really showing
> > > it is 60 ms or 75 ms or anything.
> > > 
> > > > 
> > > > o	Task A, being part of rcuperf, almost immediately does another
> > > > 	synchronize_rcu().  So ->gp_seq_needed becomes 0xc.
> > > 
> > > Yes, but before that another synchronize_rcu, it also gets the timestamp and
> > > does a diff between old/new timestamp and has captured the data, so at that
> > > point the data captured should only be for around 1GPs worth give or take.
> > 
> > But in this case, the "give or take" is almost a full grace period, for
> > a total of almost two grace periods.
> > 
> > > > If you play out the rest of this sequence, you should see how Task A
> > > > waits for almost two full grace periods.
> > > 
> > > I tried to play this out, still didn't get it :-|. Not that it is an issue
> > > per-se, but still would like to understand it better.
> > 
> > Let's try it a different way.  Let's skip the state portion of ->gp_seq
> > and just focus on the number of elapsed grace periods.
> > 
> > o	Let's start with RCU idle, having completed grace-period number
> > 	zero and not yet having started grace-period number 1.
> > 	Let's also abbreviate: GP0 and GP1.
> > 
> > o	Task A wants a grace period.  Because RCU is idle, Task A's
> > 	GP will complete at completion of GP1.
> > 
> > o	RCU starts GP1.
> > 
> > o	Almost immediately thereafter, Task B wants a grace period.
> > 	But it cannot use GP1, because GP1 has already started, even
> > 	if just barely, because some CPU or tasks might already have
> > 	reported a quiescent state for GP1.
> > 
> > 	So Task B's GP will not complete until the end of GP2.	And that
> > 	is almost two GPs worth of time from now.
> > 
> > And on a system where RCU is busy, Task B's experience is the common
> > case if it is in a tight loop doing synchronize_rcu().  Because RCU is
> > busy, by the end of each grace period, there will always be a request
> > for another grace period.  And therefore RCU's grace-period kthread
> > will immediately start a new grace period upon completion of each
> > grace period.  So the sequence of events from Task B's viewpoint will
> > be as follows:
> > 
> > o	Task B executes synchronize_rcu(), which requests a new grace
> > 	period.
> > 
> > o	This requested grace period ends, and another immediately starts.
> > 
> > o	Task B's RCU callback is invoked, which does a wakeup.
> > 
> > o	Task B executes another synchronize_rcu(), but just after a new
> > 	grace period has started.  Task B thus has to wait almost two
> > 	full grace periods.
> > 
> > This could be perceived as a problem, but my question back to you would
> > be "Why on earth are you invoking synchronize_rcu() in a tight loop???"
> > "Oh, because you are running rcuperf?  Well, that is why rcuperf uses
> > ftrace data!"
> > 
> > Which might be the point on which we are talking past each other.  You
> > might have been thinking of the latency of RCU's grace-period computation.
> > I was thinking of the length of time between the call to synchronize_rcu()
> > and the corresponding return.
> 
> This registered perfectly with me now, I was having a hard time visualizing
> but it makes sense to me now based on your description. Thanks a lot!
> 
> Ascii art:
> (Basically the synch rcu is done just after a slight offset)
> 
>                synchronize_rcu Wait     synchronize_rcu Wait 
>               |<-------------------->|  |--------------------|
>               v                      v  v                    v
> <----------> <----------> <----------> <---------> <--------->
>      GP            GP          GP         GP          GP
> 
> GP = long preempt disable duration

You got it!

							Thanx, Paul