Question about ->head field of rcu_segcblist

Question about ->head field of rcu_segcblist

[Joel Fernandes]

Hi Paul,

I was parsing the Data-Structures document and had a question about
the following "Important note" text.

Could it be clarified in the below text better why "remaining
callbacks are placed back on the RCU_DONE_TAIL segment", is a reason
for not depending on ->head for determining if no callbacks are
associated with the rcu_segcblist? If callbacks are added back to the
DONE_TAIL segment, then I would think rcu_head should be != NULL.
Infact the "rsclp->head = *rsclp->tails[RCU_DONE_TAIL];" in
rcu_segcblist_extract_done_cbs should set the ->head to  NULL if I
understand correctly.

Important note: It is the ->len field that determines whether or not
there are callbacks associated with this rcu_segcblist structure, not
the ->head pointer. The reason for this is that all the
ready-to-invoke callbacks (that is, those in the RCU_DONE_TAIL
segment) are extracted all at once at callback-invocation time. If
callback invocation must be postponed, for example, because a
high-priority process just woke up on this CPU, then the remaining
callbacks are placed back on the RCU_DONE_TAIL segment. Either way,
the ->len and ->len_lazy counts are adjusted after the corresponding
callbacks have been invoked, and so again it is the ->lencount that
accurately reflects whether or not there are callbacks associated with
this rcu_segcblist structure. Of course, off-CPU sampling of the ->len
count requires the use of appropriate synchronization, for example,
memory barriers. This synchronization can be a bit subtle,
particularly in the case of rcu_barrier().

Thanks!

 - Joel

Re: Question about ->head field of rcu_segcblist

[Joel Fernandes]

On Sun, Sep 23, 2018 at 7:30 PM Joel Fernandes <joel@joelfernandes.org> wrote:
>
> Hi Paul,
>
> I was parsing the Data-Structures document and had a question about
> the following "Important note" text.
>
> Could it be clarified in the below text better why "remaining
> callbacks are placed back on the RCU_DONE_TAIL segment", is a reason
> for not depending on ->head for determining if no callbacks are
> associated with the rcu_segcblist? If callbacks are added back to the
> DONE_TAIL segment, then I would think rcu_head should be != NULL.
> Infact the "rsclp->head = *rsclp->tails[RCU_DONE_TAIL];" in
> rcu_segcblist_extract_done_cbs should set the ->head to  NULL if I
> understand correctly.

Just to clarify, I meant set to NULL assuming all cbs were done
waiting and ready to be invoked.

Re: Question about ->head field of rcu_segcblist

[Paul E. McKenney]

On Sun, Sep 23, 2018 at 07:30:30PM -0400, Joel Fernandes wrote:
> Hi Paul,
> 
> I was parsing the Data-Structures document and had a question about
> the following "Important note" text.
> 
> Could it be clarified in the below text better why "remaining
> callbacks are placed back on the RCU_DONE_TAIL segment", is a reason
> for not depending on ->head for determining if no callbacks are
> associated with the rcu_segcblist? If callbacks are added back to the
> DONE_TAIL segment, then I would think rcu_head should be != NULL.
> Infact the "rsclp->head = *rsclp->tails[RCU_DONE_TAIL];" in
> rcu_segcblist_extract_done_cbs should set the ->head to  NULL if I
> understand correctly.

The rcu_segcblist_extract_done_cbs() function will set rsclp->head
to NULL only if there were no non-done callbacks on the rsclp list.
Otherwise, if there are non-done callbacks, then rsclp->head will
be set to the first non-done callback.

Either way, the problem is that the done callbacks can be removed
and re-added, but the count is not adjusted until the re-add.  So
you have to look at the count to see if there are callbacks.

Testing rsclp->head fails because it can be temporarily NULL, even
though there are callbacks hanging off of a pointer in rcu_do_batch()'s
stack frame.

Or am I misunderstanding your question?

							Thanx, Paul

> Important note: It is the ->len field that determines whether or not
> there are callbacks associated with this rcu_segcblist structure, not
> the ->head pointer. The reason for this is that all the
> ready-to-invoke callbacks (that is, those in the RCU_DONE_TAIL
> segment) are extracted all at once at callback-invocation time. If
> callback invocation must be postponed, for example, because a
> high-priority process just woke up on this CPU, then the remaining
> callbacks are placed back on the RCU_DONE_TAIL segment. Either way,
> the ->len and ->len_lazy counts are adjusted after the corresponding
> callbacks have been invoked, and so again it is the ->lencount that
> accurately reflects whether or not there are callbacks associated with
> this rcu_segcblist structure. Of course, off-CPU sampling of the ->len
> count requires the use of appropriate synchronization, for example,
> memory barriers. This synchronization can be a bit subtle,
> particularly in the case of rcu_barrier().
> 
> Thanks!
> 
>  - Joel
> 

Re: Question about ->head field of rcu_segcblist

[Paul E. McKenney]

On Sun, Sep 23, 2018 at 07:31:37PM -0400, Joel Fernandes wrote:
> On Sun, Sep 23, 2018 at 7:30 PM Joel Fernandes <joel@joelfernandes.org> wrote:
> >
> > Hi Paul,
> >
> > I was parsing the Data-Structures document and had a question about
> > the following "Important note" text.
> >
> > Could it be clarified in the below text better why "remaining
> > callbacks are placed back on the RCU_DONE_TAIL segment", is a reason
> > for not depending on ->head for determining if no callbacks are
> > associated with the rcu_segcblist? If callbacks are added back to the
> > DONE_TAIL segment, then I would think rcu_head should be != NULL.
> > Infact the "rsclp->head = *rsclp->tails[RCU_DONE_TAIL];" in
> > rcu_segcblist_extract_done_cbs should set the ->head to  NULL if I
> > understand correctly.
> 
> Just to clarify, I meant set to NULL assuming all cbs were done
> waiting and ready to be invoked.

Ah, good, then that is correct.  But even then, being NULL doesn't mean
no callbacks because they might be temporarily held by rcu_do_batch().

							Thanx, Paul

Re: Question about ->head field of rcu_segcblist

[Joel Fernandes]

On Sun, Sep 23, 2018 at 7:54 PM Paul E. McKenney <paulmck@linux.ibm.com> wrote:
>
> On Sun, Sep 23, 2018 at 07:30:30PM -0400, Joel Fernandes wrote:
> > Hi Paul,
> >
> > I was parsing the Data-Structures document and had a question about
> > the following "Important note" text.
> >
> > Could it be clarified in the below text better why "remaining
> > callbacks are placed back on the RCU_DONE_TAIL segment", is a reason
> > for not depending on ->head for determining if no callbacks are
> > associated with the rcu_segcblist? If callbacks are added back to the
> > DONE_TAIL segment, then I would think rcu_head should be != NULL.
> > Infact the "rsclp->head = *rsclp->tails[RCU_DONE_TAIL];" in
> > rcu_segcblist_extract_done_cbs should set the ->head to  NULL if I
> > understand correctly.
>
> The rcu_segcblist_extract_done_cbs() function will set rsclp->head
> to NULL only if there were no non-done callbacks on the rsclp list.
> Otherwise, if there are non-done callbacks, then rsclp->head will
> be set to the first non-done callback.
>
> Either way, the problem is that the done callbacks can be removed
> and re-added, but the count is not adjusted until the re-add.  So
> you have to look at the count to see if there are callbacks.
>
> Testing rsclp->head fails because it can be temporarily NULL, even
> though there are callbacks hanging off of a pointer in rcu_do_batch()'s
> stack frame.
>
> Or am I misunderstanding your question?

Thanks yes that clears it up, I see what you mean that that ->head
field is temporarily volatile and really the ->len tells the real
story :-)

- Joel