Re: 8-CPU (SMP) #s for lockfree rtcache

Dipankar Sarma
2002-05-28T17:06:45Z

On Tue, May 28, 2002 at 09:55:35PM +0530, Dipankar Sarma wrote:
> Hi Robert,
>
> On Tue, May 28, 2002 at 08:49:58AM -0700, Robert Love wrote:
> >
> > > Well, the last time RCU was discussed, Linus said that he would
> > > like to see someplace where RCU clearly helps.
> >
> > I agree the numbers posted are nice, but I remain skeptical like Linus.
> > Sure, the locking overhead is nearly gone in the profiled function where
> > RCU is used. But the overhead has just been _moved_ to wherever the RCU
> > work is now done. Any benchmark needs to include the damage done there,
> > too.
>
> Have you looked at the rt_rcu patch ? Where do you think there
> is overhead compared to what route cache alread does ? In my
> profiles, rcu routines and kernel mechanisms that it uses
> don't show high up. If you have any suggestions, then I can
> do an investigation.

Hi Robert,

While we are at it, I think this is good point to analyze.
So here is an brief analysis of rt_rcu patch from the overhead
standpoint -

1. Read side has no overhead, we just don't take the per-bucket lock.
2. For just the route cache portion of code, RCU comes into picture
only when dst entries are deleted. This however has two issues -
a> expiry of dst entries is checked through a non-frequent
timer b>lease for recently used dst entries are extended.
So we don't do frequent RCU based deletion of dst entries.
Periodically a set of dst entries expire and instead of
freeing them immediately, we just put them in RCU queue(s)
for freeing after the grace period (call_rcu() in rt_free()).

Coming to the RCU mechanism -

1. Grace period detection : Different RCU algorithms do it
differently, however if there is no RCU pending *nothing*
is done regarding this. One rcu implementation uses
a 10ms timer to check for grace period completion and another
rcu_poll uses a repeating tasklet to poll for it. The grace period
detection is based on a per-cpu context switch counter. I have not seen
signficant profile counts for grace period detection scheme, but
nevertheless I will put up the profile counts for Dave's test
at the LSE website.

2. Actual update : RCU processes the batched update callbacks from tasklet
context. The rt_rcu callbacks don't do anything other than
call dst_free(), which would have been called by non-RCU
code under lock in any case. I am not sure doing this from
tasklet context adds any overhead and I suspect that it doesn't.

Comments/suggestions ?

Thanks
--
Dipankar Sarma <dipankar@in.ibm.com> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.