Re: Userspace RCU+rtth hack (was Re: [patch 3/3] radix-tree: RCU lockless readside)

Nick Piggin
2006-06-24T10:20:31Z

On Thu, Jun 22, 2006 at 01:25:53PM -0700, Paul E. McKenney wrote:
> On Thu, Jun 22, 2006 at 08:23:43PM +0200, Nick Piggin wrote:
> >
> > Just out of interest, attached is my userspace RCU implementation
> > and RCU radix-tree concurrent tests for Andrew Morton's radix-tree
> > test harness.
> >
> > The RCU implementation is only 100 lines. Awful performance, of
> > course, but I've stretched the rcu_read_lock/unlock over large
> > periods so that we can get full concurrency at the cost of a
> > bit of memory build up. And it still seems to catch use-after
> > RCU-freed errors pretty easily.
>
> Interesting approach! One caution -- this approach can result in
> RCU callbacks being invoked in the context of either call_rcu() or
> rcu_read_unlock(). In some legitimate uses of RCU, this can result
> in deadlock. See Documentation/RCU/UP.txt for more info.
>
> One solution is to have some other context (perhaps just a separate
> pthread, given that performance is not critical) to invoke the callbacks.

Ah that's true. And I knew that, but it didn't occur to me ;)

>
> Another user-level RCU implementation is available here:
>
> http://www.cs.toronto.edu/~tomhart/perflab/ipdps06.tgz

Interesting, thanks.

> I have a few user-mode implementations myself, but the lawyers won't
> let me release them. :-(

I imagine they're quite a bit faster than my quick hack, too ;)

>
> > Question - our kernel's call_rcu implies a smp_wmb, right? Because
> > that did catch me out initially, because I initially had no barrier
> > to prevent the freeing of the object becoming visible before
> > removal of its last reference becoming visible (fixed by adding
> > smp_wmb() in my call_rcu).
>
> No and yes... The kernel's call_rcu() itself does not have an smp_wmb(),
> but the Classic RCU grace-period mechanism forces a memory barrier on each
> CPU as part of grace-period detection -- which is why rcu_read_lock()
> and rcu_read_unlock() don't need memory barriers. Looks like your need
> for an smp_wmb() in call_rcu() itself is due to the fact that you can
> execute callbacks in the context of the call_rcu() itself.

That makes sense. Thanks for clearing that up.