Re: [PATCH] generic dynamic per cpu refcounting

[Kent Overstreet <koverstreet@google.com>]

On Mon, Jan 28, 2013 at 10:27:37AM -0800, Tejun Heo wrote:
> Hello, guys.
> 
> On Mon, Jan 28, 2013 at 10:15:28AM -0800, Kent Overstreet wrote:
> > > 	percpu_ref_kill();
> > > 	put_and_dsetroy();
> > > 
> > > And this can race with another holder which drops the last reference,
> > > its put_and_dsetroy() can see PCPU_REF_DYING and return false.
> > > 
> > > Or I misunderstood the code/interface?
> > 
> > Nope, nailed it :) That should _definitely_ be in the documentation.
> 
> Can we just combine kill initiation and base ref put and make that the
> responsibility of the owner?  Extra features on basic constructs may
> seem good for certain use cases but tend to bring more confusion than
> good in the long run.  If a user needs to synchronize among multiple
> killers, let the user deal with the issue.

Don't follow...

Something I forgot to mention in the last mail though is that often the
caller will need its own synchronize_rcu()/call_rcu() -
percpu_ref_kill() corresponds to when you make the object unavailable
(i.e. deleting it from the rcu protected hash table in aio) and you need
a synchronize_rcu() before you drop your initial ref.

So letting the caller do it means the caller can merge the two
synchronize_rcu()s.

> 
> > Actually - I think it'd be better to have the default percpu_ref_kill()
> > do the second synchronize_rcu(), and have an unsafe version that skips
> > it.
> 
> Note that synchronize_rcu/sched() can be very slow and cause problems
> in paths which are frequently traveled and visible to userland.  It's
> fine for things like module destruction but can be a problem even
> during device destruction - blkcg had synchronize_rcu() in
> request_queue destruction which led to huge latencies during boot
> because SCSI wants to create and then destroy request_queues for all
> possible LUNs on certain configurations.  So, if you put
> synchronize_rcu/sched() in percpu_ref_kill(), that better not be used
> from e.g. close(2).

Yeah. It'd be really nice if it was doable without synchronize_rcu(),
but it'd definitely make get/put heavier.

Though, re. close() - considering we only need a synchronize_rcu() if
the ref was in percpu mode, I wonder if that would be a dealbreaker. I
have no clue myself.

Getting rid of synchronize_rcu would basically require turning get and
put into cmpxchg() loops - even in the percpu fastpath. However, percpu
mode would still be getting rid of the shared cacheline contention, we'd
just be adding another branch that can be safely marked unlikely() - and
my current version has one of those already, so two branches instead of
one in the fast path.

I suppose I should give it a shot.

As long as I'm going down that route I could probably make the bare non
percpu ref 8 bytes instead of 16, too...