Re: [PATCH] generic dynamic per cpu refcounting

[Kent Overstreet <koverstreet@google.com>]

On Mon, Jan 28, 2013 at 01:28:14PM -0800, Tejun Heo wrote:
> On Mon, Jan 28, 2013 at 01:24:07PM -0800, Kent Overstreet wrote:
> > > 	set dying;
> > > 	synchronize_sched();
> > > 	collect percpu refs into global atomic_t;
> > > 	put the base ref;
> > 
> > After you set state := dying, percpu_ref_put() decrements the atomic_t,
> > but it can't check if it's 0 yet because the thread that's collecting
> > the percpu refs might not be done yet.
> > 
> > So percpu_ref_put can't check for ref == 0 until after state == dead.
> > But the put in your example might have made ref 0. When did you set
> > state to dead?
> 
> But at that point, the operation is already global, so there gotta be
> a lighter way to synchronize stuff than going through full grace
> period.  ie. You can add a bias value before marking dead so that the
> counter never reaches zero before all percpu counters are collected
> and then unbias it right before putting the base ref, that way the
> only way you can hit zero ref is all refs are actually zero.

Ahh. Bias value sounds... hacky (i.e. harder to convince myself it's
correct) but I see what you're getting at.

Something to consider is wrapping; after we set state to dying but
before we've collected the percpu counters, the atomic counter may be
negative.

But since the atomic counter is 64 bits, we can use 1 << 32 for the bias
value (and just include that when we first initialize it). Which makes
me feel like it's less of a hack too.

I'll have to think about it some more but seems like it ought t owork...