From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S941388AbcKQLD1 (ORCPT <rfc822;w@1wt.eu>);
        Thu, 17 Nov 2016 06:03:27 -0500
Received: from mail.linuxfoundation.org ([140.211.169.12]:53416 "EHLO
        mail.linuxfoundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1753881AbcKQLDX (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Thu, 17 Nov 2016 06:03:23 -0500
Date: Thu, 17 Nov 2016 12:03:33 +0100
From: Greg KH <gregkh@linuxfoundation.org>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>, keescook@chromium.org,
        will.deacon@arm.com, elena.reshetova@intel.com, arnd@arndb.de,
        tglx@linutronix.de, mingo@kernel.org, hpa@zytor.com, dave@progbits.org,
        linux-kernel@vger.kernel.org
Subject: Re: [RFC][PATCH 7/7] kref: Implement using refcount_t
Message-ID: <20161117110333.GA19999@kroah.com>
References: <20161114173946.501528675@infradead.org>
 <20161114174446.832175072@infradead.org>
 <20161115123337.GD12110@tardis.cn.ibm.com>
 <20161115130154.GX3117@twins.programming.kicks-ass.net>
 <20161115141909.GJ27541@tardis.cn.ibm.com>
 <20161117092800.GD3142@twins.programming.kicks-ass.net>
 <20161117094851.GA5227@tardis.cn.ibm.com>
 <20161117102959.GH3142@twins.programming.kicks-ass.net>
 <20161117103927.GU3157@twins.programming.kicks-ass.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20161117103927.GU3157@twins.programming.kicks-ass.net>
User-Agent: Mutt/1.7.1 (2016-10-04)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Nov 17, 2016 at 11:39:27AM +0100, Peter Zijlstra wrote:
> On Thu, Nov 17, 2016 at 11:29:59AM +0100, Peter Zijlstra wrote:
> > On Thu, Nov 17, 2016 at 05:48:51PM +0800, Boqun Feng wrote:
> > > > > But as I said, we actually only need the pairing of orderings:
> > > > > 
> > > > > 1) load part of cmpxchg -> free() 
> > > > > 2) object accesses -> store part of cmpxchg
> > > > > 
> > > > > Ordering #1 can be achieved via control dependency as you pointed out
> > > > > that free()s very much includes stores. And ordering #2 can be achieved
> > > > > with RELEASE.
> > > > > 
> > > > > So the code is right, I just thought the comment may be misleading. The
> > > > > reason we use cmpxchg_release() is just for achieving ordering #2, and
> > > > > not to order "prior loads and stores" with "a subsequent free".
> > > > > 
> > > > > Am I missing some subtle orderings here?
> > > > 
> > > > I would want to further quality 1), it must be no earlier than the load
> > > > of the last / successful ll/sc round.
> > > > 
> > > 
> > > Great, that's more accurate!
> > > 
> > > > At that point we're guaranteed a reference count of 1 that _will_ drop
> > > > to 0, and thus nobody else (should) reference that memory anymore.
> > > > 
> > > > If we agree on this, I'll update the comment :-) Will, do you too agree?
> > > 
> > > Agreed ;-)
> > > 
> > > Control dependencies and RELEASE are totally enough for the internal
> > > correctness of refcount_t along with its interactivity with free().
> > > People better not reply order guarantees other than this ;-)
> > 
> > Hurm.. let me ruin my own argument.
> > 
> > Since the free() stores could leak upwards until that ll, and object
> > stores can be delayed until the sc, we still have a problem. Just not
> > with the thread that free()s or any other thread that knew about the
> > object.
> > 
> > The problem comes from any other thread doing an allocation, since its
> > possible to observe the memory as freed while there are stores pending
> > to it, we can have those delayed stores trample on our freshly allocated
> > and initialized object.
> > 
> > The stores must really not be before the SC, so I fear we must either
> > add an smp_wmb() after the release, or punt and use the fully ordered
> > cmpxchg().
> 
> And let me note here that RCU users can use a fully relaxed put, because
> call_rcu() guarantees a grace-period between the call_rcu and the
> free(), which in turn provides a full memory barrier that orders things.
> 
> We could actually expose that to driver writers by doing something like:
> 
> 	kref_put_rcu(struct kref *kref, struct rcu_head *head, rcu_callback_t func)
> 	{
> 		if (refcount_dec_and_test_relaxed(&kref->refcount))
> 			call_rcu(head, func);
> 	}
> 
> Do we want to go there?

No, please no...