From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754403AbbHCRAq (ORCPT ); Mon, 3 Aug 2015 13:00:46 -0400 Received: from terminus.zytor.com ([198.137.202.10]:40111 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754254AbbHCRAo (ORCPT ); Mon, 3 Aug 2015 13:00:44 -0400 Date: Mon, 3 Aug 2015 09:59:46 -0700 From: tip-bot for Will Deacon Message-ID: Cc: tglx@linutronix.de, paulmck@linux.vnet.ibm.com, will.deacon@arm.com, akpm@linux-foundation.org, hpa@zytor.com, torvalds@linux-foundation.org, Waiman.Long@hp.com, peterz@infradead.org, mingo@kernel.org, Steve.Capper@arm.com, linux-kernel@vger.kernel.org Reply-To: mingo@kernel.org, peterz@infradead.org, Steve.Capper@arm.com, linux-kernel@vger.kernel.org, tglx@linutronix.de, paulmck@linux.vnet.ibm.com, will.deacon@arm.com, akpm@linux-foundation.org, hpa@zytor.com, Waiman.Long@hp.com, torvalds@linux-foundation.org In-Reply-To: <20150713155830.GL2632@arm.com> References: <20150713155830.GL2632@arm.com> To: linux-tip-commits@vger.kernel.org Subject: [tip:locking/core] locking/pvqspinlock: Order pv_unhash() after cmpxchg() on unlock slowpath Git-Commit-ID: 3b3fdf10a8add87ef0050138d51bfee9ab4983df X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: 3b3fdf10a8add87ef0050138d51bfee9ab4983df Gitweb: http://git.kernel.org/tip/3b3fdf10a8add87ef0050138d51bfee9ab4983df Author: Will Deacon AuthorDate: Mon, 13 Jul 2015 16:58:30 +0100 Committer: Ingo Molnar CommitDate: Mon, 3 Aug 2015 10:57:09 +0200 locking/pvqspinlock: Order pv_unhash() after cmpxchg() on unlock slowpath When we unlock in __pv_queued_spin_unlock(), a failed cmpxchg() on the lock value indicates that we need to take the slow-path and unhash the corresponding node blocked on the lock. Since a failed cmpxchg() does not provide any memory-ordering guarantees, it is possible that the node data could be read before the cmpxchg() on weakly-ordered architectures and therefore return a stale value, leading to hash corruption and/or a BUG(). This patch adds an smb_rmb() following the failed cmpxchg operation, so that the unhashing is ordered after the lock has been checked. Reported-by: Peter Zijlstra Signed-off-by: Will Deacon [ Added more comments] Signed-off-by: Peter Zijlstra (Intel) Acked-by: Waiman Long Cc: Andrew Morton Cc: Linus Torvalds Cc: Paul E. McKenney Cc: Paul McKenney Cc: Steve Capper Cc: Thomas Gleixner Link: http://lkml.kernel.org/r/20150713155830.GL2632@arm.com Signed-off-by: Ingo Molnar --- kernel/locking/qspinlock_paravirt.h | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/kernel/locking/qspinlock_paravirt.h b/kernel/locking/qspinlock_paravirt.h index 489a878..ab8b1bb 100644 --- a/kernel/locking/qspinlock_paravirt.h +++ b/kernel/locking/qspinlock_paravirt.h @@ -244,13 +244,17 @@ static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node) if (!lp) { /* ONCE */ lp = pv_hash(lock, pn); /* - * lp must be set before setting _Q_SLOW_VAL + * We must hash before setting _Q_SLOW_VAL, such that + * when we observe _Q_SLOW_VAL in __pv_queued_spin_unlock() + * we'll be sure to be able to observe our hash entry. * - * [S] lp = lock [RmW] l = l->locked = 0 - * MB MB - * [S] l->locked = _Q_SLOW_VAL [L] lp + * [S] pn->state + * [S] [Rmw] l->locked == _Q_SLOW_VAL + * MB RMB + * [RmW] l->locked = _Q_SLOW_VAL [L] + * [L] pn->state * - * Matches the cmpxchg() in __pv_queued_spin_unlock(). + * Matches the smp_rmb() in __pv_queued_spin_unlock(). */ if (!cmpxchg(&l->locked, _Q_LOCKED_VAL, _Q_SLOW_VAL)) { /* @@ -306,6 +310,15 @@ __visible void __pv_queued_spin_unlock(struct qspinlock *lock) } /* + * A failed cmpxchg doesn't provide any memory-ordering guarantees, + * so we need a barrier to order the read of the node data in + * pv_unhash *after* we've read the lock being _Q_SLOW_VAL. + * + * Matches the cmpxchg() in pv_wait_head() setting _Q_SLOW_VAL. + */ + smp_rmb(); + + /* * Since the above failed to release, this must be the SLOW path. * Therefore start by looking up the blocked node and unhashing it. */