From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755174AbbBRRYi (ORCPT ); Wed, 18 Feb 2015 12:24:38 -0500 Received: from terminus.zytor.com ([198.137.202.10]:39265 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752592AbbBRRMk (ORCPT ); Wed, 18 Feb 2015 12:12:40 -0500 Date: Wed, 18 Feb 2015 09:11:56 -0800 From: tip-bot for Davidlohr Bueso Message-ID: Cc: paulmck@linux.vnet.ibm.com, dave@stgolabs.net, mingo@kernel.org, dbueso@suse.de, torvalds@linux-foundation.org, tim.c.chen@linux.intel.com, linux-kernel@vger.kernel.org, hpa@zytor.com, peterz@infradead.org, tglx@linutronix.de, jason.low2@hp.com, walken@google.com Reply-To: tim.c.chen@linux.intel.com, paulmck@linux.vnet.ibm.com, dave@stgolabs.net, dbueso@suse.de, mingo@kernel.org, torvalds@linux-foundation.org, tglx@linutronix.de, walken@google.com, jason.low2@hp.com, linux-kernel@vger.kernel.org, hpa@zytor.com, peterz@infradead.org In-Reply-To: <1422609267-15102-4-git-send-email-dave@stgolabs.net> References: <1422609267-15102-4-git-send-email-dave@stgolabs.net> To: linux-tip-commits@vger.kernel.org Subject: [tip:locking/core] locking/rwsem: Set lock ownership ASAP Git-Commit-ID: 7a215f89a0335582292ec6f3edaa3abd570da75a X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: 7a215f89a0335582292ec6f3edaa3abd570da75a Gitweb: http://git.kernel.org/tip/7a215f89a0335582292ec6f3edaa3abd570da75a Author: Davidlohr Bueso AuthorDate: Fri, 30 Jan 2015 01:14:25 -0800 Committer: Ingo Molnar CommitDate: Wed, 18 Feb 2015 16:57:13 +0100 locking/rwsem: Set lock ownership ASAP In order to optimize the spinning step, we need to set the lock owner as soon as the lock is acquired; after a successful counter cmpxchg operation, that is. This is particularly useful as rwsems need to set the owner to nil for readers, so there is a greater chance of falling out of the spinning. Currently we only set the owner much later in the game, in the more generic level -- latency can be specially bad when waiting for a node->next pointer when releasing the osq in up_write calls. As such, update the owner inside rwsem_try_write_lock (when the lock is obtained after blocking) and rwsem_try_write_lock_unqueued (when the lock is obtained while spinning). This requires creating a new internal rwsem.h header to share the owner related calls. Also cleanup some headers for mutex and rwsem. Suggested-by: Peter Zijlstra Signed-off-by: Davidlohr Bueso Signed-off-by: Peter Zijlstra (Intel) Cc: Jason Low Cc: Linus Torvalds Cc: Michel Lespinasse Cc: Paul E. McKenney Cc: Tim Chen Link: http://lkml.kernel.org/r/1422609267-15102-4-git-send-email-dave@stgolabs.net Signed-off-by: Ingo Molnar --- kernel/locking/mutex.c | 2 +- kernel/locking/rwsem-xadd.c | 8 ++++++-- kernel/locking/rwsem.c | 22 +--------------------- kernel/locking/rwsem.h | 20 ++++++++++++++++++++ 4 files changed, 28 insertions(+), 24 deletions(-) diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 59cd6c3..43bf25e 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -25,7 +25,7 @@ #include #include #include -#include "mcs_spinlock.h" +#include /* * In the DEBUG case we are using the "NULL fastpath" for mutexes, diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c index 82aba46..07713e5 100644 --- a/kernel/locking/rwsem-xadd.c +++ b/kernel/locking/rwsem-xadd.c @@ -14,8 +14,9 @@ #include #include #include +#include -#include "mcs_spinlock.h" +#include "rwsem.h" /* * Guide to the rw_semaphore's count field for common values. @@ -265,6 +266,7 @@ static inline bool rwsem_try_write_lock(long count, struct rw_semaphore *sem) RWSEM_ACTIVE_WRITE_BIAS) == RWSEM_WAITING_BIAS) { if (!list_is_singular(&sem->wait_list)) rwsem_atomic_update(RWSEM_WAITING_BIAS, sem); + rwsem_set_owner(sem); return true; } @@ -284,8 +286,10 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem) return false; old = cmpxchg(&sem->count, count, count + RWSEM_ACTIVE_WRITE_BIAS); - if (old == count) + if (old == count) { + rwsem_set_owner(sem); return true; + } count = old; } diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c index e2d3bc7..205be0c 100644 --- a/kernel/locking/rwsem.c +++ b/kernel/locking/rwsem.c @@ -9,29 +9,9 @@ #include #include #include - #include -#ifdef CONFIG_RWSEM_SPIN_ON_OWNER -static inline void rwsem_set_owner(struct rw_semaphore *sem) -{ - sem->owner = current; -} - -static inline void rwsem_clear_owner(struct rw_semaphore *sem) -{ - sem->owner = NULL; -} - -#else -static inline void rwsem_set_owner(struct rw_semaphore *sem) -{ -} - -static inline void rwsem_clear_owner(struct rw_semaphore *sem) -{ -} -#endif +#include "rwsem.h" /* * lock for reading diff --git a/kernel/locking/rwsem.h b/kernel/locking/rwsem.h new file mode 100644 index 0000000..870ed9a --- /dev/null +++ b/kernel/locking/rwsem.h @@ -0,0 +1,20 @@ +#ifdef CONFIG_RWSEM_SPIN_ON_OWNER +static inline void rwsem_set_owner(struct rw_semaphore *sem) +{ + sem->owner = current; +} + +static inline void rwsem_clear_owner(struct rw_semaphore *sem) +{ + sem->owner = NULL; +} + +#else +static inline void rwsem_set_owner(struct rw_semaphore *sem) +{ +} + +static inline void rwsem_clear_owner(struct rw_semaphore *sem) +{ +} +#endif