From mboxrd@z Thu Jan 1 00:00:00 1970 From: Waiman Long Subject: Re: [PATCH v9 04/19] qspinlock: Extract out the exchange of tail code word Date: Fri, 18 Apr 2014 14:13:57 -0400 Message-ID: <53516B65.1020600@hp.com> References: <1397747051-15401-1-git-send-email-Waiman.Long@hp.com> <1397747051-15401-5-git-send-email-Waiman.Long@hp.com> <20140417154902.GO11096@twins.programming.kicks-ass.net> <53504771.9070102@hp.com> <20140418081517.GY11096@twins.programming.kicks-ass.net> <535161BF.90405@hp.com> <20140418175339.GZ11182@twins.programming.kicks-ass.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20140418175339.GZ11182@twins.programming.kicks-ass.net> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Peter Zijlstra Cc: linux-arch@vger.kernel.org, Rik van Riel , Raghavendra K T , Gleb Natapov , kvm@vger.kernel.org, Konrad Rzeszutek Wilk , Scott J Norton , x86@kernel.org, Paolo Bonzini , linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Ingo Molnar , Chegu Vinod , David Vrabel , "H. Peter Anvin" , xen-devel@lists.xenproject.org, Thomas Gleixner , "Paul E. McKenney" , Linus Torvalds , Oleg Nesterov List-Id: linux-arch.vger.kernel.org On 04/18/2014 01:53 PM, Peter Zijlstra wrote: > On Fri, Apr 18, 2014 at 01:32:47PM -0400, Waiman Long wrote: >> On 04/18/2014 04:15 AM, Peter Zijlstra wrote: >>> On Thu, Apr 17, 2014 at 05:28:17PM -0400, Waiman Long wrote: >>>> On 04/17/2014 11:49 AM, Peter Zijlstra wrote: >>>>> On Thu, Apr 17, 2014 at 11:03:56AM -0400, Waiman Long wrote: >>>>>> @@ -192,36 +220,25 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, u32 val) >>>>>> node->next = NULL; >>>>>> >>>>>> /* >>>>>> + * We touched a (possibly) cold cacheline; attempt the trylock once >>>>>> + * more in the hope someone let go while we weren't watching as long >>>>>> + * as no one was queuing. >>>>>> */ >>>>>> + if (!(val& _Q_TAIL_MASK)&& queue_spin_trylock(lock)) >>>>>> + goto release; >>>>> But you just did a potentially very expensive op; @val isn't >>>>> representative anymore! >>>> That is not true. I pass in a pointer to val to trylock_pending() (the >>>> pointer thing) so that it will store the latest value that it reads from the >>>> lock back into val. I did miss one in the PV qspinlock exit loop. I will add >>>> it back when I do the next version. >>> But you did that read _before_ you touched a cold cacheline, that's 100s >>> of cycles. Whatever value you read back then is now complete nonsense. >> For spin_lock(), the lock cacheline is touched by a cmpxchg(). It can takes >> 100s of cycles whether it is hot or cold. > Its not the lock cacheline, you just touched the per-cpu node cacheline > for the first time, setting up the node. > Thank for the clarification, now I know what you mean. -Longman From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from g4t3427.houston.hp.com ([15.201.208.55]:27557 "EHLO g4t3427.houston.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752305AbaDRSOP (ORCPT ); Fri, 18 Apr 2014 14:14:15 -0400 Message-ID: <53516B65.1020600@hp.com> Date: Fri, 18 Apr 2014 14:13:57 -0400 From: Waiman Long MIME-Version: 1.0 Subject: Re: [PATCH v9 04/19] qspinlock: Extract out the exchange of tail code word References: <1397747051-15401-1-git-send-email-Waiman.Long@hp.com> <1397747051-15401-5-git-send-email-Waiman.Long@hp.com> <20140417154902.GO11096@twins.programming.kicks-ass.net> <53504771.9070102@hp.com> <20140418081517.GY11096@twins.programming.kicks-ass.net> <535161BF.90405@hp.com> <20140418175339.GZ11182@twins.programming.kicks-ass.net> In-Reply-To: <20140418175339.GZ11182@twins.programming.kicks-ass.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-arch-owner@vger.kernel.org List-ID: To: Peter Zijlstra Cc: Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , linux-arch@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, xen-devel@lists.xenproject.org, kvm@vger.kernel.org, Paolo Bonzini , Konrad Rzeszutek Wilk , "Paul E. McKenney" , Rik van Riel , Linus Torvalds , Raghavendra K T , David Vrabel , Oleg Nesterov , Gleb Natapov , Scott J Norton , Chegu Vinod Message-ID: <20140418181357.lOC4HPQNEHGoA7vQvw_IhGUSVEVAWGl5L_2lMe_Byw0@z> On 04/18/2014 01:53 PM, Peter Zijlstra wrote: > On Fri, Apr 18, 2014 at 01:32:47PM -0400, Waiman Long wrote: >> On 04/18/2014 04:15 AM, Peter Zijlstra wrote: >>> On Thu, Apr 17, 2014 at 05:28:17PM -0400, Waiman Long wrote: >>>> On 04/17/2014 11:49 AM, Peter Zijlstra wrote: >>>>> On Thu, Apr 17, 2014 at 11:03:56AM -0400, Waiman Long wrote: >>>>>> @@ -192,36 +220,25 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, u32 val) >>>>>> node->next = NULL; >>>>>> >>>>>> /* >>>>>> + * We touched a (possibly) cold cacheline; attempt the trylock once >>>>>> + * more in the hope someone let go while we weren't watching as long >>>>>> + * as no one was queuing. >>>>>> */ >>>>>> + if (!(val& _Q_TAIL_MASK)&& queue_spin_trylock(lock)) >>>>>> + goto release; >>>>> But you just did a potentially very expensive op; @val isn't >>>>> representative anymore! >>>> That is not true. I pass in a pointer to val to trylock_pending() (the >>>> pointer thing) so that it will store the latest value that it reads from the >>>> lock back into val. I did miss one in the PV qspinlock exit loop. I will add >>>> it back when I do the next version. >>> But you did that read _before_ you touched a cold cacheline, that's 100s >>> of cycles. Whatever value you read back then is now complete nonsense. >> For spin_lock(), the lock cacheline is touched by a cmpxchg(). It can takes >> 100s of cycles whether it is hot or cold. > Its not the lock cacheline, you just touched the per-cpu node cacheline > for the first time, setting up the node. > Thank for the clarification, now I know what you mean. -Longman